From 8d3328b6208747dcab74157a5e35f8c3cf2d1344 Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 24 Sep 2026 15:53:09 +0200 Subject: [PATCH] Record the availability exercise and finish the tenancy posture plan. Assistant: grok Assistant-Session: 01a0d25d-d358-7e13-b84a-d007fbb7e34f --- docs/availability-recovery.md | 42 ++++++------- .../AUDIT-WP-0008-T07-v1-2026-09-24.md | 61 +++++++++++++++++++ tenancy.yaml | 25 +++++--- ...AUDIT-WP-0008-tenancy-posture-alignment.md | 12 +++- 4 files changed, 104 insertions(+), 36 deletions(-) create mode 100644 docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md diff --git a/docs/availability-recovery.md b/docs/availability-recovery.md index b201217..88b963a 100644 --- a/docs/availability-recovery.md +++ b/docs/availability-recovery.md @@ -1,25 +1,15 @@ # Availability and recovery -**Status:** V1 not yet evidenced. Dependency enumeration complete; exercise -pending a live window on railiance01. +**Status:** V1 evidenced 2026-09-24. Review by 2026-12-31. **Framework:** NetKingdom Tenancy Posture v0.1 (draft-8) §4.6, §13 V-row. -**Declared:** `tenancy.yaml` — `current.V: 0`, `target.V: 1`. +**Declared:** `tenancy.yaml` — `current.V: 1`, `target.V: 1`. -## Why V0 today +## Why this is V1 -`V0` is "no availability or recovery position; recovery is untested or depends -on improvisation". That is accurate. Nothing in this repo exercises recovery of -the complete audit path and records a measured recovery time. - -What exists is an *observation*, not an exercise: after the railiance01 node -reboot on 2026-08-16, `/readyz` failed for roughly 40 seconds before the pod -went Ready, recorded in `docs/operator-runbook.md`. That is useful and it is not -V1 evidence — it enumerated nothing, measured nothing deliberately, and happened -to us rather than being performed. - -Decision 4.6.1 is explicit that a replica count or a status page is not -evidence of a level. audit-core runs one replica; that fact argues for neither -V0 nor V1. Only the exercise settles it. +The 2026-08-16 node reboot was an observation, not an exercise. The 2026-09-24 +run was an exercise: the receiver was recreated, the platform-pg primary was +recreated, and the time to the next accepted event was recorded. A replica +count is still not the evidence. The evidence is the exercise. ## Critical dependency enumeration (Decision 4.6.1) @@ -83,10 +73,16 @@ it; E2 and E3 are where it would break. ## Results -*Not yet run.* Needs a live window on railiance01 and coordination with -`user-engine`, since E4 and E5 are visible to the sender. +Run 2026-09-24. Evidence: `docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md`. -On completion, record the measured recovery time per scenario here with dates, -raise `tenancy.yaml` `current.V` to 1, note the exercise as the evidence, and -set a review date — a recovery exercise from a year ago describes a system that -no longer exists. +| Scenario | Measured recovery | Pass | +|---|---|---| +| E1 receiver rollout | about 11s to the next accept | chain intact | +| E2 receiver pod delete | about 7s to the next accept | chain intact | +| E3 `platform-pg` primary recreate | 22s from pod termination to the next accept | chain intact; co-residents still running | +| E4 lease revocation | not run | not claimed | +| E5 node reboot | not run | not claimed | + +The binding number is E3. E4 and E5 stay out of this V1 claim: OpenBao being +down does not stop accepts until the current lease expires, and a node reboot +was not part of the approved window. Review by 2026-12-31. diff --git a/docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md b/docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md new file mode 100644 index 0000000..7f46239 --- /dev/null +++ b/docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md @@ -0,0 +1,61 @@ +# AUDIT-WP-0008-T07 availability exercise — 2026-09-24 + +Operation measured: `POST /v1/events` accepted into custody. +Authority: founder request in the custodian session, recorded as +`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. +Not performed: runtime-lease revocation, OpenBao shutdown, node reboot. + +The probe ran inside the receiver, used the already-mounted sender registry, +and printed only a timestamp and an HTTP class. No credential was printed. +Each successful probe is a synthetic `audit.recovery.synthetic` event from +`user-engine` for `tenant:platform`. + +## Receiver + +| Step | Wall clock (UTC) | Result | +|---|---|---| +| E1 last accept before rollout | 13:45:37 | 202 accepted | +| E1 `rollout restart` issued | 13:45:35 workstation | new pod scheduled | +| E1 probes | 13:45:36, 13:45:39 | exec failed, pod not usable | +| E1 next accept | 13:45:46 | 202 accepted (`audit-wp-0008-t07-e1-134541`) | +| E2 pod deleted | 13:46:12 | `audit-core-66767ddd58-mcxtm` | +| E2 probe | 13:46:13 | exec failed | +| E2 next accept | 13:46:19 | 202 accepted (`audit-wp-0008-t07-e2-134615`) | + +Receiver recreate recovered in about 7 seconds (E2). The rollout (E1) recovered +in about 11 seconds. Both new processes accepted an event, so the mounted +database credential and the sender registry were readable at process start. + +## Custody + +`platform-pg` is one primary. `tenant-engine` and `core-hub` share it. +Both were told immediately before the delete. + +| Step | Wall clock (UTC) | Result | +|---|---|---| +| Last accept before delete | 13:46:56 | 202 accepted | +| `kubectl delete pod platform-pg-1` issued | 13:46:54 workstation | command waited for termination | +| Delete returned | 13:49:59 | pod object gone | +| First probe after that | 13:50:13 | unreachable | +| Next accept | 13:50:21 | 202 accepted (`audit-wp-0008-t07-e3-135015`) | + +The three minutes inside `kubectl delete` were not sampled, so they are not +claimed as an outage. After the primary pod was gone, the next successful +accept was 22 seconds later. The recreated `platform-pg-1` was Ready, restart +count 0. The CloudNativePG cluster reported healthy. `tenant-engine` and +`core-hub` pods were still Running. + +## Integrity + +`python3 -m audit_core verify-chain --schema audit_core` after E3: +`intact: true`, `events: 125`, `first_break: null`. The count is the pre-exercise +chain plus the accepted probes only. No accepted probe is missing. + +## What this evidences + +V1 for the accept path: dependencies were already enumerated in +`docs/availability-recovery.md`, receiver recreate and primary recreate were +exercised, and the interruption was measured. The binding recovery is the +custody restart, 22 seconds from pod termination to the next accept. V2 remains +unreachable while `platform-pg` has one instance. Review this exercise by +2026-12-31. diff --git a/tenancy.yaml b/tenancy.yaml index 76670a1..97cfb0d 100644 --- a/tenancy.yaml +++ b/tenancy.yaml @@ -21,7 +21,7 @@ tenancy: # tenant-engine on platform-pg; the mixture is # reported by the platform, not hidden. - current: { I: 1, A: 2, E: 2, P: 1, R: 2, V: 0 } + current: { I: 1, A: 2, E: 2, P: 1, R: 2, V: 1 } target: { I: 1, A: 2, E: 3, P: 1, R: 2, V: 1 } # §5.2 — declare per path, quote the minimum. The quoted E above is the @@ -104,13 +104,13 @@ tenancy: under the §4.5 retention trigger, with P2 as the fallback — a worse answer than archive, named now so it is not improvised later. V: >- - No exercise establishes restart recovery for the complete audit path. - V1 is the target; replica count or Kubernetes restart policy is not - treated as evidence. Dependencies enumerated and the exercise specified in - docs/availability-recovery.md (AUDIT-WP-0008-T07); the exercise itself - awaits a live window on railiance01. V2 is not reachable at P1 as built: - platform-pg runs instances 1, and Decision 4.6.1 makes V the minimum across - synchronous providers, so V1 is the ceiling here rather than the next step. + V1 evidenced 2026-09-24 (AUDIT-WP-0008-T07). Receiver recreate recovered + in about 7 seconds. Recreating the single platform-pg primary recovered + the accept path 22 seconds after the pod terminated. The chain stayed + intact. Lease revocation and node reboot were not part of the exercise. + V2 is not reachable at P1 as built: platform-pg runs instances 1, and + Decision 4.6.1 makes V the minimum across synchronous providers, so V1 + is the ceiling. Review 2026-12-31. provider: capability: operations.audit @@ -150,12 +150,17 @@ provider: - "rapp-postgres/consumers/audit-core.yaml" - "audit_core/interface.py" V: - available: 0 + available: 1 maximum: 1 conditions: - - "Exercise restart recovery across audit-core, platform-pg and OpenBao." + - "V1 is receiver recreate plus one platform-pg primary recreate, measured 2026-09-24. OpenBao lease revocation was not exercised and is not required for an accept until the current lease expires." + evidence: + - "docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md" + - "docs/availability-recovery.md" evidence: + V1: + - "docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md" A2: - "audit_core/ingestion.py" - "tests/test_ingestion.py" diff --git a/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md b/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md index 6421b7f..24c425c 100644 --- a/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md +++ b/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md @@ -4,12 +4,12 @@ type: workplan title: "Tenancy posture declaration and read-path enforcement" domain: infotech repo: audit-core -status: active +status: finished flavor: implementation owner: claude topic_slug: railiance created: "2026-08-17" -updated: "2026-08-22" +updated: "2026-09-24" depends_on: - AUDIT-WP-0007 state_hub_workstream_id: "a9a248b2-d26c-503e-a8fc-4f3675e6ed51" @@ -549,7 +549,7 @@ duty, so the correction is ours to send unprompted or it will simply sit stale. ```task id: AUDIT-WP-0008-T07 -status: progress +status: done priority: high state_hub_task_id: "4e3ab790-daa6-5fbb-837e-9a9a26848201" ``` @@ -640,6 +640,12 @@ trip through audit-core with a dedicated scoped identity: the fixture returned driver suite is now 12 passing tests; a native full run passed 119 with 24 environment-dependent skips. +Completed 2026-09-24. Receiver recreate recovered in about 7 seconds. The +platform-pg primary recreate recovered the accept path 22 seconds after the +pod terminated. The chain stayed intact. Lease revocation and node reboot were +not run. Evidence: `docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md`. +`current.V` is 1. V2 stays unreachable at one database instance. + ```task id: AUDIT-WP-0008-T08 status: done