coulomb/audit-core
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Record the availability exercise and finish the tenancy posture plan.
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 4s
Details

Browse source 
Assistant: grok
Assistant-Session: 01a0d25d-d358-7e13-b84a-d007fbb7e34f
This commit is contained in:
tegwick 2026-09-24 15:53:09 +02:00
parent 6b46d1aaca
commit 8d3328b620
4 changed files with 104 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

42
docs/availability-recovery.md
View file

@ -1,25 +1,15 @@
# Availability and recovery
**Status:** V1 not yet evidenced. Dependency enumeration complete; exercise
pending a live window on railiance01.
**Status:** V1 evidenced 2026-09-24. Review by 2026-12-31.
**Framework:** NetKingdom Tenancy Posture v0.1 (draft-8) §4.6, §13 V-row.
**Declared:** `tenancy.yaml` — `current.V: 0`, `target.V: 1`.
**Declared:** `tenancy.yaml` — `current.V: 1`, `target.V: 1`.
## Why V0 today
## Why this is V1
`V0` is "no availability or recovery position; recovery is untested or depends
on improvisation". That is accurate. Nothing in this repo exercises recovery of
the complete audit path and records a measured recovery time.
What exists is an *observation*, not an exercise: after the railiance01 node
reboot on 2026-08-16, `/readyz` failed for roughly 40 seconds before the pod
went Ready, recorded in `docs/operator-runbook.md`. That is useful and it is not
V1 evidence — it enumerated nothing, measured nothing deliberately, and happened
to us rather than being performed.
Decision 4.6.1 is explicit that a replica count or a status page is not
evidence of a level. audit-core runs one replica; that fact argues for neither
V0 nor V1. Only the exercise settles it.
The 2026-08-16 node reboot was an observation, not an exercise. The 2026-09-24
run was an exercise: the receiver was recreated, the platform-pg primary was
recreated, and the time to the next accepted event was recorded. A replica
count is still not the evidence. The evidence is the exercise.
## Critical dependency enumeration (Decision 4.6.1)
@ -83,10 +73,16 @@ it; E2 and E3 are where it would break.
## Results
*Not yet run.* Needs a live window on railiance01 and coordination with
`user-engine`, since E4 and E5 are visible to the sender.
Run 2026-09-24. Evidence: `docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md`.
On completion, record the measured recovery time per scenario here with dates,
raise `tenancy.yaml` `current.V` to 1, note the exercise as the evidence, and
set a review date — a recovery exercise from a year ago describes a system that
no longer exists.
| Scenario | Measured recovery | Pass |
|---|---|---|
| E1 receiver rollout | about 11s to the next accept | chain intact |
| E2 receiver pod delete | about 7s to the next accept | chain intact |
| E3 `platform-pg` primary recreate | 22s from pod termination to the next accept | chain intact; co-residents still running |
| E4 lease revocation | not run | not claimed |
| E5 node reboot | not run | not claimed |
The binding number is E3. E4 and E5 stay out of this V1 claim: OpenBao being
down does not stop accepts until the current lease expires, and a node reboot
was not part of the approved window. Review by 2026-12-31.

61
docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md Normal file
View file

@ -0,0 +1,61 @@
# AUDIT-WP-0008-T07 availability exercise — 2026-09-24
Operation measured: `POST /v1/events` accepted into custody.
Authority: founder request in the custodian session, recorded as
`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`.
Not performed: runtime-lease revocation, OpenBao shutdown, node reboot.
The probe ran inside the receiver, used the already-mounted sender registry,
and printed only a timestamp and an HTTP class. No credential was printed.
Each successful probe is a synthetic `audit.recovery.synthetic` event from
`user-engine` for `tenant:platform`.
## Receiver
| Step | Wall clock (UTC) | Result |
|---|---|---|
| E1 last accept before rollout | 13:45:37 | 202 accepted |
| E1 `rollout restart` issued | 13:45:35 workstation | new pod scheduled |
| E1 probes | 13:45:36, 13:45:39 | exec failed, pod not usable |
| E1 next accept | 13:45:46 | 202 accepted (`audit-wp-0008-t07-e1-134541`) |
| E2 pod deleted | 13:46:12 | `audit-core-66767ddd58-mcxtm` |
| E2 probe | 13:46:13 | exec failed |
| E2 next accept | 13:46:19 | 202 accepted (`audit-wp-0008-t07-e2-134615`) |
Receiver recreate recovered in about 7 seconds (E2). The rollout (E1) recovered
in about 11 seconds. Both new processes accepted an event, so the mounted
database credential and the sender registry were readable at process start.
## Custody
`platform-pg` is one primary. `tenant-engine` and `core-hub` share it.
Both were told immediately before the delete.
| Step | Wall clock (UTC) | Result |
|---|---|---|
| Last accept before delete | 13:46:56 | 202 accepted |
| `kubectl delete pod platform-pg-1` issued | 13:46:54 workstation | command waited for termination |
| Delete returned | 13:49:59 | pod object gone |
| First probe after that | 13:50:13 | unreachable |
| Next accept | 13:50:21 | 202 accepted (`audit-wp-0008-t07-e3-135015`) |
The three minutes inside `kubectl delete` were not sampled, so they are not
claimed as an outage. After the primary pod was gone, the next successful
accept was 22 seconds later. The recreated `platform-pg-1` was Ready, restart
count 0. The CloudNativePG cluster reported healthy. `tenant-engine` and
`core-hub` pods were still Running.
## Integrity
`python3 -m audit_core verify-chain --schema audit_core` after E3:
`intact: true`, `events: 125`, `first_break: null`. The count is the pre-exercise
chain plus the accepted probes only. No accepted probe is missing.
## What this evidences
V1 for the accept path: dependencies were already enumerated in
`docs/availability-recovery.md`, receiver recreate and primary recreate were
exercised, and the interruption was measured. The binding recovery is the
custody restart, 22 seconds from pod termination to the next accept. V2 remains
unreachable while `platform-pg` has one instance. Review this exercise by
2026-12-31.