61 lines
2.8 KiB
Markdown
61 lines
2.8 KiB
Markdown
# AUDIT-WP-0008-T07 availability exercise — 2026-09-24
|
|
|
|
Operation measured: `POST /v1/events` accepted into custody.
|
|
Authority: founder request in the custodian session, recorded as
|
|
`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`.
|
|
Not performed: runtime-lease revocation, OpenBao shutdown, node reboot.
|
|
|
|
The probe ran inside the receiver, used the already-mounted sender registry,
|
|
and printed only a timestamp and an HTTP class. No credential was printed.
|
|
Each successful probe is a synthetic `audit.recovery.synthetic` event from
|
|
`user-engine` for `tenant:platform`.
|
|
|
|
## Receiver
|
|
|
|
| Step | Wall clock (UTC) | Result |
|
|
|---|---|---|
|
|
| E1 last accept before rollout | 13:45:37 | 202 accepted |
|
|
| E1 `rollout restart` issued | 13:45:35 workstation | new pod scheduled |
|
|
| E1 probes | 13:45:36, 13:45:39 | exec failed, pod not usable |
|
|
| E1 next accept | 13:45:46 | 202 accepted (`audit-wp-0008-t07-e1-134541`) |
|
|
| E2 pod deleted | 13:46:12 | `audit-core-66767ddd58-mcxtm` |
|
|
| E2 probe | 13:46:13 | exec failed |
|
|
| E2 next accept | 13:46:19 | 202 accepted (`audit-wp-0008-t07-e2-134615`) |
|
|
|
|
Receiver recreate recovered in about 7 seconds (E2). The rollout (E1) recovered
|
|
in about 11 seconds. Both new processes accepted an event, so the mounted
|
|
database credential and the sender registry were readable at process start.
|
|
|
|
## Custody
|
|
|
|
`platform-pg` is one primary. `tenant-engine` and `core-hub` share it.
|
|
Both were told immediately before the delete.
|
|
|
|
| Step | Wall clock (UTC) | Result |
|
|
|---|---|---|
|
|
| Last accept before delete | 13:46:56 | 202 accepted |
|
|
| `kubectl delete pod platform-pg-1` issued | 13:46:54 workstation | command waited for termination |
|
|
| Delete returned | 13:49:59 | pod object gone |
|
|
| First probe after that | 13:50:13 | unreachable |
|
|
| Next accept | 13:50:21 | 202 accepted (`audit-wp-0008-t07-e3-135015`) |
|
|
|
|
The three minutes inside `kubectl delete` were not sampled, so they are not
|
|
claimed as an outage. After the primary pod was gone, the next successful
|
|
accept was 22 seconds later. The recreated `platform-pg-1` was Ready, restart
|
|
count 0. The CloudNativePG cluster reported healthy. `tenant-engine` and
|
|
`core-hub` pods were still Running.
|
|
|
|
## Integrity
|
|
|
|
`python3 -m audit_core verify-chain --schema audit_core` after E3:
|
|
`intact: true`, `events: 125`, `first_break: null`. The count is the pre-exercise
|
|
chain plus the accepted probes only. No accepted probe is missing.
|
|
|
|
## What this evidences
|
|
|
|
V1 for the accept path: dependencies were already enumerated in
|
|
`docs/availability-recovery.md`, receiver recreate and primary recreate were
|
|
exercised, and the interruption was measured. The binding recovery is the
|
|
custody restart, 22 seconds from pod termination to the next accept. V2 remains
|
|
unreachable while `platform-pg` has one instance. Review this exercise by
|
|
2026-12-31.
|