# Availability and recovery

**Status:** V1 evidenced 2026-09-24. Review by 2026-12-31.
**Framework:** NetKingdom Tenancy Posture v0.1 (draft-8) §4.6, §13 V-row.
**Declared:** `tenancy.yaml` — `current.V: 1`, `target.V: 1`.

## Why this is V1

The 2026-08-16 node reboot was an observation, not an exercise. The 2026-09-24
run was an exercise: the receiver was recreated, the platform-pg primary was
recreated, and the time to the next accepted event was recorded. A replica
count is still not the evidence. The evidence is the exercise.

## Critical dependency enumeration (Decision 4.6.1)

V is **end-to-end**: the minimum across the components and synchronous providers
required to serve the operation. An application that restarts in five seconds
over a database that takes ninety is not V1 at five seconds.

The operation being declared is **accept an audit event** (`POST /v1/events`).
Read surfaces are operator-facing and are not the availability claim.

| # | Dependency | Role in the accept path | Failure shows as |
|---|---|---|---|
| 1 | `audit-core` Deployment (1 replica, namespace `audit-core`) | Serves the receiver | Connection refused; Service has no endpoint |
| 2 | `platform-pg` (CNPG, `instances: 1`) | Custody. Synchronous — an accept is not accepted until committed | `/readyz` 503, request 503 `unavailable` |
| 3 | Secret `audit-core-database` at `/etc/audit-core/db` | Runtime lease, re-read per connection | 503 `unavailable` on lease expiry or revocation |
| 4 | ESO + OpenBao (`database/creds/audit-core-runtime`) | Refreshes 3 every 15 min | Delayed: works until the current lease expires |
| 5 | Secret `audit-core-senders` + scope ConfigMap | Sender authentication; read at process start | 401 for every sender after a restart with a bad Secret |
| 6 | CoreDNS | Resolves `platform-pg` | 503 `unavailable`, indistinguishable from 2 without logs |
| 7 | NetworkPolicy (default-deny + sender ingress) | Admits `user-engine` | Sender timeouts; receiver sees nothing at all |

**The binding constraint is 2.** `platform-pg` runs `instances: 1` — no HA, single
node. §17 of the framework already records that no P1 tenant has HA. audit-core
cannot exceed `platform-pg`'s availability level regardless of what it does to
its own Deployment, so **V2 is not reachable from P1 as built** and is not a
target. V1 is.

Dependency 4 is worth separating: OpenBao being down does not stop accepts. It
stops *renewal*, so the failure is deferred to lease expiry. An exercise that
takes OpenBao down and declares success after two minutes has proved nothing —
this is the invisible-failure shape §12 warns about.

## The V1 exercise

§13's V1 row asks for three things, all mechanical: dependencies enumerated
(above), restart/recreate recovery exercised, interruption and measured recovery
time recorded.

**Rules for the run.** Measure from the last successful accept to the next
successful accept, not from pod restart to Ready — Ready is our own opinion of
ourselves. Drive a real `POST /v1/events` on a fixed interval throughout and
count what the sender sees, because the sender's experience is the availability
being claimed. Record wall-clock times, not durations someone computed later.

| # | Scenario | Action | Record |
|---|---|---|---|
| E1 | Receiver restart | `kubectl -n audit-core rollout restart deploy/audit-core` | Accepts lost; time to first successful accept |
| E2 | Receiver recreate | Delete the pod | As E1, plus whether any accepted event is missing after recovery |
| E3 | Custody restart | Restart the `platform-pg` primary | Time to first successful accept; 503 count; chain intact after |
| E4 | Lease expiry under load | Revoke the runtime lease, wait for ESO refresh | Time to recovery **without** a pod restart — the property `credentials.py` exists to provide |
| E5 | Node reboot | Reboot railiance01 | End-to-end recovery; compare against the 2026-08-16 observation of ~40s |

**Integrity is part of the pass condition, not a separate check.** After every
scenario, `python -m audit_core verify-chain` must report intact, and every
event the driver believes was accepted must be present. A recovery that loses an
accepted event, or forks the chain, is a custody defect and fails the exercise
regardless of how fast it was.

**RPO.** An accept is committed before the sender is told, so the expected RPO
for accepted events is zero. The exercise tests that claim rather than assuming
it; E2 and E3 are where it would break.

## Results

Run 2026-09-24. Evidence: `docs/evidence/AUDIT-WP-0008-T07-v1-2026-09-24.md`.

| Scenario | Measured recovery | Pass |
|---|---|---|
| E1 receiver rollout | about 11s to the next accept | chain intact |
| E2 receiver pod delete | about 7s to the next accept | chain intact |
| E3 `platform-pg` primary recreate | 22s from pod termination to the next accept | chain intact; co-residents still running |
| E4 lease revocation | not run | not claimed |
| E5 node reboot | not run | not claimed |

The binding number is E3. E4 and E5 stay out of this V1 claim: OpenBao being
down does not stop accepts until the current lease expires, and a node reboot
was not part of the approved window. Review by 2026-12-31.