# Audit-core restart-free database lease recovery This is the review and attended-execution contract for `RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by audit-core causes retryable unavailability and that External Secrets plus the mounted credential reader restore service without replacing or restarting the pod. It does not authorize a live revocation. ## Fixed scope and owners | Boundary | Exact scope | Responsible owner | | --- | --- | --- | | Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` | | Lease | `database/creds/audit-core-runtime`; the one handle issued for the current ExternalSecret refresh (older predecessors are left alone) | `railiance-platform` custody operator | | Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` | | Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner | | Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` | The live window requires audit-core and rapp-postgres acknowledgement, a named abort operator, and an approved synthetic-load identifier. Run the read-only gate before requesting that approval: ```bash python3 scripts/audit-core-recovery-preflight.py database-lease ``` The result must have `automated_checks_passed: true`. It will remain `ready_for_live_execution: false` until the explicit owner/window fields are provided. The helper reads Secret metadata and key names only; it never reads Secret data. The live harness is `scripts/audit-core-database-lease-recovery.py`. Start from the deliberately pending `docs/audit-core-database-lease-approval.example.json`; the actual value-safe receipt must name a ≤15-minute window, approval id, abort operator, State Hub acknowledgement ids from audit-core/rapp-postgres/platform, and an approved synthetic-load contract plus driver revision. Validate without mutation: ```bash make audit-database-lease-approval-check \ AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json ``` The separately approved load driver is an executable that accepts `baseline`, `expect-unavailable`, `expect-recovered`, and `cleanup`, plus `--contract-id ID`. Each call returns only its exact value-safe JSON evidence shape. Unexpected keys—including a bearer accidentally returned as `token`— make the harness fail without echoing driver output. ## Synthetic sender projection lane The load driver (`audit-core` `scripts/t02_synthetic_load_driver.py`, registered at `8c8bcf4`) needs exactly one temporary audit-core sender identity. The reviewed, reusable lane is `scripts/audit_core_synthetic_sender_lane.py`. It reuses the live-proven registry/reload helpers of the dated Whitehat `-02` procedure by import and the T02 receipt, attest-chain and preflight gates of the lease harness; the dated `-01`/`-02` procedures are not copied. **Receipt binding.** The identity is named only in the approved T02 receipt (see `docs/audit-core-database-lease-approval.example.json`), in a `synthetic_sender` block with exactly these keys: ```json "synthetic_sender": { "identity": "t02-synthetic-", "source": "", "tenant": "tenant:trial:", "expires_at": "" } ``` The lane accepts the receipt only if it also passes the lease harness's own approval validation (approved status, ≤15-minute window, abort operator, all three owner acknowledgements) and `synthetic_load.contract_id` equals the registered driver interface `RAILIANCE-WP-0024-T02-SYNTHETIC-01`. The registry entry it adds is `sources=[source]`, `tenants=[tenant]`, `may_write=true`, `may_read=true`, `expires_at=`; projection refuses when the name, source or tenant already appears on any existing identity. The same `identity`, `source` and `tenant` values are what the operator passes to the driver as `AUDIT_T02_SOURCE`/`AUDIT_T02_TENANT`; the confirm string is `audit-core-synthetic-sender::attended` (printed by preflight). **Ordering.** The bearer reaches audit-core only through the registry at `platform/workloads/audit-core/senders` → ExternalSecret `audit-core-senders` → env `AUDIT_CORE_SENDERS`, so both projection and cleanup restart `deploy/audit-core`. The restarts must stay outside the restart-free proof: project → restart → T02 exercise → evidence → cleanup → restart. **Contained OpenBao access only.** Every live command (lane `project` and `cleanup`, harness `exercise`) requires `--contained` and refuses unless it runs as the child of `scripts/openbao-attended-exec.py`, i.e. inside `warden access openbao-platform-admin-login --exec`: `WARDEN_ROUTING_CATALOG` must be set, `BAO_ADDR` must be the operator tunnel `http://127.0.0.1:18200`, and `BAO_TOKEN`/`VAULT_TOKEN`/`OPENBAO_TOKEN` must be absent; `bao` authenticates through Warden's isolated token helper. There is no token-file path any more: the harness's former default `https://bao.coulomb.social` was retracted on 2026-09-15 (RMASTER-WP-0020-T09), and a standing 0600 platform-admin token file is outside the accepted custody path. Outside the login, the lane's `preflight` runs the cluster checks only; `--contained` adds registry metadata and `--inspect-registry` (attended, in-memory) adds the conflict check. **One login per window.** `scripts/t02-attended-window.py` is the single reviewed child of the contained login, so the ≤15-minute window needs one MFA login: ```bash make audit-t02-attended-window \ AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json \ AUDIT_DATABASE_LEASE_DRIVER=/path/to/audit-core/scripts/t02_synthetic_load_driver.py \ AUDIT_SYNTHETIC_SENDER_CONFIRM=audit-core-synthetic-sender::attended \ AUDIT_T02_TOKEN_FILE=/abs/0700-dir/t02.bearer \ AUDIT_T02_EVIDENCE_DIR=/abs/0700-dir/evidence ``` It runs, in order: 1. lane `project` — only within the first 3 minutes of the window and with ≥10 minutes of identity lifetime left. Writes the bearer to a new mode-`0600` file (absolute, operator-owned `0700` directory outside every Git work tree; never stdout), adds the one identity, forces `audit-core-senders` sync, runs `rollout restart deploy/audit-core`, and waits for one Ready restarted pod. Any failure removes the identity (restarting again) and shreds the file. 2. port-forward — the lane and harness reach the cluster only via `ssh railiance01 kubectl`, so the window uses the same path: `ssh -L 127.0.0.1::127.0.0.1: railiance01 timeout 900 kubectl -n audit-core port-forward --address 127.0.0.1 deploy/audit-core :8080` (default port 18081, `AUDIT_T02_PORT` to override). No local kubeconfig is needed; the remote forward is bounded by `timeout` even if SSH is cut. audit-core's NetworkPolicies admit only named namespaces, which is why the operator-side driver needs the forward. 3. harness `exercise --contained` with the registered driver; the driver env is derived from the receipt: `AUDIT_T02_BASE_URL=http://127.0.0.1:`, `AUDIT_T02_TOKEN_FILE`, `AUDIT_T02_SOURCE`/`AUDIT_T02_TENANT` from `synthetic_sender`, `AUDIT_T02_FIXTURE_ID=t02-`, `AUDIT_T02_OCCURRED_AT=window.start`. Its baseline captures the post-projection pod UID. 4. evidence — the exercise JSON is written `0600` as `/-exercise.json`; a value-safe window summary is written as `-window.json` on success and on abort. 5. lane `cleanup` with that evidence — refuses unless it names this `approval_id`. Removes only the exact named identity (refusing if the name carries a different scope), restarts audit-core only if something was removed, shreds the bearer file, and confirms 1/1 Ready. The port-forward is stopped before cleanup and on every exit. Any failure after projection was attempted (including a failed `project`, whose registry write may already have reached OpenBao) runs `cleanup --abort-reason` before the window script exits non-zero. Children's stdout is captured and only their value-safe JSON is re-emitted. The single-step targets (`audit-synthetic-sender-project`, `audit-database-lease-recovery-exercise`, `audit-synthetic-sender-cleanup`) remain for the abort path; each costs its own contained login. A missed cleanup is run as `make audit-synthetic-sender-cleanup AUDIT_SYNTHETIC_SENDER_ABORT_REASON=...` and is a no-op if nothing remains. If cleanup is missed, audit-core's `expires_at` enforcement makes the identity inert at window end; the attended cleanup is still required to remove it. Projection and cleanup each restart audit-core once (Recreate, no surge), so production senders see two short retryable gaps per window; announce them with the window. The contained login discards child output and fails closed (exit 12, token revoked) on any byte the child prints. The window script therefore prints nothing: its summary, exercise evidence or `window-error-*.json` land only in the evidence directory. Single-step targets require `AUDIT_T02_RESULT_FILE` so the lane/harness write their JSON there instead of stdout. ## Attended sequence Keep one terminal attached from baseline through recovery. A value-handling helper used in the approved window must suppress OpenBao/Kubernetes response bodies and clear all transient JSON from memory or mode-`0600` temporary files. 1. Record the audit-core pod UID, restart count, database Secret `resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness, ExternalSecret condition, and ClusterSecretStore condition. 2. Start bounded retrying synthetic traffic. Its evidence records only request identifier, attempt time, response class, and final accepted/duplicate result—never the bearer or payload. 3. Inside the attended process, list only lease handles below `sys/leases/lookup/database/creds/audit-core-runtime`. OpenBao's lease lookup returns issue/expiry/TTL metadata but not the leased username, so exact selection is deliberately strict: at most 8 handles may exist under that exact runtime prefix; exactly one must have been issued within 5 seconds of the current database Secret refresh (the mounted credential), with at least five minutes of TTL left; every other handle must be an older predecessor. ESO refreshes every 15 minutes while the role TTL is 15 minutes (max 1 hour), so a predecessor can legitimately still be live; it is never revoked. Abort if no handle, several handles, or a newer unmatched handle match the refresh. (2026-10-14: the first attended window aborted here under the earlier "exactly one handle" rule with two live handles.) Retain only a one-way handle fingerprint in evidence and clear the lookup response immediately. 4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have not changed. Revoke only that handle. Do not revoke the role prefix and do not revoke the ESO parent token. 5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes 503, at least one synthetic attempt receives retryable 503, pod UID is unchanged, and restart count is unchanged. Abort if liveness fails, the pod restarts, a non-synthetic request is involved, or 503 is not retryable. 6. Annotate only ExternalSecret `audit-core/audit-core-database` to force reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret `resourceVersion`, and Kubernetes' mounted `..data` generation to advance. Do not restart or roll out audit-core. 7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches accepted or duplicate, the pre-exercise event remains readable, pod UID is unchanged, and restart count is unchanged. 8. Stop the load, confirm the predecessor lease is revoked, record the evidence, then remove the identity with the synthetic sender lane cleanup (which restarts audit-core after the evidence is captured). The replacement remains managed by ESO and expires through its normal database role TTL. Only inside the approved window, with the receipt and reviewed driver, run the whole window through `make audit-t02-attended-window` (see the synthetic sender section above); the exercise step alone is `make audit-database-lease-recovery-exercise`, which also runs inside the contained login. The exact confirmation string is embedded in the target. The harness rechecks the Secret/pod/lease immediately before revocation, requires at least five minutes of remaining lease TTL to avoid the normal ESO refresh race, performs best-effort forced reconciliation on an interrupted post-revocation run, and never issues a rollout/restart command. ## Time bounds and abort path - Planned outage observation: at most 120 seconds. - Full exercise: at most 10 minutes. - If `/readyz` has not recovered 120 seconds after forced reconciliation, stop load and page both custody and database owners. Do not bounce the pod. - If ESO is not `SecretSynced`, check OpenBao seal state and the exact store. Restart External Secrets only for the known post-unseal stale-provider case and only with the platform owner present. - If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without revocation. ## Value-safe evidence Record timestamps, approved window/load identifiers, owner acknowledgements, lease accessor or one-way handle fingerprint, Secret resource versions, mount generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO conditions, and cleanup outcome. Never record usernames, passwords, DSNs, Secret data, bearer values, request payloads, OpenBao response bodies, or unseal material.