# Audit-core restart-free database lease recovery

This is the review and attended-execution contract for
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
audit-core causes retryable unavailability and that External Secrets plus the
mounted credential reader restore service without replacing or restarting the
pod. It does not authorize a live revocation.

## Fixed scope and owners

| Boundary | Exact scope | Responsible owner |
| --- | --- | --- |
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
| Lease | `database/creds/audit-core-runtime`; only the lease matching the current mounted username | `railiance-platform` custody operator |
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |

The live window requires audit-core and rapp-postgres acknowledgement, a named
abort operator, and an approved synthetic-load identifier. Run the read-only
gate before requesting that approval:

```bash
python3 scripts/audit-core-recovery-preflight.py database-lease
```

The result must have `automated_checks_passed: true`. It will remain
`ready_for_live_execution: false` until the explicit owner/window fields are
provided. The helper reads Secret metadata and key names only; it never reads
Secret data.

## Attended sequence

Keep one terminal attached from baseline through recovery. A value-handling
helper used in the approved window must suppress OpenBao/Kubernetes response
bodies and clear all transient JSON from memory or mode-`0600` temporary files.

1. Record the audit-core pod UID, restart count, database Secret
   `resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
   ExternalSecret condition, and ClusterSecretStore condition.
2. Start bounded retrying synthetic traffic. Its evidence records only request
   identifier, attempt time, response class, and final accepted/duplicate
   result—never the bearer or payload.
3. Inside the attended process, read the current mounted username without
   emitting it. List only lease handles below
   `sys/leases/lookup/database/creds/audit-core-runtime`, look each up, and
   retain the one handle whose lease username matches. Abort unless exactly one
   live handle matches. Clear every lookup response and username immediately.
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
   not changed. Revoke only that handle. Do not revoke the role prefix and do
   not revoke the ESO parent token.
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
   503, at least one synthetic attempt receives retryable 503, pod UID is
   unchanged, and restart count is unchanged. Abort if liveness fails, the pod
   restarts, a non-synthetic request is involved, or 503 is not retryable.
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
   reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
   `resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
   Do not restart or roll out audit-core.
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
   accepted or duplicate, the pre-exercise event remains readable, pod UID is
   unchanged, and restart count is unchanged.
8. Stop the load, remove its separately approved fixture/identity, and confirm
   the predecessor lease is revoked. The replacement remains managed by ESO
   and expires through its normal database role TTL.

## Time bounds and abort path

- Planned outage observation: at most 120 seconds.
- Full exercise: at most 10 minutes.
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
  stop load and page both custody and database owners. Do not bounce the pod.
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
  Restart External Secrets only for the known post-unseal stale-provider case
  and only with the platform owner present.
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
  revocation.

## Value-safe evidence

Record timestamps, approved window/load identifiers, owner acknowledgements,
lease accessor or one-way handle fingerprint, Secret resource versions, mount
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
Secret data, bearer values, request payloads, OpenBao response bodies, or
unseal material.