railiance-platform/docs/audit-core-database-lease-recovery.md

# Audit-core restart-free database lease recovery

This is the review and attended-execution contract for
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
audit-core causes retryable unavailability and that External Secrets plus the
mounted credential reader restore service without replacing or restarting the
pod. It does not authorize a live revocation.

## Fixed scope and owners

| Boundary | Exact scope | Responsible owner |
| --- | --- | --- |
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
| Lease | `database/creds/audit-core-runtime`; exactly one live handle whose issue time coheres with the current ExternalSecret refresh | `railiance-platform` custody operator |
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |

The live window requires audit-core and rapp-postgres acknowledgement, a named
abort operator, and an approved synthetic-load identifier. Run the read-only
gate before requesting that approval:

```bash
python3 scripts/audit-core-recovery-preflight.py database-lease
```

The result must have `automated_checks_passed: true`. It will remain
`ready_for_live_execution: false` until the explicit owner/window fields are
provided. The helper reads Secret metadata and key names only; it never reads
Secret data.

The live harness is
`scripts/audit-core-database-lease-recovery.py`. Start from the deliberately
pending `docs/audit-core-database-lease-approval.example.json`; the actual
value-safe receipt must name a ≤15-minute window, approval id, abort operator,
State Hub acknowledgement ids from audit-core/rapp-postgres/platform, and an
approved synthetic-load contract plus driver revision. Validate without
mutation:

```bash
make audit-database-lease-approval-check \
  AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json
```

The separately approved load driver is an executable that accepts `baseline`,
`expect-unavailable`, `expect-recovered`, and `cleanup`, plus
`--contract-id ID`. Each call returns only its exact value-safe JSON evidence
shape. Unexpected keys—including a bearer accidentally returned as `token`—
make the harness fail without echoing driver output.

## Synthetic sender projection lane

The load driver (`audit-core` `scripts/t02_synthetic_load_driver.py`, registered
at `8c8bcf4`) needs exactly one temporary audit-core sender identity. The
reviewed, reusable lane is `scripts/audit_core_synthetic_sender_lane.py`. It
reuses the live-proven registry/reload helpers of the dated Whitehat `-02`
procedure by import and the T02 receipt, attest-chain and preflight gates of
the lease harness; the dated `-01`/`-02` procedures are not copied.

**Receipt binding.** The identity is named only in the approved T02 receipt
(see `docs/audit-core-database-lease-approval.example.json`), in a
`synthetic_sender` block with exactly these keys:

```json
"synthetic_sender": {
  "identity": "t02-synthetic-<window>",
  "source": "<named synthetic source, contains 'synthetic'>",
  "tenant": "tenant:trial:<name>",
  "expires_at": "<RFC3339 UTC, not later than window.end>"
}
```

The lane accepts the receipt only if it also passes the lease harness's own
approval validation (approved status, ≤15-minute window, abort operator, all
three owner acknowledgements) and `synthetic_load.contract_id` equals the
registered driver interface `RAILIANCE-WP-0024-T02-SYNTHETIC-01`. The registry
entry it adds is `sources=[source]`, `tenants=[tenant]`, `may_write=true`,
`may_read=true`, `expires_at=<receipt value>`; projection refuses when the
name, source or tenant already appears on any existing identity. The same
`identity`, `source` and `tenant` values are what the operator passes to the
driver as `AUDIT_T02_SOURCE`/`AUDIT_T02_TENANT`; the confirm string is
`audit-core-synthetic-sender:<approval_id>:attended` (printed by preflight).

**Ordering.** The bearer reaches audit-core only through the registry at
`platform/workloads/audit-core/senders` → ExternalSecret `audit-core-senders`
→ env `AUDIT_CORE_SENDERS`, so both projection and cleanup restart
`deploy/audit-core`. The restarts must stay outside the restart-free proof:

1. `make audit-synthetic-sender-preflight` — read-only; reads registry metadata
   only (add `AUDIT_SYNTHETIC_SENDER_PREFLIGHT_ARGS=--inspect-registry` for an
   attended in-memory conflict check).
2. `make audit-synthetic-sender-project` — only within the first 3 minutes of
   the window and with ≥10 minutes of identity lifetime left. Writes the
   bearer to a new mode-`0600` file (`AUDIT_T02_TOKEN_FILE`, absolute, in an
   operator-owned `0700` directory outside every Git work tree; never stdout),
   adds the one identity, forces `audit-core-senders` sync and runs
   `rollout restart deploy/audit-core`, then waits for one Ready restarted pod
   and prints its UID. Any failure removes the identity (restarting again) and
   shreds the bearer file.
3. The T02 exercise (`make audit-database-lease-recovery-exercise`). Its
   baseline must capture the post-projection pod UID; never project while the
   exercise is running.
4. Record the exercise's JSON result (it is the evidence file).
5. `make audit-synthetic-sender-cleanup AUDIT_T02_EVIDENCE=<result.json>` —
   refuses unless that evidence names this `approval_id`, or an explicit
   `AUDIT_SYNTHETIC_SENDER_ABORT_REASON` is given for an aborted run. Removes
   only the exact named identity (refusing if the name carries a different
   scope), restarts audit-core only if something was removed, shreds the
   bearer file, and confirms 1/1 Ready. Re-running is a no-op.

If cleanup is missed, audit-core's `expires_at` enforcement makes the identity
inert at window end; the attended cleanup is still required to remove it.

The driver runs operator-side. audit-core's NetworkPolicies admit only named
namespaces, so reach the pod with `kubectl -n audit-core port-forward
deploy/audit-core <local-port>:8080` and set `AUDIT_T02_BASE_URL` to that local
URL; record the base URL and the driver's source/tenant (which must equal the
receipt's `synthetic_sender` block) in the window notes. Projection and cleanup
each restart audit-core once (Recreate, no surge), so production senders see two
short retryable gaps per window; announce them with the window.

If `project` fails after the registry write may already have reached OpenBao (for
example a transport error on the write itself), treat the identity as possibly
present: run `cleanup --abort-reason "<reason>"` before leaving the window.
The lane's `--token-file` is the attended operator token for the registry write;
it must come from the contained ops-warden attended login accepted under RPF-WP-0017
(isolated 0600 helper state, removed afterwards), not from a long-lived token file.

## Attended sequence

Keep one terminal attached from baseline through recovery. A value-handling
helper used in the approved window must suppress OpenBao/Kubernetes response
bodies and clear all transient JSON from memory or mode-`0600` temporary files.

1. Record the audit-core pod UID, restart count, database Secret
   `resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
   ExternalSecret condition, and ClusterSecretStore condition.
2. Start bounded retrying synthetic traffic. Its evidence records only request
   identifier, attempt time, response class, and final accepted/duplicate
   result—never the bearer or payload.
3. Inside the attended process, list only lease handles below
   `sys/leases/lookup/database/creds/audit-core-runtime`. OpenBao's lease lookup
   returns issue/expiry/TTL metadata but not the leased username, so exact
   selection is deliberately stricter: there must be exactly one live handle
   under that exact runtime prefix, and its issue/expiry interval must contain
   the current database Secret refresh time. Abort for database-owner
   disambiguation if zero or multiple handles exist. Retain only a one-way
   handle fingerprint in evidence and clear the lookup response immediately.
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
   not changed. Revoke only that handle. Do not revoke the role prefix and do
   not revoke the ESO parent token.
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
   503, at least one synthetic attempt receives retryable 503, pod UID is
   unchanged, and restart count is unchanged. Abort if liveness fails, the pod
   restarts, a non-synthetic request is involved, or 503 is not retryable.
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
   reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
   `resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
   Do not restart or roll out audit-core.
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
   accepted or duplicate, the pre-exercise event remains readable, pod UID is
   unchanged, and restart count is unchanged.
8. Stop the load, confirm the predecessor lease is revoked, record the
   evidence, then remove the identity with the synthetic sender lane cleanup
   (which restarts audit-core after the evidence is captured). The replacement remains managed by ESO
   and expires through its normal database role TTL.

Only inside the approved window, with the receipt and reviewed driver:

```bash
make audit-database-lease-recovery-exercise \
  AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json \
  AUDIT_DATABASE_LEASE_DRIVER=/path/to/approved-load-driver
```

The exact confirmation string is embedded in the target. The harness rechecks
the Secret/pod/lease immediately before revocation, requires at least five
minutes of remaining lease TTL to avoid the normal ESO refresh race, performs
best-effort forced reconciliation on an interrupted post-revocation run, and
never issues a rollout/restart command.

## Time bounds and abort path

- Planned outage observation: at most 120 seconds.
- Full exercise: at most 10 minutes.
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
  stop load and page both custody and database owners. Do not bounce the pod.
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
  Restart External Secrets only for the known post-unseal stale-provider case
  and only with the platform owner present.
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
  revocation.

## Value-safe evidence

Record timestamps, approved window/load identifiers, owner acknowledgements,
lease accessor or one-way handle fingerprint, Secret resource versions, mount
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
Secret data, bearer values, request payloads, OpenBao response bodies, or
unseal material.