railiance-platform/docs/audit-core-database-lease-recovery.md

Audit-core restart-free database lease recovery

This is the review and attended-execution contract for RAILIANCE-WP-0024-T02. It proves that revoking the lease currently used by audit-core causes retryable unavailability and that External Secrets plus the mounted credential reader restore service without replacing or restarting the pod. It does not authorize a live revocation.

Fixed scope and owners

Boundary	Exact scope	Responsible owner
Consumer	Deployment audit-core/audit-core; Secret audit-core/audit-core-database; ExternalSecret audit-core/audit-core-database	audit-core
Lease	database/creds/audit-core-runtime; exactly one live handle whose issue time coheres with the current ExternalSecret refresh	railiance-platform custody operator
Database	CNPG databases/platform-pg, database audit_core, runtime group audit_core_app	rapp-postgres
Load	One separately approved synthetic sender/tenant contract; no production event or identity	audit-core + load owner
Coordination and evidence	RAILIANCE-WP-0024-T02	railiance-platform

The live window requires audit-core and rapp-postgres acknowledgement, a named abort operator, and an approved synthetic-load identifier. Run the read-only gate before requesting that approval:

python3 scripts/audit-core-recovery-preflight.py database-lease

The result must have automated_checks_passed: true. It will remain ready_for_live_execution: false until the explicit owner/window fields are provided. The helper reads Secret metadata and key names only; it never reads Secret data.

The live harness is scripts/audit-core-database-lease-recovery.py. Start from the deliberately pending docs/audit-core-database-lease-approval.example.json; the actual value-safe receipt must name a ≤15-minute window, approval id, abort operator, State Hub acknowledgement ids from audit-core/rapp-postgres/platform, and an approved synthetic-load contract plus driver revision. Validate without mutation:

make audit-database-lease-approval-check \
  AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json

The separately approved load driver is an executable that accepts baseline, expect-unavailable, expect-recovered, and cleanup, plus --contract-id ID. Each call returns only its exact value-safe JSON evidence shape. Unexpected keys—including a bearer accidentally returned as token— make the harness fail without echoing driver output.

Synthetic sender projection lane

The load driver (audit-core scripts/t02_synthetic_load_driver.py, registered at 8c8bcf4) needs exactly one temporary audit-core sender identity. The reviewed, reusable lane is scripts/audit_core_synthetic_sender_lane.py. It reuses the live-proven registry/reload helpers of the dated Whitehat -02 procedure by import and the T02 receipt, attest-chain and preflight gates of the lease harness; the dated -01/-02 procedures are not copied.

Receipt binding. The identity is named only in the approved T02 receipt (see docs/audit-core-database-lease-approval.example.json), in a synthetic_sender block with exactly these keys:

"synthetic_sender": {
  "identity": "t02-synthetic-<window>",
  "source": "<named synthetic source, contains 'synthetic'>",
  "tenant": "tenant:trial:<name>",
  "expires_at": "<RFC3339 UTC, not later than window.end>"
}

The lane accepts the receipt only if it also passes the lease harness's own approval validation (approved status, ≤15-minute window, abort operator, all three owner acknowledgements) and synthetic_load.contract_id equals the registered driver interface RAILIANCE-WP-0024-T02-SYNTHETIC-01. The registry entry it adds is sources=[source], tenants=[tenant], may_write=true, may_read=true, expires_at=<receipt value>; projection refuses when the name, source or tenant already appears on any existing identity. The same identity, source and tenant values are what the operator passes to the driver as AUDIT_T02_SOURCE/AUDIT_T02_TENANT; the confirm string is audit-core-synthetic-sender:<approval_id>:attended (printed by preflight).

Ordering. The bearer reaches audit-core only through the registry at platform/workloads/audit-core/senders → ExternalSecret audit-core-senders → env AUDIT_CORE_SENDERS, so both projection and cleanup restart deploy/audit-core. The restarts must stay outside the restart-free proof:

1. make audit-synthetic-sender-preflight — read-only; reads registry metadata only (add AUDIT_SYNTHETIC_SENDER_PREFLIGHT_ARGS=--inspect-registry for an attended in-memory conflict check).
2. make audit-synthetic-sender-project — only within the first 3 minutes of the window and with ≥10 minutes of identity lifetime left. Writes the bearer to a new mode-0600 file (AUDIT_T02_TOKEN_FILE, absolute, in an operator-owned 0700 directory outside every Git work tree; never stdout), adds the one identity, forces audit-core-senders sync and runs rollout restart deploy/audit-core, then waits for one Ready restarted pod and prints its UID. Any failure removes the identity (restarting again) and shreds the bearer file.
3. The T02 exercise (make audit-database-lease-recovery-exercise). Its baseline must capture the post-projection pod UID; never project while the exercise is running.
4. Record the exercise's JSON result (it is the evidence file).
5. make audit-synthetic-sender-cleanup AUDIT_T02_EVIDENCE=<result.json> — refuses unless that evidence names this approval_id, or an explicit AUDIT_SYNTHETIC_SENDER_ABORT_REASON is given for an aborted run. Removes only the exact named identity (refusing if the name carries a different scope), restarts audit-core only if something was removed, shreds the bearer file, and confirms 1/1 Ready. Re-running is a no-op.

If cleanup is missed, audit-core's expires_at enforcement makes the identity inert at window end; the attended cleanup is still required to remove it.

The driver runs operator-side. audit-core's NetworkPolicies admit only named namespaces, so reach the pod with kubectl -n audit-core port-forward deploy/audit-core <local-port>:8080 and set AUDIT_T02_BASE_URL to that local URL; record the base URL and the driver's source/tenant (which must equal the receipt's synthetic_sender block) in the window notes. Projection and cleanup each restart audit-core once (Recreate, no surge), so production senders see two short retryable gaps per window; announce them with the window.

If project fails after the registry write may already have reached OpenBao (for example a transport error on the write itself), treat the identity as possibly present: run cleanup --abort-reason "<reason>" before leaving the window. The lane's --token-file is the attended operator token for the registry write; it must come from the contained ops-warden attended login accepted under RPF-WP-0017 (isolated 0600 helper state, removed afterwards), not from a long-lived token file.

Attended sequence

Keep one terminal attached from baseline through recovery. A value-handling helper used in the approved window must suppress OpenBao/Kubernetes response bodies and clear all transient JSON from memory or mode-0600 temporary files.

1. Record the audit-core pod UID, restart count, database Secret resourceVersion, /healthz status, /readyz status, CNPG readiness, ExternalSecret condition, and ClusterSecretStore condition.
2. Start bounded retrying synthetic traffic. Its evidence records only request identifier, attempt time, response class, and final accepted/duplicate result—never the bearer or payload.
3. Inside the attended process, list only lease handles below sys/leases/lookup/database/creds/audit-core-runtime. OpenBao's lease lookup returns issue/expiry/TTL metadata but not the leased username, so exact selection is deliberately stricter: there must be exactly one live handle under that exact runtime prefix, and its issue/expiry interval must contain the current database Secret refresh time. Abort for database-owner disambiguation if zero or multiple handles exist. Retain only a one-way handle fingerprint in evidence and clear the lookup response immediately.
4. Recheck that the Secret resourceVersion, pod UID, and selected handle have not changed. Revoke only that handle. Do not revoke the role prefix and do not revoke the ESO parent token.
5. Prove the bounded failure state: /healthz remains 200, /readyz becomes 503, at least one synthetic attempt receives retryable 503, pod UID is unchanged, and restart count is unchanged. Abort if liveness fails, the pod restarts, a non-synthetic request is involved, or 503 is not retryable.
6. Annotate only ExternalSecret audit-core/audit-core-database to force reconciliation. Wait for Ready=True/SecretSynced, a changed Secret resourceVersion, and Kubernetes' mounted ..data generation to advance. Do not restart or roll out audit-core.
7. Prove recovery: /readyz returns 200, the retrying synthetic request reaches accepted or duplicate, the pre-exercise event remains readable, pod UID is unchanged, and restart count is unchanged.
8. Stop the load, confirm the predecessor lease is revoked, record the evidence, then remove the identity with the synthetic sender lane cleanup (which restarts audit-core after the evidence is captured). The replacement remains managed by ESO and expires through its normal database role TTL.

Only inside the approved window, with the receipt and reviewed driver:

make audit-database-lease-recovery-exercise \
  AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json \
  AUDIT_DATABASE_LEASE_DRIVER=/path/to/approved-load-driver

The exact confirmation string is embedded in the target. The harness rechecks the Secret/pod/lease immediately before revocation, requires at least five minutes of remaining lease TTL to avoid the normal ESO refresh race, performs best-effort forced reconciliation on an interrupted post-revocation run, and never issues a rollout/restart command.

Time bounds and abort path

• Planned outage observation: at most 120 seconds.
• Full exercise: at most 10 minutes.
• If /readyz has not recovered 120 seconds after forced reconciliation, stop load and page both custody and database owners. Do not bounce the pod.
• If ESO is not SecretSynced, check OpenBao seal state and the exact store. Restart External Secrets only for the known post-unseal stale-provider case and only with the platform owner present.
• If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without revocation.

Value-safe evidence

Record timestamps, approved window/load identifiers, owner acknowledgements, lease accessor or one-way handle fingerprint, Secret resource versions, mount generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO conditions, and cleanup outcome. Never record usernames, passwords, DSNs, Secret data, bearer values, request payloads, OpenBao response bodies, or unseal material.