defb2a556d
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
122 lines
6.4 KiB
Markdown
122 lines
6.4 KiB
Markdown
# Audit-core restart-free database lease recovery
|
|
|
|
This is the review and attended-execution contract for
|
|
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
|
|
audit-core causes retryable unavailability and that External Secrets plus the
|
|
mounted credential reader restore service without replacing or restarting the
|
|
pod. It does not authorize a live revocation.
|
|
|
|
## Fixed scope and owners
|
|
|
|
| Boundary | Exact scope | Responsible owner |
|
|
| --- | --- | --- |
|
|
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
|
|
| Lease | `database/creds/audit-core-runtime`; exactly one live handle whose issue time coheres with the current ExternalSecret refresh | `railiance-platform` custody operator |
|
|
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
|
|
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
|
|
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |
|
|
|
|
The live window requires audit-core and rapp-postgres acknowledgement, a named
|
|
abort operator, and an approved synthetic-load identifier. Run the read-only
|
|
gate before requesting that approval:
|
|
|
|
```bash
|
|
python3 scripts/audit-core-recovery-preflight.py database-lease
|
|
```
|
|
|
|
The result must have `automated_checks_passed: true`. It will remain
|
|
`ready_for_live_execution: false` until the explicit owner/window fields are
|
|
provided. The helper reads Secret metadata and key names only; it never reads
|
|
Secret data.
|
|
|
|
The live harness is
|
|
`scripts/audit-core-database-lease-recovery.py`. Start from the deliberately
|
|
pending `docs/audit-core-database-lease-approval.example.json`; the actual
|
|
value-safe receipt must name a ≤15-minute window, approval id, abort operator,
|
|
State Hub acknowledgement ids from audit-core/rapp-postgres/platform, and an
|
|
approved synthetic-load contract plus driver revision. Validate without
|
|
mutation:
|
|
|
|
```bash
|
|
make audit-database-lease-approval-check \
|
|
AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json
|
|
```
|
|
|
|
The separately approved load driver is an executable that accepts `baseline`,
|
|
`expect-unavailable`, `expect-recovered`, and `cleanup`, plus
|
|
`--contract-id ID`. Each call returns only its exact value-safe JSON evidence
|
|
shape. Unexpected keys—including a bearer accidentally returned as `token`—
|
|
make the harness fail without echoing driver output.
|
|
|
|
## Attended sequence
|
|
|
|
Keep one terminal attached from baseline through recovery. A value-handling
|
|
helper used in the approved window must suppress OpenBao/Kubernetes response
|
|
bodies and clear all transient JSON from memory or mode-`0600` temporary files.
|
|
|
|
1. Record the audit-core pod UID, restart count, database Secret
|
|
`resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
|
|
ExternalSecret condition, and ClusterSecretStore condition.
|
|
2. Start bounded retrying synthetic traffic. Its evidence records only request
|
|
identifier, attempt time, response class, and final accepted/duplicate
|
|
result—never the bearer or payload.
|
|
3. Inside the attended process, list only lease handles below
|
|
`sys/leases/lookup/database/creds/audit-core-runtime`. OpenBao's lease lookup
|
|
returns issue/expiry/TTL metadata but not the leased username, so exact
|
|
selection is deliberately stricter: there must be exactly one live handle
|
|
under that exact runtime prefix, and its issue/expiry interval must contain
|
|
the current database Secret refresh time. Abort for database-owner
|
|
disambiguation if zero or multiple handles exist. Retain only a one-way
|
|
handle fingerprint in evidence and clear the lookup response immediately.
|
|
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
|
|
not changed. Revoke only that handle. Do not revoke the role prefix and do
|
|
not revoke the ESO parent token.
|
|
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
|
|
503, at least one synthetic attempt receives retryable 503, pod UID is
|
|
unchanged, and restart count is unchanged. Abort if liveness fails, the pod
|
|
restarts, a non-synthetic request is involved, or 503 is not retryable.
|
|
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
|
|
reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
|
|
`resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
|
|
Do not restart or roll out audit-core.
|
|
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
|
|
accepted or duplicate, the pre-exercise event remains readable, pod UID is
|
|
unchanged, and restart count is unchanged.
|
|
8. Stop the load, remove its separately approved fixture/identity, and confirm
|
|
the predecessor lease is revoked. The replacement remains managed by ESO
|
|
and expires through its normal database role TTL.
|
|
|
|
Only inside the approved window, with the receipt and reviewed driver:
|
|
|
|
```bash
|
|
make audit-database-lease-recovery-exercise \
|
|
AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json \
|
|
AUDIT_DATABASE_LEASE_DRIVER=/path/to/approved-load-driver
|
|
```
|
|
|
|
The exact confirmation string is embedded in the target. The harness rechecks
|
|
the Secret/pod/lease immediately before revocation, requires at least five
|
|
minutes of remaining lease TTL to avoid the normal ESO refresh race, performs
|
|
best-effort forced reconciliation on an interrupted post-revocation run, and
|
|
never issues a rollout/restart command.
|
|
|
|
## Time bounds and abort path
|
|
|
|
- Planned outage observation: at most 120 seconds.
|
|
- Full exercise: at most 10 minutes.
|
|
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
|
|
stop load and page both custody and database owners. Do not bounce the pod.
|
|
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
|
|
Restart External Secrets only for the known post-unseal stale-provider case
|
|
and only with the platform owner present.
|
|
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
|
|
revocation.
|
|
|
|
## Value-safe evidence
|
|
|
|
Record timestamps, approved window/load identifiers, owner acknowledgements,
|
|
lease accessor or one-way handle fingerprint, Secret resource versions, mount
|
|
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
|
|
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
|
|
Secret data, bearer values, request payloads, OpenBao response bodies, or
|
|
unseal material.
|