Add audit recovery exercise preflights
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
This commit is contained in:
codex
parent
dca3d87994
commit
bd25f7fa40
7 changed files with 684 additions and 7 deletions
86
docs/audit-core-database-lease-recovery.md
Normal file
86
docs/audit-core-database-lease-recovery.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
# Audit-core restart-free database lease recovery
|
||||
|
||||
This is the review and attended-execution contract for
|
||||
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
|
||||
audit-core causes retryable unavailability and that External Secrets plus the
|
||||
mounted credential reader restore service without replacing or restarting the
|
||||
pod. It does not authorize a live revocation.
|
||||
|
||||
## Fixed scope and owners
|
||||
|
||||
| Boundary | Exact scope | Responsible owner |
|
||||
| --- | --- | --- |
|
||||
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
|
||||
| Lease | `database/creds/audit-core-runtime`; only the lease matching the current mounted username | `railiance-platform` custody operator |
|
||||
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
|
||||
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
|
||||
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |
|
||||
|
||||
The live window requires audit-core and rapp-postgres acknowledgement, a named
|
||||
abort operator, and an approved synthetic-load identifier. Run the read-only
|
||||
gate before requesting that approval:
|
||||
|
||||
```bash
|
||||
python3 scripts/audit-core-recovery-preflight.py database-lease
|
||||
```
|
||||
|
||||
The result must have `automated_checks_passed: true`. It will remain
|
||||
`ready_for_live_execution: false` until the explicit owner/window fields are
|
||||
provided. The helper reads Secret metadata and key names only; it never reads
|
||||
Secret data.
|
||||
|
||||
## Attended sequence
|
||||
|
||||
Keep one terminal attached from baseline through recovery. A value-handling
|
||||
helper used in the approved window must suppress OpenBao/Kubernetes response
|
||||
bodies and clear all transient JSON from memory or mode-`0600` temporary files.
|
||||
|
||||
1. Record the audit-core pod UID, restart count, database Secret
|
||||
`resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
|
||||
ExternalSecret condition, and ClusterSecretStore condition.
|
||||
2. Start bounded retrying synthetic traffic. Its evidence records only request
|
||||
identifier, attempt time, response class, and final accepted/duplicate
|
||||
result—never the bearer or payload.
|
||||
3. Inside the attended process, read the current mounted username without
|
||||
emitting it. List only lease handles below
|
||||
`sys/leases/lookup/database/creds/audit-core-runtime`, look each up, and
|
||||
retain the one handle whose lease username matches. Abort unless exactly one
|
||||
live handle matches. Clear every lookup response and username immediately.
|
||||
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
|
||||
not changed. Revoke only that handle. Do not revoke the role prefix and do
|
||||
not revoke the ESO parent token.
|
||||
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
|
||||
503, at least one synthetic attempt receives retryable 503, pod UID is
|
||||
unchanged, and restart count is unchanged. Abort if liveness fails, the pod
|
||||
restarts, a non-synthetic request is involved, or 503 is not retryable.
|
||||
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
|
||||
reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
|
||||
`resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
|
||||
Do not restart or roll out audit-core.
|
||||
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
|
||||
accepted or duplicate, the pre-exercise event remains readable, pod UID is
|
||||
unchanged, and restart count is unchanged.
|
||||
8. Stop the load, remove its separately approved fixture/identity, and confirm
|
||||
the predecessor lease is revoked. The replacement remains managed by ESO
|
||||
and expires through its normal database role TTL.
|
||||
|
||||
## Time bounds and abort path
|
||||
|
||||
- Planned outage observation: at most 120 seconds.
|
||||
- Full exercise: at most 10 minutes.
|
||||
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
|
||||
stop load and page both custody and database owners. Do not bounce the pod.
|
||||
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
|
||||
Restart External Secrets only for the known post-unseal stale-provider case
|
||||
and only with the platform owner present.
|
||||
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
|
||||
revocation.
|
||||
|
||||
## Value-safe evidence
|
||||
|
||||
Record timestamps, approved window/load identifiers, owner acknowledgements,
|
||||
lease accessor or one-way handle fingerprint, Secret resource versions, mount
|
||||
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
|
||||
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
|
||||
Secret data, bearer values, request payloads, OpenBao response bodies, or
|
||||
unseal material.
|
||||
121
docs/railiance01-coordinated-reboot.md
Normal file
121
docs/railiance01-coordinated-reboot.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# Coordinated `railiance01` reboot and recovery
|
||||
|
||||
This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a
|
||||
single-node platform outage, not a Kubernetes rolling operation and not an HA
|
||||
claim. This document does not authorize a reboot.
|
||||
|
||||
## Responsibility map
|
||||
|
||||
| Phase | Accountable owner | Required evidence |
|
||||
| --- | --- | --- |
|
||||
| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator |
|
||||
| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns |
|
||||
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled |
|
||||
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
|
||||
| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary |
|
||||
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
|
||||
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |
|
||||
|
||||
The last observed reboot on 2026-08-16 showed the expected dependency order:
|
||||
the host and k3s returned, audit-core was unready for about 40 seconds while
|
||||
DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted
|
||||
Secrets remained, but new leases could not refresh until unseal.
|
||||
|
||||
## Fail-closed preflight
|
||||
|
||||
Run the value-safe, read-only gate:
|
||||
|
||||
```bash
|
||||
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||
```
|
||||
|
||||
The automated checks require one Ready node, active k3s, healthy 1/1
|
||||
`platform-pg`, continuous archiving, a successful backup no older than 26
|
||||
hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and
|
||||
the reviewed 1/1 Ready audit-core image. The following are human gates and make
|
||||
`ready_for_live_execution` remain false until supplied:
|
||||
|
||||
- approved window id and acknowledgements from all five owners;
|
||||
- a named attended driver/abort operator;
|
||||
- independent provider-console access, not SSH alone;
|
||||
- a current non-secret OpenBao Raft snapshot evidence file;
|
||||
- explicit availability of two of three Shamir shares through their approved
|
||||
out-of-band custody paths.
|
||||
|
||||
Do not reuse old restore-drill templates as current snapshot evidence. Do not
|
||||
put a snapshot, unseal share, token, password, or Secret response in Git or
|
||||
State Hub.
|
||||
|
||||
## Pre-reboot hold point
|
||||
|
||||
1. Announce the outage and stop new deployment, migration, credential-change,
|
||||
backup-restore, and security-engagement mutations on `railiance01`.
|
||||
2. Confirm no temporary Whitehat or other expiring projection will cross the
|
||||
window. Clean it first if present.
|
||||
3. Capture the read-only preflight result and audit-core chain-head attestation.
|
||||
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
|
||||
is healthy, and the PVC is Bound. If the daily backup is stale, take and
|
||||
verify an on-demand backup before continuing.
|
||||
5. Create and escrow a fresh OpenBao Raft snapshot through the approved
|
||||
encrypted backup lane. Record only its evidence receipt and verification.
|
||||
6. Have both Shamir share custodians present. Prove provider-console access and
|
||||
SSH access. Record the exact restart and abort operators.
|
||||
7. Re-run the preflight with the approval and attestation flags. The result
|
||||
must say `ready_for_live_execution: true`. Pause for the final operator
|
||||
go/no-go; this is the last reversible point.
|
||||
|
||||
## Reboot and recovery order
|
||||
|
||||
1. `railiance-infra` performs one ordinary host reboot and retains the provider
|
||||
console. Do not reinstall the OS or k3s.
|
||||
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
|
||||
If SSH does not return, use the provider console; do not change firewall
|
||||
declarations as an ad hoc recovery.
|
||||
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod
|
||||
are running. Application readiness may still be false at this point.
|
||||
4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity,
|
||||
primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap.
|
||||
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
|
||||
non-recorded share ceremony. Run `make openbao-status` and
|
||||
`make openbao-verify-post-unseal` after it reports initialized and unsealed.
|
||||
6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and
|
||||
the audit-core sender/database ExternalSecrets are SecretSynced. If they
|
||||
remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
|
||||
the platform owner may restart only Deployment
|
||||
`external-secrets/external-secrets`, then recheck reconciliation.
|
||||
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
|
||||
liveness boundary; wait for `/readyz` to recover naturally after DNS,
|
||||
PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
|
||||
8. Run audit-core integrity verification against the pre-reboot chain head and
|
||||
one separately approved synthetic E2. Confirm no chain break, the prior
|
||||
marker remains readable, and the new event is accepted exactly once.
|
||||
9. Re-run the read-only preflight, end the mutation freeze, and notify all
|
||||
owners with value-safe evidence.
|
||||
|
||||
## Stop, escalation, and recovery boundaries
|
||||
|
||||
- Before reboot: abort for missing approval, owner, console access, unseal
|
||||
quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
|
||||
or any overlapping live mutation.
|
||||
- After reboot: retain the outage and escalate for a missing host, node/PVC
|
||||
identity change, CNPG re-bootstrap, OpenBao initialization loss, failed
|
||||
unseal, audit chain break, or any evidence of data loss.
|
||||
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
|
||||
recover PostgreSQL in place, delete a PVC, or weaken network policy as a
|
||||
shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to
|
||||
`railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`;
|
||||
PostgreSQL restoration always creates a new CNPG Cluster under
|
||||
`rapp-postgres`' restore procedure.
|
||||
|
||||
## Evidence contract
|
||||
|
||||
Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id
|
||||
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
|
||||
OpenBao initialized/sealed booleans and version, resource readiness conditions,
|
||||
Secret resource versions and key names, HTTP status sequence, audit chain-head
|
||||
hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts,
|
||||
and final notifications.
|
||||
|
||||
Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database
|
||||
credentials/usernames/DSNs, Secret data, sender bearer values, private keys,
|
||||
provider-console credentials, request payloads, or decrypted snapshots.
|
||||
Loading…
Add table
Add a link
Reference in a new issue