Add audit recovery exercise preflights
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
This commit is contained in:
codex
parent
dca3d87994
commit
bd25f7fa40
7 changed files with 684 additions and 7 deletions
8
Makefile
8
Makefile
|
|
@ -475,5 +475,11 @@ audit-whitehat-e2-project: ## Project two exact mount-only credentials during th
|
||||||
audit-whitehat-e2-cleanup: ## Remove the exact engagement credentials and reload audit-core
|
audit-whitehat-e2-cleanup: ## Remove the exact engagement credentials and reload audit-core
|
||||||
python3 scripts/audit-core-whitehat-e2-credentials.py cleanup --confirm WH-ENG-20260822-AUDIT-E2-01:attended
|
python3 scripts/audit-core-whitehat-e2-credentials.py cleanup --confirm WH-ENG-20260822-AUDIT-E2-01:attended
|
||||||
|
|
||||||
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup apps-pg-isolation-probe
|
audit-database-lease-recovery-preflight: ## Read-only gate for the restart-free database lease exercise
|
||||||
|
python3 scripts/audit-core-recovery-preflight.py database-lease
|
||||||
|
|
||||||
|
railiance01-reboot-preflight: ## Read-only gate for the coordinated single-node reboot
|
||||||
|
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||||
|
|
||||||
|
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight railiance01-reboot-preflight apps-pg-isolation-probe
|
||||||
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
|
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
|
||||||
|
|
|
||||||
|
|
@ -98,9 +98,9 @@
|
||||||
| task | RAILIANCE-WP-0023-T02 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
| task | RAILIANCE-WP-0023-T02 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
||||||
| task | RAILIANCE-WP-0023-T03 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
| task | RAILIANCE-WP-0023-T03 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
||||||
| task | RAILIANCE-WP-0024-T01 | wait | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
| task | RAILIANCE-WP-0024-T01 | wait | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||||
| task | RAILIANCE-WP-0024-T02 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
| task | RAILIANCE-WP-0024-T02 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||||
| task | RAILIANCE-WP-0024-T03 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
| task | RAILIANCE-WP-0024-T03 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||||
| task | RAILIANCE-WP-0024-T04 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
| task | RAILIANCE-WP-0024-T04 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||||
| task | RPF-WP-0018-T01 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
| task | RPF-WP-0018-T01 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||||
| task | RPF-WP-0018-T02 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
| task | RPF-WP-0018-T02 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||||
| task | RPF-WP-0018-T03 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
| task | RPF-WP-0018-T03 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||||
|
|
|
||||||
86
docs/audit-core-database-lease-recovery.md
Normal file
86
docs/audit-core-database-lease-recovery.md
Normal file
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Audit-core restart-free database lease recovery
|
||||||
|
|
||||||
|
This is the review and attended-execution contract for
|
||||||
|
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
|
||||||
|
audit-core causes retryable unavailability and that External Secrets plus the
|
||||||
|
mounted credential reader restore service without replacing or restarting the
|
||||||
|
pod. It does not authorize a live revocation.
|
||||||
|
|
||||||
|
## Fixed scope and owners
|
||||||
|
|
||||||
|
| Boundary | Exact scope | Responsible owner |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
|
||||||
|
| Lease | `database/creds/audit-core-runtime`; only the lease matching the current mounted username | `railiance-platform` custody operator |
|
||||||
|
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
|
||||||
|
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
|
||||||
|
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |
|
||||||
|
|
||||||
|
The live window requires audit-core and rapp-postgres acknowledgement, a named
|
||||||
|
abort operator, and an approved synthetic-load identifier. Run the read-only
|
||||||
|
gate before requesting that approval:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit-core-recovery-preflight.py database-lease
|
||||||
|
```
|
||||||
|
|
||||||
|
The result must have `automated_checks_passed: true`. It will remain
|
||||||
|
`ready_for_live_execution: false` until the explicit owner/window fields are
|
||||||
|
provided. The helper reads Secret metadata and key names only; it never reads
|
||||||
|
Secret data.
|
||||||
|
|
||||||
|
## Attended sequence
|
||||||
|
|
||||||
|
Keep one terminal attached from baseline through recovery. A value-handling
|
||||||
|
helper used in the approved window must suppress OpenBao/Kubernetes response
|
||||||
|
bodies and clear all transient JSON from memory or mode-`0600` temporary files.
|
||||||
|
|
||||||
|
1. Record the audit-core pod UID, restart count, database Secret
|
||||||
|
`resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
|
||||||
|
ExternalSecret condition, and ClusterSecretStore condition.
|
||||||
|
2. Start bounded retrying synthetic traffic. Its evidence records only request
|
||||||
|
identifier, attempt time, response class, and final accepted/duplicate
|
||||||
|
result—never the bearer or payload.
|
||||||
|
3. Inside the attended process, read the current mounted username without
|
||||||
|
emitting it. List only lease handles below
|
||||||
|
`sys/leases/lookup/database/creds/audit-core-runtime`, look each up, and
|
||||||
|
retain the one handle whose lease username matches. Abort unless exactly one
|
||||||
|
live handle matches. Clear every lookup response and username immediately.
|
||||||
|
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
|
||||||
|
not changed. Revoke only that handle. Do not revoke the role prefix and do
|
||||||
|
not revoke the ESO parent token.
|
||||||
|
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
|
||||||
|
503, at least one synthetic attempt receives retryable 503, pod UID is
|
||||||
|
unchanged, and restart count is unchanged. Abort if liveness fails, the pod
|
||||||
|
restarts, a non-synthetic request is involved, or 503 is not retryable.
|
||||||
|
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
|
||||||
|
reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
|
||||||
|
`resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
|
||||||
|
Do not restart or roll out audit-core.
|
||||||
|
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
|
||||||
|
accepted or duplicate, the pre-exercise event remains readable, pod UID is
|
||||||
|
unchanged, and restart count is unchanged.
|
||||||
|
8. Stop the load, remove its separately approved fixture/identity, and confirm
|
||||||
|
the predecessor lease is revoked. The replacement remains managed by ESO
|
||||||
|
and expires through its normal database role TTL.
|
||||||
|
|
||||||
|
## Time bounds and abort path
|
||||||
|
|
||||||
|
- Planned outage observation: at most 120 seconds.
|
||||||
|
- Full exercise: at most 10 minutes.
|
||||||
|
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
|
||||||
|
stop load and page both custody and database owners. Do not bounce the pod.
|
||||||
|
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
|
||||||
|
Restart External Secrets only for the known post-unseal stale-provider case
|
||||||
|
and only with the platform owner present.
|
||||||
|
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
|
||||||
|
revocation.
|
||||||
|
|
||||||
|
## Value-safe evidence
|
||||||
|
|
||||||
|
Record timestamps, approved window/load identifiers, owner acknowledgements,
|
||||||
|
lease accessor or one-way handle fingerprint, Secret resource versions, mount
|
||||||
|
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
|
||||||
|
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
|
||||||
|
Secret data, bearer values, request payloads, OpenBao response bodies, or
|
||||||
|
unseal material.
|
||||||
121
docs/railiance01-coordinated-reboot.md
Normal file
121
docs/railiance01-coordinated-reboot.md
Normal file
|
|
@ -0,0 +1,121 @@
|
||||||
|
# Coordinated `railiance01` reboot and recovery
|
||||||
|
|
||||||
|
This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a
|
||||||
|
single-node platform outage, not a Kubernetes rolling operation and not an HA
|
||||||
|
claim. This document does not authorize a reboot.
|
||||||
|
|
||||||
|
## Responsibility map
|
||||||
|
|
||||||
|
| Phase | Accountable owner | Required evidence |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator |
|
||||||
|
| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns |
|
||||||
|
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled |
|
||||||
|
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
|
||||||
|
| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary |
|
||||||
|
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
|
||||||
|
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |
|
||||||
|
|
||||||
|
The last observed reboot on 2026-08-16 showed the expected dependency order:
|
||||||
|
the host and k3s returned, audit-core was unready for about 40 seconds while
|
||||||
|
DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted
|
||||||
|
Secrets remained, but new leases could not refresh until unseal.
|
||||||
|
|
||||||
|
## Fail-closed preflight
|
||||||
|
|
||||||
|
Run the value-safe, read-only gate:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
The automated checks require one Ready node, active k3s, healthy 1/1
|
||||||
|
`platform-pg`, continuous archiving, a successful backup no older than 26
|
||||||
|
hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and
|
||||||
|
the reviewed 1/1 Ready audit-core image. The following are human gates and make
|
||||||
|
`ready_for_live_execution` remain false until supplied:
|
||||||
|
|
||||||
|
- approved window id and acknowledgements from all five owners;
|
||||||
|
- a named attended driver/abort operator;
|
||||||
|
- independent provider-console access, not SSH alone;
|
||||||
|
- a current non-secret OpenBao Raft snapshot evidence file;
|
||||||
|
- explicit availability of two of three Shamir shares through their approved
|
||||||
|
out-of-band custody paths.
|
||||||
|
|
||||||
|
Do not reuse old restore-drill templates as current snapshot evidence. Do not
|
||||||
|
put a snapshot, unseal share, token, password, or Secret response in Git or
|
||||||
|
State Hub.
|
||||||
|
|
||||||
|
## Pre-reboot hold point
|
||||||
|
|
||||||
|
1. Announce the outage and stop new deployment, migration, credential-change,
|
||||||
|
backup-restore, and security-engagement mutations on `railiance01`.
|
||||||
|
2. Confirm no temporary Whitehat or other expiring projection will cross the
|
||||||
|
window. Clean it first if present.
|
||||||
|
3. Capture the read-only preflight result and audit-core chain-head attestation.
|
||||||
|
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
|
||||||
|
is healthy, and the PVC is Bound. If the daily backup is stale, take and
|
||||||
|
verify an on-demand backup before continuing.
|
||||||
|
5. Create and escrow a fresh OpenBao Raft snapshot through the approved
|
||||||
|
encrypted backup lane. Record only its evidence receipt and verification.
|
||||||
|
6. Have both Shamir share custodians present. Prove provider-console access and
|
||||||
|
SSH access. Record the exact restart and abort operators.
|
||||||
|
7. Re-run the preflight with the approval and attestation flags. The result
|
||||||
|
must say `ready_for_live_execution: true`. Pause for the final operator
|
||||||
|
go/no-go; this is the last reversible point.
|
||||||
|
|
||||||
|
## Reboot and recovery order
|
||||||
|
|
||||||
|
1. `railiance-infra` performs one ordinary host reboot and retains the provider
|
||||||
|
console. Do not reinstall the OS or k3s.
|
||||||
|
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
|
||||||
|
If SSH does not return, use the provider console; do not change firewall
|
||||||
|
declarations as an ad hoc recovery.
|
||||||
|
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod
|
||||||
|
are running. Application readiness may still be false at this point.
|
||||||
|
4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity,
|
||||||
|
primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap.
|
||||||
|
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
|
||||||
|
non-recorded share ceremony. Run `make openbao-status` and
|
||||||
|
`make openbao-verify-post-unseal` after it reports initialized and unsealed.
|
||||||
|
6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and
|
||||||
|
the audit-core sender/database ExternalSecrets are SecretSynced. If they
|
||||||
|
remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
|
||||||
|
the platform owner may restart only Deployment
|
||||||
|
`external-secrets/external-secrets`, then recheck reconciliation.
|
||||||
|
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
|
||||||
|
liveness boundary; wait for `/readyz` to recover naturally after DNS,
|
||||||
|
PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
|
||||||
|
8. Run audit-core integrity verification against the pre-reboot chain head and
|
||||||
|
one separately approved synthetic E2. Confirm no chain break, the prior
|
||||||
|
marker remains readable, and the new event is accepted exactly once.
|
||||||
|
9. Re-run the read-only preflight, end the mutation freeze, and notify all
|
||||||
|
owners with value-safe evidence.
|
||||||
|
|
||||||
|
## Stop, escalation, and recovery boundaries
|
||||||
|
|
||||||
|
- Before reboot: abort for missing approval, owner, console access, unseal
|
||||||
|
quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
|
||||||
|
or any overlapping live mutation.
|
||||||
|
- After reboot: retain the outage and escalate for a missing host, node/PVC
|
||||||
|
identity change, CNPG re-bootstrap, OpenBao initialization loss, failed
|
||||||
|
unseal, audit chain break, or any evidence of data loss.
|
||||||
|
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
|
||||||
|
recover PostgreSQL in place, delete a PVC, or weaken network policy as a
|
||||||
|
shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to
|
||||||
|
`railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`;
|
||||||
|
PostgreSQL restoration always creates a new CNPG Cluster under
|
||||||
|
`rapp-postgres`' restore procedure.
|
||||||
|
|
||||||
|
## Evidence contract
|
||||||
|
|
||||||
|
Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id
|
||||||
|
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
|
||||||
|
OpenBao initialized/sealed booleans and version, resource readiness conditions,
|
||||||
|
Secret resource versions and key names, HTTP status sequence, audit chain-head
|
||||||
|
hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts,
|
||||||
|
and final notifications.
|
||||||
|
|
||||||
|
Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database
|
||||||
|
credentials/usernames/DSNs, Secret data, sender bearer values, private keys,
|
||||||
|
provider-console credentials, request payloads, or decrypted snapshots.
|
||||||
331
scripts/audit-core-recovery-preflight.py
Executable file
331
scripts/audit-core-recovery-preflight.py
Executable file
|
|
@ -0,0 +1,331 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Value-safe preflight for RAILIANCE-WP-0024 recovery exercises.
|
||||||
|
|
||||||
|
This helper is deliberately read-only. It never reads Kubernetes Secret data,
|
||||||
|
database usernames/passwords, OpenBao lease payloads, or unseal material. Live
|
||||||
|
lease revocation and a host reboot remain separate, attended actions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
EXPECTED_IMAGE = (
|
||||||
|
"forgejo.coulomb.social/coulomb/audit-core@"
|
||||||
|
"sha256:c2fe39a0185b99be3fc0cb14d2de69772b8e66e20490097c9d11d90cc39719a6"
|
||||||
|
)
|
||||||
|
EXPECTED_DB_KEYS = {"username", "password", "host", "port", "dbname"}
|
||||||
|
|
||||||
|
|
||||||
|
class PreflightError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Remote:
|
||||||
|
def __init__(self, host: str) -> None:
|
||||||
|
self.host = host
|
||||||
|
|
||||||
|
def run(self, command: list[str], *, label: str) -> str:
|
||||||
|
remote_command = " ".join(shlex.quote(part) for part in command)
|
||||||
|
completed = subprocess.run(
|
||||||
|
["ssh", "-o", "BatchMode=yes", self.host, remote_command],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
if completed.returncode != 0:
|
||||||
|
# Remote output can include provider or application detail that is
|
||||||
|
# inappropriate for durable evidence. Report only the failed step.
|
||||||
|
raise PreflightError(f"{label} failed (exit {completed.returncode})")
|
||||||
|
return completed.stdout.strip()
|
||||||
|
|
||||||
|
def kubectl(self, args: list[str], *, label: str) -> str:
|
||||||
|
return self.run(["kubectl", *args], label=label)
|
||||||
|
|
||||||
|
def kubectl_json(self, args: list[str], *, label: str) -> dict[str, Any]:
|
||||||
|
raw = self.kubectl([*args, "-o", "json"], label=label)
|
||||||
|
try:
|
||||||
|
value = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise PreflightError(f"{label} returned invalid JSON") from exc
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise PreflightError(f"{label} did not return an object")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def condition(resource: dict[str, Any], kind: str) -> str | None:
|
||||||
|
for item in resource.get("status", {}).get("conditions", []):
|
||||||
|
if item.get("type") == kind:
|
||||||
|
return item.get("status")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def rfc3339(value: str) -> datetime:
|
||||||
|
return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def latest_completed_backup(backups: dict[str, Any], now: datetime) -> dict[str, Any]:
|
||||||
|
completed = [
|
||||||
|
item
|
||||||
|
for item in backups.get("items", [])
|
||||||
|
if item.get("status", {}).get("phase") == "completed"
|
||||||
|
and item.get("status", {}).get("stoppedAt")
|
||||||
|
]
|
||||||
|
if not completed:
|
||||||
|
raise PreflightError("no completed platform-pg backup is visible")
|
||||||
|
latest = max(completed, key=lambda item: item["status"]["stoppedAt"])
|
||||||
|
stopped = rfc3339(latest["status"]["stoppedAt"])
|
||||||
|
return {
|
||||||
|
"name": latest["metadata"]["name"],
|
||||||
|
"stopped_at": stopped.isoformat().replace("+00:00", "Z"),
|
||||||
|
"age_hours": round((now - stopped).total_seconds() / 3600, 2),
|
||||||
|
"method": latest.get("spec", {}).get("method"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def resource_condition(resource: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
conditions = resource.get("status", {}).get("conditions", [])
|
||||||
|
current = conditions[0] if conditions else {}
|
||||||
|
return {
|
||||||
|
"ready": current.get("status") == "True",
|
||||||
|
"reason": current.get("reason"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def secret_key_names(description: str) -> set[str]:
|
||||||
|
"""Extract names from ``kubectl describe secret`` without reading data."""
|
||||||
|
keys: set[str] = set()
|
||||||
|
for line in description.splitlines():
|
||||||
|
match = re.match(r"^([^\s:]+):\s+\d+ bytes$", line.strip())
|
||||||
|
if match:
|
||||||
|
keys.add(match.group(1))
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def common_state(remote: Remote, now: datetime) -> dict[str, Any]:
|
||||||
|
node_list = remote.kubectl_json(["get", "nodes"], label="read node state")
|
||||||
|
nodes = node_list.get("items", [])
|
||||||
|
if len(nodes) != 1:
|
||||||
|
raise PreflightError(f"expected one railiance01 node, observed {len(nodes)}")
|
||||||
|
node = nodes[0]
|
||||||
|
|
||||||
|
cluster = remote.kubectl_json(
|
||||||
|
["-n", "databases", "get", "cluster", "platform-pg"],
|
||||||
|
label="read platform-pg state",
|
||||||
|
)
|
||||||
|
deployment = remote.kubectl_json(
|
||||||
|
["-n", "audit-core", "get", "deployment", "audit-core"],
|
||||||
|
label="read audit-core deployment",
|
||||||
|
)
|
||||||
|
pod_list = remote.kubectl_json(
|
||||||
|
["-n", "audit-core", "get", "pods", "-l", "app.kubernetes.io/name=audit-core"],
|
||||||
|
label="read audit-core pod metadata",
|
||||||
|
)
|
||||||
|
pods = pod_list.get("items", [])
|
||||||
|
if len(pods) != 1:
|
||||||
|
raise PreflightError(f"expected one audit-core pod, observed {len(pods)}")
|
||||||
|
pod = pods[0]
|
||||||
|
container_statuses = pod.get("status", {}).get("containerStatuses", [])
|
||||||
|
restart_count = sum(int(item.get("restartCount", 0)) for item in container_statuses)
|
||||||
|
|
||||||
|
stores: dict[str, Any] = {}
|
||||||
|
for name in ("openbao-audit-core", "openbao-audit-core-database"):
|
||||||
|
resource = remote.kubectl_json(
|
||||||
|
["get", "clustersecretstore", name], label=f"read {name} state"
|
||||||
|
)
|
||||||
|
stores[name] = resource_condition(resource)
|
||||||
|
|
||||||
|
external_secrets: dict[str, Any] = {}
|
||||||
|
for name in ("audit-core-database", "audit-core-senders"):
|
||||||
|
resource = remote.kubectl_json(
|
||||||
|
["-n", "audit-core", "get", "externalsecret", name],
|
||||||
|
label=f"read {name} state",
|
||||||
|
)
|
||||||
|
external_secrets[name] = resource_condition(resource)
|
||||||
|
|
||||||
|
bao_raw = remote.kubectl(
|
||||||
|
["-n", "openbao", "exec", "openbao-0", "--", "bao", "status", "-format=json"],
|
||||||
|
label="read OpenBao seal state",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
bao_status = json.loads(bao_raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise PreflightError("OpenBao seal state returned invalid JSON") from exc
|
||||||
|
image = deployment["spec"]["template"]["spec"]["containers"][0]["image"]
|
||||||
|
state = {
|
||||||
|
"captured_at": now.replace(microsecond=0).isoformat().replace("+00:00", "Z"),
|
||||||
|
"node": {
|
||||||
|
"name": node["metadata"]["name"],
|
||||||
|
"ready": condition(node, "Ready") == "True",
|
||||||
|
"k3s_active": remote.run(
|
||||||
|
["systemctl", "is-active", "k3s"], label="read k3s service state"
|
||||||
|
) == "active",
|
||||||
|
"uptime_seconds": float(
|
||||||
|
remote.run(["cat", "/proc/uptime"], label="read host uptime").split()[0]
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"platform_pg": {
|
||||||
|
"ready_instances": cluster.get("status", {}).get("readyInstances", 0),
|
||||||
|
"instances": cluster.get("spec", {}).get("instances", 0),
|
||||||
|
"phase": cluster.get("status", {}).get("phase"),
|
||||||
|
"continuous_archiving": condition(cluster, "ContinuousArchiving") == "True",
|
||||||
|
"last_backup_succeeded": condition(cluster, "LastBackupSucceeded") == "True",
|
||||||
|
},
|
||||||
|
"openbao": {
|
||||||
|
"initialized": bool(bao_status.get("initialized")),
|
||||||
|
"sealed": bool(bao_status.get("sealed")),
|
||||||
|
"seal_type": bao_status.get("type"),
|
||||||
|
"threshold": bao_status.get("t"),
|
||||||
|
"shares": bao_status.get("n"),
|
||||||
|
},
|
||||||
|
"external_secret_stores": stores,
|
||||||
|
"external_secrets": external_secrets,
|
||||||
|
"audit_core": {
|
||||||
|
"image": image,
|
||||||
|
"image_matches_reviewed_digest": image == EXPECTED_IMAGE,
|
||||||
|
"ready_replicas": deployment.get("status", {}).get("readyReplicas", 0),
|
||||||
|
"replicas": deployment.get("spec", {}).get("replicas", 0),
|
||||||
|
"pod_uid": pod["metadata"]["uid"],
|
||||||
|
"restart_count": restart_count,
|
||||||
|
},
|
||||||
|
"secret_values_observed": False,
|
||||||
|
}
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def automated_common_pass(state: dict[str, Any]) -> bool:
|
||||||
|
return all(
|
||||||
|
(
|
||||||
|
state["node"]["ready"],
|
||||||
|
state["node"]["k3s_active"],
|
||||||
|
state["platform_pg"]["ready_instances"] == state["platform_pg"]["instances"] == 1,
|
||||||
|
state["platform_pg"]["continuous_archiving"],
|
||||||
|
state["platform_pg"]["last_backup_succeeded"],
|
||||||
|
state["openbao"]["initialized"],
|
||||||
|
not state["openbao"]["sealed"],
|
||||||
|
all(item["ready"] for item in state["external_secret_stores"].values()),
|
||||||
|
all(item["ready"] for item in state["external_secrets"].values()),
|
||||||
|
state["audit_core"]["image_matches_reviewed_digest"],
|
||||||
|
state["audit_core"]["ready_replicas"] == state["audit_core"]["replicas"] == 1,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def database_lease_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
state = common_state(remote, now)
|
||||||
|
secret_rv = remote.kubectl(
|
||||||
|
["-n", "audit-core", "get", "secret", "audit-core-database", "-o", "jsonpath={.metadata.resourceVersion}"],
|
||||||
|
label="read database Secret metadata",
|
||||||
|
)
|
||||||
|
description = remote.kubectl(
|
||||||
|
["-n", "audit-core", "describe", "secret", "audit-core-database"],
|
||||||
|
label="read database Secret key names",
|
||||||
|
)
|
||||||
|
keys = secret_key_names(description)
|
||||||
|
health_code = remote.kubectl(
|
||||||
|
["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/healthz", timeout=3).status)'],
|
||||||
|
label="probe audit-core health",
|
||||||
|
)
|
||||||
|
ready_code = remote.kubectl(
|
||||||
|
["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/readyz", timeout=3).status)'],
|
||||||
|
label="probe audit-core readiness",
|
||||||
|
)
|
||||||
|
gates = {
|
||||||
|
"approved_window_id_recorded": bool(args.approved_window_id),
|
||||||
|
"audit_core_owner_acknowledged": args.audit_core_owner_ack,
|
||||||
|
"rapp_postgres_owner_acknowledged": args.rapp_postgres_owner_ack,
|
||||||
|
"approved_synthetic_load_contract": bool(args.synthetic_load_id),
|
||||||
|
"attended_abort_operator_named": bool(args.abort_operator),
|
||||||
|
}
|
||||||
|
automated = automated_common_pass(state) and keys == EXPECTED_DB_KEYS and health_code == "200" and ready_code == "200"
|
||||||
|
return {
|
||||||
|
"procedure": "audit-core-database-lease-recovery",
|
||||||
|
"preflight_only": True,
|
||||||
|
"automated_checks_passed": automated,
|
||||||
|
"ready_for_live_execution": automated and all(gates.values()),
|
||||||
|
"operator_gates": gates,
|
||||||
|
"baseline": {
|
||||||
|
"database_secret_resource_version": secret_rv,
|
||||||
|
"database_secret_keys": sorted(keys),
|
||||||
|
"health_status": int(health_code),
|
||||||
|
"readiness_status": int(ready_code),
|
||||||
|
"pod_uid": state["audit_core"]["pod_uid"],
|
||||||
|
"restart_count": state["audit_core"]["restart_count"],
|
||||||
|
},
|
||||||
|
"state": state,
|
||||||
|
"secret_values_observed": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def reboot_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
state = common_state(remote, now)
|
||||||
|
backups = remote.kubectl_json(
|
||||||
|
["-n", "databases", "get", "backup", "-l", "cnpg.io/cluster=platform-pg"],
|
||||||
|
label="read platform-pg backup metadata",
|
||||||
|
)
|
||||||
|
latest = latest_completed_backup(backups, now)
|
||||||
|
snapshot_evidence = Path(args.openbao_snapshot_evidence).resolve() if args.openbao_snapshot_evidence else None
|
||||||
|
gates = {
|
||||||
|
"approved_window_id_recorded": bool(args.approved_window_id),
|
||||||
|
"openbao_snapshot_evidence_present": bool(snapshot_evidence and snapshot_evidence.is_file()),
|
||||||
|
"unseal_quorum_attested": args.unseal_quorum_attested,
|
||||||
|
"provider_console_access_attested": args.provider_console_attested,
|
||||||
|
"host_cluster_platform_database_audit_owners_acknowledged": args.all_owners_ack,
|
||||||
|
"attended_abort_operator_named": bool(args.abort_operator),
|
||||||
|
}
|
||||||
|
automated = automated_common_pass(state) and latest["age_hours"] <= args.max_backup_age_hours
|
||||||
|
return {
|
||||||
|
"procedure": "railiance01-coordinated-reboot",
|
||||||
|
"preflight_only": True,
|
||||||
|
"automated_checks_passed": automated,
|
||||||
|
"ready_for_live_execution": automated and all(gates.values()),
|
||||||
|
"operator_gates": gates,
|
||||||
|
"latest_platform_pg_backup": latest,
|
||||||
|
"max_backup_age_hours": args.max_backup_age_hours,
|
||||||
|
"state": state,
|
||||||
|
"secret_values_observed": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("procedure", choices=["database-lease", "node-reboot"])
|
||||||
|
parser.add_argument("--remote", default="railiance01")
|
||||||
|
parser.add_argument("--approved-window-id")
|
||||||
|
parser.add_argument("--abort-operator")
|
||||||
|
parser.add_argument("--audit-core-owner-ack", action="store_true")
|
||||||
|
parser.add_argument("--rapp-postgres-owner-ack", action="store_true")
|
||||||
|
parser.add_argument("--synthetic-load-id")
|
||||||
|
parser.add_argument("--openbao-snapshot-evidence")
|
||||||
|
parser.add_argument("--unseal-quorum-attested", action="store_true")
|
||||||
|
parser.add_argument("--provider-console-attested", action="store_true")
|
||||||
|
parser.add_argument("--all-owners-ack", action="store_true")
|
||||||
|
parser.add_argument("--max-backup-age-hours", type=float, default=26.0)
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
remote = Remote(args.remote)
|
||||||
|
result = (
|
||||||
|
database_lease_preflight(remote, args)
|
||||||
|
if args.procedure == "database-lease"
|
||||||
|
else reboot_preflight(remote, args)
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, KeyError, PreflightError) as exc:
|
||||||
|
print(f"recovery preflight failed: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
print(json.dumps(result, indent=2, sort_keys=True))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
101
tests/test_audit_core_recovery_preflight.py
Normal file
101
tests/test_audit_core_recovery_preflight.py
Normal file
|
|
@ -0,0 +1,101 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import unittest
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
SPEC = importlib.util.spec_from_file_location(
|
||||||
|
"audit_core_recovery_preflight",
|
||||||
|
ROOT / "scripts" / "audit-core-recovery-preflight.py",
|
||||||
|
)
|
||||||
|
assert SPEC and SPEC.loader
|
||||||
|
module = importlib.util.module_from_spec(SPEC)
|
||||||
|
SPEC.loader.exec_module(module)
|
||||||
|
|
||||||
|
|
||||||
|
class RecoveryPreflightTests(unittest.TestCase):
|
||||||
|
def test_condition_selects_named_condition(self) -> None:
|
||||||
|
resource = {
|
||||||
|
"status": {
|
||||||
|
"conditions": [
|
||||||
|
{"type": "Ready", "status": "True"},
|
||||||
|
{"type": "ContinuousArchiving", "status": "False"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.assertEqual("True", module.condition(resource, "Ready"))
|
||||||
|
self.assertEqual("False", module.condition(resource, "ContinuousArchiving"))
|
||||||
|
self.assertIsNone(module.condition(resource, "Missing"))
|
||||||
|
|
||||||
|
def test_latest_backup_uses_completed_stop_time_and_reports_no_values(self) -> None:
|
||||||
|
backups = {
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"metadata": {"name": "older"},
|
||||||
|
"spec": {"method": "barmanObjectStore"},
|
||||||
|
"status": {"phase": "completed", "stoppedAt": "2026-08-22T01:00:00Z"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"metadata": {"name": "newer"},
|
||||||
|
"spec": {"method": "barmanObjectStore"},
|
||||||
|
"status": {"phase": "completed", "stoppedAt": "2026-08-22T02:00:00Z"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"metadata": {"name": "running"},
|
||||||
|
"status": {"phase": "running", "stoppedAt": "2026-08-22T03:00:00Z"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
result = module.latest_completed_backup(
|
||||||
|
backups, datetime(2026, 8, 22, 4, 0, tzinfo=UTC)
|
||||||
|
)
|
||||||
|
self.assertEqual("newer", result["name"])
|
||||||
|
self.assertEqual(2.0, result["age_hours"])
|
||||||
|
self.assertNotIn("data", result)
|
||||||
|
|
||||||
|
def test_common_gate_requires_every_boundary(self) -> None:
|
||||||
|
state = {
|
||||||
|
"node": {"ready": True, "k3s_active": True},
|
||||||
|
"platform_pg": {
|
||||||
|
"ready_instances": 1,
|
||||||
|
"instances": 1,
|
||||||
|
"continuous_archiving": True,
|
||||||
|
"last_backup_succeeded": True,
|
||||||
|
},
|
||||||
|
"openbao": {"initialized": True, "sealed": False},
|
||||||
|
"external_secret_stores": {"a": {"ready": True}, "b": {"ready": True}},
|
||||||
|
"external_secrets": {"a": {"ready": True}, "b": {"ready": True}},
|
||||||
|
"audit_core": {
|
||||||
|
"image_matches_reviewed_digest": True,
|
||||||
|
"ready_replicas": 1,
|
||||||
|
"replicas": 1,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
self.assertTrue(module.automated_common_pass(state))
|
||||||
|
state["openbao"]["sealed"] = True
|
||||||
|
self.assertFalse(module.automated_common_pass(state))
|
||||||
|
|
||||||
|
def test_expected_database_projection_is_exact_key_set(self) -> None:
|
||||||
|
self.assertEqual(
|
||||||
|
{"username", "password", "host", "port", "dbname"},
|
||||||
|
module.EXPECTED_DB_KEYS,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_secret_description_parser_reads_names_not_values(self) -> None:
|
||||||
|
description = """Name: audit-core-database
|
||||||
|
Data
|
||||||
|
====
|
||||||
|
username: 20 bytes
|
||||||
|
password: 36 bytes
|
||||||
|
host: 50 bytes
|
||||||
|
"""
|
||||||
|
self.assertEqual(
|
||||||
|
{"username", "password", "host"}, module.secret_key_names(description)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
|
|
@ -87,7 +87,7 @@ passed. Live projection and cleanup remain waiting for the attended window.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0024-T02
|
id: RAILIANCE-WP-0024-T02
|
||||||
status: todo
|
status: progress
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "caba7fcd-014e-5253-87f7-6993731cb0c4"
|
state_hub_task_id: "caba7fcd-014e-5253-87f7-6993731cb0c4"
|
||||||
```
|
```
|
||||||
|
|
@ -101,11 +101,23 @@ restart. Record preconditions, time bounds, abort conditions, and rollback.
|
||||||
Done when the procedure proves the property claimed by audit-core's credential
|
Done when the procedure proves the property claimed by audit-core's credential
|
||||||
reader without exposing a username/password or affecting non-synthetic data.
|
reader without exposing a username/password or affecting non-synthetic data.
|
||||||
|
|
||||||
|
**Procedure implementation (2026-08-22):**
|
||||||
|
`docs/audit-core-database-lease-recovery.md` fixes the exact consumer, lease,
|
||||||
|
database, synthetic-load, owner, time-bound, abort, and value-safe evidence
|
||||||
|
contract. It requires matching the current lease internally while retaining
|
||||||
|
only its non-usable handle, proving health 200/readiness 503/retryable request
|
||||||
|
503 after revocation, forcing only the database ExternalSecret, and proving
|
||||||
|
recovery with the same pod UID and restart count. The read-only helper
|
||||||
|
`scripts/audit-core-recovery-preflight.py database-lease` passed live automated
|
||||||
|
checks without observing Secret data. The live revocation still waits for an
|
||||||
|
approved window, audit-core/rapp-postgres acknowledgement, a separately
|
||||||
|
approved synthetic-load contract, and a named abort operator.
|
||||||
|
|
||||||
## T03 — Define the coordinated railiance01 reboot exercise
|
## T03 — Define the coordinated railiance01 reboot exercise
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0024-T03
|
id: RAILIANCE-WP-0024-T03
|
||||||
status: todo
|
status: progress
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "09cf4065-4f8a-5b5f-beec-17f623efb56f"
|
state_hub_task_id: "09cf4065-4f8a-5b5f-beec-17f623efb56f"
|
||||||
```
|
```
|
||||||
|
|
@ -120,11 +132,23 @@ abort/rollback conditions.
|
||||||
Done when an operator-approved window can execute from one checklist without
|
Done when an operator-approved window can execute from one checklist without
|
||||||
guessing or relying on credential values in the record.
|
guessing or relying on credential values in the record.
|
||||||
|
|
||||||
|
**Procedure implementation (2026-08-22):**
|
||||||
|
`docs/railiance01-coordinated-reboot.md` now provides the single ordered
|
||||||
|
preflight, hold point, reboot, recovery, escalation, and evidence checklist
|
||||||
|
across host, cluster, OpenBao/ESO, platform-pg, and audit-core owners. The
|
||||||
|
read-only helper `scripts/audit-core-recovery-preflight.py node-reboot` passed
|
||||||
|
all automated live checks: node/k3s, platform-pg 1/1, continuous archiving,
|
||||||
|
fresh completed Barman backup, unsealed OpenBao, ESO Valid/SecretSynced, and
|
||||||
|
the reviewed audit-core digest 1/1 Ready. Live readiness remains fail-closed on
|
||||||
|
the intentionally non-automated gates: an approved window and owner set,
|
||||||
|
provider-console access, a current OpenBao Raft snapshot receipt, 2-of-3 Shamir
|
||||||
|
quorum availability, and a named abort operator.
|
||||||
|
|
||||||
## T04 — Route the contracts and approval gates
|
## T04 — Route the contracts and approval gates
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0024-T04
|
id: RAILIANCE-WP-0024-T04
|
||||||
status: todo
|
status: progress
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "22f5011f-aa43-54d0-9c68-66b1b7a2d140"
|
state_hub_task_id: "22f5011f-aa43-54d0-9c68-66b1b7a2d140"
|
||||||
```
|
```
|
||||||
|
|
@ -137,6 +161,14 @@ engagement or recovery window is explicitly acknowledged.
|
||||||
Done when every participant has the same value-safe procedure, responsibility
|
Done when every participant has the same value-safe procedure, responsibility
|
||||||
map, and stop conditions.
|
map, and stop conditions.
|
||||||
|
|
||||||
|
**Routing package (2026-08-22):** the owner-review package consists of
|
||||||
|
`docs/audit-core-whitehat-e2-credential-projection.md`,
|
||||||
|
`docs/audit-core-database-lease-recovery.md`, and
|
||||||
|
`docs/railiance01-coordinated-reboot.md`. T01's live credential window remains
|
||||||
|
separate from the new T02/T03 authorization gates. Outstanding approvals are
|
||||||
|
stated in each procedure and enforced by the read-only preflight output rather
|
||||||
|
than inferred from repository access.
|
||||||
|
|
||||||
## Acceptance
|
## Acceptance
|
||||||
|
|
||||||
- [ ] E2 cannot proceed without enforced sender expiry and exact-path projection.
|
- [ ] E2 cannot proceed without enforced sender expiry and exact-path projection.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue