From bd25f7fa40e10c3664cb0162128a8717dccfb103 Mon Sep 17 00:00:00 2001 From: codex Date: Sat, 22 Aug 2026 13:16:13 +0200 Subject: [PATCH] Add audit recovery exercise preflights Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa --- Makefile | 8 +- WORK-RECORDS.md | 6 +- docs/audit-core-database-lease-recovery.md | 86 +++++ docs/railiance01-coordinated-reboot.md | 121 +++++++ scripts/audit-core-recovery-preflight.py | 331 ++++++++++++++++++ tests/test_audit_core_recovery_preflight.py | 101 ++++++ ...-core-custody-and-recovery-coordination.md | 38 +- 7 files changed, 684 insertions(+), 7 deletions(-) create mode 100644 docs/audit-core-database-lease-recovery.md create mode 100644 docs/railiance01-coordinated-reboot.md create mode 100755 scripts/audit-core-recovery-preflight.py create mode 100644 tests/test_audit_core_recovery_preflight.py diff --git a/Makefile b/Makefile index 73a99f6..6534b45 100644 --- a/Makefile +++ b/Makefile @@ -475,5 +475,11 @@ audit-whitehat-e2-project: ## Project two exact mount-only credentials during th audit-whitehat-e2-cleanup: ## Remove the exact engagement credentials and reload audit-core python3 scripts/audit-core-whitehat-e2-credentials.py cleanup --confirm WH-ENG-20260822-AUDIT-E2-01:attended -.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup apps-pg-isolation-probe +audit-database-lease-recovery-preflight: ## Read-only gate for the restart-free database lease exercise + python3 scripts/audit-core-recovery-preflight.py database-lease + +railiance01-reboot-preflight: ## Read-only gate for the coordinated single-node reboot + python3 scripts/audit-core-recovery-preflight.py node-reboot + +.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight railiance01-reboot-preflight apps-pg-isolation-probe .PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index b27e32f..d6ed1d4 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -98,9 +98,9 @@ | task | RAILIANCE-WP-0023-T02 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md | | task | RAILIANCE-WP-0023-T03 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md | | task | RAILIANCE-WP-0024-T01 | wait | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | -| task | RAILIANCE-WP-0024-T02 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | -| task | RAILIANCE-WP-0024-T03 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | -| task | RAILIANCE-WP-0024-T04 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | +| task | RAILIANCE-WP-0024-T02 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | +| task | RAILIANCE-WP-0024-T03 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | +| task | RAILIANCE-WP-0024-T04 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md | | task | RPF-WP-0018-T01 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | task | RPF-WP-0018-T02 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | task | RPF-WP-0018-T03 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | diff --git a/docs/audit-core-database-lease-recovery.md b/docs/audit-core-database-lease-recovery.md new file mode 100644 index 0000000..da25ff4 --- /dev/null +++ b/docs/audit-core-database-lease-recovery.md @@ -0,0 +1,86 @@ +# Audit-core restart-free database lease recovery + +This is the review and attended-execution contract for +`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by +audit-core causes retryable unavailability and that External Secrets plus the +mounted credential reader restore service without replacing or restarting the +pod. It does not authorize a live revocation. + +## Fixed scope and owners + +| Boundary | Exact scope | Responsible owner | +| --- | --- | --- | +| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` | +| Lease | `database/creds/audit-core-runtime`; only the lease matching the current mounted username | `railiance-platform` custody operator | +| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` | +| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner | +| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` | + +The live window requires audit-core and rapp-postgres acknowledgement, a named +abort operator, and an approved synthetic-load identifier. Run the read-only +gate before requesting that approval: + +```bash +python3 scripts/audit-core-recovery-preflight.py database-lease +``` + +The result must have `automated_checks_passed: true`. It will remain +`ready_for_live_execution: false` until the explicit owner/window fields are +provided. The helper reads Secret metadata and key names only; it never reads +Secret data. + +## Attended sequence + +Keep one terminal attached from baseline through recovery. A value-handling +helper used in the approved window must suppress OpenBao/Kubernetes response +bodies and clear all transient JSON from memory or mode-`0600` temporary files. + +1. Record the audit-core pod UID, restart count, database Secret + `resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness, + ExternalSecret condition, and ClusterSecretStore condition. +2. Start bounded retrying synthetic traffic. Its evidence records only request + identifier, attempt time, response class, and final accepted/duplicate + result—never the bearer or payload. +3. Inside the attended process, read the current mounted username without + emitting it. List only lease handles below + `sys/leases/lookup/database/creds/audit-core-runtime`, look each up, and + retain the one handle whose lease username matches. Abort unless exactly one + live handle matches. Clear every lookup response and username immediately. +4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have + not changed. Revoke only that handle. Do not revoke the role prefix and do + not revoke the ESO parent token. +5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes + 503, at least one synthetic attempt receives retryable 503, pod UID is + unchanged, and restart count is unchanged. Abort if liveness fails, the pod + restarts, a non-synthetic request is involved, or 503 is not retryable. +6. Annotate only ExternalSecret `audit-core/audit-core-database` to force + reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret + `resourceVersion`, and Kubernetes' mounted `..data` generation to advance. + Do not restart or roll out audit-core. +7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches + accepted or duplicate, the pre-exercise event remains readable, pod UID is + unchanged, and restart count is unchanged. +8. Stop the load, remove its separately approved fixture/identity, and confirm + the predecessor lease is revoked. The replacement remains managed by ESO + and expires through its normal database role TTL. + +## Time bounds and abort path + +- Planned outage observation: at most 120 seconds. +- Full exercise: at most 10 minutes. +- If `/readyz` has not recovered 120 seconds after forced reconciliation, + stop load and page both custody and database owners. Do not bounce the pod. +- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store. + Restart External Secrets only for the known post-unseal stale-provider case + and only with the platform owner present. +- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without + revocation. + +## Value-safe evidence + +Record timestamps, approved window/load identifiers, owner acknowledgements, +lease accessor or one-way handle fingerprint, Secret resource versions, mount +generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO +conditions, and cleanup outcome. Never record usernames, passwords, DSNs, +Secret data, bearer values, request payloads, OpenBao response bodies, or +unseal material. diff --git a/docs/railiance01-coordinated-reboot.md b/docs/railiance01-coordinated-reboot.md new file mode 100644 index 0000000..535b2b9 --- /dev/null +++ b/docs/railiance01-coordinated-reboot.md @@ -0,0 +1,121 @@ +# Coordinated `railiance01` reboot and recovery + +This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a +single-node platform outage, not a Kubernetes rolling operation and not an HA +claim. This document does not authorize a reboot. + +## Responsibility map + +| Phase | Accountable owner | Required evidence | +| --- | --- | --- | +| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator | +| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns | +| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled | +| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification | +| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary | +| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal | +| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result | + +The last observed reboot on 2026-08-16 showed the expected dependency order: +the host and k3s returned, audit-core was unready for about 40 seconds while +DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted +Secrets remained, but new leases could not refresh until unseal. + +## Fail-closed preflight + +Run the value-safe, read-only gate: + +```bash +python3 scripts/audit-core-recovery-preflight.py node-reboot +``` + +The automated checks require one Ready node, active k3s, healthy 1/1 +`platform-pg`, continuous archiving, a successful backup no older than 26 +hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and +the reviewed 1/1 Ready audit-core image. The following are human gates and make +`ready_for_live_execution` remain false until supplied: + +- approved window id and acknowledgements from all five owners; +- a named attended driver/abort operator; +- independent provider-console access, not SSH alone; +- a current non-secret OpenBao Raft snapshot evidence file; +- explicit availability of two of three Shamir shares through their approved + out-of-band custody paths. + +Do not reuse old restore-drill templates as current snapshot evidence. Do not +put a snapshot, unseal share, token, password, or Secret response in Git or +State Hub. + +## Pre-reboot hold point + +1. Announce the outage and stop new deployment, migration, credential-change, + backup-restore, and security-engagement mutations on `railiance01`. +2. Confirm no temporary Whitehat or other expiring projection will cross the + window. Clean it first if present. +3. Capture the read-only preflight result and audit-core chain-head attestation. +4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving + is healthy, and the PVC is Bound. If the daily backup is stale, take and + verify an on-demand backup before continuing. +5. Create and escrow a fresh OpenBao Raft snapshot through the approved + encrypted backup lane. Record only its evidence receipt and verification. +6. Have both Shamir share custodians present. Prove provider-console access and + SSH access. Record the exact restart and abort operators. +7. Re-run the preflight with the approval and attestation flags. The result + must say `ready_for_live_execution: true`. Pause for the final operator + go/no-go; this is the last reversible point. + +## Reboot and recovery order + +1. `railiance-infra` performs one ordinary host reboot and retains the provider + console. Do not reinstall the OS or k3s. +2. Recover host network/SSH, then verify `k3s` active and the single node Ready. + If SSH does not return, use the provider console; do not change firewall + declarations as an ad hoc recovery. +3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod + are running. Application readiness may still be false at this point. +4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity, + primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap. +5. Expect OpenBao to be sealed. Two custodians unseal it through the attended, + non-recorded share ceremony. Run `make openbao-status` and + `make openbao-verify-post-unseal` after it reports initialized and unsealed. +6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and + the audit-core sender/database ExternalSecrets are SecretSynced. If they + remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state, + the platform owner may restart only Deployment + `external-secrets/external-secrets`, then recheck reconciliation. +7. Do not restart audit-core to cure database timing. `/healthz` must remain the + liveness boundary; wait for `/readyz` to recover naturally after DNS, + PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count. +8. Run audit-core integrity verification against the pre-reboot chain head and + one separately approved synthetic E2. Confirm no chain break, the prior + marker remains readable, and the new event is accepted exactly once. +9. Re-run the read-only preflight, end the mutation freeze, and notify all + owners with value-safe evidence. + +## Stop, escalation, and recovery boundaries + +- Before reboot: abort for missing approval, owner, console access, unseal + quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC, + or any overlapping live mutation. +- After reboot: retain the outage and escalate for a missing host, node/PVC + identity change, CNPG re-bootstrap, OpenBao initialization loss, failed + unseal, audit chain break, or any evidence of data loss. +- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao, + recover PostgreSQL in place, delete a PVC, or weaken network policy as a + shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to + `railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`; + PostgreSQL restoration always creates a new CNPG Cluster under + `rapp-postgres`' restore procedure. + +## Evidence contract + +Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id +or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age, +OpenBao initialized/sealed booleans and version, resource readiness conditions, +Secret resource versions and key names, HTTP status sequence, audit chain-head +hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts, +and final notifications. + +Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database +credentials/usernames/DSNs, Secret data, sender bearer values, private keys, +provider-console credentials, request payloads, or decrypted snapshots. diff --git a/scripts/audit-core-recovery-preflight.py b/scripts/audit-core-recovery-preflight.py new file mode 100755 index 0000000..0f2c5ff --- /dev/null +++ b/scripts/audit-core-recovery-preflight.py @@ -0,0 +1,331 @@ +#!/usr/bin/env python3 +"""Value-safe preflight for RAILIANCE-WP-0024 recovery exercises. + +This helper is deliberately read-only. It never reads Kubernetes Secret data, +database usernames/passwords, OpenBao lease payloads, or unseal material. Live +lease revocation and a host reboot remain separate, attended actions. +""" + +from __future__ import annotations + +import argparse +import json +import re +import shlex +import subprocess +import sys +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + + +EXPECTED_IMAGE = ( + "forgejo.coulomb.social/coulomb/audit-core@" + "sha256:c2fe39a0185b99be3fc0cb14d2de69772b8e66e20490097c9d11d90cc39719a6" +) +EXPECTED_DB_KEYS = {"username", "password", "host", "port", "dbname"} + + +class PreflightError(RuntimeError): + pass + + +class Remote: + def __init__(self, host: str) -> None: + self.host = host + + def run(self, command: list[str], *, label: str) -> str: + remote_command = " ".join(shlex.quote(part) for part in command) + completed = subprocess.run( + ["ssh", "-o", "BatchMode=yes", self.host, remote_command], + text=True, + capture_output=True, + check=False, + ) + if completed.returncode != 0: + # Remote output can include provider or application detail that is + # inappropriate for durable evidence. Report only the failed step. + raise PreflightError(f"{label} failed (exit {completed.returncode})") + return completed.stdout.strip() + + def kubectl(self, args: list[str], *, label: str) -> str: + return self.run(["kubectl", *args], label=label) + + def kubectl_json(self, args: list[str], *, label: str) -> dict[str, Any]: + raw = self.kubectl([*args, "-o", "json"], label=label) + try: + value = json.loads(raw) + except json.JSONDecodeError as exc: + raise PreflightError(f"{label} returned invalid JSON") from exc + if not isinstance(value, dict): + raise PreflightError(f"{label} did not return an object") + return value + + +def condition(resource: dict[str, Any], kind: str) -> str | None: + for item in resource.get("status", {}).get("conditions", []): + if item.get("type") == kind: + return item.get("status") + return None + + +def rfc3339(value: str) -> datetime: + return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC) + + +def latest_completed_backup(backups: dict[str, Any], now: datetime) -> dict[str, Any]: + completed = [ + item + for item in backups.get("items", []) + if item.get("status", {}).get("phase") == "completed" + and item.get("status", {}).get("stoppedAt") + ] + if not completed: + raise PreflightError("no completed platform-pg backup is visible") + latest = max(completed, key=lambda item: item["status"]["stoppedAt"]) + stopped = rfc3339(latest["status"]["stoppedAt"]) + return { + "name": latest["metadata"]["name"], + "stopped_at": stopped.isoformat().replace("+00:00", "Z"), + "age_hours": round((now - stopped).total_seconds() / 3600, 2), + "method": latest.get("spec", {}).get("method"), + } + + +def resource_condition(resource: dict[str, Any]) -> dict[str, Any]: + conditions = resource.get("status", {}).get("conditions", []) + current = conditions[0] if conditions else {} + return { + "ready": current.get("status") == "True", + "reason": current.get("reason"), + } + + +def secret_key_names(description: str) -> set[str]: + """Extract names from ``kubectl describe secret`` without reading data.""" + keys: set[str] = set() + for line in description.splitlines(): + match = re.match(r"^([^\s:]+):\s+\d+ bytes$", line.strip()) + if match: + keys.add(match.group(1)) + return keys + + +def common_state(remote: Remote, now: datetime) -> dict[str, Any]: + node_list = remote.kubectl_json(["get", "nodes"], label="read node state") + nodes = node_list.get("items", []) + if len(nodes) != 1: + raise PreflightError(f"expected one railiance01 node, observed {len(nodes)}") + node = nodes[0] + + cluster = remote.kubectl_json( + ["-n", "databases", "get", "cluster", "platform-pg"], + label="read platform-pg state", + ) + deployment = remote.kubectl_json( + ["-n", "audit-core", "get", "deployment", "audit-core"], + label="read audit-core deployment", + ) + pod_list = remote.kubectl_json( + ["-n", "audit-core", "get", "pods", "-l", "app.kubernetes.io/name=audit-core"], + label="read audit-core pod metadata", + ) + pods = pod_list.get("items", []) + if len(pods) != 1: + raise PreflightError(f"expected one audit-core pod, observed {len(pods)}") + pod = pods[0] + container_statuses = pod.get("status", {}).get("containerStatuses", []) + restart_count = sum(int(item.get("restartCount", 0)) for item in container_statuses) + + stores: dict[str, Any] = {} + for name in ("openbao-audit-core", "openbao-audit-core-database"): + resource = remote.kubectl_json( + ["get", "clustersecretstore", name], label=f"read {name} state" + ) + stores[name] = resource_condition(resource) + + external_secrets: dict[str, Any] = {} + for name in ("audit-core-database", "audit-core-senders"): + resource = remote.kubectl_json( + ["-n", "audit-core", "get", "externalsecret", name], + label=f"read {name} state", + ) + external_secrets[name] = resource_condition(resource) + + bao_raw = remote.kubectl( + ["-n", "openbao", "exec", "openbao-0", "--", "bao", "status", "-format=json"], + label="read OpenBao seal state", + ) + try: + bao_status = json.loads(bao_raw) + except json.JSONDecodeError as exc: + raise PreflightError("OpenBao seal state returned invalid JSON") from exc + image = deployment["spec"]["template"]["spec"]["containers"][0]["image"] + state = { + "captured_at": now.replace(microsecond=0).isoformat().replace("+00:00", "Z"), + "node": { + "name": node["metadata"]["name"], + "ready": condition(node, "Ready") == "True", + "k3s_active": remote.run( + ["systemctl", "is-active", "k3s"], label="read k3s service state" + ) == "active", + "uptime_seconds": float( + remote.run(["cat", "/proc/uptime"], label="read host uptime").split()[0] + ), + }, + "platform_pg": { + "ready_instances": cluster.get("status", {}).get("readyInstances", 0), + "instances": cluster.get("spec", {}).get("instances", 0), + "phase": cluster.get("status", {}).get("phase"), + "continuous_archiving": condition(cluster, "ContinuousArchiving") == "True", + "last_backup_succeeded": condition(cluster, "LastBackupSucceeded") == "True", + }, + "openbao": { + "initialized": bool(bao_status.get("initialized")), + "sealed": bool(bao_status.get("sealed")), + "seal_type": bao_status.get("type"), + "threshold": bao_status.get("t"), + "shares": bao_status.get("n"), + }, + "external_secret_stores": stores, + "external_secrets": external_secrets, + "audit_core": { + "image": image, + "image_matches_reviewed_digest": image == EXPECTED_IMAGE, + "ready_replicas": deployment.get("status", {}).get("readyReplicas", 0), + "replicas": deployment.get("spec", {}).get("replicas", 0), + "pod_uid": pod["metadata"]["uid"], + "restart_count": restart_count, + }, + "secret_values_observed": False, + } + return state + + +def automated_common_pass(state: dict[str, Any]) -> bool: + return all( + ( + state["node"]["ready"], + state["node"]["k3s_active"], + state["platform_pg"]["ready_instances"] == state["platform_pg"]["instances"] == 1, + state["platform_pg"]["continuous_archiving"], + state["platform_pg"]["last_backup_succeeded"], + state["openbao"]["initialized"], + not state["openbao"]["sealed"], + all(item["ready"] for item in state["external_secret_stores"].values()), + all(item["ready"] for item in state["external_secrets"].values()), + state["audit_core"]["image_matches_reviewed_digest"], + state["audit_core"]["ready_replicas"] == state["audit_core"]["replicas"] == 1, + ) + ) + + +def database_lease_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]: + now = datetime.now(UTC) + state = common_state(remote, now) + secret_rv = remote.kubectl( + ["-n", "audit-core", "get", "secret", "audit-core-database", "-o", "jsonpath={.metadata.resourceVersion}"], + label="read database Secret metadata", + ) + description = remote.kubectl( + ["-n", "audit-core", "describe", "secret", "audit-core-database"], + label="read database Secret key names", + ) + keys = secret_key_names(description) + health_code = remote.kubectl( + ["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/healthz", timeout=3).status)'], + label="probe audit-core health", + ) + ready_code = remote.kubectl( + ["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/readyz", timeout=3).status)'], + label="probe audit-core readiness", + ) + gates = { + "approved_window_id_recorded": bool(args.approved_window_id), + "audit_core_owner_acknowledged": args.audit_core_owner_ack, + "rapp_postgres_owner_acknowledged": args.rapp_postgres_owner_ack, + "approved_synthetic_load_contract": bool(args.synthetic_load_id), + "attended_abort_operator_named": bool(args.abort_operator), + } + automated = automated_common_pass(state) and keys == EXPECTED_DB_KEYS and health_code == "200" and ready_code == "200" + return { + "procedure": "audit-core-database-lease-recovery", + "preflight_only": True, + "automated_checks_passed": automated, + "ready_for_live_execution": automated and all(gates.values()), + "operator_gates": gates, + "baseline": { + "database_secret_resource_version": secret_rv, + "database_secret_keys": sorted(keys), + "health_status": int(health_code), + "readiness_status": int(ready_code), + "pod_uid": state["audit_core"]["pod_uid"], + "restart_count": state["audit_core"]["restart_count"], + }, + "state": state, + "secret_values_observed": False, + } + + +def reboot_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]: + now = datetime.now(UTC) + state = common_state(remote, now) + backups = remote.kubectl_json( + ["-n", "databases", "get", "backup", "-l", "cnpg.io/cluster=platform-pg"], + label="read platform-pg backup metadata", + ) + latest = latest_completed_backup(backups, now) + snapshot_evidence = Path(args.openbao_snapshot_evidence).resolve() if args.openbao_snapshot_evidence else None + gates = { + "approved_window_id_recorded": bool(args.approved_window_id), + "openbao_snapshot_evidence_present": bool(snapshot_evidence and snapshot_evidence.is_file()), + "unseal_quorum_attested": args.unseal_quorum_attested, + "provider_console_access_attested": args.provider_console_attested, + "host_cluster_platform_database_audit_owners_acknowledged": args.all_owners_ack, + "attended_abort_operator_named": bool(args.abort_operator), + } + automated = automated_common_pass(state) and latest["age_hours"] <= args.max_backup_age_hours + return { + "procedure": "railiance01-coordinated-reboot", + "preflight_only": True, + "automated_checks_passed": automated, + "ready_for_live_execution": automated and all(gates.values()), + "operator_gates": gates, + "latest_platform_pg_backup": latest, + "max_backup_age_hours": args.max_backup_age_hours, + "state": state, + "secret_values_observed": False, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("procedure", choices=["database-lease", "node-reboot"]) + parser.add_argument("--remote", default="railiance01") + parser.add_argument("--approved-window-id") + parser.add_argument("--abort-operator") + parser.add_argument("--audit-core-owner-ack", action="store_true") + parser.add_argument("--rapp-postgres-owner-ack", action="store_true") + parser.add_argument("--synthetic-load-id") + parser.add_argument("--openbao-snapshot-evidence") + parser.add_argument("--unseal-quorum-attested", action="store_true") + parser.add_argument("--provider-console-attested", action="store_true") + parser.add_argument("--all-owners-ack", action="store_true") + parser.add_argument("--max-backup-age-hours", type=float, default=26.0) + args = parser.parse_args() + try: + remote = Remote(args.remote) + result = ( + database_lease_preflight(remote, args) + if args.procedure == "database-lease" + else reboot_preflight(remote, args) + ) + except (OSError, ValueError, KeyError, PreflightError) as exc: + print(f"recovery preflight failed: {exc}", file=sys.stderr) + return 1 + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_audit_core_recovery_preflight.py b/tests/test_audit_core_recovery_preflight.py new file mode 100644 index 0000000..597175d --- /dev/null +++ b/tests/test_audit_core_recovery_preflight.py @@ -0,0 +1,101 @@ +from __future__ import annotations + +import importlib.util +import unittest +from datetime import UTC, datetime +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location( + "audit_core_recovery_preflight", + ROOT / "scripts" / "audit-core-recovery-preflight.py", +) +assert SPEC and SPEC.loader +module = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(module) + + +class RecoveryPreflightTests(unittest.TestCase): + def test_condition_selects_named_condition(self) -> None: + resource = { + "status": { + "conditions": [ + {"type": "Ready", "status": "True"}, + {"type": "ContinuousArchiving", "status": "False"}, + ] + } + } + self.assertEqual("True", module.condition(resource, "Ready")) + self.assertEqual("False", module.condition(resource, "ContinuousArchiving")) + self.assertIsNone(module.condition(resource, "Missing")) + + def test_latest_backup_uses_completed_stop_time_and_reports_no_values(self) -> None: + backups = { + "items": [ + { + "metadata": {"name": "older"}, + "spec": {"method": "barmanObjectStore"}, + "status": {"phase": "completed", "stoppedAt": "2026-08-22T01:00:00Z"}, + }, + { + "metadata": {"name": "newer"}, + "spec": {"method": "barmanObjectStore"}, + "status": {"phase": "completed", "stoppedAt": "2026-08-22T02:00:00Z"}, + }, + { + "metadata": {"name": "running"}, + "status": {"phase": "running", "stoppedAt": "2026-08-22T03:00:00Z"}, + }, + ] + } + result = module.latest_completed_backup( + backups, datetime(2026, 8, 22, 4, 0, tzinfo=UTC) + ) + self.assertEqual("newer", result["name"]) + self.assertEqual(2.0, result["age_hours"]) + self.assertNotIn("data", result) + + def test_common_gate_requires_every_boundary(self) -> None: + state = { + "node": {"ready": True, "k3s_active": True}, + "platform_pg": { + "ready_instances": 1, + "instances": 1, + "continuous_archiving": True, + "last_backup_succeeded": True, + }, + "openbao": {"initialized": True, "sealed": False}, + "external_secret_stores": {"a": {"ready": True}, "b": {"ready": True}}, + "external_secrets": {"a": {"ready": True}, "b": {"ready": True}}, + "audit_core": { + "image_matches_reviewed_digest": True, + "ready_replicas": 1, + "replicas": 1, + }, + } + self.assertTrue(module.automated_common_pass(state)) + state["openbao"]["sealed"] = True + self.assertFalse(module.automated_common_pass(state)) + + def test_expected_database_projection_is_exact_key_set(self) -> None: + self.assertEqual( + {"username", "password", "host", "port", "dbname"}, + module.EXPECTED_DB_KEYS, + ) + + def test_secret_description_parser_reads_names_not_values(self) -> None: + description = """Name: audit-core-database +Data +==== +username: 20 bytes +password: 36 bytes +host: 50 bytes +""" + self.assertEqual( + {"username", "password", "host"}, module.secret_key_names(description) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md b/workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md index 94ae5bc..db9aa62 100644 --- a/workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md +++ b/workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md @@ -87,7 +87,7 @@ passed. Live projection and cleanup remain waiting for the attended window. ```task id: RAILIANCE-WP-0024-T02 -status: todo +status: progress priority: high state_hub_task_id: "caba7fcd-014e-5253-87f7-6993731cb0c4" ``` @@ -101,11 +101,23 @@ restart. Record preconditions, time bounds, abort conditions, and rollback. Done when the procedure proves the property claimed by audit-core's credential reader without exposing a username/password or affecting non-synthetic data. +**Procedure implementation (2026-08-22):** +`docs/audit-core-database-lease-recovery.md` fixes the exact consumer, lease, +database, synthetic-load, owner, time-bound, abort, and value-safe evidence +contract. It requires matching the current lease internally while retaining +only its non-usable handle, proving health 200/readiness 503/retryable request +503 after revocation, forcing only the database ExternalSecret, and proving +recovery with the same pod UID and restart count. The read-only helper +`scripts/audit-core-recovery-preflight.py database-lease` passed live automated +checks without observing Secret data. The live revocation still waits for an +approved window, audit-core/rapp-postgres acknowledgement, a separately +approved synthetic-load contract, and a named abort operator. + ## T03 — Define the coordinated railiance01 reboot exercise ```task id: RAILIANCE-WP-0024-T03 -status: todo +status: progress priority: high state_hub_task_id: "09cf4065-4f8a-5b5f-beec-17f623efb56f" ``` @@ -120,11 +132,23 @@ abort/rollback conditions. Done when an operator-approved window can execute from one checklist without guessing or relying on credential values in the record. +**Procedure implementation (2026-08-22):** +`docs/railiance01-coordinated-reboot.md` now provides the single ordered +preflight, hold point, reboot, recovery, escalation, and evidence checklist +across host, cluster, OpenBao/ESO, platform-pg, and audit-core owners. The +read-only helper `scripts/audit-core-recovery-preflight.py node-reboot` passed +all automated live checks: node/k3s, platform-pg 1/1, continuous archiving, +fresh completed Barman backup, unsealed OpenBao, ESO Valid/SecretSynced, and +the reviewed audit-core digest 1/1 Ready. Live readiness remains fail-closed on +the intentionally non-automated gates: an approved window and owner set, +provider-console access, a current OpenBao Raft snapshot receipt, 2-of-3 Shamir +quorum availability, and a named abort operator. + ## T04 — Route the contracts and approval gates ```task id: RAILIANCE-WP-0024-T04 -status: todo +status: progress priority: medium state_hub_task_id: "22f5011f-aa43-54d0-9c68-66b1b7a2d140" ``` @@ -137,6 +161,14 @@ engagement or recovery window is explicitly acknowledged. Done when every participant has the same value-safe procedure, responsibility map, and stop conditions. +**Routing package (2026-08-22):** the owner-review package consists of +`docs/audit-core-whitehat-e2-credential-projection.md`, +`docs/audit-core-database-lease-recovery.md`, and +`docs/railiance01-coordinated-reboot.md`. T01's live credential window remains +separate from the new T02/T03 authorization gates. Outstanding approvals are +stated in each procedure and enforced by the read-only preflight output rather +than inferred from repository access. + ## Acceptance - [ ] E2 cannot proceed without enforced sender expiry and exact-path projection.