Add audit recovery exercise preflights
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
This commit is contained in:
codex
parent
dca3d87994
commit
bd25f7fa40
7 changed files with 684 additions and 7 deletions
8
Makefile
8
Makefile
|
|
@ -475,5 +475,11 @@ audit-whitehat-e2-project: ## Project two exact mount-only credentials during th
|
|||
audit-whitehat-e2-cleanup: ## Remove the exact engagement credentials and reload audit-core
|
||||
python3 scripts/audit-core-whitehat-e2-credentials.py cleanup --confirm WH-ENG-20260822-AUDIT-E2-01:attended
|
||||
|
||||
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup apps-pg-isolation-probe
|
||||
audit-database-lease-recovery-preflight: ## Read-only gate for the restart-free database lease exercise
|
||||
python3 scripts/audit-core-recovery-preflight.py database-lease
|
||||
|
||||
railiance01-reboot-preflight: ## Read-only gate for the coordinated single-node reboot
|
||||
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||
|
||||
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight railiance01-reboot-preflight apps-pg-isolation-probe
|
||||
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
|
||||
|
|
|
|||
|
|
@ -98,9 +98,9 @@
|
|||
| task | RAILIANCE-WP-0023-T02 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
||||
| task | RAILIANCE-WP-0023-T03 | done | — | workplans/RAILIANCE-WP-0023-hub-core-candidate-credential-lanes.md |
|
||||
| task | RAILIANCE-WP-0024-T01 | wait | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T02 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T03 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T04 | todo | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T02 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T03 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RAILIANCE-WP-0024-T04 | progress | — | workplans/RAILIANCE-WP-0024-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RPF-WP-0018-T01 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||
| task | RPF-WP-0018-T02 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||
| task | RPF-WP-0018-T03 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
|
||||
|
|
|
|||
86
docs/audit-core-database-lease-recovery.md
Normal file
86
docs/audit-core-database-lease-recovery.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
# Audit-core restart-free database lease recovery
|
||||
|
||||
This is the review and attended-execution contract for
|
||||
`RAILIANCE-WP-0024-T02`. It proves that revoking the lease currently used by
|
||||
audit-core causes retryable unavailability and that External Secrets plus the
|
||||
mounted credential reader restore service without replacing or restarting the
|
||||
pod. It does not authorize a live revocation.
|
||||
|
||||
## Fixed scope and owners
|
||||
|
||||
| Boundary | Exact scope | Responsible owner |
|
||||
| --- | --- | --- |
|
||||
| Consumer | Deployment `audit-core/audit-core`; Secret `audit-core/audit-core-database`; ExternalSecret `audit-core/audit-core-database` | `audit-core` |
|
||||
| Lease | `database/creds/audit-core-runtime`; only the lease matching the current mounted username | `railiance-platform` custody operator |
|
||||
| Database | CNPG `databases/platform-pg`, database `audit_core`, runtime group `audit_core_app` | `rapp-postgres` |
|
||||
| Load | One separately approved synthetic sender/tenant contract; no production event or identity | `audit-core` + load owner |
|
||||
| Coordination and evidence | `RAILIANCE-WP-0024-T02` | `railiance-platform` |
|
||||
|
||||
The live window requires audit-core and rapp-postgres acknowledgement, a named
|
||||
abort operator, and an approved synthetic-load identifier. Run the read-only
|
||||
gate before requesting that approval:
|
||||
|
||||
```bash
|
||||
python3 scripts/audit-core-recovery-preflight.py database-lease
|
||||
```
|
||||
|
||||
The result must have `automated_checks_passed: true`. It will remain
|
||||
`ready_for_live_execution: false` until the explicit owner/window fields are
|
||||
provided. The helper reads Secret metadata and key names only; it never reads
|
||||
Secret data.
|
||||
|
||||
## Attended sequence
|
||||
|
||||
Keep one terminal attached from baseline through recovery. A value-handling
|
||||
helper used in the approved window must suppress OpenBao/Kubernetes response
|
||||
bodies and clear all transient JSON from memory or mode-`0600` temporary files.
|
||||
|
||||
1. Record the audit-core pod UID, restart count, database Secret
|
||||
`resourceVersion`, `/healthz` status, `/readyz` status, CNPG readiness,
|
||||
ExternalSecret condition, and ClusterSecretStore condition.
|
||||
2. Start bounded retrying synthetic traffic. Its evidence records only request
|
||||
identifier, attempt time, response class, and final accepted/duplicate
|
||||
result—never the bearer or payload.
|
||||
3. Inside the attended process, read the current mounted username without
|
||||
emitting it. List only lease handles below
|
||||
`sys/leases/lookup/database/creds/audit-core-runtime`, look each up, and
|
||||
retain the one handle whose lease username matches. Abort unless exactly one
|
||||
live handle matches. Clear every lookup response and username immediately.
|
||||
4. Recheck that the Secret `resourceVersion`, pod UID, and selected handle have
|
||||
not changed. Revoke only that handle. Do not revoke the role prefix and do
|
||||
not revoke the ESO parent token.
|
||||
5. Prove the bounded failure state: `/healthz` remains 200, `/readyz` becomes
|
||||
503, at least one synthetic attempt receives retryable 503, pod UID is
|
||||
unchanged, and restart count is unchanged. Abort if liveness fails, the pod
|
||||
restarts, a non-synthetic request is involved, or 503 is not retryable.
|
||||
6. Annotate only ExternalSecret `audit-core/audit-core-database` to force
|
||||
reconciliation. Wait for `Ready=True/SecretSynced`, a changed Secret
|
||||
`resourceVersion`, and Kubernetes' mounted `..data` generation to advance.
|
||||
Do not restart or roll out audit-core.
|
||||
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
|
||||
accepted or duplicate, the pre-exercise event remains readable, pod UID is
|
||||
unchanged, and restart count is unchanged.
|
||||
8. Stop the load, remove its separately approved fixture/identity, and confirm
|
||||
the predecessor lease is revoked. The replacement remains managed by ESO
|
||||
and expires through its normal database role TTL.
|
||||
|
||||
## Time bounds and abort path
|
||||
|
||||
- Planned outage observation: at most 120 seconds.
|
||||
- Full exercise: at most 10 minutes.
|
||||
- If `/readyz` has not recovered 120 seconds after forced reconciliation,
|
||||
stop load and page both custody and database owners. Do not bounce the pod.
|
||||
- If ESO is not `SecretSynced`, check OpenBao seal state and the exact store.
|
||||
Restart External Secrets only for the known post-unseal stale-provider case
|
||||
and only with the platform owner present.
|
||||
- If PostgreSQL is not 1/1 Ready or continuous archiving is false, abort without
|
||||
revocation.
|
||||
|
||||
## Value-safe evidence
|
||||
|
||||
Record timestamps, approved window/load identifiers, owner acknowledgements,
|
||||
lease accessor or one-way handle fingerprint, Secret resource versions, mount
|
||||
generation changes, HTTP status sequence, pod UID/restart counts, CNPG/ESO
|
||||
conditions, and cleanup outcome. Never record usernames, passwords, DSNs,
|
||||
Secret data, bearer values, request payloads, OpenBao response bodies, or
|
||||
unseal material.
|
||||
121
docs/railiance01-coordinated-reboot.md
Normal file
121
docs/railiance01-coordinated-reboot.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# Coordinated `railiance01` reboot and recovery
|
||||
|
||||
This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a
|
||||
single-node platform outage, not a Kubernetes rolling operation and not an HA
|
||||
claim. This document does not authorize a reboot.
|
||||
|
||||
## Responsibility map
|
||||
|
||||
| Phase | Accountable owner | Required evidence |
|
||||
| --- | --- | --- |
|
||||
| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator |
|
||||
| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns |
|
||||
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled |
|
||||
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
|
||||
| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary |
|
||||
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
|
||||
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |
|
||||
|
||||
The last observed reboot on 2026-08-16 showed the expected dependency order:
|
||||
the host and k3s returned, audit-core was unready for about 40 seconds while
|
||||
DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted
|
||||
Secrets remained, but new leases could not refresh until unseal.
|
||||
|
||||
## Fail-closed preflight
|
||||
|
||||
Run the value-safe, read-only gate:
|
||||
|
||||
```bash
|
||||
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||
```
|
||||
|
||||
The automated checks require one Ready node, active k3s, healthy 1/1
|
||||
`platform-pg`, continuous archiving, a successful backup no older than 26
|
||||
hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and
|
||||
the reviewed 1/1 Ready audit-core image. The following are human gates and make
|
||||
`ready_for_live_execution` remain false until supplied:
|
||||
|
||||
- approved window id and acknowledgements from all five owners;
|
||||
- a named attended driver/abort operator;
|
||||
- independent provider-console access, not SSH alone;
|
||||
- a current non-secret OpenBao Raft snapshot evidence file;
|
||||
- explicit availability of two of three Shamir shares through their approved
|
||||
out-of-band custody paths.
|
||||
|
||||
Do not reuse old restore-drill templates as current snapshot evidence. Do not
|
||||
put a snapshot, unseal share, token, password, or Secret response in Git or
|
||||
State Hub.
|
||||
|
||||
## Pre-reboot hold point
|
||||
|
||||
1. Announce the outage and stop new deployment, migration, credential-change,
|
||||
backup-restore, and security-engagement mutations on `railiance01`.
|
||||
2. Confirm no temporary Whitehat or other expiring projection will cross the
|
||||
window. Clean it first if present.
|
||||
3. Capture the read-only preflight result and audit-core chain-head attestation.
|
||||
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
|
||||
is healthy, and the PVC is Bound. If the daily backup is stale, take and
|
||||
verify an on-demand backup before continuing.
|
||||
5. Create and escrow a fresh OpenBao Raft snapshot through the approved
|
||||
encrypted backup lane. Record only its evidence receipt and verification.
|
||||
6. Have both Shamir share custodians present. Prove provider-console access and
|
||||
SSH access. Record the exact restart and abort operators.
|
||||
7. Re-run the preflight with the approval and attestation flags. The result
|
||||
must say `ready_for_live_execution: true`. Pause for the final operator
|
||||
go/no-go; this is the last reversible point.
|
||||
|
||||
## Reboot and recovery order
|
||||
|
||||
1. `railiance-infra` performs one ordinary host reboot and retains the provider
|
||||
console. Do not reinstall the OS or k3s.
|
||||
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
|
||||
If SSH does not return, use the provider console; do not change firewall
|
||||
declarations as an ad hoc recovery.
|
||||
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod
|
||||
are running. Application readiness may still be false at this point.
|
||||
4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity,
|
||||
primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap.
|
||||
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
|
||||
non-recorded share ceremony. Run `make openbao-status` and
|
||||
`make openbao-verify-post-unseal` after it reports initialized and unsealed.
|
||||
6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and
|
||||
the audit-core sender/database ExternalSecrets are SecretSynced. If they
|
||||
remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
|
||||
the platform owner may restart only Deployment
|
||||
`external-secrets/external-secrets`, then recheck reconciliation.
|
||||
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
|
||||
liveness boundary; wait for `/readyz` to recover naturally after DNS,
|
||||
PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
|
||||
8. Run audit-core integrity verification against the pre-reboot chain head and
|
||||
one separately approved synthetic E2. Confirm no chain break, the prior
|
||||
marker remains readable, and the new event is accepted exactly once.
|
||||
9. Re-run the read-only preflight, end the mutation freeze, and notify all
|
||||
owners with value-safe evidence.
|
||||
|
||||
## Stop, escalation, and recovery boundaries
|
||||
|
||||
- Before reboot: abort for missing approval, owner, console access, unseal
|
||||
quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
|
||||
or any overlapping live mutation.
|
||||
- After reboot: retain the outage and escalate for a missing host, node/PVC
|
||||
identity change, CNPG re-bootstrap, OpenBao initialization loss, failed
|
||||
unseal, audit chain break, or any evidence of data loss.
|
||||
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
|
||||
recover PostgreSQL in place, delete a PVC, or weaken network policy as a
|
||||
shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to
|
||||
`railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`;
|
||||
PostgreSQL restoration always creates a new CNPG Cluster under
|
||||
`rapp-postgres`' restore procedure.
|
||||
|
||||
## Evidence contract
|
||||
|
||||
Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id
|
||||
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
|
||||
OpenBao initialized/sealed booleans and version, resource readiness conditions,
|
||||
Secret resource versions and key names, HTTP status sequence, audit chain-head
|
||||
hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts,
|
||||
and final notifications.
|
||||
|
||||
Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database
|
||||
credentials/usernames/DSNs, Secret data, sender bearer values, private keys,
|
||||
provider-console credentials, request payloads, or decrypted snapshots.
|
||||
331
scripts/audit-core-recovery-preflight.py
Executable file
331
scripts/audit-core-recovery-preflight.py
Executable file
|
|
@ -0,0 +1,331 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Value-safe preflight for RAILIANCE-WP-0024 recovery exercises.
|
||||
|
||||
This helper is deliberately read-only. It never reads Kubernetes Secret data,
|
||||
database usernames/passwords, OpenBao lease payloads, or unseal material. Live
|
||||
lease revocation and a host reboot remain separate, attended actions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
EXPECTED_IMAGE = (
|
||||
"forgejo.coulomb.social/coulomb/audit-core@"
|
||||
"sha256:c2fe39a0185b99be3fc0cb14d2de69772b8e66e20490097c9d11d90cc39719a6"
|
||||
)
|
||||
EXPECTED_DB_KEYS = {"username", "password", "host", "port", "dbname"}
|
||||
|
||||
|
||||
class PreflightError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class Remote:
|
||||
def __init__(self, host: str) -> None:
|
||||
self.host = host
|
||||
|
||||
def run(self, command: list[str], *, label: str) -> str:
|
||||
remote_command = " ".join(shlex.quote(part) for part in command)
|
||||
completed = subprocess.run(
|
||||
["ssh", "-o", "BatchMode=yes", self.host, remote_command],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
if completed.returncode != 0:
|
||||
# Remote output can include provider or application detail that is
|
||||
# inappropriate for durable evidence. Report only the failed step.
|
||||
raise PreflightError(f"{label} failed (exit {completed.returncode})")
|
||||
return completed.stdout.strip()
|
||||
|
||||
def kubectl(self, args: list[str], *, label: str) -> str:
|
||||
return self.run(["kubectl", *args], label=label)
|
||||
|
||||
def kubectl_json(self, args: list[str], *, label: str) -> dict[str, Any]:
|
||||
raw = self.kubectl([*args, "-o", "json"], label=label)
|
||||
try:
|
||||
value = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise PreflightError(f"{label} returned invalid JSON") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise PreflightError(f"{label} did not return an object")
|
||||
return value
|
||||
|
||||
|
||||
def condition(resource: dict[str, Any], kind: str) -> str | None:
|
||||
for item in resource.get("status", {}).get("conditions", []):
|
||||
if item.get("type") == kind:
|
||||
return item.get("status")
|
||||
return None
|
||||
|
||||
|
||||
def rfc3339(value: str) -> datetime:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC)
|
||||
|
||||
|
||||
def latest_completed_backup(backups: dict[str, Any], now: datetime) -> dict[str, Any]:
|
||||
completed = [
|
||||
item
|
||||
for item in backups.get("items", [])
|
||||
if item.get("status", {}).get("phase") == "completed"
|
||||
and item.get("status", {}).get("stoppedAt")
|
||||
]
|
||||
if not completed:
|
||||
raise PreflightError("no completed platform-pg backup is visible")
|
||||
latest = max(completed, key=lambda item: item["status"]["stoppedAt"])
|
||||
stopped = rfc3339(latest["status"]["stoppedAt"])
|
||||
return {
|
||||
"name": latest["metadata"]["name"],
|
||||
"stopped_at": stopped.isoformat().replace("+00:00", "Z"),
|
||||
"age_hours": round((now - stopped).total_seconds() / 3600, 2),
|
||||
"method": latest.get("spec", {}).get("method"),
|
||||
}
|
||||
|
||||
|
||||
def resource_condition(resource: dict[str, Any]) -> dict[str, Any]:
|
||||
conditions = resource.get("status", {}).get("conditions", [])
|
||||
current = conditions[0] if conditions else {}
|
||||
return {
|
||||
"ready": current.get("status") == "True",
|
||||
"reason": current.get("reason"),
|
||||
}
|
||||
|
||||
|
||||
def secret_key_names(description: str) -> set[str]:
|
||||
"""Extract names from ``kubectl describe secret`` without reading data."""
|
||||
keys: set[str] = set()
|
||||
for line in description.splitlines():
|
||||
match = re.match(r"^([^\s:]+):\s+\d+ bytes$", line.strip())
|
||||
if match:
|
||||
keys.add(match.group(1))
|
||||
return keys
|
||||
|
||||
|
||||
def common_state(remote: Remote, now: datetime) -> dict[str, Any]:
|
||||
node_list = remote.kubectl_json(["get", "nodes"], label="read node state")
|
||||
nodes = node_list.get("items", [])
|
||||
if len(nodes) != 1:
|
||||
raise PreflightError(f"expected one railiance01 node, observed {len(nodes)}")
|
||||
node = nodes[0]
|
||||
|
||||
cluster = remote.kubectl_json(
|
||||
["-n", "databases", "get", "cluster", "platform-pg"],
|
||||
label="read platform-pg state",
|
||||
)
|
||||
deployment = remote.kubectl_json(
|
||||
["-n", "audit-core", "get", "deployment", "audit-core"],
|
||||
label="read audit-core deployment",
|
||||
)
|
||||
pod_list = remote.kubectl_json(
|
||||
["-n", "audit-core", "get", "pods", "-l", "app.kubernetes.io/name=audit-core"],
|
||||
label="read audit-core pod metadata",
|
||||
)
|
||||
pods = pod_list.get("items", [])
|
||||
if len(pods) != 1:
|
||||
raise PreflightError(f"expected one audit-core pod, observed {len(pods)}")
|
||||
pod = pods[0]
|
||||
container_statuses = pod.get("status", {}).get("containerStatuses", [])
|
||||
restart_count = sum(int(item.get("restartCount", 0)) for item in container_statuses)
|
||||
|
||||
stores: dict[str, Any] = {}
|
||||
for name in ("openbao-audit-core", "openbao-audit-core-database"):
|
||||
resource = remote.kubectl_json(
|
||||
["get", "clustersecretstore", name], label=f"read {name} state"
|
||||
)
|
||||
stores[name] = resource_condition(resource)
|
||||
|
||||
external_secrets: dict[str, Any] = {}
|
||||
for name in ("audit-core-database", "audit-core-senders"):
|
||||
resource = remote.kubectl_json(
|
||||
["-n", "audit-core", "get", "externalsecret", name],
|
||||
label=f"read {name} state",
|
||||
)
|
||||
external_secrets[name] = resource_condition(resource)
|
||||
|
||||
bao_raw = remote.kubectl(
|
||||
["-n", "openbao", "exec", "openbao-0", "--", "bao", "status", "-format=json"],
|
||||
label="read OpenBao seal state",
|
||||
)
|
||||
try:
|
||||
bao_status = json.loads(bao_raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise PreflightError("OpenBao seal state returned invalid JSON") from exc
|
||||
image = deployment["spec"]["template"]["spec"]["containers"][0]["image"]
|
||||
state = {
|
||||
"captured_at": now.replace(microsecond=0).isoformat().replace("+00:00", "Z"),
|
||||
"node": {
|
||||
"name": node["metadata"]["name"],
|
||||
"ready": condition(node, "Ready") == "True",
|
||||
"k3s_active": remote.run(
|
||||
["systemctl", "is-active", "k3s"], label="read k3s service state"
|
||||
) == "active",
|
||||
"uptime_seconds": float(
|
||||
remote.run(["cat", "/proc/uptime"], label="read host uptime").split()[0]
|
||||
),
|
||||
},
|
||||
"platform_pg": {
|
||||
"ready_instances": cluster.get("status", {}).get("readyInstances", 0),
|
||||
"instances": cluster.get("spec", {}).get("instances", 0),
|
||||
"phase": cluster.get("status", {}).get("phase"),
|
||||
"continuous_archiving": condition(cluster, "ContinuousArchiving") == "True",
|
||||
"last_backup_succeeded": condition(cluster, "LastBackupSucceeded") == "True",
|
||||
},
|
||||
"openbao": {
|
||||
"initialized": bool(bao_status.get("initialized")),
|
||||
"sealed": bool(bao_status.get("sealed")),
|
||||
"seal_type": bao_status.get("type"),
|
||||
"threshold": bao_status.get("t"),
|
||||
"shares": bao_status.get("n"),
|
||||
},
|
||||
"external_secret_stores": stores,
|
||||
"external_secrets": external_secrets,
|
||||
"audit_core": {
|
||||
"image": image,
|
||||
"image_matches_reviewed_digest": image == EXPECTED_IMAGE,
|
||||
"ready_replicas": deployment.get("status", {}).get("readyReplicas", 0),
|
||||
"replicas": deployment.get("spec", {}).get("replicas", 0),
|
||||
"pod_uid": pod["metadata"]["uid"],
|
||||
"restart_count": restart_count,
|
||||
},
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
return state
|
||||
|
||||
|
||||
def automated_common_pass(state: dict[str, Any]) -> bool:
|
||||
return all(
|
||||
(
|
||||
state["node"]["ready"],
|
||||
state["node"]["k3s_active"],
|
||||
state["platform_pg"]["ready_instances"] == state["platform_pg"]["instances"] == 1,
|
||||
state["platform_pg"]["continuous_archiving"],
|
||||
state["platform_pg"]["last_backup_succeeded"],
|
||||
state["openbao"]["initialized"],
|
||||
not state["openbao"]["sealed"],
|
||||
all(item["ready"] for item in state["external_secret_stores"].values()),
|
||||
all(item["ready"] for item in state["external_secrets"].values()),
|
||||
state["audit_core"]["image_matches_reviewed_digest"],
|
||||
state["audit_core"]["ready_replicas"] == state["audit_core"]["replicas"] == 1,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def database_lease_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]:
|
||||
now = datetime.now(UTC)
|
||||
state = common_state(remote, now)
|
||||
secret_rv = remote.kubectl(
|
||||
["-n", "audit-core", "get", "secret", "audit-core-database", "-o", "jsonpath={.metadata.resourceVersion}"],
|
||||
label="read database Secret metadata",
|
||||
)
|
||||
description = remote.kubectl(
|
||||
["-n", "audit-core", "describe", "secret", "audit-core-database"],
|
||||
label="read database Secret key names",
|
||||
)
|
||||
keys = secret_key_names(description)
|
||||
health_code = remote.kubectl(
|
||||
["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/healthz", timeout=3).status)'],
|
||||
label="probe audit-core health",
|
||||
)
|
||||
ready_code = remote.kubectl(
|
||||
["-n", "audit-core", "exec", "deploy/audit-core", "--", "python", "-c", 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:8080/readyz", timeout=3).status)'],
|
||||
label="probe audit-core readiness",
|
||||
)
|
||||
gates = {
|
||||
"approved_window_id_recorded": bool(args.approved_window_id),
|
||||
"audit_core_owner_acknowledged": args.audit_core_owner_ack,
|
||||
"rapp_postgres_owner_acknowledged": args.rapp_postgres_owner_ack,
|
||||
"approved_synthetic_load_contract": bool(args.synthetic_load_id),
|
||||
"attended_abort_operator_named": bool(args.abort_operator),
|
||||
}
|
||||
automated = automated_common_pass(state) and keys == EXPECTED_DB_KEYS and health_code == "200" and ready_code == "200"
|
||||
return {
|
||||
"procedure": "audit-core-database-lease-recovery",
|
||||
"preflight_only": True,
|
||||
"automated_checks_passed": automated,
|
||||
"ready_for_live_execution": automated and all(gates.values()),
|
||||
"operator_gates": gates,
|
||||
"baseline": {
|
||||
"database_secret_resource_version": secret_rv,
|
||||
"database_secret_keys": sorted(keys),
|
||||
"health_status": int(health_code),
|
||||
"readiness_status": int(ready_code),
|
||||
"pod_uid": state["audit_core"]["pod_uid"],
|
||||
"restart_count": state["audit_core"]["restart_count"],
|
||||
},
|
||||
"state": state,
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
|
||||
|
||||
def reboot_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]:
|
||||
now = datetime.now(UTC)
|
||||
state = common_state(remote, now)
|
||||
backups = remote.kubectl_json(
|
||||
["-n", "databases", "get", "backup", "-l", "cnpg.io/cluster=platform-pg"],
|
||||
label="read platform-pg backup metadata",
|
||||
)
|
||||
latest = latest_completed_backup(backups, now)
|
||||
snapshot_evidence = Path(args.openbao_snapshot_evidence).resolve() if args.openbao_snapshot_evidence else None
|
||||
gates = {
|
||||
"approved_window_id_recorded": bool(args.approved_window_id),
|
||||
"openbao_snapshot_evidence_present": bool(snapshot_evidence and snapshot_evidence.is_file()),
|
||||
"unseal_quorum_attested": args.unseal_quorum_attested,
|
||||
"provider_console_access_attested": args.provider_console_attested,
|
||||
"host_cluster_platform_database_audit_owners_acknowledged": args.all_owners_ack,
|
||||
"attended_abort_operator_named": bool(args.abort_operator),
|
||||
}
|
||||
automated = automated_common_pass(state) and latest["age_hours"] <= args.max_backup_age_hours
|
||||
return {
|
||||
"procedure": "railiance01-coordinated-reboot",
|
||||
"preflight_only": True,
|
||||
"automated_checks_passed": automated,
|
||||
"ready_for_live_execution": automated and all(gates.values()),
|
||||
"operator_gates": gates,
|
||||
"latest_platform_pg_backup": latest,
|
||||
"max_backup_age_hours": args.max_backup_age_hours,
|
||||
"state": state,
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("procedure", choices=["database-lease", "node-reboot"])
|
||||
parser.add_argument("--remote", default="railiance01")
|
||||
parser.add_argument("--approved-window-id")
|
||||
parser.add_argument("--abort-operator")
|
||||
parser.add_argument("--audit-core-owner-ack", action="store_true")
|
||||
parser.add_argument("--rapp-postgres-owner-ack", action="store_true")
|
||||
parser.add_argument("--synthetic-load-id")
|
||||
parser.add_argument("--openbao-snapshot-evidence")
|
||||
parser.add_argument("--unseal-quorum-attested", action="store_true")
|
||||
parser.add_argument("--provider-console-attested", action="store_true")
|
||||
parser.add_argument("--all-owners-ack", action="store_true")
|
||||
parser.add_argument("--max-backup-age-hours", type=float, default=26.0)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
remote = Remote(args.remote)
|
||||
result = (
|
||||
database_lease_preflight(remote, args)
|
||||
if args.procedure == "database-lease"
|
||||
else reboot_preflight(remote, args)
|
||||
)
|
||||
except (OSError, ValueError, KeyError, PreflightError) as exc:
|
||||
print(f"recovery preflight failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
101
tests/test_audit_core_recovery_preflight.py
Normal file
101
tests/test_audit_core_recovery_preflight.py
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import unittest
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"audit_core_recovery_preflight",
|
||||
ROOT / "scripts" / "audit-core-recovery-preflight.py",
|
||||
)
|
||||
assert SPEC and SPEC.loader
|
||||
module = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(module)
|
||||
|
||||
|
||||
class RecoveryPreflightTests(unittest.TestCase):
|
||||
def test_condition_selects_named_condition(self) -> None:
|
||||
resource = {
|
||||
"status": {
|
||||
"conditions": [
|
||||
{"type": "Ready", "status": "True"},
|
||||
{"type": "ContinuousArchiving", "status": "False"},
|
||||
]
|
||||
}
|
||||
}
|
||||
self.assertEqual("True", module.condition(resource, "Ready"))
|
||||
self.assertEqual("False", module.condition(resource, "ContinuousArchiving"))
|
||||
self.assertIsNone(module.condition(resource, "Missing"))
|
||||
|
||||
def test_latest_backup_uses_completed_stop_time_and_reports_no_values(self) -> None:
|
||||
backups = {
|
||||
"items": [
|
||||
{
|
||||
"metadata": {"name": "older"},
|
||||
"spec": {"method": "barmanObjectStore"},
|
||||
"status": {"phase": "completed", "stoppedAt": "2026-08-22T01:00:00Z"},
|
||||
},
|
||||
{
|
||||
"metadata": {"name": "newer"},
|
||||
"spec": {"method": "barmanObjectStore"},
|
||||
"status": {"phase": "completed", "stoppedAt": "2026-08-22T02:00:00Z"},
|
||||
},
|
||||
{
|
||||
"metadata": {"name": "running"},
|
||||
"status": {"phase": "running", "stoppedAt": "2026-08-22T03:00:00Z"},
|
||||
},
|
||||
]
|
||||
}
|
||||
result = module.latest_completed_backup(
|
||||
backups, datetime(2026, 8, 22, 4, 0, tzinfo=UTC)
|
||||
)
|
||||
self.assertEqual("newer", result["name"])
|
||||
self.assertEqual(2.0, result["age_hours"])
|
||||
self.assertNotIn("data", result)
|
||||
|
||||
def test_common_gate_requires_every_boundary(self) -> None:
|
||||
state = {
|
||||
"node": {"ready": True, "k3s_active": True},
|
||||
"platform_pg": {
|
||||
"ready_instances": 1,
|
||||
"instances": 1,
|
||||
"continuous_archiving": True,
|
||||
"last_backup_succeeded": True,
|
||||
},
|
||||
"openbao": {"initialized": True, "sealed": False},
|
||||
"external_secret_stores": {"a": {"ready": True}, "b": {"ready": True}},
|
||||
"external_secrets": {"a": {"ready": True}, "b": {"ready": True}},
|
||||
"audit_core": {
|
||||
"image_matches_reviewed_digest": True,
|
||||
"ready_replicas": 1,
|
||||
"replicas": 1,
|
||||
},
|
||||
}
|
||||
self.assertTrue(module.automated_common_pass(state))
|
||||
state["openbao"]["sealed"] = True
|
||||
self.assertFalse(module.automated_common_pass(state))
|
||||
|
||||
def test_expected_database_projection_is_exact_key_set(self) -> None:
|
||||
self.assertEqual(
|
||||
{"username", "password", "host", "port", "dbname"},
|
||||
module.EXPECTED_DB_KEYS,
|
||||
)
|
||||
|
||||
def test_secret_description_parser_reads_names_not_values(self) -> None:
|
||||
description = """Name: audit-core-database
|
||||
Data
|
||||
====
|
||||
username: 20 bytes
|
||||
password: 36 bytes
|
||||
host: 50 bytes
|
||||
"""
|
||||
self.assertEqual(
|
||||
{"username", "password", "host"}, module.secret_key_names(description)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -87,7 +87,7 @@ passed. Live projection and cleanup remain waiting for the attended window.
|
|||
|
||||
```task
|
||||
id: RAILIANCE-WP-0024-T02
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "caba7fcd-014e-5253-87f7-6993731cb0c4"
|
||||
```
|
||||
|
|
@ -101,11 +101,23 @@ restart. Record preconditions, time bounds, abort conditions, and rollback.
|
|||
Done when the procedure proves the property claimed by audit-core's credential
|
||||
reader without exposing a username/password or affecting non-synthetic data.
|
||||
|
||||
**Procedure implementation (2026-08-22):**
|
||||
`docs/audit-core-database-lease-recovery.md` fixes the exact consumer, lease,
|
||||
database, synthetic-load, owner, time-bound, abort, and value-safe evidence
|
||||
contract. It requires matching the current lease internally while retaining
|
||||
only its non-usable handle, proving health 200/readiness 503/retryable request
|
||||
503 after revocation, forcing only the database ExternalSecret, and proving
|
||||
recovery with the same pod UID and restart count. The read-only helper
|
||||
`scripts/audit-core-recovery-preflight.py database-lease` passed live automated
|
||||
checks without observing Secret data. The live revocation still waits for an
|
||||
approved window, audit-core/rapp-postgres acknowledgement, a separately
|
||||
approved synthetic-load contract, and a named abort operator.
|
||||
|
||||
## T03 — Define the coordinated railiance01 reboot exercise
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0024-T03
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "09cf4065-4f8a-5b5f-beec-17f623efb56f"
|
||||
```
|
||||
|
|
@ -120,11 +132,23 @@ abort/rollback conditions.
|
|||
Done when an operator-approved window can execute from one checklist without
|
||||
guessing or relying on credential values in the record.
|
||||
|
||||
**Procedure implementation (2026-08-22):**
|
||||
`docs/railiance01-coordinated-reboot.md` now provides the single ordered
|
||||
preflight, hold point, reboot, recovery, escalation, and evidence checklist
|
||||
across host, cluster, OpenBao/ESO, platform-pg, and audit-core owners. The
|
||||
read-only helper `scripts/audit-core-recovery-preflight.py node-reboot` passed
|
||||
all automated live checks: node/k3s, platform-pg 1/1, continuous archiving,
|
||||
fresh completed Barman backup, unsealed OpenBao, ESO Valid/SecretSynced, and
|
||||
the reviewed audit-core digest 1/1 Ready. Live readiness remains fail-closed on
|
||||
the intentionally non-automated gates: an approved window and owner set,
|
||||
provider-console access, a current OpenBao Raft snapshot receipt, 2-of-3 Shamir
|
||||
quorum availability, and a named abort operator.
|
||||
|
||||
## T04 — Route the contracts and approval gates
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0024-T04
|
||||
status: todo
|
||||
status: progress
|
||||
priority: medium
|
||||
state_hub_task_id: "22f5011f-aa43-54d0-9c68-66b1b7a2d140"
|
||||
```
|
||||
|
|
@ -137,6 +161,14 @@ engagement or recovery window is explicitly acknowledged.
|
|||
Done when every participant has the same value-safe procedure, responsibility
|
||||
map, and stop conditions.
|
||||
|
||||
**Routing package (2026-08-22):** the owner-review package consists of
|
||||
`docs/audit-core-whitehat-e2-credential-projection.md`,
|
||||
`docs/audit-core-database-lease-recovery.md`, and
|
||||
`docs/railiance01-coordinated-reboot.md`. T01's live credential window remains
|
||||
separate from the new T02/T03 authorization gates. Outstanding approvals are
|
||||
stated in each procedure and enforced by the read-only preflight output rather
|
||||
than inferred from repository access.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- [ ] E2 cannot proceed without enforced sender expiry and exact-path projection.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue