RPF-WP-0015: T02 synthetic sender lane, T03 checklist revision, ops-warden re-acceptance
Addresses round-1 owner review requests from audit-core, rapp-postgres, railiance-cluster and railiance-infra; refreshes the review contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 680842@bnt-lap001 Assistant-Session: 54eb9e8e-7d6b-4f0c-b7a6-a6db62075883
This commit is contained in:
codex
parent
aee6fe79d7
commit
ce8286dff9
13 changed files with 2032 additions and 52 deletions
31
Makefile
31
Makefile
|
|
@ -534,6 +534,35 @@ audit-database-lease-recovery-exercise: ## Run the separately approved attended
|
|||
--load-driver "$(AUDIT_DATABASE_LEASE_DRIVER)" \
|
||||
--confirm RAILIANCE-WP-0024-T02:attended
|
||||
|
||||
AUDIT_SYNTHETIC_SENDER_LANE = python3 scripts/audit_core_synthetic_sender_lane.py
|
||||
AUDIT_SYNTHETIC_SENDER_NEED_APPROVAL = @test -n "$(AUDIT_DATABASE_LEASE_APPROVAL)" || { echo 'set AUDIT_DATABASE_LEASE_APPROVAL=/path/to/approved-receipt.json' >&2; exit 2; }
|
||||
AUDIT_SYNTHETIC_SENDER_NEED_CONFIRM = @test -n "$(AUDIT_SYNTHETIC_SENDER_CONFIRM)" || { echo 'set AUDIT_SYNTHETIC_SENDER_CONFIRM to the confirm string printed by preflight' >&2; exit 2; }
|
||||
|
||||
audit-synthetic-sender-test: ## Unit tests for the synthetic sender lane (no network)
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest tests.test_audit_core_synthetic_sender_lane
|
||||
|
||||
audit-synthetic-sender-preflight: ## Read-only gate for the T02 synthetic sender (no registry values read)
|
||||
$(AUDIT_SYNTHETIC_SENDER_NEED_APPROVAL)
|
||||
$(AUDIT_SYNTHETIC_SENDER_LANE) preflight --approval "$(AUDIT_DATABASE_LEASE_APPROVAL)" \
|
||||
$(if $(AUDIT_T02_TOKEN_FILE),--bearer-file "$(AUDIT_T02_TOKEN_FILE)") $(AUDIT_SYNTHETIC_SENDER_PREFLIGHT_ARGS)
|
||||
|
||||
audit-synthetic-sender-project: ## Attended: add the one T02 identity, write bearer file, restart audit-core
|
||||
$(AUDIT_SYNTHETIC_SENDER_NEED_APPROVAL)
|
||||
$(AUDIT_SYNTHETIC_SENDER_NEED_CONFIRM)
|
||||
@test -n "$(AUDIT_T02_TOKEN_FILE)" || { echo 'set AUDIT_T02_TOKEN_FILE=/abs/path/outside/git (0700 dir)' >&2; exit 2; }
|
||||
$(AUDIT_SYNTHETIC_SENDER_LANE) project --approval "$(AUDIT_DATABASE_LEASE_APPROVAL)" \
|
||||
--bearer-file "$(AUDIT_T02_TOKEN_FILE)" --confirm "$(AUDIT_SYNTHETIC_SENDER_CONFIRM)"
|
||||
|
||||
audit-synthetic-sender-cleanup: ## Attended, idempotent: remove the T02 identity after evidence, restart audit-core
|
||||
$(AUDIT_SYNTHETIC_SENDER_NEED_APPROVAL)
|
||||
$(AUDIT_SYNTHETIC_SENDER_NEED_CONFIRM)
|
||||
@test -n "$(AUDIT_T02_EVIDENCE)$(AUDIT_SYNTHETIC_SENDER_ABORT_REASON)" || { echo 'set AUDIT_T02_EVIDENCE=/path/to/exercise-result.json or AUDIT_SYNTHETIC_SENDER_ABORT_REASON=...' >&2; exit 2; }
|
||||
$(AUDIT_SYNTHETIC_SENDER_LANE) cleanup --approval "$(AUDIT_DATABASE_LEASE_APPROVAL)" \
|
||||
--confirm "$(AUDIT_SYNTHETIC_SENDER_CONFIRM)" \
|
||||
$(if $(AUDIT_T02_TOKEN_FILE),--bearer-file "$(AUDIT_T02_TOKEN_FILE)") \
|
||||
$(if $(AUDIT_T02_EVIDENCE),--exercise-evidence "$(AUDIT_T02_EVIDENCE)") \
|
||||
$(if $(AUDIT_SYNTHETIC_SENDER_ABORT_REASON),--abort-reason "$(AUDIT_SYNTHETIC_SENDER_ABORT_REASON)")
|
||||
|
||||
railiance01-reboot-preflight: ## Read-only gate for the coordinated single-node reboot
|
||||
python3 scripts/audit-core-recovery-preflight.py node-reboot $(RAILIANCE01_REBOOT_PREFLIGHT_ARGS)
|
||||
|
||||
|
|
@ -560,7 +589,7 @@ wp0024-t02-driver-register: ## Register committed audit-core driver source witho
|
|||
@python3 scripts/wp0024-t02-driver-candidate.py register \
|
||||
--repo "$(abspath ../audit-core)" --driver "$(DRIVER)" --reviewer "$(REVIEWER)"
|
||||
|
||||
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight audit-database-lease-approval-check audit-database-lease-recovery-exercise railiance01-reboot-preflight wp0024-owner-show wp0024-owner-verify wp0024-owner-status wp0024-t02-driver-show wp0024-t02-driver-status wp0024-t02-driver-register apps-pg-isolation-probe
|
||||
.PHONY: audit-synthetic-sender-test audit-synthetic-sender-preflight audit-synthetic-sender-project audit-synthetic-sender-cleanup audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight audit-database-lease-approval-check audit-database-lease-recovery-exercise railiance01-reboot-preflight wp0024-owner-show wp0024-owner-verify wp0024-owner-status wp0024-t02-driver-show wp0024-t02-driver-status wp0024-t02-driver-register apps-pg-isolation-probe
|
||||
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
|
||||
|
||||
##@ S3 service assurance (read-only)
|
||||
|
|
|
|||
|
|
@ -25,5 +25,11 @@
|
|||
"synthetic_load": {
|
||||
"contract_id": null,
|
||||
"driver_revision": null
|
||||
},
|
||||
"synthetic_sender": {
|
||||
"identity": null,
|
||||
"source": null,
|
||||
"tenant": null,
|
||||
"expires_at": null
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -48,6 +48,69 @@ The separately approved load driver is an executable that accepts `baseline`,
|
|||
shape. Unexpected keys—including a bearer accidentally returned as `token`—
|
||||
make the harness fail without echoing driver output.
|
||||
|
||||
## Synthetic sender projection lane
|
||||
|
||||
The load driver (`audit-core` `scripts/t02_synthetic_load_driver.py`, registered
|
||||
at `8c8bcf4`) needs exactly one temporary audit-core sender identity. The
|
||||
reviewed, reusable lane is `scripts/audit_core_synthetic_sender_lane.py`. It
|
||||
reuses the live-proven registry/reload helpers of the dated Whitehat `-02`
|
||||
procedure by import and the T02 receipt, attest-chain and preflight gates of
|
||||
the lease harness; the dated `-01`/`-02` procedures are not copied.
|
||||
|
||||
**Receipt binding.** The identity is named only in the approved T02 receipt
|
||||
(see `docs/audit-core-database-lease-approval.example.json`), in a
|
||||
`synthetic_sender` block with exactly these keys:
|
||||
|
||||
```json
|
||||
"synthetic_sender": {
|
||||
"identity": "t02-synthetic-<window>",
|
||||
"source": "<named synthetic source, contains 'synthetic'>",
|
||||
"tenant": "tenant:trial:<name>",
|
||||
"expires_at": "<RFC3339 UTC, not later than window.end>"
|
||||
}
|
||||
```
|
||||
|
||||
The lane accepts the receipt only if it also passes the lease harness's own
|
||||
approval validation (approved status, ≤15-minute window, abort operator, all
|
||||
three owner acknowledgements) and `synthetic_load.contract_id` equals the
|
||||
registered driver interface `RAILIANCE-WP-0024-T02-SYNTHETIC-01`. The registry
|
||||
entry it adds is `sources=[source]`, `tenants=[tenant]`, `may_write=true`,
|
||||
`may_read=true`, `expires_at=<receipt value>`; projection refuses when the
|
||||
name, source or tenant already appears on any existing identity. The same
|
||||
`identity`, `source` and `tenant` values are what the operator passes to the
|
||||
driver as `AUDIT_T02_SOURCE`/`AUDIT_T02_TENANT`; the confirm string is
|
||||
`audit-core-synthetic-sender:<approval_id>:attended` (printed by preflight).
|
||||
|
||||
**Ordering.** The bearer reaches audit-core only through the registry at
|
||||
`platform/workloads/audit-core/senders` → ExternalSecret `audit-core-senders`
|
||||
→ env `AUDIT_CORE_SENDERS`, so both projection and cleanup restart
|
||||
`deploy/audit-core`. The restarts must stay outside the restart-free proof:
|
||||
|
||||
1. `make audit-synthetic-sender-preflight` — read-only; reads registry metadata
|
||||
only (add `AUDIT_SYNTHETIC_SENDER_PREFLIGHT_ARGS=--inspect-registry` for an
|
||||
attended in-memory conflict check).
|
||||
2. `make audit-synthetic-sender-project` — only within the first 3 minutes of
|
||||
the window and with ≥10 minutes of identity lifetime left. Writes the
|
||||
bearer to a new mode-`0600` file (`AUDIT_T02_TOKEN_FILE`, absolute, in an
|
||||
operator-owned `0700` directory outside every Git work tree; never stdout),
|
||||
adds the one identity, forces `audit-core-senders` sync and runs
|
||||
`rollout restart deploy/audit-core`, then waits for one Ready restarted pod
|
||||
and prints its UID. Any failure removes the identity (restarting again) and
|
||||
shreds the bearer file.
|
||||
3. The T02 exercise (`make audit-database-lease-recovery-exercise`). Its
|
||||
baseline must capture the post-projection pod UID; never project while the
|
||||
exercise is running.
|
||||
4. Record the exercise's JSON result (it is the evidence file).
|
||||
5. `make audit-synthetic-sender-cleanup AUDIT_T02_EVIDENCE=<result.json>` —
|
||||
refuses unless that evidence names this `approval_id`, or an explicit
|
||||
`AUDIT_SYNTHETIC_SENDER_ABORT_REASON` is given for an aborted run. Removes
|
||||
only the exact named identity (refusing if the name carries a different
|
||||
scope), restarts audit-core only if something was removed, shreds the
|
||||
bearer file, and confirms 1/1 Ready. Re-running is a no-op.
|
||||
|
||||
If cleanup is missed, audit-core's `expires_at` enforcement makes the identity
|
||||
inert at window end; the attended cleanup is still required to remove it.
|
||||
|
||||
## Attended sequence
|
||||
|
||||
Keep one terminal attached from baseline through recovery. A value-handling
|
||||
|
|
@ -82,8 +145,9 @@ bodies and clear all transient JSON from memory or mode-`0600` temporary files.
|
|||
7. Prove recovery: `/readyz` returns 200, the retrying synthetic request reaches
|
||||
accepted or duplicate, the pre-exercise event remains readable, pod UID is
|
||||
unchanged, and restart count is unchanged.
|
||||
8. Stop the load, remove its separately approved fixture/identity, and confirm
|
||||
the predecessor lease is revoked. The replacement remains managed by ESO
|
||||
8. Stop the load, confirm the predecessor lease is revoked, record the
|
||||
evidence, then remove the identity with the synthetic sender lane cleanup
|
||||
(which restarts audit-core after the evidence is captured). The replacement remains managed by ESO
|
||||
and expires through its normal database role TTL.
|
||||
|
||||
Only inside the approved window, with the receipt and reviewed driver:
|
||||
|
|
|
|||
|
|
@ -8,11 +8,13 @@ claim. This document does not authorize a reboot.
|
|||
|
||||
| Phase | Accountable owner | Required evidence |
|
||||
| --- | --- | --- |
|
||||
| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator |
|
||||
| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns |
|
||||
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled |
|
||||
| Window, notifications, go/no-go | operator + `railiance-platform` | approval/window id; new scenario id; fresh human decision; named driver |
|
||||
| Independent abort authority (RPF-WP-0017) | `railiance-infra` | named abort owner acknowledgement; may stop the window at any step without a platform vote |
|
||||
| Attended login containment | `railiance-platform` | `reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` accepting the exact ops-warden revision |
|
||||
| Host access, provider console, reboot, firewall, time | `railiance-infra` | SSH; same-day `RAIL01-CONSOLE-*` receipt; host returns; UFW/NTP/tunnel checks |
|
||||
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; named core platform workloads Ready; failing-pod diff |
|
||||
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
|
||||
| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary |
|
||||
| All CNPG clusters (`platform-pg` backup/WAL) | `rapp-postgres` (+ owning app for each cluster) | per-cluster UID, primary PVC UID, systemID, 1/1 primary; fresh `platform-pg` Barman backup |
|
||||
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
|
||||
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |
|
||||
|
||||
|
|
@ -29,22 +31,71 @@ Run the value-safe, read-only gate:
|
|||
python3 scripts/audit-core-recovery-preflight.py node-reboot
|
||||
```
|
||||
|
||||
The automated checks require one Ready node, active k3s, healthy 1/1
|
||||
`platform-pg`, continuous archiving, a successful backup no older than 26
|
||||
hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and
|
||||
the reviewed 1/1 Ready audit-core image. The following are human gates and make
|
||||
`ready_for_live_execution` remain false until supplied:
|
||||
The automated checks (`automated_checks` in the output) require:
|
||||
|
||||
- one Ready node, active k3s, and the boot id recorded;
|
||||
- `timedatectl` `NTPSynchronized=yes` (receipt/backup ages are untrusted otherwise);
|
||||
- UFW active with exactly the declared allow set — `OpenSSH` and Nydus
|
||||
`2224/tcp` (v4/v6) — and `6443`/`8472` not open (railiance-infra
|
||||
RAIL-HO-WP-0009 / Goss declaration);
|
||||
- the workstation `k3s-api-railiance01` tunnel answering (`bridge check`);
|
||||
- every named core platform workload Ready, plus every other Deployment and
|
||||
StatefulSet in their namespaces: CoreDNS, local-path-provisioner, traefik,
|
||||
metrics-server, cert-manager, the CNPG operator, external-secrets,
|
||||
knative-serving (activator, controller, net-kourier-controller) and the
|
||||
kourier gateway, argocd, and openbao;
|
||||
- every CNPG cluster, discovered live (today `apps-pg`, `forgejo-db`,
|
||||
`net-kingdom-pg`, `platform-pg`, `platform-pg-2`, `state-hub-db`,
|
||||
`target-revenue-pg`, `user-engine-pg`; a vanished expected cluster fails),
|
||||
1/1 Ready, current primary on a healthy PVC, `ContinuousArchiving=True` where
|
||||
backup is configured, with cluster UID, primary PVC UID, and PostgreSQL
|
||||
`systemID` (from `pg_controldata`, value-safe) recorded;
|
||||
- a completed `platform-pg` backup no older than 26 hours;
|
||||
- every ClusterSecretStore `Valid` and every ExternalSecret `SecretSynced`;
|
||||
- unsealed OpenBao and the reviewed audit-core digest in both the Deployment
|
||||
spec and the running pod (`imageID`), 1/1 Ready;
|
||||
- before the reboot, every Terminating PVC resolved or explicitly listed with
|
||||
`--accept-terminating-pvc`.
|
||||
|
||||
The output also records the all-namespace failing-pod baseline
|
||||
(`reboot_evidence.failing_pods`). On 2026-10-14 it held six pre-existing pods:
|
||||
three `coulomb/ihp-railiance-probe-*`, `forgejo/forgejo-gitea-*`,
|
||||
`forgejo/forgejo-runner-*`, and `telemetry/telemetry-smtp-proof-20260928-*`.
|
||||
Keep the pre-reboot JSON; it is the comparison input after the reboot.
|
||||
|
||||
The following are human gates and make `ready_for_live_execution` remain false
|
||||
until supplied:
|
||||
|
||||
- approved window id and acknowledgements from all five owners;
|
||||
- a named attended driver/abort operator;
|
||||
- independent provider-console access, not SSH alone;
|
||||
- a new scenario id (`--scenario-id`) and a fresh human GO decision
|
||||
(`--human-decision-id`). Consumed decisions `449a697a-…` and `85724c0c-…` and
|
||||
scenario `WARDEN-WP-0027-T02-DRILL-20260822-01` are not reusable and are
|
||||
denied by the preflight;
|
||||
- the RPF-WP-0017 re-acceptance receipt
|
||||
`reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` recording
|
||||
acceptance of the exact ops-warden revision used for the attended login. The
|
||||
2026-10-14 revision `11b2a9d…` was reviewed as `reject`, so this gate is
|
||||
currently closed;
|
||||
- a named attended driver/abort operator and the `railiance-infra`
|
||||
independent abort owner acknowledgement (`--infra-abort-owner-ack`);
|
||||
- independent provider-console access, not SSH alone: a value-safe
|
||||
`RAIL01-CONSOLE-<YYYYMMDDTHHMMSSZ>` receipt recorded by railiance-infra under
|
||||
its `docs/evidence/railiance01-console-access/` (procedure:
|
||||
railiance-infra `docs/railiance01-console-access.md`), passed with
|
||||
`--provider-console-receipt`. The preflight accepts it only when it is dated
|
||||
the window day, names the approved window id, and has `console_shows_host`
|
||||
and `independent_of_guest_network` true. No out-of-band console is
|
||||
established today, so T03 cannot pass this gate until railiance-infra
|
||||
RAIL-HO-WP-0011-T09 is done;
|
||||
- a current non-secret OpenBao Raft snapshot receipt that validates against
|
||||
`docs/openbao-reboot-snapshot-receipt.example.json`, the live cluster id and
|
||||
Raft index, and the default 24-hour freshness bound;
|
||||
- explicit availability of two of three Shamir shares through their approved
|
||||
out-of-band custody paths.
|
||||
|
||||
Do not reuse old restore-drill templates as current snapshot evidence. Do not
|
||||
Receipt validation failures are reported in `receipt_gate_failures` and never
|
||||
abort evidence capture. Do not reuse old restore-drill templates as current
|
||||
snapshot evidence. Do not
|
||||
put a snapshot, unseal share, token, password, or Secret response in Git or
|
||||
State Hub.
|
||||
|
||||
|
|
@ -54,18 +105,28 @@ State Hub.
|
|||
backup-restore, and security-engagement mutations on `railiance01`.
|
||||
2. Confirm no temporary Whitehat or other expiring projection will cross the
|
||||
window. Clean it first if present.
|
||||
3. Capture the read-only preflight result and audit-core chain-head attestation.
|
||||
3. Capture the read-only preflight result (keep the JSON as the pre-reboot
|
||||
evidence: boot id, failing-pod baseline, CNPG UIDs/PVC UIDs/systemIDs) and
|
||||
the audit-core chain-head attestation.
|
||||
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
|
||||
is healthy, and the PVC is Bound. If the daily backup is stale, take and
|
||||
verify an on-demand backup before continuing.
|
||||
5. Create and escrow a fresh OpenBao Raft snapshot through the approved
|
||||
verify an on-demand backup before continuing. `state-hub-db` and
|
||||
`user-engine-pg` have no CNPG backup configured (nor do `net-kingdom-pg` and
|
||||
`target-revenue-pg`); their owners should take and verify a fresh backup
|
||||
through their own lane (for `user-engine-pg`, the rapp-user-engine backup
|
||||
CronJob) or record an explicit waiver before the window.
|
||||
5. Resolve the stuck Terminating PVC `platform-pg-drill/platform-pg-drill-1`
|
||||
(owner: `rapp-postgres`, separately approved) or list it for the window with
|
||||
`--accept-terminating-pvc platform-pg-drill/platform-pg-drill-1`.
|
||||
6. Create and escrow a fresh OpenBao Raft snapshot through the approved
|
||||
encrypted backup lane. Record only its evidence receipt and verification.
|
||||
Pass that receipt to the preflight with
|
||||
`--openbao-snapshot-evidence /path/to/receipt.json`; a merely existing,
|
||||
stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied.
|
||||
6. Have both Shamir share custodians present. Prove provider-console access and
|
||||
SSH access. Record the exact restart and abort operators.
|
||||
7. Re-run the preflight with the approval and attestation flags. The result
|
||||
7. Have both Shamir share custodians present. Prove provider-console access and
|
||||
SSH access (same-day `RAIL01-CONSOLE-*` receipt). Record the exact restart
|
||||
operator, abort operator, and the railiance-infra independent abort owner.
|
||||
8. Re-run the preflight with the approval and attestation flags. The result
|
||||
must say `ready_for_live_execution: true`. Pause for the final operator
|
||||
go/no-go; this is the last reversible point.
|
||||
|
||||
|
|
@ -75,17 +136,33 @@ State Hub.
|
|||
console. Do not reinstall the OS or k3s.
|
||||
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
|
||||
If SSH does not return, use the provider console; do not change firewall
|
||||
declarations as an ad hoc recovery.
|
||||
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod
|
||||
are running. Application readiness may still be false at this point.
|
||||
4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity,
|
||||
primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap.
|
||||
declarations as an ad hoc recovery. `railiance-infra` verifies: the boot id
|
||||
differs from the pre-reboot capture and exactly one boot occurred since it;
|
||||
UFW is active with only `OpenSSH` + `2224/tcp`, `6443`/`8472` not open; the
|
||||
`k3s-api-railiance01` tunnel answers; and `timedatectl` reports
|
||||
`NTPSynchronized=yes`. Do not unseal, run OIDC/ESO checks, or trust any
|
||||
receipt or backup age until time is synchronized.
|
||||
3. `railiance-cluster` verifies these core platform workloads are Running and
|
||||
reconciled: CoreDNS, local-path-provisioner, traefik, metrics-server,
|
||||
cert-manager (+ cainjector/webhook), CNPG operator
|
||||
(`cnpg-system/cnpg-controller-manager`), external-secrets (+ webhook/cert
|
||||
controller), knative-serving (activator, controller, net-kourier-controller)
|
||||
and `kourier-system/3scale-kourier-gateway`, argocd, and the OpenBao pod
|
||||
(Running but sealed, so not Ready yet). Application readiness may still be
|
||||
false at this point. Images not already cached on the node cannot be pulled
|
||||
until in-cluster Forgejo (`forgejo.coulomb.social`) and `forgejo-db` are up;
|
||||
`ImagePullBackOff` on such pods is an expected lag, not a failure, until
|
||||
Forgejo is serving — escalate only if it persists after Forgejo is Ready.
|
||||
4. `rapp-postgres` (with each cluster's owner) verifies for every CNPG cluster
|
||||
the same cluster UID, primary PVC UID, and `systemID` as the pre-reboot
|
||||
evidence, primary 1/1 Ready, `ContinuousArchiving=True` where configured,
|
||||
and no re-bootstrap. A changed systemID or PVC UID is a stop condition.
|
||||
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
|
||||
non-recorded share ceremony. Run `make openbao-status` and
|
||||
`make openbao-verify-post-unseal` after it reports initialized and unsealed.
|
||||
6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and
|
||||
the audit-core sender/database ExternalSecrets are SecretSynced. If they
|
||||
remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
|
||||
6. Verify every ClusterSecretStore is `Valid` and every ExternalSecret is
|
||||
`SecretSynced` (not only `openbao-audit-core`/`openbao-audit-core-database`
|
||||
and the audit-core projections). If they remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
|
||||
the platform owner may restart only Deployment
|
||||
`external-secrets/external-secrets`, then recheck reconciliation.
|
||||
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
|
||||
|
|
@ -94,16 +171,33 @@ State Hub.
|
|||
8. Run audit-core integrity verification against the pre-reboot chain head and
|
||||
one separately approved synthetic E2. Confirm no chain break, the prior
|
||||
marker remains readable, and the new event is accepted exactly once.
|
||||
9. Re-run the read-only preflight, end the mutation freeze, and notify all
|
||||
owners with value-safe evidence.
|
||||
9. Re-run the read-only preflight in post-reboot mode:
|
||||
|
||||
```bash
|
||||
python3 scripts/audit-core-recovery-preflight.py node-reboot \
|
||||
--pre-reboot-evidence /path/to/pre-reboot-preflight.json
|
||||
```
|
||||
|
||||
`reboot_comparison.passed` must be true: boot id changed with exactly one
|
||||
boot since the pre-reboot capture, no CNPG identity change, and no failing
|
||||
pod outside the pre-reboot baseline (`failing_pod_diff.new_failing` empty).
|
||||
Then end the mutation freeze and notify all owners with value-safe evidence.
|
||||
|
||||
## Stop, escalation, and recovery boundaries
|
||||
|
||||
- Before reboot: abort for missing approval, owner, console access, unseal
|
||||
- Before reboot: abort for missing approval, owner, console receipt, unseal
|
||||
quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
|
||||
or any overlapping live mutation.
|
||||
- After reboot: retain the outage and escalate for a missing host, node/PVC
|
||||
identity change, CNPG re-bootstrap, OpenBao initialization loss, failed
|
||||
accepted ops-warden revision, new scenario id/fresh decision, NTP sync, or
|
||||
any overlapping live mutation. `railiance-infra` may abort independently.
|
||||
- Attended login: if the login emits any unexpected credential output (token,
|
||||
secret id, password, key, or unrecognised value-shaped text), stop
|
||||
immediately: self-revoke the issued credential, abort the window, and never
|
||||
retry under the same scenario or decision. A new scenario id and a fresh
|
||||
human decision are required before any further attempt.
|
||||
- After reboot: retain the outage and escalate for a missing host, more than
|
||||
one boot, firewall drift, unsynchronized time, node/PVC/systemID identity
|
||||
change, CNPG re-bootstrap, new failing pods outside the baseline (other than
|
||||
the expected pre-Forgejo image-pull lag), OpenBao initialization loss, failed
|
||||
unseal, audit chain break, or any evidence of data loss.
|
||||
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
|
||||
recover PostgreSQL in place, delete a PVC, or weaken network policy as a
|
||||
|
|
@ -114,7 +208,9 @@ State Hub.
|
|||
|
||||
## Evidence contract
|
||||
|
||||
Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id
|
||||
Allowed: approval/window ids, scenario and decision ids, receipt ids/paths,
|
||||
owner acknowledgements, CNPG cluster UIDs and systemIDs, UFW rule targets,
|
||||
NTP sync boolean, failing pod names, timestamps, host boot id
|
||||
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
|
||||
OpenBao initialized/sealed booleans and version, resource readiness conditions,
|
||||
Secret resource versions and key names, HTTP status sequence, audit chain-head
|
||||
|
|
|
|||
|
|
@ -33,7 +33,8 @@
|
|||
"docs/audit-core-database-lease-recovery.md",
|
||||
"docs/railiance01-coordinated-reboot.md",
|
||||
"scripts/audit-core-recovery-preflight.py",
|
||||
"scripts/audit-core-database-lease-recovery.py"
|
||||
"scripts/audit-core-database-lease-recovery.py",
|
||||
"scripts/audit_core_synthetic_sender_lane.py"
|
||||
],
|
||||
"assertions": [
|
||||
"The database-lease exercise uses only synthetic load and proves health 200, readiness and requests fail retryably with 503, then recover without changing the audit-core pod UID or restart count.",
|
||||
|
|
@ -120,7 +121,8 @@
|
|||
"scripts/audit-core-recovery-preflight.py",
|
||||
"scripts/audit-core-database-lease-recovery.py",
|
||||
"docs/audit-core-database-lease-approval.example.json",
|
||||
"docs/openbao-reboot-snapshot-receipt.example.json"
|
||||
"docs/openbao-reboot-snapshot-receipt.example.json",
|
||||
"scripts/audit_core_synthetic_sender_lane.py"
|
||||
],
|
||||
"assertions": [
|
||||
"T02 remains gated on a named window, abort operator, exact owner receipts and a revision-pinned synthetic-load driver; repository access alone cannot revoke a lease.",
|
||||
|
|
@ -136,11 +138,12 @@
|
|||
}
|
||||
},
|
||||
"artifact_sha256": {
|
||||
"docs/audit-core-database-lease-approval.example.json": "360d49d0bea7ca376405974403036bff080533728f133d237f30f4ee3bb943e0",
|
||||
"docs/audit-core-database-lease-recovery.md": "5aa8f2723d7216d775990e0b2cd770290284b62d185526a9dffe7d6a4f925d95",
|
||||
"docs/audit-core-database-lease-approval.example.json": "87e57e3290074508c6517938acf8c7fc1a75d977b8ff630f247a4ff5d2ee45c6",
|
||||
"docs/audit-core-database-lease-recovery.md": "630e9324d61fddc50c2c364649064a316a7b406b45cc6b4799cf0c4f8319b237",
|
||||
"docs/openbao-reboot-snapshot-receipt.example.json": "b95903a3e1aacb607eecc3f0440f08e62480d09da47bd7ab96c3eb8814f2eaee",
|
||||
"docs/railiance01-coordinated-reboot.md": "9969c2f498587710154e8d9a650fb8a13159e9e7b99918291dbdf577b131b7e7",
|
||||
"scripts/audit-core-database-lease-recovery.py": "0a214a492af863a0e396625b50fbd795623f7e4b928240cb43f9becfac4fd387",
|
||||
"scripts/audit-core-recovery-preflight.py": "248c663bad31860dd4f62870db00f6d624143d18c18f86634a39ceda12c7bc66"
|
||||
"docs/railiance01-coordinated-reboot.md": "1cbe079ca34252c1638879f0c03cf5b96d9a0078de5047ecf0cb5b3857a7121d",
|
||||
"scripts/audit-core-database-lease-recovery.py": "1de02f13dcd4e87e65de1c97ae14567f2b8e4a51aa0423e3fca78a224cf374b3",
|
||||
"scripts/audit-core-recovery-preflight.py": "41ee3889db008ed90d51b855fc3470244051e7e3d7ba6c1c9256c3d115bbb23d",
|
||||
"scripts/audit_core_synthetic_sender_lane.py": "e71b3b3d56472a1cc91fc093d327a72116d89d07c1676f44bc24a8da36154bd4"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
298
reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json
Normal file
298
reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json
Normal file
|
|
@ -0,0 +1,298 @@
|
|||
{
|
||||
"interface": "railiance.attended-login-containment-reacceptance-review",
|
||||
"workplan_id": "RPF-WP-0017",
|
||||
"source_repo": "ops-warden",
|
||||
"current_decision": "accept",
|
||||
"current_reviewed_revision": "6e1430073c64ed79204cdf676f18bfa41812d5d2",
|
||||
"reviews": [
|
||||
{
|
||||
"interface": "railiance.attended-login-containment-reacceptance-review",
|
||||
"version": 1,
|
||||
"workplan_id": "RPF-WP-0017",
|
||||
"task_id": "RPF-WP-0017-T01",
|
||||
"owner": "railiance-platform",
|
||||
"source_repo": "ops-warden",
|
||||
"baseline_revision": "0fae0904ce8d8694338dd53a8a79abec5fec788d",
|
||||
"baseline_owner_receipt": "ops-warden@bc1966da825fbeaa4d83947a4f942b5d5f085408:docs/evidence/RAILIANCE-WP-0026-T01-ops-warden-receipt.json",
|
||||
"reviewed_revision": "11b2a9d7bc97f5fe8606927c17fe8acc59b79cd0",
|
||||
"reviewed_paths": [
|
||||
"src/warden/cli.py",
|
||||
"src/warden/proxy.py",
|
||||
"tests/test_proxy.py"
|
||||
],
|
||||
"intervening_commits": [
|
||||
{
|
||||
"revision": "b4c1d3900a528bbb9f2714d59403757426185d63",
|
||||
"date": "2026-09-01",
|
||||
"summary": "Removed `login_output.strip()` from the attended-login fail-closed condition (WARDEN-WP-0036-T01). A login with returncode 0 and a populated helper now proceeds to child handoff even when the client emitted stdout/stderr."
|
||||
},
|
||||
{
|
||||
"revision": "8f01eefb1e958d060adfb78ea214464b730edd4d",
|
||||
"date": "2026-09-01",
|
||||
"summary": "Propagates the caller's ~/.config/warden/warden.yaml path as WARDEN_CONFIG into the isolated child env. Path only, not a credential; no criterion impact."
|
||||
},
|
||||
{
|
||||
"revision": "4c3a0f4d3a0b1b70c791ed03148205da6eebcfc2",
|
||||
"date": "2026-09-28",
|
||||
"summary": "Introduced AttendedLoginError with distinct exit codes 10-14 and phase outcomes in both audit files; child output now fails closed on any byte including whitespace, checked before returncode. Containment-strengthening for the child phase; no regression."
|
||||
}
|
||||
],
|
||||
"criteria": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Isolated writable token-helper location created before authentication, mode 0600 with private parents, deterministic cleanup",
|
||||
"result": "pass",
|
||||
"evidence": "_prepare_attended_login_home unchanged since baseline: HOME probe, 0700 root and session dirs, O_EXCL|O_NOFOLLOW 0600 helper with preflight write/fsync/truncate, ownership/mode asserted; finally-block rmtree + rmdir. 4c3a0f4 only reclassifies cleanup failure as AttendedLoginError(exit 14, cleanup_failed). Tests: test_attended_login_persistence_failure_revokes_and_cleans, test_cli_attended_result_and_both_audits."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Read-only or unusable default home fails before OIDC begins",
|
||||
"result": "pass",
|
||||
"evidence": "Home preflight runs before any subprocess and is unchanged since baseline. test_attended_login_refuses_read_only_home_before_auth asserts subprocess.run is never invoked."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Adapter-owned stdout/stderr containment; no inherited agent-visible stdio",
|
||||
"result": "pass",
|
||||
"evidence": "_contained_run still uses PIPE for stdout and stderr for login, child, and revoke. Login output is held only for revocation and is never printed, returned, or audited; error messages are value-free. test_attended_login_contained_success_output_never_escapes and the parametrized unexpected-output test assert the sentinel is absent from capsys, the exception text, argv, and both audit logs."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Unexpected credential-bearing or otherwise non-allowlisted output fails closed and triggers immediate self-revocation",
|
||||
"result": "fail",
|
||||
"evidence": "b4c1d39 changed proxy.py's login check from `if login.returncode != 0 or login_output.strip() or not helper_valid` to `if login.returncode != 0 or not helper_valid`. Login-phase output is no longer an independent fail-closed trigger, and there is no allowlist. A login that exits 0 with a populated helper but emits arbitrary or credential-bearing stdout/stderr now proceeds to the reviewed child instead of failing closed. The default `-format=json` makes credential-bearing login output the expected case. The output is still contained (criterion 3 holds), but this criterion requires fail-closed on non-allowlisted output, and the baseline receipt outcome `unexpected_stdout_and_stderr_fail_closed` is no longer true for the login phase. The child phase still meets the criterion (4c3a0f4: any child byte gives exit 12 child_output with revocation)."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Focused tests cover success handoff, unwritable home refusal, helper persistence failure, unexpected stdout, unexpected stderr, immediate revocation, cleanup, and value absence",
|
||||
"result": "fail",
|
||||
"evidence": "The focused suite passes and covers success, home refusal, persistence failure, revocation, cleanup, value absence, and the child-output/exit-code phases. However, test_attended_login_unexpected_output_is_contained_revoked_and_cleaned (stdout/stderr) now passes only because its fake login leaves the helper empty, so it exercises the persistence-failure path, not an unexpected-output trigger. The new test_attended_login_contained_success_output_never_escapes asserts that credential-bearing login output with a populated helper succeeds (rc 0, child runs). No test proves that unexpected login stdout or stderr fails closed on its own, so the coverage required by the criterion is absent."
|
||||
}
|
||||
],
|
||||
"tests": [
|
||||
{
|
||||
"command": "uv run pytest -q tests/test_proxy.py tests/test_plan.py",
|
||||
"passed": true,
|
||||
"passed_count": 68,
|
||||
"failed_count": 0
|
||||
},
|
||||
{
|
||||
"command": "uv run pytest -q",
|
||||
"passed": false,
|
||||
"passed_count": 610,
|
||||
"failed_count": 1,
|
||||
"deselected_count": 4,
|
||||
"note": "Single failure tests/test_agent_read_boundary_check.py::TestGeneratedArtifact::test_artifact_is_current (registry/generated/high-risk-data-paths.yaml stale vs catalog). It is unrelated to attended-login containment but means the repository is not green at the reviewed revision."
|
||||
},
|
||||
{
|
||||
"command": "uv run ruff check src/warden/proxy.py src/warden/cli.py tests/test_proxy.py",
|
||||
"passed": true
|
||||
}
|
||||
],
|
||||
"decision": "reject",
|
||||
"decision_rationale": "Criteria 1-3 still hold. Criterion 4 regressed in b4c1d39 (WARDEN-WP-0036): login-phase output is no longer a fail-closed trigger, so the behaviour railiance-platform accepted at 0fae0904 has been changed without a railiance-platform re-review. Criterion 5's unexpected-output coverage is now satisfied only incidentally through the persistence-failure path. The relaxation may be a deliberate design choice, since output is still never forwarded. If so, RPF-WP-0017 acceptance criterion 4 must be formally amended by railiance-platform, with an explicit allowlist for contained `bao login -format=json` output and railiance-infra concurrence, before this revision can be accepted. Otherwise ops-warden must restore the login-output fail-closed check and add a populated-helper unexpected-output test.",
|
||||
"observations": [
|
||||
"WARDEN-WP-0036-T02 records a live governed OpenBao platform-admin operation on 2026-09-01 using the post-b4c1d39 code. That revision had not been re-accepted under RPF-WP-0017. This review did not verify the operation; it is recorded for railiance-infra follow-up.",
|
||||
"8f01eef forwards a non-secret config path into the isolated child; acceptable, but note that the child now reads caller-home configuration despite HOME isolation."
|
||||
],
|
||||
"live_oidc_performed": false,
|
||||
"openbao_calls_performed": false,
|
||||
"live_drill_authorized": false,
|
||||
"secret_values_observed": false,
|
||||
"sensitive_material_recorded": false,
|
||||
"reviewer": "claude/railiance-platform-ops-warden-review",
|
||||
"reviewed_at": "2026-10-14T10:29:14Z"
|
||||
},
|
||||
{
|
||||
"interface": "railiance.attended-login-containment-reacceptance-review",
|
||||
"version": 1,
|
||||
"workplan_id": "RPF-WP-0017",
|
||||
"task_id": "RPF-WP-0017-T01",
|
||||
"owner": "railiance-platform",
|
||||
"source_repo": "ops-warden",
|
||||
"baseline_revision": "0fae0904ce8d8694338dd53a8a79abec5fec788d",
|
||||
"supersedes_review_of": "11b2a9d7bc97f5fe8606927c17fe8acc59b79cd0",
|
||||
"reviewed_revision": "8a426fba696c024a24430e8d3bcb239463d51f51",
|
||||
"fix_revision": "244f1ae2ca538d36b0dc43032b26f5eca85c6c66",
|
||||
"reviewed_paths": [
|
||||
"src/warden/proxy.py",
|
||||
"tests/test_proxy.py",
|
||||
"registry/generated/high-risk-data-paths.yaml"
|
||||
],
|
||||
"intervening_commits": [
|
||||
{
|
||||
"revision": "244f1ae2ca538d36b0dc43032b26f5eca85c6c66",
|
||||
"date": "2026-10-14",
|
||||
"summary": "WARDEN-WP-0036: restores login output as an independent fail-closed trigger. Login proceeds only if returncode 0, helper populated (0600, owned, non-empty) AND output allowlisted (_login_output_allowed); otherwise single self-revocation and AttendedLoginError(exit 10, login_failed), no retry."
|
||||
},
|
||||
{
|
||||
"revision": "8a426fba696c024a24430e8d3bcb239463d51f51",
|
||||
"date": "2026-10-14",
|
||||
"summary": "Regenerates registry/generated/high-risk-data-paths.yaml; diff is generated_at and catalog_revision/catalog_revision_date only. Restores test_artifact_is_current. No criterion impact."
|
||||
}
|
||||
],
|
||||
"criteria": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Isolated writable token-helper location created before authentication, mode 0600 with private parents, deterministic cleanup",
|
||||
"result": "pass",
|
||||
"evidence": "_prepare_attended_login_home and the finally-block cleanup (rmtree + rmdir, exit 14 cleanup_failed) are unchanged by 244f1ae. New tests assert the .warden-attended-login root is absent after non-allowlisted fail-closed."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Read-only or unusable default home fails before OIDC begins",
|
||||
"result": "pass",
|
||||
"evidence": "Home preflight unchanged; test_attended_login_refuses_read_only_home_before_auth still passes."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Adapter-owned stdout/stderr containment; no inherited agent-visible stdio",
|
||||
"result": "pass",
|
||||
"evidence": "_contained_run still PIPEs stdout/stderr for login, child, revoke. The allowlist functions return bool only; the new error suffix ' (login client produced non-allowlisted output)' is a fixed string with no captured bytes. Parametrized tests assert the sentinel is absent from the exception text and capsys out/err."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Unexpected credential-bearing or otherwise non-allowlisted output fails closed and triggers immediate self-revocation",
|
||||
"result": "pass",
|
||||
"evidence": "proxy.py: `if login.returncode != 0 or not helper_valid or not output_allowed` -> _revoke_contained -> AttendedLoginError(exit 10, login_failed) before child handoff; no retry loop. stdout allowlist: empty, or exactly one JSON object (json.loads full-document, duplicate keys rejected, Unicode/ValueError/RecursionError caught) whose top-level keys are within a fixed 9-key typed set, with `auth` required as an object containing `client_token` (str) and only a fixed 12-key typed auth set; bool is not accepted where int is expected; NaN/floats rejected. stderr allowlist: empty, or a fullmatch of the OpenBao OIDC prompt (browser-launch or open-link variant), exactly one `https://\\S+` URL on its own line, optional 'Waiting for OIDC authentication to complete...' line, whitespace only otherwise. Reviewer probes: two prompts, CRLF, two URLs on one line, trailing text after the wait line, and non-ASCII whitespace inside the URL are all rejected."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Focused tests cover success handoff, unwritable home refusal, helper persistence failure, unexpected stdout, unexpected stderr, immediate revocation, cleanup, and value absence",
|
||||
"result": "pass",
|
||||
"evidence": "New test_attended_login_non_allowlisted_output_with_populated_helper_fails_closed (13 cases: stdout text, stderr text, stderr appended to valid prompt, JSON + trailing text, two JSON objects, extra top-level key, extra auth key, wrong type, no auth, JSON array, duplicate key, malformed JSON, non-UTF-8) uses a populated helper so only the output trigger can fire. It asserts exit 10/login_failed, sentinel absence, call sequence exactly [login, token revoke -self] (no retry, child never runs), and removal of the private root. test_attended_login_allowlisted_output_hands_off covers the 5 allowlisted shapes. The earlier persistence-failure and CLI dual-audit tests still pass (the audit test now uses JSON-shaped login output)."
|
||||
}
|
||||
],
|
||||
"allowlist_relaxation_note": "Criterion 4 as baselined at 0fae0904 rejected any login output. This revision accepts two allowlisted shapes: one `bao login -format=json` object on stdout, and the OpenBao OIDC CLI prompt on stderr. Both are an explicit, structure-only relaxation of 'reject all login output'. They are accepted because the output stays captured and is never forwarded, logged, or audited (criterion 3), and because a successful attended login is required to return the issued token in contained output. The stderr URL group (`https://[^\\s]+`) can carry arbitrary non-whitespace data, including a token-shaped string. railiance-platform accepts this: the URL is the server-issued OIDC auth_url and is printed before any client token exists, and the stderr is contained. The relaxation does not allow output to reach the agent.",
|
||||
"tests": [
|
||||
{
|
||||
"command": "uv run pytest -q tests/test_proxy.py tests/test_plan.py",
|
||||
"passed": true,
|
||||
"passed_count": 86,
|
||||
"failed_count": 0
|
||||
},
|
||||
{
|
||||
"command": "uv run pytest -q",
|
||||
"passed": true,
|
||||
"passed_count": 629,
|
||||
"failed_count": 0,
|
||||
"deselected_count": 4
|
||||
},
|
||||
{
|
||||
"command": "uv run ruff check src/warden/proxy.py tests/test_proxy.py",
|
||||
"passed": true
|
||||
}
|
||||
],
|
||||
"decision": "accept",
|
||||
"decision_rationale": "244f1ae fixes both failures from the 11b2a9d review. Non-allowlisted login stdout or stderr with a populated helper now independently fails closed, with one self-revocation, no retry, no child handoff, cleanup, and value-free errors (criterion 4). Populated-helper tests prove the output trigger directly, not through the persistence-failure path (criterion 5). 8a426fb restores a green full suite. The allowlist is tight on key names, types, and document framing. The remaining concerns below are non-blocking because the output stays contained.",
|
||||
"concerns_non_blocking": [
|
||||
"The stdout allowlist constrains key names and top-level types only. Nested contents of data, metadata, warnings, wrap_info, and mfa_requirement are unconstrained, and client_token may be empty. A non-null wrap_info next to auth would be accepted, and a wrapping token in it would not be covered by `token revoke -self`. Recommended hardening: require wrap_info to be null, and data to be null or empty.",
|
||||
"The stderr URL group accepts arbitrary non-whitespace data (accepted above). Leading whitespace before the prompt is also accepted.",
|
||||
"Because stderr is captured, the operator never sees the open-link variant's URL. In headless use the login can only finish via the browser-launch path. This is usability, not containment.",
|
||||
"The new parametrized tests assert value absence in the exception and stdio, but not in the two audit files. Audit absence for login_failed is covered only by test_cli_attended_result_and_both_audits.",
|
||||
"Observations from the first review still stand: the WARDEN-WP-0036-T02 live operation on 2026-09-01 ran on unaccepted code and needs railiance-infra follow-up, and the WARDEN_CONFIG path is forwarded to the child."
|
||||
],
|
||||
"live_oidc_performed": false,
|
||||
"openbao_calls_performed": false,
|
||||
"live_drill_authorized": false,
|
||||
"secret_values_observed": false,
|
||||
"sensitive_material_recorded": false,
|
||||
"reviewer": "claude/railiance-platform-ops-warden-review-2",
|
||||
"reviewed_at": "2026-10-14T10:39:09Z"
|
||||
},
|
||||
{
|
||||
"interface": "railiance.attended-login-containment-reacceptance-review",
|
||||
"version": 1,
|
||||
"workplan_id": "RPF-WP-0017",
|
||||
"task_id": "RPF-WP-0017-T01",
|
||||
"owner": "railiance-platform",
|
||||
"source_repo": "ops-warden",
|
||||
"baseline_revision": "0fae0904ce8d8694338dd53a8a79abec5fec788d",
|
||||
"supersedes_review_of": "8a426fba696c024a24430e8d3bcb239463d51f51",
|
||||
"reviewed_revision": "6e1430073c64ed79204cdf676f18bfa41812d5d2",
|
||||
"reviewed_paths": [
|
||||
"src/warden/proxy.py",
|
||||
"tests/test_proxy.py"
|
||||
],
|
||||
"intervening_commits": [
|
||||
{
|
||||
"revision": "6e1430073c64ed79204cdf676f18bfa41812d5d2",
|
||||
"date": "2026-10-14",
|
||||
"summary": "WARDEN-WP-0036: tightens the attended-login allowlist. wrap_info must be null or absent, data must be null, absent or {}, auth.client_token must be a non-empty str, and leading whitespace before the stderr OIDC prompt is rejected. Populated-helper tests now run through the CLI and assert that sentinel values are absent from both audit files. Addresses non-blocking concerns 1, 2 and 4 of the second review."
|
||||
}
|
||||
],
|
||||
"criteria": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Isolated writable token-helper location created before authentication, mode 0600 with private parents, deterministic cleanup",
|
||||
"result": "pass",
|
||||
"evidence": "Helper home preparation and cleanup are unchanged since 8a426fb. CLI-level tests now assert the private root is removed on both the fail-closed and the hand-off paths."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Read-only or unusable default home fails before OIDC begins",
|
||||
"result": "pass",
|
||||
"evidence": "Unchanged; test_attended_login_refuses_read_only_home_before_auth passes."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Adapter-owned stdout/stderr containment; no inherited agent-visible stdio",
|
||||
"result": "pass",
|
||||
"evidence": "_contained_run is unchanged. New _assert_login_audits reads access-audit.log and audit.jsonl and asserts the unexpected-login, OK and helper sentinels are absent, with login exit_code/outcome recorded (10/login_failed or 0/ok). CLI output, capsys and every subprocess argv are asserted sentinel-free."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Unexpected credential-bearing or otherwise non-allowlisted output fails closed and triggers immediate self-revocation",
|
||||
"result": "pass",
|
||||
"evidence": "The fail-closed gate is unchanged from 244f1ae (returncode, helper_valid and output_allowed, then a single self-revocation and exit 10, with no retry). The allowlist is now tighter. The type of wrap_info is (NoneType,) only, so a wrapping token that would survive `token revoke -self` is rejected. `parsed.get(\"data\")` truthy is rejected, and given the (dict, None) typing that means only null, absent or {} pass. client_token must be a non-empty str. _LOGIN_OIDC_STDERR no longer starts with `\\s*`, so fullmatch rejects any leading byte. Reviewer probes: a list-typed data, a tab-prefixed prompt and a space-prefixed prompt are rejected; a null wrap_info and trailing whitespace after the prompt are accepted, as intended."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Focused tests cover success handoff, unwritable home refusal, helper persistence failure, unexpected stdout, unexpected stderr, immediate revocation, cleanup, and value absence",
|
||||
"result": "pass",
|
||||
"evidence": "The non-allowlisted parametrized test grows to 20 cases through the CLI. New cases: wrap_info carrying a token, an empty wrap_info, non-empty data, empty and null client_token, and stderr with a leading space or newline. It asserts exit 10, the error reason text, sentinel absence (output, capsys, argv, both audit files), exactly [login, revoke -self] with no retry and no child, and cleanup. A direct-call test keeps the value-free exception and exact argv assertions. The allowlisted hand-off test (6 cases, adding data-empty) now runs through the CLI and asserts the child runs, the final call is revoke -self, the sentinel is absent, cleanup happens, and the audit records 0/ok."
|
||||
}
|
||||
],
|
||||
"allowlist_relaxation_note": "Unchanged from the second review. Accepting one `bao login -format=json` object on stdout and the OpenBao OIDC prompt on stderr is an explicit, structure-only relaxation of 'reject all login output'. It is accepted because both stay captured, are never forwarded, logged or audited, and cannot reach the agent. The stderr URL group (`https://[^\\s]+`) can still carry arbitrary non-whitespace data. That is acceptable because it is the server-issued OIDC auth_url, printed before any client token exists, and it is contained.",
|
||||
"tests": [
|
||||
{
|
||||
"command": "uv run pytest -q tests/test_proxy.py tests/test_plan.py",
|
||||
"passed": true,
|
||||
"passed_count": 95,
|
||||
"failed_count": 0
|
||||
},
|
||||
{
|
||||
"command": "uv run pytest -q",
|
||||
"passed": true,
|
||||
"passed_count": 638,
|
||||
"failed_count": 0,
|
||||
"deselected_count": 4
|
||||
},
|
||||
{
|
||||
"command": "uv run ruff check src/warden/proxy.py tests/test_proxy.py",
|
||||
"passed": true
|
||||
}
|
||||
],
|
||||
"decision": "accept",
|
||||
"decision_rationale": "6e14300 only tightens the allowlist that the second review accepted at 8a426fb, and adds audit-file value-absence coverage. It loosens nothing: the fail-closed gate, revocation, no-retry and cleanup paths are unchanged. Second-review concerns 1, 2 and 4 are resolved. All five criteria pass and the full suite is green.",
|
||||
"resolved_concerns": [
|
||||
"Concern 1 is resolved: wrap_info must be null or absent, data null, absent or {}, and client_token non-empty.",
|
||||
"Concern 2 is partially resolved: leading whitespace before the prompt is rejected, but the URL group stays permissive by design (see allowlist_relaxation_note).",
|
||||
"Concern 4 is resolved: audit-file sentinel absence is now asserted on both the fail-closed and the hand-off paths."
|
||||
],
|
||||
"concerns_non_blocking": [
|
||||
"A whitespace-only client_token (e.g. \" \") passes, because the check is `token != \"\"` rather than `token.strip()`. This is cosmetic: the populated 0600 helper check is the real persistence gate.",
|
||||
"The nested contents of auth.metadata, warnings and mfa_requirement are still unconstrained. They are contained, and the self-token is what gets revoked.",
|
||||
"Concern 3 from the second review is unchanged. Because stderr is captured, the operator never sees the open-link URL, so headless login cannot complete. This is usability, not containment.",
|
||||
"Observations from the earlier reviews still stand: the WARDEN-WP-0036-T02 live operation on 2026-09-01 ran on unaccepted code and needs railiance-infra follow-up, and the WARDEN_CONFIG path is forwarded to the child."
|
||||
],
|
||||
"live_oidc_performed": false,
|
||||
"openbao_calls_performed": false,
|
||||
"live_drill_authorized": false,
|
||||
"secret_values_observed": false,
|
||||
"sensitive_material_recorded": false,
|
||||
"reviewer": "claude/railiance-platform-ops-warden-review-2",
|
||||
"reviewed_at": "2026-10-14T10:43:47Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -100,6 +100,12 @@ def load_approval(path: Path, now: datetime, *, require_open_window: bool) -> di
|
|||
load = document.get("synthetic_load") or {}
|
||||
if not load.get("contract_id") or not load.get("driver_revision"):
|
||||
raise ProcedureError("approval receipt lacks the synthetic-load contract")
|
||||
sender = document.get("synthetic_sender")
|
||||
# Strict naming/scope/expiry rules live in audit_core_synthetic_sender_lane.py.
|
||||
if not isinstance(sender, dict) or set(sender) != {"identity", "source", "tenant", "expires_at"} or not all(
|
||||
isinstance(value, str) and value for value in sender.values()
|
||||
):
|
||||
raise ProcedureError("approval receipt lacks the projected synthetic sender identity")
|
||||
return document
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -104,6 +104,23 @@ def latest_completed_backup(backups: dict[str, Any], now: datetime) -> dict[str,
|
|||
}
|
||||
|
||||
|
||||
def running_pod_image(pod: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Image actually running in the serving pod, not only the Deployment spec.
|
||||
|
||||
``status.containerStatuses[].imageID`` carries the digest the kubelet
|
||||
resolved; a spec that matches while an older ReplicaSet pod still runs must
|
||||
fail closed.
|
||||
"""
|
||||
containers = pod.get("spec", {}).get("containers", [])
|
||||
statuses = pod.get("status", {}).get("containerStatuses", [])
|
||||
name = containers[0].get("name") if containers else None
|
||||
spec_image = containers[0].get("image") if containers else None
|
||||
status = next((item for item in statuses if item.get("name") == name), None)
|
||||
image_id = (status or {}).get("imageID") or ""
|
||||
match = re.search(r"@(sha256:[0-9a-f]{64})$", image_id)
|
||||
return {"image": spec_image, "digest": match.group(1) if match else None}
|
||||
|
||||
|
||||
def resource_condition(resource: dict[str, Any]) -> dict[str, Any]:
|
||||
conditions = resource.get("status", {}).get("conditions", [])
|
||||
current = conditions[0] if conditions else {}
|
||||
|
|
@ -247,6 +264,7 @@ def common_state(remote: Remote, now: datetime) -> dict[str, Any]:
|
|||
except json.JSONDecodeError as exc:
|
||||
raise PreflightError("OpenBao seal state returned invalid JSON") from exc
|
||||
image = deployment["spec"]["template"]["spec"]["containers"][0]["image"]
|
||||
running = running_pod_image(pod)
|
||||
state = {
|
||||
"captured_at": now.replace(microsecond=0).isoformat().replace("+00:00", "Z"),
|
||||
"node": {
|
||||
|
|
@ -280,6 +298,12 @@ def common_state(remote: Remote, now: datetime) -> dict[str, Any]:
|
|||
"audit_core": {
|
||||
"image": image,
|
||||
"image_matches_reviewed_digest": image == EXPECTED_IMAGE,
|
||||
"running_pod_image": running["image"],
|
||||
"running_pod_image_digest": running["digest"],
|
||||
"running_pod_image_matches_reviewed_digest": (
|
||||
running["image"] == EXPECTED_IMAGE
|
||||
and running["digest"] == EXPECTED_IMAGE.rsplit("@", 1)[1]
|
||||
),
|
||||
"ready_replicas": deployment.get("status", {}).get("readyReplicas", 0),
|
||||
"replicas": deployment.get("spec", {}).get("replicas", 0),
|
||||
"pod_uid": pod["metadata"]["uid"],
|
||||
|
|
@ -303,6 +327,7 @@ def automated_common_pass(state: dict[str, Any]) -> bool:
|
|||
all(item["ready"] for item in state["external_secret_stores"].values()),
|
||||
all(item["ready"] for item in state["external_secrets"].values()),
|
||||
state["audit_core"]["image_matches_reviewed_digest"],
|
||||
state["audit_core"]["running_pod_image_matches_reviewed_digest"],
|
||||
state["audit_core"]["ready_replicas"] == state["audit_core"]["replicas"] == 1,
|
||||
)
|
||||
)
|
||||
|
|
@ -366,14 +391,431 @@ def database_lease_preflight(remote: Remote, args: argparse.Namespace) -> dict[s
|
|||
}
|
||||
|
||||
|
||||
# --- node-reboot specific checks -------------------------------------------
|
||||
|
||||
# Named core platform workloads (namespace, kind, name) that must be fully
|
||||
# Ready. Every other Deployment/StatefulSet in these namespaces must be Ready
|
||||
# as well, so a renamed or newly added operator component cannot hide.
|
||||
CORE_PLATFORM_WORKLOADS = (
|
||||
("kube-system", "Deployment", "coredns"),
|
||||
("kube-system", "Deployment", "local-path-provisioner"),
|
||||
("kube-system", "Deployment", "traefik"),
|
||||
("kube-system", "Deployment", "metrics-server"),
|
||||
("cert-manager", "Deployment", "cert-manager"),
|
||||
("cnpg-system", "Deployment", "cnpg-controller-manager"),
|
||||
("external-secrets", "Deployment", "external-secrets"),
|
||||
("knative-serving", "Deployment", "activator"),
|
||||
("knative-serving", "Deployment", "controller"),
|
||||
("knative-serving", "Deployment", "net-kourier-controller"),
|
||||
("kourier-system", "Deployment", "3scale-kourier-gateway"),
|
||||
("argocd", "Deployment", "argocd-repo-server"),
|
||||
("argocd", "StatefulSet", "argocd-application-controller"),
|
||||
("openbao", "StatefulSet", "openbao"),
|
||||
)
|
||||
CORE_PLATFORM_NAMESPACES = frozenset(ns for ns, _, _ in CORE_PLATFORM_WORKLOADS)
|
||||
|
||||
# CNPG clusters observed on 2026-10-14. The live list is discovered; this set
|
||||
# only makes a vanished cluster fail closed instead of silently shrinking scope.
|
||||
EXPECTED_CNPG_CLUSTERS = frozenset({
|
||||
"databases/apps-pg", "databases/forgejo-db", "databases/net-kingdom-pg",
|
||||
"databases/platform-pg", "databases/platform-pg-2", "databases/state-hub-db",
|
||||
"target-revenue/target-revenue-pg", "user-engine/user-engine-pg",
|
||||
})
|
||||
FRESH_BACKUP_REQUIRED = frozenset({"databases/platform-pg"})
|
||||
FRESH_BACKUP_RECOMMENDED = frozenset({"databases/state-hub-db", "user-engine/user-engine-pg"})
|
||||
|
||||
# Declared host firewall (railiance-infra RAIL-HO-WP-0009 / Goss).
|
||||
UFW_ALLOWED = frozenset({"OpenSSH", "2224/tcp"})
|
||||
UFW_FORBIDDEN_PORTS = ("6443", "8472")
|
||||
K3S_API_TUNNEL = "k3s-api-railiance01"
|
||||
|
||||
OPS_WARDEN_RECEIPT = "reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json"
|
||||
CONSUMED_DECISION_PREFIXES = ("449a697a", "85724c0c")
|
||||
CONSUMED_SCENARIO_IDS = frozenset({"WARDEN-WP-0027-T02-DRILL-20260822-01"})
|
||||
SAFE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{5,127}$")
|
||||
FORBIDDEN_RECEIPT_MARKERS = (
|
||||
"BEGIN PRIVATE KEY", "BEGIN OPENSSH PRIVATE KEY", "AGE-SECRET-KEY-1",
|
||||
"OPENBAO_ROOT_TOKEN", "VAULT_TOKEN", "hvs.", "password=", "://",
|
||||
)
|
||||
|
||||
|
||||
def workload_ready(resource: dict[str, Any]) -> bool:
|
||||
wanted = resource.get("spec", {}).get("replicas", 1)
|
||||
return wanted >= 1 and resource.get("status", {}).get("readyReplicas", 0) == wanted
|
||||
|
||||
|
||||
def core_platform_workloads(workloads: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Readiness of named core workloads plus every workload in core namespaces."""
|
||||
observed: dict[tuple[str, str, str], bool] = {}
|
||||
for item in workloads.get("items", []):
|
||||
meta = item.get("metadata", {})
|
||||
if meta.get("namespace") in CORE_PLATFORM_NAMESPACES:
|
||||
key = (meta["namespace"], item.get("kind"), meta.get("name"))
|
||||
observed[key] = workload_ready(item)
|
||||
missing = sorted("/".join(key) for key in CORE_PLATFORM_WORKLOADS if key not in observed)
|
||||
unready = sorted("/".join(key) for key, ready in observed.items() if not ready)
|
||||
return {
|
||||
"required": ["/".join(key) for key in CORE_PLATFORM_WORKLOADS],
|
||||
"missing": missing,
|
||||
"unready": unready,
|
||||
"ready": not missing and not unready,
|
||||
}
|
||||
|
||||
|
||||
def failing_pods(pod_list: dict[str, Any]) -> list[str]:
|
||||
"""``namespace/name`` of every pod that is neither Succeeded nor Running+Ready."""
|
||||
failing = []
|
||||
for pod in pod_list.get("items", []):
|
||||
status = pod.get("status", {})
|
||||
phase = status.get("phase")
|
||||
if phase == "Succeeded":
|
||||
continue
|
||||
statuses = status.get("containerStatuses", [])
|
||||
if phase == "Running" and statuses and all(item.get("ready") for item in statuses):
|
||||
continue
|
||||
meta = pod.get("metadata", {})
|
||||
failing.append(f"{meta.get('namespace')}/{meta.get('name')}")
|
||||
return sorted(failing)
|
||||
|
||||
|
||||
def failing_pod_diff(baseline: list[str], current: list[str]) -> dict[str, Any]:
|
||||
new = sorted(set(current) - set(baseline))
|
||||
return {
|
||||
"baseline_count": len(baseline),
|
||||
"current_count": len(current),
|
||||
"new_failing": new,
|
||||
"resolved": sorted(set(baseline) - set(current)),
|
||||
"no_new_failures": not new,
|
||||
}
|
||||
|
||||
|
||||
def system_identifier(controldata: str) -> str | None:
|
||||
match = re.search(r"^Database system identifier:\s+(\d+)\s*$", controldata, re.MULTILINE)
|
||||
return match.group(1) if match else None
|
||||
|
||||
|
||||
def cnpg_identity(cluster: dict[str, Any], pvc_uids: dict[str, str]) -> dict[str, Any]:
|
||||
"""Value-safe identity/health of one CNPG cluster (no credentials, no DSNs)."""
|
||||
meta = cluster.get("metadata", {})
|
||||
status = cluster.get("status", {})
|
||||
primary = status.get("currentPrimary")
|
||||
backup_configured = bool(cluster.get("spec", {}).get("backup"))
|
||||
archiving = condition(cluster, "ContinuousArchiving")
|
||||
return {
|
||||
"uid": meta.get("uid"),
|
||||
"instances": cluster.get("spec", {}).get("instances", 0),
|
||||
"ready_instances": status.get("readyInstances", 0),
|
||||
"phase": status.get("phase"),
|
||||
"current_primary": primary,
|
||||
"primary_pvc_uid": pvc_uids.get(primary) if primary else None,
|
||||
"healthy_pvc": sorted(status.get("healthyPVC", [])),
|
||||
"backup_configured": backup_configured,
|
||||
"continuous_archiving": archiving == "True" if backup_configured else None,
|
||||
"system_id": None,
|
||||
}
|
||||
|
||||
|
||||
def cnpg_healthy(identity: dict[str, Any]) -> bool:
|
||||
return (
|
||||
identity["ready_instances"] == identity["instances"] == 1
|
||||
and bool(identity["uid"])
|
||||
and bool(identity["primary_pvc_uid"])
|
||||
and bool(identity["system_id"])
|
||||
and identity["current_primary"] in identity["healthy_pvc"]
|
||||
and identity["continuous_archiving"] is not False
|
||||
)
|
||||
|
||||
|
||||
def latest_backups_by_cluster(backups: dict[str, Any], now: datetime) -> dict[str, Any]:
|
||||
grouped: dict[str, list[dict[str, Any]]] = {}
|
||||
for item in backups.get("items", []):
|
||||
meta = item.get("metadata", {})
|
||||
cluster = item.get("spec", {}).get("cluster", {}).get("name")
|
||||
if cluster:
|
||||
grouped.setdefault(f"{meta.get('namespace')}/{cluster}", []).append(item)
|
||||
result = {}
|
||||
for key, items in grouped.items():
|
||||
try:
|
||||
result[key] = latest_completed_backup({"items": items}, now)
|
||||
except PreflightError:
|
||||
result[key] = None
|
||||
return result
|
||||
|
||||
|
||||
def eso_inventory(stores: dict[str, Any], external_secrets: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Every ClusterSecretStore Valid and every ExternalSecret SecretSynced."""
|
||||
def bad(items: list[dict[str, Any]], reason: str, namespaced: bool) -> list[str]:
|
||||
out = []
|
||||
for item in items:
|
||||
ready = next(
|
||||
(c for c in item.get("status", {}).get("conditions", []) if c.get("type") == "Ready"),
|
||||
{},
|
||||
)
|
||||
if ready.get("status") != "True" or ready.get("reason") != reason:
|
||||
meta = item.get("metadata", {})
|
||||
out.append(f"{meta.get('namespace')}/{meta.get('name')}" if namespaced else meta.get("name"))
|
||||
return sorted(out)
|
||||
|
||||
store_items = stores.get("items", [])
|
||||
es_items = external_secrets.get("items", [])
|
||||
invalid_stores = bad(store_items, "Valid", False)
|
||||
unsynced = bad(es_items, "SecretSynced", True)
|
||||
return {
|
||||
"cluster_store_count": len(store_items),
|
||||
"invalid_cluster_stores": invalid_stores,
|
||||
"external_secret_count": len(es_items),
|
||||
"unsynced_external_secrets": unsynced,
|
||||
"ready": bool(store_items) and bool(es_items) and not invalid_stores and not unsynced,
|
||||
}
|
||||
|
||||
|
||||
def ufw_state(status_text: str) -> dict[str, Any]:
|
||||
"""Compare ``ufw status`` with the declared OpenSSH + Nydus 2224 allow set."""
|
||||
active = bool(re.search(r"^Status:\s+active\s*$", status_text, re.MULTILINE))
|
||||
targets: set[str] = set()
|
||||
in_rules = False
|
||||
for line in status_text.splitlines():
|
||||
if line.startswith("--"):
|
||||
in_rules = True
|
||||
continue
|
||||
if in_rules and line.strip():
|
||||
target = re.split(r"\s{2,}", line.strip())[0].replace(" (v6)", "")
|
||||
targets.add(target)
|
||||
forbidden = sorted(
|
||||
target for target in targets
|
||||
if any(re.match(rf"^{port}(/|:|$)", target) for port in UFW_FORBIDDEN_PORTS)
|
||||
)
|
||||
return {
|
||||
"active": active,
|
||||
"allowed_targets": sorted(targets),
|
||||
"forbidden_ports_open": forbidden,
|
||||
"matches_declaration": active and targets == UFW_ALLOWED and not forbidden,
|
||||
}
|
||||
|
||||
|
||||
def boots_since(list_boots_json: str, since: datetime) -> int:
|
||||
"""Count boots whose first entry is after ``since`` (journalctl --list-boots -o json)."""
|
||||
try:
|
||||
boots = json.loads(list_boots_json)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise PreflightError("boot list returned invalid JSON") from exc
|
||||
count = 0
|
||||
for boot in boots:
|
||||
first = boot.get("first_entry")
|
||||
if isinstance(first, int) and datetime.fromtimestamp(first / 1_000_000, UTC) > since:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def validate_ops_warden_receipt(path: Path) -> dict[str, Any]:
|
||||
"""RPF-WP-0017 re-acceptance of the ops-warden revision for the attended login."""
|
||||
try:
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
receipt = json.loads(raw)
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise PreflightError("ops-warden acceptance receipt is unavailable or invalid JSON") from exc
|
||||
if not isinstance(receipt, dict) or receipt.get("workplan_id", "RPF-WP-0017") != "RPF-WP-0017" or "RPF-WP-0017" not in raw:
|
||||
raise PreflightError("ops-warden acceptance receipt is not an RPF-WP-0017 receipt")
|
||||
if any(marker in raw for marker in FORBIDDEN_RECEIPT_MARKERS[:-1]):
|
||||
raise PreflightError("ops-warden acceptance receipt contains forbidden material")
|
||||
if receipt.get("secret_values_observed") is True or receipt.get("sensitive_material_recorded") is True:
|
||||
raise PreflightError("ops-warden acceptance receipt records sensitive material")
|
||||
disposition = next(
|
||||
(str(receipt[key]).lower() for key in ("current_decision", "disposition", "decision", "status", "result") if key in receipt),
|
||||
"",
|
||||
)
|
||||
if disposition not in {"accept", "accepted", "approve", "approved"}:
|
||||
raise PreflightError("ops-warden acceptance receipt does not record acceptance")
|
||||
revision = next(
|
||||
(
|
||||
receipt[key] for key in ("current_reviewed_revision", "reviewed_revision", "accepted_revision", "ops_warden_revision", "revision")
|
||||
if isinstance(receipt.get(key), str) and re.fullmatch(r"[0-9a-f]{40}", receipt[key])
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not revision:
|
||||
raise PreflightError("ops-warden acceptance receipt names no exact 40-hex revision")
|
||||
return {"path": str(path), "accepted_revision": revision, "verified": True}
|
||||
|
||||
|
||||
CONSOLE_RECEIPT_ID = re.compile(r"^RAIL01-CONSOLE-(\d{8})T\d{6}Z$")
|
||||
|
||||
|
||||
def validate_console_receipt(path: Path, *, window_id: str | None, now: datetime) -> dict[str, Any]:
|
||||
"""railiance-infra console attestation (docs/railiance01-console-access.md).
|
||||
|
||||
Accept only a same-day receipt for this window that proves the console
|
||||
shows the host and is independent of the guest network. Role-only fields;
|
||||
any credential-looking or placeholder content is denied.
|
||||
"""
|
||||
try:
|
||||
import yaml # type: ignore[import-untyped]
|
||||
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
receipt = yaml.safe_load(raw)
|
||||
except Exception as exc: # noqa: BLE001 - any parse/import failure is fail-closed
|
||||
raise PreflightError("provider-console receipt is unavailable or invalid YAML") from exc
|
||||
if not isinstance(receipt, dict):
|
||||
raise PreflightError("provider-console receipt has an unsupported shape")
|
||||
if any(marker in raw for marker in FORBIDDEN_RECEIPT_MARKERS) or "<" in raw:
|
||||
raise PreflightError("provider-console receipt contains forbidden material/placeholder")
|
||||
match = CONSOLE_RECEIPT_ID.fullmatch(str(receipt.get("receipt_id", "")))
|
||||
if not match or path.stem != receipt["receipt_id"]:
|
||||
raise PreflightError("provider-console receipt id is not RAIL01-CONSOLE-<YYYYMMDDTHHMMSSZ>")
|
||||
try:
|
||||
attested = rfc3339(str(receipt["attested_at"]))
|
||||
except (KeyError, ValueError) as exc:
|
||||
raise PreflightError("provider-console receipt has invalid attested_at") from exc
|
||||
today = now.astimezone(UTC).strftime("%Y%m%d")
|
||||
if match.group(1) != today or attested.strftime("%Y%m%d") != today or attested > now:
|
||||
raise PreflightError("provider-console receipt is not from the window day")
|
||||
if receipt.get("host") != "railiance01" or receipt.get("panel_login") != "ok":
|
||||
raise PreflightError("provider-console receipt does not prove panel access to railiance01")
|
||||
if receipt.get("console_shows_host") is not True or receipt.get("independent_of_guest_network") is not True:
|
||||
raise PreflightError("provider-console receipt does not prove an independent host console")
|
||||
if not window_id or str(receipt.get("reboot_window_id")) != window_id:
|
||||
raise PreflightError("provider-console receipt does not name the approved window id")
|
||||
return {
|
||||
"receipt_id": receipt["receipt_id"],
|
||||
"attested_at": attested.isoformat().replace("+00:00", "Z"),
|
||||
"attested_by_role": receipt.get("attested_by_role"),
|
||||
"console_shows_host": True,
|
||||
"independent_of_guest_network": True,
|
||||
"window_id_matches": True,
|
||||
"verified": True,
|
||||
}
|
||||
|
||||
|
||||
def fresh_decision_gates(scenario_id: str | None, decision_id: str | None) -> dict[str, bool]:
|
||||
"""Consumed drill scenario/decisions can never authorize this window."""
|
||||
return {
|
||||
"new_scenario_id_recorded": bool(scenario_id)
|
||||
and bool(SAFE_ID.fullmatch(scenario_id))
|
||||
and scenario_id not in CONSUMED_SCENARIO_IDS,
|
||||
"fresh_human_decision_recorded": bool(decision_id)
|
||||
and bool(SAFE_ID.fullmatch(decision_id))
|
||||
and not decision_id.lower().startswith(CONSUMED_DECISION_PREFIXES),
|
||||
}
|
||||
|
||||
|
||||
def reboot_comparison(pre: dict[str, Any], post: dict[str, Any], boots: int) -> dict[str, Any]:
|
||||
"""Post-reboot proof: exactly one reboot, same database identities, no new failures."""
|
||||
pre_host, post_host = pre.get("host", {}), post["host"]
|
||||
identity_changes = []
|
||||
for key, before in pre.get("cnpg_clusters", {}).items():
|
||||
after = post["cnpg_clusters"].get(key)
|
||||
for field in ("uid", "primary_pvc_uid", "system_id"):
|
||||
if after is None or after.get(field) != before.get(field):
|
||||
identity_changes.append(f"{key}:{field}")
|
||||
pod_diff = failing_pod_diff(pre.get("failing_pods", []), post["failing_pods"])
|
||||
return {
|
||||
"boot_id_changed": bool(pre_host.get("boot_id"))
|
||||
and pre_host.get("boot_id") != post_host["boot_id"],
|
||||
"boots_since_pre_capture": boots,
|
||||
"exactly_one_reboot": boots == 1 and pre_host.get("boot_id") != post_host["boot_id"],
|
||||
"database_identity_changes": sorted(identity_changes),
|
||||
"failing_pod_diff": pod_diff,
|
||||
"passed": boots == 1
|
||||
and pre_host.get("boot_id") != post_host["boot_id"]
|
||||
and not identity_changes
|
||||
and pod_diff["no_new_failures"],
|
||||
}
|
||||
|
||||
|
||||
def tunnel_answers(name: str) -> bool:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["bridge", "check", name], text=True, capture_output=True, check=False, timeout=30
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
rows = [line.split() for line in completed.stdout.splitlines() if line.startswith(name)]
|
||||
return completed.returncode == 0 and bool(rows) and rows[0][-1] == "yes"
|
||||
|
||||
|
||||
def reboot_state(remote: Remote, now: datetime) -> dict[str, Any]:
|
||||
workloads = remote.kubectl_json(
|
||||
["get", "deployments,statefulsets", "-A"], label="read core workload state"
|
||||
)
|
||||
pods = remote.kubectl_json(["get", "pods", "-A"], label="read all-namespace pod phases")
|
||||
clusters = remote.kubectl_json(
|
||||
["get", "clusters.postgresql.cnpg.io", "-A"], label="discover CNPG clusters"
|
||||
)
|
||||
pvcs = remote.kubectl_json(["get", "pvc", "-A"], label="read PVC identities")
|
||||
pvc_uids = {
|
||||
f"{item['metadata']['namespace']}/{item['metadata']['name']}": item["metadata"]["uid"]
|
||||
for item in pvcs.get("items", [])
|
||||
}
|
||||
terminating_pvcs = sorted(
|
||||
f"{item['metadata']['namespace']}/{item['metadata']['name']}"
|
||||
for item in pvcs.get("items", [])
|
||||
if item["metadata"].get("deletionTimestamp")
|
||||
)
|
||||
cnpg: dict[str, Any] = {}
|
||||
for cluster in clusters.get("items", []):
|
||||
ns, name = cluster["metadata"]["namespace"], cluster["metadata"]["name"]
|
||||
ns_pvcs = {
|
||||
key.split("/", 1)[1]: uid for key, uid in pvc_uids.items() if key.startswith(f"{ns}/")
|
||||
}
|
||||
identity = cnpg_identity(cluster, ns_pvcs)
|
||||
if identity["current_primary"]:
|
||||
controldata = remote.kubectl(
|
||||
[
|
||||
"-n", ns, "exec", identity["current_primary"], "-c", "postgres", "--",
|
||||
"pg_controldata", "/var/lib/postgresql/data/pgdata",
|
||||
],
|
||||
label=f"read {ns}/{name} system identifier",
|
||||
)
|
||||
identity["system_id"] = system_identifier(controldata)
|
||||
cnpg[f"{ns}/{name}"] = identity
|
||||
backups = remote.kubectl_json(["get", "backups.postgresql.cnpg.io", "-A"], label="read CNPG backup metadata")
|
||||
stores = remote.kubectl_json(["get", "clustersecretstores"], label="read ESO store conditions")
|
||||
projections = remote.kubectl_json(["get", "externalsecrets", "-A"], label="read ESO projection conditions")
|
||||
ufw = remote.run(["sudo", "-n", "ufw", "status"], label="read host firewall state")
|
||||
ntp = remote.run(
|
||||
["timedatectl", "show", "-p", "NTPSynchronized", "--value"], label="read time sync state"
|
||||
)
|
||||
return {
|
||||
"host": {
|
||||
"boot_id": remote.run(["cat", "/proc/sys/kernel/random/boot_id"], label="read boot id"),
|
||||
"ntp_synchronized": ntp.strip() == "yes",
|
||||
"firewall": ufw_state(ufw),
|
||||
},
|
||||
"k3s_api_tunnel_answers": tunnel_answers(K3S_API_TUNNEL),
|
||||
"core_platform_workloads": core_platform_workloads(workloads),
|
||||
"failing_pods": failing_pods(pods),
|
||||
"terminating_pvcs": terminating_pvcs,
|
||||
"cnpg_clusters": cnpg,
|
||||
"missing_expected_cnpg_clusters": sorted(EXPECTED_CNPG_CLUSTERS - set(cnpg)),
|
||||
"cnpg_backups": latest_backups_by_cluster(backups, now),
|
||||
"eso": eso_inventory(stores, projections),
|
||||
}
|
||||
|
||||
|
||||
def reboot_automated_pass(extra: dict[str, Any], max_backup_age_hours: float) -> dict[str, bool]:
|
||||
backups = extra["cnpg_backups"]
|
||||
return {
|
||||
"ntp_synchronized": extra["host"]["ntp_synchronized"],
|
||||
"firewall_matches_declaration": extra["host"]["firewall"]["matches_declaration"],
|
||||
"k3s_api_tunnel_answers": extra["k3s_api_tunnel_answers"],
|
||||
"core_platform_workloads_ready": extra["core_platform_workloads"]["ready"],
|
||||
"all_cnpg_clusters_healthy": not extra["missing_expected_cnpg_clusters"]
|
||||
and all(cnpg_healthy(item) for item in extra["cnpg_clusters"].values()),
|
||||
"required_backups_fresh": all(
|
||||
backups.get(key) is not None and backups[key]["age_hours"] <= max_backup_age_hours
|
||||
for key in FRESH_BACKUP_REQUIRED
|
||||
),
|
||||
"all_eso_stores_valid_and_synced": extra["eso"]["ready"],
|
||||
}
|
||||
|
||||
|
||||
def reboot_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]:
|
||||
now = datetime.now(UTC)
|
||||
state = common_state(remote, now)
|
||||
backups = remote.kubectl_json(
|
||||
["-n", "databases", "get", "backup", "-l", "cnpg.io/cluster=platform-pg"],
|
||||
label="read platform-pg backup metadata",
|
||||
)
|
||||
latest = latest_completed_backup(backups, now)
|
||||
extra = reboot_state(remote, now)
|
||||
latest = extra["cnpg_backups"].get("databases/platform-pg")
|
||||
if latest is None:
|
||||
raise PreflightError("no completed platform-pg backup is visible")
|
||||
snapshot_evidence = Path(args.openbao_snapshot_evidence).resolve() if args.openbao_snapshot_evidence else None
|
||||
snapshot_receipt = None
|
||||
if snapshot_evidence:
|
||||
|
|
@ -383,25 +825,75 @@ def reboot_preflight(remote: Remote, args: argparse.Namespace) -> dict[str, Any]
|
|||
now=now,
|
||||
max_age_hours=args.max_snapshot_age_hours,
|
||||
)
|
||||
# Gate failures stay value-safe gate results; they never abort evidence capture.
|
||||
receipt_errors: dict[str, str] = {}
|
||||
warden_receipt = None
|
||||
try:
|
||||
warden_receipt = validate_ops_warden_receipt(Path(args.ops_warden_acceptance_receipt))
|
||||
except PreflightError as exc:
|
||||
receipt_errors["ops_warden_acceptance_receipt"] = str(exc)
|
||||
console_receipt = None
|
||||
if args.provider_console_receipt:
|
||||
try:
|
||||
console_receipt = validate_console_receipt(
|
||||
Path(args.provider_console_receipt), window_id=args.approved_window_id, now=now
|
||||
)
|
||||
except PreflightError as exc:
|
||||
receipt_errors["provider_console_receipt"] = str(exc)
|
||||
else:
|
||||
receipt_errors["provider_console_receipt"] = "not supplied (RAIL-HO-WP-0011-T09 pending)"
|
||||
checks = reboot_automated_pass(extra, args.max_backup_age_hours)
|
||||
comparison = None
|
||||
if args.pre_reboot_evidence:
|
||||
try:
|
||||
pre = json.loads(Path(args.pre_reboot_evidence).read_text(encoding="utf-8"))["reboot_evidence"]
|
||||
except (OSError, json.JSONDecodeError, KeyError, TypeError) as exc:
|
||||
raise PreflightError("pre-reboot evidence is unavailable or has no reboot_evidence") from exc
|
||||
boot_list = remote.run(
|
||||
["sudo", "-n", "journalctl", "--list-boots", "-o", "json", "--no-pager"],
|
||||
label="read boot list",
|
||||
)
|
||||
comparison = reboot_comparison(pre, extra, boots_since(boot_list, rfc3339(pre["captured_at"])))
|
||||
checks["post_reboot_comparison_passed"] = comparison["passed"]
|
||||
else:
|
||||
# Pre-reboot: stuck Terminating PVCs must be resolved or explicitly listed.
|
||||
listed = set(args.accept_terminating_pvc or [])
|
||||
checks["terminating_pvcs_resolved_or_listed"] = set(extra["terminating_pvcs"]) <= listed
|
||||
decision = fresh_decision_gates(args.scenario_id, args.human_decision_id)
|
||||
gates = {
|
||||
"approved_window_id_recorded": bool(args.approved_window_id),
|
||||
**decision,
|
||||
"ops_warden_revision_accepted_for_attended_login": bool(warden_receipt),
|
||||
"openbao_snapshot_evidence_present": bool(snapshot_receipt),
|
||||
"unseal_quorum_attested": args.unseal_quorum_attested,
|
||||
"provider_console_access_attested": args.provider_console_attested,
|
||||
"provider_console_receipt_verified": bool(console_receipt),
|
||||
"host_cluster_platform_database_audit_owners_acknowledged": args.all_owners_ack,
|
||||
"railiance_infra_independent_abort_owner_acknowledged": args.infra_abort_owner_ack,
|
||||
"attended_abort_operator_named": bool(args.abort_operator),
|
||||
}
|
||||
automated = automated_common_pass(state) and latest["age_hours"] <= args.max_backup_age_hours
|
||||
automated = (
|
||||
automated_common_pass(state)
|
||||
and latest["age_hours"] <= args.max_backup_age_hours
|
||||
and all(checks.values())
|
||||
)
|
||||
return {
|
||||
"procedure": "railiance01-coordinated-reboot",
|
||||
"phase": "post-reboot" if comparison else "pre-reboot",
|
||||
"preflight_only": True,
|
||||
"automated_checks_passed": automated,
|
||||
"ready_for_live_execution": automated and all(gates.values()),
|
||||
"automated_checks": checks,
|
||||
"ready_for_live_execution": automated and all(gates.values()) and not comparison,
|
||||
"operator_gates": gates,
|
||||
"latest_platform_pg_backup": latest,
|
||||
"max_backup_age_hours": args.max_backup_age_hours,
|
||||
"fresh_backup_recommended_for": sorted(FRESH_BACKUP_RECOMMENDED),
|
||||
"openbao_snapshot_receipt": snapshot_receipt,
|
||||
"ops_warden_acceptance_receipt": warden_receipt,
|
||||
"provider_console_receipt": console_receipt,
|
||||
"receipt_gate_failures": receipt_errors,
|
||||
"max_snapshot_age_hours": args.max_snapshot_age_hours,
|
||||
"reboot_evidence": {"captured_at": state["captured_at"], **extra},
|
||||
"reboot_comparison": comparison,
|
||||
"state": state,
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
|
|
@ -418,7 +910,22 @@ def main() -> int:
|
|||
parser.add_argument("--synthetic-load-id")
|
||||
parser.add_argument("--openbao-snapshot-evidence")
|
||||
parser.add_argument("--unseal-quorum-attested", action="store_true")
|
||||
parser.add_argument("--provider-console-attested", action="store_true")
|
||||
parser.add_argument(
|
||||
"--provider-console-receipt",
|
||||
help="railiance-infra RAIL01-CONSOLE-<ts>.yaml receipt for this window",
|
||||
)
|
||||
parser.add_argument("--infra-abort-owner-ack", action="store_true")
|
||||
parser.add_argument("--scenario-id", help="new scenario id; consumed drill ids are denied")
|
||||
parser.add_argument("--human-decision-id", help="fresh human GO decision id")
|
||||
parser.add_argument("--ops-warden-acceptance-receipt", default=OPS_WARDEN_RECEIPT)
|
||||
parser.add_argument(
|
||||
"--accept-terminating-pvc", action="append",
|
||||
help="namespace/name of a pre-existing Terminating PVC accepted for the window",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pre-reboot-evidence",
|
||||
help="pre-reboot preflight JSON; switches node-reboot to post-reboot comparison",
|
||||
)
|
||||
parser.add_argument("--all-owners-ack", action="store_true")
|
||||
parser.add_argument("--max-backup-age-hours", type=float, default=26.0)
|
||||
parser.add_argument("--max-snapshot-age-hours", type=float, default=24.0)
|
||||
|
|
|
|||
439
scripts/audit_core_synthetic_sender_lane.py
Normal file
439
scripts/audit_core_synthetic_sender_lane.py
Normal file
|
|
@ -0,0 +1,439 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Attended, value-safe synthetic sender lane for audit-core recovery exercises.
|
||||
|
||||
One temporary audit-core sender identity is added to the registry at
|
||||
``platform/workloads/audit-core/senders`` for exactly one approved window and
|
||||
removed again afterwards. The identity is parameterized by the approved window
|
||||
receipt (``synthetic_sender`` block), not by constants in this file, so the
|
||||
dated Whitehat ``-01``/``-02`` procedures are never copied.
|
||||
|
||||
Ordering contract (RAILIANCE-WP-0024-T02):
|
||||
|
||||
project -> audit-core restart -> T02 exercise -> evidence -> cleanup -> restart
|
||||
|
||||
* ``preflight`` is read-only. It never reads the sender registry values unless
|
||||
``--inspect-registry`` is given (attended; values stay in memory only).
|
||||
* ``project`` is time-gated to the opening minutes of the approved window and
|
||||
writes the bearer only to a new mode-0600 file outside any Git work tree.
|
||||
* ``cleanup`` is idempotent and refuses to run until the T02 exercise evidence
|
||||
is recorded, unless an explicit abort reason is given.
|
||||
|
||||
No command prints a bearer, the registry, OpenBao bodies, or Kubernetes Secret
|
||||
data.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import stat
|
||||
import sys
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LANE = "audit-core-synthetic-sender"
|
||||
REGISTRY_PATH = "platform/workloads/audit-core/senders"
|
||||
DRIVER_INTERFACE = ROOT / "interfaces" / "RAILIANCE-WP-0024-T02-synthetic-load-driver.json"
|
||||
PROJECT_CUTOFF = timedelta(minutes=3)
|
||||
MIN_REMAINING = timedelta(minutes=10)
|
||||
NAME_RE = re.compile(r"^[a-z][a-z0-9-]{2,62}$")
|
||||
TENANT_RE = re.compile(r"^tenant:trial:[a-z0-9][a-z0-9-]{1,62}$")
|
||||
RFC3339_Z_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
|
||||
|
||||
|
||||
class LaneError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _load(name: str, filename: str) -> Any:
|
||||
spec = importlib.util.spec_from_file_location(name, ROOT / "scripts" / filename)
|
||||
if not spec or not spec.loader:
|
||||
raise LaneError(f"cannot load {filename}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
# Reuse, do not copy: the live-proven registry/reload helpers from the -02 lane
|
||||
# (Operator, safe_run, secure_unlink, force_sender_sync_and_restart) and the
|
||||
# reviewed T02 receipt/attest-chain/preflight gates from the lease harness.
|
||||
CREDS = _load("audit_core_whitehat_e2_credentials_02", "audit-core-whitehat-e2-credentials-02.py")
|
||||
LEASE = _load("audit_core_database_lease_recovery", "audit-core-database-lease-recovery.py")
|
||||
PREFLIGHT = LEASE.PREFLIGHT
|
||||
ERRORS = (OSError, IndexError, KeyError, LaneError, CREDS.ProcedureError, LEASE.ProcedureError, PREFLIGHT.PreflightError)
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def stamp(value: datetime) -> str:
|
||||
return value.astimezone(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def confirm_token(approval: dict[str, Any]) -> str:
|
||||
return f"{LANE}:{approval['approval_id']}:attended"
|
||||
|
||||
|
||||
def driver_contract_id() -> str:
|
||||
try:
|
||||
return json.loads(DRIVER_INTERFACE.read_text(encoding="utf-8"))["contract_id"]
|
||||
except (OSError, json.JSONDecodeError, KeyError) as exc:
|
||||
raise LaneError("synthetic-load driver interface is unavailable") from exc
|
||||
|
||||
|
||||
def sender_spec(approval: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Validate the receipt's single synthetic identity; return it normalized."""
|
||||
block = approval.get("synthetic_sender")
|
||||
if not isinstance(block, dict):
|
||||
raise LaneError("approval receipt has no synthetic_sender block")
|
||||
if set(block) != {"identity", "source", "tenant", "expires_at"}:
|
||||
raise LaneError("synthetic_sender must name exactly identity, source, tenant and expires_at")
|
||||
name, source, tenant, expires_text = (
|
||||
block["identity"], block["source"], block["tenant"], block["expires_at"]
|
||||
)
|
||||
if not isinstance(name, str) or not NAME_RE.match(name) or "synthetic" not in name:
|
||||
raise LaneError("synthetic_sender.identity must be a lowercase name containing 'synthetic'")
|
||||
if not isinstance(source, str) or not NAME_RE.match(source) or "synthetic" not in source:
|
||||
raise LaneError("synthetic_sender.source must be a named synthetic source, never '*'")
|
||||
if not isinstance(tenant, str) or not TENANT_RE.match(tenant):
|
||||
raise LaneError("synthetic_sender.tenant must be exactly one tenant:trial:<name>")
|
||||
if not isinstance(expires_text, str) or not RFC3339_Z_RE.match(expires_text):
|
||||
raise LaneError("synthetic_sender.expires_at must be RFC3339 UTC (YYYY-MM-DDTHH:MM:SSZ)")
|
||||
expires = LEASE.parse_time(expires_text)
|
||||
window_end = LEASE.parse_time(approval["window"]["end"])
|
||||
if expires > window_end:
|
||||
raise LaneError("synthetic_sender.expires_at must not be later than the window end")
|
||||
load = approval.get("synthetic_load") or {}
|
||||
if load.get("contract_id") != driver_contract_id():
|
||||
raise LaneError("synthetic_load.contract_id does not match the registered driver interface")
|
||||
return {"name": name, "source": source, "tenant": tenant, "expires_at": expires_text, "expires": expires}
|
||||
|
||||
|
||||
def load_receipt(path: Path, now: datetime, *, require_open_window: bool) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
approval = LEASE.load_approval(path, now, require_open_window=require_open_window)
|
||||
return approval, sender_spec(approval)
|
||||
|
||||
|
||||
def assert_projection_time(approval: dict[str, Any], spec: dict[str, Any], now: datetime) -> None:
|
||||
start = LEASE.parse_time(approval["window"]["start"])
|
||||
if not start <= now <= start + PROJECT_CUTOFF:
|
||||
raise LaneError(
|
||||
f"projection permitted only {stamp(start)} through {stamp(start + PROJECT_CUTOFF)}; now {stamp(now)}"
|
||||
)
|
||||
if spec["expires"] - now < MIN_REMAINING:
|
||||
raise LaneError("identity would expire before a full T02 exercise could complete")
|
||||
|
||||
|
||||
def registry_entry(spec: dict[str, Any], token: str) -> dict[str, Any]:
|
||||
return {
|
||||
"name": spec["name"],
|
||||
"tokens": [token],
|
||||
"sources": [spec["source"]],
|
||||
"tenants": [spec["tenant"]],
|
||||
"may_write": True,
|
||||
"may_read": True,
|
||||
"expires_at": spec["expires_at"],
|
||||
}
|
||||
|
||||
|
||||
def registry_conflicts(registry: list[Any], spec: dict[str, Any]) -> list[str]:
|
||||
"""Name, source, or tenant overlap with any existing identity is a conflict."""
|
||||
conflicts: list[str] = []
|
||||
for entry in registry:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
label = str(entry.get("name"))
|
||||
if entry.get("name") == spec["name"]:
|
||||
conflicts.append(f"identity {label} already exists")
|
||||
if spec["source"] in (entry.get("sources") or []):
|
||||
conflicts.append(f"source is already granted to {label}")
|
||||
if spec["tenant"] in (entry.get("tenants") or []):
|
||||
conflicts.append(f"tenant is already granted to {label}")
|
||||
return conflicts
|
||||
|
||||
|
||||
def add_identity(registry: list[Any], spec: dict[str, Any], token: str) -> list[Any]:
|
||||
conflicts = registry_conflicts(registry, spec)
|
||||
if conflicts:
|
||||
raise LaneError("registry conflict: " + "; ".join(conflicts))
|
||||
return [*registry, registry_entry(spec, token)]
|
||||
|
||||
|
||||
def remove_identity(registry: list[Any], spec: dict[str, Any]) -> tuple[list[Any], int]:
|
||||
"""Remove only the exact named identity, and only if it is ours."""
|
||||
updated: list[Any] = []
|
||||
removed = 0
|
||||
for entry in registry:
|
||||
if isinstance(entry, dict) and entry.get("name") == spec["name"]:
|
||||
if entry.get("sources") != [spec["source"]] or entry.get("tenants") != [spec["tenant"]]:
|
||||
raise LaneError("named identity exists with a different scope; refusing to remove it")
|
||||
removed += 1
|
||||
continue
|
||||
updated.append(entry)
|
||||
return updated, removed
|
||||
|
||||
|
||||
def _inside_git_tree(path: Path) -> bool:
|
||||
return any((parent / ".git").exists() for parent in path.parents)
|
||||
|
||||
|
||||
def check_bearer_path(path: Path, *, must_exist: bool) -> None:
|
||||
if not path.is_absolute():
|
||||
raise LaneError("bearer file must be an absolute path")
|
||||
parent = path.parent.resolve()
|
||||
target = parent / path.name
|
||||
if target == ROOT or ROOT in target.parents or _inside_git_tree(target):
|
||||
raise LaneError("bearer file must be outside every Git work tree")
|
||||
info = parent.stat()
|
||||
if info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) & 0o077:
|
||||
raise LaneError("bearer file directory must be owned by the operator with mode 0700")
|
||||
if must_exist:
|
||||
details = target.lstat()
|
||||
if not stat.S_ISREG(details.st_mode) or stat.S_IMODE(details.st_mode) != 0o600:
|
||||
raise LaneError("bearer file must be a regular mode-0600 file")
|
||||
elif target.exists() or target.is_symlink():
|
||||
raise LaneError("bearer file already exists; run cleanup first")
|
||||
|
||||
|
||||
def write_bearer(path: Path, token: str) -> None:
|
||||
check_bearer_path(path, must_exist=False)
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
handle.write(token)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(path, 0o600)
|
||||
|
||||
|
||||
def assert_evidence(path: Path | None, abort_reason: str | None, approval: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Cleanup restarts audit-core, so it must not precede the recorded evidence."""
|
||||
if abort_reason:
|
||||
return {"mode": "abort", "reason": abort_reason.strip()[:200]}
|
||||
if path is None:
|
||||
raise LaneError("cleanup requires --exercise-evidence (recorded T02 result) or --abort-reason")
|
||||
try:
|
||||
evidence = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise LaneError("exercise evidence is unavailable or invalid JSON") from exc
|
||||
if (
|
||||
not isinstance(evidence, dict)
|
||||
or evidence.get("procedure") != LEASE.PROCEDURE
|
||||
or evidence.get("approval_id") != approval["approval_id"]
|
||||
or evidence.get("secret_values_observed") is not False
|
||||
or not evidence.get("completed_at")
|
||||
):
|
||||
raise LaneError("exercise evidence does not belong to this approved T02 window")
|
||||
return {"mode": "evidence-recorded", "completed_at": evidence["completed_at"]}
|
||||
|
||||
|
||||
def cluster_state(remote: Any) -> dict[str, Any]:
|
||||
state = PREFLIGHT.common_state(remote, utc_now())
|
||||
return {
|
||||
"automated_checks_passed": PREFLIGHT.automated_common_pass(state),
|
||||
"openbao_unsealed": state["openbao"]["initialized"] and not state["openbao"]["sealed"],
|
||||
"senders_external_secret_ready": state["external_secrets"]["audit-core-senders"]["ready"],
|
||||
"image_matches_reviewed_digest": state["audit_core"]["image_matches_reviewed_digest"],
|
||||
"ready": f"{state['audit_core']['ready_replicas']}/{state['audit_core']['replicas']}",
|
||||
"pod_uid": state["audit_core"]["pod_uid"],
|
||||
}
|
||||
|
||||
|
||||
def settled_state(remote: Any, *, previous_uid: str | None, timeout: float = 90) -> dict[str, Any]:
|
||||
"""Wait until exactly one Ready serving pod remains (old pod terminated)."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
try:
|
||||
state = cluster_state(remote)
|
||||
if state["ready"] == "1/1" and state["pod_uid"] != previous_uid:
|
||||
return state
|
||||
except PREFLIGHT.PreflightError:
|
||||
pass
|
||||
if time.monotonic() >= deadline:
|
||||
raise LaneError("audit-core did not settle on one Ready restarted pod")
|
||||
time.sleep(3)
|
||||
|
||||
|
||||
def registry_metadata(operator: Any) -> dict[str, Any]:
|
||||
raw = operator.bao(["kv", "metadata", "get", "-format=json", REGISTRY_PATH], label="read sender registry metadata").stdout
|
||||
try:
|
||||
data = json.loads(raw)["data"]
|
||||
return {"current_version": data["current_version"], "updated_time": data["updated_time"]}
|
||||
except (json.JSONDecodeError, KeyError, TypeError) as exc:
|
||||
raise LaneError("sender registry metadata is incomplete") from exc
|
||||
|
||||
|
||||
def time_state(approval: dict[str, Any], now: datetime) -> str:
|
||||
start = LEASE.parse_time(approval["window"]["start"])
|
||||
end = LEASE.parse_time(approval["window"]["end"])
|
||||
if now < start:
|
||||
return "before-window"
|
||||
if now <= start + PROJECT_CUTOFF:
|
||||
return "projection-window-open"
|
||||
return "projection-cutoff-passed" if now <= end else "window-closed"
|
||||
|
||||
|
||||
def identity_view(spec: dict[str, Any]) -> dict[str, Any]:
|
||||
return {key: spec[key] for key in ("name", "source", "tenant", "expires_at")}
|
||||
|
||||
|
||||
def preflight(args: argparse.Namespace) -> dict[str, Any]:
|
||||
now = utc_now()
|
||||
approval, spec = load_receipt(args.approval, now, require_open_window=False)
|
||||
result: dict[str, Any] = {
|
||||
"lane": LANE,
|
||||
"approval_id": approval["approval_id"],
|
||||
"time_state": time_state(approval, now),
|
||||
"identity": identity_view(spec),
|
||||
"confirm": confirm_token(approval),
|
||||
"registry_inspected": False,
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
if args.bearer_file is not None:
|
||||
check_bearer_path(args.bearer_file, must_exist=False)
|
||||
result["bearer_file_path_ok"] = True
|
||||
if args.offline:
|
||||
result["live_checks"] = "skipped"
|
||||
return result
|
||||
remote = PREFLIGHT.Remote(args.remote)
|
||||
result["audit_core"] = cluster_state(remote)
|
||||
operator = CREDS.Operator(args.remote, args.token_file)
|
||||
result["sender_registry_metadata"] = registry_metadata(operator)
|
||||
if args.inspect_registry:
|
||||
# Attended only: values are held in memory and never emitted.
|
||||
registry = operator.registry()
|
||||
result["registry_inspected"] = True
|
||||
result["identity_present"] = any(
|
||||
isinstance(entry, dict) and entry.get("name") == spec["name"] for entry in registry
|
||||
)
|
||||
result["registry_conflicts"] = registry_conflicts(registry, spec)
|
||||
registry.clear()
|
||||
result["automated_checks_passed"] = result["audit_core"]["automated_checks_passed"]
|
||||
return result
|
||||
|
||||
|
||||
def _cleanup_registry(operator: Any, spec: dict[str, Any]) -> int:
|
||||
updated, removed = remove_identity(operator.registry(), spec)
|
||||
if removed:
|
||||
operator.write_registry(updated)
|
||||
CREDS.force_sender_sync_and_restart(operator)
|
||||
updated.clear()
|
||||
return removed
|
||||
|
||||
|
||||
def project(args: argparse.Namespace) -> dict[str, Any]:
|
||||
now = utc_now()
|
||||
approval, spec = load_receipt(args.approval, now, require_open_window=True)
|
||||
if args.confirm != confirm_token(approval):
|
||||
raise LaneError(f"project requires --confirm {confirm_token(approval)}")
|
||||
assert_projection_time(approval, spec, now)
|
||||
if args.bearer_file is None:
|
||||
raise LaneError("project requires --bearer-file outside Git")
|
||||
check_bearer_path(args.bearer_file, must_exist=False)
|
||||
remote = PREFLIGHT.Remote(args.remote)
|
||||
before = cluster_state(remote)
|
||||
if not before["automated_checks_passed"]:
|
||||
raise LaneError("audit-core recovery preflight is not green; refusing to project")
|
||||
LEASE.attest_chain_state(remote)
|
||||
operator = CREDS.Operator(args.remote, args.token_file)
|
||||
registry = operator.registry()
|
||||
conflicts = registry_conflicts(registry, spec)
|
||||
if conflicts:
|
||||
raise LaneError("registry conflict: " + "; ".join(conflicts))
|
||||
token = secrets.token_urlsafe(48)
|
||||
bearer_written = registry_written = False
|
||||
try:
|
||||
write_bearer(args.bearer_file, token)
|
||||
bearer_written = True
|
||||
operator.write_registry(add_identity(registry, spec, token))
|
||||
registry_written = True
|
||||
CREDS.force_sender_sync_and_restart(operator)
|
||||
after = settled_state(remote, previous_uid=before["pod_uid"])
|
||||
except BaseException:
|
||||
if registry_written:
|
||||
_cleanup_registry(operator, spec)
|
||||
if bearer_written:
|
||||
CREDS.secure_unlink(args.bearer_file)
|
||||
raise
|
||||
finally:
|
||||
token = ""
|
||||
registry.clear()
|
||||
return {
|
||||
"lane": LANE,
|
||||
"approval_id": approval["approval_id"],
|
||||
"projected_at": stamp(utc_now()),
|
||||
"identity": identity_view(spec),
|
||||
"bearer_file": str(args.bearer_file),
|
||||
"bearer_file_mode": "0600",
|
||||
"audit_core_restarted": True,
|
||||
"audit_core_pod_uid_after_projection": after["pod_uid"],
|
||||
"next_step": "run the T02 exercise; its baseline must capture this pod UID",
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
|
||||
|
||||
def cleanup(args: argparse.Namespace) -> dict[str, Any]:
|
||||
approval, spec = load_receipt(args.approval, utc_now(), require_open_window=False)
|
||||
if args.confirm != confirm_token(approval):
|
||||
raise LaneError(f"cleanup requires --confirm {confirm_token(approval)}")
|
||||
gate = assert_evidence(args.exercise_evidence, args.abort_reason, approval)
|
||||
operator = CREDS.Operator(args.remote, args.token_file)
|
||||
removed = _cleanup_registry(operator, spec)
|
||||
remaining = any(
|
||||
isinstance(entry, dict) and entry.get("name") == spec["name"] for entry in operator.registry()
|
||||
)
|
||||
if remaining:
|
||||
raise LaneError("temporary identity remains after cleanup")
|
||||
bearer_removed = False
|
||||
if args.bearer_file is not None and (args.bearer_file.exists() or args.bearer_file.is_symlink()):
|
||||
check_bearer_path(args.bearer_file, must_exist=True)
|
||||
CREDS.secure_unlink(args.bearer_file)
|
||||
bearer_removed = True
|
||||
after = settled_state(PREFLIGHT.Remote(args.remote), previous_uid=None)
|
||||
return {
|
||||
"lane": LANE,
|
||||
"approval_id": approval["approval_id"],
|
||||
"cleanup_gate": gate,
|
||||
"cleanup_completed_at": stamp(utc_now()),
|
||||
"identity": spec["name"],
|
||||
"identities_removed": removed,
|
||||
"audit_core_restarted": bool(removed),
|
||||
"bearer_file_removed": bearer_removed,
|
||||
"audit_core_ready": after["ready"],
|
||||
"secret_values_observed": False,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("command", choices=["preflight", "project", "cleanup"])
|
||||
parser.add_argument("--approval", type=Path, required=True, help="approved T02 window receipt")
|
||||
parser.add_argument("--bearer-file", type=Path, help="absolute path outside Git (mode-0700 directory)")
|
||||
parser.add_argument("--remote", default="railiance01")
|
||||
parser.add_argument("--token-file", type=Path, default=Path.home() / ".local/openbao/platform-admin.token")
|
||||
parser.add_argument("--confirm")
|
||||
parser.add_argument("--offline", action="store_true", help="preflight: validate receipt/path only")
|
||||
parser.add_argument("--inspect-registry", action="store_true", help="preflight: attended registry conflict check")
|
||||
parser.add_argument("--exercise-evidence", type=Path, help="cleanup: recorded T02 exercise result JSON")
|
||||
parser.add_argument("--abort-reason", help="cleanup: abort path when no exercise evidence exists")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
handler = {"preflight": preflight, "project": project, "cleanup": cleanup}[args.command]
|
||||
result = handler(args)
|
||||
except ERRORS as exc:
|
||||
print(f"synthetic sender lane failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -35,6 +35,12 @@ def approved_receipt(now: datetime) -> dict:
|
|||
for owner in ("audit-core", "rapp-postgres", "railiance-platform")
|
||||
},
|
||||
"synthetic_load": {"contract_id": "load-1", "driver_revision": "abc123"},
|
||||
"synthetic_sender": {
|
||||
"identity": "t02-synthetic-sender",
|
||||
"source": "t02-synthetic",
|
||||
"tenant": "tenant:trial:t02-synthetic",
|
||||
"expires_at": "2026-08-22T18:15:00Z",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -64,6 +70,17 @@ class DatabaseLeaseRecoveryTests(unittest.TestCase):
|
|||
|
||||
self.assertEqual("serving", module.pod_state(Remote())["uid"])
|
||||
|
||||
def test_approval_requires_projected_synthetic_sender(self) -> None:
|
||||
now = datetime(2026, 8, 22, 12, 0, tzinfo=UTC)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / "approval.json"
|
||||
path.write_text(json.dumps(approved_receipt(now)), encoding="utf-8")
|
||||
self.assertTrue(module.load_approval(path, now, require_open_window=True)["synthetic_sender"])
|
||||
for sender in (None, {}, {"identity": "", "source": "s", "tenant": "t", "expires_at": "x"}):
|
||||
path.write_text(json.dumps({**approved_receipt(now), "synthetic_sender": sender}), encoding="utf-8")
|
||||
with self.assertRaises(module.ProcedureError):
|
||||
module.load_approval(path, now, require_open_window=True)
|
||||
|
||||
def test_approval_requires_exact_owner_and_window_contract(self) -> None:
|
||||
now = datetime(2026, 8, 22, 12, 0, tzinfo=UTC)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
|
|
|
|||
|
|
@ -19,6 +19,22 @@ SPEC.loader.exec_module(module)
|
|||
|
||||
|
||||
class RecoveryPreflightTests(unittest.TestCase):
|
||||
def test_ops_warden_receipt_reads_current_decision_of_review_history(self) -> None:
|
||||
revision = "a" * 40
|
||||
history = {
|
||||
"workplan_id": "RPF-WP-0017",
|
||||
"current_decision": "accept",
|
||||
"current_reviewed_revision": revision,
|
||||
"reviews": [{"decision": "reject"}, {"decision": "accept"}],
|
||||
}
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "receipt.json"
|
||||
path.write_text(json.dumps(history), encoding="utf-8")
|
||||
self.assertEqual(revision, module.validate_ops_warden_receipt(path)["accepted_revision"])
|
||||
path.write_text(json.dumps({**history, "current_decision": "reject"}), encoding="utf-8")
|
||||
with self.assertRaises(module.PreflightError):
|
||||
module.validate_ops_warden_receipt(path)
|
||||
|
||||
def test_serving_pods_ignores_job_owned_attest_chain_pods(self) -> None:
|
||||
def pod(name: str, kind: str) -> dict:
|
||||
return {"metadata": {"name": name, "ownerReferences": [{"kind": kind}]}}
|
||||
|
|
@ -92,6 +108,7 @@ class RecoveryPreflightTests(unittest.TestCase):
|
|||
"external_secrets": {"a": {"ready": True}, "b": {"ready": True}},
|
||||
"audit_core": {
|
||||
"image_matches_reviewed_digest": True,
|
||||
"running_pod_image_matches_reviewed_digest": True,
|
||||
"ready_replicas": 1,
|
||||
"replicas": 1,
|
||||
},
|
||||
|
|
@ -99,6 +116,9 @@ class RecoveryPreflightTests(unittest.TestCase):
|
|||
self.assertTrue(module.automated_common_pass(state))
|
||||
state["openbao"]["sealed"] = True
|
||||
self.assertFalse(module.automated_common_pass(state))
|
||||
state["openbao"]["sealed"] = False
|
||||
state["audit_core"]["running_pod_image_matches_reviewed_digest"] = False
|
||||
self.assertFalse(module.automated_common_pass(state))
|
||||
|
||||
def test_expected_database_projection_is_exact_key_set(self) -> None:
|
||||
self.assertEqual(
|
||||
|
|
@ -191,5 +211,234 @@ host: 50 bytes
|
|||
)
|
||||
|
||||
|
||||
DIGEST = module.EXPECTED_IMAGE.rsplit("@", 1)[1]
|
||||
|
||||
|
||||
def workload(ns: str, kind: str, name: str, ready: int = 1, replicas: int = 1) -> dict:
|
||||
return {
|
||||
"kind": kind,
|
||||
"metadata": {"namespace": ns, "name": name},
|
||||
"spec": {"replicas": replicas},
|
||||
"status": {"readyReplicas": ready},
|
||||
}
|
||||
|
||||
|
||||
def es_item(name: str, reason: str, status: str = "True", ns: str | None = None) -> dict:
|
||||
meta = {"name": name, **({"namespace": ns} if ns else {})}
|
||||
return {"metadata": meta, "status": {"conditions": [{"type": "Ready", "status": status, "reason": reason}]}}
|
||||
|
||||
|
||||
def cnpg_cluster(primary: str = "pg-1", archiving: str | None = "True") -> dict:
|
||||
cluster = {
|
||||
"metadata": {"namespace": "databases", "name": "pg", "uid": "uid-1"},
|
||||
"spec": {"instances": 1, **({"backup": {"barmanObjectStore": {}}} if archiving else {})},
|
||||
"status": {"readyInstances": 1, "currentPrimary": primary, "healthyPVC": [primary], "phase": "Cluster in healthy state"},
|
||||
}
|
||||
if archiving:
|
||||
cluster["status"]["conditions"] = [{"type": "ContinuousArchiving", "status": archiving}]
|
||||
return cluster
|
||||
|
||||
|
||||
class NodeRebootPreflightTests(unittest.TestCase):
|
||||
def test_running_pod_image_uses_resolved_digest(self) -> None:
|
||||
pod = {
|
||||
"spec": {"containers": [{"name": "app", "image": module.EXPECTED_IMAGE}]},
|
||||
"status": {"containerStatuses": [{"name": "app", "imageID": "forgejo.coulomb.social/coulomb/audit-core@" + DIGEST}]},
|
||||
}
|
||||
self.assertEqual(DIGEST, module.running_pod_image(pod)["digest"])
|
||||
pod["status"]["containerStatuses"][0]["imageID"] = "x@sha256:" + "0" * 64
|
||||
self.assertNotEqual(DIGEST, module.running_pod_image(pod)["digest"])
|
||||
pod["status"]["containerStatuses"] = []
|
||||
self.assertIsNone(module.running_pod_image(pod)["digest"])
|
||||
|
||||
def test_core_workloads_require_named_and_all_in_namespace(self) -> None:
|
||||
items = [workload(*key) for key in module.CORE_PLATFORM_WORKLOADS]
|
||||
self.assertTrue(module.core_platform_workloads({"items": items})["ready"])
|
||||
result = module.core_platform_workloads({"items": items[1:]})
|
||||
self.assertEqual(["kube-system/Deployment/coredns"], result["missing"])
|
||||
extra = items + [workload("argocd", "Deployment", "argocd-server", ready=0)]
|
||||
result = module.core_platform_workloads({"items": extra})
|
||||
self.assertEqual(["argocd/Deployment/argocd-server"], result["unready"])
|
||||
self.assertFalse(result["ready"])
|
||||
ignored = items + [workload("coulomb", "Deployment", "probe", ready=0)]
|
||||
self.assertTrue(module.core_platform_workloads({"items": ignored})["ready"])
|
||||
|
||||
def test_failing_pods_and_baseline_diff(self) -> None:
|
||||
def pod(name: str, phase: str, ready: bool = True) -> dict:
|
||||
return {"metadata": {"namespace": "ns", "name": name},
|
||||
"status": {"phase": phase, "containerStatuses": [{"ready": ready}]}}
|
||||
pods = {"items": [pod("ok", "Running"), pod("done", "Succeeded"), pod("bad", "Failed", False),
|
||||
pod("pending", "Pending", False), pod("unready", "Running", False)]}
|
||||
current = module.failing_pods(pods)
|
||||
self.assertEqual(["ns/bad", "ns/pending", "ns/unready"], current)
|
||||
diff = module.failing_pod_diff(["ns/bad", "ns/gone"], current)
|
||||
self.assertEqual(["ns/pending", "ns/unready"], diff["new_failing"])
|
||||
self.assertEqual(["ns/gone"], diff["resolved"])
|
||||
self.assertFalse(diff["no_new_failures"])
|
||||
self.assertTrue(module.failing_pod_diff(current, current)["no_new_failures"])
|
||||
|
||||
def test_cnpg_identity_and_health(self) -> None:
|
||||
self.assertEqual("7672", module.system_identifier("x\nDatabase system identifier: 7672\ny"))
|
||||
self.assertIsNone(module.system_identifier("nothing"))
|
||||
identity = module.cnpg_identity(cnpg_cluster(), {"pg-1": "pvc-uid"})
|
||||
identity["system_id"] = "7672"
|
||||
self.assertEqual("pvc-uid", identity["primary_pvc_uid"])
|
||||
self.assertTrue(module.cnpg_healthy(identity))
|
||||
no_backup = module.cnpg_identity(cnpg_cluster(archiving=None), {"pg-1": "pvc-uid"})
|
||||
no_backup["system_id"] = "7672"
|
||||
self.assertIsNone(no_backup["continuous_archiving"])
|
||||
self.assertTrue(module.cnpg_healthy(no_backup))
|
||||
broken = module.cnpg_identity(cnpg_cluster(archiving="False"), {"pg-1": "pvc-uid"})
|
||||
broken["system_id"] = "7672"
|
||||
self.assertFalse(module.cnpg_healthy(broken))
|
||||
missing_pvc = module.cnpg_identity(cnpg_cluster(), {})
|
||||
missing_pvc["system_id"] = "7672"
|
||||
self.assertFalse(module.cnpg_healthy(missing_pvc))
|
||||
|
||||
def test_backups_grouped_per_cluster(self) -> None:
|
||||
backups = {"items": [
|
||||
{"metadata": {"namespace": "databases", "name": "b1"}, "spec": {"cluster": {"name": "platform-pg"}},
|
||||
"status": {"phase": "completed", "stoppedAt": "2026-10-14T00:00:00Z"}},
|
||||
{"metadata": {"namespace": "databases", "name": "b2"}, "spec": {"cluster": {"name": "apps-pg"}},
|
||||
"status": {"phase": "failed"}},
|
||||
]}
|
||||
result = module.latest_backups_by_cluster(backups, datetime(2026, 10, 14, 2, tzinfo=UTC))
|
||||
self.assertEqual(2.0, result["databases/platform-pg"]["age_hours"])
|
||||
self.assertIsNone(result["databases/apps-pg"])
|
||||
|
||||
def test_eso_inventory_requires_every_store_and_projection(self) -> None:
|
||||
stores = {"items": [es_item("a", "Valid"), es_item("b", "Valid")]}
|
||||
projections = {"items": [es_item("x", "SecretSynced", ns="n")]}
|
||||
self.assertTrue(module.eso_inventory(stores, projections)["ready"])
|
||||
stores["items"].append(es_item("c", "InvalidProviderConfig", "False"))
|
||||
projections["items"].append(es_item("y", "SecretSyncedError", "False", ns="n"))
|
||||
result = module.eso_inventory(stores, projections)
|
||||
self.assertEqual(["c"], result["invalid_cluster_stores"])
|
||||
self.assertEqual(["n/y"], result["unsynced_external_secrets"])
|
||||
self.assertFalse(result["ready"])
|
||||
self.assertFalse(module.eso_inventory({"items": []}, {"items": []})["ready"])
|
||||
|
||||
def test_ufw_matches_declaration_only(self) -> None:
|
||||
ok = """Status: active
|
||||
|
||||
To Action From
|
||||
-- ------ ----
|
||||
OpenSSH ALLOW Anywhere
|
||||
2224/tcp ALLOW Anywhere # nydus
|
||||
OpenSSH (v6) ALLOW Anywhere (v6)
|
||||
2224/tcp (v6) ALLOW Anywhere (v6) # nydus
|
||||
"""
|
||||
self.assertTrue(module.ufw_state(ok)["matches_declaration"])
|
||||
opened = ok + "6443/tcp ALLOW Anywhere\n"
|
||||
result = module.ufw_state(opened)
|
||||
self.assertEqual(["6443/tcp"], result["forbidden_ports_open"])
|
||||
self.assertFalse(result["matches_declaration"])
|
||||
self.assertFalse(module.ufw_state("Status: inactive\n")["matches_declaration"])
|
||||
|
||||
def test_boot_count_and_reboot_comparison(self) -> None:
|
||||
pre_time = datetime(2026, 10, 14, 8, tzinfo=UTC)
|
||||
after = int((pre_time + timedelta(minutes=5)).timestamp() * 1_000_000)
|
||||
before = int((pre_time - timedelta(days=1)).timestamp() * 1_000_000)
|
||||
boots = json.dumps([{"boot_id": "a", "first_entry": before}, {"boot_id": "b", "first_entry": after}])
|
||||
self.assertEqual(1, module.boots_since(boots, pre_time))
|
||||
db = {"databases/pg": {"uid": "u", "primary_pvc_uid": "p", "system_id": "s"}}
|
||||
pre = {"host": {"boot_id": "a"}, "cnpg_clusters": db, "failing_pods": ["ns/old"]}
|
||||
post = {"host": {"boot_id": "b"}, "cnpg_clusters": json.loads(json.dumps(db)), "failing_pods": ["ns/old"]}
|
||||
self.assertTrue(module.reboot_comparison(pre, post, 1)["passed"])
|
||||
self.assertFalse(module.reboot_comparison(pre, post, 2)["exactly_one_reboot"])
|
||||
same_boot = {**post, "host": {"boot_id": "a"}}
|
||||
self.assertFalse(module.reboot_comparison(pre, same_boot, 1)["passed"])
|
||||
post["cnpg_clusters"]["databases/pg"]["system_id"] = "rebootstrapped"
|
||||
result = module.reboot_comparison(pre, post, 1)
|
||||
self.assertEqual(["databases/pg:system_id"], result["database_identity_changes"])
|
||||
self.assertFalse(result["passed"])
|
||||
post["cnpg_clusters"] = {}
|
||||
self.assertFalse(module.reboot_comparison(pre, post, 1)["passed"])
|
||||
|
||||
def test_consumed_decisions_and_scenario_are_not_reusable(self) -> None:
|
||||
gates = module.fresh_decision_gates(
|
||||
"WARDEN-WP-0027-T02-DRILL-20260822-01", "449a697a-5303-4582-aa9e-b0bc8b35ab2d"
|
||||
)
|
||||
self.assertFalse(any(gates.values()))
|
||||
self.assertFalse(module.fresh_decision_gates("x", "85724c0c-e70f")["fresh_human_decision_recorded"])
|
||||
gates = module.fresh_decision_gates("RAILIANCE-WP-0024-T03-REBOOT-20261020-01", "0123abcd-new")
|
||||
self.assertTrue(all(gates.values()))
|
||||
self.assertFalse(any(module.fresh_decision_gates(None, None).values()))
|
||||
|
||||
def _write(self, directory: str, name: str, text: str) -> Path:
|
||||
path = Path(directory) / name
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
def test_ops_warden_receipt_requires_acceptance_and_revision(self) -> None:
|
||||
base = {"workplan_id": "RPF-WP-0017", "reviewed_revision": "a" * 40,
|
||||
"secret_values_observed": False, "sensitive_material_recorded": False}
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
ok = self._write(directory, "ok.json", json.dumps({**base, "decision": "accept"}))
|
||||
self.assertEqual("a" * 40, module.validate_ops_warden_receipt(ok)["accepted_revision"])
|
||||
for name, body in (
|
||||
("reject.json", {**base, "decision": "reject"}),
|
||||
("norev.json", {**base, "decision": "accept", "reviewed_revision": "HEAD"}),
|
||||
("other.json", {**base, "decision": "accept", "workplan_id": "RPF-WP-0099"}),
|
||||
("leak.json", {**base, "decision": "accept", "note": "hvs.abc"}),
|
||||
("sensitive.json", {**base, "decision": "accept", "sensitive_material_recorded": True}),
|
||||
):
|
||||
with self.assertRaises(module.PreflightError, msg=name):
|
||||
module.validate_ops_warden_receipt(self._write(directory, name, json.dumps(body)))
|
||||
with self.assertRaises(module.PreflightError):
|
||||
module.validate_ops_warden_receipt(Path(directory) / "missing.json")
|
||||
|
||||
def test_console_receipt_same_day_window_and_independent(self) -> None:
|
||||
now = datetime(2026, 10, 15, 9, 0, tzinfo=UTC)
|
||||
good = {
|
||||
"receipt_id": "RAIL01-CONSOLE-20261015T080000Z", "host": "railiance01",
|
||||
"attested_by_role": "host-europe-account-holder", "attested_at": "2026-10-15T08:00:00Z",
|
||||
"panel_login": "ok", "mfa_used": True, "console_mechanism": "VNC console",
|
||||
"console_shows_host": True, "independent_of_guest_network": True,
|
||||
"rescue_mode_offered": True, "hard_reboot_offered": True, "reboot_window_id": "WIN-20261015-01",
|
||||
}
|
||||
name = good["receipt_id"] + ".yaml"
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
ok = self._write(directory, name, json.dumps(good)) # JSON is valid YAML
|
||||
result = module.validate_console_receipt(ok, window_id="WIN-20261015-01", now=now)
|
||||
self.assertTrue(result["verified"])
|
||||
with self.assertRaises(module.PreflightError):
|
||||
module.validate_console_receipt(ok, window_id="WIN-OTHER", now=now)
|
||||
with self.assertRaises(module.PreflightError):
|
||||
module.validate_console_receipt(ok, window_id="WIN-20261015-01", now=now + timedelta(days=1))
|
||||
for change in (
|
||||
{"independent_of_guest_network": "unknown"},
|
||||
{"console_shows_host": False},
|
||||
{"panel_login": "failed"},
|
||||
{"console_mechanism": "<as named in the panel>"},
|
||||
{"note": "password=hunter2"},
|
||||
):
|
||||
path = self._write(directory, name, json.dumps({**good, **change}))
|
||||
with self.assertRaises(module.PreflightError, msg=str(change)):
|
||||
module.validate_console_receipt(path, window_id="WIN-20261015-01", now=now)
|
||||
renamed = self._write(directory, "RAIL01-CONSOLE-20261015T070000Z.yaml", json.dumps(good))
|
||||
with self.assertRaises(module.PreflightError):
|
||||
module.validate_console_receipt(renamed, window_id="WIN-20261015-01", now=now)
|
||||
|
||||
def test_reboot_automated_pass_names_each_boundary(self) -> None:
|
||||
extra = {
|
||||
"host": {"ntp_synchronized": True, "firewall": {"matches_declaration": True}},
|
||||
"k3s_api_tunnel_answers": True,
|
||||
"core_platform_workloads": {"ready": True},
|
||||
"missing_expected_cnpg_clusters": [],
|
||||
"cnpg_clusters": {},
|
||||
"cnpg_backups": {"databases/platform-pg": {"age_hours": 3}},
|
||||
"eso": {"ready": True},
|
||||
}
|
||||
self.assertTrue(all(module.reboot_automated_pass(extra, 26).values()))
|
||||
extra["host"]["ntp_synchronized"] = False
|
||||
extra["missing_expected_cnpg_clusters"] = ["databases/state-hub-db"]
|
||||
checks = module.reboot_automated_pass(extra, 26)
|
||||
self.assertFalse(checks["ntp_synchronized"])
|
||||
self.assertFalse(checks["all_cnpg_clusters_healthy"])
|
||||
extra["cnpg_backups"] = {}
|
||||
self.assertFalse(module.reboot_automated_pass(extra, 26)["required_backups_fresh"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
242
tests/test_audit_core_synthetic_sender_lane.py
Normal file
242
tests/test_audit_core_synthetic_sender_lane.py
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"audit_core_synthetic_sender_lane",
|
||||
ROOT / "scripts" / "audit_core_synthetic_sender_lane.py",
|
||||
)
|
||||
assert SPEC and SPEC.loader
|
||||
lane = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(lane)
|
||||
|
||||
CONTRACT = "RAILIANCE-WP-0024-T02-SYNTHETIC-01"
|
||||
|
||||
|
||||
def z(value: datetime) -> str:
|
||||
return value.astimezone(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def receipt(now: datetime, **sender: str) -> dict:
|
||||
start = now - timedelta(minutes=1)
|
||||
end = start + timedelta(minutes=15)
|
||||
return {
|
||||
"procedure": lane.LEASE.PROCEDURE,
|
||||
"task_id": lane.LEASE.TASK_ID,
|
||||
"status": "approved",
|
||||
"approval_id": "t02-window-1",
|
||||
"window": {"start": z(start), "end": z(end)},
|
||||
"abort_operator": "operator:example",
|
||||
"owners": {
|
||||
owner: {"acknowledged": True, "message_id": f"msg-{owner}"}
|
||||
for owner in ("audit-core", "rapp-postgres", "railiance-platform")
|
||||
},
|
||||
"synthetic_load": {"contract_id": CONTRACT, "driver_revision": "sha256:" + "0" * 64},
|
||||
"synthetic_sender": {
|
||||
"identity": "t02-synthetic-1",
|
||||
"source": "railiance-t02-synthetic",
|
||||
"tenant": "tenant:trial:t02-synthetic-1",
|
||||
"expires_at": z(end),
|
||||
**sender,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class FakeOperator:
|
||||
def __init__(self, registry: list) -> None:
|
||||
self.registry_value = registry
|
||||
self.writes: list[list] = []
|
||||
|
||||
def registry(self) -> list:
|
||||
return json.loads(json.dumps(self.registry_value))
|
||||
|
||||
def write_registry(self, registry: list) -> None:
|
||||
self.writes.append(registry)
|
||||
self.registry_value = json.loads(json.dumps(registry))
|
||||
|
||||
|
||||
class SyntheticSenderLaneTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.dir = Path(self.temp.name)
|
||||
os.chmod(self.dir, 0o700)
|
||||
self.now = datetime.now(UTC)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def write_receipt(self, document: dict) -> Path:
|
||||
path = self.dir / "receipt.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
return path
|
||||
|
||||
def test_receipt_yields_exactly_one_scoped_expiring_identity(self) -> None:
|
||||
_, spec = lane.load_receipt(self.write_receipt(receipt(self.now)), self.now, require_open_window=True)
|
||||
entry = lane.registry_entry(spec, "bearer")
|
||||
self.assertEqual(["railiance-t02-synthetic"], entry["sources"])
|
||||
self.assertEqual(["tenant:trial:t02-synthetic-1"], entry["tenants"])
|
||||
self.assertTrue(entry["may_write"] and entry["may_read"])
|
||||
self.assertLessEqual(lane.LEASE.parse_time(entry["expires_at"]), lane.LEASE.parse_time(receipt(self.now)["window"]["end"]))
|
||||
|
||||
def test_receipt_rejections(self) -> None:
|
||||
end = lane.LEASE.parse_time(receipt(self.now)["window"]["end"])
|
||||
bad = [
|
||||
{"source": "*"},
|
||||
{"source": "user-engine"},
|
||||
{"tenant": "*"},
|
||||
{"tenant": "tenant:friendly:binky"},
|
||||
{"identity": "user-engine"},
|
||||
{"expires_at": z(end + timedelta(seconds=1))},
|
||||
{"expires_at": "2026-10-14T10:00:00+00:00"},
|
||||
]
|
||||
for override in bad:
|
||||
with self.subTest(override=override):
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.sender_spec(receipt(self.now, **override))
|
||||
missing = receipt(self.now)
|
||||
del missing["synthetic_sender"]
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.sender_spec(missing)
|
||||
extra = receipt(self.now)
|
||||
extra["synthetic_sender"]["tokens"] = ["x"]
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.sender_spec(extra)
|
||||
wrong_contract = receipt(self.now)
|
||||
wrong_contract["synthetic_load"]["contract_id"] = "OTHER"
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.sender_spec(wrong_contract)
|
||||
|
||||
def test_pending_example_receipt_is_refused(self) -> None:
|
||||
with self.assertRaises(lane.LEASE.ProcedureError):
|
||||
lane.load_receipt(ROOT / "docs" / "audit-core-database-lease-approval.example.json", self.now, require_open_window=False)
|
||||
|
||||
def test_projection_time_gate(self) -> None:
|
||||
document = receipt(self.now)
|
||||
spec = lane.sender_spec(document)
|
||||
lane.assert_projection_time(document, spec, self.now)
|
||||
start = lane.LEASE.parse_time(document["window"]["start"])
|
||||
for when in (start - timedelta(seconds=1), start + lane.PROJECT_CUTOFF + timedelta(seconds=1)):
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.assert_projection_time(document, spec, when)
|
||||
short = receipt(self.now, expires_at=z(self.now + timedelta(minutes=5)))
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.assert_projection_time(short, lane.sender_spec(short), self.now)
|
||||
|
||||
def test_registry_add_refuses_overlap_and_remove_is_exact(self) -> None:
|
||||
spec = lane.sender_spec(receipt(self.now))
|
||||
production = [{"name": "user-engine", "tokens": ["p"], "sources": ["user-engine"], "tenants": ["*"]}]
|
||||
updated = lane.add_identity(production, spec, "bearer")
|
||||
self.assertEqual(2, len(updated))
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.add_identity(updated, spec, "again")
|
||||
shared_source = [{"name": "x", "sources": [spec["source"]], "tenants": ["tenant:trial:x"]}]
|
||||
self.assertTrue(lane.registry_conflicts(shared_source, spec))
|
||||
cleaned, removed = lane.remove_identity(updated, spec)
|
||||
self.assertEqual((production, 1), (cleaned, removed))
|
||||
self.assertEqual((production, 0), lane.remove_identity(production, spec))
|
||||
hijacked = [{"name": spec["name"], "sources": ["user-engine"], "tenants": ["*"]}]
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.remove_identity(hijacked, spec)
|
||||
|
||||
def test_bearer_file_is_new_0600_outside_git(self) -> None:
|
||||
target = self.dir / "t02.bearer"
|
||||
lane.write_bearer(target, "value")
|
||||
self.assertEqual(0o600, target.stat().st_mode & 0o777)
|
||||
lane.check_bearer_path(target, must_exist=True)
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.write_bearer(target, "value")
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.check_bearer_path(ROOT / "t02.bearer", must_exist=False)
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.check_bearer_path(Path("relative.bearer"), must_exist=False)
|
||||
open_dir = self.dir / "open"
|
||||
open_dir.mkdir(mode=0o755)
|
||||
os.chmod(open_dir, 0o755)
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.check_bearer_path(open_dir / "t02.bearer", must_exist=False)
|
||||
|
||||
def test_cleanup_requires_matching_evidence_or_abort_reason(self) -> None:
|
||||
approval = receipt(self.now)
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.assert_evidence(None, None, approval)
|
||||
evidence = self.dir / "evidence.json"
|
||||
evidence.write_text(json.dumps({
|
||||
"procedure": lane.LEASE.PROCEDURE, "approval_id": "other",
|
||||
"completed_at": z(self.now), "secret_values_observed": False,
|
||||
}), encoding="utf-8")
|
||||
with self.assertRaises(lane.LaneError):
|
||||
lane.assert_evidence(evidence, None, approval)
|
||||
evidence.write_text(json.dumps({
|
||||
"procedure": lane.LEASE.PROCEDURE, "approval_id": approval["approval_id"],
|
||||
"completed_at": z(self.now), "secret_values_observed": False,
|
||||
}), encoding="utf-8")
|
||||
self.assertEqual("evidence-recorded", lane.assert_evidence(evidence, None, approval)["mode"])
|
||||
self.assertEqual("abort", lane.assert_evidence(None, "readiness never dropped", approval)["mode"])
|
||||
|
||||
def test_cleanup_registry_is_idempotent_and_restarts_only_on_change(self) -> None:
|
||||
spec = lane.sender_spec(receipt(self.now))
|
||||
operator = FakeOperator(lane.add_identity([], spec, "bearer"))
|
||||
with mock.patch.object(lane.CREDS, "force_sender_sync_and_restart") as restart:
|
||||
self.assertEqual(1, lane._cleanup_registry(operator, spec))
|
||||
self.assertEqual(0, lane._cleanup_registry(operator, spec))
|
||||
self.assertEqual(1, restart.call_count)
|
||||
self.assertEqual([], operator.registry_value)
|
||||
|
||||
def test_project_rolls_back_registry_and_bearer_on_restart_failure(self) -> None:
|
||||
approval_path = self.write_receipt(receipt(self.now))
|
||||
approval = json.loads(approval_path.read_text())
|
||||
bearer = self.dir / "t02.bearer"
|
||||
operator = FakeOperator([{"name": "user-engine", "tokens": ["p"], "sources": ["user-engine"], "tenants": ["*"]}])
|
||||
args = argparse.Namespace(
|
||||
approval=approval_path, confirm=lane.confirm_token(approval), bearer_file=bearer,
|
||||
remote="railiance01", token_file=self.dir / "unused",
|
||||
)
|
||||
green = {"automated_checks_passed": True, "ready": "1/1", "pod_uid": "uid-1"}
|
||||
restart = mock.Mock(side_effect=[lane.CREDS.ProcedureError("rollout failed"), None])
|
||||
with mock.patch.object(lane.PREFLIGHT, "Remote"), \
|
||||
mock.patch.object(lane, "cluster_state", return_value=green), \
|
||||
mock.patch.object(lane.LEASE, "attest_chain_state"), \
|
||||
mock.patch.object(lane.CREDS, "Operator", return_value=operator), \
|
||||
mock.patch.object(lane.CREDS, "force_sender_sync_and_restart", restart):
|
||||
with self.assertRaises(lane.CREDS.ProcedureError):
|
||||
lane.project(args)
|
||||
self.assertFalse(bearer.exists())
|
||||
self.assertEqual(["user-engine"], [entry["name"] for entry in operator.registry_value])
|
||||
self.assertEqual(2, restart.call_count)
|
||||
|
||||
def test_project_requires_exact_confirmation_and_no_bearer_on_stdout(self) -> None:
|
||||
approval_path = self.write_receipt(receipt(self.now))
|
||||
stdout, stderr = io.StringIO(), io.StringIO()
|
||||
with redirect_stdout(stdout), redirect_stderr(stderr):
|
||||
code = lane.main(["project", "--approval", str(approval_path), "--confirm", "wrong",
|
||||
"--bearer-file", str(self.dir / "b")])
|
||||
self.assertEqual(1, code)
|
||||
self.assertIn("--confirm audit-core-synthetic-sender:t02-window-1:attended", stderr.getvalue())
|
||||
self.assertEqual("", stdout.getvalue())
|
||||
|
||||
def test_offline_preflight_is_value_safe(self) -> None:
|
||||
stdout = io.StringIO()
|
||||
with redirect_stdout(stdout):
|
||||
code = lane.main(["preflight", "--offline", "--approval", str(self.write_receipt(receipt(self.now))),
|
||||
"--bearer-file", str(self.dir / "t02.bearer")])
|
||||
self.assertEqual(0, code)
|
||||
result = json.loads(stdout.getvalue())
|
||||
self.assertEqual("projection-window-open", result["time_state"])
|
||||
self.assertFalse(result["secret_values_observed"])
|
||||
self.assertNotIn("tokens", json.dumps(result))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -392,3 +392,27 @@ checklist revisions requested by cluster/infra/postgres/platform (full-cluster r
|
|||
checks, pre-reboot failing-pod baseline, all CNPG clusters, firewall/boot-id/time sync,
|
||||
named console path, infra abort ownership, ops-warden login revision re-acceptance and
|
||||
stop-on-credential-output rule, PVC/cluster UID capture).
|
||||
|
||||
Revision for the second owner-review round (2026-10-14):
|
||||
- T02: attended synthetic-sender lane `scripts/audit_core_synthetic_sender_lane.py`
|
||||
(+ Makefile targets, tests) projects exactly one receipt-named
|
||||
`tenant:trial:` identity, restarts audit-core before the baseline and cleans up only
|
||||
after recorded evidence; the lease harness now requires the receipt's
|
||||
`synthetic_sender` block. The lane reuses helpers from the dated -02 script; move
|
||||
them to a shared module before that script is retired.
|
||||
- T03: `docs/railiance01-coordinated-reboot.md` and the node-reboot preflight cover
|
||||
named core workloads, a failing-pod baseline diff, every CNPG cluster (UID, PVC UID,
|
||||
systemID), all stores/ExternalSecrets, firewall/tunnel/boot-id, NTP, Forgejo pull
|
||||
lag, infra abort ownership, a same-day `RAIL01-CONSOLE-` receipt
|
||||
(railiance-infra docs/railiance01-console-access.md), new scenario/decision ids and
|
||||
an accepted ops-warden revision; running-pod image digest is now checked too.
|
||||
- ops-warden: re-acceptance first rejected HEAD 11b2a9d (b4c1d39 had dropped the
|
||||
login-output fail-closed check); fixed locally in ops-warden 244f1ae/6e14300 and
|
||||
accepted at 6e14300 (reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json).
|
||||
The OIDC stderr prompt is an allowlisted relaxation. WARDEN-WP-0036-T02's
|
||||
2026-09-01 live op ran while the relaxed check was in source (railiance-infra follow-up).
|
||||
Remaining before any window: second-round owner approvals; console access proven
|
||||
(RAIL-HO-WP-0011-T09); fresh backups or waivers for state-hub-db, user-engine-pg,
|
||||
net-kingdom-pg, target-revenue-pg; stuck PVC platform-pg-drill/platform-pg-drill-1;
|
||||
ops-warden 6e14300 published and installed; then window ids, abort operator, snapshot
|
||||
receipt and 2-of-3 unseal quorum.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue