From 9886567b409cb34df4f355d505babd33804eb889 Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 27 Sep 2026 18:47:55 +0200 Subject: [PATCH] Fix rotation review evidence and reconcile blocked S1 workplans Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e3b9-b19e-7ba1-9eb4-4faea76af3ea --- Makefile | 8 +-- docs/evidence/2026-09-27-loose-ends.md | 54 +++++++++++++++++++ ...026-09-27-railiance01-clock-inventory.json | 48 +++++++++++++++++ docs/sops-rotation-approval.example.yaml | 3 +- docs/sops-rotation.md | 7 ++- scripts/sops_rotation.py | 10 +++- tests/test_secret_and_receipt_contracts.py | 54 +++++++++++++++++++ ...reproducible-s1-declaration-and-handoff.md | 28 +++++++++- ...RAIL-HO-WP-0012-s1-backup-recovery-loop.md | 24 +++++++-- .../RAIL-HO-WP-0013-host-time-baseline.md | 37 +++++++++++-- 10 files changed, 255 insertions(+), 18 deletions(-) create mode 100644 docs/evidence/2026-09-27-loose-ends.md create mode 100644 docs/evidence/2026-09-27-railiance01-clock-inventory.json diff --git a/Makefile b/Makefile index 00ffe4d..5baa7d9 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ IMG ?= ubuntu-24.04 USER ?= admin # Decrypt Hetzner token at runtime (requires SOPS_AGE_KEY or keys.txt locally) -HCLOUD_TOKEN := $(shell sops -d --extract '["hetzner"]["token"]' secrets/hetzner-token.yaml 2>/dev/null) +HCLOUD_TOKEN = $(shell sops -d --extract '["hetzner"]["token"]' secrets/hetzner-token.yaml 2>/dev/null) # ---- Help ---- help: ## Show this help @@ -112,7 +112,7 @@ tf-destroy: ## Terraform destroy (exact approval required before init) # --- Terraform provider/lockfile helpers --- TF_DIR := terraform/hetzner -TF_TOKEN := $(HCLOUD_TOKEN) +TF_TOKEN = $(HCLOUD_TOKEN) LOCKFILE := $(TF_DIR)/.terraform.lock.hcl tf-lock-commit: ## Commit the current provider lockfile @@ -287,8 +287,8 @@ PLAY := $(ANS_DIR)/playbooks/bootstrap.yaml SSH_USER ?= ANSIBLE_USER_FLAG := $(if $(SSH_USER),-u $(SSH_USER),) -# Load your SOPS key for decryption when running playbooks (optional if you use keys.txt) -export SOPS_AGE_KEY := $(shell cat ~/.config/sops/age/keys.txt 2>/dev/null) +# SOPS reads its standard key file itself; preserve an explicitly supplied +# SOPS_AGE_KEY without reading/exporting private keys for unrelated Make targets. ansible-help: ## Show common Ansible commands @echo "Convergence targets:" diff --git a/docs/evidence/2026-09-27-loose-ends.md b/docs/evidence/2026-09-27-loose-ends.md new file mode 100644 index 0000000..8767ae3 --- /dev/null +++ b/docs/evidence/2026-09-27-loose-ends.md @@ -0,0 +1,54 @@ +# Existing workplan closeout review — 2026-09-27 + +Reviewed all root and archived workplans. The only unfinished plans are +RAIL-HO-WP-0011, 0012 and 0013; all now have state `blocked`, with eight +remaining tasks in `wait`. No new task or workplan was opened. No residual task +has been marked done without its required live acceptance evidence. + +## Implemented under WP-0011 T06/T08 + +- Rotation dry-run output exposes the exact metadata-only review plan. +- Approval binds each changed ciphertext's SHA-256 as well as recipients/path. +- Applied receipts retain the original and resulting recipient/hash evidence. +- Unrelated Make targets no longer eagerly decrypt the Hetzner token or read + and export the local age private key. + +Validation: 54 Python unit tests pass, including three new rotation regression +tests. Inventory, baseline parity, read-only handoff contract, protected secret +paths, SOPS recipient metadata and whitespace checks pass. Ansible-core 2.17.13 +host-time playbook syntax check passes in a disposable controller environment. +No production decryption, recipient rotation or credential retrieval occurred. + +## Read-only host verification + +`ansible-playbook playbooks/verify.yaml`, Ansible-core 2.17.13, reached both hosts: + +| Host | Checks completed | Blocking assertion | Changes | +| --- | --- | --- | --- | +| CoulombCore | Executable and baseline stat | `/usr/local/bin/goss` absent | 0 | +| Railiance01 | Executable and baseline stat | Installed baseline digest differs from source render | 0 | + +The initial sandboxed attempt failed writing Ansible's connection cache; the +rerun with that access produced the host findings above. Neither attempt is a +passing handoff. Host refresh includes installation/configuration and an hourly +timer, so the concrete rendered diff must be reviewed before that separate +mutation; subsequent baseline failures must also be resolved before T05 closes. + +## Backup dependency correction + +The exact S1 offsite contract remains pending: +`d150eb3e6a19d658aa76c930b32fc20ef75ffa399558fc193fbde0738551ee62`. +Warden's route reports unknown execution workload identity. Platform WP-0029's +September 15 closure resolves the old upload-share incident, but does not accept +this contract or prove S1 transfer/restore. Keep WP-0012 T05/T06 waiting for those +specific owner and recovery receipts. + +## Clock dependency check + +The existing railiance-clock collector produced +`2026-09-27-railiance01-clock-inventory.json`. Read-only `dpkg-query -W systemd +systemd-timesyncd` returned `255.4-1ubuntu8.17` for both. Effective source/poll +observations and configuration hashes are recorded in the receipt and WP-0013. +RCLK-WP-0002 still lacks completed policy review; no disposable Ubuntu VM is +declared here for reboot/outage/rollback testing. The already deployed authority +does not close those gates. No clocks or services were changed. diff --git a/docs/evidence/2026-09-27-railiance01-clock-inventory.json b/docs/evidence/2026-09-27-railiance01-clock-inventory.json new file mode 100644 index 0000000..4c650e8 --- /dev/null +++ b/docs/evidence/2026-09-27-railiance01-clock-inventory.json @@ -0,0 +1,48 @@ +{ + "schema_version": 1, + "host_alias": "railiance01", + "collector_reported_at": "2026-09-27T16:46:14.477816+00:00", + "read_only": true, + "authority_usable": null, + "synchronized_reported": true, + "limitations": [ + "OS synchronization flag is not an independent UTC error bound", + "host and collector timestamps are observations, not authenticated time samples", + "SSH uses existing caller identity/trust; no sudo, installation or time mutation" + ], + "collector_sha256": "3aadb6fb888d87cefb63e94ac87c9c640ef63ef6f404b09a01e8e8642b26784e", + "remote_script_sha256": "597e08b95855c4bd1fee40ad8e7c1a8359312c1583ac8363e5bd922b493f074a", + "inventory": { + "remote_reported_at": "2026-09-27T16:46:16.708853+00:00", + "commands": { + "clock": { + "returncode": 0, + "stdout": "Timezone=Etc/UTC\nLocalRTC=no\nNTP=yes\nNTPSynchronized=yes\n" + }, + "upstream": { + "returncode": 0, + "stdout": "LinkNTPServers=\nSystemNTPServers=ntp.ubuntu.com\nRuntimeNTPServers=\nFallbackNTPServers=ntp.ubuntu.com\nServerName=ntp.ubuntu.com\nServerAddress=2620:2d:4000:1::41\nRootDistanceMaxUSec=500ms\nPollIntervalMinUSec=32s\nPollIntervalMaxUSec=1min 4s\nPollIntervalUSec=1min 4s\nNTPMessage={ Leap=0, Version=4, Mode=4, Stratum=2, Precision=-25, RootDelay=6.454ms, RootDispersion=183us, Reference=1D586304, OriginateTimestamp=Sun 2026-09-27 16:45:18 UTC, ReceiveTimestamp=Sun 2026-09-27 16:45:18 UTC, TransmitTimestamp=Sun 2026-09-27 16:45:18 UTC, DestinationTimestamp=Sun 2026-09-27 16:45:18 UTC, Ignored=no, PacketCount=15904, Jitter=3.125ms }\nFrequency=1174453\n" + }, + "timesyncd": { + "returncode": 0, + "stdout": "User=systemd-timesync\nActiveState=active\nFragmentPath=/usr/lib/systemd/system/systemd-timesyncd.service\nDropInPaths=\nUnitFileState=enabled\n" + }, + "other_daemons": { + "returncode": 4, + "stdout": "inactive\ninactive\ninactive\n" + }, + "virtualization": { + "returncode": 0, + "stdout": "kvm\n" + }, + "ntp_listeners": { + "returncode": 0, + "stdout": "" + } + }, + "config_sha256": { + "/etc/systemd/timesyncd.conf": "e6734751f8aaf19fddfff891ad246387f5f59bd9ff1a5f0cac2c34bc81941c62", + "/etc/systemd/timesyncd.conf.d/60-railiance-clock.conf": "c787da5279c983dc6284ed16258fa2ff069cc62b53f360f4f3fed5bb94f79d5f" + } + } +} diff --git a/docs/sops-rotation-approval.example.yaml b/docs/sops-rotation-approval.example.yaml index 806236d..3868a9d 100644 --- a/docs/sops-rotation-approval.example.yaml +++ b/docs/sops-rotation-approval.example.yaml @@ -1,9 +1,10 @@ # Metadata-only example. Copy outside Git for an attended approved rotation. approved: false -approved_by: "operator-name" +approved_by: "reviewer-name" approved_at: "2026-08-23T00:00:00Z" changes: - path: secrets/hetzner-token.yaml + sha256: "replace-with-exact-ciphertext-sha256-from-plan" before_recipients: - age1old-example-not-valid after_recipients: diff --git a/docs/sops-rotation.md b/docs/sops-rotation.md index 3c58af2..91943d0 100644 --- a/docs/sops-rotation.md +++ b/docs/sops-rotation.md @@ -8,6 +8,8 @@ python3 scripts/sops_rotation.py --check It compares each protected file's public age-recipient metadata with the first matching rule in `.sops.yaml`. CI runs this check to detect recipient drift. +The JSON output includes the exact file paths, ciphertext SHA-256 hashes, and +before/after recipient sets. Run without `--check` to inspect proposed drift. An attended non-printing decryption check may emit a receipt: @@ -21,7 +23,10 @@ receipt or command output. Actual key updates require `--apply` and an approval YAML containing `approved: true`, `approved_by`, `approved_at`, and an exact `changes` list from -the current plan. The command fails if that list differs from current metadata. +the current plan (including each changed file's `sha256`). The command fails if +that list differs from current metadata or the reviewed ciphertext has changed. +Applied receipts retain the reviewed before/after recipient sets and original +ciphertext hash, plus `after_sha256` for the resulting ciphertext. Review and preserve recovery-key custody before approving recipient removal. Start from `docs/sops-rotation-approval.example.yaml`; the committed example is deliberately unapproved and contains no usable recipient. diff --git a/scripts/sops_rotation.py b/scripts/sops_rotation.py index f8ad895..33e553e 100644 --- a/scripts/sops_rotation.py +++ b/scripts/sops_rotation.py @@ -117,6 +117,7 @@ def _load_approval(path: Path, plan: list[dict[str, Any]]) -> None: expected = [ { "path": item["path"], + "sha256": item["sha256"], "before_recipients": item["before_recipients"], "after_recipients": item["after_recipients"], } @@ -202,9 +203,13 @@ def main() -> int: raise RotationError("--apply requires at least one recipient change") _load_approval(args.approval_file, plan) _apply(plan) - plan = rotation_plan() - if any(item["changed"] for item in plan): + after_plan = rotation_plan() + if any(item["changed"] for item in after_plan): raise RotationError("recipient drift remains after rotation") + if [item["path"] for item in after_plan] != [item["path"] for item in plan]: + raise RotationError("protected file inventory changed during rotation") + for before, after in zip(plan, after_plan): + before["after_sha256"] = after["sha256"] verified = _verify_decryption(protected_files()) if args.verify_decryption or args.apply else False receipt = build_receipt(plan, verified, args.apply) if args.receipt: @@ -219,6 +224,7 @@ def main() -> int: "changes": sum(1 for item in plan if item["changed"]), "decryption_verified": verified, "applied": args.apply, + "plan": plan, }, sort_keys=True, ) diff --git a/tests/test_secret_and_receipt_contracts.py b/tests/test_secret_and_receipt_contracts.py index 7b9b0ad..280fd80 100644 --- a/tests/test_secret_and_receipt_contracts.py +++ b/tests/test_secret_and_receipt_contracts.py @@ -1,11 +1,17 @@ from __future__ import annotations import copy +import contextlib +import io import json import sys +import tempfile import unittest import uuid from pathlib import Path +from unittest.mock import patch + +import yaml ROOT = Path(__file__).resolve().parents[1] sys.path.insert(0, str(ROOT / "scripts")) @@ -13,6 +19,7 @@ sys.path.insert(0, str(ROOT / "scripts")) from check_secret_paths import is_encrypted_content, is_protected_path # noqa: E402 from s1_receipt import ReceiptError, load_receipt, validate_receipt # noqa: E402 from sops_rotation import rotation_plan # noqa: E402 +import sops_rotation # noqa: E402 class SecretAndReceiptContractTests(unittest.TestCase): @@ -74,6 +81,53 @@ class SecretAndReceiptContractTests(unittest.TestCase): self.assertTrue(plan) self.assertFalse(any(item["changed"] for item in plan)) + def test_rotation_approval_binds_ciphertext(self) -> None: + plan = [{"path": "secrets/example.yaml", "sha256": "a" * 64, + "before_recipients": ["age1before"], + "after_recipients": ["age1after"], "changed": True}] + approval = {"approved": True, "approved_by": "reviewer", + "approved_at": "2026-09-27T00:00:00Z", + "changes": [{k: v for k, v in plan[0].items() if k != "changed"}]} + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "approval.yaml" + path.write_text(yaml.safe_dump(approval)) + sops_rotation._load_approval(path, plan) + plan[0]["sha256"] = "b" * 64 + with self.assertRaises(sops_rotation.RotationError): + sops_rotation._load_approval(path, plan) + + def test_applied_rotation_keeps_reviewed_before_and_after(self) -> None: + before = [{"path": "secrets/example.yaml", "sha256": "a" * 64, + "before_recipients": ["age1before"], + "after_recipients": ["age1after"], "changed": True}] + after = [{"path": "secrets/example.yaml", "sha256": "b" * 64, + "before_recipients": ["age1after"], + "after_recipients": ["age1after"], "changed": False}] + with tempfile.TemporaryDirectory() as tmp: + receipt_path = Path(tmp) / "receipt.json" + with patch.object(sys, "argv", ["rotation", "--apply", "--approval-file", + "approval.yaml", "--receipt", str(receipt_path)]), \ + patch.object(sops_rotation, "rotation_plan", side_effect=[before, after]), \ + patch.object(sops_rotation, "_load_approval"), \ + patch.object(sops_rotation, "_apply"), \ + patch.object(sops_rotation, "_verify_decryption", return_value=True), \ + contextlib.redirect_stdout(io.StringIO()) as output: + self.assertEqual(sops_rotation.main(), 0) + receipt = json.loads(receipt_path.read_text()) + self.assertEqual(receipt["before_recipients"], ["age1before"]) + self.assertEqual(receipt["after_recipients"], ["age1after"]) + self.assertEqual(receipt["file_metadata"][0]["sha256"], "a" * 64) + self.assertEqual(receipt["file_metadata"][0]["after_sha256"], "b" * 64) + self.assertEqual(json.loads(output.getvalue())["changes"], 1) + + def test_default_rotation_output_contains_reviewable_plan(self) -> None: + with patch.object(sys, "argv", ["rotation"]), \ + contextlib.redirect_stdout(io.StringIO()) as output: + self.assertEqual(sops_rotation.main(), 0) + payload = json.loads(output.getvalue()) + self.assertEqual(payload["plan"], rotation_plan(ROOT)) + self.assertFalse(payload["decryption_verified"]) + if __name__ == "__main__": unittest.main() diff --git a/workplans/RAIL-HO-WP-0011-reproducible-s1-declaration-and-handoff.md b/workplans/RAIL-HO-WP-0011-reproducible-s1-declaration-and-handoff.md index 6ea4065..9dc5580 100644 --- a/workplans/RAIL-HO-WP-0011-reproducible-s1-declaration-and-handoff.md +++ b/workplans/RAIL-HO-WP-0011-reproducible-s1-declaration-and-handoff.md @@ -4,12 +4,12 @@ type: workplan title: "Make the S1 declaration reproducible and the handoff verifiably green" domain: financials repo: railiance-infra -status: active +status: blocked flavor: implementation owner: codex topic_slug: railiance created: "2026-08-23" -updated: "2026-08-23" +updated: "2026-09-27" related: - RAIL-HO-WP-0002 - RAIL-HO-WP-0009 @@ -313,3 +313,27 @@ Current evidence (2026-08-23): - Forgejo Actions run 79 is green for revision `4f2312a` across all three jobs. - Pending before finish: an attended fresh all-host handoff receipt and an attended non-printing SOPS decryption receipt. + +## Closeout review — 2026-09-27 + +T08 source gaps repaired: the default metadata-only output now includes exact +file paths, ciphertext hashes and before/after recipient sets; approval binds +the ciphertext hash; applied receipts preserve the reviewed original recipients +and hash alongside the resulting hash. Three regression tests cover these cases. +Unrelated Make targets no longer decrypt the provider token or read/export the +private age key during Makefile evaluation. No credential was rotated. + +T05 remains `wait`: a read-only Ansible 2.17.13 run reached both hosts with +`changed=0`. CoulombCore lacks `/usr/local/bin/goss`; Railiance01's installed +baseline checksum differs from the source-rendered profile. The next step is a +reviewed Goss refresh for each host, followed by remediation of any actual +baseline failures and a clean-revision all-host handoff. A failed surface check +is not green host evidence. See `docs/evidence/2026-09-27-loose-ends.md`. + +T08 remains `wait`: source tests and metadata checks pass, but this workstation +has no SOPS executable or approved decryption session for the repository's +recipient. The existing attended non-printing decryption receipt is still +required; mock tests do not substitute for it. The host having SOPS installed +does not establish approved recovery-key custody. + +Workplan is `blocked` until those live verification prerequisites are met. diff --git a/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md b/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md index 1646f3a..4967e13 100644 --- a/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md +++ b/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md @@ -4,12 +4,12 @@ type: workplan title: "Close the encrypted S1 backup and recovery loop" domain: financials repo: railiance-infra -status: active +status: blocked flavor: implementation owner: codex topic_slug: railiance created: "2026-08-23" -updated: "2026-08-23" +updated: "2026-09-27" related: - RAIL-HO-WP-0011 state_hub_workstream_id: "5ea28f8f-376c-5230-8bb7-ca871c1a75f4" @@ -132,7 +132,7 @@ no timer is installed merely by running a verification command. ```task id: RAIL-HO-WP-0012-T05 -status: progress +status: wait priority: high state_hub_task_id: "783dff8b-849e-5ed9-a668-af1184be7bdd" ``` @@ -216,3 +216,21 @@ decrypted configuration. - T05 is source-prepared and pending owner acceptance; T06 remains `wait`. No off-host write, retained-artifact deletion, private-key access, or live-host restore occurred. + +## Closeout review — 2026-09-27 + +T05 is `wait`, not ongoing implementation. The exact S1 acceptance file remains +`pending`; no owner decision accepts this envelope/projection. Route lookup still +reports unknown workload identity. Source upload/collision/redirect/retention +and isolated fixture recovery tests pass, but no approved live S1 transfer or +owner retrieval receipt exists. T06 consequently remains `wait` for that copy +and attended recovery-key custody and isolated restore. + +The older upload-credential incident is no longer a blocker: platform +`RPF-WP-0029` closed on 2026-09-15 with predecessor-share invalidation attestation +and replacement transport/application recovery evidence. That evidence concerns +the platform's archive, not this S1 bundle, and does not accept the S1 contract. +Historical recovery-key exposure remains separately recorded by the owner. + +Workplan is `blocked` on exact owner acceptance, controlled S1 transfer and an +attended S1 restore drill. No new task or workplan was created. diff --git a/workplans/RAIL-HO-WP-0013-host-time-baseline.md b/workplans/RAIL-HO-WP-0013-host-time-baseline.md index 4eafd8d..ade0c4e 100644 --- a/workplans/RAIL-HO-WP-0013-host-time-baseline.md +++ b/workplans/RAIL-HO-WP-0013-host-time-baseline.md @@ -4,12 +4,12 @@ type: workplan title: "Declare and verify the Railiance host UTC baseline" domain: financials repo: railiance-infra -status: active +status: blocked flavor: planning owner: codex topic_slug: railiance created: "2026-09-14" -updated: "2026-09-15" +updated: "2026-09-27" related_workplans: - RCLK-WP-0005 - RCLK-WP-0002 @@ -33,7 +33,7 @@ and tools/observe_host_clock.py. No configuration or clock change was made. ```task id: RAIL-HO-WP-0013-T01 -status: todo +status: wait priority: high state_hub_task_id: "9e5db140-0642-58de-8a9c-1c5a3cfd7b6e" ``` @@ -49,7 +49,7 @@ Define what source health can honestly claim before exposing it to the clock app ```task id: RAIL-HO-WP-0013-T02 -status: progress +status: wait priority: high state_hub_task_id: "75f17ffa-b781-549d-82ce-d19d431c2618" ``` @@ -81,7 +81,7 @@ A mocked systemctl result or container-only check is not host synchronization pr ```task id: RAIL-HO-WP-0013-T04 -status: progress +status: wait priority: high state_hub_task_id: "9adae3a6-8ee6-538b-9bb3-1ee32ed185e1" ``` @@ -111,3 +111,30 @@ railiance-platform docs/evidence/2026-09-15-railiance-clock-production.json. Health collection needs read-only adjtimex; ProtectClock is disabled only on that exporter, while both services retain empty capability sets. Disposable outage/reboot/rollback rehearsals remain tracked in T03. + +## Closeout review — 2026-09-27 + +All four residual tasks are `wait`; deployed source alone does not meet their +acceptance criteria. T01 is blocked on RCLK-WP-0002 source/leap/error-model +acceptance; that owner's T01–T03 are still in progress and T04 is todo. T02 has +an implemented opt-in role and passes native Ansible syntax checking, but still +requires T01's reviewed policy and baseline/Goss health integration. T03 requires +an admitted disposable Ubuntu VM for convergence/no-op, drift, reboot, +source-outage/recovery and rollback evidence. No such test target is declared in +this repository. T04's prior live deployment evidence stands, but its required +T03 recovery evidence and steady-state handoff remain outstanding. + +Refreshed read-only inventory: +`docs/evidence/2026-09-27-railiance01-clock-inventory.json`, collected with the +existing railiance-clock collector. Installed systemd and systemd-timesyncd: +`255.4-1ubuntu8.17`. Effective system/fallback source: `ntp.ubuntu.com`; +per-link and runtime source lists empty. Poll bounds 32–64 seconds, root-distance +threshold 500ms, observed leap 0. UTC, timesyncd active, synchronization reported, +no UDP/123 listener and no observed competing daemon. These are diagnostics, +not independent accuracy or source-independence proof. Configuration hashes are +in the receipt; no host configuration was changed. + +The September 14 duplicate-workplan inbox warning is historical: the current +checkout has one WP-0013 file and already contains consolidation commit +`c402245`. Preserve its existing task UUIDs. Workplan is `blocked` on the +remaining review and native recovery prerequisites.