Fix rotation review evidence and reconcile blocked S1 workplans

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e3b9-b19e-7ba1-9eb4-4faea76af3ea

Makefile

@@ -16,7 +16,7 @@ IMG   ?= ubuntu-24.04
 USER  ?= admin
 
 # Decrypt Hetzner token at runtime (requires SOPS_AGE_KEY or keys.txt locally)
-HCLOUD_TOKEN := $(shell sops -d --extract '["hetzner"]["token"]' secrets/hetzner-token.yaml 2>/dev/null)
+HCLOUD_TOKEN = $(shell sops -d --extract '["hetzner"]["token"]' secrets/hetzner-token.yaml 2>/dev/null)
 
 # ---- Help ----
 help: ## Show this help
@@ -112,7 +112,7 @@ tf-destroy: ## Terraform destroy (exact approval required before init)
 
 # --- Terraform provider/lockfile helpers ---
 TF_DIR := terraform/hetzner
-TF_TOKEN := $(HCLOUD_TOKEN)
+TF_TOKEN = $(HCLOUD_TOKEN)
 LOCKFILE := $(TF_DIR)/.terraform.lock.hcl
 
 tf-lock-commit: ## Commit the current provider lockfile
@@ -287,8 +287,8 @@ PLAY := $(ANS_DIR)/playbooks/bootstrap.yaml
 SSH_USER ?=
 ANSIBLE_USER_FLAG := $(if $(SSH_USER),-u $(SSH_USER),)
 
-# Load your SOPS key for decryption when running playbooks (optional if you use keys.txt)
-export SOPS_AGE_KEY := $(shell cat ~/.config/sops/age/keys.txt 2>/dev/null)
+# SOPS reads its standard key file itself; preserve an explicitly supplied
+# SOPS_AGE_KEY without reading/exporting private keys for unrelated Make targets.
 
 ansible-help: ## Show common Ansible commands
 	@echo "Convergence targets:"

docs/evidence/2026-09-27-loose-ends.md

@@ -0,0 +1,54 @@
+# Existing workplan closeout review — 2026-09-27
+
+Reviewed all root and archived workplans. The only unfinished plans are
+RAIL-HO-WP-0011, 0012 and 0013; all now have state `blocked`, with eight
+remaining tasks in `wait`. No new task or workplan was opened. No residual task
+has been marked done without its required live acceptance evidence.
+
+## Implemented under WP-0011 T06/T08
+
+- Rotation dry-run output exposes the exact metadata-only review plan.
+- Approval binds each changed ciphertext's SHA-256 as well as recipients/path.
+- Applied receipts retain the original and resulting recipient/hash evidence.
+- Unrelated Make targets no longer eagerly decrypt the Hetzner token or read
+  and export the local age private key.
+
+Validation: 54 Python unit tests pass, including three new rotation regression
+tests. Inventory, baseline parity, read-only handoff contract, protected secret
+paths, SOPS recipient metadata and whitespace checks pass. Ansible-core 2.17.13
+host-time playbook syntax check passes in a disposable controller environment.
+No production decryption, recipient rotation or credential retrieval occurred.
+
+## Read-only host verification
+
+`ansible-playbook playbooks/verify.yaml`, Ansible-core 2.17.13, reached both hosts:
+
+| Host | Checks completed | Blocking assertion | Changes |
+| --- | --- | --- | --- |
+| CoulombCore | Executable and baseline stat | `/usr/local/bin/goss` absent | 0 |
+| Railiance01 | Executable and baseline stat | Installed baseline digest differs from source render | 0 |
+
+The initial sandboxed attempt failed writing Ansible's connection cache; the
+rerun with that access produced the host findings above. Neither attempt is a
+passing handoff. Host refresh includes installation/configuration and an hourly
+timer, so the concrete rendered diff must be reviewed before that separate
+mutation; subsequent baseline failures must also be resolved before T05 closes.
+
+## Backup dependency correction
+
+The exact S1 offsite contract remains pending:
+`d150eb3e6a19d658aa76c930b32fc20ef75ffa399558fc193fbde0738551ee62`.
+Warden's route reports unknown execution workload identity. Platform WP-0029's
+September 15 closure resolves the old upload-share incident, but does not accept
+this contract or prove S1 transfer/restore. Keep WP-0012 T05/T06 waiting for those
+specific owner and recovery receipts.
+
+## Clock dependency check
+
+The existing railiance-clock collector produced
+`2026-09-27-railiance01-clock-inventory.json`. Read-only `dpkg-query -W systemd
+systemd-timesyncd` returned `255.4-1ubuntu8.17` for both. Effective source/poll
+observations and configuration hashes are recorded in the receipt and WP-0013.
+RCLK-WP-0002 still lacks completed policy review; no disposable Ubuntu VM is
+declared here for reboot/outage/rollback testing. The already deployed authority
+does not close those gates. No clocks or services were changed.

docs/evidence/2026-09-27-railiance01-clock-inventory.json

@@ -0,0 +1,48 @@
+{
+  "schema_version": 1,
+  "host_alias": "railiance01",
+  "collector_reported_at": "2026-09-27T16:46:14.477816+00:00",
+  "read_only": true,
+  "authority_usable": null,
+  "synchronized_reported": true,
+  "limitations": [
+    "OS synchronization flag is not an independent UTC error bound",
+    "host and collector timestamps are observations, not authenticated time samples",
+    "SSH uses existing caller identity/trust; no sudo, installation or time mutation"
+  ],
+  "collector_sha256": "3aadb6fb888d87cefb63e94ac87c9c640ef63ef6f404b09a01e8e8642b26784e",
+  "remote_script_sha256": "597e08b95855c4bd1fee40ad8e7c1a8359312c1583ac8363e5bd922b493f074a",
+  "inventory": {
+    "remote_reported_at": "2026-09-27T16:46:16.708853+00:00",
+    "commands": {
+      "clock": {
+        "returncode": 0,
+        "stdout": "Timezone=Etc/UTC\nLocalRTC=no\nNTP=yes\nNTPSynchronized=yes\n"
+      },
+      "upstream": {
+        "returncode": 0,
+        "stdout": "LinkNTPServers=\nSystemNTPServers=ntp.ubuntu.com\nRuntimeNTPServers=\nFallbackNTPServers=ntp.ubuntu.com\nServerName=ntp.ubuntu.com\nServerAddress=2620:2d:4000:1::41\nRootDistanceMaxUSec=500ms\nPollIntervalMinUSec=32s\nPollIntervalMaxUSec=1min 4s\nPollIntervalUSec=1min 4s\nNTPMessage={ Leap=0, Version=4, Mode=4, Stratum=2, Precision=-25, RootDelay=6.454ms, RootDispersion=183us, Reference=1D586304, OriginateTimestamp=Sun 2026-09-27 16:45:18 UTC, ReceiveTimestamp=Sun 2026-09-27 16:45:18 UTC, TransmitTimestamp=Sun 2026-09-27 16:45:18 UTC, DestinationTimestamp=Sun 2026-09-27 16:45:18 UTC, Ignored=no, PacketCount=15904, Jitter=3.125ms }\nFrequency=1174453\n"
+      },
+      "timesyncd": {
+        "returncode": 0,
+        "stdout": "User=systemd-timesync\nActiveState=active\nFragmentPath=/usr/lib/systemd/system/systemd-timesyncd.service\nDropInPaths=\nUnitFileState=enabled\n"
+      },
+      "other_daemons": {
+        "returncode": 4,
+        "stdout": "inactive\ninactive\ninactive\n"
+      },
+      "virtualization": {
+        "returncode": 0,
+        "stdout": "kvm\n"
+      },
+      "ntp_listeners": {
+        "returncode": 0,
+        "stdout": ""
+      }
+    },
+    "config_sha256": {
+      "/etc/systemd/timesyncd.conf": "e6734751f8aaf19fddfff891ad246387f5f59bd9ff1a5f0cac2c34bc81941c62",
+      "/etc/systemd/timesyncd.conf.d/60-railiance-clock.conf": "c787da5279c983dc6284ed16258fa2ff069cc62b53f360f4f3fed5bb94f79d5f"
+    }
+  }
+}

docs/sops-rotation-approval.example.yaml

@@ -1,9 +1,10 @@
 # Metadata-only example. Copy outside Git for an attended approved rotation.
 approved: false
-approved_by: "operator-name"
+approved_by: "reviewer-name"
 approved_at: "2026-08-23T00:00:00Z"
 changes:
   - path: secrets/hetzner-token.yaml
+    sha256: "replace-with-exact-ciphertext-sha256-from-plan"
     before_recipients:
       - age1old-example-not-valid
     after_recipients:

docs/sops-rotation.md

@@ -8,6 +8,8 @@ python3 scripts/sops_rotation.py --check
 
 It compares each protected file's public age-recipient metadata with the first
 matching rule in `.sops.yaml`. CI runs this check to detect recipient drift.
+The JSON output includes the exact file paths, ciphertext SHA-256 hashes, and
+before/after recipient sets. Run without `--check` to inspect proposed drift.
 
 An attended non-printing decryption check may emit a receipt:
 
@@ -21,7 +23,10 @@ receipt or command output.
 
 Actual key updates require `--apply` and an approval YAML containing
 `approved: true`, `approved_by`, `approved_at`, and an exact `changes` list from
-the current plan. The command fails if that list differs from current metadata.
+the current plan (including each changed file's `sha256`). The command fails if
+that list differs from current metadata or the reviewed ciphertext has changed.
+Applied receipts retain the reviewed before/after recipient sets and original
+ciphertext hash, plus `after_sha256` for the resulting ciphertext.
 Review and preserve recovery-key custody before approving recipient removal.
 Start from `docs/sops-rotation-approval.example.yaml`; the committed example is
 deliberately unapproved and contains no usable recipient.

scripts/sops_rotation.py

@@ -117,6 +117,7 @@ def _load_approval(path: Path, plan: list[dict[str, Any]]) -> None:
     expected = [
         {
             "path": item["path"],
+            "sha256": item["sha256"],
             "before_recipients": item["before_recipients"],
             "after_recipients": item["after_recipients"],
         }
@@ -202,9 +203,13 @@ def main() -> int:
                 raise RotationError("--apply requires at least one recipient change")
             _load_approval(args.approval_file, plan)
             _apply(plan)
-            plan = rotation_plan()
-            if any(item["changed"] for item in plan):
+            after_plan = rotation_plan()
+            if any(item["changed"] for item in after_plan):
                 raise RotationError("recipient drift remains after rotation")
+            if [item["path"] for item in after_plan] != [item["path"] for item in plan]:
+                raise RotationError("protected file inventory changed during rotation")
+            for before, after in zip(plan, after_plan):
+                before["after_sha256"] = after["sha256"]
         verified = _verify_decryption(protected_files()) if args.verify_decryption or args.apply else False
         receipt = build_receipt(plan, verified, args.apply)
         if args.receipt:
@@ -219,6 +224,7 @@ def main() -> int:
                     "changes": sum(1 for item in plan if item["changed"]),
                     "decryption_verified": verified,
                     "applied": args.apply,
+                    "plan": plan,
                 },
                 sort_keys=True,
             )

tests/test_secret_and_receipt_contracts.py

@@ -1,11 +1,17 @@
 from __future__ import annotations
 
 import copy
+import contextlib
+import io
 import json
 import sys
+import tempfile
 import unittest
 import uuid
 from pathlib import Path
+from unittest.mock import patch
+
+import yaml
 
 ROOT = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(ROOT / "scripts"))
@@ -13,6 +19,7 @@ sys.path.insert(0, str(ROOT / "scripts"))
 from check_secret_paths import is_encrypted_content, is_protected_path  # noqa: E402
 from s1_receipt import ReceiptError, load_receipt, validate_receipt  # noqa: E402
 from sops_rotation import rotation_plan  # noqa: E402
+import sops_rotation  # noqa: E402
 
 
 class SecretAndReceiptContractTests(unittest.TestCase):
@@ -74,6 +81,53 @@ class SecretAndReceiptContractTests(unittest.TestCase):
         self.assertTrue(plan)
         self.assertFalse(any(item["changed"] for item in plan))
 
+    def test_rotation_approval_binds_ciphertext(self) -> None:
+        plan = [{"path": "secrets/example.yaml", "sha256": "a" * 64,
+                 "before_recipients": ["age1before"],
+                 "after_recipients": ["age1after"], "changed": True}]
+        approval = {"approved": True, "approved_by": "reviewer",
+                    "approved_at": "2026-09-27T00:00:00Z",
+                    "changes": [{k: v for k, v in plan[0].items() if k != "changed"}]}
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "approval.yaml"
+            path.write_text(yaml.safe_dump(approval))
+            sops_rotation._load_approval(path, plan)
+            plan[0]["sha256"] = "b" * 64
+            with self.assertRaises(sops_rotation.RotationError):
+                sops_rotation._load_approval(path, plan)
+
+    def test_applied_rotation_keeps_reviewed_before_and_after(self) -> None:
+        before = [{"path": "secrets/example.yaml", "sha256": "a" * 64,
+                   "before_recipients": ["age1before"],
+                   "after_recipients": ["age1after"], "changed": True}]
+        after = [{"path": "secrets/example.yaml", "sha256": "b" * 64,
+                  "before_recipients": ["age1after"],
+                  "after_recipients": ["age1after"], "changed": False}]
+        with tempfile.TemporaryDirectory() as tmp:
+            receipt_path = Path(tmp) / "receipt.json"
+            with patch.object(sys, "argv", ["rotation", "--apply", "--approval-file",
+                                           "approval.yaml", "--receipt", str(receipt_path)]), \
+                 patch.object(sops_rotation, "rotation_plan", side_effect=[before, after]), \
+                 patch.object(sops_rotation, "_load_approval"), \
+                 patch.object(sops_rotation, "_apply"), \
+                 patch.object(sops_rotation, "_verify_decryption", return_value=True), \
+                 contextlib.redirect_stdout(io.StringIO()) as output:
+                self.assertEqual(sops_rotation.main(), 0)
+            receipt = json.loads(receipt_path.read_text())
+            self.assertEqual(receipt["before_recipients"], ["age1before"])
+            self.assertEqual(receipt["after_recipients"], ["age1after"])
+            self.assertEqual(receipt["file_metadata"][0]["sha256"], "a" * 64)
+            self.assertEqual(receipt["file_metadata"][0]["after_sha256"], "b" * 64)
+            self.assertEqual(json.loads(output.getvalue())["changes"], 1)
+
+    def test_default_rotation_output_contains_reviewable_plan(self) -> None:
+        with patch.object(sys, "argv", ["rotation"]), \
+             contextlib.redirect_stdout(io.StringIO()) as output:
+            self.assertEqual(sops_rotation.main(), 0)
+        payload = json.loads(output.getvalue())
+        self.assertEqual(payload["plan"], rotation_plan(ROOT))
+        self.assertFalse(payload["decryption_verified"])
+
 
 if __name__ == "__main__":
     unittest.main()

workplans/RAIL-HO-WP-0011-reproducible-s1-declaration-and-handoff.md

@@ -4,12 +4,12 @@ type: workplan
 title: "Make the S1 declaration reproducible and the handoff verifiably green"
 domain: financials
 repo: railiance-infra
-status: active
+status: blocked
 flavor: implementation
 owner: codex
 topic_slug: railiance
 created: "2026-08-23"
-updated: "2026-08-23"
+updated: "2026-09-27"
 related:
   - RAIL-HO-WP-0002
   - RAIL-HO-WP-0009
@@ -313,3 +313,27 @@ Current evidence (2026-08-23):
 - Forgejo Actions run 79 is green for revision `4f2312a` across all three jobs.
 - Pending before finish: an attended fresh all-host handoff receipt and an
   attended non-printing SOPS decryption receipt.
+
+## Closeout review — 2026-09-27
+
+T08 source gaps repaired: the default metadata-only output now includes exact
+file paths, ciphertext hashes and before/after recipient sets; approval binds
+the ciphertext hash; applied receipts preserve the reviewed original recipients
+and hash alongside the resulting hash. Three regression tests cover these cases.
+Unrelated Make targets no longer decrypt the provider token or read/export the
+private age key during Makefile evaluation. No credential was rotated.
+
+T05 remains `wait`: a read-only Ansible 2.17.13 run reached both hosts with
+`changed=0`. CoulombCore lacks `/usr/local/bin/goss`; Railiance01's installed
+baseline checksum differs from the source-rendered profile. The next step is a
+reviewed Goss refresh for each host, followed by remediation of any actual
+baseline failures and a clean-revision all-host handoff. A failed surface check
+is not green host evidence. See `docs/evidence/2026-09-27-loose-ends.md`.
+
+T08 remains `wait`: source tests and metadata checks pass, but this workstation
+has no SOPS executable or approved decryption session for the repository's
+recipient. The existing attended non-printing decryption receipt is still
+required; mock tests do not substitute for it. The host having SOPS installed
+does not establish approved recovery-key custody.
+
+Workplan is `blocked` until those live verification prerequisites are met.

workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md

@@ -4,12 +4,12 @@ type: workplan
 title: "Close the encrypted S1 backup and recovery loop"
 domain: financials
 repo: railiance-infra
-status: active
+status: blocked
 flavor: implementation
 owner: codex
 topic_slug: railiance
 created: "2026-08-23"
-updated: "2026-08-23"
+updated: "2026-09-27"
 related:
   - RAIL-HO-WP-0011
 state_hub_workstream_id: "5ea28f8f-376c-5230-8bb7-ca871c1a75f4"
@@ -132,7 +132,7 @@ no timer is installed merely by running a verification command.
 
 ```task
 id: RAIL-HO-WP-0012-T05
-status: progress
+status: wait
 priority: high
 state_hub_task_id: "783dff8b-849e-5ed9-a668-af1184be7bdd"
 ```
@@ -216,3 +216,21 @@ decrypted configuration.
 - T05 is source-prepared and pending owner acceptance; T06 remains `wait`. No
   off-host write, retained-artifact deletion, private-key access, or live-host
   restore occurred.
+
+## Closeout review — 2026-09-27
+
+T05 is `wait`, not ongoing implementation. The exact S1 acceptance file remains
+`pending`; no owner decision accepts this envelope/projection. Route lookup still
+reports unknown workload identity. Source upload/collision/redirect/retention
+and isolated fixture recovery tests pass, but no approved live S1 transfer or
+owner retrieval receipt exists. T06 consequently remains `wait` for that copy
+and attended recovery-key custody and isolated restore.
+
+The older upload-credential incident is no longer a blocker: platform
+`RPF-WP-0029` closed on 2026-09-15 with predecessor-share invalidation attestation
+and replacement transport/application recovery evidence. That evidence concerns
+the platform's archive, not this S1 bundle, and does not accept the S1 contract.
+Historical recovery-key exposure remains separately recorded by the owner.
+
+Workplan is `blocked` on exact owner acceptance, controlled S1 transfer and an
+attended S1 restore drill. No new task or workplan was created.

workplans/RAIL-HO-WP-0013-host-time-baseline.md

@@ -4,12 +4,12 @@ type: workplan
 title: "Declare and verify the Railiance host UTC baseline"
 domain: financials
 repo: railiance-infra
-status: active
+status: blocked
 flavor: planning
 owner: codex
 topic_slug: railiance
 created: "2026-09-14"
-updated: "2026-09-15"
+updated: "2026-09-27"
 related_workplans:
   - RCLK-WP-0005
   - RCLK-WP-0002
@@ -33,7 +33,7 @@ and tools/observe_host_clock.py. No configuration or clock change was made.
 
 ```task
 id: RAIL-HO-WP-0013-T01
-status: todo
+status: wait
 priority: high
 state_hub_task_id: "9e5db140-0642-58de-8a9c-1c5a3cfd7b6e"
 ```
@@ -49,7 +49,7 @@ Define what source health can honestly claim before exposing it to the clock app
 
 ```task
 id: RAIL-HO-WP-0013-T02
-status: progress
+status: wait
 priority: high
 state_hub_task_id: "75f17ffa-b781-549d-82ce-d19d431c2618"
 ```
@@ -81,7 +81,7 @@ A mocked systemctl result or container-only check is not host synchronization pr
 
 ```task
 id: RAIL-HO-WP-0013-T04
-status: progress
+status: wait
 priority: high
 state_hub_task_id: "9adae3a6-8ee6-538b-9bb3-1ee32ed185e1"
 ```
@@ -111,3 +111,30 @@ railiance-platform docs/evidence/2026-09-15-railiance-clock-production.json.
 Health collection needs read-only adjtimex; ProtectClock is disabled only on
 that exporter, while both services retain empty capability sets. Disposable
 outage/reboot/rollback rehearsals remain tracked in T03.
+
+## Closeout review — 2026-09-27
+
+All four residual tasks are `wait`; deployed source alone does not meet their
+acceptance criteria. T01 is blocked on RCLK-WP-0002 source/leap/error-model
+acceptance; that owner's T01–T03 are still in progress and T04 is todo. T02 has
+an implemented opt-in role and passes native Ansible syntax checking, but still
+requires T01's reviewed policy and baseline/Goss health integration. T03 requires
+an admitted disposable Ubuntu VM for convergence/no-op, drift, reboot,
+source-outage/recovery and rollback evidence. No such test target is declared in
+this repository. T04's prior live deployment evidence stands, but its required
+T03 recovery evidence and steady-state handoff remain outstanding.
+
+Refreshed read-only inventory:
+`docs/evidence/2026-09-27-railiance01-clock-inventory.json`, collected with the
+existing railiance-clock collector. Installed systemd and systemd-timesyncd:
+`255.4-1ubuntu8.17`. Effective system/fallback source: `ntp.ubuntu.com`;
+per-link and runtime source lists empty. Poll bounds 32–64 seconds, root-distance
+threshold 500ms, observed leap 0. UTC, timesyncd active, synchronization reported,
+no UDP/123 listener and no observed competing daemon. These are diagnostics,
+not independent accuracy or source-independence proof. Configuration hashes are
+in the receipt; no host configuration was changed.
+
+The September 14 duplicate-workplan inbox warning is historical: the current
+checkout has one WP-0013 file and already contains consolidation commit
+`c402245`. Preserve its existing task UUIDs. Workplan is `blocked` on the
+remaining review and native recovery prerequisites.