railiance-platform/workplans/ADHOC-2026-11-03.md

id	type	title	domain	repo	status	flavor	owner	topic_slug	created	updated	related	state_hub_workstream_id
RPF-WP-ADHOC-2026-11-03	workplan	Ad hoc tasks 2026-11-03: MAS backup alerting, test-suite, admission-check and credential-test repair	financials	railiance-platform	finished	implementation	claude-code	coulomb-social	2026-11-03	2026-11-03	RPF-WP-0070	96b47a38-0d2e-595c-99e0-122625a80cad

Alert on failed or stale MAS evidence backups

id: RPF-WP-ADHOC-2026-11-03-T01
status: done
priority: medium
state_hub_task_id: "e82616f9-bf8e-50af-9ac7-580b57135d21"

RPF-WP-0070 closed with one residual: a failed MAS backup only showed up as a failed systemd unit. The fix copies the email-connect ledger pattern. OnFailure=mas-evidence-alert@%n.service on mas-evidence-backup.service and mas-evidence-freshness.service starts scripts/mas_evidence_notify.py. That script queues a notice with no values in a private directory (/var/lib/mas-evidence-backup/notifications, 0700), then posts it to the platform-root State Hub inbox at the in-cluster address 10.43.68.154:8000. Delivery counts only when the hub echoes back the exact sender, recipient, subject and body. A notice that cannot be sent stays pending, and the next successful freshness run replays it (ExecStartPost=-, which never fails the check). The subjects are MAS evidence backup failure, MAS evidence backup stale and MAS evidence notification test. Each body carries the unit, host, workplan and next steps, and a failure notice says not to scale or restart the facility and never to delete retained objects. While a backup stays stale, the hourly check repeats the notice once per hour.

Done when the bundle is installed on railiance01 and a test notice from mas-evidence-alert@mas-evidence-alert-test.service.service arrives in the platform-root inbox.

2026-11-03: committed as edd8687. The bundle is built and its hashes are fixed (install.sh SHA256 92432680…a2cb), but it is not yet installed on railiance01. No MAS evidence notification test message has reached the platform-root inbox. The next step is the attended install and test-notice command from the session.

2026-11-03 09:20Z: the founder ran install.sh (source edd8687) on railiance01. The installed notify script and the three unit files match the bundle hashes. The install's backup run passed (offsite_fetched_verified, version 1793697611356111, no retained object deleted, facility not scaled), and both timers remain enabled. The test unit succeeded, and the platform-root inbox holds MAS evidence notification test (message 8d77ed2c…caa4, 09:20:16Z) from railiance-platform. T01 is done. Receipt: docs/evidence/mas-evidence-alert-install-2026-11-03.json.

Make the platform test suite collect and finish

id: RPF-WP-ADHOC-2026-11-03-T02
status: done
priority: medium
state_hub_task_id: "f6115103-bd15-5b6e-bfee-73faff1d12ad"

The full suite did not finish locally. Two files failed at collection. test_digest_retention.py imported scripts without the repo root on sys.path, and test_verify_informed_decision_requester.py needs the informed_decision package from the informed-decision image. Run file by file (149 files), six test_task_execution_* files also errored. Some never put the sibling secrets-engine source on sys.path, so they only imported after an earlier test had loaded it. Fourteen proofs in test_task_execution_mediator.py and test_task_execution_trusted_whole.py need prebuilt fixture binaries and a pinned trusted review environment, which this checkout does not have. test_openbao_isolated_restore_native.py takes about 7 minutes; it failed once and passed on the re-run.

Fixes, tests only:

• test_digest_retention.py puts the repo root on sys.path.
• The informed-decision proof skips unless informed_decision is importable. With PYTHONPATH=~/informed-decision it still passes 3/3.
• tests/conftest.py adds SECRETS_ENGINE_SOURCE_ROOT/src (default ../secrets-engine) to sys.path, the same root the plan tests use.
• The mediator and trusted-whole proofs skip with a named reason when their binaries or environment are absent and no env var points at a build. If one is configured but incomplete, they still fail.

Result 2026-11-03: python3 -m pytest now runs to completion, with 1507 passed, 9 failed and 18 skipped in 17m58s. Plain pytest also collects. The 9 failures are not environment problems: they fail the same way with the new conftest disabled. They are assertions that have drifted from the source they check:

• test_credential_change: all repo CCRs validate; coding-agent JWT role; core-hub database lanes. This includes a principal_type that is now a list.
• test_eso_auth_recovery: a generated-policy comment check.
• test_factory_audit_delivery: three tests stop on reviewed_manifest_changed.
• test_flex_auth_signer_native_preparation: two status-text and scope assertions. These cover credential-governance rules, so they were not loosened here. Each needs an owner check of whether the source or the test is right.

Refresh the admission baseline after reviewed rapp-postgres changes

id: RPF-WP-ADHOC-2026-11-03-T03
status: done
priority: medium
state_hub_task_id: "31d34bd6-f4af-5ed9-8666-77f6483c6d9a"

make assurance-admission failed with invalid-or-unavailable-input. The report itself was unchanged: cells, ceilings, retention, limits and consumers were all equal. Only three rapp-postgres source digests had drifted from the 2026-09-15 baseline. They come from two reviewed commits:

• 208f8b6 (RAPP-POSTGRES-WP-0008) lowers CPU and memory requests on platform-pg and platform-pg-2. Disclosed limits are unchanged.
• 8d4106f (RAPP-POSTGRES-WP-0007) is a provisioner grant fix: NOSUPERUSER and runtime read on the conformance view. No capacity or retention logic. assurance/admission-baseline.json now carries the three new digests, and make assurance-admission passes.

Repair the three drifted credential-change tests

id: RPF-WP-ADHOC-2026-11-03-T04
status: done
priority: medium
state_hub_task_id: "aabbe169-82bf-55e7-a172-d12870b1c5e8"

Three of the T02 drift failures were in tests/test_credential_change.py. For each, the git history showed whether the source or the test was right:

• Core-hub database lanes: the test was stale. CCR-2026-0031 (d3377fa, applied on railiance01) deliberately added the repo-manager-forgejo-source ExternalSecret. The expected set now includes it, and the test asserts that it uses its own openbao-core-hub-forgejo-source store rather than the database lane.
• Coding-agent JWT role: the test was stale. RPF-WP-0065 (2126457) binds principal_type: [service, agent] while the NK-WP-0054 transition runs, and test_netkingdom_role_files.py already asserts this. The test now matches.
• All repo CCRs validate: the record was wrong. CCR-2026-0035 (Authelia SMTP, active) was written in a compact form that predates the full workload-kv-read schema. It is restated from the applied lane source and the 2026-10-06 owner receipt, with nothing new asserted:
  • openbao.auth comes from role_payload;
  • the new openbao/policies/authelia-smtp-eso.hcl is byte-equal to the lane.POLICY that scripts/authelia_smtp_delivery.py applied;
  • metadata_read: false and token_self_lifecycle: true make the generated policy match it;
  • the front door, risk, verification and lifecycle sections cite only the receipt fields. No live change. applier-dry-run still blocks this lane, as it does sibling CCR-2026-0056. Both are applied by attended owner scripts, not the delegated applier.

Result: test_credential_change.py passes in full. Across the 25 test files that touch policies or CCRs, 337 pass and 1 fails. That failure is test_eso_auth_recovery (a lane policy has 4 paths and the test expects 3). It was on the T02 list and fails the same way without this change.

Repair the drifted ESO recovery policy test

id: RPF-WP-ADHOC-2026-11-03-T05
status: done
priority: medium
state_hub_task_id: "c584e656-5aa1-5fe0-a3e7-f18bc6cfc01a"

test_eso_auth_recovery expected exactly three paths in each recovery lane's policy. workload-kv-read-reuse-surface-runtime-eso.hcl has a fourth: the exact data path platform/workloads/reuse/reuse-surface/forgejo-source-read. It was added by CCR-2026-0046, which is approved and was recorded as verified in 2951b69 (RPF-WP-0058). The test, written 2026-09-05, predates it. The source was right; the test was stale.

The test now allows one extra exact data path for each CCR that is approved, applied, verified or active and names the same policy_file. The bans on wildcards, metadata and list are unchanged, and the lane path and both token self paths are now asserted explicitly. A mutation check passes: with CCR-2026-0046 set back to proposed, the test fails.

The policy comment still says "CCR-2026-0046 proposed" and was deliberately left alone. repair_eso_kubernetes_auth.py requires the live policy to equal this file byte for byte (policy_drift), so editing the comment would make the next attended recovery refuse.

Repair the last five drifted tests

id: RPF-WP-ADHOC-2026-11-03-T06
status: done
priority: medium
state_hub_task_id: "ee8632ff-97f0-582c-b11a-458ec951a707"

These are the last five of the nine T02 drift failures. In each case the source was right, and two were more than test drift.

• test_factory_audit_delivery (3). The attended delivery tool itself was stale, not just the tests. 80e0539 (CUST-WP-0073, a security fix) added a metadata-only target.template to manifests/factory-audit-senders.yaml. scripts/factory_audit_delivery.py still pinned the old manifest digest and refused any template, so it would have refused a re-delivery. I reviewed the diff: it adds only Secret annotations. The manifest is re-pinned to a28aecfe…0cfe. The template rule is narrowed rather than dropped: dataFrom is still refused, and a template may only contain metadata, exactly as the reviewed manifest declares. A new test proves that data, type and dataFrom changes are still refused.
• test_flex_auth_signer_native_preparation (2). These tests froze the 2026-10-16 "prepared" state. Since then CCR-2026-0050 has recorded scoped owner reviews (RPF-WP-0035-T14). The founder and cluster-runtime-storage-owner have not approved, so the CCR stays proposed. The native plan's adapter state and storage scope changed in reviewed commits (a3e8eb6, 84c529f). The tests now check:
  • that no founder or storage-owner approval is recorded;
  • that the plan bytes still equal the reviewed digest 22306d50…;
  • the invariants that still hold: no native execution approval, mandatory gates remain, and Platform creates no storage. The plan still lists every review as "pending". I left that alone, because the plan is a pinned, reviewed artifact.

Result 2026-11-04: the full suite runs green, with 1547 passed, 18 skipped and 0 failed in 17m35s.

T04 and T05 were registered by reopening this plan briefly; C-11 does not create tasks in a finished workplan. RPF-WP-0064-T05's 422 needed an intervention_note, because needs_human requires one.