ops-warden/workplans/WARDEN-WP-0027-credential-governance-lockdown.md

id	type	title	domain	repo	status	owner	topic_slug	planning_priority	planning_order	created	updated	state_hub_workstream_id
WARDEN-WP-0027	workplan	Tamper-resistant credential governance + mass rotation/lockdown (Strand B)	infotech	ops-warden	active	codex	custodian	medium	27	2026-07-16	2026-08-23	21528e8d-a049-523d-9ae1-da7a27cb8bbf

Tamper-resistant credential governance + mass rotation/lockdown (Strand B)

Origin

Explicitly deferred from WARDEN-WP-0026 (credential disclosure hygiene, Strand A). WP-0026 "Out of scope" carves out the heavyweight governance machinery and directs that it be "capture[d] separately if/when justified." This workplan is that capture.

Strand A makes accidental disclosure structurally hard and gives every lane advisory rotation guidance. Strand B is the executable, tamper-resistant layer: turning that advisory guidance into one-command action and hardening policy governance against silent drift or malicious change.

Status: active, narrowly on T02

Activated by the operator on 2026-08-22 after the second activation gate became concrete. RAILIANCE-WP-0024-T03 now requires an approved recovery window, fresh encrypted Raft snapshot evidence, provider-console access, named abort authority, and the live 2-of-3 Shamir quorum before a coordinated reboot. That is fleet policy requiring a designed break-glass path, not speculative heavyweight machinery.

Activation is deliberately narrow. No mass-disclosure incident triggers T01, and no fleet policy currently mandates T03's signed policy-manifest reconcile. Those tasks remain cancel; cancellation here continues to mean deferred, not abandoned. T02 alone is progress.

Activation gate (promote to ready only when ≥1 holds)

• A real disclosure incident requires rotating more than a couple of secrets at once, making manual per-lane rotation (Strand A) too slow.
• Fleet policy for OpenBao mandates tamper-evident policy governance or a designed break-glass path.
• Audit/compliance requires provable "reconcile policy to a known-good commit."

Goal

Make credential response executable and trustworthy at fleet scale: one command rotates or re-establishes a set of tainted secrets; policy changes are tamper-evident and reconcilable to a known-good state; a graded lockdown / break-glass path exists with an explicit trust-root.

Scope boundary (unchanged from Strand A)

ops-warden still custodies no secret values. Strand B orchestrates the owner's tools (OpenBao, provider re-mint, railiance-platform credential broker) as the caller — it does not hold or vend secrets. Rotation execution runs the owner-native path; ops-warden sequences and verifies it.

Task: Executable mass rotation driver

id: WARDEN-WP-0027-T01
status: cancel
priority: high
state_hub_task_id: "b5691939-9d84-5115-9618-0f8839010d14"

Turn Strand A's per-lane rotation: guidance (WP-0026 T06) into an executable driver: warden rotate <id> (single lane) and warden rotate --tainted (all lanes marked EXPOSED per WP-0026 T05). Each step runs the owner-native command (provider re-mint, bao kv put, re-encrypt+reupload for re-establish lanes) as the caller, verifies via capabilities (never reads the value), and clears the exposed_at taint on success. Dry-run first; idempotent; per-lane failure isolates.

Done when: a tainted set of lanes can be rotated/re-established with one command, each verified capabilities-safe, taint cleared only on success.

Depends on: WP-0026 T05 (taint convention), T06 (rotation registry).

Task: Graded lockdown / break-glass with explicit trust-root

id: WARDEN-WP-0027-T02
status: progress
priority: medium
state_hub_task_id: "cae498ee-6307-5d32-9f1b-a471cfcc2536"

Design and document a graded lockdown: (a) soft — deny agent roles read on all high-risk lanes; (b) hard — seal + re-key with a pre-designed trust-root and quorum. Define the trust-root (who holds unseal shares, recovery keys), the break-glass invocation, and the re-entry path. Design + runbook first; any executable seal step is opt-in and attended.

Done when: a documented, rehearsed break-glass path exists with a named trust-root and quorum, and soft-lockdown is executable via capabilities-based policy toggles.

Parked input — the warden-sign AppRole (2026-08-11). secrets-engine holds a validated dry-run for a warden-sign AppRole (policy + role, update on ssh/sign/{agt,adm,atm}-role only). It was parked rather than withdrawn specifically because it is an input to this task: today the credential broker is the only autonomous path to a signing token, so if its issuer lapses, recovery is a founder OIDC act. Whether a second autonomous path should exist — and whether it is the AppRole or something this trust-root design supersedes — is a break-glass question, not a routing one. Resolve it here. Context and the counter-argument (AppRole is narrower in capability than placing the broker's issuer token on a remote host) are in workplans/ADHOC-2026-08-11.md T03 (ad-hoc finished 2026-08-15; question lives here now); secrets-engine was told to stop holding apply readiness until this task or the ops-bridge cutover fires (msg 863c7b57).

Rebaselined 2026-08-22. The July description predates the authoritative OpenBao migration. The current barrier is a rotated 2-of-3, not the aspirational 3-of-5 value still present in ops-warden's legacy posture descriptor. RMASTER-WP-0020 records two attended restart/unseal cycles on 2026-08-03 with shares supplied through hidden prompts, matching inventory, audit continuity, exact-path read, and sibling denial. That is valid rehearsal evidence for re-entry, but it is not misreported as a current production emergency-seal drill.

The soft half is also already live: RAILIANCE-WP-0022 reports total concrete high-risk path coverage and a dedicated coding-agent AppRole proving deny-wins without reading a secret value. ops-warden consumes that owner control through scripts/check_agent_read_boundary.py; it does not take ownership of the live OpenBao policy.

The consumer contract and remaining acceptance gate are now documented in docs/credential-governance-break-glass.md. One attended production emergency seal/unseal drill remains. It must use railiance-platform's evidence template and validator, have snapshot/quorum/console/driver/abort receipts before the seal hold point, and be executed by the platform owner—not by a coding agent.

Parked AppRole disposition: do not unpark the standalone warden-sign AppRole for T02 break-glass. The normal broker remains autonomous; an issuer failure deliberately crosses into attended OIDC and, below that, the 2-of-3 platform trust-root. A standing AppRole placed outside the broker would weaken that boundary and still would not recover a sealed OpenBao. A separately approved ops-bridge unattended-signing design may re-evaluate the narrow AppRole under its own workplan; that is service access, not recovery authority.

Read-only readiness 2026-08-22. The owner node-reboot preflight passed every automated check without observing a secret value: node/k3s, platform-pg and continuous archiving, a 16.37-hour successful backup, initialized/unsealed OpenBao with shares=3 and threshold=2, ESO stores/projections, and the reviewed audit-core digest were healthy. ready_for_live_execution correctly remained false because no approved window, current encrypted off-host snapshot receipt, provider-console attestation, quorum attestation, owner-ack flag, or named abort operator was supplied.

That preflight also exposed a Warden safety defect: the ceremony request matched the generic openbao-api-key template and proposed paste-once provisioning. openbao-shamir-recovery-ceremony now distinguishes the owner-operated approval ceremony from secret retrieval. It is a non-value-bearing pointer and can never offer --fetch, --exec, --out, --wrap, or paste-once share transport.

Scenario established 2026-08-22. The operator delegated preparation to the agent and reserved their involvement for the final GO/NO-GO. The bounded, non-secret scenario is docs/evidence/WARDEN-WP-0027-T02-drill-scenario-2026-08-22.md: ops-warden coordinates and enforces the hold point; railiance-platform is the proposed OpenBao driver; railiance-infra is the proposed independent console/abort owner; and railiance-master is asked only for a metadata-only 2-of-3 availability attestation. All three owner receipts and a green fully parameterized preflight are required before the final operator question.

The railiance-infra role is scoped by the authoritative review contract WARDEN-WP-0027-T02-DRILL-20260822-01-INFRA in interfaces/reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-railiance-infra.json. It binds the exact scenario artifact and expiry, requires explicit independent provider-console and distinct-abort attestations, permits metadata-only evidence, and states authorizes_execution: false.

Preparation approved 2026-08-22. Decision 9da57559-712a-4521-b46e-a4c69729f9d2 authorizes preparation only and narrows the possible live scope to one intentional OpenBao seal followed by the existing 2-of-3 unseal ceremony. It excludes a host reboot, re-key, restore, policy change, PVC mutation, credential disclosure, and general workload restart. Railiance-infra satisfied its direct contract with receipt 01a02b4b-7295-7836-b288-f29407008524. The revisioned, non-authorizing gate matrix is in docs/evidence/WARDEN-WP-0027-T02-drill-preparation-checklist-2026-08-22.md; platform snapshot/driver and master quorum contracts remain required before the fully parameterized read-only preflight.

Live attempt terminal NO-GO 2026-08-23. All owner receipts and the final read-only preflight became green, and the operator opened the exact bounded window with decision 449a697a-5303-4582-aa9e-b0bc8b35ab2d. The platform driver stopped before seal when attended OIDC succeeded but the read-only agent home prevented token-helper persistence and the underlying client emitted the short-lived credential into captured output despite -no-print. The credential was immediately self-revoked. Independent metadata confirmed OpenBao remained initialized and unsealed with zero unseal progress; no seal, unseal, reboot, restore, policy, PVC, or workload mutation occurred. NO-GO decision 85724c0c-e70f-4e2f-a8c6-a9cb1ea6331b supersedes and consumes the GO; the scenario and its receipts cannot be reused.

The direct remediation interface is RAILIANCE-WP-0026-T01. Ops-warden commit 0fae0904ce8d8694338dd53a8a79abec5fec788d replaces the persistent login-only handoff with a contained login-plus-reviewed-command envelope. It proves a private writable helper before OIDC, captures login/child/revocation stdout and stderr, fails closed on any output or persistence defect, self-revokes, and removes the helper. No live OIDC was used for verification. The value-safe owner receipt is docs/evidence/RAILIANCE-WP-0026-T01-ops-warden-receipt.json; a future drill requires owner acceptance, fresh receipts, a new scenario id, and a new human decision.

Containment review status 2026-08-23. railiance-infra independently accepted the exact implementation revision at its commit 186b030 and State Hub message 7b2846dd-4b7c-4933-b61f-921e8ccf9ff2, after rerunning the 42 focused tests and lint. Direct platform review request 9a5f8973-0e74-4df2-8089-5a5bdc42295b now carries the exact revision, receipt, test results, and infra acceptance. T02 remains progress until railiance-platform accepts or requests changes. No human GO is relevant while that owner gate is open; acceptance will permit preparation of a new scenario, not execution or reuse of the terminal one.

Owner gate CLOSED — accepted 2026-08-23, found 2026-08-28. railiance-platform accepted the exact revision 0fae0904. Recorded in their RPF-WP-0017-attended-login-output-containment.md (status: finished), which also records railiance-infra's independent approval at 186b030 and marks all five acceptance criteria met. The receipt digest was verified here rather than taken on trust: docs/evidence/RAILIANCE-WP-0026-T01-ops-warden-receipt.json at ops-warden bc1966da hashes to d2ba444ed16989590325697e69d25283dc75a9432c29a72e627e80bf9fd987e4, matching their record exactly.

Note the identifier: the remediation interface recorded above as RAILIANCE-WP-0026-T01 is RPF-WP-0017-T01 in the owner's repo. Searching for the cited id finds an unrelated workplan, which is part of why this sat unnoticed.

Their acceptance is source acceptance only and authorizes no live OIDC or drill — their words, and the boundary holds. So T02's state changes but its Done when does not: what was blocked was preparing a new scenario, and that is now permitted. The terminal NO-GO scenario and its receipts remain unusable.

Third instance of the same failure this session. The acceptance existed for five days; no message reached ops-warden. Identically, key-cape accepted the WP-0033-T04 question on 2026-08-23 with no message, and nine unread messages were sitting on already-superseded threads. The verified: field added by WP-0033-T05 was built for exactly this and it works — what does not work is waiting for a counterparty to tell you. Re-checking a blocker means reading the owner's repository.

Remaining to close T02: one attended production emergency seal/unseal drill, requiring a new scenario id, fresh owner receipts from platform/infra/master, a fully parameterized green preflight, and a new human GO. It is executed by the platform owner, never by a coding agent. That is an operator decision, not an agent one, so T02 stays progress and the decision is surfaced rather than taken.

Task: Tamper-evident policy governance + reconcile

id: WARDEN-WP-0027-T03
status: cancel
priority: medium
state_hub_task_id: "7dbedcdc-dd5a-551c-bff1-0702fea0a9cf"

Policy-as-code for OpenBao policies/roles with tamper-evidence: a signed/hashed manifest of the intended policy set, drift detection against live OpenBao, and a reconcile-to-known-good-commit path. (Drift detection may land independently even if full reconcile stays gated.) Coordinate ownership with railiance-platform (OpenBao deployment) — ops-warden contributes the ops-access policy surface, not the cluster.

Done when: live OpenBao policy state is diffable against a signed known-good manifest, and an attended reconcile can restore it.

Cross-repo: railiance-platform (OpenBao cluster + policy custody).

References

• WARDEN-WP-0026 — Strand A (disclosure hygiene, advisory rotation) — prerequisite
• .claude/rules/credential-routing.md — ops-warden custodies no secret values
• railiance-platform — OpenBao cluster, policy custody, credential broker
• wiki/AccessRouting.md — issue vs route vs assist boundary
• workplans/ADHOC-2026-08-11.md T03 — parked warden-sign AppRole (ad-hoc finished 2026-08-15); T02 resolves it as unsuitable for break-glass, while a separately approved ops-bridge unattended-signing design may still re-evaluate it cert_command cutover
• secrets-engine SECRETS-WP-0004 — the parked AppRole apply/handoff