ops-warden/workplans/WARDEN-WP-0027-credential-governance-lockdown.md

---
id: WARDEN-WP-0027
type: workplan
title: "Tamper-resistant credential governance + mass rotation/lockdown (Strand B)"
domain: infotech
repo: ops-warden
status: active
owner: codex
topic_slug: custodian
planning_priority: medium
planning_order: 27
created: "2026-07-16"
updated: "2026-08-23"
state_hub_workstream_id: "21528e8d-a049-523d-9ae1-da7a27cb8bbf"
---

# Tamper-resistant credential governance + mass rotation/lockdown (Strand B)

## Origin

Explicitly deferred from `WARDEN-WP-0026` (credential disclosure hygiene, Strand A).
WP-0026 "Out of scope" carves out the heavyweight governance machinery and directs
that it be "capture[d] separately if/when justified." This workplan is that capture.

Strand A makes accidental disclosure structurally hard and gives every lane
**advisory** rotation guidance. Strand B is the **executable, tamper-resistant**
layer: turning that advisory guidance into one-command action and hardening policy
governance against silent drift or malicious change.

## Status: active, narrowly on T02

Activated by the operator on 2026-08-22 after the second activation gate became
concrete. `RAILIANCE-WP-0024-T03` now requires an approved recovery window,
fresh encrypted Raft snapshot evidence, provider-console access, named abort
authority, and the live **2-of-3 Shamir quorum** before a coordinated reboot.
That is fleet policy requiring a designed break-glass path, not speculative
heavyweight machinery.

Activation is deliberately narrow. No mass-disclosure incident triggers T01,
and no fleet policy currently mandates T03's signed policy-manifest reconcile.
Those tasks remain `cancel`; cancellation here continues to mean deferred, not
abandoned. T02 alone is `progress`.

## Activation gate (promote to `ready` only when ≥1 holds)

- A real disclosure incident requires rotating **more than a couple** of secrets at
  once, making manual per-lane rotation (Strand A) too slow.
- Fleet policy for OpenBao mandates tamper-evident policy governance or a designed
  break-glass path.
- Audit/compliance requires provable "reconcile policy to a known-good commit."

## Goal

Make credential response **executable and trustworthy at fleet scale**: one command
rotates or re-establishes a set of tainted secrets; policy changes are tamper-evident
and reconcilable to a known-good state; a graded lockdown / break-glass path exists
with an explicit trust-root.

## Scope boundary (unchanged from Strand A)

ops-warden still **custodies no secret values**. Strand B orchestrates the owner's
tools (OpenBao, provider re-mint, railiance-platform credential broker) as the
caller — it does not hold or vend secrets. Rotation execution runs the owner-native
path; ops-warden sequences and verifies it.

## Task: Executable mass rotation driver

```task
id: WARDEN-WP-0027-T01
status: cancel
priority: high
state_hub_task_id: "b5691939-9d84-5115-9618-0f8839010d14"
```

Turn Strand A's per-lane `rotation:` guidance (WP-0026 T06) into an executable
driver: `warden rotate <id>` (single lane) and `warden rotate --tainted` (all lanes
marked EXPOSED per WP-0026 T05). Each step runs the owner-native command
(provider re-mint, `bao kv put`, re-encrypt+reupload for re-establish lanes) as the
caller, verifies via capabilities (never reads the value), and clears the
`exposed_at` taint on success. Dry-run first; idempotent; per-lane failure isolates.

**Done when:** a tainted set of lanes can be rotated/re-established with one command,
each verified capabilities-safe, taint cleared only on success.

**Depends on:** WP-0026 T05 (taint convention), T06 (rotation registry).

## Task: Graded lockdown / break-glass with explicit trust-root

```task
id: WARDEN-WP-0027-T02
status: progress
priority: medium
state_hub_task_id: "cae498ee-6307-5d32-9f1b-a471cfcc2536"
```

Design and document a graded lockdown: (a) soft — deny agent roles read on all
high-risk lanes; (b) hard — seal + re-key with a pre-designed trust-root and quorum.
Define the trust-root (who holds unseal shares, recovery keys), the break-glass
invocation, and the re-entry path. **Design + runbook first**; any executable seal
step is opt-in and attended.

**Done when:** a documented, rehearsed break-glass path exists with a named
trust-root and quorum, and soft-lockdown is executable via capabilities-based
policy toggles.

**Parked input — the warden-sign AppRole (2026-08-11).** secrets-engine holds a
validated dry-run for a `warden-sign` AppRole (policy + role, `update` on
`ssh/sign/{agt,adm,atm}-role` only). It was **parked** rather than withdrawn
specifically because it is an input to this task: today the credential broker is the
only *autonomous* path to a signing token, so if its issuer lapses, recovery is a
founder OIDC act. Whether a second autonomous path should exist — and whether it is
the AppRole or something this trust-root design supersedes — is a break-glass
question, not a routing one. Resolve it here. Context and the counter-argument
(AppRole is narrower in capability than placing the broker's issuer token on a remote
host) are in `workplans/ADHOC-2026-08-11.md` T03 (ad-hoc **finished** 2026-08-15;
question lives here now); secrets-engine was told to stop holding apply
readiness until this task or the ops-bridge cutover fires (msg `863c7b57`).

**Rebaselined 2026-08-22.** The July description predates the authoritative
OpenBao migration. The current barrier is a rotated **2-of-3**, not the
aspirational 3-of-5 value still present in ops-warden's legacy posture
descriptor. `RMASTER-WP-0020` records two attended restart/unseal cycles on
2026-08-03 with shares supplied through hidden prompts, matching inventory,
audit continuity, exact-path read, and sibling denial. That is valid rehearsal
evidence for re-entry, but it is not misreported as a current production
emergency-seal drill.

The soft half is also already live: `RAILIANCE-WP-0022` reports total concrete
high-risk path coverage and a dedicated coding-agent AppRole proving deny-wins
without reading a secret value. ops-warden consumes that owner control through
`scripts/check_agent_read_boundary.py`; it does not take ownership of the live
OpenBao policy.

The consumer contract and remaining acceptance gate are now documented in
`docs/credential-governance-break-glass.md`. One attended production emergency
seal/unseal drill remains. It must use railiance-platform's evidence template
and validator, have snapshot/quorum/console/driver/abort receipts before the
seal hold point, and be executed by the platform owner—not by a coding agent.

**Parked AppRole disposition:** do not unpark the standalone `warden-sign`
AppRole for T02 break-glass. The normal broker remains autonomous; an issuer
failure deliberately crosses into attended OIDC and, below that, the 2-of-3
platform trust-root. A standing AppRole placed outside the broker would weaken
that boundary and still would not recover a sealed OpenBao. A separately
approved ops-bridge unattended-signing design may re-evaluate the narrow
AppRole under its own workplan; that is service access, not recovery authority.

**Read-only readiness 2026-08-22.** The owner `node-reboot` preflight passed
every automated check without observing a secret value: node/k3s, platform-pg
and continuous archiving, a 16.37-hour successful backup, initialized/unsealed
OpenBao with `shares=3` and `threshold=2`, ESO stores/projections, and the
reviewed audit-core digest were healthy. `ready_for_live_execution` correctly
remained false because no approved window, current encrypted off-host snapshot
receipt, provider-console attestation, quorum attestation, owner-ack flag, or
named abort operator was supplied.

That preflight also exposed a Warden safety defect: the ceremony request matched
the generic `openbao-api-key` template and proposed paste-once provisioning.
`openbao-shamir-recovery-ceremony` now distinguishes the owner-operated approval
ceremony from secret retrieval. It is a non-value-bearing pointer and can never
offer `--fetch`, `--exec`, `--out`, `--wrap`, or paste-once share transport.

**Scenario established 2026-08-22.** The operator delegated preparation to the
agent and reserved their involvement for the final GO/NO-GO. The bounded,
non-secret scenario is
`docs/evidence/WARDEN-WP-0027-T02-drill-scenario-2026-08-22.md`: ops-warden
coordinates and enforces the hold point; railiance-platform is the proposed
OpenBao driver; railiance-infra is the proposed independent console/abort owner;
and railiance-master is asked only for a metadata-only 2-of-3 availability
attestation. All three owner receipts and a green fully parameterized preflight
are required before the final operator question.

The railiance-infra role is scoped by the authoritative review contract
`WARDEN-WP-0027-T02-DRILL-20260822-01-INFRA` in
`interfaces/reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-railiance-infra.json`.
It binds the exact scenario artifact and expiry, requires explicit independent
provider-console and distinct-abort attestations, permits metadata-only evidence,
and states `authorizes_execution: false`.

**Preparation approved 2026-08-22.** Decision
`9da57559-712a-4521-b46e-a4c69729f9d2` authorizes preparation only and narrows
the possible live scope to one intentional OpenBao seal followed by the existing
2-of-3 unseal ceremony. It excludes a host reboot, re-key, restore, policy
change, PVC mutation, credential disclosure, and general workload restart.
Railiance-infra satisfied its direct contract with receipt
`01a02b4b-7295-7836-b288-f29407008524`. The revisioned, non-authorizing gate
matrix is in
`docs/evidence/WARDEN-WP-0027-T02-drill-preparation-checklist-2026-08-22.md`;
platform snapshot/driver and master quorum contracts remain required before the
fully parameterized read-only preflight.

**Live attempt terminal NO-GO 2026-08-23.** All owner receipts and the final
read-only preflight became green, and the operator opened the exact bounded
window with decision `449a697a-5303-4582-aa9e-b0bc8b35ab2d`. The platform
driver stopped before seal when attended OIDC succeeded but the read-only agent
home prevented token-helper persistence and the underlying client emitted the
short-lived credential into captured output despite `-no-print`. The credential
was immediately self-revoked. Independent metadata confirmed OpenBao remained
initialized and unsealed with zero unseal progress; no seal, unseal, reboot,
restore, policy, PVC, or workload mutation occurred. NO-GO decision
`85724c0c-e70f-4e2f-a8c6-a9cb1ea6331b` supersedes and consumes the GO; the
scenario and its receipts cannot be reused.

The direct remediation interface is `RAILIANCE-WP-0026-T01`. Ops-warden commit
`0fae0904ce8d8694338dd53a8a79abec5fec788d` replaces the persistent login-only
handoff with a contained login-plus-reviewed-command envelope. It proves a
private writable helper before OIDC, captures login/child/revocation stdout and
stderr, fails closed on any output or persistence defect, self-revokes, and
removes the helper. No live OIDC was used for verification. The value-safe
owner receipt is
`docs/evidence/RAILIANCE-WP-0026-T01-ops-warden-receipt.json`; a future drill
requires owner acceptance, fresh receipts, a new scenario id, and a new human
decision.

**Containment review status 2026-08-23.** `railiance-infra` independently
accepted the exact implementation revision at its commit `186b030` and State
Hub message `7b2846dd-4b7c-4933-b61f-921e8ccf9ff2`, after rerunning the 42
focused tests and lint. Direct platform review request
`9a5f8973-0e74-4df2-8089-5a5bdc42295b` now carries the exact revision, receipt,
test results, and infra acceptance. T02 remains `progress` until
`railiance-platform` accepts or requests changes. No human GO is relevant while
that owner gate is open; acceptance will permit preparation of a new scenario,
not execution or reuse of the terminal one.

**Owner gate CLOSED — accepted 2026-08-23, found 2026-08-28.**
`railiance-platform` accepted the exact revision `0fae0904`. Recorded in their
`RPF-WP-0017-attended-login-output-containment.md` (`status: finished`), which
also records railiance-infra's independent approval at `186b030` and marks all
five acceptance criteria met. The receipt digest was verified here rather than
taken on trust: `docs/evidence/RAILIANCE-WP-0026-T01-ops-warden-receipt.json` at
ops-warden `bc1966da` hashes to
`d2ba444ed16989590325697e69d25283dc75a9432c29a72e627e80bf9fd987e4`, matching
their record exactly.

Note the identifier: the remediation interface recorded above as
`RAILIANCE-WP-0026-T01` is `RPF-WP-0017-T01` in the owner's repo. Searching for
the cited id finds an unrelated workplan, which is part of why this sat unnoticed.

**Their acceptance is source acceptance only and authorizes no live OIDC or
drill** — their words, and the boundary holds. So T02's state changes but its
`Done when` does not: what was blocked was *preparing a new scenario*, and that
is now permitted. The terminal NO-GO scenario and its receipts remain unusable.

**Third instance of the same failure this session.** The acceptance existed for
five days; no message reached ops-warden. Identically, `key-cape` accepted the
WP-0033-T04 question on 2026-08-23 with no message, and nine unread messages
were sitting on already-superseded threads. The `verified:` field added by
WP-0033-T05 was built for exactly this and it works — what does not work is
waiting for a counterparty to tell you. Re-checking a blocker means reading the
owner's repository.

**Remaining to close T02:** one attended production emergency seal/unseal drill,
requiring a new scenario id, fresh owner receipts from platform/infra/master, a
fully parameterized green preflight, and a new human GO. It is executed by the
platform owner, never by a coding agent. That is an operator decision, not an
agent one, so T02 stays `progress` and the decision is surfaced rather than taken.

## Task: Tamper-evident policy governance + reconcile

```task
id: WARDEN-WP-0027-T03
status: cancel
priority: medium
state_hub_task_id: "7dbedcdc-dd5a-551c-bff1-0702fea0a9cf"
```

Policy-as-code for OpenBao policies/roles with tamper-evidence: a signed/hashed
manifest of the intended policy set, drift **detection** against live OpenBao, and a
reconcile-to-known-good-commit path. (Drift detection may land independently even if
full reconcile stays gated.) Coordinate ownership with railiance-platform (OpenBao
deployment) — ops-warden contributes the ops-access policy surface, not the cluster.

**Done when:** live OpenBao policy state is diffable against a signed known-good
manifest, and an attended reconcile can restore it.

**Cross-repo:** railiance-platform (OpenBao cluster + policy custody).

## References

- `WARDEN-WP-0026` — Strand A (disclosure hygiene, advisory rotation) — prerequisite
- `.claude/rules/credential-routing.md` — ops-warden custodies no secret values
- `railiance-platform` — OpenBao cluster, policy custody, credential broker
- `wiki/AccessRouting.md` — issue vs route vs assist boundary
- `workplans/ADHOC-2026-08-11.md` T03 — parked warden-sign AppRole (ad-hoc finished
  2026-08-15); T02 resolves it as unsuitable for break-glass, while a separately
  approved ops-bridge unattended-signing design may still re-evaluate it
  cert_command cutover
- `secrets-engine` `SECRETS-WP-0004` — the parked AppRole apply/handoff