railiance-platform/docs/provisioning-windows/2026-10-04-1000/keycape/founder-runbook.md

Founder runbook: KeyCape secrets-engine-openbao client secret custody

Source only. Nothing here has been run. Every command below is run by the founder, attended, from /home/worsch/railiance-platform at the reviewed commit. No command prints a secret value. Design: ops-mason plans/keycape-secrets-engine-openbao-client-custody.md (658cdbc, MASON-WP-0009). CCRs: credential-change-requests/CCR-2026-0033.yaml (authoritative custody + issuer ESO reader) and CCR-2026-0034.yaml (attended operator OIDC reader).

Gates before any command

• A1 approve the ops-mason design.
• A2 approve CCR-2026-0033 and CCR-2026-0034, each with platform-owner, key-cape-owner and net-kingdom-owner reviews, and confirm the bound claims of both readers (bound_claims_confirmed: true):
  • CCR-2026-0033: Kubernetes SA external-secrets/external-secrets, audience openbao.
  • CCR-2026-0034: groups=net-kingdom-admins and /assurance/level=aal2 (bound_claims_type: string). KeyCape (7d96ee4) reports that MFA is not pinned on the openbao-admin client, so this aal2 binding is the lane's MFA guarantee.
• A3 a founder decision id for the one custody-child run, recorded in an approved review comment of CCR-2026-0033. It must differ from boundary decision 632a8893-0d6a-4591-a50b-40bbdb798f32. Written below as <A3-decision-id>.

Live order

1. Custody child (section B) writes KV v1 plus metadata.
2. Platform applies the policies/roles of both CCRs and the sso-only ESO (section C). Secret sso/keycape-secrets-engine-openbao-client exists.
3. NetKingdom env patch (NK-WP-0046).
4. KeyCape registration merge (KEY-WP-0037-T03).
5. KeyCape restart immediately after step 4. Between the KV write and the restart, exchanges fail closed (401).
6. Live issuance check (KEY-WP-0037-T04).
7. OpenBao JWT role/login proof (RPF-WP-0035-T02).

Steps 3 to 7 belong to their owners and each needs its own go.

A. Metadata-only survey (read-only; before step 1)

ops-mason's survey generator refuses data paths and runs under the scoped, value-blind ops-mason-build grant:

cd /home/worsch/ops-mason
./scripts/bao-session.sh grant MASON-WP-0009-T04
./scripts/plan-survey.py --kv-path platform/workloads/keycape/secrets-engine-openbao \
  --policy workload-kv-read-keycape-secrets-engine-openbao-operator \
  --k8s-role external-secrets-keycape-secrets-engine-openbao \
  --consumer secrets-engine --keyword keycape --keyword secrets-engine-openbao
./scripts/bao-session.sh revoke

Expected: a Markdown survey block. The block shows KV metadata absent at the path (otherwise the first run must not be used; see --rotate). It also shows no existing role or policy with the two proposed names, and no auth/jwt role touching the path. Paste it into MASON-WP-0009-T04. It contains no value.

B. Attended custody run (step 1)

Pre-check, offline (no OpenBao call): both CCRs validate and the child is the reviewed revision.

python3 scripts/credential-change.py validate credential-change-requests/CCR-2026-0033.yaml
python3 scripts/credential-change.py validate credential-change-requests/CCR-2026-0034.yaml
python3 -m pytest -q tests/test_keycape_secrets_engine_openbao_custody.py
git log -1 --format=%H -- scripts/keycape_secrets_engine_openbao_custody.py

Expected: [OK] for both with no "bound claim is not confirmed" warning; tests pass. The child has no separate dry-run mode. It refuses before any OpenBao call when the decision id is missing, wrong or unrecorded, or when either CCR is not approved and confirmed. It also refuses outside the attended envelope or when BAO_TOKEN/VAULT_TOKEN is set.

Run (first write, cas=0):

set +x; umask 077; unset BAO_TOKEN VAULT_TOKEN
warden access openbao-platform-admin-login --exec -- \
  python3 scripts/keycape_secrets_engine_openbao_custody.py provision \
  --approval-decision <A3-decision-id> \
  --receipt /home/worsch/.local/state/railiance/receipts/CCR-2026-0033-custody-$(date -u +%Y%m%dT%H%M%SZ).json

Rotation uses the same command with --rotate (cas=<observed current version>) under a standing rotation approval for CCR-2026-0033. It is roll-forward only.

Expected: exit 0, no output, and a new mode-0600 receipt (create its parent directory 0700 first if absent):

{"schema": "platform.keycape-secrets-engine-openbao-custody.v1", "ccr": "CCR-2026-0033",
 "status": "custody_written_pending_issuer_delivery", "phase": "done",
 "credential_values_emitted": false, "client_side_read_admitted": false,
 "approval_decision_id": "<A3-decision-id>", "ccr_sha256": "<digest>",
 "kv_written": true, "kv_version": 1, "kv_cas": 0, "kv_request_id": "<id>",
 "kv_metadata": "applied", "started_at": "...", "finished_at": "..."}

On status: failed, read error and phase. A failure with kv_written: true keeps that version. Recover only with a fresh --rotate run, never by reading the value.

C. Policies, roles and ESO (step 2)

Policies and roles go through the sanctioned delegated applier. It refuses unapproved CCRs and unconfirmed bindings, and it diffs the policy files against the generated bodies.

# dry-run (offline review of policy diff and role payload, incl. token_num_uses=4, aal2 claim)
python3 scripts/credential-change.py applier-dry-run credential-change-requests/CCR-2026-0033.yaml
python3 scripts/credential-change.py applier-dry-run credential-change-requests/CCR-2026-0034.yaml
python3 scripts/credential-change.py applier-apply credential-change-requests/CCR-2026-0034.yaml --plan-only

# apply (attended platform-admin; one phrase per CCR)
warden access openbao-platform-admin-login --exec -- \
  python3 scripts/credential-change.py applier-apply credential-change-requests/CCR-2026-0033.yaml \
  --confirm "DELEGATED APPLY CCR-2026-0033" --actor founder --record-state-hub --quiet
warden access openbao-platform-admin-login --exec -- \
  python3 scripts/credential-change.py applier-apply credential-change-requests/CCR-2026-0034.yaml \
  --confirm "DELEGATED APPLY CCR-2026-0034" --actor founder --record-state-hub --quiet

Expected: exit 0 with no output (--quiet, which the warden wrapper requires) and a value-free delegated_metadata_apply evidence entry in each CCR.

ESO: eso.draft.yaml in this directory carries railiance.io/admission: pending-review. Promote it to approved in a reviewed commit after A2 first. Then:

kubectl apply --dry-run=server -f docs/provisioning-windows/2026-10-04-1000/keycape/eso.draft.yaml
kubectl apply -f docs/provisioning-windows/2026-10-04-1000/keycape/eso.draft.yaml
kubectl -n sso get externalsecret keycape-secrets-engine-openbao-client \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
kubectl -n sso get secret keycape-secrets-engine-openbao-client -o name

Expected: ClusterSecretStore ... created (server dry run) and ExternalSecret ... created (server dry run), then the real apply, then True, then secret/keycape-secrets-engine-openbao-client. Never use another output format or a template on the Secret.

After step 2, ops-mason runs its capabilities-only verification (MASON-WP-0009-T04). It checks operator-token read on the data path and deny everywhere else, and that a non-member or non-aal2 identity cannot obtain the role. It also checks the circularity invariants and the metadata settings.

Rollback (roll-forward; never a read of an old value)

Before step 3: delete the CCR-2026-0034 OIDC role, then its policy. Delete the ExternalSecret (uid precondition), then the ESO role and policy. The KV entry is kept until a separate destroy approval. After step 3: the NetKingdom rollout helper restores the prior config/Deployment first, then the same sequence.