railiance-platform/workplans/RPF-WP-0064-isolated-original-openbao-restore.md

id	type	title	domain	repo	status	flavor	owner	created	updated	related	state_hub_workstream_id
RPF-WP-0064	workplan	Isolated original-OpenBao restore for secrets-engine consumer continuity	financials	railiance-platform	active	implementation	claude	2026-10-23	2026-10-28	SECRETS-WP-0015 NK-WP-0049 RPF-WP-0015 RPF-WP-0036 RPF-WP-0035 RAPP-OPENBAO-WP-0001	e19808db-52d6-508b-b70c-6775c4f78c4f

RPF-WP-0064 — Isolated original-OpenBao restore for secrets-engine consumer continuity

Receiving workplan for hub message 41e9af3e from net-kingdom (case secrets15-nk49-readiness). The founder accepted on 2026-10-23 that railiance-platform owns this as source preparation.

NK-WP-0049-T18 (net-kingdom a09bf20, verified receipt sha256 16d72f7c…1651, secrets-engine bfa1b3b) proved in isolation that the original KeyCape issuer, its JWKS key and the archived secrets-engine-openbao registration are intact. SECRETS-WP-0015-T04 also needs the original OpenBao, recovered on its own into an isolated, network-contained target. That recovery uses the original custody and unseal path. It must show the KV registration platform/workloads/keycape/secrets-engine-openbao, the narrow secrets-engine JWT role and the operator auth path. There is no production redirect, no fresh backend standing in for the original and no injected secret.

Scope of this plan: the restore and its evidence only. The secrets-engine consumer run (attended in-memory secret source, KeyCape exchange, OpenBao JWT login, openbao:admin refusal, session revocation) is a separate founder-admitted attempt owned by secrets-engine. This plan prepares its target.

Why not RPF-WP-0036-T03: that task owns recurring backup/restore assurance and waits on the activity-core scheduler (ACTIVITY-WP-0042). This is a one-off attended restore with no scheduler, so a separate plan avoids taking on that dependency. Its evidence can later feed T03 as one isolated restore receipt. RPF-WP-0015 keeps the live snapshot experiments. Its snapshot receipts authorize no restore.

Nothing in this plan claims restore capability or consumer continuity. Every live step stays wait until attended evidence exists.

T01 — Source survey and recorded findings

id: RPF-WP-0064-T01
status: done
priority: high
state_hub_task_id: "02b16f20-8ab3-547b-b8ac-1334139d3450"

Findings (2026-10-23, read-only source survey; no cluster, OpenBao or snapshot access)

1. Snapshot capture and location. The procedure is docs/openbao.md §Backup: bao operator raft snapshot save on openbao-0, then age encryption before the snapshot leaves the operator machine. The latest recorded capture is the attended RPF-WP-0015-T03 hold point on 2026-10-14. That snapshot was encrypted to the offsite recipient derived from the founder's offline railiance-backup.key, decrypt-verified in memory and uploaded to railiance-backups/openbao/ (HTTP 201), and the plaintext was shredded from tmpfs. An earlier 2026-08-23 drill receipt is at reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json. Receipts are validated against docs/openbao-reboot-snapshot-receipt.example.json (24 h freshness, cluster id, applied index). Those receipts authorize no restore.
2. Freshness against the needed objects. The KV path was corrected to platform/workloads/keycape/secrets-engine-openbao on 2026-10-06, and the JWT role was proven live on 2026-10-10 (CCR-2026-0038). The 2026-10-14 snapshot should therefore contain both. This is an inference until the restored metadata readback confirms it. If they are missing, T07 stops; it does not repair anything.
3. Version pin. Chart openbao/openbao 0.28.2 (Makefile OPENBAO_CHART_VERSION); image quay.io/openbao/openbao:2.5.4 (helm/openbao-values.yaml; docs/openbao.md still says app v2.5.3, which is stale doc text). The restore target must use the same image, pinned by digest at rehearsal time. Storage is integrated Raft.
4. Unseal and custody model in use. Shamir 2-of-3, unsealed by the founder in attendance (the 2026-10-14 post-reboot unseal was done outside the transcript). The transit auto-unseal stanza is still commented out. A Raft snapshot restore brings back the original barrier keyring, so the restored target unseals only with the original shares. That is the "original unseal path" the request asks for. Restoring needs a throwaway initialization of the empty target first. Its throwaway keys and token stay in process memory, are never recorded, and stop working once snapshot restore -force completes. This is not a "fresh backend" in the request's sense, because no state from it survives.
5. Role, KV and auth definitions are owned here, not in rapp-openbao. The sources are openbao/auth/secrets-engine-jwt-role.json, openbao/auth/keycape-services-config.json (mount keycape-services), docs/contracts/secrets-engine-openbao-jwt-login.yaml, the operator path openbao/auth/netkingdom-oidc-config.json plus netkingdom-platform-admin-role.json (mount netkingdom, role platform-admin), and the workload KV lane (docs/workload-kv-access-lanes.md, KeyCape provisioning window 2026-10-04). rapp-openbao's SCOPE excludes policy, custody and workload KV lanes. It owns only the package (chart, values, UI overlay), and neither of its workplans has a restore task. rapp-openbao only needs to confirm the image/chart pin used by the target.
6. Isolation constraint found. keycape-services validates JWTs through jwks_url: https://kc.coulomb.social/jwks, and netkingdom uses OIDC discovery at https://kc.coulomb.social. A networkless OpenBao can restore, unseal and read back configuration and metadata. However, a live JWT login (the consumer run) needs that hostname to resolve inside the contained network to the isolated NK49 KeyCape, with a trusted TLS chain. It must never reach the production KeyCape. An interactive OIDC operator login is out of scope for the isolated target. The operator path is proven by restored config and role readback, plus root-less login refusal where applicable.
7. Reusable patterns. These are reused as patterns, not copied:
   • Existing in-repo template docs/openbao-restore-drill-evidence.example.json and its validator make openbao-validate-restore-evidence.
   • net-kingdom docs/bootstrap-recovery-ceremony.md and tools/bootstrap-recovery/isolated-*.py: attended, hidden-key, fail-closed setup with a Docker --network none target, a safe receipt and an independent verifier.
   • secrets-engine docs/founder-admission.md for the founder SSH-signed admission with a time window.

T02 — Isolated target and adapter design

id: RPF-WP-0064-T02
status: done
priority: high
depends_on: [RPF-WP-0064-T01]
state_hub_task_id: "51adc483-5102-5723-ab64-e7a2698befe8"

Design scripts/openbao-isolated-restore/ (adapter plus doc). Fixed properties:

• Target: a disposable Docker container on the founder workstation, running the T01 image digest with an in-container Raft data directory on tmpfs. It is never on railiance01 or the cluster, and it has no kubeconfig, no host networking and no published ports.
• Network containment: phase A (restore + readback) uses --network none. Phase B (handed to the consumer run) uses a single --internal Docker network that has only the isolated KeyCape from NK49 and this OpenBao. In phase B, kc.coulomb.social resolves to the isolated KeyCape, and egress is checked to be refused before admission. Phase B is offered, not run, by this plan.
• Custody inputs, all founder-attended and never through argv, files in the repo, env dumps or logs:
  • (a) the encrypted snapshot object from railiance-backups/openbao/, named by its ciphertext sha256;
  • (b) the offline railiance-backup.key age identity, used for decryption straight into the tmpfs stream;
  • (c) 2 of 3 original Shamir shares, entered through a hidden prompt. The restore needs no root token and no secret value is read. Readback uses a token created in the restored target by an attended generate-root, used in memory only and revoked before teardown, or uses the restored operator path. Which of the two is decided in T02 review.
• Readback (metadata only): sys/seal-status, Raft applied index and cluster id; the mounts keycape-services and netkingdom and their configs (issuer and JWKS URL); role secrets-engine-openbao (bound claims and policies compared to the source JSON hash); platform/workloads/keycape/secrets-engine-openbao KV metadata (versions and timestamps, never data); and policy names. No KV data read.
• Timeout/abort/cleanup: a hard wall-clock limit of 45 minutes from the first custody input. Any fail-closed check, a missing object, an unexpected network interface, or the founder pressing abort leads to docker rm -f, tmpfs unmount, shredding of the decrypted stream, and a recorded aborted receipt. Teardown also runs on success, unless phase B was admitted in the same window. The plaintext snapshot never touches persistent disk.

Delivered (2026-10-23)

scripts/openbao-isolated-restore/ holds run-isolated-restore.sh (the attended launcher), preflight.py (metadata-only, runs before any prompt) and restore_adapter.py (the session plus a reaper). The design document is docs/openbao-isolated-restore.md. Implemented as specified above, with these recorded choices:

• Readback credential: generate-root from the two shares already entered. The OTP is decoded in memory, and the credential is revoked and verified dead before teardown. The operator OIDC path cannot work without a network. The admission names the choice (readback_credential), so the founder confirms it by signing T05.
• OpenBao 2.5 disables unauthenticated generate-root by default. The target enables it on its loopback-only listener, inside --network none. The image's VOLUME paths (/openbao/file, /openbao/logs) are overlaid with tmpfs. That is the only way to get no Docker volumes, and the first native run failed closed on exactly this.
• Custody uses hidden TTY prompts. The age identity sits in a sealed memfd (/dev/fd/N for age), and the decrypted snapshot sits in a sealed memfd streamed to snapshot restore -force /dev/stdin. Shares are held in wiped buffers. No custody value appears in argv, env, files or logs.
• The admission namespace is railiance-openbao-restore-attempt, with anchor /etc/railiance-openbao-restore/allowed_signers. Original material refuses if the anchor is writable by the runtime account, or if PINNED_IMAGE_DIGEST (T06) is unset or differs from the admission.
• The 45-minute deadline is enforced in-session (every subprocess timeout) and by a reaper on parent loss or at hard deadline + 60 s. The reaper removes the container and writes the aborted receipt.
• Phase B is a design stub only (doc §Phase B).

T03 — Safe receipt schema and independent verifier

id: RPF-WP-0064-T03
status: done
priority: high
depends_on: [RPF-WP-0064-T02]
state_hub_task_id: "69ef4a3b-7b0a-5b7b-b49a-c688f9d19554"

Extend docs/openbao-restore-drill-evidence.example.json into a v2 receipt:

• attempt id and admission hash;
• image digest;
• ciphertext sha256 of the snapshot and the source cluster id and applied index (from the RPF-WP-0015 receipt);
• restored cluster id and index;
• a network-namespace proof (interfaces and the egress-refusal result);
• presence and source-hash match for each required object;
• result restored | refused | aborted;
• the teardown proof;
• no_secret_material_recorded.

Add a verifier that runs separately from the adapter. It rejects placeholders, unknown fields, missing teardown, data fields, and anything shaped like a token or share. Validate it with adversarial fixtures.

Delivered (2026-10-23)

• The v2 receipt shape is docs/openbao-restore-receipt-v2.example.json (schema railiance-openbao-restore-receipt/v2, extends the drill template). The admission shape is docs/openbao-restore-admission.example.json. All digests are typed sha256:, and no key name sits next to a bare hex value. gitleaks 8.30.1 finds nothing in any new file.
• The independent verifier is scripts/openbao-isolated-restore/verify-restore-receipt.py, run with make openbao-verify-restore-receipt. It does not import the adapter. It takes a clean checkout at adapter_commit for the source-hash comparison, and accepts --not-after and --expect-result.
• 17 offline tests in tests/test_openbao_isolated_restore.py (make openbao-isolated-restore-test) cover:
  • 28 adversarial receipt fixtures: unknown or missing fields, missing or reaper-only teardown, data fields, KV /data/ paths, credential/share/hex/age/PEM shapes, placeholders, untyped digests, missing claims, deadline overrun, unprotected anchor for original material, result inconsistencies, phase B, source hash/commit mismatch, duplicate keys and the --not-after window;
  • admission refusals and signature namespace/byte binding;
  • launcher and preflight refusals before any prompt.

T04 — Staged rehearsal with synthetic material

id: RPF-WP-0064-T04
status: done
priority: high
depends_on: [RPF-WP-0064-T03]
state_hub_task_id: "9c10caca-d772-5608-954e-6e9f78d59a39"

Run the full adapter against a synthetic OpenBao, initialized on purpose with generated Shamir shares and seeded with stand-in mounts, role and KV metadata that mirror the T01 source files. Snapshot it, age-encrypt it to a generated identity, restore it into the T02 target, unseal it with the synthetic shares, read it back, and tear it down. Rehearse each abort path (timeout, wrong ciphertext hash, network interface present, missing object). This step uses no real material.

Rehearsal result (2026-10-23): native, generated material only

make openbao-isolated-restore-rehearsal ran 12 tests, all passing, in about 156 s, against the real quay.io/openbao/openbao@sha256:436eaf97…c115 image (the local pull of tag 2.5.4; T06 still confirms the production digest). Every container ran with --network none and tmpfs storage.

Each run generates its own material:

• an SSH admission key and anchor;
• an age identity;
• a 3-share, threshold-2 Shamir init;
• root credentials;
• a KV value.

A source OpenBao is seeded from the real source JSON/HCL shapes. The JWT and OIDC configs are written with skip_jwks_validation, because the source has no network. That source is snapshotted, age-encrypted, restored through the adapter into a fresh target, unsealed with shares 1 and 3, read back, and torn down. Each receipt is checked by the independent verifier, and its text is checked to hold none of the generated credentials, shares, identity or KV value.

Native coverage:

• restored end to end: sealed after restore, original shares unseal it, cluster id and index match, the throwaway init is invalidated, the readback credential is revoked;
• wrong age identity (aborts before any target exists);
• wrong ciphertext hash;
• wrong plaintext hash;
• wrong share (unseal_refused, target removed);
• missing JWT role and KV registration (refused/object_missing, not repaired);
• source drift (refused/object_mismatch);
• in-session timeout (deadline_exceeded);
• operator Ctrl-C (operator_abort);
• parent loss (the reaper removes a real container and writes parent_lost);
• the reaper hard deadline (kills the session, deadline_exceeded).

Mocked (reduced coverage): only the network-interface abort. A wrapper around docker adds a bridge entry to the inspect output of a real networkless container. Attaching a real network is out of bounds. The adapter aborts network_containment_failed, and the verifier refuses any receipt that claims such a network.

Not covered: the founder TTY prompt itself, run-isolated-restore.sh past its pre-prompt refusals, and preflight.py past admission refusal. All three need a protected root-owned anchor and the T06 pin. The rehearsal drives the same Session with injected prompts.

After the run, no rpf64-* container, Docker volume or /dev/shm workdir remained, and all generated material was deleted.

Retained rehearsal receipt (synthetic material, outside Git): ~/rpf64-rehearsal-20261023/rehearsal-receipt.json on the workstation, sha256:5764de67e6ed1a998838ace7843f96b3be0e4da37c8e17b8bb078372245feeae. It is bound to adapter_commit a61a07cf158320bf5fe4eb3b8f50dc8ced9c0714, and its result is restored/all_required_objects_match. Re-verify it with:

make openbao-verify-restore-receipt RECEIPT=<path> SOURCE_ROOT=<clean checkout at a61a07c>

T05 — Founder-signed attempt admission

id: RPF-WP-0064-T05
status: wait
priority: high
needs_human: true
depends_on: [RPF-WP-0064-T04]
blocking_reason: "Founder signs (ssh-keygen -Y sign, namespace railiance-openbao-restore-attempt; allowed_signers at a root-owned /etc/railiance-openbao-restore/) the v1 admission in docs/openbao-restore-admission.example.json: attempt id, phase A only, window (<=24h) + 2700 s deadline, abort operator, T06 image digest, adapter commit + source_sha256 closure, snapshot ciphertext/plaintext sha256 + bytes + age recipient + source cluster id/applied index + source receipt sha256 (RPF-WP-0015-T03), unseal 2-of-3, readback via generate-root from entered shares. T04 rehearsal done; waits on T06 pin."
state_hub_task_id: "a954eaf9-855b-5458-9f13-c6cfed289934"

2026-10-23: the founder accepted the three T02 design choices before signing: (1) readback through a temporary root credential generated from the two entered shares, revoked and checked dead before teardown; (2) unauthenticated generate-root enabled only on the target's own loopback listener inside the networkless container; (3) strict match of restored role/policy objects against current source, so a snapshot older than a later source edit yields refused. Signing T05 still confirms them per attempt.

T06 — Owner confirmation of package pin

id: RPF-WP-0064-T06
status: done
priority: low
depends_on: [RPF-WP-0064-T02]
state_hub_task_id: "cb689628-9a5a-5f2b-a3ec-be37bd6cdb37"

2026-10-24: done from the live runtime. The founder read openbao-0 read-only: spec image quay.io/openbao/openbao:2.5.4, container imageID docker.io/openbao/openbao@sha256:436eaf9778cad75507ff70ea26ace30dcbe15606e619ac3823495663d7f7c115, equal to the T04 rehearsal pull (digests are registry-independent). Written to PINNED_IMAGE_DIGEST; the offline suite (17) passes. This is the current production digest. That the same digest ran at the 2026-10-14 snapshot is not separately proven; the attended run will refuse on any restore incompatibility rather than repair. rapp-openbao was informed and may object; no rapp-openbao change requested.

T07 — Attended original restore (phase A)

id: RPF-WP-0064-T07
status: done
priority: high
needs_human: true
depends_on: [RPF-WP-0064-T05, RPF-WP-0064-T06]
state_hub_task_id: "96e88167-c484-527c-84aa-0464dd0f5597"

2026-10-24 attempt RPF-WP-0064-T07-20261024-01 (admission 796414c2…a3b9, adapter 1701a3f, window 08:00Z–14:00Z): aborted, reason unseal_refused, after 168 s; the receipt 1a7de4a4…abe6 (kept outside Git) was independently verified from a clean clone at 1701a3f as a genuine aborted result. It proved the off-host ciphertext 8069449c…8c41 (375618 bytes) intact, decryption with the offline age identity, plaintext a6312444…590b, and that the founder's two original Shamir shares unseal the restored barrier (unsealed_with_original_shares: true); networkless containment and full teardown held, with no volumes. The abort came after unseal: sys/leader never reported an active self with an applied index within 60 s, and the adapter reused the unseal_refused reason. The first hypothesis (target raft node id isolated-restore versus production voter openbao-0) was disproved by a native rehearsal, which restored and elected a leader anyway. The adapter now aligns the node id with production, waits up to 180 s, aborts as leader_not_elected, and records bounded typed sys/leader/sys/seal-status fields and a redacted first error line in restored.leader_wait. Offline 19 and native 12 pass. A new -02 admission is needed for the new adapter commit.

2026-10-24 attempt -02 (admission 33c8aab2…75ab, adapter 624843e): two runs refused at the first prompt (custody_identity_refused, clipboard content; the key file and a shape-only paste check were correct), then a third run reached aborted, leader_not_elected after 236 s. Its leader_wait diagnostics showed the cause: after the original shares unsealed the node, seal-status read sealed: true and sys/leader returned API errors for 180 s. Production declares a file audit device at /openbao/audit/openbao-audit.log (rapp-openbao values); the restored audit table re-enables it on unseal, the target had no writable /openbao/audit, and OpenBao sealed itself again. Both -02 aborted receipts (cfb2c2af…, 4c5cd362…, 904a9d95… incl. the refusals) verified independently from a clean clone at 624843e. Fix: an in-memory /openbao/audit on the target. The rehearsal source now carries production's file audit device, and a native regression case without that mount reproduces the reseal (leader_not_elected, sealed: true), while the full restore passes with it. Offline 19 and native 13 pass. A -03 admission is needed for the new adapter commit.

2026-10-24 attempt -03 (admission 3ef17fd1…4cc9, adapter 7084b57, founder attended): restored, all_required_objects_match in 46 s. Independently verified from a clean clone at 7084b57: receipt sha256:4e048deedda21aa35ea4ff811e437f9c0e546fbefe4eb8a2025a74bf65e2579d, retained outside Git on the founder workstation (mode 0400). Original cluster id fd28df5d… matched, applied index 6040604 (not behind 6040594), leader elected in 2 s, the throwaway init was invalidated, the original shares unsealed, and all nine required objects were present and equal to the committed sources: auth mounts, secrets mounts, keycape-services JWT config and role secrets-engine-openbao, netkingdom OIDC config and role platform-admin, policy secrets-engine-login-self, and KV metadata platform/workloads/keycape/secrets-engine-openbao (no KV data read). The readback credential was revoked; networkless (lo only), no volumes, full teardown. Claims not made: consumer continuity, live JWT login, phase B network, owner acceptance, recurring restore assurance. T08 handoff: recorded in secrets-engine SECRETS-WP-0015-T04 and net-kingdom NK-WP-0049-T18 the same day. RPF-WP-0036-T03 may point to this receipt as a one-off; it is not recurring assurance.

Run the admitted adapter once. The outcome is a T03 receipt verified independently. The receipt is kept outside Git and referenced by its hash. A missing object or failed check is a valid refused outcome; it is not repaired in this attempt.

T08 — Bounded evidence handoff

id: RPF-WP-0064-T08
status: done
priority: medium
depends_on: [RPF-WP-0064-T07]
state_hub_task_id: "94139cb1-037a-58ef-883c-dde71fe6bfde"

Hand to secrets-engine (SECRETS-WP-0015-T04) and net-kingdom (NK-WP-0049-T18) the receipt hash, the verifier result, the image digest and the recovered-object list. Also hand over the phase B offer terms (contained network with the isolated KeyCape, same custody, its own admission). The handoff makes no continuity claim. A pointer to the receipt goes to RPF-WP-0036-T03 as one isolated restore receipt; it does not count as recurring assurance.

Phase B plan and confirmation (2026-10-27, reply to net-kingdom message 5da58a3d)

Source: net-kingdom docs/nk49-phase-b-plan.md (NK-WP-0049-T43); founder decision of 2026-10-27 names railiance-platform as creator and tearer-down of the single --internal network and as keeper of the restored OpenBao for the admitted window. Preparation only. This confirmation authorizes nothing, starts nothing and creates no network. Every start needs a new founder-signed window; the -03 admission (3ef17fd1…4cc9) and the T40 window carry nothing over.

Confirmed: railiance-platform can own (a) network create/inspect/egress-refusal proof/removal, and (b) the phase A restore followed by keeping OpenBao up unsealed on that network for the admitted window, with a phase B receipt. Subject to the changes and reviews below.

What the adapter does today (7084b57, attempt -03): --network none, lo only, listener 127.0.0.1:8200 with tls_disable, api_addr/cluster_addr on loopback, generate-root enabled unauthenticated on that loopback listener, check_network requires none, receipt schema lists phase_b_network as a not-claimed field, the verifier rejects phase B, teardown always runs (or by reaper), 45 minute deadline from first custody input, no volumes.

What must change (new mode, phase A path stays byte-identical in behaviour):

1. Admission and preflight: new signed phase B admission naming the network name, subnet, OpenBao alias (bao-restored), admitted container set, window and adapter commit; preflight refuses a pre-existing network or stale rpf64-* objects.
2. Network lifecycle: create docker network create --internal with a label naming the attempt; prove Internal=true, no gateway, empty attached set; prove egress refused (public address, non-admitted name) from a probe container before admission continues; at the end remove the network last and show it gone. The reaper removes only objects carrying this attempt's label, never a network it did not create.
3. Listener: OpenBao must listen on the internal address (0.0.0.0 inside the namespace, reachable only via the internal network), api_addr accordingly, --network <internal> --network-alias bao-restored. Raft cluster_addr stays loopback (single node).
4. check_network variant: exactly one named internal network, inspected Internal=true, attached set equals the admitted containers, subnet route and no default route; the lo-only check is replaced for this mode.
5. Readback credential: phase A uses generate-root from the entered shares. In phase B the unauthenticated-generate-root setting must not be exposed on the internal listener. Plan: keep it on a separate loopback-only listener used for readback, or disable it after readback; the credential is revoked and shown dead before the consumer is admitted to the network. The consumer only ever uses the JWT role.
6. Lifetime: do not tear down after readback; stay up until the founder window ends, a consumer-done signal, or abort. The 45 minute limit runs from first custody input, so OpenBao starts last among long-lived pieces (as the plan orders), and the consumer run follows at once. Hard deadline plus reaper stay.
7. Receipt schema v-next, verifier and adversarial fixtures for the phase B receipt (network facts, egress refusal, revocation before consumer admission, teardown including network removal); no secret values, no KV data. phase_b_network moves from not-claimed to claimed only in that mode.
8. TLS trust (net-kingdom item 4): OpenBao must trust the ISRG Root X1 chain to fetch https://kc.coulomb.social/jwks; verify in the native rehearsal whether the pinned image's CA bundle suffices and do not assume it. No self-signed substitute and no injected trust file without owner review.
9. Rehearsal: native run with synthetic material on a throwaway internal network plus a stand-in JWKS server; offline tests extended.

Needs review: all of the above is a new adapter commit and a new pin (T06 owner confirmation); the listener and generate-root change (item 5) is the security-sensitive point and wants the same review depth as the phase A adapter; admission schema change; independent verification of the phase B receipt from a clean clone. Network subnet and container naming must be agreed with net-kingdom so the attached-set check matches.

Objections / conditions:

• OpenBao stays up with an unsealed backend reachable by the consumer for hours versus the 45 minute phase A bound; the founder must accept the longer exposure explicitly in the window (24 h maximum proposed; I suggest a much shorter window).
• Ordering: network exists before the isolated KeyCape starts; whichever party starts second must not alter the attached set. If NetKingdom's KeyCape cannot attach to a pre-created network, the plan fails closed.
• Docker is the only supported runtime here; the consumer runner is attached by the consumer owner and checked by me by inspection only.
• No live action, no network creation, no custody input until the window, the new admission and pin exist.

id: RPF-WP-0064-T09
status: todo
priority: medium
depends_on: [RPF-WP-0064-T08]
state_hub_task_id: "ef4ded2e-af0c-5e3f-8b8c-aad80e8be074"

Phase B source preparation: implement items 1-9 above as a separate mode, rehearse natively with synthetic material, obtain owner review and the new pin, then wait for a new founder-signed phase B window (T05-style admission). No live run is part of this task.

Owner answers to NK-WP-0049-T43 (2026-10-27, reply to net-kingdom message 43a545e2)

Source and value-free evidence only. No docker, kubectl or OpenBao action; no value read.

1. keycape-services JWT trust fields (presence only).

Field	Committed source (openbao/auth/keycape-services-config.json)	Restored original (T07 -03 receipt)
jwks_url	present	proven present and equal to source
jwks_ca_pem	absent	not proven either way
oidc_discovery_url	absent	inferred absent
oidc_discovery_ca_pem	absent	not proven either way

• Proven: the source file has three keys only (bound_issuer, jwks_url, jwt_supported_algs). The -03 receipt matched the restored config to the source, but restore_adapter.project() compares only keys present in the source file. Extra restored fields are not covered by the receipt.
• Inferred, high confidence: the CCR-2026-0038 applier writes the whole file (bao write auth/keycape-services/config @file, scripts/service_jwt_lane.py). A config write replaces the stored config, so both CA fields were unset when it was applied. OpenBao also refuses jwks_url together with oidc_discovery_url, so the proven jwks_url excludes a discovery URL. A later out-of-band write is not excluded by the evidence. The netkingdom mount (operator path, not used in phase B) uses oidc_discovery_url and has no oidc_discovery_ca_pem in source.
• Consequence: JWKS fetch trust comes from the image's system CA bundle.

ISRG Root X1 in the pinned image (sha256:436eaf97…c115). Not proven, and the bundle was not inspected. Inferred: the production pod used this config, with no CA field, for the live login proof on 2026-10-10. Writing the config with JWKS validation also needs a TLS fetch. So the production image trusted the chain then served for kc.coulomb.social through its system roots. This assumes the chain is Let's Encrypt, which is not recorded in this repo, and that the 2026-10-10 image digest equals the T06 pin, which is not proven. Settle both in the phase B native rehearsal (T09 item 8): run a value-free /jwks fetch from the pinned image against the real letsencrypt-prod chain served on the internal network, and record the result, the chain's root subject and whether a CA field is set. Add a presence-only check of the four fields to the phase B readback.

2. Proposed phase B names, so the attached-set check can match.

Item	Live window	Synthetic rehearsal
Network (--internal, IPv4 only, label railiance.rpf64.attempt=<attempt-id>)	rpf64-nk49-pb	rpf64-nk49-pb-rh
Subnet	10.249.64.0/28	10.249.65.0/28
Restored OpenBao: container, alias, IP	rpf64-nk49-pb-openbao, bao-restored, .2	rpf64-nk49-pb-rh-openbao, same alias
Isolated KeyCape worker (NetKingdom): container, alias, IP	nk49-pb-keycape, kc.coulomb.social, .3	nk49-pb-rh-keycape
Consumer runner (secrets-engine): container, IP	secrets15-pb-consumer, .4	secrets15-pb-rh-consumer
Egress probe (railiance-platform; removed before KeyCape attaches)	rpf64-nk49-pb-probe	rpf64-nk49-pb-rh-probe

• Admitted attached set after the probe is removed: exactly the three named containers, in the order network, probe and removal, KeyCape, OpenBao, consumer.
• KeyCape's dependent containers share the worker namespace (container:<worker>), so they are not attached separately.
• Fixed IPs (--ip) make the attached-set and route checks exact. The subnet avoids Docker's default pools and k3s 10.42/10.43; preflight refuses on any overlap with a host route.
• --internal still gives the host bridge an address. Whether to add com.docker.network.bridge.inhibit_ipv4=true ("no gateway") is decided in the rehearsal. Until then, the check is no default route plus refused egress.
• Names are proposals. Each owner's admission binds them.

3. Phase A evidence as T15's OpenBao dependency (platform-owner view, builder mode). Acceptable. Original material restored in isolation: cluster id matched, applied index not behind, the founder's original shares unsealed it, and all nine required objects matched source (mounts, both auth configs, both roles, policy, KV metadata). The receipt was independently verified from a clean clone, with containment and teardown proven. That is proportionate for a builder-mode estate.

Named residuals:

• (R1) Config match is a projection over source keys; extra fields such as the CA fields are unproven.
• (R2) No live JWT login against the restored target. That is phase B, T09.
• (R3) The operator OIDC path was checked by config and role readback only, with no interactive login.
• (R4) KV metadata only, by design. Value integrity is not shown.
• (R5) The snapshot dates from 2026-10-14. That the image digest at snapshot time equals the pin is not proven.
• (R6) This was one attended restore, not recurring assurance (RPF-WP-0036-T03).
• (R7) The receipt is a single copy outside Git on the founder workstation, referenced by hash 4e048dee…579d.
• (R8) Restore depends on the file audit-device mount (found and fixed in -02/-03). Any future target needs it.

None of these blocks T15's OpenBao line. R1/R2 close in phase B. This is not owner acceptance of the full T15 packet or of the other three services.

T10 — Break-glass drill: generate-root ceremony and held root-token check (NK-WP-0055-T13)

id: RPF-WP-0064-T10
status: done
priority: high
depends_on: [RPF-WP-0064-T08]
state_hub_task_id: "5facf182-91fb-58bc-a2dd-c6724049e70f"

Source: NetKingdom NK-WP-0055-T13. The ceremony is written in docs/openbao.md, section "Break-glass: OpenBao with SSO down". The founder runs this drill against an isolated restored OpenBao only, never production. KeyCape and Authelia are not used, which simulates SSO being down.

Drill mode (new adapter mode; the phase A path keeps its current behaviour).

The drill runs as a separate --mode break-glass-drill with its own signed admission (drill: "break-glass", attempt id RPF-WP-0064-T10-<id>). It does the phase A restore and unseal, then:

1. Held root-token check. A hidden prompt takes the founder's held initial root token. The drill calls auth/token/lookup-self with it and records only valid, policies, ttl == 0, expire_time == null and orphan. It never revokes or otherwise uses the held token.
2. Fresh ceremony. A sys/generate-root/attempt is started with a server-generated OTP. The founder enters two shares again at hidden prompts, so the unseal shares are not reused silently as in phase A. The token is decoded in memory and lookup-self must show ["root"].
3. Minimal action. One read-only action: sys/auth metadata.
4. Revoke. revoke-self must return 204, and a follow-up lookup-self must return 403.
5. Cancelled attempt. Start an attempt and cancel it (DELETE sys/generate-root/attempt), so the cancel step is drilled too.

What the adapter must allow.

• The isolated listener. The adapter keeps disable_unauthed_generate_root_endpoints = false on the --network none loopback listener only, as it does today.
• ALLOWED_WRITES. sys/generate-root/attempt needs both PUT and DELETE, and a second sys/generate-root/update sequence must be allowed.
• ALLOWED_READS. GET sys/generate-root/attempt (status) must be added. auth/token/lookup-self must be allowed with the held token and with the generated token.
• Prompts. One extra hidden prompt each for the held token and for the two re-entered shares.
• Deadline. Everything stays inside the 45 minute deadline, with the same teardown.
• Unchanged rules. No network, no KV /data/ reads, no change to production.

Receipt fields (all free of secret values). The drill writes a receipt v2 extension with:

• drill: "break-glass"
• held_root_token: valid, policies, ttl_zero and expire_time_null
• generate_root: started, shares_entered: 2, complete, otp_decode_ok, policies: ["root"], minimal_action_path and minimal_action_ok
• revocation: revoke_status: 204 and post_revoke_lookup: 403
• cancel_drill: started_then_cancelled
• claims_not_made, which additionally lists:
  • production_root_token_validity: the result is valid only as of the snapshot. Name the snapshot (the 2026-10-14 snapshot or a newer RPF-WP-0066 scheduled one) and its plaintext sha256.
  • production_generate_root_availability
  • sso_down_access_path

The verifier gets matching adversarial fixtures that reject token, accessor, OTP, nonce, encoded-token and share shapes. Offline tests and a native rehearsal use synthetic material and a fake held token.

Founder-run SSO-down walk (production, read-only). Over k3s-api-railiance01 with the admin kubeconfig, the founder runs:

• kubectl -n openbao get pod openbao-0
• bao status (a seal check)
• a kubectl port-forward reachability check

There is no unseal and no generate-root on production. Record the result in the NK-WP-0055-T13 post-event review record.

Owner decision for rapp-openbao. The production listener has unauthenticated generate-root disabled (OpenBao 2.5.4 default). This decision is a precondition for the ceremony to work on production without a valid token. The choices are:

• (a) an incident-time ConfigMap edit plus a restart
• (b) a standing loopback-only listener (127.0.0.1:8210) reachable only through kubectl port-forward or exec

Follow-up (founder decision). If the held initial root token is valid, revoke it on production after the ceremony drill succeeds, or record why it is kept.

Done when:

• the drill mode is implemented and reviewed, with tests
• the native rehearsal passes
• a founder-signed admission exists for the window
• the founder has run the drill and the independent verifier passes on the receipt
• NetKingdom has a value-free post-event review record and the rapp-openbao listener decision

No live run is part of the source preparation.

Implementation (2026-10-28, source only; no live action)

--mode break-glass-drill is in scripts/openbao-isolated-restore/restore_adapter.py, with --mode in preflight.py and --break-glass-drill in run-isolated-restore.sh. The phase A path is unchanged.

• Admission. The v1 admission plus drill: "break-glass", snapshot_label (lowercase object name containing its date) and readback_credential: "generate-root-from-reentered-shares". The attempt id is RPF-WP-0064-T10-*. Phase A and drill admissions are disjoint and refuse each other's mode.
• Allowed surfaces. One added read: GET sys/generate-root/attempt. One added delete: ALLOWED_DELETES = {sys/generate-root/attempt}. The writes sys/generate-root/attempt and sys/generate-root/update were already allowed, as were auth/token/lookup-self and sys/auth. These are unchanged: --network none, the loopback listener, 2700 s, teardown, the reaper and no /data/ reads.
• Order. Phase A restore and unseal. The unseal shares are then wiped. Next come the held-token lookup-self, the generate-root attempt (refused if one is already in progress), the two re-entered shares, the OTP decode in memory, lookup-self = ["root"], GET sys/auth, revoke-self (204: the CLI prints no body) and lookup-self (403). Last, an attempt is started, deleted, and the status shows started: false.
• Held-token outcome. An invalid held token (403) is a finding, valid: false, not an abort.
• Results. The result is drilled (break_glass_drill_complete) or aborted. The new abort reasons are held_token_refused, generate_root_failed, minimal_action_failed, revocation_failed and cancel_drill_failed.
• Receipt. The receipt adds drill, held_root_token, generate_root, revocation, cancel_drill and claim_limits.production_root_token_validity {as_of_snapshot, snapshot_plaintext_sha256}. It also adds three claims_not_made entries. objects is [].
• Verifier. The verifier types every extension field and binds snapshot_plaintext_sha256 to snapshot.plaintext_sha256. It rejects these keys: token, accessor, client_token, otp, nonce, encoded_*, share(s) and unseal_keys*. It also rejects any 20+ character mixed-case alphanumeric run, which covers tokens, accessors, OTPs and encoded tokens.
• Tests. Offline (python3 -m unittest tests.test_openbao_isolated_restore tests.test_openbao_break_glass_drill): 34 pass, 15 of them new in tests/test_openbao_break_glass_drill.py. They use a fake OpenBao and adversarial token, accessor, OTP, nonce, encoded-token and share fixtures. Native (make openbao-isolated-restore-rehearsal): 16 pass, 3 of them new. The new native cases are a synthetic held root token (drilled, valid, ["root"], ttl 0), a fake held token (drilled, valid: false) and a wrong re-entered share (aborted, generate_root_failed). The native rehearsal uses local Docker with --network none, the pinned 2.5.4 image and generated material.

Drill result (2026-10-28)

Attempt RPF-WP-0064-T10-20261027-01, admission sha256:741fa571…ce7a, adapter 3727f8b, founder-run: drilled (break_glass_drill_complete) in 101 s. Receipt sha256:e4591346e99b23e405f1cb69f6f5158dbdb7b5d69b2d1d7709c563a74cc18397 (outside Git, mode 0400) verified PASS in-run and again from a clean clone at 3727f8b. Held initial root token valid as of the 2026-10-14 snapshot (root, ttl 0, no expiry, orphan); fresh generate-root complete with re-entered shares, sys/auth read-only action ok, revoke-self 204, post-revoke lookup 403, cancel drilled; teardown complete, no volumes. The snapshot ciphertext was retrieved again from Storage Share (/dev/shm had been cleared) and the admission was rebuilt from the template with the -03 receipt's snapshot facts. NetKingdom post-event record: net-kingdom/docs/evidence/2026-10-28-king-path-drill.md. Production SSO-down walk (2026-10-28, founder-run, read-only, static-key tunnel plus admin kubeconfig): openbao-0 Running, OpenBao 2.5.4 initialized and unsealed, threshold 2 of 3, HA enabled; break-glass listener 8210 generate-root status 200, main listener 8200 405. Root-token disposition: founder decision 7ce684ec, escrowed until production entry, then revoked with scripts/openbao_revoke_initial_root_token.py (NK-WP-0055-T16). T10 is done.

Founder runbook: drill (isolated restore on the workstation, never production)

Run each block in your own terminal. Nothing below needs the cluster. Snapshot facts are reused from the signed T07 -03 admission (the 2026-10-14 snapshot, plaintext a6312444…590b). If a newer RPF-WP-0066 scheduled snapshot is used instead, put its facts in the admission and its name in snapshot_label.

# 0. Clean checkout at the reviewed commit; offline tests pass
cd ~/railiance-platform && git pull --ff-only && git status --porcelain   # must print nothing
python3 -m unittest tests.test_openbao_isolated_restore tests.test_openbao_break_glass_drill
COMMIT=$(git rev-parse HEAD); echo "$COMMIT"

# 1. Build the drill admission from the T07 -03 admission (same snapshot facts)
W=/dev/shm/rpf64-t10; mkdir -m 700 -p "$W"
T07_ADM=/path/to/RPF-WP-0064-T07-20261024-03.json        # your signed -03 admission
CIPHERTEXT=/path/to/the-2026-10-14-snapshot.age           # same file as T07 (8069449c…8c41, 375618 bytes)
LABEL=openbao-raft-20261014...                            # its object name in railiance-backups/openbao/, lowercase
ID=RPF-WP-0064-T10-$(date -u +%Y%m%d)-01
NB=$(date -u +%Y-%m-%dT%H:%M:00Z); NA=$(date -u -d '+6 hours' +%Y-%m-%dT%H:%M:00Z)
SRC=$(python3 -c 'import importlib.util as u,json;s=u.spec_from_file_location("a","scripts/openbao-isolated-restore/restore_adapter.py");m=u.module_from_spec(s);s.loader.exec_module(m);print(json.dumps(m.source_digests(m.ROOT)))')
jq --arg id "$ID" --arg nb "$NB" --arg na "$NA" --arg c "$COMMIT" --arg label "$LABEL" --argjson src "$SRC" \
  '.attempt_id=$id | .not_before=$nb | .not_after=$na | .adapter_commit=$c | .source_sha256=$src
   | .drill="break-glass" | .snapshot_label=$label | .readback_credential="generate-root-from-reentered-shares"' \
  "$T07_ADM" > "$W/$ID.json"
chmod 600 "$W/$ID.json"; jq '{attempt_id, drill, snapshot_label, not_before, not_after, adapter_commit}' "$W/$ID.json"

# 2. Sign it (founder key in /etc/railiance-openbao-restore/allowed_signers; fixed namespace)
ssh-keygen -Y sign -f ~/.ssh/<founder-signing-key> -n railiance-openbao-restore-attempt "$W/$ID.json"
chmod 600 "$W/$ID.json.sig"
ADM_SHA=sha256:$(sha256sum "$W/$ID.json" | cut -d' ' -f1); echo "$ADM_SHA"

# 3. Run the drill (hidden prompts, in this order: backup age identity; unseal share 1; unseal share 2;
#    held initial root token; share 1 AGAIN; share 2 AGAIN). Clear the clipboard after each paste.
mkdir -m 700 -p ~/rpf64-receipts
bash scripts/openbao-isolated-restore/run-isolated-restore.sh --break-glass-drill \
  "$W/$ID.json" "$W/$ID.json.sig" "$ADM_SHA" "$CIPHERTEXT" ~/rpf64-receipts

# 4. Verify independently from a clean clone at the admitted commit, then show the value-free facts
RECEIPT=$(ls -t ~/rpf64-receipts/rpf64-break-glass-drill-*.json | head -1); chmod 400 "$RECEIPT"
rm -rf /dev/shm/rpf64-verify && git clone -q ~/railiance-platform /dev/shm/rpf64-verify \
  && git -C /dev/shm/rpf64-verify checkout -q "$COMMIT"
python3 /dev/shm/rpf64-verify/scripts/openbao-isolated-restore/verify-restore-receipt.py \
  --receipt "$RECEIPT" --source-root /dev/shm/rpf64-verify --expect-result drilled
jq '{attempt_id, result, reason, elapsed_seconds, restored: {cluster_id_match: .restored.cluster_id_match,
     applied_index: .restored.applied_index}, held_root_token, generate_root, revocation, cancel_drill,
     claim_limits, claims_not_made, teardown}' "$RECEIPT"
rm -rf "$W" /dev/shm/rpf64-verify

Paste back:

• the admission sha256 ($ADM_SHA) and the commit;
• the verifier line, which is PASS: drilled receipt sha256:… or FAIL: <code>;
• the jq output from step 4.

Never paste the receipt path contents beyond that, nor any token, share or OTP. A FAIL or aborted is still evidence: paste the same lines. The receipt stays outside Git (mode 0400).

Founder runbook: production SSO-down walk (read-only)

The walk uses the static-key k3s-api-railiance01 tunnel and the admin kubeconfig. Local port 18200 is taken by a tunnel, so the port-forward maps the break-glass listener 8210 to 18210 and the main listener 8200 to 18201 (RAPP-OPENBAO-WP-0003, live since 2026-10-28). There is no unseal, no generate-root attempt and no token on production.

bridge status | grep k3s-api-railiance01           # up; restore with: bridge up k3s-api-railiance01
export KUBECONFIG=~/.kube/config-railiance01
kubectl -n openbao get pod openbao-0 -o wide
kubectl -n openbao exec openbao-0 -- bao status -format=json \
  | jq '{initialized, sealed, t, n, version, ha_enabled}'          # exit 2 only means sealed
kubectl -n openbao port-forward pod/openbao-0 18210:8210 18201:8200 >/dev/null 2>&1 & PF=$!; sleep 3
curl -sS http://127.0.0.1:18201/v1/sys/seal-status | jq '{initialized, sealed, t, n}'
curl -sS -o /dev/null -w 'break-glass listener generate-root status GET: %{http_code}\n' \
  http://127.0.0.1:18210/v1/sys/generate-root/attempt                # expect 200: ceremony available
curl -sS -o /dev/null -w 'main listener generate-root status GET: %{http_code}\n' \
  http://127.0.0.1:18201/v1/sys/generate-root/attempt                # expect 405: refused without a token
kill "$PF"; unset KUBECONFIG

Paste back the pod line, the two jq outputs and both generate-root status GET codes. Record them in the NK-WP-0055-T13 post-event review record. The GET only reads status and starts nothing.