railiance-platform/workplans/RPF-WP-0064-isolated-original-openbao-restore.md

id	type	title	domain	repo	status	flavor	owner	created	updated	related	state_hub_workstream_id
RPF-WP-0064	workplan	Isolated original-OpenBao restore for secrets-engine consumer continuity	financials	railiance-platform	active	implementation	claude	2026-10-23	2026-10-23	SECRETS-WP-0015 NK-WP-0049 RPF-WP-0015 RPF-WP-0036 RPF-WP-0035 RAPP-OPENBAO-WP-0001	e19808db-52d6-508b-b70c-6775c4f78c4f

RPF-WP-0064 — Isolated original-OpenBao restore for secrets-engine consumer continuity

Receiving workplan for hub message 41e9af3e from net-kingdom (case secrets15-nk49-readiness). The founder accepted on 2026-10-23 that railiance-platform owns this as source preparation.

NK-WP-0049-T18 (net-kingdom a09bf20, verified receipt sha256 16d72f7c…1651, secrets-engine bfa1b3b) proved in isolation that the original KeyCape issuer, its JWKS key and the archived secrets-engine-openbao registration are intact. SECRETS-WP-0015-T04 also needs the original OpenBao, recovered on its own into an isolated, network-contained target. That recovery uses the original custody and unseal path. It must show the KV registration platform/workloads/keycape/secrets-engine-openbao, the narrow secrets-engine JWT role and the operator auth path. There is no production redirect, no fresh backend standing in for the original and no injected secret.

Scope of this plan: the restore and its evidence only. The secrets-engine consumer run (attended in-memory secret source, KeyCape exchange, OpenBao JWT login, openbao:admin refusal, session revocation) is a separate founder-admitted attempt owned by secrets-engine. This plan prepares its target.

Why not RPF-WP-0036-T03: that task owns recurring backup/restore assurance and waits on the activity-core scheduler (ACTIVITY-WP-0042). This is a one-off attended restore with no scheduler, so a separate plan avoids taking on that dependency. Its evidence can later feed T03 as one isolated restore receipt. RPF-WP-0015 keeps the live snapshot experiments. Its snapshot receipts authorize no restore.

Nothing in this plan claims restore capability or consumer continuity. Every live step stays wait until attended evidence exists.

T01 — Source survey and recorded findings

id: RPF-WP-0064-T01
status: done
priority: high
state_hub_task_id: "02b16f20-8ab3-547b-b8ac-1334139d3450"

Findings (2026-10-23, read-only source survey; no cluster, OpenBao or snapshot access)

1. Snapshot capture and location. The procedure is docs/openbao.md §Backup: bao operator raft snapshot save on openbao-0, then age encryption before the snapshot leaves the operator machine. The latest recorded capture is the attended RPF-WP-0015-T03 hold point on 2026-10-14. That snapshot was encrypted to the offsite recipient derived from the founder's offline railiance-backup.key, decrypt-verified in memory and uploaded to railiance-backups/openbao/ (HTTP 201), and the plaintext was shredded from tmpfs. An earlier 2026-08-23 drill receipt is at reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json. Receipts are validated against docs/openbao-reboot-snapshot-receipt.example.json (24 h freshness, cluster id, applied index). Those receipts authorize no restore.
2. Freshness against the needed objects. The KV path was corrected to platform/workloads/keycape/secrets-engine-openbao on 2026-10-06, and the JWT role was proven live on 2026-10-10 (CCR-2026-0038). The 2026-10-14 snapshot should therefore contain both. This is an inference until the restored metadata readback confirms it. If they are missing, T07 stops; it does not repair anything.
3. Version pin. Chart openbao/openbao 0.28.2 (Makefile OPENBAO_CHART_VERSION); image quay.io/openbao/openbao:2.5.4 (helm/openbao-values.yaml; docs/openbao.md still says app v2.5.3, which is stale doc text). The restore target must use the same image, pinned by digest at rehearsal time. Storage is integrated Raft.
4. Unseal and custody model in use. Shamir 2-of-3, unsealed by the founder in attendance (the 2026-10-14 post-reboot unseal was done outside the transcript). The transit auto-unseal stanza is still commented out. A Raft snapshot restore brings back the original barrier keyring, so the restored target unseals only with the original shares. That is the "original unseal path" the request asks for. Restoring needs a throwaway initialization of the empty target first. Its throwaway keys and token stay in process memory, are never recorded, and stop working once snapshot restore -force completes. This is not a "fresh backend" in the request's sense, because no state from it survives.
5. Role, KV and auth definitions are owned here, not in rapp-openbao. The sources are openbao/auth/secrets-engine-jwt-role.json, openbao/auth/keycape-services-config.json (mount keycape-services), docs/contracts/secrets-engine-openbao-jwt-login.yaml, the operator path openbao/auth/netkingdom-oidc-config.json plus netkingdom-platform-admin-role.json (mount netkingdom, role platform-admin), and the workload KV lane (docs/workload-kv-access-lanes.md, KeyCape provisioning window 2026-10-04). rapp-openbao's SCOPE excludes policy, custody and workload KV lanes. It owns only the package (chart, values, UI overlay), and neither of its workplans has a restore task. rapp-openbao only needs to confirm the image/chart pin used by the target.
6. Isolation constraint found. keycape-services validates JWTs through jwks_url: https://kc.coulomb.social/jwks, and netkingdom uses OIDC discovery at https://kc.coulomb.social. A networkless OpenBao can restore, unseal and read back configuration and metadata. However, a live JWT login (the consumer run) needs that hostname to resolve inside the contained network to the isolated NK49 KeyCape, with a trusted TLS chain. It must never reach the production KeyCape. An interactive OIDC operator login is out of scope for the isolated target. The operator path is proven by restored config and role readback, plus root-less login refusal where applicable.
7. Reusable patterns. These are reused as patterns, not copied:
   • Existing in-repo template docs/openbao-restore-drill-evidence.example.json and its validator make openbao-validate-restore-evidence.
   • net-kingdom docs/bootstrap-recovery-ceremony.md and tools/bootstrap-recovery/isolated-*.py: attended, hidden-key, fail-closed setup with a Docker --network none target, a safe receipt and an independent verifier.
   • secrets-engine docs/founder-admission.md for the founder SSH-signed admission with a time window.

T02 — Isolated target and adapter design

id: RPF-WP-0064-T02
status: done
priority: high
depends_on: [RPF-WP-0064-T01]
state_hub_task_id: "51adc483-5102-5723-ab64-e7a2698befe8"

Design scripts/openbao-isolated-restore/ (adapter plus doc). Fixed properties:

• Target: a disposable Docker container on the founder workstation, running the T01 image digest with an in-container Raft data directory on tmpfs. It is never on railiance01 or the cluster, and it has no kubeconfig, no host networking and no published ports.
• Network containment: phase A (restore + readback) uses --network none. Phase B (handed to the consumer run) uses a single --internal Docker network that has only the isolated KeyCape from NK49 and this OpenBao. In phase B, kc.coulomb.social resolves to the isolated KeyCape, and egress is checked to be refused before admission. Phase B is offered, not run, by this plan.
• Custody inputs, all founder-attended and never through argv, files in the repo, env dumps or logs:
  • (a) the encrypted snapshot object from railiance-backups/openbao/, named by its ciphertext sha256;
  • (b) the offline railiance-backup.key age identity, used for decryption straight into the tmpfs stream;
  • (c) 2 of 3 original Shamir shares, entered through a hidden prompt. The restore needs no root token and no secret value is read. Readback uses a token created in the restored target by an attended generate-root, used in memory only and revoked before teardown, or uses the restored operator path. Which of the two is decided in T02 review.
• Readback (metadata only): sys/seal-status, Raft applied index and cluster id; the mounts keycape-services and netkingdom and their configs (issuer and JWKS URL); role secrets-engine-openbao (bound claims and policies compared to the source JSON hash); platform/workloads/keycape/secrets-engine-openbao KV metadata (versions and timestamps, never data); and policy names. No KV data read.
• Timeout/abort/cleanup: a hard wall-clock limit of 45 minutes from the first custody input. Any fail-closed check, a missing object, an unexpected network interface, or the founder pressing abort leads to docker rm -f, tmpfs unmount, shredding of the decrypted stream, and a recorded aborted receipt. Teardown also runs on success, unless phase B was admitted in the same window. The plaintext snapshot never touches persistent disk.

Delivered (2026-10-23)

scripts/openbao-isolated-restore/ holds run-isolated-restore.sh (the attended launcher), preflight.py (metadata-only, runs before any prompt) and restore_adapter.py (the session plus a reaper). The design document is docs/openbao-isolated-restore.md. Implemented as specified above, with these recorded choices:

• Readback credential: generate-root from the two shares already entered. The OTP is decoded in memory, and the credential is revoked and verified dead before teardown. The operator OIDC path cannot work without a network. The admission names the choice (readback_credential), so the founder confirms it by signing T05.
• OpenBao 2.5 disables unauthenticated generate-root by default. The target enables it on its loopback-only listener, inside --network none. The image's VOLUME paths (/openbao/file, /openbao/logs) are overlaid with tmpfs. That is the only way to get no Docker volumes, and the first native run failed closed on exactly this.
• Custody uses hidden TTY prompts. The age identity sits in a sealed memfd (/dev/fd/N for age), and the decrypted snapshot sits in a sealed memfd streamed to snapshot restore -force /dev/stdin. Shares are held in wiped buffers. No custody value appears in argv, env, files or logs.
• The admission namespace is railiance-openbao-restore-attempt, with anchor /etc/railiance-openbao-restore/allowed_signers. Original material refuses if the anchor is writable by the runtime account, or if PINNED_IMAGE_DIGEST (T06) is unset or differs from the admission.
• The 45-minute deadline is enforced in-session (every subprocess timeout) and by a reaper on parent loss or at hard deadline + 60 s. The reaper removes the container and writes the aborted receipt.
• Phase B is a design stub only (doc §Phase B).

T03 — Safe receipt schema and independent verifier

id: RPF-WP-0064-T03
status: done
priority: high
depends_on: [RPF-WP-0064-T02]
state_hub_task_id: "69ef4a3b-7b0a-5b7b-b49a-c688f9d19554"

Extend docs/openbao-restore-drill-evidence.example.json into a v2 receipt:

• attempt id and admission hash;
• image digest;
• ciphertext sha256 of the snapshot and the source cluster id and applied index (from the RPF-WP-0015 receipt);
• restored cluster id and index;
• a network-namespace proof (interfaces and the egress-refusal result);
• presence and source-hash match for each required object;
• result restored | refused | aborted;
• the teardown proof;
• no_secret_material_recorded.

Add a verifier that runs separately from the adapter. It rejects placeholders, unknown fields, missing teardown, data fields, and anything shaped like a token or share. Validate it with adversarial fixtures.

Delivered (2026-10-23)

• The v2 receipt shape is docs/openbao-restore-receipt-v2.example.json (schema railiance-openbao-restore-receipt/v2, extends the drill template). The admission shape is docs/openbao-restore-admission.example.json. All digests are typed sha256:, and no key name sits next to a bare hex value. gitleaks 8.30.1 finds nothing in any new file.
• The independent verifier is scripts/openbao-isolated-restore/verify-restore-receipt.py, run with make openbao-verify-restore-receipt. It does not import the adapter. It takes a clean checkout at adapter_commit for the source-hash comparison, and accepts --not-after and --expect-result.
• 17 offline tests in tests/test_openbao_isolated_restore.py (make openbao-isolated-restore-test) cover:
  • 28 adversarial receipt fixtures: unknown or missing fields, missing or reaper-only teardown, data fields, KV /data/ paths, credential/share/hex/age/PEM shapes, placeholders, untyped digests, missing claims, deadline overrun, unprotected anchor for original material, result inconsistencies, phase B, source hash/commit mismatch, duplicate keys and the --not-after window;
  • admission refusals and signature namespace/byte binding;
  • launcher and preflight refusals before any prompt.

T04 — Staged rehearsal with synthetic material

id: RPF-WP-0064-T04
status: done
priority: high
depends_on: [RPF-WP-0064-T03]
state_hub_task_id: "9c10caca-d772-5608-954e-6e9f78d59a39"

Run the full adapter against a synthetic OpenBao, initialized on purpose with generated Shamir shares and seeded with stand-in mounts, role and KV metadata that mirror the T01 source files. Snapshot it, age-encrypt it to a generated identity, restore it into the T02 target, unseal it with the synthetic shares, read it back, and tear it down. Rehearse each abort path (timeout, wrong ciphertext hash, network interface present, missing object). This step uses no real material.

Rehearsal result (2026-10-23): native, generated material only

make openbao-isolated-restore-rehearsal ran 12 tests, all passing, in about 156 s, against the real quay.io/openbao/openbao@sha256:436eaf97…c115 image (the local pull of tag 2.5.4; T06 still confirms the production digest). Every container ran with --network none and tmpfs storage.

Each run generates its own material:

• an SSH admission key and anchor;
• an age identity;
• a 3-share, threshold-2 Shamir init;
• root credentials;
• a KV value.

A source OpenBao is seeded from the real source JSON/HCL shapes. The JWT and OIDC configs are written with skip_jwks_validation, because the source has no network. That source is snapshotted, age-encrypted, restored through the adapter into a fresh target, unsealed with shares 1 and 3, read back, and torn down. Each receipt is checked by the independent verifier, and its text is checked to hold none of the generated credentials, shares, identity or KV value.

Native coverage:

• restored end to end: sealed after restore, original shares unseal it, cluster id and index match, the throwaway init is invalidated, the readback credential is revoked;
• wrong age identity (aborts before any target exists);
• wrong ciphertext hash;
• wrong plaintext hash;
• wrong share (unseal_refused, target removed);
• missing JWT role and KV registration (refused/object_missing, not repaired);
• source drift (refused/object_mismatch);
• in-session timeout (deadline_exceeded);
• operator Ctrl-C (operator_abort);
• parent loss (the reaper removes a real container and writes parent_lost);
• the reaper hard deadline (kills the session, deadline_exceeded).

Mocked (reduced coverage): only the network-interface abort. A wrapper around docker adds a bridge entry to the inspect output of a real networkless container. Attaching a real network is out of bounds. The adapter aborts network_containment_failed, and the verifier refuses any receipt that claims such a network.

Not covered: the founder TTY prompt itself, run-isolated-restore.sh past its pre-prompt refusals, and preflight.py past admission refusal. All three need a protected root-owned anchor and the T06 pin. The rehearsal drives the same Session with injected prompts.

After the run, no rpf64-* container, Docker volume or /dev/shm workdir remained, and all generated material was deleted.

Retained rehearsal receipt (synthetic material, outside Git): ~/rpf64-rehearsal-20261023/rehearsal-receipt.json on the workstation, sha256:5764de67e6ed1a998838ace7843f96b3be0e4da37c8e17b8bb078372245feeae. It is bound to adapter_commit a61a07cf158320bf5fe4eb3b8f50dc8ced9c0714, and its result is restored/all_required_objects_match. Re-verify it with:

make openbao-verify-restore-receipt RECEIPT=<path> SOURCE_ROOT=<clean checkout at a61a07c>

T05 — Founder-signed attempt admission

id: RPF-WP-0064-T05
status: wait
priority: high
needs_human: true
depends_on: [RPF-WP-0064-T04]
blocking_reason: "Founder signs (ssh-keygen -Y sign, namespace railiance-openbao-restore-attempt; allowed_signers at a root-owned /etc/railiance-openbao-restore/) the v1 admission in docs/openbao-restore-admission.example.json: attempt id, phase A only, window (<=24h) + 2700 s deadline, abort operator, T06 image digest, adapter commit + source_sha256 closure, snapshot ciphertext/plaintext sha256 + bytes + age recipient + source cluster id/applied index + source receipt sha256 (RPF-WP-0015-T03), unseal 2-of-3, readback via generate-root from entered shares. T04 rehearsal done; waits on T06 pin."
state_hub_task_id: "a954eaf9-855b-5458-9f13-c6cfed289934"

2026-10-23: the founder accepted the three T02 design choices before signing: (1) readback through a temporary root credential generated from the two entered shares, revoked and checked dead before teardown; (2) unauthenticated generate-root enabled only on the target's own loopback listener inside the networkless container; (3) strict match of restored role/policy objects against current source, so a snapshot older than a later source edit yields refused. Signing T05 still confirms them per attempt.

T06 — Owner confirmation of package pin

id: RPF-WP-0064-T06
status: done
priority: low
depends_on: [RPF-WP-0064-T02]
state_hub_task_id: "cb689628-9a5a-5f2b-a3ec-be37bd6cdb37"

2026-10-24: done from the live runtime. The founder read openbao-0 read-only: spec image quay.io/openbao/openbao:2.5.4, container imageID docker.io/openbao/openbao@sha256:436eaf9778cad75507ff70ea26ace30dcbe15606e619ac3823495663d7f7c115, equal to the T04 rehearsal pull (digests are registry-independent). Written to PINNED_IMAGE_DIGEST; the offline suite (17) passes. This is the current production digest. That the same digest ran at the 2026-10-14 snapshot is not separately proven; the attended run will refuse on any restore incompatibility rather than repair. rapp-openbao was informed and may object; no rapp-openbao change requested.

T07 — Attended original restore (phase A)

id: RPF-WP-0064-T07
status: done
priority: high
needs_human: true
depends_on: [RPF-WP-0064-T05, RPF-WP-0064-T06]
state_hub_task_id: "96e88167-c484-527c-84aa-0464dd0f5597"

2026-10-24 attempt RPF-WP-0064-T07-20261024-01 (admission 796414c2…a3b9, adapter 1701a3f, window 08:00Z–14:00Z): aborted, reason unseal_refused, after 168 s; the receipt 1a7de4a4…abe6 (kept outside Git) was independently verified from a clean clone at 1701a3f as a genuine aborted result. It proved the off-host ciphertext 8069449c…8c41 (375618 bytes) intact, decryption with the offline age identity, plaintext a6312444…590b, and that the founder's two original Shamir shares unseal the restored barrier (unsealed_with_original_shares: true); networkless containment and full teardown held, with no volumes. The abort came after unseal: sys/leader never reported an active self with an applied index within 60 s, and the adapter reused the unseal_refused reason. The first hypothesis (target raft node id isolated-restore versus production voter openbao-0) was disproved by a native rehearsal, which restored and elected a leader anyway. The adapter now aligns the node id with production, waits up to 180 s, aborts as leader_not_elected, and records bounded typed sys/leader/sys/seal-status fields and a redacted first error line in restored.leader_wait. Offline 19 and native 12 pass. A new -02 admission is needed for the new adapter commit.

2026-10-24 attempt -02 (admission 33c8aab2…75ab, adapter 624843e): two runs refused at the first prompt (custody_identity_refused, clipboard content; the key file and a shape-only paste check were correct), then a third run reached aborted, leader_not_elected after 236 s. Its leader_wait diagnostics showed the cause: after the original shares unsealed the node, seal-status read sealed: true and sys/leader returned API errors for 180 s. Production declares a file audit device at /openbao/audit/openbao-audit.log (rapp-openbao values); the restored audit table re-enables it on unseal, the target had no writable /openbao/audit, and OpenBao sealed itself again. Both -02 aborted receipts (cfb2c2af…, 4c5cd362…, 904a9d95… incl. the refusals) verified independently from a clean clone at 624843e. Fix: an in-memory /openbao/audit on the target. The rehearsal source now carries production's file audit device, and a native regression case without that mount reproduces the reseal (leader_not_elected, sealed: true), while the full restore passes with it. Offline 19 and native 13 pass. A -03 admission is needed for the new adapter commit.

2026-10-24 attempt -03 (admission 3ef17fd1…4cc9, adapter 7084b57, founder attended): restored, all_required_objects_match in 46 s. Independently verified from a clean clone at 7084b57: receipt sha256:4e048deedda21aa35ea4ff811e437f9c0e546fbefe4eb8a2025a74bf65e2579d, retained outside Git on the founder workstation (mode 0400). Original cluster id fd28df5d… matched, applied index 6040604 (not behind 6040594), leader elected in 2 s, the throwaway init was invalidated, the original shares unsealed, and all nine required objects were present and equal to the committed sources: auth mounts, secrets mounts, keycape-services JWT config and role secrets-engine-openbao, netkingdom OIDC config and role platform-admin, policy secrets-engine-login-self, and KV metadata platform/workloads/keycape/secrets-engine-openbao (no KV data read). The readback credential was revoked; networkless (lo only), no volumes, full teardown. Claims not made: consumer continuity, live JWT login, phase B network, owner acceptance, recurring restore assurance. T08 handoff: recorded in secrets-engine SECRETS-WP-0015-T04 and net-kingdom NK-WP-0049-T18 the same day. RPF-WP-0036-T03 may point to this receipt as a one-off; it is not recurring assurance.

Run the admitted adapter once. The outcome is a T03 receipt verified independently. The receipt is kept outside Git and referenced by its hash. A missing object or failed check is a valid refused outcome; it is not repaired in this attempt.

T08 — Bounded evidence handoff

id: RPF-WP-0064-T08
status: done
priority: medium
depends_on: [RPF-WP-0064-T07]
state_hub_task_id: "94139cb1-037a-58ef-883c-dde71fe6bfde"

Hand to secrets-engine (SECRETS-WP-0015-T04) and net-kingdom (NK-WP-0049-T18) the receipt hash, the verifier result, the image digest and the recovered-object list. Also hand over the phase B offer terms (contained network with the isolated KeyCape, same custody, its own admission). The handoff makes no continuity claim. A pointer to the receipt goes to RPF-WP-0036-T03 as one isolated restore receipt; it does not count as recurring assurance.