RPF-WP-0064-T10: break-glass drill mode for the isolated restore adapter

Adds --mode break-glass-drill with its own signed admission (drill: "break-glass", attempt id RPF-WP-0064-T10-*, snapshot_label). After the phase A restore and unseal it checks the held root token with lookup-self only, runs a fresh generate-root ceremony with re-entered shares, does one read-only sys/auth action, revokes the generated token (204 then 403) and drills a cancelled attempt. Phase A is unchanged. The verifier types the receipt extension and rejects token, accessor, OTP, nonce, encoded-token and share shapes. Offline 34 and native 16 pass. Founder runbooks for the drill and the read-only production SSO-down walk are in the T10 section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 4515@bnt-lap001 Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354
2026-10-28 00:02:10 +01:00 · 2026-10-28 00:02:10 +01:00 · a2da4085e0
commit a2da4085e0
parent 05ff9f8678
11 changed files with 867 additions and 49 deletions
--- a/workplans/RPF-WP-0064-isolated-original-openbao-restore.md
+++ b/workplans/RPF-WP-0064-isolated-original-openbao-restore.md
@ -8,7 +8,7 @@ status: active
 flavor: implementation
 owner: claude
 created: "2026-10-23"
-updated: "2026-10-23"
+updated: "2026-10-28"
 related:
  - SECRETS-WP-0015
  - NK-WP-0049
@ -588,7 +588,7 @@ packet or of the other three services.

 ```task
 id: RPF-WP-0064-T10
-status: todo
+status: progress
 priority: high
 depends_on: [RPF-WP-0064-T08]
 state_hub_task_id: "5facf182-91fb-58bc-a2dd-c6724049e70f"
@ -673,3 +673,117 @@ the ceremony drill succeeds, or record why it is kept.
 - NetKingdom has a value-free post-event review record and the rapp-openbao listener decision

 No live run is part of the source preparation.
+
+### Implementation (2026-10-28, source only; no live action)
+
+`--mode break-glass-drill` is in `scripts/openbao-isolated-restore/restore_adapter.py`, with `--mode` in
+`preflight.py` and `--break-glass-drill` in `run-isolated-restore.sh`. The phase A path is unchanged.
+
+- **Admission.** The v1 admission plus `drill: "break-glass"`, `snapshot_label` (lowercase object name
+  containing its date) and `readback_credential: "generate-root-from-reentered-shares"`. The attempt id is
+  `RPF-WP-0064-T10-*`. Phase A and drill admissions are disjoint and refuse each other's mode.
+- **Allowed surfaces.** One added read: `GET sys/generate-root/attempt`. One added delete:
+  `ALLOWED_DELETES = {sys/generate-root/attempt}`. The writes `sys/generate-root/attempt` and
+  `sys/generate-root/update` were already allowed, as were `auth/token/lookup-self` and `sys/auth`. These are
+  unchanged: `--network none`, the loopback listener, 2700 s, teardown, the reaper and no `/data/` reads.
+- **Order.** Phase A restore and unseal. The unseal shares are then wiped. Next come the held-token
+  `lookup-self`, the generate-root attempt (refused if one is already in progress), the two re-entered shares,
+  the OTP decode in memory, `lookup-self` = `["root"]`, `GET sys/auth`, `revoke-self` (204: the CLI prints no
+  body) and `lookup-self` (403). Last, an attempt is started, deleted, and the status shows `started: false`.
+- **Held-token outcome.** An invalid held token (403) is a finding, `valid: false`, not an abort.
+- **Results.** The result is `drilled` (`break_glass_drill_complete`) or `aborted`. The new abort reasons are
+  `held_token_refused`, `generate_root_failed`, `minimal_action_failed`, `revocation_failed` and
+  `cancel_drill_failed`.
+- **Receipt.** The receipt adds `drill`, `held_root_token`, `generate_root`, `revocation`, `cancel_drill` and
+  `claim_limits.production_root_token_validity {as_of_snapshot, snapshot_plaintext_sha256}`. It also adds three
+  `claims_not_made` entries. `objects` is `[]`.
+- **Verifier.** The verifier types every extension field and binds `snapshot_plaintext_sha256` to
+  `snapshot.plaintext_sha256`. It rejects these keys: `token`, `accessor`, `client_token`, `otp`, `nonce`,
+  `encoded_*`, `share(s)` and `unseal_keys*`. It also rejects any 20+ character mixed-case alphanumeric run,
+  which covers tokens, accessors, OTPs and encoded tokens.
+- **Tests.** Offline (`python3 -m unittest tests.test_openbao_isolated_restore
+  tests.test_openbao_break_glass_drill`): 34 pass, 15 of them new in `tests/test_openbao_break_glass_drill.py`. They use a fake OpenBao and adversarial token, accessor, OTP,
+  nonce, encoded-token and share fixtures. Native (`make openbao-isolated-restore-rehearsal`): 16 pass, 3 of
+  them new. The new native cases are a synthetic held root token (`drilled`, valid, `["root"]`, ttl 0), a
+  fake held token (`drilled`, `valid: false`) and a wrong re-entered share (`aborted`, `generate_root_failed`).
+  The native rehearsal uses local Docker with `--network none`, the pinned 2.5.4 image and generated material.
+
+### Founder runbook: drill (isolated restore on the workstation, never production)
+
+Run each block in your own terminal. Nothing below needs the cluster. Snapshot facts are reused from the
+signed T07 `-03` admission (the 2026-10-14 snapshot, plaintext `a6312444…590b`). If a newer RPF-WP-0066
+scheduled snapshot is used instead, put its facts in the admission and its name in `snapshot_label`.
+
+```bash
+# 0. Clean checkout at the reviewed commit; offline tests pass
+cd ~/railiance-platform && git pull --ff-only && git status --porcelain   # must print nothing
+python3 -m unittest tests.test_openbao_isolated_restore tests.test_openbao_break_glass_drill
+COMMIT=$(git rev-parse HEAD); echo "$COMMIT"
+
+# 1. Build the drill admission from the T07 -03 admission (same snapshot facts)
+W=/dev/shm/rpf64-t10; mkdir -m 700 -p "$W"
+T07_ADM=/path/to/RPF-WP-0064-T07-20261024-03.json        # your signed -03 admission
+CIPHERTEXT=/path/to/the-2026-10-14-snapshot.age           # same file as T07 (8069449c…8c41, 375618 bytes)
+LABEL=openbao-raft-20261014...                            # its object name in railiance-backups/openbao/, lowercase
+ID=RPF-WP-0064-T10-$(date -u +%Y%m%d)-01
+NB=$(date -u +%Y-%m-%dT%H:%M:00Z); NA=$(date -u -d '+6 hours' +%Y-%m-%dT%H:%M:00Z)
+SRC=$(python3 -c 'import importlib.util as u,json;s=u.spec_from_file_location("a","scripts/openbao-isolated-restore/restore_adapter.py");m=u.module_from_spec(s);s.loader.exec_module(m);print(json.dumps(m.source_digests(m.ROOT)))')
+jq --arg id "$ID" --arg nb "$NB" --arg na "$NA" --arg c "$COMMIT" --arg label "$LABEL" --argjson src "$SRC" \
+  '.attempt_id=$id | .not_before=$nb | .not_after=$na | .adapter_commit=$c | .source_sha256=$src
+   | .drill="break-glass" | .snapshot_label=$label | .readback_credential="generate-root-from-reentered-shares"' \
+  "$T07_ADM" > "$W/$ID.json"
+chmod 600 "$W/$ID.json"; jq '{attempt_id, drill, snapshot_label, not_before, not_after, adapter_commit}' "$W/$ID.json"
+
+# 2. Sign it (founder key in /etc/railiance-openbao-restore/allowed_signers; fixed namespace)
+ssh-keygen -Y sign -f ~/.ssh/<founder-signing-key> -n railiance-openbao-restore-attempt "$W/$ID.json"
+chmod 600 "$W/$ID.json.sig"
+ADM_SHA=sha256:$(sha256sum "$W/$ID.json" | cut -d' ' -f1); echo "$ADM_SHA"
+
+# 3. Run the drill (hidden prompts, in this order: backup age identity; unseal share 1; unseal share 2;
+#    held initial root token; share 1 AGAIN; share 2 AGAIN). Clear the clipboard after each paste.
+mkdir -m 700 -p ~/rpf64-receipts
+bash scripts/openbao-isolated-restore/run-isolated-restore.sh --break-glass-drill \
+  "$W/$ID.json" "$W/$ID.json.sig" "$ADM_SHA" "$CIPHERTEXT" ~/rpf64-receipts
+
+# 4. Verify independently from a clean clone at the admitted commit, then show the value-free facts
+RECEIPT=$(ls -t ~/rpf64-receipts/rpf64-break-glass-drill-*.json | head -1); chmod 400 "$RECEIPT"
+rm -rf /dev/shm/rpf64-verify && git clone -q ~/railiance-platform /dev/shm/rpf64-verify \
+  && git -C /dev/shm/rpf64-verify checkout -q "$COMMIT"
+python3 /dev/shm/rpf64-verify/scripts/openbao-isolated-restore/verify-restore-receipt.py \
+  --receipt "$RECEIPT" --source-root /dev/shm/rpf64-verify --expect-result drilled
+jq '{attempt_id, result, reason, elapsed_seconds, restored: {cluster_id_match: .restored.cluster_id_match,
+     applied_index: .restored.applied_index}, held_root_token, generate_root, revocation, cancel_drill,
+     claim_limits, claims_not_made, teardown}' "$RECEIPT"
+rm -rf "$W" /dev/shm/rpf64-verify
+```
+
+**Paste back:**
+
+- the admission sha256 (`$ADM_SHA`) and the commit;
+- the verifier line, which is `PASS: drilled receipt sha256:…` or `FAIL: <code>`;
+- the `jq` output from step 4.
+
+Never paste the receipt path contents beyond that, nor any token, share or OTP. A `FAIL` or `aborted` is still
+evidence: paste the same lines. The receipt stays outside Git (mode 0400).
+
+### Founder runbook: production SSO-down walk (read-only)
+
+The walk uses the static-key `k3s-api-railiance01` tunnel and the admin kubeconfig. Local port 18200 is taken by
+a tunnel, so the port-forward uses 18210. There is no unseal, no generate-root attempt and no token on
+production.
+
+```bash
+bridge status k3s-api-railiance01                      # up; restore with: bridge up k3s-api-railiance01
+export KUBECONFIG=~/.kube/config-railiance01
+kubectl -n openbao get pod openbao-0 -o wide
+kubectl -n openbao exec openbao-0 -- bao status -format=json \
+  | jq '{initialized, sealed, t, n, version, ha_enabled}'          # exit 2 only means sealed
+kubectl -n openbao port-forward pod/openbao-0 18210:8200 >/dev/null 2>&1 & PF=$!; sleep 3
+curl -sS http://127.0.0.1:18210/v1/sys/seal-status | jq '{initialized, sealed, t, n}'
+curl -sS -o /dev/null -w 'generate-root status GET: %{http_code}\n' \
+  http://127.0.0.1:18210/v1/sys/generate-root/attempt                # 405 = unauthenticated generate-root disabled
+kill "$PF"; unset KUBECONFIG
+```
+
+**Paste back** the pod line, the two `jq` outputs and the `generate-root status GET` code. Record them in the
+NK-WP-0055-T13 post-event review record. The GET only reads status and starts nothing.