From 8016f56f18c11534dfe7867784e342e8e959db02 Mon Sep 17 00:00:00 2001
From: codex <codex@local>
Date: Tue, 27 Oct 2026 23:31:58 +0100
Subject: [PATCH] NK-WP-0055-T13: OpenBao break-glass ceremony with SSO down;
 RPF-WP-0064-T10 drill plan

Adds the docs/openbao.md section: unseal, value-free held-token check,
generate-root with in-memory OTP decode, minimal action, revoke-self with
verification, post-event review record. Records that production's listener
has unauthenticated generate-root disabled (OpenBao 2.5.4 default) and the
owner decision this needs. Commands tested against a mock API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 4515@bnt-lap001
Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354
---
 WORK-RECORDS.md                               |   1 +
 docs/openbao.md                               | 229 ++++++++++++++++++
 ...-0064-isolated-original-openbao-restore.md |  89 +++++++
 3 files changed, 319 insertions(+)

diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md
index cf2b810..7fe3b66 100644
--- a/WORK-RECORDS.md
+++ b/WORK-RECORDS.md
@@ -204,6 +204,7 @@
 | task | RPF-WP-0064-T07 | done | — | workplans/RPF-WP-0064-isolated-original-openbao-restore.md |
 | task | RPF-WP-0064-T08 | done | — | workplans/RPF-WP-0064-isolated-original-openbao-restore.md |
 | task | RPF-WP-0064-T09 | todo | — | workplans/RPF-WP-0064-isolated-original-openbao-restore.md |
+| task | RPF-WP-0064-T10 | todo | — | workplans/RPF-WP-0064-isolated-original-openbao-restore.md |
 | task | RPF-WP-0065-T01 | done | — | workplans/RPF-WP-0065-netkingdom-role-claim-bindings.md |
 | task | RPF-WP-0065-T02 | done | — | workplans/RPF-WP-0065-netkingdom-role-claim-bindings.md |
 | task | RPF-WP-0065-T03 | done | — | workplans/RPF-WP-0065-netkingdom-role-claim-bindings.md |
diff --git a/docs/openbao.md b/docs/openbao.md
index 054c71d..3df1d6b 100644
--- a/docs/openbao.md
+++ b/docs/openbao.md
@@ -193,6 +193,235 @@ escrow owner through an out-of-band channel. The initial root token is either
 revoked after a non-root platform-admin token exists or stored as offline
 break-glass material with the same handling as unseal shares.
 
+## Break-glass: OpenBao with SSO down
+
+Use this when KeyCape or Authelia is down and an OpenBao action can't wait. It
+belongs to NetKingdom NK-WP-0055-T13 and is drilled against an isolated restored
+OpenBao only (RPF-WP-0064-T10). Never drill it on production.
+
+**What fails with SSO down.** Operator OIDC logins through `auth/netkingdom`
+fail, and so does the warden lane. ops-warden SSH certificates fail too,
+because the warden backend is OpenBao. The OpenBao UI tunnel
+`openbao-ui-railiance01` is only usable if its SSH credential does not come
+from warden.
+
+**What still works.** Cluster access uses the static-key tunnel
+`k3s-api-railiance01` (as `tegwick`) plus the admin kubeconfig client
+certificate. Both are founder-workstation material, with an age-sealed copy on
+the Storage Share (net-kingdom crisis list items 4 and 9). Through that path
+you can use `kubectl exec` into `openbao-0` and
+`kubectl -n openbao port-forward pod/openbao-0 8200:8200`.
+
+**Custody facts.** The mechanism is manual Shamir unseal: 3 shares, threshold
+2. The founder (platform-root) holds all three shares in the replicated browser
+password safe, under the `kingpath.credentials` entry (crisis list item 3). The
+founder also holds the initial root token offline. Its validity is unconfirmed.
+
+**Rules for every step.**
+
+- Enter values only at hidden prompts: `read -rs` or OpenBao's own
+  `(will be hidden)` prompt.
+- Never put a share, token or OTP in argv, an exported environment variable,
+  a file, shell history, chat, State Hub or Git.
+- Keep these values in unexported shell variables in one terminal and `unset`
+  them at the end.
+- Clear the clipboard after each paste from the password safe.
+- Run with `set +o history` (or a leading space with
+  `HISTCONTROL=ignorespace`) as defence in depth.
+- Output is reduced with `jq` to fields that are not secret. Never print a
+  token, `accessor`, `encoded_token`, `otp` or share.
+
+### 1. Reach and unseal
+
+```bash
+kubectl --context railiance01 -n openbao get pod openbao-0
+kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{initialized, sealed, t, n, progress}'
+```
+
+If `sealed` is `true`, enter two shares. Each one goes into OpenBao's hidden
+prompt, which needs a TTY (`-ti`):
+
+```bash
+kubectl -n openbao exec -ti openbao-0 -- bao operator unseal   # share 1 at "Unseal Key (will be hidden):"
+kubectl -n openbao exec -ti openbao-0 -- bao operator unseal   # share 2
+kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{sealed, progress}'
+```
+
+Do not pass a share as an argument (`bao operator unseal <key>`). That puts it
+in argv on the workstation and in the pod.
+
+### 2. API session in one terminal
+
+```bash
+set +o history
+kubectl -n openbao port-forward pod/openbao-0 8200:8200 >/dev/null & PF_PID=$!
+ADDR=http://127.0.0.1:8200
+bao_api() {  # bao_api METHOD PATH [TOKEN_VAR [curl args...]]; JSON body on stdin (</dev/null for none)
+  local m=$1 p=$2 tv=${3:-}; shift $(( $# < 3 ? $# : 3 ))
+  if [ -n "$tv" ]; then
+    curl -sS -X "$m" -H @<(printf 'X-Vault-Token: %s\n' "${!tv}") \
+      -H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
+  else
+    curl -sS -X "$m" -H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
+  fi
+}
+```
+
+`printf` is a shell builtin and the header reaches `curl` through a process
+substitution (`/dev/fd/N`), so no token appears in argv. `curl -H @file` needs
+curl 7.55 or later. The body is always read from stdin.
+
+### 3. Value-free validity check of a held token
+
+This works for the held initial root token, or any token you have in hand.
+
+```bash
+read -rsp 'Token (hidden): ' T; echo
+bao_api GET auth/token/lookup-self T </dev/null \
+  | jq '{valid: (.data != null), policies: .data.policies, ttl: .data.ttl,
+         expire_time: .data.expire_time, orphan: .data.orphan, errors}'
+```
+
+- **A valid initial root token** shows `policies: ["root"]`, `ttl: 0` and
+  `expire_time: null`.
+- **A revoked or unknown token** shows `valid: false` and
+  `errors: ["permission denied"]`.
+- **Recording.** Record only that output. Never record `id`, `accessor` or
+  `display_name`.
+
+A held root token that is still valid is a standing root credential. Once
+generate-root has been drilled successfully, the founder decides whether to
+revoke it on production with the step 6 commands.
+
+To use a valid held token, skip step 4 and use it the way step 5 uses `ROOT`
+(with `T` in place of `ROOT`). Do not run `revoke-self` on the held token unless
+you have decided to retire it. Run `unset T` at the end.
+
+### 4. `generate-root` ceremony
+
+**Prerequisite: the listener must allow unauthenticated generate-root.**
+OpenBao 2.x (production runs 2.5.4) disables unauthenticated generate-root by
+default. The production listener in `rapp-openbao/helm/openbao-values.yaml`
+does not set `disable_unauthed_generate_root_endpoints = false`. So without a
+valid token this ceremony is **not available on production today**. The
+isolated restore adapter enables it on its loopback-only listener, which is
+why the drill passes there.
+
+There are two ways to close the gap. Option (b) is the owner decision for
+rapp-openbao and is recorded in RPF-WP-0064-T10.
+
+- **(a) At incident time.**
+  1. Add a second listener to the OpenBao config ConfigMap with
+     `kubectl -n openbao edit configmap openbao-config` (confirm the name with
+     `kubectl -n openbao get cm`):
+     `listener "tcp" { address = "127.0.0.1:8210" tls_disable = 1 disable_unauthed_generate_root_endpoints = false }`
+  2. Delete `openbao-0`, unseal again (step 1), and port-forward `8210`.
+  3. Afterwards, revert the ConfigMap and restart.
+  4. If ArgoCD manages the release, it may revert the edit. Suspend sync for
+     the window first.
+- **(b) Standing.** Declare that loopback-only listener in the rapp-openbao
+  values. It is reachable only through `kubectl port-forward` or `exec`, which
+  already require cluster-admin. Even then, an attempt still needs two shares.
+
+Run the ceremony against the listener that allows it (set `ADDR`
+accordingly). The CLI forms are:
+
+- `bao operator generate-root -status`
+- `bao operator generate-root -init`: OpenBao generates and prints the OTP.
+- `bao operator generate-root -init -otp=<otp>` and
+  `bao operator generate-root -generate-otp`: these put the OTP in argv.
+- `bao operator generate-root -nonce=<nonce>`: prompts for a share, hidden.
+- `bao operator generate-root -decode=<encoded> -otp=<otp>`: puts both values
+  in argv and prints the token.
+- `bao operator generate-root -cancel`
+- PGP mode: `-init -pgp-key=keybase:<user>|<file>`, then
+  `gpg --decrypt` of the base64-decoded `encoded_token`.
+
+Because `-decode` exposes the encoded token and OTP in argv and prints the
+token, run the equivalent through the API with the OTP kept in memory:
+
+```bash
+bao_api GET sys/generate-root/attempt </dev/null | jq '{started, progress, required}'
+# started must be false; otherwise cancel a stale attempt:
+#   bao_api DELETE sys/generate-root/attempt </dev/null
+INIT=$(echo '{}' | bao_api PUT sys/generate-root/attempt)          # server generates the OTP
+NONCE=$(jq -r .nonce <<<"$INIT"); OTP=$(jq -r .otp <<<"$INIT"); unset INIT
+for i in 1 2; do
+  read -rsp "Unseal share $i (hidden): " K; echo
+  R=$(printf '{"key":"%s","nonce":"%s"}' "$K" "$NONCE" | bao_api PUT sys/generate-root/update)
+  unset K
+  jq '{progress, required, complete, errors}' <<<"$R"
+done
+ENC=$(jq -r .encoded_token <<<"$R"); unset R
+ROOT=$(printf '%s\n%s\n' "$ENC" "$OTP" | python3 -c 'import sys,base64
+e,o=sys.stdin.read().split(); r=base64.b64decode(e+"="*(-len(e)%4))
+assert len(r)==len(o); sys.stdout.write(bytes(a^b for a,b in zip(r,o.encode())).decode())')
+unset ENC OTP NONCE
+bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{policies: .data.policies, ttl: .data.ttl}'   # expect ["root"], 0
+```
+
+The share reaches `curl` only on stdin through the `printf` builtin; do not
+build the body with an external command such as `jq --arg`, which puts it in
+argv. The decode is the XOR used by
+`scripts/openbao-isolated-restore/restore_adapter.py` (`generate_credential`),
+which was proven against OpenBao 2.5.4 in RPF-WP-0064-T07.
+
+### 5. Minimal action
+
+Do only the one action the incident needs, with `ROOT` in the header. For
+example:
+
+```bash
+bao_api PUT <path> ROOT </path/to/reviewed-body.json | jq '{errors, warnings}'
+```
+
+The body file must hold no secret value. Do not log in to the UI with the
+root token.
+
+**Example: fix the operator OIDC path.** To repair
+`auth/netkingdom/role/platform-admin`, use the rollback body written by
+`scripts/openbao-platform-admin-aal2.sh rollback-body`. For a workload secret
+value, read it at a hidden prompt and pipe a `printf`-built body, as in step 4.
+
+### 6. Revoke and verify
+
+```bash
+bao_api POST auth/token/revoke-self ROOT -o /dev/null -w '%{http_code}\n' </dev/null   # expect 204
+bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{revoked: (.errors == ["permission denied"])}'  # expect true
+unset ROOT T; kill "$PF_PID"; set -o history
+```
+
+The CLI equivalent of the revoke is `bao token revoke -self`, run with the
+token in the CLI's token helper, so prefer the API form. If the attempt was
+abandoned, cancel it with `bao_api DELETE sys/generate-root/attempt </dev/null`
+and confirm that `started` is `false`. If option (a) was used, revert the
+listener now.
+
+### 7. Post-event review record
+
+Within 24 hours, write a value-free record to
+`docs/evidence/<date>-openbao-break-glass.json` (or the drill receipt for a
+drill) and send NetKingdom a State Hub message. Fields:
+
+- **What:**
+  - each action and API path, with the reviewed body's sha256
+  - whether the pod was unsealed
+  - the token check result from step 3
+  - whether generate-root completed
+  - the HTTP status of `revoke-self`
+  - `revoked: true`
+- **Why:** the triggering outage and why the OIDC path could not be used.
+- **Who:** the operator and any witness, by role (`platform-root`, `tegwick`).
+- **When:** UTC start and end, and the audit-log window. The audit device
+  records HMAC'd tokens only.
+- **Follow-ups:**
+  - generate a new root token (no standing root)
+  - a decision on the held initial root token
+  - re-sealing of material on the Storage Share if anything moved
+
+The record never contains a share, token, accessor, OTP, nonce or encoded
+token.
+
 ## Auto-Unseal via Transit Seal (optional, NET-WP-0020 T4)
 
 `helm/openbao-values.yaml` carries a commented `seal "transit"` stanza inside
diff --git a/workplans/RPF-WP-0064-isolated-original-openbao-restore.md b/workplans/RPF-WP-0064-isolated-original-openbao-restore.md
index 5e2b12f..d84c99f 100644
--- a/workplans/RPF-WP-0064-isolated-original-openbao-restore.md
+++ b/workplans/RPF-WP-0064-isolated-original-openbao-restore.md
@@ -583,3 +583,92 @@ Named residuals:
 
 None of these blocks T15's OpenBao line. R1/R2 close in phase B. This is not owner acceptance of the full T15
 packet or of the other three services.
+
+## T10 — Break-glass drill: generate-root ceremony and held root-token check (NK-WP-0055-T13)
+
+```task
+id: RPF-WP-0064-T10
+status: todo
+priority: high
+depends_on: [RPF-WP-0064-T08]
+```
+
+Source: NetKingdom NK-WP-0055-T13. The ceremony is written in `docs/openbao.md`, section "Break-glass:
+OpenBao with SSO down". The founder runs this drill against an **isolated restored OpenBao only, never
+production**. KeyCape and Authelia are not used, which simulates SSO being down.
+
+**Drill mode (new adapter mode; the phase A path keeps its current behaviour).**
+
+The drill runs as a separate `--mode break-glass-drill` with its own signed admission
+(`drill: "break-glass"`, attempt id `RPF-WP-0064-T10-<id>`). It does the phase A restore and unseal, then:
+
+1. **Held root-token check.** A hidden prompt takes the founder's held initial root token. The drill calls
+   `auth/token/lookup-self` with it and records only `valid`, `policies`, `ttl == 0`, `expire_time == null`
+   and `orphan`. It never revokes or otherwise uses the held token.
+2. **Fresh ceremony.** A `sys/generate-root/attempt` is started with a server-generated OTP. The founder
+   enters two shares **again** at hidden prompts, so the unseal shares are not reused silently as in phase A.
+   The token is decoded in memory and `lookup-self` must show `["root"]`.
+3. **Minimal action.** One read-only action: `sys/auth` metadata.
+4. **Revoke.** `revoke-self` must return 204, and a follow-up `lookup-self` must return 403.
+5. **Cancelled attempt.** Start an attempt and cancel it (`DELETE sys/generate-root/attempt`), so the
+   cancel step is drilled too.
+
+**What the adapter must allow.**
+
+- **The isolated listener.** The adapter keeps `disable_unauthed_generate_root_endpoints = false` on the
+  `--network none` loopback listener only, as it does today.
+- **`ALLOWED_WRITES`.** `sys/generate-root/attempt` needs both PUT and DELETE, and a second
+  `sys/generate-root/update` sequence must be allowed.
+- **`ALLOWED_READS`.** `GET sys/generate-root/attempt` (status) must be added. `auth/token/lookup-self`
+  must be allowed with the held token and with the generated token.
+- **Prompts.** One extra hidden prompt each for the held token and for the two re-entered shares.
+- **Deadline.** Everything stays inside the 45 minute deadline, with the same teardown.
+- **Unchanged rules.** No network, no KV `/data/` reads, no change to production.
+
+**Receipt fields (all free of secret values).** The drill writes a receipt v2 extension with:
+
+- `drill: "break-glass"`
+- `held_root_token`: `valid`, `policies`, `ttl_zero` and `expire_time_null`
+- `generate_root`: `started`, `shares_entered: 2`, `complete`, `otp_decode_ok`, `policies: ["root"]`,
+  `minimal_action_path` and `minimal_action_ok`
+- `revocation`: `revoke_status: 204` and `post_revoke_lookup: 403`
+- `cancel_drill`: `started_then_cancelled`
+- `claims_not_made`, which additionally lists:
+  - `production_root_token_validity`: the result is valid only as of the snapshot. Name the snapshot (the
+    2026-10-14 snapshot or a newer RPF-WP-0066 scheduled one) and its plaintext sha256.
+  - `production_generate_root_availability`
+  - `sso_down_access_path`
+
+The verifier gets matching adversarial fixtures that reject token, accessor, OTP, nonce, encoded-token and
+share shapes. Offline tests and a native rehearsal use synthetic material and a fake held token.
+
+**Founder-run SSO-down walk (production, read-only).** Over `k3s-api-railiance01` with the admin kubeconfig,
+the founder runs:
+
+- `kubectl -n openbao get pod openbao-0`
+- `bao status` (a seal check)
+- a `kubectl port-forward` reachability check
+
+There is no unseal and no generate-root on production. Record the result in the NK-WP-0055-T13 post-event
+review record.
+
+**Owner decision for rapp-openbao.** The production listener has unauthenticated generate-root disabled
+(OpenBao 2.5.4 default). This decision is a precondition for the ceremony to work on production without a
+valid token. The choices are:
+
+- **(a)** an incident-time ConfigMap edit plus a restart
+- **(b)** a standing loopback-only listener (`127.0.0.1:8210`) reachable only through `kubectl
+  port-forward` or `exec`
+
+**Follow-up (founder decision).** If the held initial root token is valid, revoke it on production after
+the ceremony drill succeeds, or record why it is kept.
+
+**Done when:**
+
+- the drill mode is implemented and reviewed, with tests
+- the native rehearsal passes
+- a founder-signed admission exists for the window
+- the founder has run the drill and the independent verifier passes on the receipt
+- NetKingdom has a value-free post-event review record and the rapp-openbao listener decision
+
+No live run is part of the source preparation.