NK-WP-0055-T13: OpenBao break-glass ceremony with SSO down; RPF-WP-0064-T10 drill plan
Adds the docs/openbao.md section: unseal, value-free held-token check, generate-root with in-memory OTP decode, minimal action, revoke-self with verification, post-event review record. Records that production's listener has unauthenticated generate-root disabled (OpenBao 2.5.4 default) and the owner decision this needs. Commands tested against a mock API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 4515@bnt-lap001 Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354
This commit is contained in:
codex
parent
e3bc1ab32b
commit
8016f56f18
3 changed files with 319 additions and 0 deletions
229
docs/openbao.md
229
docs/openbao.md
|
|
@ -193,6 +193,235 @@ escrow owner through an out-of-band channel. The initial root token is either
|
|||
revoked after a non-root platform-admin token exists or stored as offline
|
||||
break-glass material with the same handling as unseal shares.
|
||||
|
||||
## Break-glass: OpenBao with SSO down
|
||||
|
||||
Use this when KeyCape or Authelia is down and an OpenBao action can't wait. It
|
||||
belongs to NetKingdom NK-WP-0055-T13 and is drilled against an isolated restored
|
||||
OpenBao only (RPF-WP-0064-T10). Never drill it on production.
|
||||
|
||||
**What fails with SSO down.** Operator OIDC logins through `auth/netkingdom`
|
||||
fail, and so does the warden lane. ops-warden SSH certificates fail too,
|
||||
because the warden backend is OpenBao. The OpenBao UI tunnel
|
||||
`openbao-ui-railiance01` is only usable if its SSH credential does not come
|
||||
from warden.
|
||||
|
||||
**What still works.** Cluster access uses the static-key tunnel
|
||||
`k3s-api-railiance01` (as `tegwick`) plus the admin kubeconfig client
|
||||
certificate. Both are founder-workstation material, with an age-sealed copy on
|
||||
the Storage Share (net-kingdom crisis list items 4 and 9). Through that path
|
||||
you can use `kubectl exec` into `openbao-0` and
|
||||
`kubectl -n openbao port-forward pod/openbao-0 8200:8200`.
|
||||
|
||||
**Custody facts.** The mechanism is manual Shamir unseal: 3 shares, threshold
|
||||
2. The founder (platform-root) holds all three shares in the replicated browser
|
||||
password safe, under the `kingpath.credentials` entry (crisis list item 3). The
|
||||
founder also holds the initial root token offline. Its validity is unconfirmed.
|
||||
|
||||
**Rules for every step.**
|
||||
|
||||
- Enter values only at hidden prompts: `read -rs` or OpenBao's own
|
||||
`(will be hidden)` prompt.
|
||||
- Never put a share, token or OTP in argv, an exported environment variable,
|
||||
a file, shell history, chat, State Hub or Git.
|
||||
- Keep these values in unexported shell variables in one terminal and `unset`
|
||||
them at the end.
|
||||
- Clear the clipboard after each paste from the password safe.
|
||||
- Run with `set +o history` (or a leading space with
|
||||
`HISTCONTROL=ignorespace`) as defence in depth.
|
||||
- Output is reduced with `jq` to fields that are not secret. Never print a
|
||||
token, `accessor`, `encoded_token`, `otp` or share.
|
||||
|
||||
### 1. Reach and unseal
|
||||
|
||||
```bash
|
||||
kubectl --context railiance01 -n openbao get pod openbao-0
|
||||
kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{initialized, sealed, t, n, progress}'
|
||||
```
|
||||
|
||||
If `sealed` is `true`, enter two shares. Each one goes into OpenBao's hidden
|
||||
prompt, which needs a TTY (`-ti`):
|
||||
|
||||
```bash
|
||||
kubectl -n openbao exec -ti openbao-0 -- bao operator unseal # share 1 at "Unseal Key (will be hidden):"
|
||||
kubectl -n openbao exec -ti openbao-0 -- bao operator unseal # share 2
|
||||
kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{sealed, progress}'
|
||||
```
|
||||
|
||||
Do not pass a share as an argument (`bao operator unseal <key>`). That puts it
|
||||
in argv on the workstation and in the pod.
|
||||
|
||||
### 2. API session in one terminal
|
||||
|
||||
```bash
|
||||
set +o history
|
||||
kubectl -n openbao port-forward pod/openbao-0 8200:8200 >/dev/null & PF_PID=$!
|
||||
ADDR=http://127.0.0.1:8200
|
||||
bao_api() { # bao_api METHOD PATH [TOKEN_VAR [curl args...]]; JSON body on stdin (</dev/null for none)
|
||||
local m=$1 p=$2 tv=${3:-}; shift $(( $# < 3 ? $# : 3 ))
|
||||
if [ -n "$tv" ]; then
|
||||
curl -sS -X "$m" -H @<(printf 'X-Vault-Token: %s\n' "${!tv}") \
|
||||
-H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
|
||||
else
|
||||
curl -sS -X "$m" -H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
|
||||
fi
|
||||
}
|
||||
```
|
||||
|
||||
`printf` is a shell builtin and the header reaches `curl` through a process
|
||||
substitution (`/dev/fd/N`), so no token appears in argv. `curl -H @file` needs
|
||||
curl 7.55 or later. The body is always read from stdin.
|
||||
|
||||
### 3. Value-free validity check of a held token
|
||||
|
||||
This works for the held initial root token, or any token you have in hand.
|
||||
|
||||
```bash
|
||||
read -rsp 'Token (hidden): ' T; echo
|
||||
bao_api GET auth/token/lookup-self T </dev/null \
|
||||
| jq '{valid: (.data != null), policies: .data.policies, ttl: .data.ttl,
|
||||
expire_time: .data.expire_time, orphan: .data.orphan, errors}'
|
||||
```
|
||||
|
||||
- **A valid initial root token** shows `policies: ["root"]`, `ttl: 0` and
|
||||
`expire_time: null`.
|
||||
- **A revoked or unknown token** shows `valid: false` and
|
||||
`errors: ["permission denied"]`.
|
||||
- **Recording.** Record only that output. Never record `id`, `accessor` or
|
||||
`display_name`.
|
||||
|
||||
A held root token that is still valid is a standing root credential. Once
|
||||
generate-root has been drilled successfully, the founder decides whether to
|
||||
revoke it on production with the step 6 commands.
|
||||
|
||||
To use a valid held token, skip step 4 and use it the way step 5 uses `ROOT`
|
||||
(with `T` in place of `ROOT`). Do not run `revoke-self` on the held token unless
|
||||
you have decided to retire it. Run `unset T` at the end.
|
||||
|
||||
### 4. `generate-root` ceremony
|
||||
|
||||
**Prerequisite: the listener must allow unauthenticated generate-root.**
|
||||
OpenBao 2.x (production runs 2.5.4) disables unauthenticated generate-root by
|
||||
default. The production listener in `rapp-openbao/helm/openbao-values.yaml`
|
||||
does not set `disable_unauthed_generate_root_endpoints = false`. So without a
|
||||
valid token this ceremony is **not available on production today**. The
|
||||
isolated restore adapter enables it on its loopback-only listener, which is
|
||||
why the drill passes there.
|
||||
|
||||
There are two ways to close the gap. Option (b) is the owner decision for
|
||||
rapp-openbao and is recorded in RPF-WP-0064-T10.
|
||||
|
||||
- **(a) At incident time.**
|
||||
1. Add a second listener to the OpenBao config ConfigMap with
|
||||
`kubectl -n openbao edit configmap openbao-config` (confirm the name with
|
||||
`kubectl -n openbao get cm`):
|
||||
`listener "tcp" { address = "127.0.0.1:8210" tls_disable = 1 disable_unauthed_generate_root_endpoints = false }`
|
||||
2. Delete `openbao-0`, unseal again (step 1), and port-forward `8210`.
|
||||
3. Afterwards, revert the ConfigMap and restart.
|
||||
4. If ArgoCD manages the release, it may revert the edit. Suspend sync for
|
||||
the window first.
|
||||
- **(b) Standing.** Declare that loopback-only listener in the rapp-openbao
|
||||
values. It is reachable only through `kubectl port-forward` or `exec`, which
|
||||
already require cluster-admin. Even then, an attempt still needs two shares.
|
||||
|
||||
Run the ceremony against the listener that allows it (set `ADDR`
|
||||
accordingly). The CLI forms are:
|
||||
|
||||
- `bao operator generate-root -status`
|
||||
- `bao operator generate-root -init`: OpenBao generates and prints the OTP.
|
||||
- `bao operator generate-root -init -otp=<otp>` and
|
||||
`bao operator generate-root -generate-otp`: these put the OTP in argv.
|
||||
- `bao operator generate-root -nonce=<nonce>`: prompts for a share, hidden.
|
||||
- `bao operator generate-root -decode=<encoded> -otp=<otp>`: puts both values
|
||||
in argv and prints the token.
|
||||
- `bao operator generate-root -cancel`
|
||||
- PGP mode: `-init -pgp-key=keybase:<user>|<file>`, then
|
||||
`gpg --decrypt` of the base64-decoded `encoded_token`.
|
||||
|
||||
Because `-decode` exposes the encoded token and OTP in argv and prints the
|
||||
token, run the equivalent through the API with the OTP kept in memory:
|
||||
|
||||
```bash
|
||||
bao_api GET sys/generate-root/attempt </dev/null | jq '{started, progress, required}'
|
||||
# started must be false; otherwise cancel a stale attempt:
|
||||
# bao_api DELETE sys/generate-root/attempt </dev/null
|
||||
INIT=$(echo '{}' | bao_api PUT sys/generate-root/attempt) # server generates the OTP
|
||||
NONCE=$(jq -r .nonce <<<"$INIT"); OTP=$(jq -r .otp <<<"$INIT"); unset INIT
|
||||
for i in 1 2; do
|
||||
read -rsp "Unseal share $i (hidden): " K; echo
|
||||
R=$(printf '{"key":"%s","nonce":"%s"}' "$K" "$NONCE" | bao_api PUT sys/generate-root/update)
|
||||
unset K
|
||||
jq '{progress, required, complete, errors}' <<<"$R"
|
||||
done
|
||||
ENC=$(jq -r .encoded_token <<<"$R"); unset R
|
||||
ROOT=$(printf '%s\n%s\n' "$ENC" "$OTP" | python3 -c 'import sys,base64
|
||||
e,o=sys.stdin.read().split(); r=base64.b64decode(e+"="*(-len(e)%4))
|
||||
assert len(r)==len(o); sys.stdout.write(bytes(a^b for a,b in zip(r,o.encode())).decode())')
|
||||
unset ENC OTP NONCE
|
||||
bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{policies: .data.policies, ttl: .data.ttl}' # expect ["root"], 0
|
||||
```
|
||||
|
||||
The share reaches `curl` only on stdin through the `printf` builtin; do not
|
||||
build the body with an external command such as `jq --arg`, which puts it in
|
||||
argv. The decode is the XOR used by
|
||||
`scripts/openbao-isolated-restore/restore_adapter.py` (`generate_credential`),
|
||||
which was proven against OpenBao 2.5.4 in RPF-WP-0064-T07.
|
||||
|
||||
### 5. Minimal action
|
||||
|
||||
Do only the one action the incident needs, with `ROOT` in the header. For
|
||||
example:
|
||||
|
||||
```bash
|
||||
bao_api PUT <path> ROOT </path/to/reviewed-body.json | jq '{errors, warnings}'
|
||||
```
|
||||
|
||||
The body file must hold no secret value. Do not log in to the UI with the
|
||||
root token.
|
||||
|
||||
**Example: fix the operator OIDC path.** To repair
|
||||
`auth/netkingdom/role/platform-admin`, use the rollback body written by
|
||||
`scripts/openbao-platform-admin-aal2.sh rollback-body`. For a workload secret
|
||||
value, read it at a hidden prompt and pipe a `printf`-built body, as in step 4.
|
||||
|
||||
### 6. Revoke and verify
|
||||
|
||||
```bash
|
||||
bao_api POST auth/token/revoke-self ROOT -o /dev/null -w '%{http_code}\n' </dev/null # expect 204
|
||||
bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{revoked: (.errors == ["permission denied"])}' # expect true
|
||||
unset ROOT T; kill "$PF_PID"; set -o history
|
||||
```
|
||||
|
||||
The CLI equivalent of the revoke is `bao token revoke -self`, run with the
|
||||
token in the CLI's token helper, so prefer the API form. If the attempt was
|
||||
abandoned, cancel it with `bao_api DELETE sys/generate-root/attempt </dev/null`
|
||||
and confirm that `started` is `false`. If option (a) was used, revert the
|
||||
listener now.
|
||||
|
||||
### 7. Post-event review record
|
||||
|
||||
Within 24 hours, write a value-free record to
|
||||
`docs/evidence/<date>-openbao-break-glass.json` (or the drill receipt for a
|
||||
drill) and send NetKingdom a State Hub message. Fields:
|
||||
|
||||
- **What:**
|
||||
- each action and API path, with the reviewed body's sha256
|
||||
- whether the pod was unsealed
|
||||
- the token check result from step 3
|
||||
- whether generate-root completed
|
||||
- the HTTP status of `revoke-self`
|
||||
- `revoked: true`
|
||||
- **Why:** the triggering outage and why the OIDC path could not be used.
|
||||
- **Who:** the operator and any witness, by role (`platform-root`, `tegwick`).
|
||||
- **When:** UTC start and end, and the audit-log window. The audit device
|
||||
records HMAC'd tokens only.
|
||||
- **Follow-ups:**
|
||||
- generate a new root token (no standing root)
|
||||
- a decision on the held initial root token
|
||||
- re-sealing of material on the Storage Share if anything moved
|
||||
|
||||
The record never contains a share, token, accessor, OTP, nonce or encoded
|
||||
token.
|
||||
|
||||
## Auto-Unseal via Transit Seal (optional, NET-WP-0020 T4)
|
||||
|
||||
`helm/openbao-values.yaml` carries a commented `seal "transit"` stanza inside
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue