railiance-platform/docs/openbao.md

OpenBao - Platform Secrets Service

Chart: openbao/openbao Chart version: 0.28.2 App version: v2.5.4 (image tag in helm/openbao-values.yaml) Namespace: openbao Managed by: railiance-platform (S3) Workplan: RAIL-PL-WP-0002 Initial target: Railiance01 (92.205.62.239)

---

Ownership note: the deployable OpenBao package surface now has a wave-1 home in rapp-openbao. Platform custody, policy, credential lanes, delegated automation, and root-of-trust handling remain railiance-platform responsibilities. Use docs/rapp-openbao-compatibility-handoff.md for the migration-window split between package-owned deploy/smoke surfaces and retained S3 bootstrap/governance work.

---

Architecture

S5 workloads / operators
  -> openbao.openbao.svc.cluster.local:8200
       -> openbao-0
            -> integrated Raft storage on local-path PVC
            -> audit storage PVC mounted at /openbao/audit

Platform operators with approved admin identity
  -> bridge up openbao-ui-railiance01
       -> http://127.0.0.1:18200
       -> SSH local forward to the ClusterIP
       -> openbao-ui service
       -> OpenBao UI/API
       -> KeyCape OIDC at https://kc.coulomb.social for login

• OpenBao is the canonical Railiance S3 secrets service.
• SOPS/age remains the Git-at-rest bootstrap mechanism.
• The first Railiance01 deployment is single-replica Raft, not true HA.
• Browser UI exposure is operator-only through the named openbao-ui-railiance01 tunnel. Operators authenticate through KeyCape/OIDC with MFA and the platform-admin role. Do not use the root token through the browser UI.
• kubectl exec and port-forwarding remain valid break-glass/operator paths for maintenance and non-browser verification.
• TLS is disabled inside the pod listener for this internal-only bootstrap. Add cert-manager-backed internal TLS before relying on cluster-internal traffic from untrusted namespaces.

Deployment

The official OpenBao project recommends the Helm chart for Kubernetes deployments and warns to run Helm with --dry-run before install or upgrade.

From a host with kubeconfig access:

make openbao-dry-run
make openbao-deploy
make openbao-status

make openbao-deploy upgrades the OpenBao Helm release, then applies the KeyCape login overlay gateway (helm/openbao-ui-overlay-k8s.yaml) as a ClusterIP-only surface. The chart ingress stays disabled. Public-listener retraction and rollback are separate guarded commands documented in docs/openbao-public-listener-transition.md.

On Railiance01 directly:

cd ~/railiance-platform
sudo env KUBECONFIG=/etc/rancher/k3s/k3s.yaml make openbao-dry-run
sudo env KUBECONFIG=/etc/rancher/k3s/k3s.yaml make openbao-deploy
sudo env KUBECONFIG=/etc/rancher/k3s/k3s.yaml make openbao-status

If the repo is not present on Railiance01 yet, copy only the non-secret values and middleware files, then run Helm directly:

scp helm/openbao-values.yaml tegwick@92.205.62.239:/tmp/openbao-values.yaml
scp helm/openbao-middleware.yaml tegwick@92.205.62.239:/tmp/openbao-middleware.yaml
ssh tegwick@92.205.62.239 \
  'sudo env KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply -f /tmp/openbao-middleware.yaml'
ssh tegwick@92.205.62.239 \
  'sudo env KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm upgrade --install openbao openbao/openbao \
     --version 0.28.2 \
     --namespace openbao \
     --create-namespace \
     -f /tmp/openbao-values.yaml \
     --dry-run'

Repeat without --dry-run to deploy.

Verification

kubectl get pods,svc,pvc -n openbao -o wide
kubectl exec -n openbao openbao-0 -- bao status

Expected immediately after install:

• openbao-0 is Running.
• openbao, openbao-active, openbao-internal, and openbao-ui services exist as cluster-internal services.
• With openbao-ui-railiance01 healthy, http://127.0.0.1:18200 serves the operator UI.
• data and audit PVCs are Bound.
• bao status reports Initialized: false and Sealed: true.

That state is intentional until the bootstrap ceremony is completed. bao status may return exit code 2 while sealed; this is expected for the pre-init state and does not by itself indicate a deployment failure.

Bootstrap Ceremony

Do not initialize OpenBao in a casual shell session. Initialization emits the unseal keys and initial root token. Treat this as a break-glass event.

Setup Operator And King Credential

The initial accountable setup operator/contact is tegwick (bernd.worsch@gmail.com), with Forgejo identity tegwick. This identity can assemble early infrastructure, receive notifications, and operate day-to-day Git/Forgejo workflows, but it is not the desired long-term platform root of trust.

The actual platform-root target is a separate king credential created through the NetKingdom bootstrap path before OpenBao becomes live secret custody. Email may receive notifications, but Forgejo, Git, State Hub, chat, tickets, shell history, and email must not store or transfer OpenBao unseal keys, root tokens, private keys, OTP seeds, recovery codes, or screenshots of secret output.

The canonical custody policy is in net-kingdom/docs/platform-root-custody.md. The preferred production posture is independent two-of-three custody. Temporary single-operator king custody is feasible for pre-production bootstrap only when second-factor protection, offline recovery storage, and a low-friction upgrade path to additional custodians are in place.

Pre-flight checks:

make openbao-status
make openbao-verify

Proceed only when:

• openbao-0 is Running.
• data and audit PVCs are Bound.
• bao status reports Initialized: false and Sealed: true.
• Railiance01 host/cluster backup posture is understood for this maintenance window.
• the guided NetKingdom bootstrap path exists for creating or importing the king credential.
• the OpenBao custody mode is recorded: preferred independent custody, or an explicit temporary single-custodian king bootstrap exception.

Recommended ceremony:

1. Confirm the Railiance01 backup posture first.

2. Prepare the king credential and approved escrow holders or offline single-custody locations.

3. Run initialization once:

   kubectl exec -n openbao openbao-0 -- \
     bao operator init -key-shares=3 -key-threshold=2
   

4. Give each unseal share to its escrow owner or approved king-custody location through an out-of-band channel.

5. Unseal with two shares:

   kubectl exec -n openbao openbao-0 -- bao operator unseal
   

6. Log in with the initial root token only long enough to create durable admin auth, enable audit, and prepare policies.

7. Revoke or tightly escrow the initial root token.

Do not paste unseal keys, root tokens, screenshots, or command output into Git, State Hub, chat, shell history, or issue trackers. Each unseal share goes to one escrow owner through an out-of-band channel. The initial root token is either revoked after a non-root platform-admin token exists or stored as offline break-glass material with the same handling as unseal shares.

Break-glass: OpenBao with SSO down

Use this when KeyCape or Authelia is down and an OpenBao action can't wait. It belongs to NetKingdom NK-WP-0055-T13 and is drilled against an isolated restored OpenBao only (RPF-WP-0064-T10). Never drill it on production. The drill runs as scripts/openbao-isolated-restore/run-isolated-restore.sh --break-glass-drill with its own signed admission; the founder runbook is in the RPF-WP-0064 workplan, T10.

What fails with SSO down. Operator OIDC logins through auth/netkingdom fail, and so does the warden lane. ops-warden SSH certificates fail too, because the warden backend is OpenBao. The OpenBao UI tunnel openbao-ui-railiance01 is only usable if its SSH credential does not come from warden.

What still works. Cluster access uses the static-key tunnel k3s-api-railiance01 (as tegwick) plus the admin kubeconfig client certificate. Both are founder-workstation material, with an age-sealed copy on the Storage Share (net-kingdom crisis list items 4 and 9). Through that path you can use kubectl exec into openbao-0 and kubectl -n openbao port-forward pod/openbao-0 8200:8200.

Custody facts. The mechanism is manual Shamir unseal: 3 shares, threshold 2. The founder (platform-root) holds all three shares in the replicated browser password safe, under the kingpath.credentials entry (crisis list item 3). The founder also holds the initial root token offline. Its validity is unconfirmed.

Rules for every step.

• Enter values only at hidden prompts: read -rs or OpenBao's own (will be hidden) prompt.
• Never put a share, token or OTP in argv, an exported environment variable, a file, shell history, chat, State Hub or Git.
• Keep these values in unexported shell variables in one terminal and unset them at the end.
• Clear the clipboard after each paste from the password safe.
• Run with set +o history (or a leading space with HISTCONTROL=ignorespace) as defence in depth.
• Output is reduced with jq to fields that are not secret. Never print a token, accessor, encoded_token, otp or share.

1. Reach and unseal

kubectl --context railiance01 -n openbao get pod openbao-0
kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{initialized, sealed, t, n, progress}'

If sealed is true, enter two shares. Each one goes into OpenBao's hidden prompt, which needs a TTY (-ti):

kubectl -n openbao exec -ti openbao-0 -- bao operator unseal   # share 1 at "Unseal Key (will be hidden):"
kubectl -n openbao exec -ti openbao-0 -- bao operator unseal   # share 2
kubectl -n openbao exec openbao-0 -- bao status -format=json | jq '{sealed, progress}'

Do not pass a share as an argument (bao operator unseal <key>). That puts it in argv on the workstation and in the pod.

2. API session in one terminal

set +o history
kubectl -n openbao port-forward pod/openbao-0 8200:8200 >/dev/null & PF_PID=$!
ADDR=http://127.0.0.1:8200
bao_api() {  # bao_api METHOD PATH [TOKEN_VAR [curl args...]]; JSON body on stdin (</dev/null for none)
  local m=$1 p=$2 tv=${3:-}; shift $(( $# < 3 ? $# : 3 ))
  if [ -n "$tv" ]; then
    curl -sS -X "$m" -H @<(printf 'X-Vault-Token: %s\n' "${!tv}") \
      -H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
  else
    curl -sS -X "$m" -H 'Content-Type: application/json' --data-binary @- "$@" "$ADDR/v1/$p"
  fi
}

printf is a shell builtin and the header reaches curl through a process substitution (/dev/fd/N), so no token appears in argv. curl -H @file needs curl 7.55 or later. The body is always read from stdin.

3. Value-free validity check of a held token

This works for the held initial root token, or any token you have in hand.

read -rsp 'Token (hidden): ' T; echo
bao_api GET auth/token/lookup-self T </dev/null \
  | jq '{valid: (.data != null), policies: .data.policies, ttl: .data.ttl,
         expire_time: .data.expire_time, orphan: .data.orphan, errors}'

• A valid initial root token shows policies: ["root"], ttl: 0 and expire_time: null.
• A revoked or unknown token shows valid: false and errors: ["permission denied"].
• Recording. Record only that output. Never record id, accessor or display_name.

A held root token that is still valid is a standing root credential. Once generate-root has been drilled successfully, the founder decides whether to revoke it on production with the step 6 commands.

To use a valid held token, skip step 4 and use it the way step 5 uses ROOT (with T in place of ROOT). Do not run revoke-self on the held token unless you have decided to retire it. Run unset T at the end.

4. generate-root ceremony

Prerequisite: the listener must allow unauthenticated generate-root. OpenBao 2.x (production runs 2.5.4) disables unauthenticated generate-root by default. The production listener in rapp-openbao/helm/openbao-values.yaml does not set disable_unauthed_generate_root_endpoints = false. So without a valid token this ceremony is not available on production today. The isolated restore adapter enables it on its loopback-only listener, which is why the drill passes there.

There are two ways to close the gap. Option (b) is the owner decision for rapp-openbao and is recorded in RPF-WP-0064-T10.

• (a) At incident time.
  1. Add a second listener to the OpenBao config ConfigMap with kubectl -n openbao edit configmap openbao-config (confirm the name with kubectl -n openbao get cm): listener "tcp" { address = "127.0.0.1:8210" tls_disable = 1 disable_unauthed_generate_root_endpoints = false }
  2. Delete openbao-0, unseal again (step 1), and port-forward 8210.
  3. Afterwards, revert the ConfigMap and restart.
  4. If ArgoCD manages the release, it may revert the edit. Suspend sync for the window first.
• (b) Standing. Declare that loopback-only listener in the rapp-openbao values. It is reachable only through kubectl port-forward or exec, which already require cluster-admin. Even then, an attempt still needs two shares.

Run the ceremony against the listener that allows it (set ADDR accordingly). The CLI forms are:

• bao operator generate-root -status
• bao operator generate-root -init: OpenBao generates and prints the OTP.
• bao operator generate-root -init -otp=<otp> and bao operator generate-root -generate-otp: these put the OTP in argv.
• bao operator generate-root -nonce=<nonce>: prompts for a share, hidden.
• bao operator generate-root -decode=<encoded> -otp=<otp>: puts both values in argv and prints the token.
• bao operator generate-root -cancel
• PGP mode: -init -pgp-key=keybase:<user>|<file>, then gpg --decrypt of the base64-decoded encoded_token.

Because -decode exposes the encoded token and OTP in argv and prints the token, run the equivalent through the API with the OTP kept in memory:

bao_api GET sys/generate-root/attempt </dev/null | jq '{started, progress, required}'
# started must be false; otherwise cancel a stale attempt:
#   bao_api DELETE sys/generate-root/attempt </dev/null
INIT=$(echo '{}' | bao_api PUT sys/generate-root/attempt)          # server generates the OTP
NONCE=$(jq -r .nonce <<<"$INIT"); OTP=$(jq -r .otp <<<"$INIT"); unset INIT
for i in 1 2; do
  read -rsp "Unseal share $i (hidden): " K; echo
  R=$(printf '{"key":"%s","nonce":"%s"}' "$K" "$NONCE" | bao_api PUT sys/generate-root/update)
  unset K
  jq '{progress, required, complete, errors}' <<<"$R"
done
ENC=$(jq -r .encoded_token <<<"$R"); unset R
ROOT=$(printf '%s\n%s\n' "$ENC" "$OTP" | python3 -c 'import sys,base64
e,o=sys.stdin.read().split(); r=base64.b64decode(e+"="*(-len(e)%4))
assert len(r)==len(o); sys.stdout.write(bytes(a^b for a,b in zip(r,o.encode())).decode())')
unset ENC OTP NONCE
bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{policies: .data.policies, ttl: .data.ttl}'   # expect ["root"], 0

The share reaches curl only on stdin through the printf builtin; do not build the body with an external command such as jq --arg, which puts it in argv. The decode is the XOR used by scripts/openbao-isolated-restore/restore_adapter.py (generate_credential), which was proven against OpenBao 2.5.4 in RPF-WP-0064-T07.

5. Minimal action

Do only the one action the incident needs, with ROOT in the header. For example:

bao_api PUT <path> ROOT </path/to/reviewed-body.json | jq '{errors, warnings}'

The body file must hold no secret value. Do not log in to the UI with the root token.

Example: fix the operator OIDC path. To repair auth/netkingdom/role/platform-admin, use the rollback body written by scripts/openbao-platform-admin-aal2.sh rollback-body. For a workload secret value, read it at a hidden prompt and pipe a printf-built body, as in step 4.

6. Revoke and verify

bao_api POST auth/token/revoke-self ROOT -o /dev/null -w '%{http_code}\n' </dev/null   # expect 204
bao_api GET auth/token/lookup-self ROOT </dev/null | jq '{revoked: (.errors == ["permission denied"])}'  # expect true
unset ROOT T; kill "$PF_PID"; set -o history

The CLI equivalent of the revoke is bao token revoke -self, run with the token in the CLI's token helper, so prefer the API form. If the attempt was abandoned, cancel it with bao_api DELETE sys/generate-root/attempt </dev/null and confirm that started is false. If option (a) was used, revert the listener now.

7. Post-event review record

Within 24 hours, write a value-free record to docs/evidence/<date>-openbao-break-glass.json (or the drill receipt for a drill) and send NetKingdom a State Hub message. Fields:

• What:
  • each action and API path, with the reviewed body's sha256
  • whether the pod was unsealed
  • the token check result from step 3
  • whether generate-root completed
  • the HTTP status of revoke-self
  • revoked: true
• Why: the triggering outage and why the OIDC path could not be used.
• Who: the operator and any witness, by role (platform-root, tegwick).
• When: UTC start and end, and the audit-log window. The audit device records HMAC'd tokens only.
• Follow-ups:
  • generate a new root token (no standing root)
  • a decision on the held initial root token
  • re-sealing of material on the Storage Share if anything moved

The record never contains a share, token, accessor, OTP, nonce or encoded token.

Auto-Unseal via Transit Seal (optional, NET-WP-0020 T4)

helm/openbao-values.yaml carries a commented seal "transit" stanza inside the server config. When an external transit OpenBao (or cloud KMS) is available, enabling it lets pods unseal automatically after restart — no manual share ceremony per restart. Shamir shares become recovery keys and keep the same escrow handling as unseal shares.

Steps:

1. Provision the transit backend and unseal key; store the transit token in a Kubernetes secret referenced through server.extraSecretEnvironmentVars (BAO_SEAL_TRANSIT_TOKEN). The token never enters Git.
2. Uncomment the seal stanza, upgrade the release, and run the seal migration from the attended ceremony posture: bao operator unseal -migrate with threshold shares.
3. Prove auto-unseal: delete the pod, confirm it returns initialized=true sealed=false without shares.
4. In the net-kingdom bootstrap console, select the auto-unseal-transit custody model and set openbao_transit_seal_configured and openbao_auto_unseal_verified in the non-secret metadata.

Initial Configuration After Unseal

File audit is configured declaratively in helm/openbao-values.yaml with a server config audit "file" "file" stanza that writes to /openbao/audit/openbao-audit.log on the audit PVC.

Enable the first KV v2 mount:

kubectl exec -n openbao openbao-0 -- \
  bao secrets enable -path=platform kv-v2

Kubernetes auth, database dynamic credentials, PKI, CSI, and External Secrets integration are follow-up tasks in RAIL-PL-WP-0002. Do not migrate live application secrets until those policies and restore drills are documented.

The repo now includes a non-secret helper for the first post-unseal configuration:

make openbao-configure-initial

The target prompts for a token, verifies the declarative file audit device is visible, enables the platform/ KV v2 mount, enables Kubernetes auth, configures Kubernetes auth from the in-pod service account, and loads:

• openbao/policies/platform-admin.hcl
• openbao/policies/platform-readonly.hcl

It does not print or store the token. You may also set OPENBAO_TOKEN_FILE=/path/to/token-file for an operator-local, uncommitted token file.

OpenBao audit is a production gate. If bao audit list does not show file/, fix the declarative audit stanza or Helm rollout before moving production secrets into OpenBao.

The helper is idempotent. Re-running it should report existing platform/ and kubernetes/ paths as already enabled instead of failing the ceremony.

After the helper succeeds, create a non-root admin token:

kubectl exec -n openbao openbao-0 -- \
  bao token create -policy=platform-admin -period=24h -orphan

Store that token through the approved operator secret path, then revoke or tightly escrow the initial root token. The root token should not become the normal operator credential.

SSH Secrets Engine (ops-warden)

After openbao-configure-initial, enable the SSH user CA used by ops-warden (warden sign via backend: vault). This is NET-WP-0020 T5 / WP-0008 T2 prerequisite.

Declarative artifacts:

• openbao/ssh/roles-spec.yaml — adm-role, agt-role, atm-role TTLs
• openbao/policies/warden-sign.hcl — least-privilege signing policy
• scripts/openbao-apply-ssh-engine.sh — idempotent apply via kubectl exec
• scripts/openbao-verify-ssh-engine.sh — non-mutating verification

Apply (requires platform-admin or equivalent token with ssh/* admin):

mkdir -p ~/.local/openbao
# Store platform-admin token locally (mode 600, never commit):
# echo '<token>' > ~/.local/openbao/platform-admin.token && chmod 600 ~/.local/openbao/platform-admin.token

OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token   OPENBAO_SSH_CA_PUBKEY_OUT=/tmp/openbao-ssh-ca.pub   make openbao-configure-ssh

OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token make openbao-verify-ssh

The apply script exports the CA public key to OPENBAO_SSH_CA_PUBKEY_OUT and updates K8s secret openbao/openbao-ssh-ca-pub (non-secret pubkey only).

Create a dedicated warden signing token (do not use platform-admin daily):

kubectl exec -n openbao openbao-0 --   bao token create -policy=warden-sign -period=8h -orphan

Host trust and principals are railiance-infra scope:

cd ~/railiance-infra
make bootstrap-ssh-ca SSH_CA_PUBKEY=/tmp/openbao-ssh-ca.pub

Then on the workstation: bao login (or export VAULT_TOKEN from the warden-sign token) and run warden sign per ops-warden/wiki/OpenBaoSshEngineChecklist.md.

Auth And Workload Integration

Initial auth model:

Actor	Method	Notes
Setup operator/contact	Forgejo tegwick / bernd.worsch@gmail.com	low-trust assembly and notifications; not platform root of trust
King credential	NetKingdom custody record for dedicated platform-root identity	accountable bootstrap/recovery authority; not a Git or email secret store
Bootstrap operator	one-time root token	only for initial audit, mounts, auth, policies, and non-root token creation
Platform operator	token with platform-admin	temporary until NetKingdom OIDC/admin integration is ready
Read-only reviewer	token with platform-readonly	metadata and health visibility, no secret reads
Kubernetes workload	Kubernetes auth role	namespace/service-account bound, policy per workload
Human identity	NetKingdom IAM Profile/OIDC	target model; OpenBao is not the identity provider
Automation	Kubernetes auth or short-lived operator token	no root tokens in automation

Browser UI Login

The browser operator surface is the named local tunnel:

http://127.0.0.1:18200

Run bridge up openbao-ui-railiance01 first. Operators then see a streamlined Sign in with KeyCape mask. The raw OpenBao fields (namespace, method, mount path, role) are hidden presets applied by the UI overlay in helm/openbao-ui-overlay/. The ClusterIP-only openbao-ui-gateway nginx proxy injects overlay assets and forwards to the OpenBao service.

Hidden defaults (also in helm/openbao-ui-overlay/presets.json):

method: OIDC
namespace: leave blank
mount path: netkingdom
role: platform-admin

Deploy or refresh the overlay:

make openbao-overlay-apply
make openbao-verify-login-overlay
make openbao-verify-login-overlay OPENBAO_VERIFY_LOGIN_OVERLAY_ARGS=--check-upstream-drift

After an OpenBao image or chart upgrade, follow helm/openbao-ui-overlay/README.md to refresh overlay selectors and patches/<version>/manifest.sha256 fingerprints if upstream login markup changed.

OIDC mounts must be visible to the unauthenticated UI listing or Ember falls back to token auth (?with=token). Apply once per cluster:

OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token \
  scripts/openbao-tune-auth-listing.sh

The gateway serves a standalone KeyCape login page at /ui/vault/auth so Ember never handles the bare auth route (avoids ?with=token / ?with=netkingdom/ bounce when OIDC mounts are hidden from the unauthenticated listing). Clicking Sign in with KeyCape calls auth_url and redirects to KeyCape directly. OIDC callbacks under /ui/vault/auth/<mount>/oidc/callback are handled by a standalone page that exchanges the authorization code, stores the UI session token, and redirects into the Ember app (no popup/window.opener flow).

The OpenBao UI redirects the browser to KeyCape at kc.coulomb.social, then returns through the same local tunnel to:

http://127.0.0.1:18200/ui/vault/auth/netkingdom/oidc/callback

The legacy keycape mount remains a compatibility alias for existing operator notes and CLI experiments. The preferred browser mount is netkingdom.

The browser callback URI must be present in both:

• KeyCape openbao-admin client redirect URIs; and
• OpenBao auth/netkingdom/role/platform-admin allowed_redirect_uris.

Declared state of the administrator login (verified live 2026-09-23, docs/evidence/2026-09-23-openbao-platform-admin-check.json):

• role auth/netkingdom/role/platform-admin: openbao/auth/netkingdom-platform-admin-role.json. It is written by railiance-platform only. Since net-kingdom a356f64, its configure-openbao-oidc.sh no longer writes the role. The retired bao.coulomb.social callbacks were pruned on 2026-09-23 (docs/evidence/2026-09-24-platform-admin-callback-prune.json).
• OIDC mount config: openbao/auth/netkingdom-oidc-config.json and openbao/auth/keycape-oidc-config.json, owned by railiance-platform. The compatibility oidc_client_secret is not declared.
• policies openbao/policies/platform-admin.hcl and openbao/policies/operator-custody.hcl, plus the built-in default (token_no_default_policy: false). default grants auth/token/revoke-self and lookup-self, so an attended session can always revoke itself.
• Re-check with the silent attended helper scripts/openbao-platform-admin-check.sh --receipt <new-file>. Run it through scripts/openbao-attended-exec.py. It only reads.

If the compatibility alias is kept enabled, also keep http://127.0.0.1:18200/ui/vault/auth/keycape/oidc/callback in the KeyCape client and auth/keycape/role/platform-admin.

Use the browser UI for metadata inspection and attended operator workflows. Do not use the OpenBao root token through the browser UI. Do not copy secret values, Inter-Hub keys, unseal shares, root tokens, OIDC client secrets, or screenshots of secret values into Git, State Hub, chat, or workplans.

For HF-WP-0001, prefer metadata-only inspection of candidate paths such as:

platform/
platform/operators/
platform/operators/inter-hub/

Workload delivery choice (see also docs/argocd-gitops.md for the GitOps tenant contract):

• Prefer External Secrets Operator for values that should become Kubernetes Secrets consumed by ordinary Helm charts.
• Use CSI-mounted files for workloads that need file references, sharper mount-level boundaries, or secret refresh without rewriting application manifests.
• Do not use the OpenBao injector in the current deployment; the Helm values leave it disabled.
• Application repositories request paths and policies; railiance-platform owns platform mounts, policy shape, and delivery mechanisms.

Path convention:

platform/workloads/<tenant-or-org>/<workload>/<secret-purpose>
platform/object-storage/<consumer>
platform/databases/<consumer>
platform/operators/<purpose>

The template policy for workload KV reads is openbao/policies/workload-kv-read-template.hcl.

Concrete workload access lanes used by ops-warden and similar front doors are tracked in docs/workload-kv-access-lanes.md. These docs carry non-secret path, field, policy, auth-role, and verification pointers only.

Backup, Restore, Audit, And Monitoring

Before any live application secrets move into OpenBao:

1. Confirm file audit is enabled and an audit file is written under /openbao/audit/openbao-audit.log.

2. Create an OpenBao Raft snapshot from the unsealed pod:

   kubectl exec -n openbao openbao-0 -- \
     bao operator raft snapshot save /tmp/openbao-raft.snap
   kubectl cp openbao/openbao-0:/tmp/openbao-raft.snap ./openbao-raft.snap
   

3. Encrypt the snapshot with age/SOPS-compatible custody before it leaves the operator machine.

4. Run an isolated restore drill before treating OpenBao as live secret custody. The drill must prove that a fresh OpenBao instance can restore the snapshot, unseal, and read a test secret. Record only non-secret evidence using docs/openbao-restore-drill-evidence.example.json as a template, replace every placeholder with real drill evidence, then validate it with:

   make openbao-validate-restore-evidence \
     OPENBAO_RESTORE_EVIDENCE=/path/to/evidence.json
   

5. Decide where audit logs are shipped durably. The audit PVC alone is not a durable audit sink. The interim audit-core mock file backend can prove API and setup wiring, but it writes to /tmp and is not production retention.

6. Run:

   make openbao-verify-post-unseal
   

Authenticated verification, after the KeyCape-backed platform-admin path or another approved operator token is available:

make openbao-verify-authenticated

The target prompts for the token without echoing it, never puts the token on the command line, and only runs non-mutating checks. It verifies that bao audit list shows file/, bao secrets list shows platform/, bao auth list shows kubernetes/, netkingdom/, and keycape/, and that the file audit log is non-empty.

If a previous attended OIDC login stored a still-valid token in the pod token helper, use:

make openbao-verify-authenticated OPENBAO_VERIFY_AUTH_ARGS=--use-token-helper

Current durable audit status: the file audit device writes to the audit PVC, which is necessary but not enough for production trust. Before application secrets move into OpenBao, choose and test a durable audit sink beyond that PVC such as an encrypted platform backup/export path or the future centralized logging stack. Do not treat non-secret hashes, screenshots, or State Hub notes as substitutes for retained audit log custody.

Interim integration status: /home/worsch/audit-core provides a mock Audit Core backend that writes JSONL records under /tmp/audit-core/audit-YYYYMMDDTHH.jsonl and deletes files older than seven days. Use it only to wire interfaces and setup validation before the durable Audit Core archive exists.

Emergency seal/unseal drills are disruptive and must only run in an attended window with threshold unseal shares available. Record non-secret drill evidence using docs/openbao-emergency-drill-evidence.example.json as a template, replace every placeholder with real drill evidence, then validate it with:

make openbao-validate-emergency-evidence \
  OPENBAO_EMERGENCY_EVIDENCE=/path/to/evidence.json

Monitoring baseline:

• pod readiness and liveness from Kubernetes probes
• bao status seal/init state
• PVC capacity for data and audit storage
• audit log write success
• future Prometheus scraping once the cluster monitoring stack exists

Artifact-Store Object Storage Handoff

artifact-store is the consumer-facing artifact preservation service for generated outputs, evidence packages, reports, logs, snapshots, exports, and release artifacts. It already has an S3-compatible backend with env:NAME and file:/mounted/path credential references, plus an artifactstore storage verify --backend s3 smoke path.

Railiance should avoid building a parallel object-storage client or credential vending flow in OpenBao. The ownership split is:

• railiance-platform / OpenBao owns bootstrap secret custody, policy, audit, break-glass access, and workload secret delivery.
• artifact-store owns artifact package manifests, the S3 backend, storage verification, and whether temporary credentials require backend refresh support or a sidecar/controller.
• net-kingdom owns the identity issuer and role-claim model if object storage adopts STS with AssumeRoleWithWebIdentity.

Initial static-credential bridge, before STS is proven:

1. Create a scoped object-store access key limited to the artifact-store bucket and prefix. Do not use object-store root credentials.

2. Store the key pair in OpenBao under a platform-owned path such as platform/object-storage/artifact-store.

3. Deliver the values to the artifact-store pod through CSI or External Secrets as mounted files.

4. Configure artifact-store with file references:

   export ARTIFACTSTORE_S3_ACCESS_KEY_REF=file:/run/secrets/artifactstore/s3-access-key
   export ARTIFACTSTORE_S3_SECRET_KEY_REF=file:/run/secrets/artifactstore/s3-secret-key
   

5. Verify from artifact-store:

   artifactstore storage verify --backend s3
   

STS credential vending remains linked to ARTIFACT-STORE-WP-0007 - MinIO Compatibility, MaxIO Fork Assessment, And STS Credential Vending. If that workstream chooses MinIO-compatible AssumeRoleWithWebIdentity, OpenBao should not become the identity provider by default. Use the NetKingdom OIDC issuer for workload/user identity, map object storage roles and policies there, and keep OpenBao responsible for bootstrap, break-glass, audit, and delivery of any controller configuration.

Current artifact-store configuration exposes access key and secret key refs, but no session-token ref. ARTIFACT-STORE-WP-0007-T004 must either add temporary-session-token support to the S3 backend or choose a sidecar/secret controller pattern that keeps refreshed credentials available through the existing env/file reference contract.

Upgrade And Rollback

1. Read the OpenBao chart release notes.
2. Update OPENBAO_CHART_VERSION in Makefile.
3. Run make openbao-dry-run.
4. Confirm current backup and audit log posture.
5. Run make openbao-deploy.
6. Run make openbao-status.

For rollback, run helm rollback openbao <REVISION> -n openbao on Railiance01 and re-check bao status.

Scaling To Three Nodes

When Railiance02 and Railiance03 join:

1. Move storage from local-path to distributed storage.
2. Set server.affinity back to anti-affinity.
3. Set server.ha.replicas: 3.
4. Re-enable a PodDisruptionBudget.
5. Run an unseal, failover, backup, and restore drill before migrating secrets.