From b690ce03fa9c1a19e166d6cfb9d79748fae4169d Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 23 Sep 2026 17:47:11 +0200 Subject: [PATCH 1/2] Retire static OpenBao ESO token bootstrap (RPF-WP-0045-T06) The openbao-activity-core store has used OpenBao Kubernetes auth since 2026-09-21 and synced cleanly for more than 24 hours. Remove the openbao-eso-token-apply script and Makefile target, point the manifest comments and runbooks at the Kubernetes-auth store, and guard against reintroducing the script. Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 151606@bnt-lap001 Assistant-Session: 3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48 --- Makefile | 5 +- docs/runbook.md | 2 +- .../15-externalsecret-backup-offsite.yaml | 7 +- .../15-externalsecret-forgejo-admin.yaml | 9 +-- .../15-externalsecret-issue-core.yaml | 5 +- k8s/railiance/README.md | 12 +-- scripts/openbao-eso-token-apply.sh | 80 ------------------- tests/test_openbao_eso_token_contract.py | 20 ++--- 8 files changed, 26 insertions(+), 114 deletions(-) delete mode 100755 scripts/openbao-eso-token-apply.sh diff --git a/Makefile b/Makefile index 23c3cdb..82cda4b 100644 --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ export prod-automation-status refresh-live-images \ dev-up dev-down railiance-up railiance-down \ start-worker start-api start-event-router \ - openbao-eso-token-apply agents-list agents-update agents-validate + agents-list agents-update agents-validate # ── Help ────────────────────────────────────────────────────────────────────── @@ -102,9 +102,6 @@ railiance-up: ## Build image and start full railiance stack (no Elasticsearch) railiance-down: ## Stop and remove railiance stack containers docker compose -f docker-compose.railiance.yml down -openbao-eso-token-apply: ## Mint coulombcore OpenBao ESO token for railiance01 activity-core - OPENBAO_TOKEN_FILE="$(HOME)/.local/openbao/platform-admin.token" bash scripts/openbao-eso-token-apply.sh - # ── Local dev processes ─────────────────────────────────────────────────────── start-worker: ## Start Temporal worker (reads env from .env if present) diff --git a/docs/runbook.md b/docs/runbook.md index 9b04f0d..4318e55 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -585,7 +585,7 @@ mount of `~/railiance-platform` on the worker) with `apply: true` and posts | Retention | newest **3** versions per package (`container`, `pypi`, `npm`, `generic`) | | Org | `coulomb` | | Protected | live cluster image tags + Helm values (`--live-images-file` / live scan) | -| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` via ExternalSecret `actcore-forgejo-admin` (OpenBao `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`). ESO token: `scripts/openbao-eso-token-apply.sh` (includes `workload-kv-read-forgejo-admin`). | +| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` via ExternalSecret `actcore-forgejo-admin` (OpenBao `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`). Store: `ClusterSecretStore/openbao-activity-core` on OpenBao Kubernetes auth (RPF-WP-0045); its role policy includes `workload-kv-read-forgejo-admin`. | | Rollback | restore package versions from Nextcloud `forgejo dump` if a needed tag was removed | **Enabled 2026-07-21** after dry-run + first apply evidence diff --git a/k8s/railiance/15-externalsecret-backup-offsite.yaml b/k8s/railiance/15-externalsecret-backup-offsite.yaml index d36f74d..5a30c89 100644 --- a/k8s/railiance/15-externalsecret-backup-offsite.yaml +++ b/k8s/railiance/15-externalsecret-backup-offsite.yaml @@ -1,9 +1,10 @@ # Sync Option A offsite backup credentials into actcore-runtime-secret. # # Prereqs on railiance01: -# - ClusterSecretStore openbao-activity-core -# - ESO token includes policy workload-kv-read-railiance-backup-offsite-lane -# (scripts/openbao-eso-token-apply.sh defaults updated in RAILIANCE-WP-0016) +# - ClusterSecretStore openbao-activity-core (railiance-platform addon; OpenBao +# Kubernetes auth via ServiceAccount activity-core/activity-core-eso since +# RPF-WP-0045, 2026-09-21 — no static ESO token) +# - Store role policy includes workload-kv-read-railiance-backup-offsite-lane # # OpenBao path: platform/workloads/railiance/backup/offsite-lane # Fields synced (never AGE_PRIVATE_KEY): diff --git a/k8s/railiance/15-externalsecret-forgejo-admin.yaml b/k8s/railiance/15-externalsecret-forgejo-admin.yaml index 310de41..0961f51 100644 --- a/k8s/railiance/15-externalsecret-forgejo-admin.yaml +++ b/k8s/railiance/15-externalsecret-forgejo-admin.yaml @@ -2,11 +2,10 @@ # # Prereqs on railiance01: # - External Secrets Operator (namespace external-secrets) -# - ClusterSecretStore openbao-activity-core (railiance-platform addon) -# - Secret external-secrets/openbao-activity-core-eso-token with policies: -# workload-kv-read-issue-core-runtime -# workload-kv-read-forgejo-admin -# (scripts/openbao-eso-token-apply.sh) +# - ClusterSecretStore openbao-activity-core (railiance-platform addon; OpenBao +# Kubernetes auth via ServiceAccount activity-core/activity-core-eso since +# RPF-WP-0045, 2026-09-21 — no static ESO token) +# - Store role policy includes workload-kv-read-forgejo-admin # # OpenBao path: platform/workloads/forgejo/forgejo-admin # property: API_TOKEN → secret key FORGEJO_TOKEN diff --git a/k8s/railiance/15-externalsecret-issue-core.yaml b/k8s/railiance/15-externalsecret-issue-core.yaml index dcd8732..8ca132d 100644 --- a/k8s/railiance/15-externalsecret-issue-core.yaml +++ b/k8s/railiance/15-externalsecret-issue-core.yaml @@ -2,8 +2,9 @@ # # Prereqs on railiance01: # - External Secrets Operator (namespace external-secrets) -# - ClusterSecretStore openbao-activity-core (railiance-platform addon) -# - Secret external-secrets/openbao-activity-core-eso-token (scripts/openbao-eso-token-apply.sh) +# - ClusterSecretStore openbao-activity-core (railiance-platform addon; OpenBao +# Kubernetes auth via ServiceAccount activity-core/activity-core-eso since +# RPF-WP-0045, 2026-09-21 — no static ESO token) # # OpenBao path: platform/workloads/issue-core/issue-core/issue-core-runtime # property: ISSUE_CORE_API_KEY (shared with issue-core ingestion) diff --git a/k8s/railiance/README.md b/k8s/railiance/README.md index aa1938a..6b1286e 100644 --- a/k8s/railiance/README.md +++ b/k8s/railiance/README.md @@ -57,9 +57,12 @@ ACTIVITY-WP-0039. | `actcore-backup-offsite` | `platform/workloads/railiance/backup/offsite-lane` | `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL` | | `llm-connect-provider-secrets` | `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` | `OPENROUTER_API_KEY` | -Prereqs: `ClusterSecretStore/openbao-activity-core` and ESO token bootstrap with -the four exact-path read policies (`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token -./scripts/openbao-eso-token-apply.sh`). The llm-connect `ExternalSecret` remains +Prereqs: `ClusterSecretStore/openbao-activity-core`, owned by railiance-platform. +Since RPF-WP-0045 (2026-09-21) it uses OpenBao Kubernetes auth through the +`activity-core/activity-core-eso` ServiceAccount, with a 15-minute token and a +role policy limited to exact paths. The static-token bootstrap +(`scripts/openbao-eso-token-apply.sh`) is retired; do not recreate a static +`openbao-activity-core-eso-token`. The llm-connect `ExternalSecret` remains owned by the llm-connect deployment package rather than activity-core. Roll back to audit mode by setting `ISSUE_SINK_TYPE=null` and restarting worker and event-router deployments. See `docs/issue-core-emission-boundary.md`. @@ -97,8 +100,7 @@ ssh railiance01 cd ~/activity-core bash k8s/railiance/bootstrap-secrets.sh kubectl apply -f k8s/railiance/10-infrastructure.yaml -# Bootstrap OpenBao ESO token + apply ExternalSecrets (once per cluster): -OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token ./scripts/openbao-eso-token-apply.sh +# Apply the store (railiance-platform addon, Kubernetes auth) + ExternalSecrets: kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml kubectl apply -f k8s/railiance/15-externalsecret-issue-core.yaml kubectl apply -f k8s/railiance/15-externalsecret-forgejo-admin.yaml diff --git a/scripts/openbao-eso-token-apply.sh b/scripts/openbao-eso-token-apply.sh deleted file mode 100755 index 86e1ed0..0000000 --- a/scripts/openbao-eso-token-apply.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash -# Mint a read-limited OpenBao token on coulombcore (bao.coulomb.social) and store -# it on railiance01 for ClusterSecretStore openbao-activity-core / ExternalSecrets: -# - actcore-issue-core-runtime (ISSUE_CORE_API_KEY) -# - actcore-forgejo-admin (FORGEJO_TOKEN for package prune) -# - actcore-backup-offsite (NC_WEBDAV_* for Option A backups) -# - llm-connect-provider-secrets (OPENROUTER_API_KEY for llm-connect) -# -# Default policies (already present in OpenBao; files re-written when present): -# workload-kv-read-issue-core-runtime -# workload-kv-read-forgejo-admin -# workload-kv-read-railiance-backup-offsite-lane -# workload-kv-read-llm-connect-provider-secrets -set -euo pipefail - -# Space-separated policy names. Override with OPENBAO_ACTIVITY_CORE_POLICIES. -DEFAULT_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane workload-kv-read-llm-connect-provider-secrets" -POLICIES="${OPENBAO_ACTIVITY_CORE_POLICIES:-$DEFAULT_POLICIES}" -POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}" -BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}" -RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}" -SECRET_NAME="${OPENBAO_ACTIVITY_CORE_ESO_SECRET:-openbao-activity-core-eso-token}" -SECRET_NS="${OPENBAO_ACTIVITY_CORE_ESO_NAMESPACE:-external-secrets}" - -if ! command -v bao >/dev/null 2>&1; then - echo "ERROR: bao CLI not found" >&2 - exit 1 -fi - -echo "OpenBao addr: $BAO_ADDR" -echo "Policies: $POLICIES" - -if [[ -n "${BAO_TOKEN:-}" ]]; then - : -elif [[ -n "${OPENBAO_TOKEN_FILE:-}" && -f "${OPENBAO_TOKEN_FILE}" ]]; then - BAO_TOKEN="$(head -n 1 "${OPENBAO_TOKEN_FILE}")" -else - read -r -s -p "OpenBao operator token (coulombcore / bao.coulomb.social): " BAO_TOKEN - echo >&2 -fi - -if [[ -z "${BAO_TOKEN:-}" ]]; then - echo "ERROR: empty OpenBao token" >&2 - exit 1 -fi - -export BAO_ADDR BAO_TOKEN - -health="$(curl -fsS "$BAO_ADDR/v1/sys/health")" -if echo "$health" | grep -q '"sealed":true'; then - echo "ERROR: OpenBao at $BAO_ADDR reports sealed" >&2 - exit 1 -fi - -policy_flags=() -for policy in $POLICIES; do - policy_file="$POLICY_DIR/${policy}.hcl" - if [[ -f "$policy_file" ]]; then - bao policy write "$policy" "$policy_file" - else - echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2 - fi - policy_flags+=(-policy="$policy") -done - -token_json="$(bao token create "${policy_flags[@]}" \ - -display-name="eso-activity-core-runtime" \ - -period=720h \ - -format=json)" -token="$(python3 -c "import json,sys; print(json.load(sys.stdin)['auth']['client_token'])" <<<"$token_json")" - -KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create namespace "$SECRET_NS" --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f - -KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create secret generic "$SECRET_NAME" \ - --namespace "$SECRET_NS" \ - --from-literal=token="$token" \ - --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f - - -unset BAO_TOKEN token token_json -echo "ok: applied $SECRET_NS/$SECRET_NAME on railiance01 (token not printed)" -echo "ok: policies attached: $POLICIES" diff --git a/tests/test_openbao_eso_token_contract.py b/tests/test_openbao_eso_token_contract.py index 3f61aa7..7ff1101 100644 --- a/tests/test_openbao_eso_token_contract.py +++ b/tests/test_openbao_eso_token_contract.py @@ -4,20 +4,12 @@ from pathlib import Path ROOT = Path(__file__).resolve().parents[1] -def test_activity_core_eso_token_covers_every_declared_runtime_lane(): - script = (ROOT / "scripts" / "openbao-eso-token-apply.sh").read_text(encoding="utf-8") - required_policies = { - "workload-kv-read-issue-core-runtime", - "workload-kv-read-forgejo-admin", - "workload-kv-read-railiance-backup-offsite-lane", - "workload-kv-read-llm-connect-provider-secrets", - } - - default_line = next( - line for line in script.splitlines() if line.startswith("DEFAULT_POLICIES=") - ) - for policy in required_policies: - assert policy in default_line +def test_static_eso_token_bootstrap_stays_retired(): + # RPF-WP-0045 moved the store to OpenBao Kubernetes auth; a static token + # would expire again and must not be reintroduced. + assert not (ROOT / "scripts" / "openbao-eso-token-apply.sh").exists() + makefile = (ROOT / "Makefile").read_text(encoding="utf-8") + assert "openbao-eso-token-apply" not in makefile def test_railiance_runbook_applies_and_waits_for_llm_connect_external_secret(): From a129249d258d3ca3e80a995653d1fbc74b593819 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 23 Sep 2026 17:48:51 +0200 Subject: [PATCH 2/2] WP-0039-T03: record option (b) and add silent seeding script The founder chose to mint fresh tokens for both worker identities. Add the reviewed, idempotent, silent script for the attended OpenBao admin lane, and the cutover order that keeps the claim loop's gap to seconds. Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 151606@bnt-lap001 Assistant-Session: 3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48 --- scripts/wp0039-seed-worker-tokens.sh | 40 +++++++++++++++++++ ...multi-worker-identity-and-token-custody.md | 31 ++++++++++++++ 2 files changed, 71 insertions(+) create mode 100755 scripts/wp0039-seed-worker-tokens.sh diff --git a/scripts/wp0039-seed-worker-tokens.sh b/scripts/wp0039-seed-worker-tokens.sh new file mode 100755 index 0000000..efa6764 --- /dev/null +++ b/scripts/wp0039-seed-worker-tokens.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# ACTIVITY-WP-0039-T03 (option b): mint fresh ops_run worker tokens into OpenBao. +# +# Founder-attended only, through the silent admin lane (orientation section 5): +# +# BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \ +# warden access openbao-platform-admin-login --exec -- \ +# sh scripts/wp0039-seed-worker-tokens.sh +# +# Silent by design: warden fails closed on any child output. The value is +# generated inside a pipe and sent to bao on stdin, so it never appears in +# argv, output, Git, or the hub. Idempotent: an existing path is never +# overwritten. Rerunning after success is a no-op that re-verifies. +# +# Exit codes: 0 both paths present and verified · 3 a path exists without a +# usable token field (left untouched) · 4 write failed · 5 verification failed. +exec >/dev/null 2>&1 +set -u + +MOUNT=platform +BASE=workloads/activity-core/ops-run-workers +PATHS="rein-aharness-railiance01 rein-aharness-metered-railiance01" + +token_len() { + bao kv get -mount="$MOUNT" -field=token "$BASE/$1" 2>/dev/null | tr -d '\n' | wc -c +} + +for slug in $PATHS; do + if bao kv metadata get -mount="$MOUNT" "$BASE/$slug"; then + [ "$(token_len "$slug")" -eq 64 ] || exit 3 + continue + fi + openssl rand -hex 32 | tr -d '\n' \ + | bao kv put -mount="$MOUNT" "$BASE/$slug" token=- || exit 4 +done + +for slug in $PATHS; do + [ "$(token_len "$slug")" -eq 64 ] || exit 5 +done +exit 0 diff --git a/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md index 42b4bcd..f6d80dc 100644 --- a/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md +++ b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md @@ -123,6 +123,24 @@ Founder-attended, through `warden access openbao-platform-admin-login --exec` Done when both paths hold a value and no value has been printed or logged. +**Decision 2026-09-23 (founder): option (b).** Both tokens are minted fresh, so +the hand-generated claim-loop value is retired at cutover. The reviewed script +`scripts/wp0039-seed-worker-tokens.sh` does the minting. It runs silently, never +overwrites an existing path, and exits 0 when both paths are verified, 3 on an +unusable existing path, 4 on a write failure, and 5 on a verification failure. +It generates each value inside a pipe and sends it to `bao kv put ... token=-` +on stdin. The founder runs it in their own terminal: + +```bash +BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \ + warden access openbao-platform-admin-login --exec -- \ + sh scripts/wp0039-seed-worker-tokens.sh +``` + +Read warden's printed line rather than its exit code (orientation section 5). +Minting does not depend on T02, but ESO cannot read the paths until the policy +lands. + ## Roll out and prove both identities ```task @@ -143,4 +161,17 @@ Prove four things: - A metered token paired with the loop identity is rejected with HTTP 403. - The previous hand-set Secret key is now owned by ESO. +Cutover order for option (b), which keeps the claim loop's gap to seconds. +The pods read `actcore-runtime-secret` only at start. + +1. Apply `15-externalsecret-worker-tokens.yaml`, then force a refresh. The + Secret now holds the new tokens, and the running API still uses the old one. +2. On railiance01, write the new `ACTIVITY_CORE_WORKER_TOKEN` from the Secret + into `~/.config/rein-aharness/claim-loop.env`. Use a go-template read piped + into the file, and never print the value. +3. Apply `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, then restart + `deployment/actcore-api`. +4. Restart `rein-aharness-claim-loop.service`, and confirm its first poll + returns HTTP 200. + Report the revision to secrets-engine on threads `914853d9` and `6e694682`.