--- id: RPF-WP-0045 type: workplan title: "Move openbao-activity-core and openbao-email-connect off expired static tokens to Kubernetes auth" domain: financials repo: railiance-platform status: ready flavor: implementation owner: railiance-platform topic_slug: railiance created: "2026-09-21" updated: "2026-09-21" related: [RPF-WP-0037] state_hub_workstream_id: "0edc6426-9cc8-5bbe-88ea-c27d9862d96e" --- Two ClusterSecretStores on railiance01 fail validation: OpenBao answers 403 on `auth/token/lookup-self` for the static tokens they reference (`openbao-email-connect` since 2026-09-13, `openbao-activity-core` since 2026-09-19). Five ExternalSecrets are `SecretSyncedError`. This plan moves both stores to the method RPF-WP-0037 used for `openbao-forgejo`, `openbao-reuse` and `openbao-target-revenue`. It is an authentication migration, not a credential rotation. No KV value changes. **This plan changes nothing live by itself.** Every live task is `wait` on the founder's go-ahead. T01 is `ADMINISTER @ realm:openbao/railiance01` with the platform-admin credential, `activation=APPROVED`. T03, T05 and T06 are `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. No step needs `BREAK_GLASS`. Evidence is `EvidenceBoundary target-audited` (kubectl status and OpenBao audit log on railiance01). ## Read-only evidence, 2026-09-21 Taken over `ssh railiance01 'kubectl get …'` and `kubectl auth can-i`. No secret value was requested. - Recovered method: `spec.provider.vault.auth.kubernetes`, mountPath `kubernetes`, a **dedicated ServiceAccount in the consumer namespace**, `audiences: [openbao]`, a per-lane OpenBao role. Source: `openbao/eso-auth-recovery/lanes.json`, `scripts/repair_eso_kubernetes_auth.py`. | Store | SA | OpenBao role | Policy | | --- | --- | --- | --- | | openbao-forgejo | forgejo/forgejo-mailer-eso | forgejo-mailer-eso | workload-kv-read-forgejo-mailer-eso | | openbao-reuse | reuse/reuse-surface-eso | reuse-surface-runtime-eso | workload-kv-read-reuse-surface-runtime-eso | | openbao-target-revenue | target-revenue/target-revenue-eso | target-revenue-runtime-eso | workload-kv-read-target-revenue-runtime-eso | Role shape: bound to exactly one SA name and one namespace, audience `openbao`, ttl/max_ttl/explicit_max_ttl 15m, `token_no_default_policy=true`, one policy. Policy: exact `platform/data/...` read plus `auth/token/lookup-self` (read) and `auth/token/revoke-self` (update). All three stores are Ready. - Failing ExternalSecrets and the KV paths they read (live `spec.data[].remoteRef.key`): | Namespace | ExternalSecret | KV path | Target Secret | | --- | --- | --- | --- | | activity-core | actcore-backup-offsite | platform/workloads/railiance/backup/offsite-lane | actcore-runtime-secret (Merge) | | activity-core | actcore-forgejo-admin | platform/workloads/forgejo/forgejo-admin | actcore-runtime-secret (Merge) | | activity-core | actcore-issue-core-runtime | platform/workloads/issue-core/issue-core/issue-core-runtime | actcore-runtime-secret (Merge) | | activity-core | llm-connect-provider-secrets | platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets | llm-connect-provider-secrets (Owner) | | email-connect | email-connect-runtime | platform/workloads/email-connect/transactional | email-connect-runtime | The activity-core four last refreshed on 2026-09-19 (so all four KV paths existed then). Target Secrets are retained with their last synced values; workloads are running on those. - ESO's controller SA `external-secrets/external-secrets` may `create serviceaccount --subresource=token` in `activity-core` and `email-connect` (`kubectl auth can-i` = yes). No new RBAC is needed. - ServiceAccounts `activity-core/activity-core-eso` and `email-connect/email-connect-eso` do not exist yet. - OpenBao's `auth/kubernetes/config` is written by `scripts/openbao-apply-initial-config.sh` with `kubernetes_host` only, so it reviews JWTs with the OpenBao pod's own projected token. No static reviewer JWT that could expire. ## Declared in the repository (this commit) - `openbao/policies/workload-kv-read-activity-core-eso.hcl`: the four data paths above, lookup-self, revoke-self. Nothing else. (The older `external-secrets-activity-core.hcl` also grants metadata reads; it stays for its existing callers and is not attached to the new role.) - `openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl`: one data path, lookup-self, revoke-self. - `openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml`: the two SAs. - `argocd/platform-addons/openbao-secretstore/openbao-{activity-core,email-connect}.clustersecretstore.yaml`: target specs with Kubernetes auth. **These differ from live** until T03. Nothing syncs them (excluded from `kustomization.yaml`; the railiance01 ArgoCD draft for `openbao-secretstore` is pinned to `d2dbc19`). ## T01 OpenBao roles and policies (founder, admin credential) ```task id: RPF-WP-0045-T01 status: wait priority: high state_hub_task_id: "766ec8d9-68de-54ff-9be7-4549383890cc" ``` Waits on the founder's go-ahead. **The only step that needs the OpenBao platform-admin credential**, one attended session, four writes plus read-backs. Run from a checkout of `railiance-platform` at the commit that holds this plan. No token appears on the command line; `warden access` supplies it to the child process. ```sh cd /home/worsch/railiance-platform warden access openbao-platform-admin-login --exec -- sh -euc ' bao policy write workload-kv-read-activity-core-eso \ openbao/policies/workload-kv-read-activity-core-eso.hcl bao policy write workload-kv-read-email-connect-transactional-eso \ openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl bao write auth/kubernetes/role/activity-core-eso \ bound_service_account_names=activity-core-eso \ bound_service_account_namespaces=activity-core \ audience=openbao token_policies=workload-kv-read-activity-core-eso \ token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \ token_no_default_policy=true bao write auth/kubernetes/role/email-connect-transactional-eso \ bound_service_account_names=email-connect-eso \ bound_service_account_namespaces=email-connect \ audience=openbao token_policies=workload-kv-read-email-connect-transactional-eso \ token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \ token_no_default_policy=true bao read auth/kubernetes/role/activity-core-eso bao read auth/kubernetes/role/email-connect-transactional-eso ' ``` Before writing, confirm neither role nor policy exists (`bao read auth/kubernetes/role/` returns 404); if one does, stop and compare rather than overwrite. Done when both read-backs show the fields above. ## T02 Confirm the declared manifests are the ones to apply ```task id: RPF-WP-0045-T02 status: todo priority: medium state_hub_task_id: "27819462-9bae-57ec-95ba-2b90302bd376" ``` Repository and read-only. `ssh railiance01 'kubectl diff -f -' < ` for the SA file and both store manifests (client-side; nothing applied). Expected diff: two new SAs, and in each store only `spec.provider.vault.auth` changes. Any other difference stops T03. ## T03 Apply the ServiceAccounts and switch the two stores (founder go-ahead) ```task id: RPF-WP-0045-T03 status: wait priority: high state_hub_task_id: "ea84219a-d5f4-59da-af83-8b1de8564650" ``` Waits on T01 and the founder's go-ahead. No OpenBao credential. Kubernetes admin on railiance01. ```sh cd /home/worsch/railiance-platform ssh railiance01 'kubectl apply -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml ``` Optional pre-check before the store switch, no admin credential (login with a short-lived SA JWT; the output holds a 15-minute client token, so discard it): ```sh ssh railiance01 'J=$(kubectl -n email-connect create token email-connect-eso --audience=openbao --duration=10m); kubectl -n openbao exec -i openbao-0 -- bao write -field=token_policies auth/kubernetes/login role=email-connect-transactional-eso jwt="$J"' ``` Expected output: `[workload-kv-read-email-connect-transactional-eso]`. Same for `activity-core-eso` in `activity-core`. A wrong namespace or audience must fail. ## T04 Verify ```task id: RPF-WP-0045-T04 status: wait priority: high state_hub_task_id: "8416600e-635e-550b-9e59-72081a3fd7f7" ``` Read-only after T03. Force a refresh, then check. ```sh ssh railiance01 'kubectl annotate clustersecretstore openbao-activity-core openbao-email-connect force-sync=$(date +%s) --overwrite' ssh railiance01 'kubectl -n activity-core annotate externalsecret --all force-sync=$(date +%s) --overwrite; kubectl -n email-connect annotate externalsecret email-connect-runtime force-sync=$(date +%s) --overwrite' ssh railiance01 'kubectl wait --for=condition=Ready clustersecretstore/openbao-activity-core clustersecretstore/openbao-email-connect --timeout=60s' ssh railiance01 'kubectl -n activity-core wait --for=condition=Ready externalsecret --all --timeout=90s; kubectl -n email-connect wait --for=condition=Ready externalsecret/email-connect-runtime --timeout=90s' ssh railiance01 'kubectl get externalsecret -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[0].reason,AT:.status.refreshTime | grep -v SecretSynced$' ``` (The annotate calls are writes to ESO objects; they belong to the T03 go-ahead.) Done when both stores are Ready, the five ExternalSecrets show `SecretSynced` with a refreshTime after T03, and the last command lists no ExternalSecret other than the header. The earlier 37-count had 5 failing; expect 0. ## T05 Rollback (only if T03/T04 fails) ```task id: RPF-WP-0045-T05 status: wait priority: medium state_hub_task_id: "6bccc229-3428-588a-9488-3fd5d5a07e92" ``` The static tokens are already invalid, so rollback cannot restore sync; it restores the prior, known-failing spec. Workload Secrets are unaffected either way (ESO keeps the last synced values). Founder go-ahead: ```sh ssh railiance01 'kubectl patch clustersecretstore openbao-activity-core --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-activity-core-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"' ssh railiance01 'kubectl patch clustersecretstore openbao-email-connect --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-email-connect-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"' ssh railiance01 'kubectl delete -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml ``` OpenBao side (admin credential, only if the roles themselves are wrong): `warden access openbao-platform-admin-login --exec -- sh -euc 'bao delete auth/kubernetes/role/activity-core-eso; bao delete auth/kubernetes/role/email-connect-transactional-eso; bao policy delete workload-kv-read-activity-core-eso; bao policy delete workload-kv-read-email-connect-transactional-eso'`. Then `git revert` this plan's manifest commit. Correct forward in preference, as RPF-WP-0037 does. ## T06 Retire the two static-token Secrets ```task id: RPF-WP-0045-T06 status: wait priority: medium state_hub_task_id: "f56d7096-e005-5dd5-afa0-b1affe247fa7" ``` Waits on T04 passing for 24 hours (a few 1h refreshes) and the founder's go-ahead. First prove nothing references them: ```sh ssh railiance01 'kubectl get clustersecretstores,secretstores -A -o jsonpath="{range .items[*]}{.metadata.name} {.spec.provider.vault.auth.tokenSecretRef.name}{\"\n\"}{end}" | grep -E "openbao-(activity-core|email-connect)-eso-token"' ``` Expected: no output. Both tokens already fail `lookup-self` with 403, so no revocation is needed; the founder may confirm invalidity with the RPF-WP-0037 `cleanup` pattern (the helper reads the value in memory only). Then delete: ```sh ssh railiance01 'kubectl -n external-secrets delete secret openbao-activity-core-eso-token openbao-email-connect-eso-token' ``` Also retire the bootstrap targets in the owning repos: `activity-core` `make openbao-eso-token-apply` / `scripts/openbao-eso-token-apply.sh` and `email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh`, and update `docs/argocd-gitops.md`. Their owners do that in their repos. ## Risks - **Token values in Secret metadata.** Both token Secrets were created with `kubectl apply`, so `kubectl.kubernetes.io/last-applied-configuration` carries the token value. A metadata-only read shows it. These two tokens are already invalid; T06 deletes them. Every other `*-eso-token` Secret in `external-secrets` that was `kubectl apply`-ed (audit-core, core-hub, canned-prompts, sbom-nexus, tenant-engine) is likely the same with **live** tokens. Not inspected. Needs its own finding. - Dependencies of the new method: OpenBao unsealed and reachable (same as today), the Kubernetes auth mount's JWT review with OpenBao's own pod token (auto-rotated, working for the three recovered stores and six more), and ESO's TokenRequest right (present). Nothing found that expires. - The forgejo-admin PAT and the offsite-lane values read here have their own lifetimes (CCR-2026-0006, CCR-2026-0004). A later sync failure on those is a value problem, not this auth problem. - Why the static tokens died is not established (TTL expiry or revocation). Other stores still on static tokens (audit-core, core-hub-database, canned-prompts, sbom-nexus, tenant-engine) may fail the same way. Migrating them is out of scope here.