railiance-platform/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md

id	type	title	domain	repo	status	flavor	owner	topic_slug	created	updated	related	state_hub_workstream_id
RPF-WP-0045	workplan	Move openbao-activity-core and openbao-email-connect off expired static tokens to Kubernetes auth	financials	railiance-platform	active	implementation	railiance-platform	railiance	2026-09-21	2026-09-21	RPF-WP-0037	0edc6426-9cc8-5bbe-88ea-c27d9862d96e

Two ClusterSecretStores on railiance01 fail validation: OpenBao answers 403 on auth/token/lookup-self for the static tokens they reference (openbao-email-connect since 2026-09-13, openbao-activity-core since 2026-09-19). Five ExternalSecrets are SecretSyncedError. This plan moves both stores to the method RPF-WP-0037 used for openbao-forgejo, openbao-reuse and openbao-target-revenue. It is an authentication migration, not a credential rotation. No KV value changes.

This plan changes nothing live by itself. Every live task is wait on the founder's go-ahead. T01 is ADMINISTER @ realm:openbao/railiance01 with the platform-admin credential, activation=APPROVED. T03, T05 and T06 are ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED. No step needs BREAK_GLASS. Evidence is EvidenceBoundary target-audited (kubectl status and OpenBao audit log on railiance01).

Read-only evidence, 2026-09-21

Taken over ssh railiance01 'kubectl get …' and kubectl auth can-i. No secret value was requested.

• Recovered method: spec.provider.vault.auth.kubernetes, mountPath kubernetes, a dedicated ServiceAccount in the consumer namespace, audiences: [openbao], a per-lane OpenBao role. Source: openbao/eso-auth-recovery/lanes.json, scripts/repair_eso_kubernetes_auth.py.

  Store	SA	OpenBao role	Policy
  openbao-forgejo	forgejo/forgejo-mailer-eso	forgejo-mailer-eso	workload-kv-read-forgejo-mailer-eso
  openbao-reuse	reuse/reuse-surface-eso	reuse-surface-runtime-eso	workload-kv-read-reuse-surface-runtime-eso
  openbao-target-revenue	target-revenue/target-revenue-eso	target-revenue-runtime-eso	workload-kv-read-target-revenue-runtime-eso

  Role shape: bound to exactly one SA name and one namespace, audience openbao, ttl/max_ttl/explicit_max_ttl 15m, token_no_default_policy=true, one policy. Policy: exact platform/data/... read plus auth/token/lookup-self (read) and auth/token/revoke-self (update). All three stores are Ready.

• Failing ExternalSecrets and the KV paths they read (live spec.data[].remoteRef.key):

  Namespace	ExternalSecret	KV path	Target Secret
  activity-core	actcore-backup-offsite	platform/workloads/railiance/backup/offsite-lane	actcore-runtime-secret (Merge)
  activity-core	actcore-forgejo-admin	platform/workloads/forgejo/forgejo-admin	actcore-runtime-secret (Merge)
  activity-core	actcore-issue-core-runtime	platform/workloads/issue-core/issue-core/issue-core-runtime	actcore-runtime-secret (Merge)
  activity-core	llm-connect-provider-secrets	platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets	llm-connect-provider-secrets (Owner)
  email-connect	email-connect-runtime	platform/workloads/email-connect/transactional	email-connect-runtime

  The activity-core four last refreshed on 2026-09-19 (so all four KV paths existed then). Target Secrets are retained with their last synced values; workloads are running on those.

• ESO's controller SA external-secrets/external-secrets may create serviceaccount --subresource=token in activity-core and email-connect (kubectl auth can-i = yes). No new RBAC is needed.

• ServiceAccounts activity-core/activity-core-eso and email-connect/email-connect-eso do not exist yet.

• OpenBao's auth/kubernetes/config is written by scripts/openbao-apply-initial-config.sh with kubernetes_host only, so it reviews JWTs with the OpenBao pod's own projected token. No static reviewer JWT that could expire.

Declared in the repository (this commit)

• openbao/policies/workload-kv-read-activity-core-eso.hcl: the four data paths above, lookup-self, revoke-self. Nothing else. (The older external-secrets-activity-core.hcl also grants metadata reads; it stays for its existing callers and is not attached to the new role.)
• openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl: one data path, lookup-self, revoke-self.
• openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml: the two SAs.
• argocd/platform-addons/openbao-secretstore/openbao-{activity-core,email-connect}.clustersecretstore.yaml: target specs with Kubernetes auth. These differ from live until T03. Nothing syncs them (excluded from kustomization.yaml; the railiance01 ArgoCD draft for openbao-secretstore is pinned to d2dbc19).

T01 OpenBao roles and policies (founder, admin credential)

id: RPF-WP-0045-T01
status: done
priority: high
state_hub_task_id: "766ec8d9-68de-54ff-9be7-4549383890cc"

Waits on the founder's go-ahead. The only step that needs the OpenBao platform-admin credential, one attended session, four writes plus read-backs. Run from a checkout of railiance-platform at the commit that holds this plan. No token appears on the command line; warden access supplies it to the child process.

cd /home/worsch/railiance-platform
warden access openbao-platform-admin-login --exec -- sh -euc '
  bao policy write workload-kv-read-activity-core-eso \
    openbao/policies/workload-kv-read-activity-core-eso.hcl
  bao policy write workload-kv-read-email-connect-transactional-eso \
    openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl
  bao write auth/kubernetes/role/activity-core-eso \
    bound_service_account_names=activity-core-eso \
    bound_service_account_namespaces=activity-core \
    audience=openbao token_policies=workload-kv-read-activity-core-eso \
    token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
    token_no_default_policy=true
  bao write auth/kubernetes/role/email-connect-transactional-eso \
    bound_service_account_names=email-connect-eso \
    bound_service_account_namespaces=email-connect \
    audience=openbao token_policies=workload-kv-read-email-connect-transactional-eso \
    token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
    token_no_default_policy=true
  bao read auth/kubernetes/role/activity-core-eso
  bao read auth/kubernetes/role/email-connect-transactional-eso
'

Before writing, confirm neither role nor policy exists (bao read auth/kubernetes/role/<name> returns 404); if one does, stop and compare rather than overwrite. Done when both read-backs show the fields above.

T02 Confirm the declared manifests are the ones to apply

id: RPF-WP-0045-T02
status: done
priority: medium
state_hub_task_id: "27819462-9bae-57ec-95ba-2b90302bd376"

Repository and read-only. ssh railiance01 'kubectl diff -f -' < <file> for the SA file and both store manifests (client-side; nothing applied). Expected diff: two new SAs, and in each store only spec.provider.vault.auth changes. Any other difference stops T03.

T03 Apply the ServiceAccounts and switch the two stores (founder go-ahead)

id: RPF-WP-0045-T03
status: done
priority: high
state_hub_task_id: "ea84219a-d5f4-59da-af83-8b1de8564650"

Waits on T01 and the founder's go-ahead. No OpenBao credential. Kubernetes admin on railiance01.

cd /home/worsch/railiance-platform
ssh railiance01 'kubectl apply -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml

Optional pre-check before the store switch, no admin credential (login with a short-lived SA JWT; the output holds a 15-minute client token, so discard it):

ssh railiance01 'J=$(kubectl -n email-connect create token email-connect-eso --audience=openbao --duration=10m); kubectl -n openbao exec -i openbao-0 -- bao write -field=token_policies auth/kubernetes/login role=email-connect-transactional-eso jwt="$J"'

Expected output: [workload-kv-read-email-connect-transactional-eso]. Same for activity-core-eso in activity-core. A wrong namespace or audience must fail.

T04 Verify

id: RPF-WP-0045-T04
status: done
priority: high
state_hub_task_id: "8416600e-635e-550b-9e59-72081a3fd7f7"

Read-only after T03. Force a refresh, then check.

ssh railiance01 'kubectl annotate clustersecretstore openbao-activity-core openbao-email-connect force-sync=$(date +%s) --overwrite'
ssh railiance01 'kubectl -n activity-core annotate externalsecret --all force-sync=$(date +%s) --overwrite; kubectl -n email-connect annotate externalsecret email-connect-runtime force-sync=$(date +%s) --overwrite'
ssh railiance01 'kubectl wait --for=condition=Ready clustersecretstore/openbao-activity-core clustersecretstore/openbao-email-connect --timeout=60s'
ssh railiance01 'kubectl -n activity-core wait --for=condition=Ready externalsecret --all --timeout=90s; kubectl -n email-connect wait --for=condition=Ready externalsecret/email-connect-runtime --timeout=90s'
ssh railiance01 'kubectl get externalsecret -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[0].reason,AT:.status.refreshTime | grep -v SecretSynced$'

(The annotate calls are writes to ESO objects; they belong to the T03 go-ahead.) Done when both stores are Ready, the five ExternalSecrets show SecretSynced with a refreshTime after T03, and the last command lists no ExternalSecret other than the header. The earlier 37-count had 5 failing; expect 0.

T05 Rollback (only if T03/T04 fails)

id: RPF-WP-0045-T05
status: wait
priority: medium
state_hub_task_id: "6bccc229-3428-588a-9488-3fd5d5a07e92"

The static tokens are already invalid, so rollback cannot restore sync; it restores the prior, known-failing spec. Workload Secrets are unaffected either way (ESO keeps the last synced values). Founder go-ahead:

ssh railiance01 'kubectl patch clustersecretstore openbao-activity-core --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-activity-core-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
ssh railiance01 'kubectl patch clustersecretstore openbao-email-connect --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-email-connect-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
ssh railiance01 'kubectl delete -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml

OpenBao side (admin credential, only if the roles themselves are wrong): warden access openbao-platform-admin-login --exec -- sh -euc 'bao delete auth/kubernetes/role/activity-core-eso; bao delete auth/kubernetes/role/email-connect-transactional-eso; bao policy delete workload-kv-read-activity-core-eso; bao policy delete workload-kv-read-email-connect-transactional-eso'. Then git revert this plan's manifest commit. Correct forward in preference, as RPF-WP-0037 does.

T06 Retire the two static-token Secrets

id: RPF-WP-0045-T06
status: wait
priority: medium
state_hub_task_id: "f56d7096-e005-5dd5-afa0-b1affe247fa7"

Waits on T04 passing for 24 hours (a few 1h refreshes) and the founder's go-ahead. First prove nothing references them:

ssh railiance01 'kubectl get clustersecretstores,secretstores -A -o jsonpath="{range .items[*]}{.metadata.name} {.spec.provider.vault.auth.tokenSecretRef.name}{\"\n\"}{end}" | grep -E "openbao-(activity-core|email-connect)-eso-token"'

Expected: no output. Both tokens already fail lookup-self with 403, so no revocation is needed; the founder may confirm invalidity with the RPF-WP-0037 cleanup pattern (the helper reads the value in memory only). Then delete:

ssh railiance01 'kubectl -n external-secrets delete secret openbao-activity-core-eso-token openbao-email-connect-eso-token'

Also retire the bootstrap targets in the owning repos: activity-core make openbao-eso-token-apply / scripts/openbao-eso-token-apply.sh and email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh, and update docs/argocd-gitops.md. Their owners do that in their repos.

Risks

• Token values in Secret metadata. Both token Secrets were created with kubectl apply, so kubectl.kubernetes.io/last-applied-configuration carries the token value. A metadata-only read shows it. These two tokens are already invalid; T06 deletes them. Every other *-eso-token Secret in external-secrets that was kubectl apply-ed (audit-core, core-hub, canned-prompts, sbom-nexus, tenant-engine) is likely the same with live tokens. Not inspected. Needs its own finding.
• Dependencies of the new method: OpenBao unsealed and reachable (same as today), the Kubernetes auth mount's JWT review with OpenBao's own pod token (auto-rotated, working for the three recovered stores and six more), and ESO's TokenRequest right (present). Nothing found that expires.
• The forgejo-admin PAT and the offsite-lane values read here have their own lifetimes (CCR-2026-0006, CCR-2026-0004). A later sync failure on those is a value problem, not this auth problem.
• Why the static tokens died is not established (TTL expiry or revocation). Other stores still on static tokens (audit-core, core-hub-database, canned-prompts, sbom-nexus, tenant-engine) may fail the same way. Migrating them is out of scope here.

Execution record, 2026-09-21 (founder go-ahead; custodian session)

• T01 done. Run via warden access openbao-platform-admin-login --exec as a silent, idempotent reviewed script (the lane fails closed on any child output; the first founder run printed bao's "Success!" lines and failed for that reason). Script: compare-before-write (exit 3 if an object differs), write only what is missing, verify all four objects; it exited 0. Login needed BAO_ADDR=http://127.0.0.1:18200 (the shell default https://bao.coulomb.social is unreachable) and an xdg-open shim on WSL. warden reported "completed but session revocation could not be confirmed" (exit 1): see ops-warden message 6a1ce1bb and railiance-platform message 641673a4.
• T02 done. Diff exactly as expected: two new SAs; each store changes only spec.provider.vault.auth.
• T03 done. SAs created; both stores switched to Kubernetes auth.
• T04 done. Both stores Ready=True ("store validated"); forced refresh; all five ExternalSecrets SecretSynced at 17:39Z; 37/37 ExternalSecrets ready cluster-wide.
• T06 (delete the two dead token Secrets) is due after 24 h of clean syncs, i.e. not before 2026-09-22T17:40Z.