RPF-WP-0045: plan Kubernetes-auth migration for openbao-activity-core and openbao-email-connect

Declares target store specs, two exact-path policies and two ESO ServiceAccounts. Nothing applied; live steps wait on the founder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 63291@bnt-lap001 Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703
2026-09-21 19:18:59 +02:00 · 2026-09-21 19:18:59 +02:00 · 6e399bbe64
commit 6e399bbe64
parent 2d79c2e4e5
6 changed files with 347 additions and 25 deletions
--- a/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
+++ b/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
@ -1,10 +1,7 @@
-# activity-core reads runtime secrets from the private railiance01 OpenBao
-# service. Kubernetes auth remains the preferred follow-up identity.
-#
-# Prereq: Secret external-secrets/openbao-activity-core-eso-token (key: token)
-# with a policy-limited OpenBao token that can read
-# platform/workloads/issue-core/issue-core/issue-core-runtime (ISSUE_CORE_API_KEY).
-# Bootstrap: activity-core scripts/openbao-eso-token-apply.sh
+# RPF-WP-0045: exact namespace/SA/audience Kubernetes auth; no static token.
+# DECLARED TARGET, NOT YET LIVE. Live still references the expired
+# external-secrets/openbao-activity-core-eso-token until RPF-WP-0045-T03.
+# Nothing syncs this file (excluded from the kustomization).
 apiVersion: external-secrets.io/v1beta1
 kind: ClusterSecretStore
 metadata:
@ -19,10 +16,14 @@ spec:
      path: platform
      version: v2
      auth:
-        tokenSecretRef:
-          name: openbao-activity-core-eso-token
-          namespace: external-secrets
-          key: token
+        kubernetes:
+          mountPath: kubernetes
+          role: activity-core-eso
+          serviceAccountRef:
+            name: activity-core-eso
+            namespace: activity-core
+            audiences:
+            - openbao
  conditions:
    - namespaces:
        - activity-core
--- a/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
+++ b/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
@ -1,13 +1,7 @@
-# email-connect transactional receiver reads SMTP + ingest token from OpenBao.
-# Deploy on railiance01 (not coulombcore ArgoCD kustomization — same interim
-# pattern as openbao-activity-core / openbao-reuse).
-#
-# Prereq: Secret external-secrets/openbao-email-connect-eso-token (key: token)
-# with policy external-secrets-email-connect (or
-# workload-kv-read-email-connect-transactional) able to read
-# platform/workloads/email-connect/transactional.
-# Bootstrap: email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh
-# CCR: CCR-2026-0010
+# RPF-WP-0045: exact namespace/SA/audience Kubernetes auth; no static token.
+# DECLARED TARGET, NOT YET LIVE. Live still references the expired
+# external-secrets/openbao-email-connect-eso-token until RPF-WP-0045-T03.
+# Nothing syncs this file (excluded from the kustomization). CCR: CCR-2026-0010.
 apiVersion: external-secrets.io/v1beta1
 kind: ClusterSecretStore
 metadata:
@ -23,10 +17,14 @@ spec:
      path: platform
      version: v2
      auth:
-        tokenSecretRef:
-          name: openbao-email-connect-eso-token
-          namespace: external-secrets
-          key: token
+        kubernetes:
+          mountPath: kubernetes
+          role: email-connect-transactional-eso
+          serviceAccountRef:
+            name: email-connect-eso
+            namespace: email-connect
+            audiences:
+            - openbao
  conditions:
    - namespaces:
        - email-connect
--- a/openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
+++ b/openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
@ -0,0 +1,16 @@
+# RPF-WP-0045: dedicated ESO identities for the two stores still on static
+# tokens. Not applied by the RPF-WP-0037 helper (it reads serviceaccounts.yaml).
+# Apply only in RPF-WP-0045-T03, after the founder's go-ahead.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: activity-core-eso
+  namespace: activity-core
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: email-connect-eso
+  namespace: email-connect
+automountServiceAccountToken: false
--- a/openbao/policies/workload-kv-read-activity-core-eso.hcl
+++ b/openbao/policies/workload-kv-read-activity-core-eso.hcl
@ -0,0 +1,28 @@
+# RPF-WP-0045: ESO delivery for ClusterSecretStore openbao-activity-core via
+# Kubernetes auth role activity-core-eso (SA activity-core/activity-core-eso).
+# Exactly the four KV paths the four activity-core ExternalSecrets read; data
+# read only, no metadata/list/write. ESO also needs lookup-self and revoke-self.
+
+path "platform/data/workloads/activity-core/llm-connect/llm-connect-provider-secrets" {
+  capabilities = ["read"]
+}
+
+path "platform/data/workloads/issue-core/issue-core/issue-core-runtime" {
+  capabilities = ["read"]
+}
+
+path "platform/data/workloads/forgejo/forgejo-admin" {
+  capabilities = ["read"]
+}
+
+path "platform/data/workloads/railiance/backup/offsite-lane" {
+  capabilities = ["read"]
+}
+
+path "auth/token/lookup-self" {
+  capabilities = ["read"]
+}
+
+path "auth/token/revoke-self" {
+  capabilities = ["update"]
+}
--- a/openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl
+++ b/openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl
@ -0,0 +1,15 @@
+# RPF-WP-0045: ESO delivery for ClusterSecretStore openbao-email-connect via
+# Kubernetes auth role email-connect-transactional-eso
+# (SA email-connect/email-connect-eso). One exact KV data read.
+
+path "platform/data/workloads/email-connect/transactional" {
+  capabilities = ["read"]
+}
+
+path "auth/token/lookup-self" {
+  capabilities = ["read"]
+}
+
+path "auth/token/revoke-self" {
+  capabilities = ["update"]
+}
--- a/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md
+++ b/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md
@ -0,0 +1,264 @@
+---
+id: RPF-WP-0045
+type: workplan
+title: "Move openbao-activity-core and openbao-email-connect off expired static tokens to Kubernetes auth"
+domain: financials
+repo: railiance-platform
+status: ready
+flavor: implementation
+owner: railiance-platform
+topic_slug: railiance
+created: "2026-09-21"
+updated: "2026-09-21"
+related: [RPF-WP-0037]
+---
+
+Two ClusterSecretStores on railiance01 fail validation: OpenBao answers 403 on
+`auth/token/lookup-self` for the static tokens they reference
+(`openbao-email-connect` since 2026-09-13, `openbao-activity-core` since
+2026-09-19). Five ExternalSecrets are `SecretSyncedError`. This plan moves both
+stores to the method RPF-WP-0037 used for `openbao-forgejo`, `openbao-reuse`
+and `openbao-target-revenue`. It is an authentication migration, not a
+credential rotation. No KV value changes.
+
+**This plan changes nothing live by itself.** Every live task is `wait` on the
+founder's go-ahead. T01 is `ADMINISTER @ realm:openbao/railiance01` with the
+platform-admin credential, `activation=APPROVED`. T03, T05 and T06 are
+`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. No step
+needs `BREAK_GLASS`. Evidence is `EvidenceBoundary target-audited`
+(kubectl status and OpenBao audit log on railiance01).
+
+## Read-only evidence, 2026-09-21
+
+Taken over `ssh railiance01 'kubectl get …'` and `kubectl auth can-i`. No
+secret value was requested.
+
+- Recovered method: `spec.provider.vault.auth.kubernetes`, mountPath
+  `kubernetes`, a **dedicated ServiceAccount in the consumer namespace**,
+  `audiences: [openbao]`, a per-lane OpenBao role. Source:
+  `openbao/eso-auth-recovery/lanes.json`, `scripts/repair_eso_kubernetes_auth.py`.
+
+  | Store | SA | OpenBao role | Policy |
+  | --- | --- | --- | --- |
+  | openbao-forgejo | forgejo/forgejo-mailer-eso | forgejo-mailer-eso | workload-kv-read-forgejo-mailer-eso |
+  | openbao-reuse | reuse/reuse-surface-eso | reuse-surface-runtime-eso | workload-kv-read-reuse-surface-runtime-eso |
+  | openbao-target-revenue | target-revenue/target-revenue-eso | target-revenue-runtime-eso | workload-kv-read-target-revenue-runtime-eso |
+
+  Role shape: bound to exactly one SA name and one namespace, audience
+  `openbao`, ttl/max_ttl/explicit_max_ttl 15m, `token_no_default_policy=true`,
+  one policy. Policy: exact `platform/data/...` read plus `auth/token/lookup-self`
+  (read) and `auth/token/revoke-self` (update). All three stores are Ready.
+- Failing ExternalSecrets and the KV paths they read (live `spec.data[].remoteRef.key`):
+
+  | Namespace | ExternalSecret | KV path | Target Secret |
+  | --- | --- | --- | --- |
+  | activity-core | actcore-backup-offsite | platform/workloads/railiance/backup/offsite-lane | actcore-runtime-secret (Merge) |
+  | activity-core | actcore-forgejo-admin | platform/workloads/forgejo/forgejo-admin | actcore-runtime-secret (Merge) |
+  | activity-core | actcore-issue-core-runtime | platform/workloads/issue-core/issue-core/issue-core-runtime | actcore-runtime-secret (Merge) |
+  | activity-core | llm-connect-provider-secrets | platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets | llm-connect-provider-secrets (Owner) |
+  | email-connect | email-connect-runtime | platform/workloads/email-connect/transactional | email-connect-runtime |
+
+  The activity-core four last refreshed on 2026-09-19 (so all four KV paths
+  existed then). Target Secrets are retained with their last synced values;
+  workloads are running on those.
+- ESO's controller SA `external-secrets/external-secrets` may
+  `create serviceaccount --subresource=token` in `activity-core` and
+  `email-connect` (`kubectl auth can-i` = yes). No new RBAC is needed.
+- ServiceAccounts `activity-core/activity-core-eso` and
+  `email-connect/email-connect-eso` do not exist yet.
+- OpenBao's `auth/kubernetes/config` is written by
+  `scripts/openbao-apply-initial-config.sh` with `kubernetes_host` only, so it
+  reviews JWTs with the OpenBao pod's own projected token. No static reviewer
+  JWT that could expire.
+
+## Declared in the repository (this commit)
+
+- `openbao/policies/workload-kv-read-activity-core-eso.hcl`: the four data
+  paths above, lookup-self, revoke-self. Nothing else. (The older
+  `external-secrets-activity-core.hcl` also grants metadata reads; it stays for
+  its existing callers and is not attached to the new role.)
+- `openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl`: one
+  data path, lookup-self, revoke-self.
+- `openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml`: the two SAs.
+- `argocd/platform-addons/openbao-secretstore/openbao-{activity-core,email-connect}.clustersecretstore.yaml`:
+  target specs with Kubernetes auth. **These differ from live** until T03.
+  Nothing syncs them (excluded from `kustomization.yaml`; the railiance01
+  ArgoCD draft for `openbao-secretstore` is pinned to `d2dbc19`).
+
+## T01 OpenBao roles and policies (founder, admin credential)
+
+```task
+id: RPF-WP-0045-T01
+status: wait
+priority: high
+```
+
+Waits on the founder's go-ahead. **The only step that needs the OpenBao
+platform-admin credential**, one attended session, four writes plus
+read-backs. Run from a checkout of `railiance-platform` at the commit that
+holds this plan. No token appears on the command line; `warden access`
+supplies it to the child process.
+
+```sh
+cd /home/worsch/railiance-platform
+warden access openbao-platform-admin-login --exec -- sh -euc '
+  bao policy write workload-kv-read-activity-core-eso \
+    openbao/policies/workload-kv-read-activity-core-eso.hcl
+  bao policy write workload-kv-read-email-connect-transactional-eso \
+    openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl
+  bao write auth/kubernetes/role/activity-core-eso \
+    bound_service_account_names=activity-core-eso \
+    bound_service_account_namespaces=activity-core \
+    audience=openbao token_policies=workload-kv-read-activity-core-eso \
+    token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
+    token_no_default_policy=true
+  bao write auth/kubernetes/role/email-connect-transactional-eso \
+    bound_service_account_names=email-connect-eso \
+    bound_service_account_namespaces=email-connect \
+    audience=openbao token_policies=workload-kv-read-email-connect-transactional-eso \
+    token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
+    token_no_default_policy=true
+  bao read auth/kubernetes/role/activity-core-eso
+  bao read auth/kubernetes/role/email-connect-transactional-eso
+'
+```
+
+Before writing, confirm neither role nor policy exists
+(`bao read auth/kubernetes/role/<name>` returns 404); if one does, stop and
+compare rather than overwrite. Done when both read-backs show the fields above.
+
+## T02 Confirm the declared manifests are the ones to apply
+
+```task
+id: RPF-WP-0045-T02
+status: todo
+priority: medium
+```
+
+Repository and read-only. `ssh railiance01 'kubectl diff -f -' < <file>` for the
+SA file and both store manifests (client-side; nothing applied). Expected diff:
+two new SAs, and in each store only `spec.provider.vault.auth` changes. Any
+other difference stops T03.
+
+## T03 Apply the ServiceAccounts and switch the two stores (founder go-ahead)
+
+```task
+id: RPF-WP-0045-T03
+status: wait
+priority: high
+```
+
+Waits on T01 and the founder's go-ahead. No OpenBao credential. Kubernetes
+admin on railiance01.
+
+```sh
+cd /home/worsch/railiance-platform
+ssh railiance01 'kubectl apply -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
+ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
+ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
+```
+
+Optional pre-check before the store switch, no admin credential (login with a
+short-lived SA JWT; the output holds a 15-minute client token, so discard it):
+
+```sh
+ssh railiance01 'J=$(kubectl -n email-connect create token email-connect-eso --audience=openbao --duration=10m); kubectl -n openbao exec -i openbao-0 -- bao write -field=token_policies auth/kubernetes/login role=email-connect-transactional-eso jwt="$J"'
+```
+
+Expected output: `[workload-kv-read-email-connect-transactional-eso]`. Same for
+`activity-core-eso` in `activity-core`. A wrong namespace or audience must fail.
+
+## T04 Verify
+
+```task
+id: RPF-WP-0045-T04
+status: wait
+priority: high
+```
+
+Read-only after T03. Force a refresh, then check.
+
+```sh
+ssh railiance01 'kubectl annotate clustersecretstore openbao-activity-core openbao-email-connect force-sync=$(date +%s) --overwrite'
+ssh railiance01 'kubectl -n activity-core annotate externalsecret --all force-sync=$(date +%s) --overwrite; kubectl -n email-connect annotate externalsecret email-connect-runtime force-sync=$(date +%s) --overwrite'
+ssh railiance01 'kubectl wait --for=condition=Ready clustersecretstore/openbao-activity-core clustersecretstore/openbao-email-connect --timeout=60s'
+ssh railiance01 'kubectl -n activity-core wait --for=condition=Ready externalsecret --all --timeout=90s; kubectl -n email-connect wait --for=condition=Ready externalsecret/email-connect-runtime --timeout=90s'
+ssh railiance01 'kubectl get externalsecret -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[0].reason,AT:.status.refreshTime | grep -v SecretSynced$'
+```
+
+(The annotate calls are writes to ESO objects; they belong to the T03 go-ahead.)
+Done when both stores are Ready, the five ExternalSecrets show `SecretSynced`
+with a refreshTime after T03, and the last command lists no ExternalSecret
+other than the header. The earlier 37-count had 5 failing; expect 0.
+
+## T05 Rollback (only if T03/T04 fails)
+
+```task
+id: RPF-WP-0045-T05
+status: wait
+priority: medium
+```
+
+The static tokens are already invalid, so rollback cannot restore sync; it
+restores the prior, known-failing spec. Workload Secrets are unaffected either
+way (ESO keeps the last synced values). Founder go-ahead:
+
+```sh
+ssh railiance01 'kubectl patch clustersecretstore openbao-activity-core --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-activity-core-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
+ssh railiance01 'kubectl patch clustersecretstore openbao-email-connect --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-email-connect-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
+ssh railiance01 'kubectl delete -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
+```
+
+OpenBao side (admin credential, only if the roles themselves are wrong):
+`warden access openbao-platform-admin-login --exec -- sh -euc 'bao delete auth/kubernetes/role/activity-core-eso; bao delete auth/kubernetes/role/email-connect-transactional-eso; bao policy delete workload-kv-read-activity-core-eso; bao policy delete workload-kv-read-email-connect-transactional-eso'`.
+Then `git revert` this plan's manifest commit. Correct forward in preference, as
+RPF-WP-0037 does.
+
+## T06 Retire the two static-token Secrets
+
+```task
+id: RPF-WP-0045-T06
+status: wait
+priority: medium
+```
+
+Waits on T04 passing for 24 hours (a few 1h refreshes) and the founder's
+go-ahead. First prove nothing references them:
+
+```sh
+ssh railiance01 'kubectl get clustersecretstores,secretstores -A -o jsonpath="{range .items[*]}{.metadata.name} {.spec.provider.vault.auth.tokenSecretRef.name}{\"\n\"}{end}" | grep -E "openbao-(activity-core|email-connect)-eso-token"'
+```
+
+Expected: no output. Both tokens already fail `lookup-self` with 403, so no
+revocation is needed; the founder may confirm invalidity with the RPF-WP-0037
+`cleanup` pattern (the helper reads the value in memory only). Then delete:
+
+```sh
+ssh railiance01 'kubectl -n external-secrets delete secret openbao-activity-core-eso-token openbao-email-connect-eso-token'
+```
+
+Also retire the bootstrap targets in the owning repos:
+`activity-core` `make openbao-eso-token-apply` / `scripts/openbao-eso-token-apply.sh`
+and `email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh`, and update
+`docs/argocd-gitops.md`. Their owners do that in their repos.
+
+## Risks
+
+- **Token values in Secret metadata.** Both token Secrets were created with
+  `kubectl apply`, so `kubectl.kubernetes.io/last-applied-configuration`
+  carries the token value. A metadata-only read shows it. These two tokens are
+  already invalid; T06 deletes them. Every other `*-eso-token` Secret in
+  `external-secrets` that was `kubectl apply`-ed (audit-core, core-hub,
+  canned-prompts, sbom-nexus, tenant-engine) is likely the same with **live**
+  tokens. Not inspected. Needs its own finding.
+- Dependencies of the new method: OpenBao unsealed and reachable (same as
+  today), the Kubernetes auth mount's JWT review with OpenBao's own pod token
+  (auto-rotated, working for the three recovered stores and six more), and
+  ESO's TokenRequest right (present). Nothing found that expires.
+- The forgejo-admin PAT and the offsite-lane values read here have their own
+  lifetimes (CCR-2026-0006, CCR-2026-0004). A later sync failure on those is a
+  value problem, not this auth problem.
+- Why the static tokens died is not established (TTL expiry or revocation).
+  Other stores still on static tokens (audit-core, core-hub-database,
+  canned-prompts, sbom-nexus, tenant-engine) may fail the same way. Migrating
+  them is out of scope here.