diff --git a/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml b/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml index b25fcc4..6b51413 100644 --- a/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml +++ b/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml @@ -1,10 +1,7 @@ -# activity-core reads runtime secrets from the private railiance01 OpenBao -# service. Kubernetes auth remains the preferred follow-up identity. -# -# Prereq: Secret external-secrets/openbao-activity-core-eso-token (key: token) -# with a policy-limited OpenBao token that can read -# platform/workloads/issue-core/issue-core/issue-core-runtime (ISSUE_CORE_API_KEY). -# Bootstrap: activity-core scripts/openbao-eso-token-apply.sh +# RPF-WP-0045: exact namespace/SA/audience Kubernetes auth; no static token. +# DECLARED TARGET, NOT YET LIVE. Live still references the expired +# external-secrets/openbao-activity-core-eso-token until RPF-WP-0045-T03. +# Nothing syncs this file (excluded from the kustomization). apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: @@ -19,10 +16,14 @@ spec: path: platform version: v2 auth: - tokenSecretRef: - name: openbao-activity-core-eso-token - namespace: external-secrets - key: token + kubernetes: + mountPath: kubernetes + role: activity-core-eso + serviceAccountRef: + name: activity-core-eso + namespace: activity-core + audiences: + - openbao conditions: - namespaces: - activity-core diff --git a/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml b/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml index b961efa..752a174 100644 --- a/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml +++ b/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml @@ -1,13 +1,7 @@ -# email-connect transactional receiver reads SMTP + ingest token from OpenBao. -# Deploy on railiance01 (not coulombcore ArgoCD kustomization — same interim -# pattern as openbao-activity-core / openbao-reuse). -# -# Prereq: Secret external-secrets/openbao-email-connect-eso-token (key: token) -# with policy external-secrets-email-connect (or -# workload-kv-read-email-connect-transactional) able to read -# platform/workloads/email-connect/transactional. -# Bootstrap: email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh -# CCR: CCR-2026-0010 +# RPF-WP-0045: exact namespace/SA/audience Kubernetes auth; no static token. +# DECLARED TARGET, NOT YET LIVE. Live still references the expired +# external-secrets/openbao-email-connect-eso-token until RPF-WP-0045-T03. +# Nothing syncs this file (excluded from the kustomization). CCR: CCR-2026-0010. apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: @@ -23,10 +17,14 @@ spec: path: platform version: v2 auth: - tokenSecretRef: - name: openbao-email-connect-eso-token - namespace: external-secrets - key: token + kubernetes: + mountPath: kubernetes + role: email-connect-transactional-eso + serviceAccountRef: + name: email-connect-eso + namespace: email-connect + audiences: + - openbao conditions: - namespaces: - email-connect diff --git a/openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml b/openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml new file mode 100644 index 0000000..fbb87f3 --- /dev/null +++ b/openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml @@ -0,0 +1,16 @@ +# RPF-WP-0045: dedicated ESO identities for the two stores still on static +# tokens. Not applied by the RPF-WP-0037 helper (it reads serviceaccounts.yaml). +# Apply only in RPF-WP-0045-T03, after the founder's go-ahead. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: activity-core-eso + namespace: activity-core +automountServiceAccountToken: false +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: email-connect-eso + namespace: email-connect +automountServiceAccountToken: false diff --git a/openbao/policies/workload-kv-read-activity-core-eso.hcl b/openbao/policies/workload-kv-read-activity-core-eso.hcl new file mode 100644 index 0000000..e470532 --- /dev/null +++ b/openbao/policies/workload-kv-read-activity-core-eso.hcl @@ -0,0 +1,28 @@ +# RPF-WP-0045: ESO delivery for ClusterSecretStore openbao-activity-core via +# Kubernetes auth role activity-core-eso (SA activity-core/activity-core-eso). +# Exactly the four KV paths the four activity-core ExternalSecrets read; data +# read only, no metadata/list/write. ESO also needs lookup-self and revoke-self. + +path "platform/data/workloads/activity-core/llm-connect/llm-connect-provider-secrets" { + capabilities = ["read"] +} + +path "platform/data/workloads/issue-core/issue-core/issue-core-runtime" { + capabilities = ["read"] +} + +path "platform/data/workloads/forgejo/forgejo-admin" { + capabilities = ["read"] +} + +path "platform/data/workloads/railiance/backup/offsite-lane" { + capabilities = ["read"] +} + +path "auth/token/lookup-self" { + capabilities = ["read"] +} + +path "auth/token/revoke-self" { + capabilities = ["update"] +} diff --git a/openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl b/openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl new file mode 100644 index 0000000..efadd99 --- /dev/null +++ b/openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl @@ -0,0 +1,15 @@ +# RPF-WP-0045: ESO delivery for ClusterSecretStore openbao-email-connect via +# Kubernetes auth role email-connect-transactional-eso +# (SA email-connect/email-connect-eso). One exact KV data read. + +path "platform/data/workloads/email-connect/transactional" { + capabilities = ["read"] +} + +path "auth/token/lookup-self" { + capabilities = ["read"] +} + +path "auth/token/revoke-self" { + capabilities = ["update"] +} diff --git a/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md b/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md new file mode 100644 index 0000000..05e906f --- /dev/null +++ b/workplans/RPF-WP-0045-eso-kubernetes-auth-activity-core-email-connect.md @@ -0,0 +1,264 @@ +--- +id: RPF-WP-0045 +type: workplan +title: "Move openbao-activity-core and openbao-email-connect off expired static tokens to Kubernetes auth" +domain: financials +repo: railiance-platform +status: ready +flavor: implementation +owner: railiance-platform +topic_slug: railiance +created: "2026-09-21" +updated: "2026-09-21" +related: [RPF-WP-0037] +--- + +Two ClusterSecretStores on railiance01 fail validation: OpenBao answers 403 on +`auth/token/lookup-self` for the static tokens they reference +(`openbao-email-connect` since 2026-09-13, `openbao-activity-core` since +2026-09-19). Five ExternalSecrets are `SecretSyncedError`. This plan moves both +stores to the method RPF-WP-0037 used for `openbao-forgejo`, `openbao-reuse` +and `openbao-target-revenue`. It is an authentication migration, not a +credential rotation. No KV value changes. + +**This plan changes nothing live by itself.** Every live task is `wait` on the +founder's go-ahead. T01 is `ADMINISTER @ realm:openbao/railiance01` with the +platform-admin credential, `activation=APPROVED`. T03, T05 and T06 are +`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. No step +needs `BREAK_GLASS`. Evidence is `EvidenceBoundary target-audited` +(kubectl status and OpenBao audit log on railiance01). + +## Read-only evidence, 2026-09-21 + +Taken over `ssh railiance01 'kubectl get …'` and `kubectl auth can-i`. No +secret value was requested. + +- Recovered method: `spec.provider.vault.auth.kubernetes`, mountPath + `kubernetes`, a **dedicated ServiceAccount in the consumer namespace**, + `audiences: [openbao]`, a per-lane OpenBao role. Source: + `openbao/eso-auth-recovery/lanes.json`, `scripts/repair_eso_kubernetes_auth.py`. + + | Store | SA | OpenBao role | Policy | + | --- | --- | --- | --- | + | openbao-forgejo | forgejo/forgejo-mailer-eso | forgejo-mailer-eso | workload-kv-read-forgejo-mailer-eso | + | openbao-reuse | reuse/reuse-surface-eso | reuse-surface-runtime-eso | workload-kv-read-reuse-surface-runtime-eso | + | openbao-target-revenue | target-revenue/target-revenue-eso | target-revenue-runtime-eso | workload-kv-read-target-revenue-runtime-eso | + + Role shape: bound to exactly one SA name and one namespace, audience + `openbao`, ttl/max_ttl/explicit_max_ttl 15m, `token_no_default_policy=true`, + one policy. Policy: exact `platform/data/...` read plus `auth/token/lookup-self` + (read) and `auth/token/revoke-self` (update). All three stores are Ready. +- Failing ExternalSecrets and the KV paths they read (live `spec.data[].remoteRef.key`): + + | Namespace | ExternalSecret | KV path | Target Secret | + | --- | --- | --- | --- | + | activity-core | actcore-backup-offsite | platform/workloads/railiance/backup/offsite-lane | actcore-runtime-secret (Merge) | + | activity-core | actcore-forgejo-admin | platform/workloads/forgejo/forgejo-admin | actcore-runtime-secret (Merge) | + | activity-core | actcore-issue-core-runtime | platform/workloads/issue-core/issue-core/issue-core-runtime | actcore-runtime-secret (Merge) | + | activity-core | llm-connect-provider-secrets | platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets | llm-connect-provider-secrets (Owner) | + | email-connect | email-connect-runtime | platform/workloads/email-connect/transactional | email-connect-runtime | + + The activity-core four last refreshed on 2026-09-19 (so all four KV paths + existed then). Target Secrets are retained with their last synced values; + workloads are running on those. +- ESO's controller SA `external-secrets/external-secrets` may + `create serviceaccount --subresource=token` in `activity-core` and + `email-connect` (`kubectl auth can-i` = yes). No new RBAC is needed. +- ServiceAccounts `activity-core/activity-core-eso` and + `email-connect/email-connect-eso` do not exist yet. +- OpenBao's `auth/kubernetes/config` is written by + `scripts/openbao-apply-initial-config.sh` with `kubernetes_host` only, so it + reviews JWTs with the OpenBao pod's own projected token. No static reviewer + JWT that could expire. + +## Declared in the repository (this commit) + +- `openbao/policies/workload-kv-read-activity-core-eso.hcl`: the four data + paths above, lookup-self, revoke-self. Nothing else. (The older + `external-secrets-activity-core.hcl` also grants metadata reads; it stays for + its existing callers and is not attached to the new role.) +- `openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl`: one + data path, lookup-self, revoke-self. +- `openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml`: the two SAs. +- `argocd/platform-addons/openbao-secretstore/openbao-{activity-core,email-connect}.clustersecretstore.yaml`: + target specs with Kubernetes auth. **These differ from live** until T03. + Nothing syncs them (excluded from `kustomization.yaml`; the railiance01 + ArgoCD draft for `openbao-secretstore` is pinned to `d2dbc19`). + +## T01 OpenBao roles and policies (founder, admin credential) + +```task +id: RPF-WP-0045-T01 +status: wait +priority: high +``` + +Waits on the founder's go-ahead. **The only step that needs the OpenBao +platform-admin credential**, one attended session, four writes plus +read-backs. Run from a checkout of `railiance-platform` at the commit that +holds this plan. No token appears on the command line; `warden access` +supplies it to the child process. + +```sh +cd /home/worsch/railiance-platform +warden access openbao-platform-admin-login --exec -- sh -euc ' + bao policy write workload-kv-read-activity-core-eso \ + openbao/policies/workload-kv-read-activity-core-eso.hcl + bao policy write workload-kv-read-email-connect-transactional-eso \ + openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl + bao write auth/kubernetes/role/activity-core-eso \ + bound_service_account_names=activity-core-eso \ + bound_service_account_namespaces=activity-core \ + audience=openbao token_policies=workload-kv-read-activity-core-eso \ + token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \ + token_no_default_policy=true + bao write auth/kubernetes/role/email-connect-transactional-eso \ + bound_service_account_names=email-connect-eso \ + bound_service_account_namespaces=email-connect \ + audience=openbao token_policies=workload-kv-read-email-connect-transactional-eso \ + token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \ + token_no_default_policy=true + bao read auth/kubernetes/role/activity-core-eso + bao read auth/kubernetes/role/email-connect-transactional-eso +' +``` + +Before writing, confirm neither role nor policy exists +(`bao read auth/kubernetes/role/` returns 404); if one does, stop and +compare rather than overwrite. Done when both read-backs show the fields above. + +## T02 Confirm the declared manifests are the ones to apply + +```task +id: RPF-WP-0045-T02 +status: todo +priority: medium +``` + +Repository and read-only. `ssh railiance01 'kubectl diff -f -' < ` for the +SA file and both store manifests (client-side; nothing applied). Expected diff: +two new SAs, and in each store only `spec.provider.vault.auth` changes. Any +other difference stops T03. + +## T03 Apply the ServiceAccounts and switch the two stores (founder go-ahead) + +```task +id: RPF-WP-0045-T03 +status: wait +priority: high +``` + +Waits on T01 and the founder's go-ahead. No OpenBao credential. Kubernetes +admin on railiance01. + +```sh +cd /home/worsch/railiance-platform +ssh railiance01 'kubectl apply -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml +ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml +ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml +``` + +Optional pre-check before the store switch, no admin credential (login with a +short-lived SA JWT; the output holds a 15-minute client token, so discard it): + +```sh +ssh railiance01 'J=$(kubectl -n email-connect create token email-connect-eso --audience=openbao --duration=10m); kubectl -n openbao exec -i openbao-0 -- bao write -field=token_policies auth/kubernetes/login role=email-connect-transactional-eso jwt="$J"' +``` + +Expected output: `[workload-kv-read-email-connect-transactional-eso]`. Same for +`activity-core-eso` in `activity-core`. A wrong namespace or audience must fail. + +## T04 Verify + +```task +id: RPF-WP-0045-T04 +status: wait +priority: high +``` + +Read-only after T03. Force a refresh, then check. + +```sh +ssh railiance01 'kubectl annotate clustersecretstore openbao-activity-core openbao-email-connect force-sync=$(date +%s) --overwrite' +ssh railiance01 'kubectl -n activity-core annotate externalsecret --all force-sync=$(date +%s) --overwrite; kubectl -n email-connect annotate externalsecret email-connect-runtime force-sync=$(date +%s) --overwrite' +ssh railiance01 'kubectl wait --for=condition=Ready clustersecretstore/openbao-activity-core clustersecretstore/openbao-email-connect --timeout=60s' +ssh railiance01 'kubectl -n activity-core wait --for=condition=Ready externalsecret --all --timeout=90s; kubectl -n email-connect wait --for=condition=Ready externalsecret/email-connect-runtime --timeout=90s' +ssh railiance01 'kubectl get externalsecret -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[0].reason,AT:.status.refreshTime | grep -v SecretSynced$' +``` + +(The annotate calls are writes to ESO objects; they belong to the T03 go-ahead.) +Done when both stores are Ready, the five ExternalSecrets show `SecretSynced` +with a refreshTime after T03, and the last command lists no ExternalSecret +other than the header. The earlier 37-count had 5 failing; expect 0. + +## T05 Rollback (only if T03/T04 fails) + +```task +id: RPF-WP-0045-T05 +status: wait +priority: medium +``` + +The static tokens are already invalid, so rollback cannot restore sync; it +restores the prior, known-failing spec. Workload Secrets are unaffected either +way (ESO keeps the last synced values). Founder go-ahead: + +```sh +ssh railiance01 'kubectl patch clustersecretstore openbao-activity-core --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-activity-core-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"' +ssh railiance01 'kubectl patch clustersecretstore openbao-email-connect --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-email-connect-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"' +ssh railiance01 'kubectl delete -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml +``` + +OpenBao side (admin credential, only if the roles themselves are wrong): +`warden access openbao-platform-admin-login --exec -- sh -euc 'bao delete auth/kubernetes/role/activity-core-eso; bao delete auth/kubernetes/role/email-connect-transactional-eso; bao policy delete workload-kv-read-activity-core-eso; bao policy delete workload-kv-read-email-connect-transactional-eso'`. +Then `git revert` this plan's manifest commit. Correct forward in preference, as +RPF-WP-0037 does. + +## T06 Retire the two static-token Secrets + +```task +id: RPF-WP-0045-T06 +status: wait +priority: medium +``` + +Waits on T04 passing for 24 hours (a few 1h refreshes) and the founder's +go-ahead. First prove nothing references them: + +```sh +ssh railiance01 'kubectl get clustersecretstores,secretstores -A -o jsonpath="{range .items[*]}{.metadata.name} {.spec.provider.vault.auth.tokenSecretRef.name}{\"\n\"}{end}" | grep -E "openbao-(activity-core|email-connect)-eso-token"' +``` + +Expected: no output. Both tokens already fail `lookup-self` with 403, so no +revocation is needed; the founder may confirm invalidity with the RPF-WP-0037 +`cleanup` pattern (the helper reads the value in memory only). Then delete: + +```sh +ssh railiance01 'kubectl -n external-secrets delete secret openbao-activity-core-eso-token openbao-email-connect-eso-token' +``` + +Also retire the bootstrap targets in the owning repos: +`activity-core` `make openbao-eso-token-apply` / `scripts/openbao-eso-token-apply.sh` +and `email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh`, and update +`docs/argocd-gitops.md`. Their owners do that in their repos. + +## Risks + +- **Token values in Secret metadata.** Both token Secrets were created with + `kubectl apply`, so `kubectl.kubernetes.io/last-applied-configuration` + carries the token value. A metadata-only read shows it. These two tokens are + already invalid; T06 deletes them. Every other `*-eso-token` Secret in + `external-secrets` that was `kubectl apply`-ed (audit-core, core-hub, + canned-prompts, sbom-nexus, tenant-engine) is likely the same with **live** + tokens. Not inspected. Needs its own finding. +- Dependencies of the new method: OpenBao unsealed and reachable (same as + today), the Kubernetes auth mount's JWT review with OpenBao's own pod token + (auto-rotated, working for the three recovered stores and six more), and + ESO's TokenRequest right (present). Nothing found that expires. +- The forgejo-admin PAT and the offsite-lane values read here have their own + lifetimes (CCR-2026-0006, CCR-2026-0004). A later sync failure on those is a + value problem, not this auth problem. +- Why the static tokens died is not established (TTL expiry or revocation). + Other stores still on static tokens (audit-core, core-hub-database, + canned-prompts, sbom-nexus, tenant-engine) may fail the same way. Migrating + them is out of scope here.