RPF-WP-0045: plan Kubernetes-auth migration for openbao-activity-core and openbao-email-connect
Declares target store specs, two exact-path policies and two ESO ServiceAccounts. Nothing applied; live steps wait on the founder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 63291@bnt-lap001 Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703
This commit is contained in:
codex
parent
2d79c2e4e5
commit
6e399bbe64
6 changed files with 347 additions and 25 deletions
|
|
@ -0,0 +1,264 @@
|
|||
---
|
||||
id: RPF-WP-0045
|
||||
type: workplan
|
||||
title: "Move openbao-activity-core and openbao-email-connect off expired static tokens to Kubernetes auth"
|
||||
domain: financials
|
||||
repo: railiance-platform
|
||||
status: ready
|
||||
flavor: implementation
|
||||
owner: railiance-platform
|
||||
topic_slug: railiance
|
||||
created: "2026-09-21"
|
||||
updated: "2026-09-21"
|
||||
related: [RPF-WP-0037]
|
||||
---
|
||||
|
||||
Two ClusterSecretStores on railiance01 fail validation: OpenBao answers 403 on
|
||||
`auth/token/lookup-self` for the static tokens they reference
|
||||
(`openbao-email-connect` since 2026-09-13, `openbao-activity-core` since
|
||||
2026-09-19). Five ExternalSecrets are `SecretSyncedError`. This plan moves both
|
||||
stores to the method RPF-WP-0037 used for `openbao-forgejo`, `openbao-reuse`
|
||||
and `openbao-target-revenue`. It is an authentication migration, not a
|
||||
credential rotation. No KV value changes.
|
||||
|
||||
**This plan changes nothing live by itself.** Every live task is `wait` on the
|
||||
founder's go-ahead. T01 is `ADMINISTER @ realm:openbao/railiance01` with the
|
||||
platform-admin credential, `activation=APPROVED`. T03, T05 and T06 are
|
||||
`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. No step
|
||||
needs `BREAK_GLASS`. Evidence is `EvidenceBoundary target-audited`
|
||||
(kubectl status and OpenBao audit log on railiance01).
|
||||
|
||||
## Read-only evidence, 2026-09-21
|
||||
|
||||
Taken over `ssh railiance01 'kubectl get …'` and `kubectl auth can-i`. No
|
||||
secret value was requested.
|
||||
|
||||
- Recovered method: `spec.provider.vault.auth.kubernetes`, mountPath
|
||||
`kubernetes`, a **dedicated ServiceAccount in the consumer namespace**,
|
||||
`audiences: [openbao]`, a per-lane OpenBao role. Source:
|
||||
`openbao/eso-auth-recovery/lanes.json`, `scripts/repair_eso_kubernetes_auth.py`.
|
||||
|
||||
| Store | SA | OpenBao role | Policy |
|
||||
| --- | --- | --- | --- |
|
||||
| openbao-forgejo | forgejo/forgejo-mailer-eso | forgejo-mailer-eso | workload-kv-read-forgejo-mailer-eso |
|
||||
| openbao-reuse | reuse/reuse-surface-eso | reuse-surface-runtime-eso | workload-kv-read-reuse-surface-runtime-eso |
|
||||
| openbao-target-revenue | target-revenue/target-revenue-eso | target-revenue-runtime-eso | workload-kv-read-target-revenue-runtime-eso |
|
||||
|
||||
Role shape: bound to exactly one SA name and one namespace, audience
|
||||
`openbao`, ttl/max_ttl/explicit_max_ttl 15m, `token_no_default_policy=true`,
|
||||
one policy. Policy: exact `platform/data/...` read plus `auth/token/lookup-self`
|
||||
(read) and `auth/token/revoke-self` (update). All three stores are Ready.
|
||||
- Failing ExternalSecrets and the KV paths they read (live `spec.data[].remoteRef.key`):
|
||||
|
||||
| Namespace | ExternalSecret | KV path | Target Secret |
|
||||
| --- | --- | --- | --- |
|
||||
| activity-core | actcore-backup-offsite | platform/workloads/railiance/backup/offsite-lane | actcore-runtime-secret (Merge) |
|
||||
| activity-core | actcore-forgejo-admin | platform/workloads/forgejo/forgejo-admin | actcore-runtime-secret (Merge) |
|
||||
| activity-core | actcore-issue-core-runtime | platform/workloads/issue-core/issue-core/issue-core-runtime | actcore-runtime-secret (Merge) |
|
||||
| activity-core | llm-connect-provider-secrets | platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets | llm-connect-provider-secrets (Owner) |
|
||||
| email-connect | email-connect-runtime | platform/workloads/email-connect/transactional | email-connect-runtime |
|
||||
|
||||
The activity-core four last refreshed on 2026-09-19 (so all four KV paths
|
||||
existed then). Target Secrets are retained with their last synced values;
|
||||
workloads are running on those.
|
||||
- ESO's controller SA `external-secrets/external-secrets` may
|
||||
`create serviceaccount --subresource=token` in `activity-core` and
|
||||
`email-connect` (`kubectl auth can-i` = yes). No new RBAC is needed.
|
||||
- ServiceAccounts `activity-core/activity-core-eso` and
|
||||
`email-connect/email-connect-eso` do not exist yet.
|
||||
- OpenBao's `auth/kubernetes/config` is written by
|
||||
`scripts/openbao-apply-initial-config.sh` with `kubernetes_host` only, so it
|
||||
reviews JWTs with the OpenBao pod's own projected token. No static reviewer
|
||||
JWT that could expire.
|
||||
|
||||
## Declared in the repository (this commit)
|
||||
|
||||
- `openbao/policies/workload-kv-read-activity-core-eso.hcl`: the four data
|
||||
paths above, lookup-self, revoke-self. Nothing else. (The older
|
||||
`external-secrets-activity-core.hcl` also grants metadata reads; it stays for
|
||||
its existing callers and is not attached to the new role.)
|
||||
- `openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl`: one
|
||||
data path, lookup-self, revoke-self.
|
||||
- `openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml`: the two SAs.
|
||||
- `argocd/platform-addons/openbao-secretstore/openbao-{activity-core,email-connect}.clustersecretstore.yaml`:
|
||||
target specs with Kubernetes auth. **These differ from live** until T03.
|
||||
Nothing syncs them (excluded from `kustomization.yaml`; the railiance01
|
||||
ArgoCD draft for `openbao-secretstore` is pinned to `d2dbc19`).
|
||||
|
||||
## T01 OpenBao roles and policies (founder, admin credential)
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T01
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Waits on the founder's go-ahead. **The only step that needs the OpenBao
|
||||
platform-admin credential**, one attended session, four writes plus
|
||||
read-backs. Run from a checkout of `railiance-platform` at the commit that
|
||||
holds this plan. No token appears on the command line; `warden access`
|
||||
supplies it to the child process.
|
||||
|
||||
```sh
|
||||
cd /home/worsch/railiance-platform
|
||||
warden access openbao-platform-admin-login --exec -- sh -euc '
|
||||
bao policy write workload-kv-read-activity-core-eso \
|
||||
openbao/policies/workload-kv-read-activity-core-eso.hcl
|
||||
bao policy write workload-kv-read-email-connect-transactional-eso \
|
||||
openbao/policies/workload-kv-read-email-connect-transactional-eso.hcl
|
||||
bao write auth/kubernetes/role/activity-core-eso \
|
||||
bound_service_account_names=activity-core-eso \
|
||||
bound_service_account_namespaces=activity-core \
|
||||
audience=openbao token_policies=workload-kv-read-activity-core-eso \
|
||||
token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
|
||||
token_no_default_policy=true
|
||||
bao write auth/kubernetes/role/email-connect-transactional-eso \
|
||||
bound_service_account_names=email-connect-eso \
|
||||
bound_service_account_namespaces=email-connect \
|
||||
audience=openbao token_policies=workload-kv-read-email-connect-transactional-eso \
|
||||
token_ttl=15m token_max_ttl=15m token_explicit_max_ttl=15m \
|
||||
token_no_default_policy=true
|
||||
bao read auth/kubernetes/role/activity-core-eso
|
||||
bao read auth/kubernetes/role/email-connect-transactional-eso
|
||||
'
|
||||
```
|
||||
|
||||
Before writing, confirm neither role nor policy exists
|
||||
(`bao read auth/kubernetes/role/<name>` returns 404); if one does, stop and
|
||||
compare rather than overwrite. Done when both read-backs show the fields above.
|
||||
|
||||
## T02 Confirm the declared manifests are the ones to apply
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Repository and read-only. `ssh railiance01 'kubectl diff -f -' < <file>` for the
|
||||
SA file and both store manifests (client-side; nothing applied). Expected diff:
|
||||
two new SAs, and in each store only `spec.provider.vault.auth` changes. Any
|
||||
other difference stops T03.
|
||||
|
||||
## T03 Apply the ServiceAccounts and switch the two stores (founder go-ahead)
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T03
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Waits on T01 and the founder's go-ahead. No OpenBao credential. Kubernetes
|
||||
admin on railiance01.
|
||||
|
||||
```sh
|
||||
cd /home/worsch/railiance-platform
|
||||
ssh railiance01 'kubectl apply -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
|
||||
ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
|
||||
ssh railiance01 'kubectl apply -f -' < argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
|
||||
```
|
||||
|
||||
Optional pre-check before the store switch, no admin credential (login with a
|
||||
short-lived SA JWT; the output holds a 15-minute client token, so discard it):
|
||||
|
||||
```sh
|
||||
ssh railiance01 'J=$(kubectl -n email-connect create token email-connect-eso --audience=openbao --duration=10m); kubectl -n openbao exec -i openbao-0 -- bao write -field=token_policies auth/kubernetes/login role=email-connect-transactional-eso jwt="$J"'
|
||||
```
|
||||
|
||||
Expected output: `[workload-kv-read-email-connect-transactional-eso]`. Same for
|
||||
`activity-core-eso` in `activity-core`. A wrong namespace or audience must fail.
|
||||
|
||||
## T04 Verify
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T04
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Read-only after T03. Force a refresh, then check.
|
||||
|
||||
```sh
|
||||
ssh railiance01 'kubectl annotate clustersecretstore openbao-activity-core openbao-email-connect force-sync=$(date +%s) --overwrite'
|
||||
ssh railiance01 'kubectl -n activity-core annotate externalsecret --all force-sync=$(date +%s) --overwrite; kubectl -n email-connect annotate externalsecret email-connect-runtime force-sync=$(date +%s) --overwrite'
|
||||
ssh railiance01 'kubectl wait --for=condition=Ready clustersecretstore/openbao-activity-core clustersecretstore/openbao-email-connect --timeout=60s'
|
||||
ssh railiance01 'kubectl -n activity-core wait --for=condition=Ready externalsecret --all --timeout=90s; kubectl -n email-connect wait --for=condition=Ready externalsecret/email-connect-runtime --timeout=90s'
|
||||
ssh railiance01 'kubectl get externalsecret -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[0].reason,AT:.status.refreshTime | grep -v SecretSynced$'
|
||||
```
|
||||
|
||||
(The annotate calls are writes to ESO objects; they belong to the T03 go-ahead.)
|
||||
Done when both stores are Ready, the five ExternalSecrets show `SecretSynced`
|
||||
with a refreshTime after T03, and the last command lists no ExternalSecret
|
||||
other than the header. The earlier 37-count had 5 failing; expect 0.
|
||||
|
||||
## T05 Rollback (only if T03/T04 fails)
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T05
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
The static tokens are already invalid, so rollback cannot restore sync; it
|
||||
restores the prior, known-failing spec. Workload Secrets are unaffected either
|
||||
way (ESO keeps the last synced values). Founder go-ahead:
|
||||
|
||||
```sh
|
||||
ssh railiance01 'kubectl patch clustersecretstore openbao-activity-core --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-activity-core-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
|
||||
ssh railiance01 'kubectl patch clustersecretstore openbao-email-connect --type=json -p "[{\"op\":\"replace\",\"path\":\"/spec/provider/vault/auth\",\"value\":{\"tokenSecretRef\":{\"name\":\"openbao-email-connect-eso-token\",\"namespace\":\"external-secrets\",\"key\":\"token\"}}}]"'
|
||||
ssh railiance01 'kubectl delete -f -' < openbao/eso-auth-recovery/rpf-wp-0045-serviceaccounts.yaml
|
||||
```
|
||||
|
||||
OpenBao side (admin credential, only if the roles themselves are wrong):
|
||||
`warden access openbao-platform-admin-login --exec -- sh -euc 'bao delete auth/kubernetes/role/activity-core-eso; bao delete auth/kubernetes/role/email-connect-transactional-eso; bao policy delete workload-kv-read-activity-core-eso; bao policy delete workload-kv-read-email-connect-transactional-eso'`.
|
||||
Then `git revert` this plan's manifest commit. Correct forward in preference, as
|
||||
RPF-WP-0037 does.
|
||||
|
||||
## T06 Retire the two static-token Secrets
|
||||
|
||||
```task
|
||||
id: RPF-WP-0045-T06
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Waits on T04 passing for 24 hours (a few 1h refreshes) and the founder's
|
||||
go-ahead. First prove nothing references them:
|
||||
|
||||
```sh
|
||||
ssh railiance01 'kubectl get clustersecretstores,secretstores -A -o jsonpath="{range .items[*]}{.metadata.name} {.spec.provider.vault.auth.tokenSecretRef.name}{\"\n\"}{end}" | grep -E "openbao-(activity-core|email-connect)-eso-token"'
|
||||
```
|
||||
|
||||
Expected: no output. Both tokens already fail `lookup-self` with 403, so no
|
||||
revocation is needed; the founder may confirm invalidity with the RPF-WP-0037
|
||||
`cleanup` pattern (the helper reads the value in memory only). Then delete:
|
||||
|
||||
```sh
|
||||
ssh railiance01 'kubectl -n external-secrets delete secret openbao-activity-core-eso-token openbao-email-connect-eso-token'
|
||||
```
|
||||
|
||||
Also retire the bootstrap targets in the owning repos:
|
||||
`activity-core` `make openbao-eso-token-apply` / `scripts/openbao-eso-token-apply.sh`
|
||||
and `email-connect/deploy/k8s/railiance/openbao-eso-token-apply.sh`, and update
|
||||
`docs/argocd-gitops.md`. Their owners do that in their repos.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Token values in Secret metadata.** Both token Secrets were created with
|
||||
`kubectl apply`, so `kubectl.kubernetes.io/last-applied-configuration`
|
||||
carries the token value. A metadata-only read shows it. These two tokens are
|
||||
already invalid; T06 deletes them. Every other `*-eso-token` Secret in
|
||||
`external-secrets` that was `kubectl apply`-ed (audit-core, core-hub,
|
||||
canned-prompts, sbom-nexus, tenant-engine) is likely the same with **live**
|
||||
tokens. Not inspected. Needs its own finding.
|
||||
- Dependencies of the new method: OpenBao unsealed and reachable (same as
|
||||
today), the Kubernetes auth mount's JWT review with OpenBao's own pod token
|
||||
(auto-rotated, working for the three recovered stores and six more), and
|
||||
ESO's TokenRequest right (present). Nothing found that expires.
|
||||
- The forgejo-admin PAT and the offsite-lane values read here have their own
|
||||
lifetimes (CCR-2026-0006, CCR-2026-0004). A later sync failure on those is a
|
||||
value problem, not this auth problem.
|
||||
- Why the static tokens died is not established (TTL expiry or revocation).
|
||||
Other stores still on static tokens (audit-core, core-hub-database,
|
||||
canned-prompts, sbom-nexus, tenant-engine) may fail the same way. Migrating
|
||||
them is out of scope here.
|
||||
Loading…
Add table
Add a link
Reference in a new issue