railiance-platform/workplans/RPF-WP-0046-eso-database-token-renewal.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related	state_hub_workstream_id
RPF-WP-0046	workplan	Keep the dynamic-database ESO parent tokens alive: periodic tokens and a renewer	financials	railiance-platform	active	railiance-platform	railiance	2026-09-23	2026-09-23	RPF-WP-0045 RPF-WP-0037 RPF-WP-0036	cdb309e0-adb4-58aa-9c98-5ea270780e8d

Problem

Five ClusterSecretStores read OpenBao dynamic database credentials (path: database) with a static tokenSecretRef. The stores are openbao-audit-core-database, -canned-prompts-database, -core-hub-database, -sbom-nexus-database and -tenant-engine-database. Each token is a 768h renewable orphan. Nothing renews it, and ESO never renews a tokenSecretRef token. When a token reaches its max TTL, OpenBao revokes every lease the token created. On 2026-09-23 this took down sbom-nexus (State Hub /repos returned 502), tenant-engine (742 restarts) and core-hub-api-candidate (43h unready). activity-core restored all three with fresh 768h tokens (rapp-postgres 3ebd984, hub messages 200e2aae and 42b0365c). Those tokens expire again around 2026-10-25T17:00Z.

Decision (founder, 2026-09-23)

Auto-renew, rather than switching to Kubernetes auth or static DB roles.

Why Kubernetes auth is not a drop-in fix here: ESO v0.16.1 (pkg/provider/vault/client.go, Close()) revokes a login-obtained token after each reconcile unless token caching is enabled. Revoking that token revokes the dynamic leases it just created, so the delivered DB password would die within seconds. RPF-WP-0045 was safe only because its stores read KV. Static database roles would remove the parent-token dependency. That change crosses rapp-postgres and four consumer repos, and remains a possible later plan.

Design:

• Re-mint each token once as a periodic token (-period=168h, renewable, orphan, same policies). A periodic token has no max TTL; each renewal resets it to the period.
• A daily CronJob external-secrets/eso-token-renewer mounts the five token Secrets read-only and calls auth/token/renew-self. The job fails if any lane fails or ends below 72h. A token that was not re-minted as periodic trips that threshold near its max TTL, so it cannot expire silently.
• The CronJob ships through ArgoCD (railiance-platform-addons), not by direct apply, per the change gate.

T01 Renewer worker and manifests

id: RPF-WP-0046-T01
status: done
priority: high
state_hub_task_id: "25592a83-d2e5-5aa2-b23e-5d563a621f19"

argocd/platform-addons/eso-token-renewer/ holds renew.py, the ServiceAccount (no RBAC, no API token), the CronJob and the kustomization. The kustomization generates the worker ConfigMap. The draft Application is argocd/railiance01/drafts/eso-token-renewer.application.yaml. The railiance-platform-addons AppProject gains batch/CronJob in git. That AppProject is applied by hand, so the git change alone is not live. Tests: tests/test_eso_token_renewer.py. A server-side dry run on railiance01 on 2026-09-23 passed for all three objects.

T02 Periodic attended mint script

id: RPF-WP-0046-T02
status: done
priority: high
state_hub_task_id: "a318792f-82ba-568b-8633-fba3513aa978"

The script is scripts/openbao-eso-db-token-periodic-attended.sh. It is rapp-postgres's reviewed silent contract, changed only to use -period and to cover all five lanes. It verifies period > 0, renewable and the exact policy set before it writes the Secret with server-side apply. Lane-to-Secret parity with the CronJob mounts is tested.

T03 Re-mint the five tokens as periodic (live, founder)

id: RPF-WP-0046-T03
status: wait
priority: high
state_hub_task_id: "211bc548-34af-5f3a-ab2c-bf29d33f97ad"

This step is ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED, and needs an attended OIDC/MFA login:

python3 /home/worsch/railiance-platform/scripts/openbao-attended-exec.py -- \
  /home/worsch/railiance-platform/scripts/openbao-eso-db-token-periodic-attended.sh \
  --confirm RPF-WP-0046-PERIODIC-ESO-TOKEN \
  --status /home/worsch/railiance-platform/docs/evidence/<date>-eso-db-token-periodic.status \
  audit-core canned-prompts core-hub sbom-nexus tenant-engine

Then force a refresh and check that all five stores are Valid and their ExternalSecrets are SecretSynced.

T04 Adopt the renewer through ArgoCD (live, founder)

id: RPF-WP-0046-T04
status: wait
priority: high
state_hub_task_id: "28726b1b-3777-5945-859b-9e563b12fee6"

1. Apply argocd/railiance01/bootstrap/02-railiance-platform-addons-project.yaml by hand, which adds batch/CronJob.
2. Pin targetRevision in the draft to the reviewed commit, move the draft to argocd/railiance01/applications/, merge, and sync the root by hand.
3. Sync eso-token-renewer manually. Then run one Job by hand: kubectl -n external-secrets create job --from=cronjob/eso-token-renewer eso-token-renewer-first. Every lane must print ok: true with a TTL of 604800.

T04 progress, 2026-09-23 (founder go-ahead):

• A kubectl diff of the AppProject showed only + batch/CronJob. The session's permission guard blocked the apply ("Shared Cluster Mutation"), so the founder runs step 1.
• Step 2 is done in git. The Application was moved to argocd/railiance01/applications/ and pinned to b2ebe10. The root has no automated sync, so nothing changes until the manual root sync.

T05 Move consumers off leases from the old tokens before 2026-10-25

id: RPF-WP-0046-T05
status: wait
priority: high
state_hub_task_id: "acf49bee-e40a-533c-b1f0-4152c989211d"

The replaced 768h tokens still expire around 2026-10-25T17:00Z, and they revoke their leases when they do. ESO refreshes every 5 minutes, so within minutes of T03 each target Secret holds credentials from the new token. The mounted files update, but a process that reads its password only at startup keeps the old one. core-hub runs a credential watcher. Restart the others once after T03, at least 10 minutes later and well before 2026-10-25: audit-core/audit-core, canned-prompts/canned-prompts, sbom-nexus/sbom-nexus, tenant-engine/tenant-engine. Check readiness after each restart. Then confirm that core-hub-api and core-hub-api-candidate stay ready.

T06 Hand-offs and assurance

id: RPF-WP-0046-T06
status: todo
priority: medium
state_hub_task_id: "2f528e56-ebf0-547e-8d4f-3cb34c348d46"

• rapp-postgres: the 768h mint scripts (apply-*-eso-token*.sh) would undo T03 if rerun. Point them at this plan's periodic script, or change them to -period.
• audit-core: scripts/renew-runtime-lease.sh re-mints a 768h token. It needs the same alignment.
• railiance-telemetry: alert on a failed eso-token-renewer Job, or on no successful run in 48h.
• RPF-WP-0036: add the renewer's last successful run to assurance-capture.

Risks

• Renewer outage. Seven days of slack at the 168h period, and the job fails loudly. The alert (T06) is what makes it safe.
• Lease max TTL is unchanged. Each dynamic lease is still capped by its role or mount max TTL. A pod that reads its password once and runs longer than that loses the lease, whatever the parent token does. This is a consumer reload issue, noted in the T06 hand-offs to the consumer owners.
• Credential lifetime. A periodic token lives as long as it is renewed. The scope stays exact per lane, and the Secrets carry no last-applied annotation. Revocation is bao token revoke on the accessor, or deleting the Secret and letting the period lapse.
• Rollback. Suspend the CronJob, or delete the Application with prune. Re-minting 768h tokens with rapp-postgres apply-eso-token-attended.sh restores today's state.