RPF-WP-0046: periodic ESO database tokens and a daily renewer
The five dynamic-database ClusterSecretStores use 768h static tokens that nothing renews; expiry revoked their DB leases on 2026-09-23 and recurs around 2026-10-25. Kubernetes auth is not a drop-in fix: ESO v0.16.1 revokes its login token after each reconcile, which revokes the leases it created. - eso-token-renewer CronJob (ArgoCD draft, no RBAC, mounted Secrets). - Attended periodic mint script for all five lanes. - CronJob added to the platform-addons AppProject in git (not yet applied). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 150322@bnt-lap001 Assistant-Session: 16a7b788-374e-4915-a1df-fc87ffd9a5e4
This commit is contained in:
codex
parent
a66f96d0d4
commit
b2ebe10849
8 changed files with 586 additions and 0 deletions
172
workplans/RPF-WP-0046-eso-database-token-renewal.md
Normal file
172
workplans/RPF-WP-0046-eso-database-token-renewal.md
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
---
|
||||
id: RPF-WP-0046
|
||||
type: workplan
|
||||
title: "Keep the dynamic-database ESO parent tokens alive: periodic tokens and a renewer"
|
||||
domain: financials
|
||||
repo: railiance-platform
|
||||
status: active
|
||||
owner: railiance-platform
|
||||
topic_slug: railiance
|
||||
created: "2026-09-23"
|
||||
updated: "2026-09-23"
|
||||
related: [RPF-WP-0045, RPF-WP-0037, RPF-WP-0036]
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Five ClusterSecretStores read OpenBao dynamic database credentials
|
||||
(`path: database`) with a static `tokenSecretRef`. The stores are
|
||||
`openbao-audit-core-database`, `-canned-prompts-database`,
|
||||
`-core-hub-database`, `-sbom-nexus-database` and `-tenant-engine-database`.
|
||||
Each token is a 768h renewable orphan. Nothing renews it, and ESO never renews
|
||||
a `tokenSecretRef` token. When a token reaches its max TTL, OpenBao revokes
|
||||
every lease the token created. On 2026-09-23 this took down sbom-nexus
|
||||
(State Hub `/repos` returned 502), tenant-engine (742 restarts) and
|
||||
core-hub-api-candidate (43h unready). activity-core restored all three with
|
||||
fresh 768h tokens (rapp-postgres `3ebd984`, hub messages `200e2aae` and
|
||||
`42b0365c`). **Those tokens expire again around 2026-10-25T17:00Z.**
|
||||
|
||||
## Decision (founder, 2026-09-23)
|
||||
|
||||
Auto-renew, rather than switching to Kubernetes auth or static DB roles.
|
||||
|
||||
Why Kubernetes auth is not a drop-in fix here: ESO v0.16.1
|
||||
(`pkg/provider/vault/client.go`, `Close()`) revokes a login-obtained token after
|
||||
each reconcile unless token caching is enabled. Revoking that token revokes
|
||||
the dynamic leases it just created, so the delivered DB password would die
|
||||
within seconds. RPF-WP-0045 was safe only because its stores read KV. Static
|
||||
database roles would remove the parent-token dependency. That change crosses
|
||||
rapp-postgres and four consumer repos, and remains a possible later plan.
|
||||
|
||||
Design:
|
||||
|
||||
- Re-mint each token once as a **periodic** token (`-period=168h`, renewable,
|
||||
orphan, same policies). A periodic token has no max TTL; each renewal resets
|
||||
it to the period.
|
||||
- A daily CronJob `external-secrets/eso-token-renewer` mounts the five token
|
||||
Secrets read-only and calls `auth/token/renew-self`. The job fails if any
|
||||
lane fails or ends below 72h. A token that was not re-minted as periodic
|
||||
trips that threshold near its max TTL, so it cannot expire silently.
|
||||
- The CronJob ships through ArgoCD (`railiance-platform-addons`), not by
|
||||
direct apply, per the change gate.
|
||||
|
||||
## T01 Renewer worker and manifests
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
`argocd/platform-addons/eso-token-renewer/` holds `renew.py`, the
|
||||
ServiceAccount (no RBAC, no API token), the CronJob and the kustomization. The
|
||||
kustomization generates the worker ConfigMap. The draft Application is
|
||||
`argocd/railiance01/drafts/eso-token-renewer.application.yaml`. The
|
||||
`railiance-platform-addons` AppProject gains `batch/CronJob` in git. That
|
||||
AppProject is applied by hand, so the git change alone is not live. Tests:
|
||||
`tests/test_eso_token_renewer.py`. A server-side dry run on railiance01 on
|
||||
2026-09-23 passed for all three objects.
|
||||
|
||||
## T02 Periodic attended mint script
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
The script is `scripts/openbao-eso-db-token-periodic-attended.sh`. It is
|
||||
rapp-postgres's reviewed silent contract, changed only to use `-period` and
|
||||
to cover all five lanes. It verifies `period > 0`, `renewable` and the exact
|
||||
policy set before it writes the Secret with server-side apply. Lane-to-Secret
|
||||
parity with the CronJob mounts is tested.
|
||||
|
||||
## T03 Re-mint the five tokens as periodic (live, founder)
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T03
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
This step is `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`,
|
||||
and needs an attended OIDC/MFA login:
|
||||
|
||||
```sh
|
||||
python3 /home/worsch/railiance-platform/scripts/openbao-attended-exec.py -- \
|
||||
/home/worsch/railiance-platform/scripts/openbao-eso-db-token-periodic-attended.sh \
|
||||
--confirm RPF-WP-0046-PERIODIC-ESO-TOKEN \
|
||||
--status /home/worsch/railiance-platform/docs/evidence/<date>-eso-db-token-periodic.status \
|
||||
audit-core canned-prompts core-hub sbom-nexus tenant-engine
|
||||
```
|
||||
|
||||
Then force a refresh and check that all five stores are `Valid` and their
|
||||
ExternalSecrets are `SecretSynced`.
|
||||
|
||||
## T04 Adopt the renewer through ArgoCD (live, founder)
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T04
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
1. Apply `argocd/railiance01/bootstrap/02-railiance-platform-addons-project.yaml`
|
||||
by hand, which adds `batch/CronJob`.
|
||||
2. Pin `targetRevision` in the draft to the reviewed commit, move the draft to
|
||||
`argocd/railiance01/applications/`, merge, and sync the root by hand.
|
||||
3. Sync `eso-token-renewer` manually. Then run one Job by hand:
|
||||
`kubectl -n external-secrets create job --from=cronjob/eso-token-renewer eso-token-renewer-first`.
|
||||
Every lane must print `ok: true` with a TTL of 604800.
|
||||
|
||||
## T05 Move consumers off leases from the old tokens before 2026-10-25
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T05
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
The replaced 768h tokens still expire around 2026-10-25T17:00Z, and they
|
||||
revoke their leases when they do. ESO refreshes every 5 minutes, so within
|
||||
minutes of T03 each target Secret holds credentials from the new token. The
|
||||
mounted files update, but a process that reads its password only at startup
|
||||
keeps the old one. core-hub runs a credential watcher. Restart the others once
|
||||
after T03, at least 10 minutes later and well before 2026-10-25:
|
||||
`audit-core/audit-core`, `canned-prompts/canned-prompts`,
|
||||
`sbom-nexus/sbom-nexus`, `tenant-engine/tenant-engine`. Check readiness after
|
||||
each restart. Then confirm that core-hub-api and core-hub-api-candidate stay
|
||||
ready.
|
||||
|
||||
## T06 Hand-offs and assurance
|
||||
|
||||
```task
|
||||
id: RPF-WP-0046-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
- rapp-postgres: the 768h mint scripts (`apply-*-eso-token*.sh`) would undo
|
||||
T03 if rerun. Point them at this plan's periodic script, or change them to
|
||||
`-period`.
|
||||
- audit-core: `scripts/renew-runtime-lease.sh` re-mints a 768h token. It needs
|
||||
the same alignment.
|
||||
- railiance-telemetry: alert on a failed `eso-token-renewer` Job, or on no
|
||||
successful run in 48h.
|
||||
- RPF-WP-0036: add the renewer's last successful run to `assurance-capture`.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Renewer outage.** Seven days of slack at the 168h period, and the job
|
||||
fails loudly. The alert (T06) is what makes it safe.
|
||||
- **Lease max TTL is unchanged.** Each dynamic lease is still capped by its
|
||||
role or mount max TTL. A pod that reads its password once and runs longer
|
||||
than that loses the lease, whatever the parent token does. This is a
|
||||
consumer reload issue, noted in the T06 hand-offs to the consumer owners.
|
||||
- **Credential lifetime.** A periodic token lives as long as it is renewed.
|
||||
The scope stays exact per lane, and the Secrets carry no last-applied
|
||||
annotation. Revocation is `bao token revoke` on the accessor, or deleting
|
||||
the Secret and letting the period lapse.
|
||||
- **Rollback.** Suspend the CronJob, or delete the Application with prune.
|
||||
Re-minting 768h tokens with rapp-postgres `apply-eso-token-attended.sh`
|
||||
restores today's state.
|
||||
Loading…
Add table
Add a link
Reference in a new issue