1918128f8e
Earliest old-token expiry is canned-prompts (~2026-10-10), not 10-25. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 150322@bnt-lap001 Assistant-Session: 16a7b788-374e-4915-a1df-fc87ffd9a5e4
196 lines
8.2 KiB
Markdown
196 lines
8.2 KiB
Markdown
---
|
|
id: RPF-WP-0046
|
|
type: workplan
|
|
title: "Keep the dynamic-database ESO parent tokens alive: periodic tokens and a renewer"
|
|
domain: financials
|
|
repo: railiance-platform
|
|
status: active
|
|
owner: railiance-platform
|
|
topic_slug: railiance
|
|
created: "2026-09-23"
|
|
updated: "2026-09-23"
|
|
related: [RPF-WP-0045, RPF-WP-0037, RPF-WP-0036]
|
|
state_hub_workstream_id: "cdb309e0-adb4-58aa-9c98-5ea270780e8d"
|
|
---
|
|
|
|
## Problem
|
|
|
|
Five ClusterSecretStores read OpenBao dynamic database credentials
|
|
(`path: database`) with a static `tokenSecretRef`. The stores are
|
|
`openbao-audit-core-database`, `-canned-prompts-database`,
|
|
`-core-hub-database`, `-sbom-nexus-database` and `-tenant-engine-database`.
|
|
Each token is a 768h renewable orphan. Nothing renews it, and ESO never renews
|
|
a `tokenSecretRef` token. When a token reaches its max TTL, OpenBao revokes
|
|
every lease the token created. On 2026-09-23 this took down sbom-nexus
|
|
(State Hub `/repos` returned 502), tenant-engine (742 restarts) and
|
|
core-hub-api-candidate (43h unready). activity-core restored all three with
|
|
fresh 768h tokens (rapp-postgres `3ebd984`, hub messages `200e2aae` and
|
|
`42b0365c`). **Those tokens expire again around 2026-10-25T17:00Z.**
|
|
|
|
## Decision (founder, 2026-09-23)
|
|
|
|
Auto-renew, rather than switching to Kubernetes auth or static DB roles.
|
|
|
|
Why Kubernetes auth is not a drop-in fix here: ESO v0.16.1
|
|
(`pkg/provider/vault/client.go`, `Close()`) revokes a login-obtained token after
|
|
each reconcile unless token caching is enabled. Revoking that token revokes
|
|
the dynamic leases it just created, so the delivered DB password would die
|
|
within seconds. RPF-WP-0045 was safe only because its stores read KV. Static
|
|
database roles would remove the parent-token dependency. That change crosses
|
|
rapp-postgres and four consumer repos, and remains a possible later plan.
|
|
|
|
Design:
|
|
|
|
- Re-mint each token once as a **periodic** token (`-period=168h`, renewable,
|
|
orphan, same policies). A periodic token has no max TTL; each renewal resets
|
|
it to the period.
|
|
- A daily CronJob `external-secrets/eso-token-renewer` mounts the five token
|
|
Secrets read-only and calls `auth/token/renew-self`. The job fails if any
|
|
lane fails or ends below 72h. A token that was not re-minted as periodic
|
|
trips that threshold near its max TTL, so it cannot expire silently.
|
|
- The CronJob ships through ArgoCD (`railiance-platform-addons`), not by
|
|
direct apply, per the change gate.
|
|
|
|
## T01 Renewer worker and manifests
|
|
|
|
```task
|
|
id: RPF-WP-0046-T01
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "25592a83-d2e5-5aa2-b23e-5d563a621f19"
|
|
```
|
|
|
|
`argocd/platform-addons/eso-token-renewer/` holds `renew.py`, the
|
|
ServiceAccount (no RBAC, no API token), the CronJob and the kustomization. The
|
|
kustomization generates the worker ConfigMap. The draft Application is
|
|
`argocd/railiance01/drafts/eso-token-renewer.application.yaml`. The
|
|
`railiance-platform-addons` AppProject gains `batch/CronJob` in git. That
|
|
AppProject is applied by hand, so the git change alone is not live. Tests:
|
|
`tests/test_eso_token_renewer.py`. A server-side dry run on railiance01 on
|
|
2026-09-23 passed for all three objects.
|
|
|
|
## T02 Periodic attended mint script
|
|
|
|
```task
|
|
id: RPF-WP-0046-T02
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "a318792f-82ba-568b-8633-fba3513aa978"
|
|
```
|
|
|
|
The script is `scripts/openbao-eso-db-token-periodic-attended.sh`. It is
|
|
rapp-postgres's reviewed silent contract, changed only to use `-period` and
|
|
to cover all five lanes. It verifies `period > 0`, `renewable` and the exact
|
|
policy set before it writes the Secret with server-side apply. Lane-to-Secret
|
|
parity with the CronJob mounts is tested.
|
|
|
|
## T03 Re-mint the five tokens as periodic (live, founder)
|
|
|
|
```task
|
|
id: RPF-WP-0046-T03
|
|
status: wait
|
|
priority: high
|
|
state_hub_task_id: "211bc548-34af-5f3a-ab2c-bf29d33f97ad"
|
|
```
|
|
|
|
This step is `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`,
|
|
and needs an attended OIDC/MFA login:
|
|
|
|
```sh
|
|
python3 /home/worsch/railiance-platform/scripts/openbao-attended-exec.py -- \
|
|
/home/worsch/railiance-platform/scripts/openbao-eso-db-token-periodic-attended.sh \
|
|
--confirm RPF-WP-0046-PERIODIC-ESO-TOKEN \
|
|
--status /home/worsch/railiance-platform/docs/evidence/<date>-eso-db-token-periodic.status \
|
|
audit-core canned-prompts core-hub sbom-nexus tenant-engine
|
|
```
|
|
|
|
Then force a refresh and check that all five stores are `Valid` and their
|
|
ExternalSecrets are `SecretSynced`.
|
|
|
|
## T04 Adopt the renewer through ArgoCD (live, founder)
|
|
|
|
```task
|
|
id: RPF-WP-0046-T04
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "28726b1b-3777-5945-859b-9e563b12fee6"
|
|
```
|
|
|
|
1. Apply `argocd/railiance01/bootstrap/02-railiance-platform-addons-project.yaml`
|
|
by hand, which adds `batch/CronJob`.
|
|
2. Pin `targetRevision` in the draft to the reviewed commit, move the draft to
|
|
`argocd/railiance01/applications/`, merge, and sync the root by hand.
|
|
3. Sync `eso-token-renewer` manually. Then run one Job by hand:
|
|
`kubectl -n external-secrets create job --from=cronjob/eso-token-renewer eso-token-renewer-first`.
|
|
Every lane must print `ok: true` with a TTL of 604800.
|
|
|
|
T04 progress, 2026-09-23 (founder go-ahead):
|
|
|
|
- A `kubectl diff` of the AppProject showed only `+ batch/CronJob`. The
|
|
session's permission guard blocked the apply ("Shared Cluster Mutation"), so
|
|
the founder runs step 1.
|
|
- Step 2 is done in git. The Application was moved to
|
|
`argocd/railiance01/applications/` and pinned to `b2ebe10`. The root has no
|
|
automated sync, so nothing changes until the manual root sync.
|
|
|
|
T04 done, 2026-09-24. The founder applied the AppProject and hand-synced the
|
|
root (`8d878cc`) and `eso-token-renewer` (`b2ebe10`). The app is
|
|
Synced/Healthy. A first manual Job completed in 9 seconds, and all five lanes
|
|
returned `ok: true`. The tokens are still pre-T03, so their TTLs cannot be
|
|
extended. **The earliest expiry is canned-prompts, about 2026-10-10**, not
|
|
2026-10-25. Then audit-core, about 2026-10-17. Evidence:
|
|
`docs/evidence/2026-09-24-eso-token-renewer-first-run.json`.
|
|
|
|
## T05 Move consumers off leases from the old tokens before 2026-10-25
|
|
|
|
```task
|
|
id: RPF-WP-0046-T05
|
|
status: wait
|
|
priority: high
|
|
state_hub_task_id: "acf49bee-e40a-533c-b1f0-4152c989211d"
|
|
```
|
|
|
|
The replaced 768h tokens still expire around 2026-10-25T17:00Z, and they
|
|
revoke their leases when they do. ESO refreshes every 5 minutes, so within
|
|
minutes of T03 each target Secret holds credentials from the new token. The
|
|
mounted files update, but a process that reads its password only at startup
|
|
keeps the old one. core-hub runs a credential watcher. Restart the others once
|
|
after T03, at least 10 minutes later and well before the earliest old-token expiry (canned-prompts, about 2026-10-10):
|
|
`audit-core/audit-core`, `canned-prompts/canned-prompts`,
|
|
`sbom-nexus/sbom-nexus`, `tenant-engine/tenant-engine`. Check readiness after
|
|
each restart. Then confirm that core-hub-api and core-hub-api-candidate stay
|
|
ready.
|
|
|
|
## T06 Hand-offs and assurance
|
|
|
|
```task
|
|
id: RPF-WP-0046-T06
|
|
status: todo
|
|
priority: medium
|
|
state_hub_task_id: "2f528e56-ebf0-547e-8d4f-3cb34c348d46"
|
|
```
|
|
|
|
- rapp-postgres: the 768h mint scripts (`apply-*-eso-token*.sh`) would undo
|
|
T03 if rerun. Point them at this plan's periodic script, or change them to
|
|
`-period`.
|
|
- audit-core: `scripts/renew-runtime-lease.sh` re-mints a 768h token. It needs
|
|
the same alignment.
|
|
- railiance-telemetry: alert on a failed `eso-token-renewer` Job, or on no
|
|
successful run in 48h.
|
|
- RPF-WP-0036: add the renewer's last successful run to `assurance-capture`.
|
|
|
|
## Risks
|
|
|
|
- **Renewer outage.** Seven days of slack at the 168h period, and the job
|
|
fails loudly. The alert (T06) is what makes it safe.
|
|
- **Lease max TTL is unchanged.** Each dynamic lease is still capped by its
|
|
role or mount max TTL. A pod that reads its password once and runs longer
|
|
than that loses the lease, whatever the parent token does. This is a
|
|
consumer reload issue, noted in the T06 hand-offs to the consumer owners.
|
|
- **Credential lifetime.** A periodic token lives as long as it is renewed.
|
|
The scope stays exact per lane, and the Secrets carry no last-applied
|
|
annotation. Revocation is `bao token revoke` on the accessor, or deleting
|
|
the Secret and letting the period lapse.
|
|
- **Rollback.** Suspend the CronJob, or delete the Application with prune.
|
|
Re-minting 768h tokens with rapp-postgres `apply-eso-token-attended.sh`
|
|
restores today's state.
|