railiance-platform/workplans/RPF-WP-0046-eso-database-token-renewal.md

---
id: RPF-WP-0046
type: workplan
title: "Keep the dynamic-database ESO parent tokens alive: periodic tokens and a renewer"
domain: financials
repo: railiance-platform
status: finished
owner: railiance-platform
topic_slug: railiance
created: "2026-09-23"
updated: "2026-09-23"
related: [RPF-WP-0045, RPF-WP-0037, RPF-WP-0036]
state_hub_workstream_id: "cdb309e0-adb4-58aa-9c98-5ea270780e8d"
---

## Problem

Five ClusterSecretStores read OpenBao dynamic database credentials
(`path: database`) with a static `tokenSecretRef`. The stores are
`openbao-audit-core-database`, `-canned-prompts-database`,
`-core-hub-database`, `-sbom-nexus-database` and `-tenant-engine-database`.
Each token is a 768h renewable orphan. Nothing renews it, and ESO never renews
a `tokenSecretRef` token. When a token reaches its max TTL, OpenBao revokes
every lease the token created. On 2026-09-23 this took down sbom-nexus
(State Hub `/repos` returned 502), tenant-engine (742 restarts) and
core-hub-api-candidate (43h unready). activity-core restored all three with
fresh 768h tokens (rapp-postgres `3ebd984`, hub messages `200e2aae` and
`42b0365c`). **Those tokens expire again around 2026-10-25T17:00Z.**

## Decision (founder, 2026-09-23)

Auto-renew, rather than switching to Kubernetes auth or static DB roles.

Why Kubernetes auth is not a drop-in fix here: ESO v0.16.1
(`pkg/provider/vault/client.go`, `Close()`) revokes a login-obtained token after
each reconcile unless token caching is enabled. Revoking that token revokes
the dynamic leases it just created, so the delivered DB password would die
within seconds. RPF-WP-0045 was safe only because its stores read KV. Static
database roles would remove the parent-token dependency. That change crosses
rapp-postgres and four consumer repos, and remains a possible later plan.

Design:

- Re-mint each token once as a **periodic** token (`-period=168h`, renewable,
  orphan, same policies). A periodic token has no max TTL; each renewal resets
  it to the period.
- A daily CronJob `external-secrets/eso-token-renewer` mounts the five token
  Secrets read-only and calls `auth/token/renew-self`. The job fails if any
  lane fails or ends below 72h. A token that was not re-minted as periodic
  trips that threshold near its max TTL, so it cannot expire silently.
- The CronJob ships through ArgoCD (`railiance-platform-addons`), not by
  direct apply, per the change gate.

## T01 Renewer worker and manifests

```task
id: RPF-WP-0046-T01
status: done
priority: high
state_hub_task_id: "25592a83-d2e5-5aa2-b23e-5d563a621f19"
```

`argocd/platform-addons/eso-token-renewer/` holds `renew.py`, the
ServiceAccount (no RBAC, no API token), the CronJob and the kustomization. The
kustomization generates the worker ConfigMap. The draft Application is
`argocd/railiance01/drafts/eso-token-renewer.application.yaml`. The
`railiance-platform-addons` AppProject gains `batch/CronJob` in git. That
AppProject is applied by hand, so the git change alone is not live. Tests:
`tests/test_eso_token_renewer.py`. A server-side dry run on railiance01 on
2026-09-23 passed for all three objects.

## T02 Periodic attended mint script

```task
id: RPF-WP-0046-T02
status: done
priority: high
state_hub_task_id: "a318792f-82ba-568b-8633-fba3513aa978"
```

The script is `scripts/openbao-eso-db-token-periodic-attended.sh`. It is
rapp-postgres's reviewed silent contract, changed only to use `-period` and
to cover all five lanes. It verifies `period > 0`, `renewable` and the exact
policy set before it writes the Secret with server-side apply. Lane-to-Secret
parity with the CronJob mounts is tested.

## T03 Re-mint the five tokens as periodic (live, founder)

```task
id: RPF-WP-0046-T03
status: done
priority: high
state_hub_task_id: "211bc548-34af-5f3a-ab2c-bf29d33f97ad"
```

This step is `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`,
and needs an attended OIDC/MFA login:

```sh
python3 /home/worsch/railiance-platform/scripts/openbao-attended-exec.py -- \
  /home/worsch/railiance-platform/scripts/openbao-eso-db-token-periodic-attended.sh \
  --confirm RPF-WP-0046-PERIODIC-ESO-TOKEN \
  --status /home/worsch/railiance-platform/docs/evidence/<date>-eso-db-token-periodic.status \
  audit-core canned-prompts core-hub sbom-nexus tenant-engine
```

Then force a refresh and check that all five stores are `Valid` and their
ExternalSecrets are `SecretSynced`.

## T04 Adopt the renewer through ArgoCD (live, founder)

```task
id: RPF-WP-0046-T04
status: done
priority: high
state_hub_task_id: "28726b1b-3777-5945-859b-9e563b12fee6"
```

1. Apply `argocd/railiance01/bootstrap/02-railiance-platform-addons-project.yaml`
   by hand, which adds `batch/CronJob`.
2. Pin `targetRevision` in the draft to the reviewed commit, move the draft to
   `argocd/railiance01/applications/`, merge, and sync the root by hand.
3. Sync `eso-token-renewer` manually. Then run one Job by hand:
   `kubectl -n external-secrets create job --from=cronjob/eso-token-renewer eso-token-renewer-first`.
   Every lane must print `ok: true` with a TTL of 604800.

T04 progress, 2026-09-23 (founder go-ahead):

- A `kubectl diff` of the AppProject showed only `+ batch/CronJob`. The
  session's permission guard blocked the apply ("Shared Cluster Mutation"), so
  the founder runs step 1.
- Step 2 is done in git. The Application was moved to
  `argocd/railiance01/applications/` and pinned to `b2ebe10`. The root has no
  automated sync, so nothing changes until the manual root sync.

T04 done, 2026-09-24. The founder applied the AppProject and hand-synced the
root (`8d878cc`) and `eso-token-renewer` (`b2ebe10`). The app is
Synced/Healthy. A first manual Job completed in 9 seconds, and all five lanes
returned `ok: true`. The tokens are still pre-T03, so their TTLs cannot be
extended. **The earliest expiry is canned-prompts, about 2026-10-10**, not
2026-10-25. Then audit-core, about 2026-10-17. Evidence:
`docs/evidence/2026-09-24-eso-token-renewer-first-run.json`.

## T05 Move consumers off leases from the old tokens before 2026-10-25

```task
id: RPF-WP-0046-T05
status: done
priority: high
state_hub_task_id: "acf49bee-e40a-533c-b1f0-4152c989211d"
```

The replaced 768h tokens still expire around 2026-10-25T17:00Z, and they
revoke their leases when they do. ESO refreshes every 5 minutes, so within
minutes of T03 each target Secret holds credentials from the new token. The
mounted files update, but a process that reads its password only at startup
keeps the old one. core-hub runs a credential watcher. Restart the others once
after T03, at least 10 minutes later and well before the earliest old-token expiry (canned-prompts, about 2026-10-10):
`audit-core/audit-core`, `canned-prompts/canned-prompts`,
`sbom-nexus/sbom-nexus`, `tenant-engine/tenant-engine`. Check readiness after
each restart. Then confirm that core-hub-api and core-hub-api-candidate stay
ready.

## T06 Hand-offs and assurance

```task
id: RPF-WP-0046-T06
status: done
priority: medium
state_hub_task_id: "2f528e56-ebf0-547e-8d4f-3cb34c348d46"
```

- rapp-postgres: the 768h mint scripts (`apply-*-eso-token*.sh`) would undo
  T03 if rerun. Point them at this plan's periodic script, or change them to
  `-period`.
- audit-core: `scripts/renew-runtime-lease.sh` re-mints a 768h token. It needs
  the same alignment.
- railiance-telemetry: alert on a failed `eso-token-renewer` Job, or on no
  successful run in 48h.
- RPF-WP-0036: add the renewer's last successful run to `assurance-capture`.

## Risks

- **Renewer outage.** Seven days of slack at the 168h period, and the job
  fails loudly. The alert (T06) is what makes it safe.
- **Lease max TTL is unchanged.** Each dynamic lease is still capped by its
  role or mount max TTL. A pod that reads its password once and runs longer
  than that loses the lease, whatever the parent token does. This is a
  consumer reload issue, noted in the T06 hand-offs to the consumer owners.
- **Credential lifetime.** A periodic token lives as long as it is renewed.
  The scope stays exact per lane, and the Secrets carry no last-applied
  annotation. Revocation is `bao token revoke` on the accessor, or deleting
  the Secret and letting the period lapse.
- **Rollback.** Suspend the CronJob, or delete the Application with prune.
  Re-minting 768h tokens with rapp-postgres `apply-eso-token-attended.sh`
  restores today's state.

## Execution record — 2026-09-23 (founder go-ahead)

- **T03 done.** Attended periodic re-mint at about 22:40Z. For all five lanes:
  `renewable=True`, `period=604800`, policies unchanged, `orphan=True`, and
  each Secret was applied clean (no last-applied annotation). The script
  exited 0. Trace: `docs/evidence/2026-09-23-eso-db-token-periodic.status`.
  After a forced refresh, all six stores are `Valid` and 13 ExternalSecrets
  synced at 22:40:51–55Z. A second renewer Job returned `ttl: 604800` for
  every lane.
- **T05 done.** Rolling restarts, one at a time, all rolled out and 1/1 Ready:
  canned-prompts, audit-core, sbom-nexus, tenant-engine. core-hub-api and
  core-hub-api-candidate stayed 1/1 without a restart (credential watcher).
  State Hub `/repos` returns 200. No workload now depends on a lease from the
  replaced max-TTL tokens. Those tokens expire on their own, from about
  2026-10-10.
- **T06 done.** Hand-offs sent 2026-09-23. rapp-postgres and audit-core were
  asked to retire or change their 768h mint scripts. railiance-telemetry was
  asked for an alert on a failed run or no success for 48h. activity-core was
  told the October re-mint is not needed. `assurance-capture` now emits
  `eso.token-renewal` (CronJob status timestamps only, 36h budget), and it is
  healthy on the first live check. The owners' follow-through is tracked in
  their own repos.