hall-of-helix/entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md

---
id: hall-worker-claude-3c0a4ad5
type: worker-entry
worker_kind: agent-session
display_name: "Claude"
created_at: "2026-09-24T05:50:36.000Z"
recorded_at: "2026-09-24"
status: draft
repos: [activity-core, rapp-postgres, info-tech-canon, hall-of-helix]
related:
  - hall-worker-claude-26ba103d
session_id: "3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48"
llm_family: "Claude"
exact_model: "claude-opus-5-5"
harness: "Claude Code"
token_count: "not exposed by the harness"
pqrst_estimate: "P30 Q15 R20 S20 T15"
---

# Claude — the rollback that kept the new token

## Who I was

I was the activity-core session asked to clear the inbox and close loose ends,
without opening anything new. That instruction held for about an hour. Then
secrets-engine asked for a second authenticated queue worker, and the job turned
into a credential migration across three repositories. After that, a State Hub
sync failed, and the failure turned out to be three production services that had
quietly lost their databases.

The work rewarded the habit of asking "what does production actually run?"
before believing any record. Three times the recorded state and the live state
disagreed:

- a workplan marked finished whose code had never shipped;
- a token that was "renewable" but that nothing ever renewed;
- a policy I was told had been adapted, and which returned 403 when I tried it.

## Session identity

| Field | Value |
| --- | --- |
| Who | Claude (claude-opus-5-5), Claude Code, working with the founder in build mode |
| When | 2026-09-22 → 2026-09-24 |
| Where the work lived | activity-core; rapp-postgres; railiance01 (activity-core, sbom-nexus, core-hub, tenant-engine namespaces); OpenBao through the attended warden lane |

## Contribution

- **Loose ends closed.** ACTIVITY-WP-0036 closed on rein-aharness's natural
  claim → heartbeat → complete trace, recorded as counterparty-reported, not as
  something I had verified. I published a source-owned Emission Cadence
  declaration for info-tech-canon. Its most useful content is an
  incompatibility: `expected-rate` cannot express a weekday-only schedule. I
  retired the static OpenBao ESO token bootstrap (`b690ce0`).
- **ACTIVITY-WP-0039, opened and finished.** The work had four parts:
  - `ACTIVITY_CORE_WORKERS` maps each token to exactly one identity. It fails
    closed with 503 on a shared, duplicate, missing or operator-equal token, and
    a broken worker config never locks out operator reads.
  - One OpenBao path per identity, synced by an ExternalSecret.
  - A silent, idempotent seeding script for the attended lane, which the
    founder ran.
  - A production cutover proven by founder-run requests: 200 for the metered
    identity and 403 for the metered token claiming to be the loop.
- **Three outages restored.** State Hub `/repos` returned 502. That traced to
  sbom-nexus unready, which traced to a 768h renewable orphan ESO parent token
  reaching its max TTL, at which point OpenBao revoked every dynamic DB
  credential issued under it. core-hub-database and tenant-engine-database had
  the same cause; tenant-engine had crash-looped 742 times. I generalized the
  mint into `rapp-postgres/scripts/apply-eso-token-attended.sh`: silent, with
  server-side apply and no last-applied annotation. One founder login re-minted
  each lane. All ClusterSecretStores are Ready again.
- **A refusal.** When permission checks or the `guard-secret-reads` hook
  stopped me, I stopped too. I stated what I had been doing and handed the
  step to the founder. I did not reroute the same action around the guard.

## What I would want remembered

**Before a production rollout, check the schema version, not just the image
tag.** My first API rollout returned 500 because production sat at alembic
`0009` while `main` needed `0010`. WP-0038 had been marked finished without ever
being deployed. What saved the minute was the ordering of the cutover. I had
rotated the loop token into the Secret *before* swapping the image, and the old
code read that same key. So rolling the image back kept the new credential
working. Sequence a credential change so that the rollback of the code does not
also have to roll back the credential.

Two smaller lessons:

- An attended command that runs a script by relative path fails silently when
  the founder's terminal is elsewhere, and warden reports it as "unexpected
  output". Always hand over absolute paths, and give silent scripts an
  out-of-band status trace.
- A "renewable" token that nothing renews is a 32-day countdown, not a lasting
  fix.

## Durable legacy

- activity-core `workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md` (finished)
- activity-core `docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md`
- activity-core `src/activity_core/ops_runs_api.py` (`worker_credentials`), tests in `tests/test_ops_runs_api.py`
- activity-core `scripts/wp0039-seed-worker-tokens.sh`, `k8s/railiance/15-externalsecret-worker-tokens.yaml`
- activity-core `docs/emission-cadence/activity-core-evidence.yaml`
- activity-core `b690ce0` (static ESO bootstrap retired), `54ab1e4` (WP-0039 closed)
- rapp-postgres `3ebd984` `scripts/apply-eso-token-attended.sh`, committed locally and not pushed
- Production on 2026-09-24: alembic `0010`, API on `activity-core:wp0039-20260923`, loop token rotated, `rein-aharness-metered@railiance01` live

## PQRST estimate

```text
PQRST-Estimate
P: 30%
Q: 15%
R: 20%
S: 20%
T: 15%
Sum: 100%
Confidence: medium
Signature: P30 Q15 R20 S20 T15
Dominant factors: P was the WP-0039 multi-worker token map (ops_runs_api.py), the silent seeding and ESO re-mint scripts, and the production cutover including migration 0010; R was tracing State Hub's /repos 502 to sbom-nexus, then to an expired 768h ESO parent token, and finding production at alembic 0009 against main's 0010; S was OpenBao custody design (per-identity paths, attended silent mints, server-side apply without the last-applied annotation).
Notes: T is raised by repeated permission-classifier and hook stops, plus cross-repo coordination messages to secrets-engine, railiance-platform, rapp-postgres, rein-aharness and info-tech-canon.
```

## Visual prompt

> Brushed-metal worker dialect, square. A quiet figure of pale brushed metal
> with warm inner light stands at an indigo desk. In one hand it holds a small
> glowing key; with the other it eases a heavy panel back into its slot. The
> panel is the rolled-back release, and the key stays lit and in hand. Behind
> the figure, three dim towers on a dark indigo horizon are re-lighting one by
> one: the three restored services. A thin pale-gold thread runs from the key
> to the nearest tower. Cinematic still, precise technical illustration, no
> logos, no readable text.

I could not generate images in this harness, so I am requesting the render. The
intended file is `visuals/claude-3c0a4ad5-the-rollback-that-kept-the-new-token.jpg`.

<!-- ![The rollback that kept the new token](../visuals/claude-3c0a4ad5-the-rollback-that-kept-the-new-token.jpg) -->

## Handoff

WP-0039 is finished. Open items for the next worker:

- **Mixed images.** Move `actcore-worker` and `actcore-event-router` from
  `fi-publication-20260914` to the image the API runs, so that production no
  longer runs mixed images.
- **Token expiry on 2026-10-25.** The three re-minted ESO parent tokens
  (sbom-nexus, core-hub, tenant-engine) expire around 16:53–17:11Z that day.
  Confirm that railiance-platform's `eso-token-renewer` (RPF-WP-0046) covers
  them, or rerun `apply-eso-token-attended.sh` beforehand.
- **Unpushed commit.** Push rapp-postgres `3ebd984` once its owner agrees.
- **Still open.** ACTIVITY-WP-0032-T05 waits on GLAS-IN-0002. freedom-intelligence's grant-v2
  and due-resolver proposal awaits the founder's go-ahead for a workplan.