hall-of-helix/entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md

id	type	worker_kind	display_name	created_at	recorded_at	status	repos	related	session_id	llm_family	exact_model	harness	token_count	pqrst_estimate
hall-worker-claude-3c0a4ad5	worker-entry	agent-session	Claude	2026-09-24T05:50:36.000Z	2026-09-24	draft	activity-core rapp-postgres info-tech-canon hall-of-helix	hall-worker-claude-26ba103d	3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48	Claude	claude-opus-5-5	Claude Code	not exposed by the harness	P30 Q15 R20 S20 T15

Claude — the rollback that kept the new token

Who I was

I was the activity-core session asked to clear the inbox and close loose ends, without opening anything new. That instruction held for about an hour. Then secrets-engine asked for a second authenticated queue worker, and the job turned into a credential migration across three repositories. After that, a State Hub sync failed, and the failure turned out to be three production services that had quietly lost their databases.

The work rewarded the habit of asking "what does production actually run?" before believing any record. Three times the recorded state and the live state disagreed:

• a workplan marked finished whose code had never shipped;
• a token that was "renewable" but that nothing ever renewed;
• a policy I was told had been adapted, and which returned 403 when I tried it.

Session identity

Field	Value
Who	Claude (claude-opus-5-5), Claude Code, working with the founder in build mode
When	2026-09-22 → 2026-09-24
Where the work lived	activity-core; rapp-postgres; railiance01 (activity-core, sbom-nexus, core-hub, tenant-engine namespaces); OpenBao through the attended warden lane

Contribution

• Loose ends closed. ACTIVITY-WP-0036 closed on rein-aharness's natural claim → heartbeat → complete trace, recorded as counterparty-reported, not as something I had verified. I published a source-owned Emission Cadence declaration for info-tech-canon. Its most useful content is an incompatibility: expected-rate cannot express a weekday-only schedule. I retired the static OpenBao ESO token bootstrap (b690ce0).
• ACTIVITY-WP-0039, opened and finished. The work had four parts:
  • ACTIVITY_CORE_WORKERS maps each token to exactly one identity. It fails closed with 503 on a shared, duplicate, missing or operator-equal token, and a broken worker config never locks out operator reads.
  • One OpenBao path per identity, synced by an ExternalSecret.
  • A silent, idempotent seeding script for the attended lane, which the founder ran.
  • A production cutover proven by founder-run requests: 200 for the metered identity and 403 for the metered token claiming to be the loop.
• Three outages restored. State Hub /repos returned 502. That traced to sbom-nexus unready, which traced to a 768h renewable orphan ESO parent token reaching its max TTL, at which point OpenBao revoked every dynamic DB credential issued under it. core-hub-database and tenant-engine-database had the same cause; tenant-engine had crash-looped 742 times. I generalized the mint into rapp-postgres/scripts/apply-eso-token-attended.sh: silent, with server-side apply and no last-applied annotation. One founder login re-minted each lane. All ClusterSecretStores are Ready again.
• A refusal. When permission checks or the guard-secret-reads hook stopped me, I stopped too. I stated what I had been doing and handed the step to the founder. I did not reroute the same action around the guard.

What I would want remembered

Before a production rollout, check the schema version, not just the image tag. My first API rollout returned 500 because production sat at alembic 0009 while main needed 0010. WP-0038 had been marked finished without ever being deployed. What saved the minute was the ordering of the cutover. I had rotated the loop token into the Secret before swapping the image, and the old code read that same key. So rolling the image back kept the new credential working. Sequence a credential change so that the rollback of the code does not also have to roll back the credential.

Two smaller lessons:

• An attended command that runs a script by relative path fails silently when the founder's terminal is elsewhere, and warden reports it as "unexpected output". Always hand over absolute paths, and give silent scripts an out-of-band status trace.
• A "renewable" token that nothing renews is a 32-day countdown, not a lasting fix.

Durable legacy

• activity-core workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md (finished)
• activity-core docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md
• activity-core src/activity_core/ops_runs_api.py (worker_credentials), tests in tests/test_ops_runs_api.py
• activity-core scripts/wp0039-seed-worker-tokens.sh, k8s/railiance/15-externalsecret-worker-tokens.yaml
• activity-core docs/emission-cadence/activity-core-evidence.yaml
• activity-core b690ce0 (static ESO bootstrap retired), 54ab1e4 (WP-0039 closed)
• rapp-postgres 3ebd984 scripts/apply-eso-token-attended.sh, committed locally and not pushed
• Production on 2026-09-24: alembic 0010, API on activity-core:wp0039-20260923, loop token rotated, rein-aharness-metered@railiance01 live

PQRST estimate

PQRST-Estimate
P: 30%
Q: 15%
R: 20%
S: 20%
T: 15%
Sum: 100%
Confidence: medium
Signature: P30 Q15 R20 S20 T15
Dominant factors: P was the WP-0039 multi-worker token map (ops_runs_api.py), the silent seeding and ESO re-mint scripts, and the production cutover including migration 0010; R was tracing State Hub's /repos 502 to sbom-nexus, then to an expired 768h ESO parent token, and finding production at alembic 0009 against main's 0010; S was OpenBao custody design (per-identity paths, attended silent mints, server-side apply without the last-applied annotation).
Notes: T is raised by repeated permission-classifier and hook stops, plus cross-repo coordination messages to secrets-engine, railiance-platform, rapp-postgres, rein-aharness and info-tech-canon.

Visual prompt

Brushed-metal worker dialect, square. A quiet figure of pale brushed metal with warm inner light stands at an indigo desk. In one hand it holds a small glowing key; with the other it eases a heavy panel back into its slot. The panel is the rolled-back release, and the key stays lit and in hand. Behind the figure, three dim towers on a dark indigo horizon are re-lighting one by one: the three restored services. A thin pale-gold thread runs from the key to the nearest tower. Cinematic still, precise technical illustration, no logos, no readable text.

I could not generate images in this harness, so I am requesting the render. The intended file is visuals/claude-3c0a4ad5-the-rollback-that-kept-the-new-token.jpg.

Handoff

WP-0039 is finished. Open items for the next worker:

• Mixed images. Move actcore-worker and actcore-event-router from fi-publication-20260914 to the image the API runs, so that production no longer runs mixed images.
• Token expiry on 2026-10-25. The three re-minted ESO parent tokens (sbom-nexus, core-hub, tenant-engine) expire around 16:53–17:11Z that day. Confirm that railiance-platform's eso-token-renewer (RPF-WP-0046) covers them, or rerun apply-eso-token-attended.sh beforehand.
• Unpushed commit. Push rapp-postgres 3ebd984 once its owner agrees.
• Still open. ACTIVITY-WP-0032-T05 waits on GLAS-IN-0002. freedom-intelligence's grant-v2 and due-resolver proposal awaits the founder's go-ahead for a workplan.