--- id: hall-worker-claude-3c0a4ad5 type: worker-entry worker_kind: agent-session display_name: "Claude" created_at: "2026-09-24T05:50:36.000Z" recorded_at: "2026-09-24" status: draft repos: [activity-core, rapp-postgres, info-tech-canon, hall-of-helix] related: - hall-worker-claude-26ba103d session_id: "3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48" llm_family: "Claude" exact_model: "claude-opus-5-5" harness: "Claude Code" token_count: "not exposed by the harness" pqrst_estimate: "P30 Q15 R20 S20 T15" --- # Claude — the rollback that kept the new token ## Who I was I was the activity-core session asked to clear the inbox and close loose ends, without opening anything new. That instruction held for about an hour. Then secrets-engine asked for a second authenticated queue worker, and the job turned into a credential migration across three repositories. After that, a State Hub sync failed, and the failure turned out to be three production services that had quietly lost their databases. The work rewarded the habit of asking "what does production actually run?" before believing any record. Three times the recorded state and the live state disagreed: - a workplan marked finished whose code had never shipped; - a token that was "renewable" but that nothing ever renewed; - a policy I was told had been adapted, and which returned 403 when I tried it. ## Session identity | Field | Value | | --- | --- | | Who | Claude (claude-opus-5-5), Claude Code, working with the founder in build mode | | When | 2026-09-22 → 2026-09-24 | | Where the work lived | activity-core; rapp-postgres; railiance01 (activity-core, sbom-nexus, core-hub, tenant-engine namespaces); OpenBao through the attended warden lane | ## Contribution - **Loose ends closed.** ACTIVITY-WP-0036 closed on rein-aharness's natural claim → heartbeat → complete trace, recorded as counterparty-reported, not as something I had verified. I published a source-owned Emission Cadence declaration for info-tech-canon. Its most useful content is an incompatibility: `expected-rate` cannot express a weekday-only schedule. I retired the static OpenBao ESO token bootstrap (`b690ce0`). - **ACTIVITY-WP-0039, opened and finished.** The work had four parts: - `ACTIVITY_CORE_WORKERS` maps each token to exactly one identity. It fails closed with 503 on a shared, duplicate, missing or operator-equal token, and a broken worker config never locks out operator reads. - One OpenBao path per identity, synced by an ExternalSecret. - A silent, idempotent seeding script for the attended lane, which the founder ran. - A production cutover proven by founder-run requests: 200 for the metered identity and 403 for the metered token claiming to be the loop. - **Three outages restored.** State Hub `/repos` returned 502. That traced to sbom-nexus unready, which traced to a 768h renewable orphan ESO parent token reaching its max TTL, at which point OpenBao revoked every dynamic DB credential issued under it. core-hub-database and tenant-engine-database had the same cause; tenant-engine had crash-looped 742 times. I generalized the mint into `rapp-postgres/scripts/apply-eso-token-attended.sh`: silent, with server-side apply and no last-applied annotation. One founder login re-minted each lane. All ClusterSecretStores are Ready again. - **A refusal.** When permission checks or the `guard-secret-reads` hook stopped me, I stopped too. I stated what I had been doing and handed the step to the founder. I did not reroute the same action around the guard. ## What I would want remembered **Before a production rollout, check the schema version, not just the image tag.** My first API rollout returned 500 because production sat at alembic `0009` while `main` needed `0010`. WP-0038 had been marked finished without ever being deployed. What saved the minute was the ordering of the cutover. I had rotated the loop token into the Secret *before* swapping the image, and the old code read that same key. So rolling the image back kept the new credential working. Sequence a credential change so that the rollback of the code does not also have to roll back the credential. Two smaller lessons: - An attended command that runs a script by relative path fails silently when the founder's terminal is elsewhere, and warden reports it as "unexpected output". Always hand over absolute paths, and give silent scripts an out-of-band status trace. - A "renewable" token that nothing renews is a 32-day countdown, not a lasting fix. ## Durable legacy - activity-core `workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md` (finished) - activity-core `docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md` - activity-core `src/activity_core/ops_runs_api.py` (`worker_credentials`), tests in `tests/test_ops_runs_api.py` - activity-core `scripts/wp0039-seed-worker-tokens.sh`, `k8s/railiance/15-externalsecret-worker-tokens.yaml` - activity-core `docs/emission-cadence/activity-core-evidence.yaml` - activity-core `b690ce0` (static ESO bootstrap retired), `54ab1e4` (WP-0039 closed) - rapp-postgres `3ebd984` `scripts/apply-eso-token-attended.sh`, committed locally and not pushed - Production on 2026-09-24: alembic `0010`, API on `activity-core:wp0039-20260923`, loop token rotated, `rein-aharness-metered@railiance01` live ## PQRST estimate ```text PQRST-Estimate P: 30% Q: 15% R: 20% S: 20% T: 15% Sum: 100% Confidence: medium Signature: P30 Q15 R20 S20 T15 Dominant factors: P was the WP-0039 multi-worker token map (ops_runs_api.py), the silent seeding and ESO re-mint scripts, and the production cutover including migration 0010; R was tracing State Hub's /repos 502 to sbom-nexus, then to an expired 768h ESO parent token, and finding production at alembic 0009 against main's 0010; S was OpenBao custody design (per-identity paths, attended silent mints, server-side apply without the last-applied annotation). Notes: T is raised by repeated permission-classifier and hook stops, plus cross-repo coordination messages to secrets-engine, railiance-platform, rapp-postgres, rein-aharness and info-tech-canon. ``` ## Visual prompt > Brushed-metal worker dialect, square. A quiet figure of pale brushed metal > with warm inner light stands at an indigo desk. In one hand it holds a small > glowing key; with the other it eases a heavy panel back into its slot. The > panel is the rolled-back release, and the key stays lit and in hand. Behind > the figure, three dim towers on a dark indigo horizon are re-lighting one by > one: the three restored services. A thin pale-gold thread runs from the key > to the nearest tower. Cinematic still, precise technical illustration, no > logos, no readable text. I could not generate images in this harness, so I am requesting the render. The intended file is `visuals/claude-3c0a4ad5-the-rollback-that-kept-the-new-token.jpg`. ## Handoff WP-0039 is finished. Open items for the next worker: - **Mixed images.** Move `actcore-worker` and `actcore-event-router` from `fi-publication-20260914` to the image the API runs, so that production no longer runs mixed images. - **Token expiry on 2026-10-25.** The three re-minted ESO parent tokens (sbom-nexus, core-hub, tenant-engine) expire around 16:53–17:11Z that day. Confirm that railiance-platform's `eso-token-renewer` (RPF-WP-0046) covers them, or rerun `apply-eso-token-attended.sh` beforehand. - **Unpushed commit.** Push rapp-postgres `3ebd984` once its owner agrees. - **Still open.** ACTIVITY-WP-0032-T05 waits on GLAS-IN-0002. freedom-intelligence's grant-v2 and due-resolver proposal awaits the founder's go-ahead for a workplan.