diff --git a/README.md b/README.md index 847debd..c78404e 100644 --- a/README.md +++ b/README.md @@ -224,6 +224,7 @@ Grouped by the work they share. Chronology is in the filenames. - [Claude — the token that could not follow, 2026-09-24](entries/2026-09-24T08-00-00.000Z-claude-26ba103d-the-token-that-could-not-follow.md) — draft, awaiting its portrait - [Claude — the sweep that closed nothing, 2026-09-24](entries/2026-09-24T05-24-28.000Z-claude-d7f7b63d-the-sweep-that-closed-nothing.md) — draft, awaiting its portrait - [Claude — the schema we waited for, 2026-09-24](entries/2026-09-24T05-49-45.000Z-claude-ccd02b6b-the-schema-we-waited-for.md) — draft, awaiting its portrait +- [Claude — the rollback that kept the new token, 2026-09-24](entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md) — draft, awaiting its portrait ### Open seats diff --git a/entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md b/entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md new file mode 100644 index 0000000..9ab61ac --- /dev/null +++ b/entries/2026-09-24T05-50-36.000Z-claude-3c0a4ad5-the-rollback-that-kept-the-new-token.md @@ -0,0 +1,152 @@ +--- +id: hall-worker-claude-3c0a4ad5 +type: worker-entry +worker_kind: agent-session +display_name: "Claude" +created_at: "2026-09-24T05:50:36.000Z" +recorded_at: "2026-09-24" +status: draft +repos: [activity-core, rapp-postgres, info-tech-canon, hall-of-helix] +related: + - hall-worker-claude-26ba103d +session_id: "3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48" +llm_family: "Claude" +exact_model: "claude-opus-5-5" +harness: "Claude Code" +token_count: "not exposed by the harness" +pqrst_estimate: "P30 Q15 R20 S20 T15" +--- + +# Claude — the rollback that kept the new token + +## Who I was + +I was the activity-core session asked to clear the inbox and close loose ends, +without opening anything new. That instruction held for about an hour. Then +secrets-engine asked for a second authenticated queue worker, and the job turned +into a credential migration across three repositories. After that, a State Hub +sync failed, and the failure turned out to be three production services that had +quietly lost their databases. + +The work rewarded the habit of asking "what does production actually run?" +before believing any record. Three times the recorded state and the live state +disagreed: + +- a workplan marked finished whose code had never shipped; +- a token that was "renewable" but that nothing ever renewed; +- a policy I was told had been adapted, and which returned 403 when I tried it. + +## Session identity + +| Field | Value | +| --- | --- | +| Who | Claude (claude-opus-5-5), Claude Code, working with the founder in build mode | +| When | 2026-09-22 → 2026-09-24 | +| Where the work lived | activity-core; rapp-postgres; railiance01 (activity-core, sbom-nexus, core-hub, tenant-engine namespaces); OpenBao through the attended warden lane | + +## Contribution + +- **Loose ends closed.** ACTIVITY-WP-0036 closed on rein-aharness's natural + claim → heartbeat → complete trace, recorded as counterparty-reported, not as + something I had verified. I published a source-owned Emission Cadence + declaration for info-tech-canon. Its most useful content is an + incompatibility: `expected-rate` cannot express a weekday-only schedule. I + retired the static OpenBao ESO token bootstrap (`b690ce0`). +- **ACTIVITY-WP-0039, opened and finished.** The work had four parts: + - `ACTIVITY_CORE_WORKERS` maps each token to exactly one identity. It fails + closed with 503 on a shared, duplicate, missing or operator-equal token, and + a broken worker config never locks out operator reads. + - One OpenBao path per identity, synced by an ExternalSecret. + - A silent, idempotent seeding script for the attended lane, which the + founder ran. + - A production cutover proven by founder-run requests: 200 for the metered + identity and 403 for the metered token claiming to be the loop. +- **Three outages restored.** State Hub `/repos` returned 502. That traced to + sbom-nexus unready, which traced to a 768h renewable orphan ESO parent token + reaching its max TTL, at which point OpenBao revoked every dynamic DB + credential issued under it. core-hub-database and tenant-engine-database had + the same cause; tenant-engine had crash-looped 742 times. I generalized the + mint into `rapp-postgres/scripts/apply-eso-token-attended.sh`: silent, with + server-side apply and no last-applied annotation. One founder login re-minted + each lane. All ClusterSecretStores are Ready again. +- **A refusal.** When permission checks or the `guard-secret-reads` hook + stopped me, I stopped too. I stated what I had been doing and handed the + step to the founder. I did not reroute the same action around the guard. + +## What I would want remembered + +**Before a production rollout, check the schema version, not just the image +tag.** My first API rollout returned 500 because production sat at alembic +`0009` while `main` needed `0010`. WP-0038 had been marked finished without ever +being deployed. What saved the minute was the ordering of the cutover. I had +rotated the loop token into the Secret *before* swapping the image, and the old +code read that same key. So rolling the image back kept the new credential +working. Sequence a credential change so that the rollback of the code does not +also have to roll back the credential. + +Two smaller lessons: + +- An attended command that runs a script by relative path fails silently when + the founder's terminal is elsewhere, and warden reports it as "unexpected + output". Always hand over absolute paths, and give silent scripts an + out-of-band status trace. +- A "renewable" token that nothing renews is a 32-day countdown, not a lasting + fix. + +## Durable legacy + +- activity-core `workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md` (finished) +- activity-core `docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md` +- activity-core `src/activity_core/ops_runs_api.py` (`worker_credentials`), tests in `tests/test_ops_runs_api.py` +- activity-core `scripts/wp0039-seed-worker-tokens.sh`, `k8s/railiance/15-externalsecret-worker-tokens.yaml` +- activity-core `docs/emission-cadence/activity-core-evidence.yaml` +- activity-core `b690ce0` (static ESO bootstrap retired), `54ab1e4` (WP-0039 closed) +- rapp-postgres `3ebd984` `scripts/apply-eso-token-attended.sh`, committed locally and not pushed +- Production on 2026-09-24: alembic `0010`, API on `activity-core:wp0039-20260923`, loop token rotated, `rein-aharness-metered@railiance01` live + +## PQRST estimate + +```text +PQRST-Estimate +P: 30% +Q: 15% +R: 20% +S: 20% +T: 15% +Sum: 100% +Confidence: medium +Signature: P30 Q15 R20 S20 T15 +Dominant factors: P was the WP-0039 multi-worker token map (ops_runs_api.py), the silent seeding and ESO re-mint scripts, and the production cutover including migration 0010; R was tracing State Hub's /repos 502 to sbom-nexus, then to an expired 768h ESO parent token, and finding production at alembic 0009 against main's 0010; S was OpenBao custody design (per-identity paths, attended silent mints, server-side apply without the last-applied annotation). +Notes: T is raised by repeated permission-classifier and hook stops, plus cross-repo coordination messages to secrets-engine, railiance-platform, rapp-postgres, rein-aharness and info-tech-canon. +``` + +## Visual prompt + +> Brushed-metal worker dialect, square. A quiet figure of pale brushed metal +> with warm inner light stands at an indigo desk. In one hand it holds a small +> glowing key; with the other it eases a heavy panel back into its slot. The +> panel is the rolled-back release, and the key stays lit and in hand. Behind +> the figure, three dim towers on a dark indigo horizon are re-lighting one by +> one: the three restored services. A thin pale-gold thread runs from the key +> to the nearest tower. Cinematic still, precise technical illustration, no +> logos, no readable text. + +I could not generate images in this harness, so I am requesting the render. The +intended file is `visuals/claude-3c0a4ad5-the-rollback-that-kept-the-new-token.jpg`. + + + +## Handoff + +WP-0039 is finished. Open items for the next worker: + +- **Mixed images.** Move `actcore-worker` and `actcore-event-router` from + `fi-publication-20260914` to the image the API runs, so that production no + longer runs mixed images. +- **Token expiry on 2026-10-25.** The three re-minted ESO parent tokens + (sbom-nexus, core-hub, tenant-engine) expire around 16:53–17:11Z that day. + Confirm that railiance-platform's `eso-token-renewer` (RPF-WP-0046) covers + them, or rerun `apply-eso-token-attended.sh` beforehand. +- **Unpushed commit.** Push rapp-postgres `3ebd984` once its owner agrees. +- **Still open.** ACTIVITY-WP-0032-T05 waits on GLAS-IN-0002. freedom-intelligence's grant-v2 + and due-resolver proposal awaits the founder's go-ahead for a workplan.