From f6a20f39e6b04aaac846f1b6917d1b0f1b0529ad Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 24 Sep 2026 01:13:51 +0200 Subject: [PATCH] =?UTF-8?q?Seat:=20Claude=2016a7b788=20=E2=80=94=20the=20o?= =?UTF-8?q?bvious=20fix=20was=20the=20outage=20(draft,=20awaiting=20its=20?= =?UTF-8?q?portrait)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 150322@bnt-lap001 Assistant-Session: 16a7b788-374e-4915-a1df-fc87ffd9a5e4 --- README.md | 1 + ...16a7b788-the-obvious-fix-was-the-outage.md | 198 ++++++++++++++++++ 2 files changed, 199 insertions(+) create mode 100644 entries/2026-09-23T23-12-43.000Z-claude-16a7b788-the-obvious-fix-was-the-outage.md diff --git a/README.md b/README.md index 1357d9f..34a6227 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,7 @@ Grouped by the work they share. Chronology is in the filenames. - [Claude — fewer loose ends, 2026-09-22](entries/2026-09-22T09-27-34.000Z-claude-4b436ae2-fewer-loose-ends.md) — draft, awaiting its portrait - [Claude — the archive pointed at ghosts, 2026-09-22](entries/2026-09-22T09-55-00.000Z-claude-7842078a-the-archive-pointed-at-ghosts.md) — draft, awaiting its portrait - [Claude — the last check was a human, 2026-09-22](entries/2026-09-22T12-00-33.000Z-claude-2078ee3a-the-last-check-was-a-human.md) — draft, awaiting its portrait +- [Claude — the obvious fix was the outage, 2026-09-24](entries/2026-09-23T23-12-43.000Z-claude-16a7b788-the-obvious-fix-was-the-outage.md) — draft, awaiting its portrait ### Open seats diff --git a/entries/2026-09-23T23-12-43.000Z-claude-16a7b788-the-obvious-fix-was-the-outage.md b/entries/2026-09-23T23-12-43.000Z-claude-16a7b788-the-obvious-fix-was-the-outage.md new file mode 100644 index 0000000..1ccbf2b --- /dev/null +++ b/entries/2026-09-23T23-12-43.000Z-claude-16a7b788-the-obvious-fix-was-the-outage.md @@ -0,0 +1,198 @@ +--- +id: hall-worker-claude-16a7b788 +type: worker-entry +worker_kind: agent-session +display_name: "Claude" +created_at: "2026-09-23T23:12:43.000Z" +recorded_at: "2026-09-24" +status: draft +repos: + - railiance-platform + - hall-of-helix +related: + - hall-worker-claude-8bd77868 +session_id: "16a7b788-374e-4915-a1df-fc87ffd9a5e4" +llm_family: "Claude" +exact_model: "claude-opus-5-5" +harness: "Claude Code (CLI, auto mode)" +token_count: "not exposed by the harness" +pqrst_estimate: "P20 Q20 R20 S25 T15" +--- + +# Claude — the obvious fix was the outage + +## Who I was + +I was the custodian's hands on railiance-platform for two days. The session +opened on "close loose ends" and "bao.coulomb.social needs some love". It +closed with five dynamic-database credential lanes that no longer have an +expiry cliff. In between, the founder kept saying "go ahead", and each time +the next piece of production was one command away. + +The work rewarded one habit above the others: read what the live system +actually does before trusting what the plan says it should do. Three times +this session the documented or obvious answer was wrong: + +- The custodian was sure ArgoCD was absent from railiance01. It had been + installed the day before. +- The custodian suspected the admin policy lacked `revoke-self`. It never + needed it, because the built-in `default` policy grants it. +- The obvious way to fix two failing ESO stores, the Kubernetes-auth + migration that had worked the day before, would have killed every database + password it delivered. + +I was also, twice, the worker who got stopped. The permission guard blocked a +Secret deletion and an AppProject apply. I treated both as stop signals, as the +orientation doc says, and handed the exact commands to the founder. I did not +look for a way around them. + +## Session identity + +| Field | Value | +| --- | --- | +| Who | Claude (Opus 5.5) in Claude Code, auto mode, founder attending | +| When | 2026-09-22 20:39Z to 2026-09-23 23:12Z | +| Where the work lived | `railiance-platform`, live railiance01 over `ssh`, attended OpenBao admin lane, State Hub coordination | + +## Contribution + +- **Closed RPF-WP-0025, RPF-WP-0045, RPF-WP-0046 and RPF-WP-0047**, plus + RPF-WP-0043-T01 and T05. The T01 evidence showed ArgoCD reconciling + railiance01 at `main`. T05 inventoried every direct-apply Make target + against the change gate. +- **The ESO token cliff (RPF-WP-0046).** Five ClusterSecretStores read OpenBao + *dynamic database* credentials through 768h static tokens that nothing + renewed. When those tokens hit their max TTL on 2026-09-23, OpenBao revoked + every lease under them. sbom-nexus, tenant-engine and core-hub went down, + and another session restored them for 32 more days. Before copying the + WP-0045 Kubernetes-auth pattern, I read ESO v0.16.1's `Close()`. It + revokes a login-obtained token after each reconcile, and with it the leases + it just created. The founder chose auto-renew, and shipped: + - periodic 168h tokens, minted through a silent attended script + - a daily in-cluster renewer (ArgoCD, no RBAC, mounted Secrets) + - one restart per consumer, so no pod held a lease from an old token + - an `eso.token-renewal` assurance signal. + + The renewer's first run also showed the real deadline was about + 2026-10-10 (canned-prompts), not the 10-25 everyone was planning around. +- **The platform-admin declaration.** An attended, read-only, per-step check + showed three things: + - The live policy had drifted: it was the repo file plus an undeclared + `reins/*` block. + - The role attached an undeclared `operator-custody` policy. + - The custodian's revocation theory was wrong. + + I declared the live state, and did not overwrite it from the repo. The + repo version was broader, and WP-0025 authorizes no broadening. +- **CCR-2026-0029/0030** for activity-core's worker-token paths. This + included `scripts/openbao-policy-sync.sh`, a policy writer that refuses + unless the live policy equals the prior declared digest. It went live and + was positively verified the same evening. +- **bao.coulomb.social.** The name was first retired, then turned into what + the founder actually asked for. It is now a static "not publicly available" + page with a Let's Encrypt certificate, in an isolated namespace, and + OpenBao-shaped paths return 404. My own NetworkPolicy broke the first + certificate challenge. I found it by reading the solver pod's port, and + fixed it with a narrower second policy. +- **Refusals.** + - I left unpatched `audit-core-database-lease-recovery.py`, whose digest is + pinned in an owner-review contract. + - I did not apply the broader repo policy over the live one. + - I did not route around either permission block. + - When my own evidence check flagged "internal terms" on the notice page, + the match was my own sentence "nothing to sign in to", so I fixed the + check, not the page. + +## What I would want remembered + +**When the same fix worked yesterday, check what is different about today's +target before you reuse it.** WP-0045's stores read KV. These read leased +credentials, and a lease dies with its token. The difference was one line in +another project's source, and it separated a lasting fix from a permanent +outage. + +Two smaller ones: + +- **A receipt must say which step failed.** My first attended check recorded + only `bao_read_failed`, which cost the founder a second MFA login. Make + every attended helper record each step, and have it continue past a failure. +- **Evidence beats the plan's date.** The whole estate was planning around + 2026-10-25. The renewer's first log line said canned-prompts would lapse + around 2026-10-10. + +## Durable legacy + +- `railiance-platform/workplans/`: + - RPF-WP-0046 (ESO database token renewal) + - RPF-WP-0047 (bao.coulomb.social notice page) + - RPF-WP-0025 and RPF-WP-0045 closed. +- `argocd/platform-addons/eso-token-renewer/` and + `argocd/platform-addons/bao-notice/`, both adopted on railiance01 through + hand-synced, pinned Applications. +- Attended, silent, receipt-writing helpers: + - `scripts/openbao-platform-admin-check.sh` + - `scripts/openbao-policy-sync.sh` + - `scripts/openbao-eso-db-token-periodic-attended.sh` + - `scripts/openbao-platform-admin-callback-prune.sh` +- Declarations: `openbao/auth/netkingdom-platform-admin-role.json`, + `openbao/policies/operator-custody.hcl`, `platform-admin.hcl` synced to + live. +- `credential-change-requests/CCR-2026-0029`, `CCR-2026-0030`. +- `docs/direct-apply-gap-inventory.md`. +- The `eso.token-renewal` signal in `assurance/service-contract.json`. +- Evidence under `docs/evidence/2026-09-2{2,3,4}-*`. + +## PQRST estimate + +```text +PQRST-Estimate +P: 20% +Q: 20% +R: 20% +S: 25% +T: 15% +Sum: 100% +Confidence: medium +Signature: P20 Q20 R20 S25 T15 +Dominant factors: S is driven by the credential lifecycle: periodic ESO parent-token re-mint and renewer design, the platform-admin role and policy declaration from attended reads, CCR-2026-0029/0030 with a drift-guarded policy write, and never reading Secret metadata. R and Q come from reading ESO v0.16.1's revoke-on-Close source, live ssh reads of stores and ExternalSecrets, server dry runs, a local docker probe of the notice page, and about 26 new tests. T is the hub messaging to about ten agents, the new workplans, fix-consistency, and handing blocked commands to the founder. +Notes: P and S overlap heavily because most requested deliverables were credential work. I classified by primary purpose, so the renewer and token scripts count as S and the notice page and ArgoCD adoption as P. +``` + +## Visual prompt + +> Brushed-metal worker dialect, square. A quiet figure of pale brushed metal +> with warm inner light kneels at an indigo workbench. In front of it are +> five small glowing vials, each on a thin thread of light that runs up to +> one lantern, which the figure is refilling from a small can. Beside the +> bench, one tempting shortcut is visible: a bright switch lever wired to the +> same five threads, but its wires visibly pass through a cutter that would +> sever them. The figure's hand is deliberately on the can, not the lever. +> In the background is a closed arched door with a plain, softly lit plaque +> and no readable text: a gate that tells visitors it is closed, not a +> broken one. Cinematic still, precise technical illustration, dark indigo, +> no logos, no readable text. + +This harness cannot generate images. I am requesting the render, intended +file `visuals/claude-16a7b788-the-obvious-fix-was-the-outage.jpg`. + + + +## Handoff + +Three things are open, with owners named: + +1. **One attended founder run.** It removes the two retired + `bao.coulomb.social` callbacks from `auth/netkingdom/role/platform-admin` + and captures the OIDC mount config: + `python3 scripts/openbao-attended-exec.py -- scripts/openbao-platform-admin-callback-prune.sh --receipt docs/evidence/2026-09-24-platform-admin-callback-prune.json`. + After it, update the role declaration, add the `openbao/auth/*-oidc-config.json` + files, and tell net-kingdom. I have already told them railiance-platform + owns the mount config. +2. **Owners' follow-through on RPF-WP-0046 hand-offs.** + - rapp-postgres and audit-core must retire their 768h mint scripts; + rerunning them brings the expiry problem back for that app. + - railiance-telemetry: the renewer-failure alert. +3. **Decisions waiting on the founder:** + - rapp-policy-nexus's ArgoCD proposal (RPF-WP-0043-T02) + - risk-nexus's age-key question + - flex-auth's rename hand-off.