At the deployed commit, registry/federation/sources.yaml points at gitea.coulomb.social for 50 of 61 sources; all 61 are forgejo at HEAD after d1c1313. Production is actively federating from the host being switched off, so it breaks on 2026-08-31 with no restart required — a different and worse failure mode than the ImagePullBackOff risk it was first filed under. This also inverts the conservative option: rebuilding the pinned commit onto forgejo does not fix federation, because the gitea URLs are in the code at that commit. Deploying HEAD is the smaller intervention once the runtime dependency is counted. Side finding routed to the repo: REUSE-WP-0019 is archived as finished and records a production deploy of T01/T02, but T04-T06 — including hub freshness monitoring — landed after the deployed commit and appear never to have shipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
408 lines
20 KiB
Markdown
408 lines
20 KiB
Markdown
---
|
||
id: SHR-WP-0002
|
||
type: workplan
|
||
title: "Generation 3 serves from the host being decommissioned"
|
||
domain: infotech
|
||
repo: prj-state-hub-retirement
|
||
status: proposed
|
||
owner: unassigned
|
||
topic_slug: state-hub-retirement
|
||
created: "2026-08-20"
|
||
updated: "2026-08-20"
|
||
state_hub_workstream_id: "c94c3ba1-6c31-4989-a2eb-650726add747"
|
||
---
|
||
|
||
# SHR-WP-0002 — Predecessor generations and deployment reality
|
||
|
||
## The problem in one sentence
|
||
|
||
**Core Hub's production runtime — the generation-3 replacement, cut over
|
||
deliberately and correctly in July — serves from `hub.coulomb.social`, which
|
||
resolves to CoulombCore: the host being decommissioned.** No plan in any
|
||
repository accounts for that.
|
||
|
||
## Correction to this workplan's first draft
|
||
|
||
The first draft of `SHR-WP-0002` (2026-08-20, superseded before any task ran)
|
||
claimed generation 2 had "retired itself by attrition". **That was wrong**, and
|
||
the evidence contradicting it is in `core-hub`'s own archive:
|
||
|
||
- `CORE-WP-0005` finished **2026-07-03**: `hub.coulomb.social` ingress serves
|
||
Core Hub; Inter-Hub compatibility, staging import, dual-run smokes, and
|
||
production cutover gates all closed.
|
||
- `CORE-WP-0007` finished **2026-07-08**: Haskell/IHP infrastructure retired,
|
||
the production Inter-Hub repo renamed/archived, `ihp-railiance-probe`
|
||
archived — after *"a short post-cutover stabilization window"* and explicit
|
||
operator approval to retire the Inter-Hub rollback deployment.
|
||
|
||
The `inter-hub` Deployment scaled `0/0` on railiance01 is that **rollback
|
||
deployment**, held at zero exactly as the workplan describes. It is not a
|
||
lapse; it is the designed end state of a careful retirement.
|
||
|
||
The lesson is the opposite of the one first drafted: **generation 2 was retired
|
||
well.** What nobody checked was whether the successor's runtime was standing on
|
||
durable ground.
|
||
|
||
## What was observed, 2026-08-20
|
||
|
||
| Generation | Position | Observed |
|
||
| --- | --- | --- |
|
||
| 1 — `state-hub` | Being retired by this project, under gates | Running; what the estate uses hourly |
|
||
| 2 — `inter-hub` | Retired `CORE-WP-0005`/`0007`, Jul 2026 | Correctly gone. Rollback deployment at `0/0` as designed |
|
||
| 3 — `core-hub` | The replacement, in production since 2026-07-03 | `hub.coulomb.social` → `92.205.130.254` = **CoulombCore**. No core-hub namespace or Deployment on railiance01 |
|
||
| — `hub-core` | The surviving repository (GOAL §1) | A reusable Python package, not a running service |
|
||
|
||
Three consequences, none recorded anywhere:
|
||
|
||
1. **The gen-1 retirement is being planned onto a runtime that has no home.**
|
||
`G-HUB-RUNTIME` and `G-CORE-ABSORB` presume somewhere to absorb traffic into.
|
||
That somewhere is currently a host with a decommission date, and the gate
|
||
model does not represent that date at all.
|
||
2. **`core-hub/SCOPE.md` is stale in a way that hides this.** It still places
|
||
"retiring production Inter-Hub before migration and smoke evidence exists"
|
||
out of scope and lists `/api/v2` compatibility and "cutover planning from
|
||
Inter-Hub" as in scope — work its own archive shows finished in July. A
|
||
reader cannot tell from the live documents that the cutover already happened.
|
||
3. **`inventory/capabilities.yaml` does not mention `inter-hub`.** The migration
|
||
was real, so this is a smaller gap than first thought — but `G-DISP` still
|
||
has no record of where gen-2 capabilities went, and the estate cannot
|
||
currently answer "does anything still provide the unified operator surface?"
|
||
without reading an archived workplan.
|
||
|
||
## Why this belongs to the project and not to a child repo
|
||
|
||
No single repository can see it. `core-hub` knows its own migration plan;
|
||
`llm-connect` knows its consumers; `ops-warden` knows its access lanes. Only this
|
||
project holds the cross-repository authority, the disposition inventory, and the
|
||
gates — and `G-DISP` ("full disposition inventory"), `G-CORE-ABSORB` and
|
||
`G-HUB-RUNTIME` are all currently gated on assumptions that the observations
|
||
above contradict.
|
||
|
||
Per `SCOPE.md`, this workplan **decides and inventories; it does not implement**.
|
||
Every implementation task it identifies is routed to the owning repository by
|
||
identifier.
|
||
|
||
## Tasks
|
||
|
||
```task
|
||
id: SHR-WP-0002-T01
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "7bf75488-59c9-47ab-a2ff-6ae459dd1444"
|
||
```
|
||
|
||
**Done 2026-08-20 — there was a proper cutover.** `CORE-WP-0005` closed the
|
||
production cutover gates on 2026-07-03; `CORE-WP-0007` retired the Haskell/IHP
|
||
infrastructure by 2026-07-08 with operator approval and a stabilization window.
|
||
The `0/0` rollback deployment is the designed end state.
|
||
|
||
Remaining sub-task: record this in `DECISIONS.md` **with the correction**, so
|
||
the project's own record shows that the first reading was wrong and why. A
|
||
retirement project that misreads a completed retirement as an attrition should
|
||
keep the evidence trail for the next reader.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T02
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "d2dbe425-345f-46cf-af70-0ca4b69a336e"
|
||
```
|
||
|
||
**Extend the disposition inventory backwards to generation 2.** Smaller than
|
||
first thought — the migration was real — but still unrecorded here. `G-DISP` demands
|
||
that every capability reach an explicit destination or retirement decision.
|
||
Applied only to State Hub, it lets gen-2 capabilities vanish unrecorded.
|
||
|
||
Produce `inventory/inter-hub-disposition.yaml` in the shape of the existing
|
||
capability rollup: for each capability gen 2 was to provide — domain hubs, shared
|
||
manifests, widgets, registries, events, unified operator surface — record whether
|
||
it is **served today** (and by what), **planned** (and by whom, under which
|
||
workplan), or **dropped** (and by whose decision).
|
||
|
||
The expected output is uncomfortable and worth having: a list of things the
|
||
estate intended to have, does not have, and has not decided to do without.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T03
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "3254b7d3-e319-44a6-ae63-23b0d080247a"
|
||
```
|
||
|
||
**This is now the workplan's centre of gravity: get the gen-3 runtime off
|
||
CoulombCore.** `G-HUB-RUNTIME` and `G-CORE-ABSORB` presume a runtime to absorb
|
||
traffic into. That runtime is `hub.coulomb.social` on CoulombCore — production
|
||
since 2026-07-03 — plus a `core-hub-staging` tunnel to the same host. There is
|
||
no core-hub presence on railiance01, and `hub-core` is a package, not a service.
|
||
|
||
Decide and record: does core-hub's runtime move to railiance01 first, or does
|
||
consolidation into `hub-core` (GOAL §1) happen directly, skipping a migration to
|
||
a host that is itself scheduled to disappear? Sequence it explicitly **against
|
||
the CoulombCore decommission date**, which is the real constraint and is not
|
||
currently represented in the gate model.
|
||
|
||
Precedent worth reusing: `issue-core` was migrated off CoulombCore on 2026-08-19
|
||
via `ISSUE-WP-0007` and a `rapp-issue-core` package. That is the pattern, and it
|
||
is one week old.
|
||
|
||
**Routed to `core-hub` 2026-08-20; waiting on their answer.** The vehicle is
|
||
`CORE-WP-0010` (runtime absorption into hub-core, `proposed`, all tasks `todo`),
|
||
which depends on `HUB-WP-0004` — itself `proposed`, and still carrying an open
|
||
decision about whether `hub-core` stays an importable library or becomes a
|
||
library plus a permanent thin host.
|
||
|
||
**So the chain that would move production off CoulombCore bottoms out in an
|
||
unanswered architecture question.** That is a sequencing problem rather than an
|
||
implementation one, which is why it sits here and not in a child repo — but the
|
||
answer is core-hub's.
|
||
|
||
Two shapes were put to them:
|
||
|
||
- **(a) Interim move** — package Core Hub as-is onto railiance01, absorb into
|
||
hub-core later on a calm schedule. Costs a migration that would otherwise not
|
||
happen; buys independence from the decommission date.
|
||
- **(b) Absorb directly** — skip the interim host and let `CORE-WP-0010` /
|
||
`HUB-WP-0004` be the migration. Cheaper in total work; couples production
|
||
continuity to finishing an architecture decision under a hard external
|
||
deadline.
|
||
|
||
The project does not choose between them, but records the reasoning either way.
|
||
The one position taken: **leaving it implicit is not acceptable**, because the
|
||
decommission date is a real constraint currently represented in no plan —
|
||
including this project's own gate model, which `T06` addresses.
|
||
|
||
Also asked: whether the decommission date changes `HUB-WP-0004`'s open
|
||
library-vs-thin-host decision. **Pinned 2026-08-20 by the operator: CoulombCore retires by 2026-08-31.**
|
||
Eleven days. See `DECISIONS.md`. On that constraint the project recommends the
|
||
interim move — an architecture decision plus a production migration inside
|
||
eleven days is not a plan. The choice remains core-hub's.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T04
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "b2196340-fa22-4531-81a2-a34bc217d66c"
|
||
```
|
||
|
||
**Reconcile stated scope against reality across participating repositories.**
|
||
The lineage confusion is visible in the documents: `core-hub/SCOPE.md` guards a
|
||
predecessor that is not running; `llm-connect/SCOPE.md` names a consumer
|
||
relationship that cannot exist; `ops-hub` describes itself as an extension *for
|
||
Core Hub*, a repository this project intends to retire into `hub-core`.
|
||
|
||
Produce a table of every participating repository's stated position versus
|
||
observed reality, and route each correction to its owner. Do **not** edit their
|
||
files from here — the project's authority is the ledger, not the prose.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T05
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "2b97a7c5-f877-4020-ba83-8eb00c3690e6"
|
||
```
|
||
|
||
**Sweep the downstream lanes that outlived their subject.** A retired generation
|
||
leaves access and credential lanes behind, and they do not expire on their own.
|
||
|
||
Known instance: ops-warden's routing catalog carries `inter-hub-bootstrap-ssh` —
|
||
`status: active`, `risk: high`, last reviewed 2026-06-24 — a bootstrap SSH
|
||
envelope for a system that runs nowhere, whose runbook points at "the ops-hub
|
||
production activation lane tracked by `CUST-WP-0049`". ops-warden cannot retire
|
||
it alone: the lane may still serve ops-hub activation independently of inter-hub's
|
||
runtime, and only this project can see both sides.
|
||
|
||
Ask each participating repository for lanes, tunnels, credentials and scheduled
|
||
jobs whose subject is a retired or lapsed generation. Route retirements to owners;
|
||
record the sweep here so the next generation change has a checklist rather than a
|
||
memory.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T06
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "9b390f80-1cbb-4485-a958-14b0e24a0b26"
|
||
```
|
||
|
||
**Add a generation-transition gate, so this cannot recur.** The project has gates
|
||
for retiring State Hub deliberately. It has none that would have caught a
|
||
predecessor lapsing, or a successor running only on a host scheduled for
|
||
shutdown.
|
||
|
||
Propose `G-GEN` for `architecture/retirement-gates_v0.1.md`: no generation is
|
||
considered superseded until its capabilities carry explicit dispositions, its
|
||
consumers are reconciled, its downstream lanes are swept, and its successor is
|
||
deployed somewhere that will still exist. Take it through the project's normal
|
||
decision route rather than adding it unilaterally.
|
||
|
||
```task
|
||
id: SHR-WP-0002-T07
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "07636ddf-a26b-4760-8353-2532af9fd938"
|
||
```
|
||
|
||
**The registry dependency, found 2026-08-20 and not previously tracked.**
|
||
`gitea.coulomb.social` is a live container registry on CoulombCore, and two
|
||
railiance01 workloads pull images from it:
|
||
`reuse/Deployment/reuse-surface` and `state-hub/Job/state-hub-alembic-init`
|
||
(checked across Deployments, StatefulSets, DaemonSets, Jobs and CronJobs — that
|
||
is the complete set).
|
||
|
||
Nothing breaks on decommission day, because running pods already hold their
|
||
images. It breaks at the next restart, reschedule or scale-up, as
|
||
`ImagePullBackOff`.
|
||
|
||
**The two are not equally urgent** (corrected 2026-08-20 after checking each
|
||
reference's liveness):
|
||
|
||
- **`reuse-surface` — worse than stale: a live runtime dependency.** A running
|
||
Deployment on the old registry, pinned to a 2026-07-07 commit and **22 commits
|
||
behind** main. Its CI switched to forgejo three hours after that commit and has
|
||
published there ever since; the deployment was never repointed.
|
||
|
||
**Escalated 2026-08-20.** At the deployed commit,
|
||
`registry/federation/sources.yaml` points at `gitea.coulomb.social` for **50 of
|
||
its 61 sources** (all 61 are forgejo at HEAD — migrated in `d1c1313`,
|
||
`RAIL-HO-WP-0006`). So production is not merely running an old image: it is
|
||
**actively federating from the host being switched off**, and that breaks *on
|
||
2026-08-31 with no restart required*. This is a different and worse failure
|
||
mode than the `ImagePullBackOff` risk it was first filed under.
|
||
|
||
It also inverts the conservative option. Rebuilding the pinned commit onto
|
||
forgejo — the apparently low-risk choice — does **not** fix federation, because
|
||
the gitea URLs are in the code at that commit. Deploying HEAD is the smaller
|
||
intervention once the runtime dependency is counted.
|
||
|
||
Side finding for that repo: `REUSE-WP-0019` is archived as finished and records
|
||
a production deploy of its T01/T02 work, but T04–T06 (telemetry store,
|
||
aggregation, **hub freshness monitoring**) landed after the deployed commit and
|
||
appear never to have shipped. A workplan closed as done contains a
|
||
freshness-monitoring feature production does not have.
|
||
- **`state-hub` — mostly a false alarm.** The Deployment is already on
|
||
`forgejo.coulomb.social/coulomb/state-hub:main-d8808bf`. The gitea reference is
|
||
a *completed* one-shot Job that will not re-run by itself; the residual risk is
|
||
a chart template recreating it. Worth cleaning, not urgent.
|
||
|
||
There was **no repository rename**. This is a registry migration (gitea →
|
||
forgejo, early July) in which producers moved and some consumers did not.
|
||
|
||
### Full sweep, 2026-08-20 — CoulombCore runs *two* registry services
|
||
|
||
The first pass looked at container images. `gitea.coulomb.social` also serves a
|
||
**PyPI package index** (`/api/packages/coulomb/pypi`), confirmed by
|
||
`railiance-fabric/fabric/interfaces/railiance-forge-python-package-index.yaml`.
|
||
Both die on 2026-08-31.
|
||
|
||
Sweeping every repo for `gitea.coulomb.social` in operational files (yaml, sh,
|
||
py, Makefile, Dockerfile, toml, json) and classifying by whether anything can
|
||
still act on it:
|
||
|
||
**Live, must move before 2026-08-31**
|
||
|
||
| Reference | What breaks |
|
||
| --- | --- |
|
||
| `reuse-surface` Deployment image | Next restart → `ImagePullBackOff`. Also 22 commits stale |
|
||
| `kaizen-agentic` Makefile + README + 5 docs | **Never migrated at all.** Publishes to gitea PyPI and instructs users to `pip install --extra-index-url …gitea…`. No forgejo equivalent exists in the repo |
|
||
|
||
**Live but already dual-pathed — retire the old half**
|
||
|
||
| Reference | Note |
|
||
| --- | --- |
|
||
| `issue-core/Makefile:240` publishes to gitea PyPI | Line 250 already publishes to forgejo. The gitea target is legacy and should go |
|
||
|
||
**Owned by the thing being decommissioned — expected**
|
||
|
||
`railiance-forge` (`manifests/gitea-ingress.yaml`, `helm/gitea-registry-values.yaml`,
|
||
`tools/gitea-runner-status.sh`) owns gitea itself; these retire *with* it.
|
||
|
||
**Not live — leave alone**
|
||
|
||
Tests asserting historical facts (`railiance-fabric`, `markitect-main`,
|
||
`reuse-surface/tests/test_forge_host.py`, `railiance-platform/tests/…`),
|
||
inventory snapshots and asset/data registers (`disaster-control`,
|
||
`domain-tree`, `railiance-fabric` snapshots), agent session blobs, and
|
||
`issue-core/Dockerfile:5` (a comment). These *record* that gitea existed, which
|
||
stays true after it is switched off. Rewriting them would destroy history to
|
||
tidy a grep.
|
||
|
||
**One to check, not ours:** `railiance-platform/argocd/bootstrap/01-railiance-tenants-project.yaml:14`
|
||
permits `sourceRepos: https://gitea.coulomb.social/coulomb/*.git`. railiance01
|
||
has no ArgoCD `applications` resource type, so this appears inert — but it is a
|
||
bootstrap file and should be confirmed rather than assumed.
|
||
|
||
### Credential-lane sweep, 2026-08-20 — a *third* registry service
|
||
|
||
Checking ops-warden's routing catalog found what the file sweep could not: the
|
||
host serves **three** package services, not two.
|
||
|
||
`gitea.coulomb.social/api/packages/coulomb/npm/` is an **npm registry**, and
|
||
ops-warden's catalog lane `whynot-design-npm-publish` — `risk: high`,
|
||
production-exercised (WP-0018 published `@whynot/design@0.4.0` through it) —
|
||
vends the `NPM_AUTH_TOKEN` that publishes to it.
|
||
|
||
So the complete CoulombCore package surface is:
|
||
|
||
| Service | Endpoint | Known consumers |
|
||
| --- | --- | --- |
|
||
| OCI container registry | `gitea.coulomb.social/coulomb/…` | `reuse-surface` (live Deployment) |
|
||
| PyPI index | `…/api/packages/coulomb/pypi` | `kaizen-agentic` (never migrated), `issue-core` (legacy half) |
|
||
| **npm registry** | `…/api/packages/coulomb/npm/` | `@whynot/design` via ops-warden lane `whynot-design-npm-publish` |
|
||
|
||
**Consequence for the credential lane:** after 2026-08-31 that lane routes to a
|
||
registry that does not exist. It does not fail safe — an operator following it
|
||
gets a token for a dead endpoint and debugs the token. `forgejo-admin-api-token`
|
||
already exists as the forgejo-side equivalent and its keywords include
|
||
`forgejo-npm`, so the destination is plausibly in place; ops-warden must confirm
|
||
rather than assume, and cannot repoint the lane before the packages are on
|
||
forgejo — the same publish-before-repoint rule as `KAIZEN-WP-0010`.
|
||
|
||
Routed to ops-warden (lane owner for the catalog entry; `railiance-platform`
|
||
owns the credential itself).
|
||
|
||
### CI runner sweep, 2026-08-20 — clear, and the sweep is now closed
|
||
|
||
86 repositories carry `.forgejo/workflows`, with jobs on `self-hosted` (83),
|
||
`container-build` (11), `ubuntu-latest` (87) and `docker` (1). Those labels
|
||
resolve to registered runners, so the question was where the runners live.
|
||
|
||
**The only runner in the estate is on railiance01**: `forgejo/forgejo-runner`,
|
||
a Deployment up 48 days, registering against `${FORGEJO_INSTANCE}` — forgejo,
|
||
not gitea. No runner exists on CoulombCore.
|
||
|
||
`railiance-forge/tools/gitea-runner-status.sh`, which prompted this check, is a
|
||
**legacy artifact**: it defaults to `RUNNER_HOST=haskelseed` and probes
|
||
`INTER_HUB_IMAGE`. Both are already retired — haskelseed's bridge on 2026-08-19,
|
||
inter-hub in July. So the gitea-era runner lived on haskelseed and died with it a
|
||
day ago, and nothing broke: evidence in itself that gitea-based CI is no longer
|
||
in use.
|
||
|
||
**CoulombCore's dependency surface is therefore fully enumerated** across four
|
||
methods — tunnels, service DNS, workload image references, operational-file
|
||
grep, credential-lane catalog, and CI runners. Each method found something the
|
||
previous one structurally could not see, which is the finding `T06`'s `G-GEN`
|
||
gate should encode: **an inventory is only as complete as the number of
|
||
independent ways you looked.**
|
||
|
||
`forgejo.coulomb.social` is already on railiance01, so the work is retag, push,
|
||
update manifest. Routed to `railiance-platform`; ownership of the `reuse-surface`
|
||
manifest sits with that repo.
|
||
|
||
**Why this is a project task and not a platform ticket:** it was missed because
|
||
the decommission inventory was assembled from tunnels and workplans. A host is
|
||
not free of dependents because nothing tunnels to it. Feed that into `T06`'s
|
||
`G-GEN` gate — a generation or host transition must enumerate what *pulls,
|
||
resolves and authenticates* against the thing being switched off, not only what
|
||
connects to it.
|
||
|
||
## Related
|
||
|
||
- `SHR-WP-0001` — foundation and architecture baseline (finished)
|
||
- `architecture/retirement-gates_v0.1.md` — `G-DISP`, `G-CORE-ABSORB`, `G-HUB-RUNTIME`
|
||
- `inventory/capabilities.yaml` — the gen-1 rollup this extends
|
||
- `core-hub/INTENT.md` — the stated lineage
|
||
- `core-hub` `CORE-WP-0005` / archived `CORE-WP-0007` — the July cutover and
|
||
Haskell retirement, the evidence that corrected this workplan's first draft
|
||
- `core-hub` `CORE-WP-0010` — runtime absorption into `hub-core`, all tasks
|
||
`todo`; the likely vehicle for T03
|
||
- `issue-core` `ISSUE-WP-0007` — the CoulombCore migration precedent
|
||
- ops-warden `inter-hub-bootstrap-ssh` — a lane outliving its subject
|