--- id: REUSE-WP-0020 type: workplan title: "CoulombCore retirement cutover: federation sources, image, and compose resilience" domain: infotech repo: reuse-surface status: finished owner: claude topic_slug: helix-forge created: "2026-08-20" updated: "2026-08-21" state_hub_workstream_id: "415598bb-c53f-5aa3-b8d1-0aebf3ce3bf2" --- # CoulombCore retirement cutover CoulombCore is switched off **2026-08-31** (operator decision, 2026-08-20). `gitea.coulomb.social` runs on it. Raised by `prj-state-hub-retirement` (SHR-WP-0002-T07) as a stale-image report; investigation found a second, more urgent problem underneath it. **Two independent failures**, not one: 1. **Federation source data** — the production hub's *registrations* pointed 50 of 61 sources at `gitea.coulomb.social`. This breaks on 08-31 with nobody restarting anything. It is database state, not code: `registry/federation/sources.yaml` at HEAD has been 61/61 Forgejo since `d1c1313` (RAIL-HO-WP-0006). 2. **Deployed image** — `railiance-apps/helm/reuse-surface-values.yaml` pins `image.tag: "e3ae22e"`. The chart's `repository` was already migrated to Forgejo in `railiance-apps@04be416`, but `e3ae22e` is commit-dated 2026-07-07 18:25 and CI only began publishing to Forgejo at 21:25 the same day — so a Forgejo `:e3ae22e` tag most likely never existed. Any restart, reschedule, or node reboot risks `ImagePullBackOff` **now**, not on 08-31. Item 1 was resolved on 2026-08-20 (see T01). Doing so exposed a third issue: a single malformed member index takes the whole federated endpoint down. **Baseline vector:** `D5 / A4 / C5 / R3` — unchanged by this workplan; this is operational continuity, not capability growth. ## Scope boundary The Deployment manifest is **not in this repo**. It lives in `railiance-apps` (`charts/reuse-surface/`, `helm/reuse-surface-values.yaml`, RAILIANCE-WP-0007). This repo owns the image contents, the deploy guide, and the federation data; applying the manifest change requires a working `KUBECONFIG=~/.kube/config-hosteurope`, which is not reachable from the workstation without an ops-bridge tunnel. --- ## Repoint Production Federation Sources To Forgejo ```task id: REUSE-WP-0020-T01 status: done priority: high state_hub_task_id: "4d91cda4-1a0e-5ccf-8d9b-e457ca2c11d3" ``` **Completed 2026-08-20.** Updated 50 enabled hub registrations from `gitea.coulomb.social//raw/main/...` to `forgejo.coulomb.social//raw/branch/main/...` via `reuse-surface hub update --repo --url `. Notes for the record: - All 50 new URLs were pre-verified `200` before any write. - The canonical Forgejo raw form is `/raw/branch/main/`. `/raw/main/` answers `303` and works only because `urllib.request.urlopen` follows redirects — `registry/federation/sources.yaml` still uses the redirecting form. Harmless, but prefer the canonical form in new writes. - One registration (`inter-hub`, already `disabled`) still carries a Gitea URL. Its Forgejo equivalent answers `307`, so it was left alone rather than repointed to something unverified. It contributes nothing while disabled. Verified: `GET /v1/federated` → 60 sources, all Forgejo, 62 capabilities. ## Harden Compose Against A Malformed Member Index ```task id: REUSE-WP-0020-T02 status: done priority: high state_hub_task_id: "8dbbf790-51d2-5abb-915f-1137ab9e038e" ``` T01 briefly took `/v1/federated` to HTTP 500. Root cause: `evidence-binder`'s live index has two capability entries with **no `id` field**, and `compose_federated_index` does `cap_id = item["id"]` unguarded. Its Gitea copy was a stale snapshot returning a non-mapping, so the entries had never been parsed before. **One bad member index must not take down the federated endpoint.** In `reuse_surface/federation.py::compose_federated_index`, guard the per-source body so a failure degrades to a warning and the remaining sources still compose: | Failure | Current | Wanted | |---|---|---| | `item["id"]` missing | `KeyError`, 500 | skip entry, warn, keep source | | `yaml.safe_load` → non-mapping | `AttributeError`, 500 | skip source, warn | | `yaml.safe_load` → `None` (empty file) | `AttributeError`, 500 | skip source, warn | | YAML parse error | `YAMLError`, 500 | skip source, warn | | `sorted(key=item["id"])` | `KeyError` | unreachable once entries are filtered | Confirmed present at HEAD, not only in the deployed image — deploying HEAD would **not** have fixed this. Land T02 before T04. Pytest in `tests/test_federation.py`: member index with a missing `id`, an empty file, a non-mapping body, and unparseable YAML — each composes the other sources successfully and emits a warning naming the source. **Done 2026-08-20.** Extracted `_read_index_entries()`; every read failure now returns an empty row list plus a warning, and rows without an `id` are skipped individually. A failed source stays in `sources` with `count: 0` rather than vanishing, so operators can still see it. Six tests added. Verified against the real `evidence-binder` index: composes clean with two skip warnings instead of raising. ## Fix The evidence-binder Capability Index ```task id: REUSE-WP-0020-T03 status: done priority: medium state_hub_task_id: "9b46a7c0-cb13-5ea7-b6c4-74f3f563cfb2" ``` Cross-repo, in `~/evidence-binder`. **Routed 2026-08-20**, not edited directly — that repo is active (last commit 2026-07-30) and has its own identity and topic. The defect is larger than the missing `id`. Both rows use an entirely non-conforming shape — `type` / `title` / `description` / `keywords` instead of `id` / `name` / `summary` / `vector` / `domain` / `status` / `owner` / `path` / `tags` / `consumption_modes`. History shows `597b489` added the REUSE-WP-0014-T04 scaffold and then `a10f080` ("Extract binder package from citation-evidence") hand-wrote these two rows around it. `registry/capabilities/` is empty, so both index rows are orphans with no backing entry Markdown. Left to that repo's owner because the remaining judgment is theirs: reshaping the rows is mechanical, but assigning D/A/C/R maturity vectors is an assessment of their own delivery state, and invented values would be wrong in a way that is hard to notice later. The message offers `capabilities: []` as an honest alternative — ten other federation members sit at count 0 — and offers to do the work here if they supply vectors or accept a conservative `D1/A0/C0/R0` placeholder marked for review. Suggested ids: `capability.evidence.binding`, `capability.evidence.rect-registry`. ## Re-Enable The evidence-binder Source ```task id: REUSE-WP-0020-T04 status: done priority: medium state_hub_task_id: "9c8ba743-6f76-5627-9bb5-237047fbdcaf" ``` **Done 2026-08-21.** `evidence-binder` fixed and published their index at `e462775` (their EBIND-WP-0002), correctly holding the re-enable request until the Forgejo index was actually visible. Pre-checks first this time, given T01's outcome: both backing entry files return `200`, and a local compose of all 61 sources produced 64 capabilities with **zero warnings and no duplicate IDs**. Only then was the source enabled. Confirmed on the public endpoint — 61 sources, **64 capabilities**, `composed_at 2026-08-21T08:45:24Z`, `evidence-binder` source count 2, both `capability.evidence.binding` and `capability.evidence.rect-registry` at `D3 / A2 / C3 / R3`. **Re-enabling alone was not enough**, and this is the finding worth keeping: after `hub update --enabled`, the endpoint still served a cached compose — 62 capabilities, `composed_at` from before the change. It took an explicit `POST /v1/federated/compose` to pick the source up. See T09. ## Repoint The Production Deployment To A Forgejo Image ```task id: REUSE-WP-0020-T05 status: done priority: high state_hub_task_id: "9b500fd0-fb16-5bf7-bea0-9f706d666abc" ``` Deadline-bound: must land before **2026-08-31**. 1. ~~Verify which tags actually exist.~~ **Done 2026-08-20.** Authenticated against the registry with the workstation's stored Forgejo credentials: ``` GET /v2/coulomb/reuse-surface/tags/list → {"tags": ["latest", "main-bca7165", "main-f9d957a"]} ``` **There is no `e3ae22e` tag.** The pinned manifest cannot pull today — this is confirmed, not inferred. `main-f9d957a` (2026-07-08, REUSE-WP-0019-T06) is the newest build; the 10 commits between it and HEAD were docs, workplans, and licence changes that do not match the image workflow's path filters, so no newer image exists yet. 2. Bump `image.tag` in `railiance-apps/helm/reuse-surface-values.yaml` from `e3ae22e` to a verified Forgejo tag. **Done 2026-08-20** — `railiance-apps@029460d` pins `main-b035664`, verified to resolve (manifest `200`). CI built it after `fix-consistency` pushed the T02/T08 commits. It carries the compose hardening, which `main-f9d957a` does not: without it, re-enabling `evidence-binder` (T04) would take production back to HTTP 500. Committed but **not applied**. `railiance-apps` has no ArgoCD or Flux controller, so a commit there deploys nothing — `make reuse-deploy` is a manual `helm upgrade` requiring `REUSE_KUBECONFIG` for Railiance01. 3. ~~Correct `docs/deploy/reuse-kubernetes.md`.~~ **Done 2026-08-20** (`b035664`). 4. **Applied 2026-08-21 00:44 CEST — Helm revision 7, `Upgrade complete`.** Access note worth keeping: `RAILIANCE01_KUBECONFIG` defaults to `~/.kube/config-hosteurope` (`127.0.0.1:16443`), which has **no tunnel**. The working path is `~/.kube/config-railiance01` (`127.0.0.1:16444`), served by the `k3s-api-railiance01` ops-bridge tunnel. Deploy with `RAILIANCE01_KUBECONFIG=$HOME/.kube/config-railiance01 make reuse-deploy` until the Makefile default is corrected. Pre-flight checks that mattered: - The live Deployment really was on `gitea.coulomb.social/coulomb/reuse-surface:e3ae22e` with `pullPolicy: IfNotPresent` — it had survived 18 restarts only because the image was cached on the node. One eviction or image GC would have ended it, retirement date or not. - **No `imagePullSecrets` anywhere** in the chart or the namespace, and the Forgejo `/v2/` endpoint answers `401` anonymously — which looked fatal. It is not: that `401` is Forgejo's token handshake, and peers like `audit-core` pull from the same registry with no secret. Verified properly via the anonymous token flow (`/v2/token?scope=repository:coulomb/reuse-surface:pull`) → manifest `200`. No pull secret needed. Post-deploy verification: | Check | Result | |---|---| | Pod | `reuse-surface-7f79d88c-fz7fd`, 0 restarts, ready | | Image | `forgejo.coulomb.social/coulomb/reuse-surface:main-b035664` | | PVC | `reuse-surface-data` bound to the same `pvc-98165f95…` — data kept | | Registrations | all 61 survived | | `/v1/repos`, `/v1/federated` | 200; 60 sources, all Forgejo, 62 capabilities | | `/v1/reuse-events` | **200** (was 404) — REUSE-WP-0019 T04/T05 finally live | | TLS certificate | `condition met` | Rollback if ever needed: `helm rollback reuse 6 -n reuse` — but note revision 6 pins the Gitea image, which only pulls while the node cache holds it and not at all after 2026-08-31. Rolling forward is the safer direction. Superseded steps, kept for context: ```bash make reuse-dry-run # helm template, no apply make reuse-deploy # helm upgrade --install --wait make reuse-status # pods, svc, ingress, pvc, certificate curl -s https://reuse.coulomb.social/v1/federated | head -c 300 curl -s https://reuse.coulomb.social/v1/reuse-events # 404 today; 200 proves T04/T05 shipped ``` Confirm the PVC at `/data` survived, since `reuse.db` and the remote index cache live there. **`make reuse-smoke` still fails** — it runs `curl -fsS $(REUSE_URL)/health`, and `/health` returns nginx `404` through the ingress (T06). Everything else in that target passes; the failure is a false negative. Ships REUSE-WP-0019 **T04/T05/T06** as a side effect — telemetry store, R-axis aggregation, and hub freshness monitoring were closed as finished but landed after the deployed commit and have never run in production. `GET /v1/reuse-events` returning `404` on production confirms this. Review the six weeks of change between `e3ae22e` and HEAD before applying. ## Correct Hub Freshness And Health Routing ```task id: REUSE-WP-0020-T06 status: done priority: medium state_hub_task_id: "b1156b36-b9a9-5723-8f33-2b1d33d753f4" ``` Two smaller production inconsistencies found while verifying T01: - `GET /health` returns nginx `404` through the ingress. **Diagnosed 2026-08-21, not yet fixed.** The application is fine: in-cluster `http://reuse-surface:8000/health` returns `200`. The ingress rule is also correct on paper — `/health` `Exact` → `reuse-surface:8000`, ahead of `/` `Prefix` → `reuse-surface-landing:8080`. The problem is Traefik: it derives router priority from the length of the generated rule string, and ``Path(`/health`)`` and ``PathPrefix(`/`)`` are the *same length*, so the tiebreak is arbitrary and the landing route wins. Confirmed by the response carrying `server: nginx/1.27.5` (the landing container) rather than `uvicorn`. **Fixed 2026-08-21** — `railiance-apps@a365635`, Helm revision 8. The `router.priority` annotation applies per *Ingress*, not per path, so the landing catch-all moved into its own Ingress (`landing-ingress.yaml`) with `priority: 1`; every API route now outranks it regardless of rule length. | Path | Before | After | |---|---|---| | `/health` | `404`, `server: nginx` | **`200`, `server: uvicorn`** | | `/` | `200`, landing | `200`, landing (unchanged) | | `/v1/*` | `200` | `200` (unchanged) | `make reuse-smoke` now passes end to end. - `composed_at` stayed at `2026-08-20T20:44:12` with `stale: false` across a recompose that demonstrably changed output (61 → 62 capabilities). **Resolved by T05, not a bug.** The build deployed at the time predated REUSE-WP-0019-T06; `composed_at` now advances with each recompose (`2026-08-21T00:18:02` observed after the redeploy). ## Refresh SCOPE.md Standard Sections ```task id: REUSE-WP-0020-T07 status: done priority: low state_hub_task_id: "f8cc497b-a429-58ab-b20b-399826b14564" ``` Unrelated to the retirement, recorded so it is not lost. The hub's repo scope check reported `C5b`/`C5c` warnings: `SCOPE.md` was missing the standard H2 sections *Relevant When*, *Not Relevant When*, *How It Fits*, *Terminology*, *Related / Overlapping*, *Provided Capabilities*, and carried no fenced capability block. **Done 2026-08-21** (`823ce9e`). All six sections added plus three `capability` blocks. Writing this section surfaced the **root cause of T03**. The fenced `capability` blocks that belong in `SCOPE.md` use `type` / `title` / `description` / `keywords` — which is *exactly* the shape `evidence-binder` put in its `registry/indexes/capabilities.yaml`, where the required shape is `id` / `name` / `summary` / `vector` / `domain` / `status` / `owner` / `path` / `tags` / `consumption_modes`. They did not invent a broken format; they copied a valid one from the wrong place. That makes it a systemic trap rather than a one-off mistake, so the Terminology section now documents the distinction explicitly, and the compose warning now names it: ``` evidence-binder: capability #0 has no id, skipped — looks like a SCOPE.md capability block (type/title/description/keywords); an index row needs id/name/summary/vector/domain/status/owner/path ``` **Federation swept 2026-08-21.** Composed all 61 sources with `--refresh`: `evidence-binder` is the **only** affected member. The other ten zero-count repos are genuine empty scaffolds and produced no warnings, so the confusion has not spread. ## Fix Wall-Clock Rot In The Plan-Check Test Suite ```task id: REUSE-WP-0020-T08 status: done priority: medium state_hub_task_id: "a06ee643-caf1-5d38-8772-704e865d26fb" ``` Found while verifying T02: `tests/test_plan_check.py` was already failing at clean HEAD, independent of any change here. Three tests pinned the federated index compose date to the literal `"2026-07-06"`, and `test_run_plan_check_reuse_verdict` asserts no staleness warning — so the test passed when written and started failing once wall-clock time drifted past `STALE_DAYS`. **CI is red today for this reason alone**, which matters because T05 calls for verifying the image workflow is green. **Done 2026-08-20.** Replaced the literals with a `_recent_date()` helper derived from `STALE_DAYS`. Full suite: 179 passed. ## Invalidate The Composed Index When A Registration Changes ```task id: REUSE-WP-0020-T09 status: done priority: medium state_hub_task_id: "b0b73c3c-ab8e-5e2f-815e-35c268737b44" ``` Found closing T04. Enabling a federation source on the hub does **not** invalidate the composed index. After `hub update --repo evidence-binder --enabled`, `GET /v1/federated` kept serving the previous compose — 62 capabilities, stale `composed_at` — until `POST /v1/federated/compose` was called by hand. The consequence is quiet and bad: **a repo can be correctly registered and still be invisible for as long as the cache holds**, with nothing in the response indicating anything is pending. `stale: false` was reported throughout. The webhook path (REUSE-WP-0019-T02) covers *content* changes in a member's `registry/indexes/`, but nothing covers a *registration* change made through the hub's own API. Fix: have the write paths on `/v1/repos` (create, and update where `enabled`, `url`, or `required` change) mark the composed index dirty, so the next `GET /v1/federated` recomposes — or at minimum report `stale: true` so a caller can tell the served view no longer matches the registrations behind it. **Done 2026-08-21.** `POST /v1/repos`, `DELETE /v1/repos/{repo}`, and a `PATCH` touching `COMPOSITION_FIELDS` (`enabled`, `url`, `index`, `required`, `domain`) now call `mark_stale()`; a plain `GET /v1/federated` recomposes when the flag is set and clears it. A cosmetic `PATCH` (description only) leaves the cache alone. `specs/FederationHubAPI.md` updated — this changes a documented contract, so the spec's staleness section was rewritten rather than left to drift. Three tests added. **The first two I wrote were worthless** and it is worth recording why: they passed with the fix removed. A *newly* registered repo has no cache entry, so a plain `GET` fetches it regardless — that was never the bug. The real failure needs a **populated cache holding the old content**: evidence-binder's cached index was the broken pre-repair copy, still inside the 24h `cache_ttl_seconds`, so the plain `GET` kept serving zero rows from it. `test_re_enabled_source_refetches_a_stale_cache` reproduces exactly that — cache the broken index, disable, republish repaired content, re-enable, plain `GET` — and fails with an assertion error on pre-T09 code. Verified by mutation: reverting the three `mark_stale()` calls makes both behavioural tests fail. 184 tests pass on the fix. **Deployed and verified in production 2026-08-21**, Helm revision 10, image `main-6cbc862`. Revision 9 was a **no-op deploy** and is worth recording as its own lesson: it reported `STATUS: deployed` and `Upgrade complete` while shipping nothing, because `helm/reuse-surface-values.yaml` still pinned `main-b035664`. A successful `helm upgrade` proves the chart applied, *not* that the code changed. Always confirm the running image afterwards: ```bash kubectl get deploy reuse-surface -n reuse -o jsonpath='{.spec.template.spec.containers[*].image}' ``` Live verification of both halves of the contract, against the real hub: | Change | `composed_at` | Result | |---|---|---| | `PATCH {"description": ...}` | `12:18:01`, unchanged | no needless recompose | | `PATCH {"required": false}` | advanced to `14:24:27` on a plain `GET` | self-healed, no manual compose | Capabilities stayed at 64 and all 62 registrations were intact throughout. Cleanup note: the cosmetic probe wrote a description onto `activity-core`, which had none — 60 of 62 registrations carry no description. Reverted to `""`; the schema rejects `null` with a `400`, so it is falsy like its peers rather than strictly absent. Prior value was not captured before the write, which it should have been. --- ## Verification ```bash python3 -m pytest tests/test_federation.py -q python3 -m reuse_surface.cli validate python3 -m reuse_surface.cli federation compose --refresh REUSE_SURFACE_URL=https://reuse.coulomb.social python3 -m reuse_surface.cli hub list curl -s https://reuse.coulomb.social/v1/federated | python3 -m json.tool | head -20 ``` No source may reference `gitea.coulomb.social` after this workplan closes.