reuse-surface/workplans/REUSE-WP-0020-coulombcore-retirement-cutover.md

---
id: REUSE-WP-0020
type: workplan
title: "CoulombCore retirement cutover: federation sources, image, and compose resilience"
domain: infotech
repo: reuse-surface
status: active
owner: claude
topic_slug: helix-forge
created: "2026-08-20"
updated: "2026-08-20"
---

# CoulombCore retirement cutover

CoulombCore is switched off **2026-08-31** (operator decision, 2026-08-20).
`gitea.coulomb.social` runs on it. Raised by `prj-state-hub-retirement`
(SHR-WP-0002-T07) as a stale-image report; investigation found a second,
more urgent problem underneath it.

**Two independent failures**, not one:

1. **Federation source data** — the production hub's *registrations* pointed
   50 of 61 sources at `gitea.coulomb.social`. This breaks on 08-31 with
   nobody restarting anything. It is database state, not code:
   `registry/federation/sources.yaml` at HEAD has been 61/61 Forgejo since
   `d1c1313` (RAIL-HO-WP-0006).
2. **Deployed image** — `railiance-apps/helm/reuse-surface-values.yaml` pins
   `image.tag: "e3ae22e"`. The chart's `repository` was already migrated to
   Forgejo in `railiance-apps@04be416`, but `e3ae22e` is commit-dated
   2026-07-07 18:25 and CI only began publishing to Forgejo at 21:25 the same
   day — so a Forgejo `:e3ae22e` tag most likely never existed. Any restart,
   reschedule, or node reboot risks `ImagePullBackOff` **now**, not on 08-31.

Item 1 was resolved on 2026-08-20 (see T01). Doing so exposed a third issue:
a single malformed member index takes the whole federated endpoint down.

**Baseline vector:** `D5 / A4 / C5 / R3` — unchanged by this workplan;
this is operational continuity, not capability growth.

## Scope boundary

The Deployment manifest is **not in this repo**. It lives in `railiance-apps`
(`charts/reuse-surface/`, `helm/reuse-surface-values.yaml`, RAILIANCE-WP-0007).
This repo owns the image contents, the deploy guide, and the federation data;
applying the manifest change requires a working `KUBECONFIG=~/.kube/config-hosteurope`,
which is not reachable from the workstation without an ops-bridge tunnel.

---

## Repoint Production Federation Sources To Forgejo

```task
id: REUSE-WP-0020-T01
status: done
priority: high
```

**Completed 2026-08-20.** Updated 50 enabled hub registrations from
`gitea.coulomb.social/<repo>/raw/main/...` to
`forgejo.coulomb.social/<repo>/raw/branch/main/...` via
`reuse-surface hub update --repo <r> --url <new>`.

Notes for the record:

- All 50 new URLs were pre-verified `200` before any write.
- The canonical Forgejo raw form is `/raw/branch/main/`. `/raw/main/` answers
  `303` and works only because `urllib.request.urlopen` follows redirects —
  `registry/federation/sources.yaml` still uses the redirecting form. Harmless,
  but prefer the canonical form in new writes.
- One registration (`inter-hub`, already `disabled`) still carries a Gitea URL.
  Its Forgejo equivalent answers `307`, so it was left alone rather than
  repointed to something unverified. It contributes nothing while disabled.

Verified: `GET /v1/federated` → 60 sources, all Forgejo, 62 capabilities.

## Harden Compose Against A Malformed Member Index

```task
id: REUSE-WP-0020-T02
status: done
priority: high
```

T01 briefly took `/v1/federated` to HTTP 500. Root cause: `evidence-binder`'s
live index has two capability entries with **no `id` field**, and
`compose_federated_index` does `cap_id = item["id"]` unguarded. Its Gitea copy
was a stale snapshot returning a non-mapping, so the entries had never been
parsed before.

**One bad member index must not take down the federated endpoint.** In
`reuse_surface/federation.py::compose_federated_index`, guard the per-source
body so a failure degrades to a warning and the remaining sources still
compose:

| Failure | Current | Wanted |
|---|---|---|
| `item["id"]` missing | `KeyError`, 500 | skip entry, warn, keep source |
| `yaml.safe_load` → non-mapping | `AttributeError`, 500 | skip source, warn |
| `yaml.safe_load` → `None` (empty file) | `AttributeError`, 500 | skip source, warn |
| YAML parse error | `YAMLError`, 500 | skip source, warn |
| `sorted(key=item["id"])` | `KeyError` | unreachable once entries are filtered |

Confirmed present at HEAD, not only in the deployed image — deploying HEAD
would **not** have fixed this. Land T02 before T04.

Pytest in `tests/test_federation.py`: member index with a missing `id`, an
empty file, a non-mapping body, and unparseable YAML — each composes the
other sources successfully and emits a warning naming the source.

**Done 2026-08-20.** Extracted `_read_index_entries()`; every read failure now
returns an empty row list plus a warning, and rows without an `id` are skipped
individually. A failed source stays in `sources` with `count: 0` rather than
vanishing, so operators can still see it. Six tests added. Verified against the
real `evidence-binder` index: composes clean with two skip warnings instead of
raising.

## Fix The evidence-binder Capability Index

```task
id: REUSE-WP-0020-T03
status: wait
priority: medium
```

Cross-repo, in `~/evidence-binder`. **Routed 2026-08-20**, not edited directly —
that repo is active (last commit 2026-07-30) and has its own identity and topic.

The defect is larger than the missing `id`. Both rows use an entirely
non-conforming shape — `type` / `title` / `description` / `keywords` instead of
`id` / `name` / `summary` / `vector` / `domain` / `status` / `owner` / `path` /
`tags` / `consumption_modes`. History shows `597b489` added the
REUSE-WP-0014-T04 scaffold and then `a10f080` ("Extract binder package from
citation-evidence") hand-wrote these two rows around it. `registry/capabilities/`
is empty, so both index rows are orphans with no backing entry Markdown.

Left to that repo's owner because the remaining judgment is theirs: reshaping
the rows is mechanical, but assigning D/A/C/R maturity vectors is an assessment
of their own delivery state, and invented values would be wrong in a way that is
hard to notice later. The message offers `capabilities: []` as an honest
alternative — ten other federation members sit at count 0 — and offers to do the
work here if they supply vectors or accept a conservative `D1/A0/C0/R0`
placeholder marked for review.

Suggested ids: `capability.evidence.binding`,
`capability.evidence.rect-registry`.

## Re-Enable The evidence-binder Source

```task
id: REUSE-WP-0020-T04
status: wait
priority: medium
```

Blocked on T02 and T03. `evidence-binder` was set `--no-enabled` on the
production hub to restore service. Once its index is valid and compose is
hardened, re-enable and confirm `/v1/federated` returns 61 sources.

## Repoint The Production Deployment To A Forgejo Image

```task
id: REUSE-WP-0020-T05
status: done
priority: high
```

Deadline-bound: must land before **2026-08-31**.

1. ~~Verify which tags actually exist.~~ **Done 2026-08-20.** Authenticated
   against the registry with the workstation's stored Forgejo credentials:

   ```
   GET /v2/coulomb/reuse-surface/tags/list
   → {"tags": ["latest", "main-bca7165", "main-f9d957a"]}
   ```

   **There is no `e3ae22e` tag.** The pinned manifest cannot pull today — this
   is confirmed, not inferred. `main-f9d957a` (2026-07-08, REUSE-WP-0019-T06)
   is the newest build; the 10 commits between it and HEAD were docs, workplans,
   and licence changes that do not match the image workflow's path filters, so
   no newer image exists yet.
2. Bump `image.tag` in `railiance-apps/helm/reuse-surface-values.yaml` from
   `e3ae22e` to a verified Forgejo tag.

   **Done 2026-08-20** — `railiance-apps@029460d` pins `main-b035664`, verified
   to resolve (manifest `200`). CI built it after `fix-consistency` pushed the
   T02/T08 commits. It carries the compose hardening, which `main-f9d957a` does
   not: without it, re-enabling `evidence-binder` (T04) would take production
   back to HTTP 500.

   Committed but **not applied**. `railiance-apps` has no ArgoCD or Flux
   controller, so a commit there deploys nothing — `make reuse-deploy` is a
   manual `helm upgrade` requiring `REUSE_KUBECONFIG` for Railiance01.
3. ~~Correct `docs/deploy/reuse-kubernetes.md`.~~ **Done 2026-08-20** (`b035664`).
4. **Applied 2026-08-21 00:44 CEST — Helm revision 7, `Upgrade complete`.**

   Access note worth keeping: `RAILIANCE01_KUBECONFIG` defaults to
   `~/.kube/config-hosteurope` (`127.0.0.1:16443`), which has **no tunnel**.
   The working path is `~/.kube/config-railiance01` (`127.0.0.1:16444`), served
   by the `k3s-api-railiance01` ops-bridge tunnel. Deploy with
   `RAILIANCE01_KUBECONFIG=$HOME/.kube/config-railiance01 make reuse-deploy`
   until the Makefile default is corrected.

   Pre-flight checks that mattered:

   - The live Deployment really was on `gitea.coulomb.social/coulomb/reuse-surface:e3ae22e`
     with `pullPolicy: IfNotPresent` — it had survived 18 restarts only because
     the image was cached on the node. One eviction or image GC would have
     ended it, retirement date or not.
   - **No `imagePullSecrets` anywhere** in the chart or the namespace, and the
     Forgejo `/v2/` endpoint answers `401` anonymously — which looked fatal.
     It is not: that `401` is Forgejo's token handshake, and peers like
     `audit-core` pull from the same registry with no secret. Verified properly
     via the anonymous token flow (`/v2/token?scope=repository:coulomb/reuse-surface:pull`)
     → manifest `200`. No pull secret needed.

   Post-deploy verification:

   | Check | Result |
   |---|---|
   | Pod | `reuse-surface-7f79d88c-fz7fd`, 0 restarts, ready |
   | Image | `forgejo.coulomb.social/coulomb/reuse-surface:main-b035664` |
   | PVC | `reuse-surface-data` bound to the same `pvc-98165f95…` — data kept |
   | Registrations | all 61 survived |
   | `/v1/repos`, `/v1/federated` | 200; 60 sources, all Forgejo, 62 capabilities |
   | `/v1/reuse-events` | **200** (was 404) — REUSE-WP-0019 T04/T05 finally live |
   | TLS certificate | `condition met` |

   Rollback if ever needed: `helm rollback reuse 6 -n reuse` — but note revision 6
   pins the Gitea image, which only pulls while the node cache holds it and not
   at all after 2026-08-31. Rolling forward is the safer direction.

   Superseded steps, kept for context:

   ```bash
   make reuse-dry-run          # helm template, no apply
   make reuse-deploy           # helm upgrade --install --wait
   make reuse-status           # pods, svc, ingress, pvc, certificate
   curl -s https://reuse.coulomb.social/v1/federated | head -c 300
   curl -s https://reuse.coulomb.social/v1/reuse-events   # 404 today; 200 proves T04/T05 shipped
   ```

   Confirm the PVC at `/data` survived, since `reuse.db` and the remote index
   cache live there.

   **`make reuse-smoke` still fails** — it runs `curl -fsS $(REUSE_URL)/health`,
   and `/health` returns nginx `404` through the ingress (T06). Everything else
   in that target passes; the failure is a false negative.

Ships REUSE-WP-0019 **T04/T05/T06** as a side effect — telemetry store,
R-axis aggregation, and hub freshness monitoring were closed as finished but
landed after the deployed commit and have never run in production.
`GET /v1/reuse-events` returning `404` on production confirms this.

Review the six weeks of change between `e3ae22e` and HEAD before applying.

## Correct Hub Freshness And Health Routing

```task
id: REUSE-WP-0020-T06
status: done
priority: medium
```

Two smaller production inconsistencies found while verifying T01:

- `GET /health` returns nginx `404` through the ingress. **Diagnosed
  2026-08-21, not yet fixed.** The application is fine: in-cluster
  `http://reuse-surface:8000/health` returns `200`. The ingress rule is also
  correct on paper — `/health` `Exact` → `reuse-surface:8000`, ahead of `/`
  `Prefix` → `reuse-surface-landing:8080`. The problem is Traefik: it derives
  router priority from the length of the generated rule string, and
  ``Path(`/health`)`` and ``PathPrefix(`/`)`` are the *same length*, so the
  tiebreak is arbitrary and the landing route wins. Confirmed by the response
  carrying `server: nginx/1.27.5` (the landing container) rather than
  `uvicorn`.

  **Fixed 2026-08-21** — `railiance-apps@a365635`, Helm revision 8. The
  `router.priority` annotation applies per *Ingress*, not per path, so the
  landing catch-all moved into its own Ingress (`landing-ingress.yaml`) with
  `priority: 1`; every API route now outranks it regardless of rule length.

  | Path | Before | After |
  |---|---|---|
  | `/health` | `404`, `server: nginx` | **`200`, `server: uvicorn`** |
  | `/` | `200`, landing | `200`, landing (unchanged) |
  | `/v1/*` | `200` | `200` (unchanged) |

  `make reuse-smoke` now passes end to end.
- `composed_at` stayed at `2026-08-20T20:44:12` with `stale: false` across a
  recompose that demonstrably changed output (61 → 62 capabilities). **Resolved
  by T05, not a bug.** The build deployed at the time predated
  REUSE-WP-0019-T06; `composed_at` now advances with each recompose
  (`2026-08-21T00:18:02` observed after the redeploy).

## Refresh SCOPE.md Standard Sections

```task
id: REUSE-WP-0020-T07
status: done
priority: low
```

Unrelated to the retirement, recorded so it is not lost. The hub's repo scope
check reported `C5b`/`C5c` warnings: `SCOPE.md` was missing the standard H2
sections *Relevant When*, *Not Relevant When*, *How It Fits*, *Terminology*,
*Related / Overlapping*, *Provided Capabilities*, and carried no fenced
capability block.

**Done 2026-08-21** (`823ce9e`). All six sections added plus three `capability`
blocks.

Writing this section surfaced the **root cause of T03**. The fenced `capability`
blocks that belong in `SCOPE.md` use `type` / `title` / `description` /
`keywords` — which is *exactly* the shape `evidence-binder` put in its
`registry/indexes/capabilities.yaml`, where the required shape is `id` / `name`
/ `summary` / `vector` / `domain` / `status` / `owner` / `path` / `tags` /
`consumption_modes`. They did not invent a broken format; they copied a valid
one from the wrong place.

That makes it a systemic trap rather than a one-off mistake, so the Terminology
section now documents the distinction explicitly, and the compose warning now
names it:

```
evidence-binder: capability #0 has no id, skipped — looks like a SCOPE.md
capability block (type/title/description/keywords); an index row needs
id/name/summary/vector/domain/status/owner/path
```

**Federation swept 2026-08-21.** Composed all 61 sources with `--refresh`:
`evidence-binder` is the **only** affected member. The other ten zero-count
repos are genuine empty scaffolds and produced no warnings, so the confusion has
not spread.

## Fix Wall-Clock Rot In The Plan-Check Test Suite

```task
id: REUSE-WP-0020-T08
status: done
priority: medium
```

Found while verifying T02: `tests/test_plan_check.py` was already failing at
clean HEAD, independent of any change here. Three tests pinned the federated
index compose date to the literal `"2026-07-06"`, and
`test_run_plan_check_reuse_verdict` asserts no staleness warning — so the test
passed when written and started failing once wall-clock time drifted past
`STALE_DAYS`. **CI is red today for this reason alone**, which matters because
T05 calls for verifying the image workflow is green.

**Done 2026-08-20.** Replaced the literals with a `_recent_date()` helper
derived from `STALE_DAYS`. Full suite: 179 passed.

---

## Verification

```bash
python3 -m pytest tests/test_federation.py -q
python3 -m reuse_surface.cli validate
python3 -m reuse_surface.cli federation compose --refresh
REUSE_SURFACE_URL=https://reuse.coulomb.social python3 -m reuse_surface.cli hub list
curl -s https://reuse.coulomb.social/v1/federated | python3 -m json.tool | head -20
```

No source may reference `gitea.coulomb.social` after this workplan closes.