reuse-surface/workplans/REUSE-WP-0020-coulombcore-retirement-cutover.md
tegwick 0c6b1e2538 Harden federated compose against malformed member indexes (REUSE-WP-0020)
Repointing the production hub's 50 Gitea-hosted federation sources to Forgejo
ahead of the 2026-08-31 CoulombCore retirement took /v1/federated to HTTP 500.
One member index (evidence-binder) has capability rows with no `id`, and
compose_federated_index dereferenced item["id"] unguarded. Its Gitea copy was a
stale snapshot returning a non-mapping, so those rows had never been parsed.

A single malformed member index must not take down the whole endpoint. Extract
_read_index_entries(): unparseable YAML, a non-mapping body, an empty file, and
a non-list `capabilities` each degrade to a warning and an empty row list, and
rows without an `id` are skipped individually. A failed source stays listed with
count 0 so it remains visible to operators rather than silently disappearing.

Also fix wall-clock rot in tests/test_plan_check.py, which was already failing
at clean HEAD: three tests pinned the compose date to a literal that has now
aged past STALE_DAYS.

Add workplan REUSE-WP-0020 covering the full cutover.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 00:04:39 +02:00

9.2 KiB

id type title domain repo status owner topic_slug created updated
REUSE-WP-0020 workplan CoulombCore retirement cutover: federation sources, image, and compose resilience infotech reuse-surface active claude helix-forge 2026-08-20 2026-08-20

CoulombCore retirement cutover

CoulombCore is switched off 2026-08-31 (operator decision, 2026-08-20). gitea.coulomb.social runs on it. Raised by prj-state-hub-retirement (SHR-WP-0002-T07) as a stale-image report; investigation found a second, more urgent problem underneath it.

Two independent failures, not one:

  1. Federation source data — the production hub's registrations pointed 50 of 61 sources at gitea.coulomb.social. This breaks on 08-31 with nobody restarting anything. It is database state, not code: registry/federation/sources.yaml at HEAD has been 61/61 Forgejo since d1c1313 (RAIL-HO-WP-0006).
  2. Deployed imagerailiance-apps/helm/reuse-surface-values.yaml pins image.tag: "e3ae22e". The chart's repository was already migrated to Forgejo in railiance-apps@04be416, but e3ae22e is commit-dated 2026-07-07 18:25 and CI only began publishing to Forgejo at 21:25 the same day — so a Forgejo :e3ae22e tag most likely never existed. Any restart, reschedule, or node reboot risks ImagePullBackOff now, not on 08-31.

Item 1 was resolved on 2026-08-20 (see T01). Doing so exposed a third issue: a single malformed member index takes the whole federated endpoint down.

Baseline vector: D5 / A4 / C5 / R3 — unchanged by this workplan; this is operational continuity, not capability growth.

Scope boundary

The Deployment manifest is not in this repo. It lives in railiance-apps (charts/reuse-surface/, helm/reuse-surface-values.yaml, RAILIANCE-WP-0007). This repo owns the image contents, the deploy guide, and the federation data; applying the manifest change requires a working KUBECONFIG=~/.kube/config-hosteurope, which is not reachable from the workstation without an ops-bridge tunnel.


Repoint Production Federation Sources To Forgejo

id: REUSE-WP-0020-T01
status: done
priority: high

Completed 2026-08-20. Updated 50 enabled hub registrations from gitea.coulomb.social/<repo>/raw/main/... to forgejo.coulomb.social/<repo>/raw/branch/main/... via reuse-surface hub update --repo <r> --url <new>.

Notes for the record:

  • All 50 new URLs were pre-verified 200 before any write.
  • The canonical Forgejo raw form is /raw/branch/main/. /raw/main/ answers 303 and works only because urllib.request.urlopen follows redirects — registry/federation/sources.yaml still uses the redirecting form. Harmless, but prefer the canonical form in new writes.
  • One registration (inter-hub, already disabled) still carries a Gitea URL. Its Forgejo equivalent answers 307, so it was left alone rather than repointed to something unverified. It contributes nothing while disabled.

Verified: GET /v1/federated → 60 sources, all Forgejo, 62 capabilities.

Harden Compose Against A Malformed Member Index

id: REUSE-WP-0020-T02
status: done
priority: high

T01 briefly took /v1/federated to HTTP 500. Root cause: evidence-binder's live index has two capability entries with no id field, and compose_federated_index does cap_id = item["id"] unguarded. Its Gitea copy was a stale snapshot returning a non-mapping, so the entries had never been parsed before.

One bad member index must not take down the federated endpoint. In reuse_surface/federation.py::compose_federated_index, guard the per-source body so a failure degrades to a warning and the remaining sources still compose:

Failure Current Wanted
item["id"] missing KeyError, 500 skip entry, warn, keep source
yaml.safe_load → non-mapping AttributeError, 500 skip source, warn
yaml.safe_loadNone (empty file) AttributeError, 500 skip source, warn
YAML parse error YAMLError, 500 skip source, warn
sorted(key=item["id"]) KeyError unreachable once entries are filtered

Confirmed present at HEAD, not only in the deployed image — deploying HEAD would not have fixed this. Land T02 before T04.

Pytest in tests/test_federation.py: member index with a missing id, an empty file, a non-mapping body, and unparseable YAML — each composes the other sources successfully and emits a warning naming the source.

Done 2026-08-20. Extracted _read_index_entries(); every read failure now returns an empty row list plus a warning, and rows without an id are skipped individually. A failed source stays in sources with count: 0 rather than vanishing, so operators can still see it. Six tests added. Verified against the real evidence-binder index: composes clean with two skip warnings instead of raising.

Fix The evidence-binder Capability Index

id: REUSE-WP-0020-T03
status: todo
priority: medium

Cross-repo, in ~/evidence-binder. Both entries in registry/indexes/capabilities.yaml ("Evidence-to-target binding", "Visual-guide rect registry") lack the required id. Add ids consistent with the registry naming convention, confirm against schemas/capability.schema.yaml, and check whether registry/capabilities/*.md entries exist behind them or whether the index rows are orphans.

Route rather than edit directly if that repo has an owning agent.

Re-Enable The evidence-binder Source

id: REUSE-WP-0020-T04
status: wait
priority: medium

Blocked on T02 and T03. evidence-binder was set --no-enabled on the production hub to restore service. Once its index is valid and compose is hardened, re-enable and confirm /v1/federated returns 61 sources.

Repoint The Production Deployment To A Forgejo Image

id: REUSE-WP-0020-T05
status: todo
priority: high

Deadline-bound: must land before 2026-08-31.

  1. Verify which tags actually exist in forgejo.coulomb.social/coulomb/reuse-surface (registry API needs auth; anonymous GET /v2/.../tags/list returns 401) and that .forgejo/workflows/image.yaml is green on main.
  2. Bump image.tag in railiance-apps/helm/reuse-surface-values.yaml from e3ae22e to a verified Forgejo tag at current HEAD.
  3. Correct docs/deploy/reuse-kubernetes.md, which still describes Gitea as the live registry and its manual docker build/push commands still target gitea.coulomb.social.
  4. Apply from a host with cluster access; verify rollout, /v1/federated, and that the PVC at /data survived.

Ships REUSE-WP-0019 T04/T05/T06 as a side effect — telemetry store, R-axis aggregation, and hub freshness monitoring were closed as finished but landed after the deployed commit and have never run in production. GET /v1/reuse-events returning 404 on production confirms this.

Review the six weeks of change between e3ae22e and HEAD before applying.

Correct Hub Freshness And Health Routing

id: REUSE-WP-0020-T06
status: todo
priority: low

Two smaller production inconsistencies found while verifying T01:

  • GET /health returns nginx 404 through the ingress, though docs/deploy/reuse-kubernetes.md documents it as the liveness path and the landing page links it. Only /v1/* routes. Pod probes hit the container directly so the service is unaffected, but the documented URL is wrong — fix the ingress route or the docs, whichever matches intent.
  • composed_at stayed at 2026-08-20T20:44:12 with stale: false across a recompose that demonstrably changed output (61 → 62 capabilities). The freshness timestamp is not tracking recomposes. Re-check after T05, since REUSE-WP-0019-T06 touches exactly this and is not deployed.

Refresh SCOPE.md Standard Sections

id: REUSE-WP-0020-T07
status: todo
priority: low

Unrelated to the retirement, recorded so it is not lost. The hub's repo scope check reports C5b/C5c warnings: SCOPE.md is missing the standard H2 sections Relevant When, Not Relevant When, How It Fits, Terminology, Related / Overlapping, Provided Capabilities, and carries no fenced capability block.

Fix Wall-Clock Rot In The Plan-Check Test Suite

id: REUSE-WP-0020-T08
status: done
priority: medium

Found while verifying T02: tests/test_plan_check.py was already failing at clean HEAD, independent of any change here. Three tests pinned the federated index compose date to the literal "2026-07-06", and test_run_plan_check_reuse_verdict asserts no staleness warning — so the test passed when written and started failing once wall-clock time drifted past STALE_DAYS. CI is red today for this reason alone, which matters because T05 calls for verifying the image workflow is green.

Done 2026-08-20. Replaced the literals with a _recent_date() helper derived from STALE_DAYS. Full suite: 179 passed.


Verification

python3 -m pytest tests/test_federation.py -q
python3 -m reuse_surface.cli validate
python3 -m reuse_surface.cli federation compose --refresh
REUSE_SURFACE_URL=https://reuse.coulomb.social python3 -m reuse_surface.cli hub list
curl -s https://reuse.coulomb.social/v1/federated | python3 -m json.tool | head -20

No source may reference gitea.coulomb.social after this workplan closes.