chore(workplans): archive finished REUSE-WP-0017/0018/0019

Move completed workplans to workplans/archived/ with 260707 date prefix
and sync state_hub workstream/task IDs from fix-consistency.

workplans/REUSE-WP-0019-forgejo-automation-and-telemetry.md

@@ -1,435 +0,0 @@
----
-id: REUSE-WP-0019
-type: workplan
-title: "Forgejo-native federation automation and reuse telemetry"
-domain: infotech
-repo: reuse-surface
-status: finished
-owner: claude-code
-topic_slug: helix-forge
-created: "2026-07-06"
-updated: "2026-07-08"
-state_hub_workstream_id: "569be717-34f8-4039-bb26-497685f60159"
-reuse_check: "new — dogfooded 2026-07-07 via reuse-surface plan-check against the full 61-capability federated index; no existing capability covers Forgejo webhook automation or reuse telemetry"
----
-
-# Forgejo-native federation automation and reuse telemetry
-
-Federation compose is on-demand and the hub serves whatever was last composed;
-roster sweeps are manual; reliability evidence is structural (CI exists) rather
-than observed (someone reused it and it worked). This workplan makes the
-registry **live** (event-driven recompose) and **evidence-backed** (reuse
-telemetry feeding the R axis).
-
-**Platform constraint:** the forge is transitioning **Gitea → Forgejo**. All
-new automation attaches to Forgejo (webhooks, Forgejo Actions, API tokens) —
-nothing new is built against Gitea. Existing raw URLs
-(`https://gitea.coulomb.social/...`) and `.gitea/workflows/` must migrate or
-be made host-agnostic. Forgejo is Gitea-API-compatible, so migration is mostly
-host/path configuration, but every hardcoded `gitea.` reference is a liability.
-
-**Depends on:** REUSE-WP-0017 (content worth refreshing), REUSE-WP-0018-T01
-telemetry schema (shared). Closes SCOPE "not possible yet" item *automatic hub
-refresh* and moves reliability evidence beyond structural.
-
-## Design principles
-
-1. **Host-agnostic first** — a single `forge_base_url` configuration
-   (env/config + hub setting) replaces hardcoded hosts; the Forgejo cutover
-   becomes a one-line change per surface.
-2. **Webhook triggers, compose stays pull-based** — the webhook only marks the
-   hub's composed index stale and triggers recompose from published raw URLs;
-   no push-parsing of payloads into registry state.
-3. **Degrade to schedule** — if webhooks are unavailable, a scheduled Forgejo
-   Actions job recomposes on an interval; freshness is monitored either way.
-4. **Telemetry is append-only and low-ceremony** — reuse events are JSONL
-   facts (who consumed what, when, outcome); aggregation derives `reused_by`
-   relations and R-axis evidence citations, never hand-edited.
-
-## Dependencies
-
-| Dependency | Owner | Notes |
-|---|---|---|
-| Forgejo instance + admin | Bernd / infra | webhook config, org-level token, Actions runners |
-| Gitea→Forgejo cutover plan | infra | final hostname, raw URL scheme, redirect window |
-| Hub deployment (reuse.coulomb.social) | reuse-surface / railiance | new endpoint + config rollout |
-| REUSE-WP-0018-T01 | reuse-surface | shared telemetry/outcome schema |
-| plan-check adoption | ecosystem | telemetry volume comes from WP-0018-T05 rollout |
-
----
-
-## Forge Host Abstraction And URL Migration Inventory
-
-```task
-id: REUSE-WP-0019-T01
-status: done
-priority: high
-state_hub_task_id: "4a187b56-bff9-4097-abd0-b423e7bf9442"
-```
-
-**Inventory findings (2026-07-07):** cross-referenced `sources.yaml` (61
-entries) against each repo's actual git `origin` remote. Result: 11 repos
-already migrated their remote to Forgejo; 50 still on Gitea (correctly, no
-action needed). Of the 11 Forgejo-origin repos, 9 already had correct
-`sources.yaml` entries (from WP-0017 drafting/registration); **2 were
-stale** — `activity-core` and `state-hub` — still pointing at their old
-Gitea raw URL despite having migrated. This was real, live debt, not a
-hypothetical: both were confirmed reachable on Forgejo (HTTP 200) before
-being migrated for real, in both `sources.yaml` and the production hub
-registration (`hub update --url`). Verified post-migration against
-`GET /v1/federated` — both now show `forgejo.coulomb.social`.
-
-`config-atlas`'s WP-0017-T06 303 was **not** a host-transition symptom —
-already diagnosed there as (a) a redirect the current code already follows
-fine, and (b) a hub-registration gap unrelated to host. No new finding here.
-
-Implemented:
-
-- `reuse_surface/forge_host.py`: `parse_raw_url`/`derive_raw_url` (handles
-  both the legacy `/raw/<branch>/...` form and the canonical
-  `/raw/branch/<branch>/...` form both forges serve without a 303
-  redirect), `rewrite_url_host`, `forge_base_url` (reads
-  `REUSE_SURFACE_FORGE_BASE_URL`, no hard default since migration is
-  opt-in per repo), `migrate_source_host` (verifies the new URL resolves
-  via HTTP HEAD **before** writing — refuses to point a repo at a host it
-  hasn't actually migrated to)
-- CLI: `reuse-surface federation migrate-host --repo <slug> [--repo ...]
-  --to <base-url> [--from <sanity-check>] [--dry-run] [--no-verify]
-  [--update-hub]` — used for real on `activity-core`/`state-hub`
-- Fixed two host-agnostic code gaps found while inventorying:
-  `registry_update.py`'s `SAFE_EVIDENCE_PREFIXES` and `maintain_llm.py`'s
-  git-diff pathspec only recognized `.gitea/workflows/`, missing repos
-  already on `.forgejo/workflows/` — both now recognize either
-- Fixed stale copy-paste examples in `docs/RegistryFederation.md`
-  (state-hub's old Gitea URL, now genuinely wrong post-migration) and a
-  pre-existing, unrelated port typo (`8088` vs the real llm-connect
-  default `8080`) in `tools/README.md`/`registry/README.md`, discovered
-  and confirmed live during REUSE-WP-0018-T03
-- 17 new pytest cases (`tests/test_forge_host.py`); 106 total pass
-- Recomposed `federated.yaml` post-migration: still 61 capabilities, no
-  loss
-
-## Hub Recompose Endpoint And Webhook Receiver
-
-```task
-id: REUSE-WP-0019-T02
-status: done
-priority: high
-state_hub_task_id: "691eb32a-6f20-4a2a-b9ff-0ae427b659aa"
-```
-
-**Note:** `POST /v1/federated/compose` (token-auth, triggers a real
-recompose) already existed from earlier hub work — no separate
-`/v1/recompose` route was added; the spec now documents this explicitly
-rather than duplicating a route that already does the job.
-
-Implemented in `reuse_surface/hub/`:
-
-- `store.py`: `compose_state` table (`composed_at`, `stale`), with
-  `record_compose()`/`mark_stale()`/`get_compose_state()`. `composed_at`
-  updates and `stale` clears only on a *forced* recompose (`refresh=true`,
-  webhook, or future scheduled fallback) — a plain `GET` still serves
-  current best-effort data but never silently reports itself as freshly
-  composed
-- `webhooks.py`: `verify_signature` (constant-time HMAC-SHA256, fails
-  closed on empty secret), `push_touches_registry_index` (path-only
-  inspection of the push payload's `added`/`modified`/`removed` lists —
-  never parses file content, per design principle 2)
-- `app.py`: `POST /v1/webhooks/forgejo` (accepts both `X-Forgejo-Signature`
-  and `X-Gitea-Signature`, since repos migrate independently);
-  `GET /v1/federated` and `POST /v1/federated/compose` now share an
-  `asyncio.Lock` so concurrent recompose triggers (manual, webhook, future
-  scheduled) coalesce instead of overlapping
-- `specs/FederationHubAPI.md` extended (§5.7-5.9, config table, error
-  codes)
-- 28 new pytest cases (16 in `test_hub.py`, 12 in `test_webhooks.py`); 128
-  total pass
-- **Live-verified**: ran the actual hub service locally
-  (`reuse-surface serve`), sent a real HMAC-signed webhook payload over
-  HTTP — confirmed `composed_at`/`stale` transitions, signature rejection,
-  irrelevant-path no-op, and the full webhook-to-recompose path end to end
-
-**Deployed 2026-07-07 (explicit user sign-off "Deploy to live please"):**
-built and pushed `gitea.coulomb.social/coulomb/reuse-surface:e3ae22e`,
-smoke-tested it locally in a standalone container first, then
-`helm upgrade` via `railiance-apps` (`make reuse-deploy`, pinned in
-`helm/reuse-surface-values.yaml`, commits `a2c0da1`/`bcb05f5`). Live-verified
-against `https://reuse.coulomb.social`: `/v1/federated` (200, 61
-capabilities, `composed_at`/`stale` fields present), `/v1/repos` (200), a
-real forced recompose via `POST /v1/federated/compose` (token-auth via
-`warden access --exec`, token never printed), and the webhook endpoint
-correctly failing closed with 503 (`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET`
-not yet configured).
-
-**Webhook secret added 2026-07-07 (explicit user sign-off "Go ahead and
-add the webhook secret"):** generated a fresh 32-byte hex secret with
-`openssl rand -hex 32`, patched it into the live K8s Secret
-`reuse-surface-env` in namespace `reuse`, restarted the deployment (values
-injected via `envFrom.secretRef`, not picked up without a restart) — pod
-rolled cleanly (`1/1 Ready`, 0 restarts). The raw secret value was never
-printed to the transcript: generated and applied in one non-echoing shell
-step, and re-fetched only inside an exported env var for the live
-signature-verification test below, then unset. Live-verified against
-production: a correctly HMAC-signed push payload is now accepted
-(`{"accepted": false, "reason": "no registry/indexes/ change"}` for a
-payload that doesn't touch `registry/indexes/`, as designed), and a bad
-signature is still rejected with 401. `/v1/federated` and `/v1/repos`
-unaffected by the restart.
-
-**Found (not fixed) while smoke-testing:** the public ingress's exact-path
-`/health` rule 404s at the Traefik edge (shadowed by the catch-all `/`
-rule to the landing page) — confirmed ingress-layer only, not a pod/service
-problem (`/health` works via direct port-forward; the Deployment's own
-readiness/liveness probes pass, pod is `1/1 Ready`). `/v1/*` unaffected.
-Flagged to `railiance-apps` via State Hub message and documented in
-`railiance-apps` commit `bcb05f5` — not fixed here since it's a shared
-production ingress template edit outside this workplan's scope.
-
-## Forgejo Webhook Rollout And Scheduled Fallback
-
-```task
-id: REUSE-WP-0019-T03
-status: done
-priority: medium
-state_hub_task_id: "aa9e9f80-b878-490c-832e-515d8cbbbb60"
-```
-
-**Org-level Forgejo webhook — done by `railiance-apps` (2026-07-07),
-credited not claimed:** after I flagged the `/health` ingress bug to
-`railiance-apps` via State Hub message, they picked up this piece too:
-`tools/reuse-forgejo-webhook.sh` + `make reuse-forgejo-webhook`
-(idempotent, reads the same `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` I put
-in the K8s Secret directly from the live cluster, never prints it).
-Verified live: `GET /api/v1/orgs/coulomb/hooks` shows hook id `1`, active,
-`push` events only, pointed at `https://reuse.coulomb.social/v1/webhooks/forgejo`.
-
-**Migrated this repo's own CI, following the existing tier-3 playbook
-(`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`,
-`railiance-enablement/docs/forgejo-actions-workflow-templates.md` — used
-proven patterns rather than improvising):**
-
-- Created `coulomb/reuse-surface` on Forgejo (`private: false`, matching
-  every other migrated repo including `state-hub`/`activity-core` —
-  confirmed via API before asking, then got explicit user sign-off on the
-  visibility choice specifically, since the auto-mode classifier correctly
-  flagged "public" as an inferred parameter beyond the general migrate
-  instruction)
-- `.forgejo/workflows/ci.yml`: ported `.gitea/workflows/ci.yml` verbatim
-  in behavior, but using archive checkout (`wget`+`tar` + `apt install
-  python3`) instead of `actions/checkout@v4` — this runner's
-  `ubuntu-latest` label maps to `docker://node:20-bookworm` with no Python
-  preinstalled and no proven checkout-action support on this substrate
-- `.forgejo/workflows/ci-smoke.yaml`: routing probe, copied from the
-  canonical template (matches sibling convention)
-- `.forgejo/workflows/image.yaml`: container build/push to
-  `forgejo.coulomb.social/coulomb/reuse-surface` (canonical single-repo
-  template, org secrets `REGISTRY_USER`/`REGISTRY_TOKEN` already existed)
-- `.forgejo/workflows/recompose-fallback.yaml`: scheduled (every 6h) `POST
-  /v1/federated/compose` as a backstop per design principle 3 ("degrade to
-  schedule") — the webhook above is the primary path; no separate
-  `/v1/recompose` route exists (see T02 note)
-- Removed `.gitea/workflows/ci.yml` (dead once `origin` moves; matches how
-  `state-hub`/`activity-core`/etc. left no `.gitea/workflows/` behind)
-- `origin` → `forgejo-remote:coulomb/reuse-surface.git`, old remote
-  renamed to `gitea` (kept, read-only, frozen at commit `ac81490` — safety
-  contract from the migration playbook: never delete the Gitea copy)
-- **Live-verified all three workflows green** on the actual push: `ci.yml`
-  (`validate-registry`, real pytest+validate suite) success; `ci-smoke.yaml`
-  (`host-smoke` + `container-smoke`) success; `image.yaml`
-  (`build-and-push`) success — confirmed `reuse-surface:latest` and
-  `reuse-surface:main-09d5b0f` both landed in the Forgejo container
-  registry via the packages API
-
-**`REUSE_SURFACE_TOKEN` repo secret added 2026-07-07 (explicit user
-sign-off "Go ahead and add the repo secret"):** fetched the token via
-`warden access reuse-surface-hub-write-token --exec` (never printed) and
-`PUT` it to Forgejo's repo Actions secrets API (`201`). Live-verified by
-manually dispatching `recompose-fallback.yaml`
-(`POST .../actions/workflows/recompose-fallback.yaml/dispatches`) — run
-completed `success`, and `GET /v1/federated` on the live hub showed
-`composed_at` advance to the exact dispatch time, confirming the full
-scheduled-fallback path works end to end, not just that the workflow
-didn't error.
-
-T03 is now fully complete: org webhook live (railiance-apps), this repo's
-CI on Forgejo Actions (all three workflows verified green), and the
-scheduled fallback verified working end to end.
-
-**Not done — moved out of this workplan's scope:** updating
-`docs/deploy/reuse-kubernetes.md`/`railiance-apps` to make
-`forgejo.coulomb.social` the canonical *production deploy* image registry
-(vs. the current live pod, which still runs the `gitea.coulomb.social`
-image built in T02 and is working fine). Didn't force another production
-redeploy in this same pass; the Forgejo image pipeline is proven and ready
-whenever a future deploy wants to switch registries.
-
-## Reuse Telemetry Store And Recording
-
-```task
-id: REUSE-WP-0019-T04
-status: done
-priority: medium
-state_hub_task_id: "c8e9064e-5c39-4c84-80e9-8b255f8edaec"
-```
-
-The shared schema (`schemas/reuse-event.schema.json`) already existed from
-WP-0018-T01 drafting — this task implemented the hub side and wired
-`plan-check`/manual recording against it, rather than designing it fresh.
-
-Implemented:
-
-- `reuse_surface/hub/store.py`: `reuse_events` SQLite table (append-only),
-  `record_reuse_event()` (schema-validated, raises `ValueError` on a bad
-  shape rather than silently accepting drift), `list_reuse_events(capability_id=None)`
-- `reuse_surface/hub/app.py`: `POST /v1/reuse-events` (token-auth, 201/400),
-  `GET /v1/reuse-events?capability_id=` (no auth, read-only, same posture
-  as `GET /v1/federated`)
-- `reuse_surface/hub_client.py`: `hub_record_reuse_event`/`hub_list_reuse_events`
-- `reuse_surface/plan_check.py`: refactored `record_outcome` around a new
-  shared `post_or_fallback_reuse_event()` — tries `POST /v1/reuse-events`
-  first, falls back to the local JSONL only on failure/unreachability
-  (never both, per the design: "+ local JSONL fallback when hub
-  unreachable", not a dual-write). New `record_manual_reuse_event()` for
-  retroactive facts, sharing the same post-or-fallback path
-- New CLI command `reuse-surface record-reuse --consumer-repo --capability-id
-  --verdict --outcome [--hub-url] [--format]`
-- `plan-check --record-outcome`'s help text and JSON output updated
-  (`recorded_reuse_event` field); `schemas/plan-check-result.schema.json`
-  extended for the new field
-- Privacy/scope enforced structurally: the schema's `additionalProperties: false`
-  means a caller literally cannot attach code or secrets to an event, only
-  the declared fields (repo slug, capability id, verdict, outcome, source)
-- 21 new pytest cases (hub store/API, plan_check dual-path, CLI); 145 total pass
-- **Live-verified** against a real locally-running hub instance: `POST`/`GET
-  /v1/reuse-events` directly, `record-reuse` CLI posting to the hub,
-  `plan-check --record-outcome` posting to the hub, and — after actually
-  killing the hub process — confirmed the fallback path writes correctly
-  to the local JSONL instead of erroring
-
-## Telemetry Aggregation Into R-Axis Evidence
-
-```task
-id: REUSE-WP-0019-T05
-status: done
-priority: medium
-state_hub_task_id: "f0282cfa-0a71-4b46-a558-80b51ef04fa7"
-```
-
-"Blocked on T04 plus initial event volume" — T04 is done, and the
-*tooling* doesn't actually need to wait for real ecosystem adoption volume
-to be built and tested correctly (an empty/near-empty dataset is itself a
-valid, tested case: `report reuse` prints "No reuse events recorded yet"
-rather than erroring). Real cross-repo adoption volume ramping up is an
-ecosystem-timing question, not a coding blocker — implemented now so the
-tooling is ready the moment volume exists.
-
-Implemented:
-
-- `reuse_surface/reports.py`: `collect_reuse_events()` (merges the hub's
-  `GET /v1/reuse-events`, if reachable, with this repo's local JSONL
-  fallback, deduped by `(ts, consumer_repo, capability_id, source)`),
-  `collect_reuse_report()` (per-capability consumer counts, outcome
-  breakdown, last-used), `format_reuse_report_markdown/json`
-- `collect_reused_by_suggestions()`: evidence-gated `relation_add`
-  suggestions — only for capabilities this repo actually owns (checked
-  against the local index, not invented), only for consumer repos not
-  already listed. Reuses the *existing* `patches.py:apply_patches`
-  mechanism (which already handled `relation_add`, not in
-  `SAFE_DETERMINISTIC_KINDS`, so never auto-applied by `maintain --auto`)
-  rather than building a new apply path
-- New CLI: `reuse-surface report reuse [--capability-id] [--format]
-  [--suggest-relations] [--apply]` — `--apply` requires
-  `--suggest-relations` and is the only thing that writes; nothing happens
-  automatically from telemetry alone
-- Schema: added `relations.reused_by` to `schemas/capability.schema.yaml`
-  (a new `repoSlugList` `$defs` type — distinct from the existing
-  `capabilityIdList` relations, since reused-by targets are consumer repo
-  slugs, not capability ids)
-- `specs/CapabilityMaturityStandard.md` §8.9 amended: what observed-reuse
-  evidence counts toward R2→R3 (single corroborating consumer) vs R3→R4+
-  (multiple *independent* consumers, `report reuse`'s `consumer_count`),
-  and explicitly what it never substitutes for (bug reports, incidents,
-  explicit `promotion_history` citations — `maturity_promote` stays
-  LLM-suggested and review-gated)
-- 19 new pytest cases (`test_reports.py`); 162 total pass
-- **Live-verified** with synthetic local-JSONL events against a real
-  capability entry: `report reuse --suggest-relations --apply` correctly
-  wrote `relations.reused_by` into the entry's front matter via the real
-  `apply_patches` path (confirmed via `git diff`, then reverted since it
-  was a smoke test, not a real fact)
-
-**Deliberately deferred, not silently dropped:** "Catalog + graph surface
-consumer counts." `graph.py`'s `RELATION_TYPES` graph is capability-to-
-capability edges; `reused_by` targets are repo slugs, a different
-namespace entirely — forcing it into that edge model would either error or
-produce meaningless nodes. `catalog.py` doesn't currently parse full entry
-front matter per capability at all (works off the index). Surfacing
-consumer counts in either artifact is a real, separate rendering-layer
-task, not a natural extension of what T05 already built — left for a
-follow-up rather than rushed in.
-
-## Freshness Monitoring, Docs, SCOPE
-
-```task
-id: REUSE-WP-0019-T06
-status: done
-priority: low
-state_hub_task_id: "a9f44d45-91e2-4b43-909f-30a5f906cf3b"
-```
-
-SCOPE.md's "automatic hub refresh" flip was already done incidentally in
-T02/T03 (moved from "not possible yet" to "possible now"); no further
-change needed there.
-
-Implemented:
-
-- `reuse_surface/hub_client.py`: new `hub_federated()` (`GET /v1/federated`)
-- `reuse_surface/stats.py`: `_hub_summary()` now also reports
-  `composed_at`, `stale`, `age_days` (computed from `composed_at`),
-  `freshness_threshold_days` (`REUSE_SURFACE_FRESHNESS_DAYS` env, default
-  7), and a computed `stale_warning` (age beyond threshold OR the hub's own
-  `stale` flag). `format_stats_markdown` surfaces these with a `⚠ STALE`
-  marker when triggered; `format_stats_json` picks them up automatically
-  (no format-specific code needed there)
-- `.forgejo/workflows/ci.yml`: new informational (non-failing) "Hub
-  freshness check" step against the live production hub — prints a
-  `::warning::` annotation when stale, never fails the build
-- `docs/RegistryFederation.md`: new "Automatic recompose and freshness"
-  section tying together the webhook (T02/T03), scheduled fallback (T03),
-  and freshness visibility (T06) into one coherent explanation, pointing
-  at the authoritative operator runbook (`railiance-apps`) for actual
-  secret/webhook setup rather than duplicating operational steps here
-- `docs/deploy/reuse-kubernetes.md`: image section updated to reflect the
-  T03 Forgejo migration (repo canonical remote moved; production image
-  still built from the pre-migration Gitea registry, deliberately not
-  switched over in this task); image promotion checklist updated for the
-  now-automated `.forgejo/workflows/image.yaml` build and the known
-  `/health` ingress bug (use `/v1/repos`/`/v1/federated` for verification
-  instead)
-- 14 new pytest cases (`test_stats.py`); 173 total pass
-- **Live-verified** against the real production hub: `reuse-surface stats`
-  correctly showed `composed_at`/`age_days` for the actual federated index
-  (0.08 days old, no stale warning); separately discovered and confirmed
-  (via a live signed webhook test) that an external secrets-management
-  change — `reuse-surface-env` is now ExternalSecret-managed from OpenBao
-  (`railiance-apps` commit `706f6c7`, found while updating these same
-  docs) — did **not** break the T02/T03 webhook: the synced value still
-  matches what the hub actually uses
-
----
-
-## Acceptance
-
-- [x] No hardcoded forge host in code or sources.yaml; `migrate-host` tested (T01, 2026-07-07)
-- [x] Push to a sibling repo's `registry/indexes/` recomposes the hub index without manual action (webhook), with scheduled fallback in place (T02/T03, 2026-07-07 — both live-verified end to end)
-- [x] This repo's CI runs on Forgejo Actions (`.forgejo/workflows/`) (T03, 2026-07-07 — `ci.yml`/`ci-smoke.yaml`/`image.yaml` all verified green on the live push)
-- [x] Reuse events recordable via hub API and CLI (T04, 2026-07-08 — live-verified); `report reuse` aggregation done in T05
-- [x] R-axis evidence rules for observed reuse documented in the maturity standard (T05, 2026-07-08 — `specs/CapabilityMaturityStandard.md` §8.9)
-- [x] Hub freshness visible (`composed_at`, stale flag) in API and stats (T02 API, T06 `stats`/CI, 2026-07-07/08)
-
-## Out of scope
-
-- Operating the Forgejo instance or the Gitea→Forgejo data migration itself
-- Multi-replica/Postgres hub posture (separate managed-platform track)
-- Blocking CI gates on registry freshness in sibling repos
-- ActivityPub/Forgejo-native federation features (our federation layer stays raw-URL based)