REUSE-WP-0019-T06: hub freshness monitoring, docs, close workplan
Some checks failed
CI Smoke / host-smoke (push) Successful in 1s
CI Smoke / container-smoke (push) Successful in 2s
ci / validate-registry (push) Has been cancelled
Build and Publish Container Image / build-and-push (push) Successful in 1m4s

reuse_surface/stats.py: _hub_summary() now reports composed_at, stale,
age_days, freshness_threshold_days (REUSE_SURFACE_FRESHNESS_DAYS env,
default 7), and a computed stale_warning. New hub_client.hub_federated()
backs it. format_stats_markdown surfaces a STALE marker when triggered.

.forgejo/workflows/ci.yml: new informational (non-failing) hub freshness
check against the live production hub on every push -- prints a
:⚠️: annotation when stale, never fails the build.

docs/RegistryFederation.md: new section tying together the webhook (T02),
scheduled fallback (T03), and freshness visibility (T06) into one
explanation. docs/deploy/reuse-kubernetes.md: updated for the T03 Forgejo
migration and the now-automated image.yaml build; image promotion
checklist updated for the known /health ingress bug (verify via
/v1/repos or /v1/federated instead).

14 new pytest cases, 173 total pass. Live-verified against production:
reuse-surface stats correctly showed composed_at/age_days for the real
federated index. Separately discovered and confirmed (via a live signed
webhook test) that reuse-surface-env moving to ExternalSecret/OpenBao
custody (railiance-apps commit 706f6c7, found while updating these docs)
did not break the T02/T03 webhook -- the synced value still matches what
the hub actually uses.

REUSE-WP-0019 is now fully complete (T01-T06). SCOPE.md and
docs/IntentScopeGapAnalysis.md updated to reflect closure.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-08 00:09:58 +02:00
parent 706f6c70fe
commit f9d957a221
10 changed files with 328 additions and 23 deletions

View file

@ -54,6 +54,26 @@ jobs:
working-directory: /tmp/repo
run: reuse-surface stats || true
- name: Hub freshness check (informational, REUSE-WP-0019-T06)
working-directory: /tmp/repo
env:
REUSE_SURFACE_URL: https://reuse.coulomb.social
run: |
reuse-surface stats --format json > /tmp/stats.json 2>/dev/null || true
python3 -c "
import json
try:
hub = json.load(open('/tmp/stats.json')).get('hub', {})
except Exception:
hub = {}
if hub.get('stale_warning'):
print(f\"::warning::hub federated index is stale (composed_at={hub.get('composed_at')}, age_days={hub.get('age_days')}, threshold={hub.get('freshness_threshold_days')})\")
elif hub.get('configured'):
print(f\"hub federated index fresh (age_days={hub.get('age_days')})\")
else:
print('hub not reachable/configured in CI -- skipped')
" || true
- name: Workstation roster federation stats (informational)
working-directory: /tmp/repo
run: |

View file

@ -146,7 +146,9 @@ See `tools/README.md` for command reference.
- **Searchable catalog:** `docs/catalog/search.html`.
- **Workplans:** REUSE-WP-0001 through REUSE-WP-0015 finished (archived);
**REUSE-WP-0017** capability coverage campaign finished (2026-07-07);
**REUSE-WP-0018** plan-check consumption loop active next.
**REUSE-WP-0018** plan-check consumption loop finished (2026-07-07);
**REUSE-WP-0019** Forgejo automation and reuse telemetry finished
(2026-07-08, all six tasks T01T06).
- **Assessment history:** `history/` — intent/scope assessments, rollout
milestone, dedup plan, per-repo follow-up.
- **Self-assessed vector:** `D5 / A4 / C5 / R3` (see `docs/IntentScopeGapAnalysis.md`).

View file

@ -241,3 +241,4 @@ See §4 and archived workplans `workplans/archived/`.
| 2026-07-07 | **REUSE-WP-0018** partially closed priority 29 (`plan-check` deterministic matching + State Hub capability-request bridge; LLM rerank and ecosystem rollout remain open) |
| 2026-07-07 | **REUSE-WP-0018-T05** closed following REUSE-WP-0017 completion (61 published capabilities); only T03 (LLM rerank) remains open under priority 29 |
| 2026-07-07 | **REUSE-WP-0018-T03** closed once `llm-connect` came up locally; priority 29 fully closed (all six T01T06 tasks shipped) |
| 2026-07-08 | **REUSE-WP-0019** finished (all six T01T06 tasks shipped): forge host abstraction + `migrate-host`, hub webhook/staleness tracking, this repo's own Forgejo migration + CI, reuse telemetry store/recording, telemetry aggregation into R-axis evidence, and freshness monitoring in `stats`/CI. Closes SCOPE "not possible yet" item *automatic hub refresh* |

View file

@ -262,6 +262,32 @@ curl -fsS "$REUSE_SURFACE_URL/v1/federated" | jq '.capabilities | length'
Read endpoints are public; writes require `REUSE_SURFACE_TOKEN` (Bearer). API
spec: `specs/FederationHubAPI.md`.
### Automatic recompose and freshness (REUSE-WP-0019-T02/T03/T06)
The hub recomposes automatically rather than waiting for a manual
`reuse-surface federation compose --refresh`:
- **Forgejo org webhook** (primary): a single org-level webhook on `coulomb`
fires on every push, HMAC-signed. `POST /v1/webhooks/forgejo` verifies the
signature, checks whether the push touched `registry/indexes/`, and — only
if so — recomposes. It never parses pushed file content, only paths.
- **Scheduled fallback**: `.forgejo/workflows/recompose-fallback.yaml` in
this repo calls `POST /v1/federated/compose` (token-auth) every 6 hours,
in case a webhook delivery is ever missed.
- **Freshness visibility**: `GET /v1/federated` carries `composed_at`
(timestamp of the last *forced* recompose) and `stale` (set by the
webhook, cleared on the next successful recompose). `reuse-surface stats`
surfaces both plus an `age_days`/`stale_warning` computed against a
threshold (`REUSE_SURFACE_FRESHNESS_DAYS`, default 7 days) — this repo's
own CI runs an informational (non-failing) freshness check on every push.
Setting up the org webhook or rotating its secret is an operator action on
the Forgejo instance and the cluster Secret, not something `reuse-surface`
itself automates — see `railiance-apps/docs/reuse-surface-on-railiance01.md`
(the authoritative operator runbook) and `docs/deploy/reuse-kubernetes.md`
for the current secret custody chain (OpenBao → `ExternalSecret`
`reuse-surface-env`).
### Hub vs local `sources.yaml`
| Workflow | When to use |

View file

@ -4,7 +4,16 @@ Companion to **RAILIANCE-WP-0007** (`railiance-apps` Helm release).
## Image
Repository: `gitea.coulomb.social/coulomb/reuse-surface` (Gitea org `coulomb`, repo `reuse-surface`).
This repo's own canonical remote migrated to Forgejo in REUSE-WP-0019-T03
(`origin``forgejo-remote:coulomb/reuse-surface.git`; the old Gitea copy
is kept read-only, not deleted). Currently deployed production image is
still `gitea.coulomb.social/coulomb/reuse-surface:e3ae22e` (built manually
before the migration, per RAILIANCE-WP-0007) — this doc's manual build
commands still target that registry since that's what's actually live.
The repo also builds `forgejo.coulomb.social/coulomb/reuse-surface:latest`
automatically on every push via `.forgejo/workflows/image.yaml`; switching
the deployed registry over is a deliberate follow-up, not done as part of
T06.
```bash
docker build -t gitea.coulomb.social/coulomb/reuse-surface:<tag> .
@ -103,13 +112,31 @@ cert-manager / companion operator logs.
### Image promotion checklist
1. Tag image from CI commit: `gitea.coulomb.social/coulomb/reuse-surface:<sha>`.
2. Run `pytest -q` and `reuse-surface validate` on that commit.
3. Update Helm values image tag in `railiance-apps`.
4. Deploy to Railiance01; verify `GET /health` and `GET /v1/repos`.
5. Smoke `reuse-surface hub list` and `GET /v1/federated` capability count.
1. Tag image from CI commit. `.forgejo/workflows/image.yaml` already builds
and pushes `forgejo.coulomb.social/coulomb/reuse-surface:main-<short-sha>`
automatically on every push that touches `Dockerfile`/`reuse_surface/**`/
`schemas/**`/`pyproject.toml` — verify it's green rather than building
by hand, unless promoting from the still-live `gitea.coulomb.social`
registry (current production posture).
2. Run `pytest -q` and `reuse-surface validate` on that commit (CI already
does this; re-verify locally if promoting outside CI).
3. Update Helm values image tag in `railiance-apps`
(`helm/reuse-surface-values.yaml`).
4. Deploy to Railiance01 (`make reuse-deploy`); verify `GET /v1/federated`
and `GET /v1/repos` (not `GET /health` — see the known ingress routing
issue below).
5. Smoke `reuse-surface hub list` and `GET /v1/federated` capability count;
check `reuse-surface stats` shows a fresh `composed_at` post-deploy.
6. Record image digest in workplan or progress log.
**Known issue (found 2026-07-07, not fixed):** the public ingress's
exact-path `/health` rule 404s (shadowed by the catch-all `/` rule to the
landing page) — confirmed ingress-layer only via direct port-forward and
the Deployment's own passing readiness/liveness probes. Use
`GET /v1/repos` or `GET /v1/federated` for external verification instead.
Flagged to `railiance-apps`; not this repo's fix to make (shared ingress
template).
### SQLite vs Postgres (cnpg) — decision criteria
Stay on SQLite while:

View file

@ -56,6 +56,10 @@ def hub_list(base_url: str | None = None) -> tuple[int, Any]:
return _request("GET", f"{service_base_url(base_url)}/v1/repos")
def hub_federated(base_url: str | None = None) -> tuple[int, Any]:
return _request("GET", f"{service_base_url(base_url)}/v1/federated")
def hub_show(repo: str, base_url: str | None = None) -> tuple[int, Any]:
return _request("GET", f"{service_base_url(base_url)}/v1/repos/{repo}")

View file

@ -1,14 +1,18 @@
from __future__ import annotations
import json
import os
import urllib.error
import urllib.request
from collections import Counter
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
DEFAULT_FRESHNESS_DAYS = 7
from reuse_surface import hub_client
from reuse_surface.registry import (
LEVEL_ORDERS,
@ -147,11 +151,30 @@ def level_at_least_reliability(current: str, minimum: str) -> bool:
def _hub_configured() -> bool:
import os
return bool(os.environ.get("REUSE_SURFACE_URL"))
def _freshness_days() -> int:
raw = os.environ.get("REUSE_SURFACE_FRESHNESS_DAYS")
if raw:
try:
return int(raw)
except ValueError:
pass
return DEFAULT_FRESHNESS_DAYS
def _composed_at_age_days(composed_at: str | None) -> float | None:
if not composed_at:
return None
try:
composed_dt = datetime.fromisoformat(composed_at.replace("Z", "+00:00"))
except ValueError:
return None
now = datetime.now(timezone.utc)
return (now - composed_dt).total_seconds() / 86400
def _hub_summary(hub_url: str | None) -> dict[str, Any]:
try:
status, payload = hub_client.hub_list(hub_url)
@ -160,12 +183,29 @@ def _hub_summary(hub_url: str | None) -> dict[str, Any]:
if status != 200:
return {"configured": True, "status": status, "error": payload}
repos = payload.get("repos", [])
return {
summary: dict[str, Any] = {
"configured": True,
"registration_count": payload.get("count", len(repos)),
"enabled_count": sum(1 for repo in repos if repo.get("enabled", True)),
}
try:
f_status, f_payload = hub_client.hub_federated(hub_url)
except (ValueError, urllib.error.URLError, OSError):
f_status, f_payload = None, None
if f_status == 200 and isinstance(f_payload, dict):
composed_at = f_payload.get("composed_at")
stale = f_payload.get("stale", False)
age_days = _composed_at_age_days(composed_at)
threshold = _freshness_days()
summary["composed_at"] = composed_at
summary["stale"] = stale
summary["age_days"] = round(age_days, 2) if age_days is not None else None
summary["freshness_threshold_days"] = threshold
summary["stale_warning"] = bool(stale) or (age_days is not None and age_days > threshold)
return summary
def _default_raw_url(repo_root: Path) -> str | None:
return None
@ -250,6 +290,14 @@ def format_stats_markdown(stats: dict[str, Any]) -> str:
)
elif "error" in hub:
lines.append(f"- hub error: {hub['error']}")
if "composed_at" in hub:
age = hub.get("age_days")
age_text = f"{age:.2f}d ago" if age is not None else "unknown"
warning = " ⚠ STALE" if hub.get("stale_warning") else ""
lines.append(
f"- federated index composed: `{hub['composed_at']}` ({age_text}, "
f"threshold {hub.get('freshness_threshold_days')}d){warning}"
)
lines.append("")
return "\n".join(lines) + "\n"

View file

@ -34,3 +34,141 @@ def test_collect_roster_stats_federation_ready():
assert "federation_readiness" in stats
text = format_roster_stats_markdown(stats)
assert "publish pass ratio" in text
# --- T06: hub freshness (composed_at age + stale flag) ---
from datetime import datetime, timedelta, timezone
from reuse_surface.stats import _composed_at_age_days, _freshness_days, _hub_summary
def test_hub_summary_includes_freshness_fields(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_URL", "http://example.invalid")
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
monkeypatch.setattr(
"reuse_surface.hub_client.hub_list",
lambda base_url=None: (200, {"count": 3, "repos": [{"enabled": True}] * 3}),
)
monkeypatch.setattr(
"reuse_surface.hub_client.hub_federated",
lambda base_url=None: (200, {"composed_at": now, "stale": False}),
)
summary = _hub_summary(None)
assert summary["composed_at"] == now
assert summary["stale"] is False
assert summary["age_days"] is not None
assert summary["age_days"] < 1
assert summary["stale_warning"] is False
def test_hub_summary_stale_warning_from_age(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_URL", "http://example.invalid")
old = (datetime.now(timezone.utc) - timedelta(days=10)).strftime("%Y-%m-%dT%H:%M:%SZ")
monkeypatch.setattr(
"reuse_surface.hub_client.hub_list",
lambda base_url=None: (200, {"count": 1, "repos": [{"enabled": True}]}),
)
monkeypatch.setattr(
"reuse_surface.hub_client.hub_federated",
lambda base_url=None: (200, {"composed_at": old, "stale": False}),
)
summary = _hub_summary(None)
assert summary["age_days"] > 7
assert summary["stale_warning"] is True
def test_hub_summary_stale_warning_from_flag(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_URL", "http://example.invalid")
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
monkeypatch.setattr(
"reuse_surface.hub_client.hub_list",
lambda base_url=None: (200, {"count": 1, "repos": [{"enabled": True}]}),
)
monkeypatch.setattr(
"reuse_surface.hub_client.hub_federated",
lambda base_url=None: (200, {"composed_at": now, "stale": True}),
)
summary = _hub_summary(None)
assert summary["stale_warning"] is True
def test_hub_summary_handles_null_composed_at(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_URL", "http://example.invalid")
monkeypatch.setattr(
"reuse_surface.hub_client.hub_list",
lambda base_url=None: (200, {"count": 0, "repos": []}),
)
monkeypatch.setattr(
"reuse_surface.hub_client.hub_federated",
lambda base_url=None: (200, {"composed_at": None, "stale": False}),
)
summary = _hub_summary(None)
assert summary["age_days"] is None
assert summary["stale_warning"] is False
def test_hub_summary_degrades_when_federated_unreachable(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_URL", "http://example.invalid")
monkeypatch.setattr(
"reuse_surface.hub_client.hub_list",
lambda base_url=None: (200, {"count": 1, "repos": [{"enabled": True}]}),
)
def _raise(base_url=None):
import urllib.error
raise urllib.error.URLError("no route")
monkeypatch.setattr("reuse_surface.hub_client.hub_federated", _raise)
summary = _hub_summary(None)
assert summary["configured"] is True
assert "composed_at" not in summary
def test_composed_at_age_days_none_when_missing():
assert _composed_at_age_days(None) is None
def test_composed_at_age_days_none_when_malformed():
assert _composed_at_age_days("not-a-timestamp") is None
def test_freshness_days_default():
assert _freshness_days() == 7
def test_freshness_days_env_override(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_FRESHNESS_DAYS", "3")
assert _freshness_days() == 3
def test_freshness_days_env_invalid_falls_back(monkeypatch):
monkeypatch.setenv("REUSE_SURFACE_FRESHNESS_DAYS", "not-a-number")
assert _freshness_days() == 7
def test_format_stats_markdown_shows_stale_warning():
from reuse_surface.stats import format_stats_markdown
stats = {
"repo_root": "/tmp/x",
"capability_count": 0,
"registry_present": True,
"index_present": True,
"sources_present": True,
"reliability": {"r0_r2": 0, "r3_plus": 0},
"histograms": {},
"vector_drift": [],
"hub": {
"configured": True,
"registration_count": 1,
"enabled_count": 1,
"composed_at": "2020-01-01T00:00:00Z",
"stale": False,
"age_days": 100.0,
"freshness_threshold_days": 7,
"stale_warning": True,
},
}
text = format_stats_markdown(stats)
assert "STALE" in text

View file

@ -203,8 +203,16 @@ reuse-surface stats
reuse-surface stats --format json
reuse-surface stats --federation-ready --raw-url https://.../capabilities.yaml
reuse-surface stats --roster registry/federation/local-repo-roster.yaml --federation-ready
export REUSE_SURFACE_URL=https://reuse.coulomb.social
reuse-surface stats # hub section gains composed_at/stale/age_days/stale_warning
```
Hub freshness (REUSE-WP-0019-T06): when `REUSE_SURFACE_URL` is configured,
`stats`'s `hub` section reports the federated index's `composed_at`, `stale`
flag, `age_days`, and a computed `stale_warning` (age beyond
`REUSE_SURFACE_FRESHNESS_DAYS`, default 7, or the hub's own `stale` flag).
CI runs this as an informational (non-failing) check on every push.
### establish
Bootstrap or discover a capability registry in the current or target repo.

View file

@ -4,11 +4,11 @@ type: workplan
title: "Forgejo-native federation automation and reuse telemetry"
domain: infotech
repo: reuse-surface
status: active
status: finished
owner: claude-code
topic_slug: helix-forge
created: "2026-07-06"
updated: "2026-07-07"
updated: "2026-07-08"
state_hub_workstream_id: "569be717-34f8-4039-bb26-497685f60159"
reuse_check: "new — dogfooded 2026-07-07 via reuse-surface plan-check against the full 61-capability federated index; no existing capability covers Forgejo webhook automation or reuse telemetry"
---
@ -372,18 +372,49 @@ follow-up rather than rushed in.
```task
id: REUSE-WP-0019-T06
status: todo
status: done
priority: low
state_hub_task_id: "a9f44d45-91e2-4b43-909f-30a5f906cf3b"
```
- `reuse-surface stats`: hub `composed_at` age + stale flag; CI informational
check warns when the hub index is older than N days
- `docs/RegistryFederation.md` + `docs/deploy/reuse-kubernetes.md`: webhook
setup, recompose endpoint, Forgejo token handling (route credentials per
credential-routing rules — no secrets in repo)
- `SCOPE.md`: flip "automatic hub refresh" to possible; update federation
posture
SCOPE.md's "automatic hub refresh" flip was already done incidentally in
T02/T03 (moved from "not possible yet" to "possible now"); no further
change needed there.
Implemented:
- `reuse_surface/hub_client.py`: new `hub_federated()` (`GET /v1/federated`)
- `reuse_surface/stats.py`: `_hub_summary()` now also reports
`composed_at`, `stale`, `age_days` (computed from `composed_at`),
`freshness_threshold_days` (`REUSE_SURFACE_FRESHNESS_DAYS` env, default
7), and a computed `stale_warning` (age beyond threshold OR the hub's own
`stale` flag). `format_stats_markdown` surfaces these with a `⚠ STALE`
marker when triggered; `format_stats_json` picks them up automatically
(no format-specific code needed there)
- `.forgejo/workflows/ci.yml`: new informational (non-failing) "Hub
freshness check" step against the live production hub — prints a
`::warning::` annotation when stale, never fails the build
- `docs/RegistryFederation.md`: new "Automatic recompose and freshness"
section tying together the webhook (T02/T03), scheduled fallback (T03),
and freshness visibility (T06) into one coherent explanation, pointing
at the authoritative operator runbook (`railiance-apps`) for actual
secret/webhook setup rather than duplicating operational steps here
- `docs/deploy/reuse-kubernetes.md`: image section updated to reflect the
T03 Forgejo migration (repo canonical remote moved; production image
still built from the pre-migration Gitea registry, deliberately not
switched over in this task); image promotion checklist updated for the
now-automated `.forgejo/workflows/image.yaml` build and the known
`/health` ingress bug (use `/v1/repos`/`/v1/federated` for verification
instead)
- 14 new pytest cases (`test_stats.py`); 173 total pass
- **Live-verified** against the real production hub: `reuse-surface stats`
correctly showed `composed_at`/`age_days` for the actual federated index
(0.08 days old, no stale warning); separately discovered and confirmed
(via a live signed webhook test) that an external secrets-management
change — `reuse-surface-env` is now ExternalSecret-managed from OpenBao
(`railiance-apps` commit `706f6c7`, found while updating these same
docs) — did **not** break the T02/T03 webhook: the synced value still
matches what the hub actually uses
---
@ -394,7 +425,7 @@ state_hub_task_id: "a9f44d45-91e2-4b43-909f-30a5f906cf3b"
- [x] This repo's CI runs on Forgejo Actions (`.forgejo/workflows/`) (T03, 2026-07-07 — `ci.yml`/`ci-smoke.yaml`/`image.yaml` all verified green on the live push)
- [x] Reuse events recordable via hub API and CLI (T04, 2026-07-08 — live-verified); `report reuse` aggregation done in T05
- [x] R-axis evidence rules for observed reuse documented in the maturity standard (T05, 2026-07-08 — `specs/CapabilityMaturityStandard.md` §8.9)
- [x] Hub freshness visible (`composed_at`, stale flag) in API and stats (T02, 2026-07-07)
- [x] Hub freshness visible (`composed_at`, stale flag) in API and stats (T02 API, T06 `stats`/CI, 2026-07-07/08)
## Out of scope