Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002

- SCOPE.md rewritten with repo-specific boundaries and current state - INTENT.md: consumer contract and capability id - AGENTS.md: developer workflow (test/run/validate) - registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1) - workplans: WP-0001 finished, WP-0002 proposed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 13:51:40 +02:00 · 2026-10-19 13:51:40 +02:00 · 2ea8c1337a
commit 2ea8c1337a
parent 5fd7ec7257
8 changed files with 321 additions and 17 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@ -259,3 +259,22 @@ To create a new workplan:
 1. Write the file following the format above
 2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`.
 3. Run `statehub fix-consistency` only when a separate deep audit is needed.
+
+## Developer Workflow
+
+Standard library only (Python 3.10+); there is no build or lint toolchain and
+no dependency install step.
+
+| Step | Command |
+|------|---------|
+| Install (optional, editable) | `pip install -e .` — or use `WEB_FETCHER_PATH`/`PYTHONPATH` from consumers |
+| Test (offline, fixtures) | `python3 -B -m unittest discover -s tests` |
+| Run | `python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl` |
+| Byte-compile check | `python3 -m compileall -q web_fetcher` |
+| Validate capability registry | `uv run --project ~/reuse-surface reuse-surface validate --root .` |
+
+Rules: every new adapter ships with a recorded fixture in `tests/fixtures/` and
+an offline test; tests must not hit the network. Do not add third-party
+dependencies. When behaviour, adapters or maturity change, update
+`registry/capabilities/capability.infotech.provenance-fetch.md` and its index
+row (keep `summary`/`tags` identical, or `validate` reports drift).
--- a/INTENT.md
+++ b/INTENT.md
@ -17,6 +17,11 @@ to pull data from the web and from public APIs.
 5. **Minimal footprint.** Standard library only, so it runs inside executors
   (for example rein-aharness on railiance01) without dependency management.

+**Contract.** Consumers depend on the `Observation` fields and the spec
+format documented in `README.md`; adapters may be added freely, but changing
+those fields is a breaking change. Discoverable through reuse-surface as
+`capability.infotech.provenance-fetch`.
+
 Out of scope: crawling, scraping behind logins, LLM-based extraction,
 scheduling (activity-core owns *when*), and execution orchestration
 (rein-aharness owns *how it runs*).
--- a/SCOPE.md
+++ b/SCOPE.md
@ -1,32 +1,49 @@
 # SCOPE

-> This file was generated by `statehub register`. Refine it as the repository
-> boundaries become clearer.
-
 ## One-liner

 Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.

 ## Core Idea

-web-fetcher exists to provide the capability described in INTENT.md.
+Research repos declare *what* they need as JSON fetch specs; web-fetcher returns
+`Observation` records that carry the value together with source, URL, the
+source's own publication time, retrieval time and the SHA-256 of the raw
+payload. Missing data is an explicit `not_found`, failures an explicit `error` —
+never a guess. See `INTENT.md` for the commitments.

 ## In Scope

- Maintain the repository's primary implementation.
- Keep docs, tests, and operational metadata current.
+- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`).
+- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`).
+- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`,
+  and the `manual` not_found placeholder. New public sources as adapters with
+  offline fixture tests.
+- Optional content-addressed payload cache for replayable runs.
+- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API.
+- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`).

 ## Out of Scope

- Own unrelated adjacent systems.
- Make irreversible operational decisions without human approval.
+- Crawling, scraping behind logins, JavaScript-rendered pages
+  (use `capability.infotech.headless-browser`).
+- LLM-based extraction or estimation of missing values.
+- Scheduling — activity-core owns *when*.
+- Execution orchestration — rein-aharness owns *how it runs*.
+- Storing or analysing fetched data; that belongs to the consuming research repo.
+- Third-party runtime dependencies.

 ## Current State

- Status: active; implementation and stability should be verified by the repo agent.
+- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
+- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
+- Known gaps: no retry/backoff, auth or pagination; no tagged release yet
+  (tracked in `WEBFETCH-WP-0002`).
+- Reuse vector: D2 / A2 / C1 / R1.

 ## Getting Oriented

- Start with: INTENT.md
- Agent instructions: AGENTS.md
- Workplans: workplans/
+- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
+- Agent instructions and dev commands: `AGENTS.md`
+- Capability record: `registry/capabilities/`
+- Workplans: `workplans/`
--- a/registry/README.md
+++ b/registry/README.md
@ -0,0 +1,12 @@
+# Capability Registry
+
+Markdown-first capability index for federation and reuse planning.
+
+## Authoring
+
+1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
+2. Add the row to `indexes/capabilities.yaml`.
+3. Run `reuse-surface validate` from a checkout with the CLI installed.
+4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
+
+Federation contract: reuse-surface `docs/RegistryFederation.md`.
--- a/registry/capabilities/capability.infotech.provenance-fetch.md
+++ b/registry/capabilities/capability.infotech.provenance-fetch.md
@ -0,0 +1,137 @@
+---
+id: capability.infotech.provenance-fetch
+name: Provenance-first Web and API Data Fetching
+summary: >-
+  Standard-library Python fetcher that turns declarative JSON specs into
+  Observation records carrying value, source, URL, source publication time,
+  retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic
+  JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found,
+  never estimated. Optional content-addressed cache replays a run from the
+  exact bytes it used.
+owner: web-fetcher
+status: draft
+domain: infotech
+tags: [fetch, http, api, provenance, research, statistics, eurostat, csv, json, xlsx, cache, replay, data-ingestion, stdlib]
+maturity:
+  discovery:
+    current: D2
+    target: D3
+    confidence: medium
+    rationale: >
+      Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md)
+      and one consumer (assembly-economics reality checkpoints) drove the
+      design. No survey of alternative fetching libraries in the estate yet.
+  availability:
+    current: A2
+    target: A3
+    confidence: medium
+    rationale: >
+      Installable source package (pyproject, CLI entry point, importable API)
+      on Forgejo with offline fixture tests. Not published to a package index
+      or as a versioned release artifact; consumers vendor via path or pip
+      install from the repository.
+external_evidence:
+  completeness:
+    level: C1
+    confidence: medium
+    basis: scope_vs_intent_and_consumer_expectations
+    satisfied_expectations:
+      - Every value carries source, URL, retrieval time and payload hash.
+      - Failures and missing values are explicit not_found/error records, never exceptions per spec.
+      - Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources.
+    broken_expectations:
+      - No pagination, authentication or rate-limit handling for APIs that need it.
+      - Only four adapters; new statistics APIs require code.
+    out_of_scope_expectations:
+      - Crawling, scraping behind logins, LLM-based extraction.
+      - Scheduling (activity-core) and execution orchestration (rein-aharness).
+  reliability:
+    level: R1
+    confidence: low
+    basis: consumer_quality_signals
+    known_reliability_risks:
+      - Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time.
+      - Live endpoints are not exercised by the test suite.
+      - Single consumer so far.
+    evidence:
+      satisfied_signals:
+        - 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures.
+        - assembly-economics run_checkpoint.py consumes it for light checkpoint runs.
+discovery:
+  intent: >
+    Give research repositories one small, trustworthy way to pull numbers from
+    public statistics APIs and web data files with full provenance, instead of
+    each repo writing ad-hoc urllib or pandas download code.
+  includes:
+    - Declarative JSON fetch specs and a CLI emitting observations as JSONL.
+    - Python API (run_spec, run_specs, Observation).
+    - Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder.
+    - Optional content-addressed payload cache for replayable runs.
+  excludes:
+    - Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping.
+    - LLM extraction or estimation of missing values.
+    - Scheduling and execution orchestration.
+  assumptions:
+    - Consumers run Python 3.10+; no third-party dependencies are needed.
+    - Sources are public and reachable over plain HTTPS.
+  use_cases:
+    - Reality checkpoints comparing forecasts against official statistics.
+    - Reproducible research runs replayed from cached source bytes.
+availability:
+  current_level: A2
+  target_level: A3
+  current_artifacts:
+    - https://forgejo.coulomb.social/coulomb/web-fetcher
+    - web_fetcher/__main__.py
+    - web_fetcher/registry.py
+  target_artifacts:
+    - Tagged, versioned release installable with pip from Forgejo.
+  consumption_modes: [Python library, cli, source]
+relations:
+  depends_on: []
+  supports: []
+  related_to: [capability.infotech.headless-browser]
+evidence:
+  documentation:
+    - README.md
+    - INTENT.md
+    - SCOPE.md
+  tests:
+    - tests/test_web_fetcher.py
+  consumer_feedback: []
+consumer_guidance:
+  recommended_for:
+    - Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos.
+    - Runs inside constrained executors where installing dependencies is undesirable.
+  not_recommended_for:
+    - JavaScript-rendered pages or sites requiring login.
+    - High-volume crawling or bulk mirroring.
+  known_limitations:
+    - No auth, pagination or retry/backoff policy beyond a single request.
+promotion_history: []
+---
+
+# Provenance-first Web and API Data Fetching
+
+`web-fetcher` runs declarative fetch specs and returns `Observation` records
+with the value **and** its provenance. Values are never estimated: each spec
+yields a sourced value or an explicit `not_found` / `error` record.
+
+## Try it
+
+```bash
+git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher
+python3 -B -m unittest discover -s tests          # offline fixtures
+python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl
+```
+
+```python
+from web_fetcher import run_spec
+obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
+                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
+                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})
+```
+
+New sources are added as adapters in `web_fetcher/adapters/` with offline
+fixture tests. Consumer reference: assembly-economics
+`checkpoints/run_checkpoint.py`.
--- a/registry/indexes/capabilities.yaml
+++ b/registry/indexes/capabilities.yaml
@ -0,0 +1,35 @@
+version: 1
+updated: '2026-10-19'
+domain: infotech
+capabilities:
+- id: capability.infotech.provenance-fetch
+  name: Provenance-first Web and API Data Fetching
+  summary: Standard-library Python fetcher that turns declarative JSON specs into Observation
+    records carrying value, source, URL, source publication time, retrieval time and payload
+    SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing
+    data is an explicit not_found, never estimated. Optional content-addressed cache replays
+    a run from the exact bytes it used.
+  domain: infotech
+  status: draft
+  owner: web-fetcher
+  tags:
+  - fetch
+  - http
+  - api
+  - provenance
+  - research
+  - statistics
+  - eurostat
+  - csv
+  - json
+  - xlsx
+  - cache
+  - replay
+  - data-ingestion
+  - stdlib
+  vector: D2 / A2 / C1 / R1
+  path: registry/capabilities/capability.infotech.provenance-fetch.md
+  consumption_modes:
+  - Python library
+  - cli
+  - source
--- a/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md
+++ b/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md
@ -4,13 +4,13 @@ type: workplan
 title: "Bootstrap State Hub integration"
 domain: infotech
 repo: web-fetcher
-status: ready
+status: finished
 flavor: planning
 depends_on: []
 owner: codex
 topic_slug: stack
 created: "2026-10-13"
-updated: "2026-10-13"
+updated: "2026-10-19"
 state_hub_workstream_id: "29736794-d8ef-5d40-bf8c-8282f89e9e9c"
 ---

@ -22,7 +22,7 @@ Provenance-first, standard-library fetching of data from public APIs and web sou

 ```task
 id: WEBFETCH-WP-0001-T01
-status: todo
+status: done
 priority: high
 state_hub_task_id: "3f92d202-2519-52f5-bfd4-1dccf5d2ea7d"
 ```
@ -34,7 +34,7 @@ Replace generated placeholders with repo-specific facts where needed.

 ```task
 id: WEBFETCH-WP-0001-T02
-status: todo
+status: done
 priority: high
 state_hub_task_id: "91ff2856-e756-5716-9837-b7d25e3cad22"
 ```
@ -47,7 +47,7 @@ changes confidently.

 ```task
 id: WEBFETCH-WP-0001-T03
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "6af171f2-f9b4-5530-912a-46dd81da6b2c"
 ```
@ -59,3 +59,14 @@ checkout:
 ```bash
 statehub fix-consistency
 ```
+
+## Outcome (2026-10-19)
+
+- T01: `SCOPE.md` rewritten with repo-specific boundaries; `INTENT.md` gained the
+  consumer contract; `AGENTS.md` and `.custodian-brief.md` reviewed.
+- T02: Developer workflow table added to `AGENTS.md` (tests: 8 passing offline).
+- T03: Seeded `WEBFETCH-WP-0002` (versioned release and fetch hardening).
+- Also: registered `capability.infotech.provenance-fetch` with reuse-surface
+  (`registry/`) and added web-fetcher to reuse-surface federation sources.
+
+Residuals: carried in `WEBFETCH-WP-0002`.
--- a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md
+++ b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md
@ -0,0 +1,68 @@
+---
+id: WEBFETCH-WP-0002
+type: workplan
+title: "Versioned release and fetch hardening"
+domain: infotech
+repo: web-fetcher
+status: proposed
+flavor: implementation
+depends_on: [WEBFETCH-WP-0001]
+owner: codex
+topic_slug: stack
+created: "2026-10-19"
+updated: "2026-10-19"
+---
+
+# Versioned release and fetch hardening
+
+Move `capability.infotech.provenance-fetch` from A2 (source) to A3 (versioned,
+installable release) and close the reliability gaps listed in the capability
+record, without adding third-party dependencies.
+
+## Retry and Backoff in fetch_bytes
+
+```task
+id: WEBFETCH-WP-0002-T01
+status: todo
+priority: high
+```
+
+Add bounded retry with exponential backoff for transient failures (connection
+errors, HTTP 429/5xx), honouring `Retry-After`. Retries must not change
+provenance semantics: `retrieved_at` and `payload_sha256` describe the attempt
+that succeeded. Offline tests with a fake getter.
+
+## Opt-in Live Smoke Test
+
+```task
+id: WEBFETCH-WP-0002-T02
+status: todo
+priority: medium
+```
+
+Add a live smoke test (skipped unless `WEB_FETCHER_LIVE=1`) that runs one spec
+per adapter against the real endpoints, to detect upstream shape changes that
+fixtures cannot.
+
+## Tagged Release Installable from Forgejo
+
+```task
+id: WEBFETCH-WP-0002-T03
+status: todo
+priority: high
+```
+
+Add a CHANGELOG, tag `v0.1.0` (or `v0.2.0` with T01), and document
+`pip install git+https://forgejo.coulomb.social/coulomb/web-fetcher@<tag>`.
+Point assembly-economics at the tag instead of `WEB_FETCHER_PATH`.
+
+## Update Capability Record
+
+```task
+id: WEBFETCH-WP-0002-T04
+status: todo
+priority: medium
+```
+
+Promote availability A2 → A3 with evidence citations and a `promotion_history`
+entry; update `SCOPE.md` current state.