From 2ea8c1337a8a1486c2ca8e57b331efb57384c9e9 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 19 Oct 2026 13:51:40 +0200 Subject: [PATCH] Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 - SCOPE.md rewritten with repo-specific boundaries and current state - INTENT.md: consumer contract and capability id - AGENTS.md: developer workflow (test/run/validate) - registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1) - workplans: WP-0001 finished, WP-0002 proposed Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24 --- AGENTS.md | 19 +++ INTENT.md | 5 + SCOPE.md | 41 ++++-- registry/README.md | 12 ++ .../capability.infotech.provenance-fetch.md | 137 ++++++++++++++++++ registry/indexes/capabilities.yaml | 35 +++++ .../WEBFETCH-WP-0001-statehub-bootstrap.md | 21 ++- ...TCH-WP-0002-release-and-fetch-hardening.md | 68 +++++++++ 8 files changed, 321 insertions(+), 17 deletions(-) create mode 100644 registry/README.md create mode 100644 registry/capabilities/capability.infotech.provenance-fetch.md create mode 100644 registry/indexes/capabilities.yaml create mode 100644 workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md diff --git a/AGENTS.md b/AGENTS.md index a14b9a2..72a7d88 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -259,3 +259,22 @@ To create a new workplan: 1. Write the file following the format above 2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`. 3. Run `statehub fix-consistency` only when a separate deep audit is needed. + +## Developer Workflow + +Standard library only (Python 3.10+); there is no build or lint toolchain and +no dependency install step. + +| Step | Command | +|------|---------| +| Install (optional, editable) | `pip install -e .` — or use `WEB_FETCHER_PATH`/`PYTHONPATH` from consumers | +| Test (offline, fixtures) | `python3 -B -m unittest discover -s tests` | +| Run | `python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl` | +| Byte-compile check | `python3 -m compileall -q web_fetcher` | +| Validate capability registry | `uv run --project ~/reuse-surface reuse-surface validate --root .` | + +Rules: every new adapter ships with a recorded fixture in `tests/fixtures/` and +an offline test; tests must not hit the network. Do not add third-party +dependencies. When behaviour, adapters or maturity change, update +`registry/capabilities/capability.infotech.provenance-fetch.md` and its index +row (keep `summary`/`tags` identical, or `validate` reports drift). diff --git a/INTENT.md b/INTENT.md index 903d9e8..944e3da 100644 --- a/INTENT.md +++ b/INTENT.md @@ -17,6 +17,11 @@ to pull data from the web and from public APIs. 5. **Minimal footprint.** Standard library only, so it runs inside executors (for example rein-aharness on railiance01) without dependency management. +**Contract.** Consumers depend on the `Observation` fields and the spec +format documented in `README.md`; adapters may be added freely, but changing +those fields is a breaking change. Discoverable through reuse-surface as +`capability.infotech.provenance-fetch`. + Out of scope: crawling, scraping behind logins, LLM-based extraction, scheduling (activity-core owns *when*), and execution orchestration (rein-aharness owns *how it runs*). diff --git a/SCOPE.md b/SCOPE.md index 071fa12..01b99c9 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -1,32 +1,49 @@ # SCOPE -> This file was generated by `statehub register`. Refine it as the repository -> boundaries become clearer. - ## One-liner Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories. ## Core Idea -web-fetcher exists to provide the capability described in INTENT.md. +Research repos declare *what* they need as JSON fetch specs; web-fetcher returns +`Observation` records that carry the value together with source, URL, the +source's own publication time, retrieval time and the SHA-256 of the raw +payload. Missing data is an explicit `not_found`, failures an explicit `error` — +never a guess. See `INTENT.md` for the commitments. ## In Scope -- Maintain the repository's primary implementation. -- Keep docs, tests, and operational metadata current. +- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`). +- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`). +- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`, + and the `manual` not_found placeholder. New public sources as adapters with + offline fixture tests. +- Optional content-addressed payload cache for replayable runs. +- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API. +- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`). ## Out of Scope -- Own unrelated adjacent systems. -- Make irreversible operational decisions without human approval. +- Crawling, scraping behind logins, JavaScript-rendered pages + (use `capability.infotech.headless-browser`). +- LLM-based extraction or estimation of missing values. +- Scheduling — activity-core owns *when*. +- Execution orchestration — rein-aharness owns *how it runs*. +- Storing or analysing fetched data; that belongs to the consuming research repo. +- Third-party runtime dependencies. ## Current State -- Status: active; implementation and stability should be verified by the repo agent. +- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19). +- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`). +- Known gaps: no retry/backoff, auth or pagination; no tagged release yet + (tracked in `WEBFETCH-WP-0002`). +- Reuse vector: D2 / A2 / C1 / R1. ## Getting Oriented -- Start with: INTENT.md -- Agent instructions: AGENTS.md -- Workplans: workplans/ +- Start with: `INTENT.md`, then `README.md` for spec fields and usage. +- Agent instructions and dev commands: `AGENTS.md` +- Capability record: `registry/capabilities/` +- Workplans: `workplans/` diff --git a/registry/README.md b/registry/README.md new file mode 100644 index 0000000..569abe9 --- /dev/null +++ b/registry/README.md @@ -0,0 +1,12 @@ +# Capability Registry + +Markdown-first capability index for federation and reuse planning. + +## Authoring + +1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`). +2. Add the row to `indexes/capabilities.yaml`. +3. Run `reuse-surface validate` from a checkout with the CLI installed. +4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`. + +Federation contract: reuse-surface `docs/RegistryFederation.md`. diff --git a/registry/capabilities/capability.infotech.provenance-fetch.md b/registry/capabilities/capability.infotech.provenance-fetch.md new file mode 100644 index 0000000..3fbcf39 --- /dev/null +++ b/registry/capabilities/capability.infotech.provenance-fetch.md @@ -0,0 +1,137 @@ +--- +id: capability.infotech.provenance-fetch +name: Provenance-first Web and API Data Fetching +summary: >- + Standard-library Python fetcher that turns declarative JSON specs into + Observation records carrying value, source, URL, source publication time, + retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic + JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found, + never estimated. Optional content-addressed cache replays a run from the + exact bytes it used. +owner: web-fetcher +status: draft +domain: infotech +tags: [fetch, http, api, provenance, research, statistics, eurostat, csv, json, xlsx, cache, replay, data-ingestion, stdlib] +maturity: + discovery: + current: D2 + target: D3 + confidence: medium + rationale: > + Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md) + and one consumer (assembly-economics reality checkpoints) drove the + design. No survey of alternative fetching libraries in the estate yet. + availability: + current: A2 + target: A3 + confidence: medium + rationale: > + Installable source package (pyproject, CLI entry point, importable API) + on Forgejo with offline fixture tests. Not published to a package index + or as a versioned release artifact; consumers vendor via path or pip + install from the repository. +external_evidence: + completeness: + level: C1 + confidence: medium + basis: scope_vs_intent_and_consumer_expectations + satisfied_expectations: + - Every value carries source, URL, retrieval time and payload hash. + - Failures and missing values are explicit not_found/error records, never exceptions per spec. + - Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources. + broken_expectations: + - No pagination, authentication or rate-limit handling for APIs that need it. + - Only four adapters; new statistics APIs require code. + out_of_scope_expectations: + - Crawling, scraping behind logins, LLM-based extraction. + - Scheduling (activity-core) and execution orchestration (rein-aharness). + reliability: + level: R1 + confidence: low + basis: consumer_quality_signals + known_reliability_risks: + - Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time. + - Live endpoints are not exercised by the test suite. + - Single consumer so far. + evidence: + satisfied_signals: + - 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures. + - assembly-economics run_checkpoint.py consumes it for light checkpoint runs. +discovery: + intent: > + Give research repositories one small, trustworthy way to pull numbers from + public statistics APIs and web data files with full provenance, instead of + each repo writing ad-hoc urllib or pandas download code. + includes: + - Declarative JSON fetch specs and a CLI emitting observations as JSONL. + - Python API (run_spec, run_specs, Observation). + - Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder. + - Optional content-addressed payload cache for replayable runs. + excludes: + - Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping. + - LLM extraction or estimation of missing values. + - Scheduling and execution orchestration. + assumptions: + - Consumers run Python 3.10+; no third-party dependencies are needed. + - Sources are public and reachable over plain HTTPS. + use_cases: + - Reality checkpoints comparing forecasts against official statistics. + - Reproducible research runs replayed from cached source bytes. +availability: + current_level: A2 + target_level: A3 + current_artifacts: + - https://forgejo.coulomb.social/coulomb/web-fetcher + - web_fetcher/__main__.py + - web_fetcher/registry.py + target_artifacts: + - Tagged, versioned release installable with pip from Forgejo. + consumption_modes: [Python library, cli, source] +relations: + depends_on: [] + supports: [] + related_to: [capability.infotech.headless-browser] +evidence: + documentation: + - README.md + - INTENT.md + - SCOPE.md + tests: + - tests/test_web_fetcher.py + consumer_feedback: [] +consumer_guidance: + recommended_for: + - Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos. + - Runs inside constrained executors where installing dependencies is undesirable. + not_recommended_for: + - JavaScript-rendered pages or sites requiring login. + - High-volume crawling or bulk mirroring. + known_limitations: + - No auth, pagination or retry/backoff policy beyond a single request. +promotion_history: [] +--- + +# Provenance-first Web and API Data Fetching + +`web-fetcher` runs declarative fetch specs and returns `Observation` records +with the value **and** its provenance. Values are never estimated: each spec +yields a sourced value or an explicit `not_found` / `error` record. + +## Try it + +```bash +git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher +python3 -B -m unittest discover -s tests # offline fixtures +python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl +``` + +```python +from web_fetcher import run_spec +obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai", + "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10", + "indic_is": "E_AI_TANY"}, "split_by": ["geo"]}) +``` + +New sources are added as adapters in `web_fetcher/adapters/` with offline +fixture tests. Consumer reference: assembly-economics +`checkpoints/run_checkpoint.py`. diff --git a/registry/indexes/capabilities.yaml b/registry/indexes/capabilities.yaml new file mode 100644 index 0000000..091a86b --- /dev/null +++ b/registry/indexes/capabilities.yaml @@ -0,0 +1,35 @@ +version: 1 +updated: '2026-10-19' +domain: infotech +capabilities: +- id: capability.infotech.provenance-fetch + name: Provenance-first Web and API Data Fetching + summary: Standard-library Python fetcher that turns declarative JSON specs into Observation + records carrying value, source, URL, source publication time, retrieval time and payload + SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing + data is an explicit not_found, never estimated. Optional content-addressed cache replays + a run from the exact bytes it used. + domain: infotech + status: draft + owner: web-fetcher + tags: + - fetch + - http + - api + - provenance + - research + - statistics + - eurostat + - csv + - json + - xlsx + - cache + - replay + - data-ingestion + - stdlib + vector: D2 / A2 / C1 / R1 + path: registry/capabilities/capability.infotech.provenance-fetch.md + consumption_modes: + - Python library + - cli + - source diff --git a/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md b/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md index 420b032..2916eaf 100644 --- a/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md +++ b/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md @@ -4,13 +4,13 @@ type: workplan title: "Bootstrap State Hub integration" domain: infotech repo: web-fetcher -status: ready +status: finished flavor: planning depends_on: [] owner: codex topic_slug: stack created: "2026-10-13" -updated: "2026-10-13" +updated: "2026-10-19" state_hub_workstream_id: "29736794-d8ef-5d40-bf8c-8282f89e9e9c" --- @@ -22,7 +22,7 @@ Provenance-first, standard-library fetching of data from public APIs and web sou ```task id: WEBFETCH-WP-0001-T01 -status: todo +status: done priority: high state_hub_task_id: "3f92d202-2519-52f5-bfd4-1dccf5d2ea7d" ``` @@ -34,7 +34,7 @@ Replace generated placeholders with repo-specific facts where needed. ```task id: WEBFETCH-WP-0001-T02 -status: todo +status: done priority: high state_hub_task_id: "91ff2856-e756-5716-9837-b7d25e3cad22" ``` @@ -47,7 +47,7 @@ changes confidently. ```task id: WEBFETCH-WP-0001-T03 -status: todo +status: done priority: medium state_hub_task_id: "6af171f2-f9b4-5530-912a-46dd81da6b2c" ``` @@ -59,3 +59,14 @@ checkout: ```bash statehub fix-consistency ``` + +## Outcome (2026-10-19) + +- T01: `SCOPE.md` rewritten with repo-specific boundaries; `INTENT.md` gained the + consumer contract; `AGENTS.md` and `.custodian-brief.md` reviewed. +- T02: Developer workflow table added to `AGENTS.md` (tests: 8 passing offline). +- T03: Seeded `WEBFETCH-WP-0002` (versioned release and fetch hardening). +- Also: registered `capability.infotech.provenance-fetch` with reuse-surface + (`registry/`) and added web-fetcher to reuse-surface federation sources. + +Residuals: carried in `WEBFETCH-WP-0002`. diff --git a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md new file mode 100644 index 0000000..836fe35 --- /dev/null +++ b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md @@ -0,0 +1,68 @@ +--- +id: WEBFETCH-WP-0002 +type: workplan +title: "Versioned release and fetch hardening" +domain: infotech +repo: web-fetcher +status: proposed +flavor: implementation +depends_on: [WEBFETCH-WP-0001] +owner: codex +topic_slug: stack +created: "2026-10-19" +updated: "2026-10-19" +--- + +# Versioned release and fetch hardening + +Move `capability.infotech.provenance-fetch` from A2 (source) to A3 (versioned, +installable release) and close the reliability gaps listed in the capability +record, without adding third-party dependencies. + +## Retry and Backoff in fetch_bytes + +```task +id: WEBFETCH-WP-0002-T01 +status: todo +priority: high +``` + +Add bounded retry with exponential backoff for transient failures (connection +errors, HTTP 429/5xx), honouring `Retry-After`. Retries must not change +provenance semantics: `retrieved_at` and `payload_sha256` describe the attempt +that succeeded. Offline tests with a fake getter. + +## Opt-in Live Smoke Test + +```task +id: WEBFETCH-WP-0002-T02 +status: todo +priority: medium +``` + +Add a live smoke test (skipped unless `WEB_FETCHER_LIVE=1`) that runs one spec +per adapter against the real endpoints, to detect upstream shape changes that +fixtures cannot. + +## Tagged Release Installable from Forgejo + +```task +id: WEBFETCH-WP-0002-T03 +status: todo +priority: high +``` + +Add a CHANGELOG, tag `v0.1.0` (or `v0.2.0` with T01), and document +`pip install git+https://forgejo.coulomb.social/coulomb/web-fetcher@`. +Point assembly-economics at the tag instead of `WEB_FETCHER_PATH`. + +## Update Capability Record + +```task +id: WEBFETCH-WP-0002-T04 +status: todo +priority: medium +``` + +Promote availability A2 → A3 with evidence citations and a `promotion_history` +entry; update `SCOPE.md` current state.