From 7d6d5a04c884822e9f3880e63fa8e511b9506a93 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 19 Oct 2026 13:58:27 +0200 Subject: [PATCH] Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002 SCOPE.md now states only what exists (write-only cache, retry on any URLError, optional source, string-compared latest period). INTENT.md sharpens commitments 1 and 2. README and the capability record drop the replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05 (enforce source), T06 (replay from cache), T07 (latest-period rule). Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24 --- INTENT.md | 13 ++-- README.md | 4 +- SCOPE.md | 63 ++++++++++++------- .../capability.infotech.provenance-fetch.md | 23 ++++--- registry/indexes/capabilities.yaml | 4 +- ...TCH-WP-0002-release-and-fetch-hardening.md | 56 +++++++++++++++-- 6 files changed, 121 insertions(+), 42 deletions(-) diff --git a/INTENT.md b/INTENT.md index 944e3da..8782fc6 100644 --- a/INTENT.md +++ b/INTENT.md @@ -6,9 +6,12 @@ to pull data from the web and from public APIs. **Core commitments** 1. **Provenance first.** No value without source, URL, retrieval time and - payload hash. -2. **Never invent.** Missing data is an explicit `not_found`, never a guess. - Language models are not a data source here. + payload hash — plus the source's own publication time wherever the source + states one. +2. **Never invent.** Missing data is an explicit `not_found`, never a guess, + and when a source offers several candidates (for example the "latest" + period) the choice follows a stated rule visible in the record. Language + models are not a data source here. 3. **Reusable adapters.** Statistics APIs, CSV/JSON endpoints and spreadsheets behind one small interface. New sources are added as adapters with offline fixture tests. @@ -19,8 +22,8 @@ to pull data from the web and from public APIs. **Contract.** Consumers depend on the `Observation` fields and the spec format documented in `README.md`; adapters may be added freely, but changing -those fields is a breaking change. Discoverable through reuse-surface as -`capability.infotech.provenance-fetch`. +those fields is a breaking change. The capability is discoverable through +reuse-surface so research repos reuse it instead of writing their own fetchers. Out of scope: crawling, scraping behind logins, LLM-based extraction, scheduling (activity-core owns *when*), and execution orchestration diff --git a/README.md b/README.md index 7bf6aba..c86fc0a 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,8 @@ Every fetch returns `Observation` records with the value **and** where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit `not_found` or `error` record. An -optional content-addressed cache lets a research run be replayed from the -exact bytes it used. +optional content-addressed cache stores the exact bytes each run used (replay +from the cache is not implemented yet; see `WEBFETCH-WP-0002`). Standard library only. Python 3.10+. diff --git a/SCOPE.md b/SCOPE.md index 01b99c9..a2b168d 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -1,27 +1,47 @@ # SCOPE +> What the repository **is** today. What it aspires to be is in `INTENT.md`; +> the gap between the two is tracked in `workplans/`. + ## One-liner -Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories. +Standard-library fetching of data from public APIs and web data files into +provenance-carrying `Observation` records, for research repositories. ## Core Idea -Research repos declare *what* they need as JSON fetch specs; web-fetcher returns -`Observation` records that carry the value together with source, URL, the -source's own publication time, retrieval time and the SHA-256 of the raw -payload. Missing data is an explicit `not_found`, failures an explicit `error` — -never a guess. See `INTENT.md` for the commitments. +Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches +each URL and returns `Observation` records with the value plus URL, retrieval +time and the SHA-256 of the raw payload. Missing data becomes `not_found`, +failures become `error`; a failing spec never aborts the run. -## In Scope +## What Exists -- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`). -- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`). -- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`, - and the `manual` not_found placeholder. New public sources as adapters with - offline fixture tests. -- Optional content-addressed payload cache for replayable runs. -- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API. -- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`). +- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`): + sets `retrieved_at` and `payload_sha256` on every fetch; retries any + `URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this + includes permanent HTTP 4xx such as 404; no `Retry-After` handling. +- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures + and parse errors become `error` observations. +- Adapters: + - `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`. + - `http_json` — dotted path lookup. + - `http_csv` — row filter; "latest" row chosen by **string** comparison of the + period column, or the last row if none is given; non-numeric values are + returned as strings with `status: ok`. + - `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`). + - `manual` — always `not_found` with a note. +- `source` is copied from the optional spec field; the HTTP adapters can emit + `status: ok` with `source: null`. `published` is unset outside `eurostat`. +- Cache (`--cache DIR`): **write-only**. Stores `.bin` and appends to + `index.jsonl`; nothing reads it back, so runs cannot yet be replayed. +- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]` + emitting JSONL; exit code 1 if any observation is `error`. Python API + `run_spec`, `run_specs`. +- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake + getters; no live-endpoint tests. +- Reuse-surface capability record `capability.infotech.provenance-fetch` + (`registry/`), vector D2 / A2 / C1 / R1. ## Out of Scope @@ -31,15 +51,16 @@ never a guess. See `INTENT.md` for the commitments. - Scheduling — activity-core owns *when*. - Execution orchestration — rein-aharness owns *how it runs*. - Storing or analysing fetched data; that belongs to the consuming research repo. -- Third-party runtime dependencies. +- Authentication, pagination, third-party runtime dependencies. ## Current State -- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19). -- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`). -- Known gaps: no retry/backoff, auth or pagination; no tagged release yet - (tracked in `WEBFETCH-WP-0002`). -- Reuse vector: D2 / A2 / C1 / R1. +- Version 0.1.0, ~400 lines of Python, untagged; consumed from source. +- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`, + via `WEB_FETCHER_PATH`). +- Running inside rein-aharness is plausible (stdlib only) but unverified. +- Gaps against `INTENT.md` (source enforcement, replay, latest-period + selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`. ## Getting Oriented diff --git a/registry/capabilities/capability.infotech.provenance-fetch.md b/registry/capabilities/capability.infotech.provenance-fetch.md index 3fbcf39..4f0f248 100644 --- a/registry/capabilities/capability.infotech.provenance-fetch.md +++ b/registry/capabilities/capability.infotech.provenance-fetch.md @@ -6,8 +6,8 @@ summary: >- Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found, - never estimated. Optional content-addressed cache replays a run from the - exact bytes it used. + never estimated. Optional content-addressed cache stores the exact bytes a + run used; replay from it is not implemented yet. owner: web-fetcher status: draft domain: infotech @@ -36,10 +36,14 @@ external_evidence: confidence: medium basis: scope_vs_intent_and_consumer_expectations satisfied_expectations: - - Every value carries source, URL, retrieval time and payload hash. + - Every value carries URL, retrieval time and payload hash. - Failures and missing values are explicit not_found/error records, never exceptions per spec. - Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources. broken_expectations: + - Runs cannot be replayed; the cache is write-only. + - source is optional in specs, so HTTP adapters can return ok values with source null. + - Source publication time is only captured by the Eurostat adapter. + - Latest-period selection compares period strings and can pick the wrong row for mixed formats. - No pagination, authentication or rate-limit handling for APIs that need it. - Only four adapters; new statistics APIs require code. out_of_scope_expectations: @@ -50,6 +54,7 @@ external_evidence: confidence: low basis: consumer_quality_signals known_reliability_risks: + - Retries cover every URLError, including permanent 4xx; 429/Retry-After is not honoured. - Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time. - Live endpoints are not exercised by the test suite. - Single consumer so far. @@ -66,7 +71,7 @@ discovery: - Declarative JSON fetch specs and a CLI emitting observations as JSONL. - Python API (run_spec, run_specs, Observation). - Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder. - - Optional content-addressed payload cache for replayable runs. + - Optional content-addressed payload cache (write-only; replay planned in WEBFETCH-WP-0002). excludes: - Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping. - LLM extraction or estimation of missing values. @@ -76,7 +81,7 @@ discovery: - Sources are public and reachable over plain HTTPS. use_cases: - Reality checkpoints comparing forecasts against official statistics. - - Reproducible research runs replayed from cached source bytes. + - Retaining the exact source bytes behind a research run for later audit. availability: current_level: A2 target_level: A3 @@ -107,7 +112,9 @@ consumer_guidance: - JavaScript-rendered pages or sites requiring login. - High-volume crawling or bulk mirroring. known_limitations: - - No auth, pagination or retry/backoff policy beyond a single request. + - No auth or pagination; retries are two fixed linear attempts on any network/HTTP error. + - Cache is write-only; replay is planned in WEBFETCH-WP-0002. + - Supply "source" in every spec; it is not enforced yet. promotion_history: [] --- @@ -115,7 +122,9 @@ promotion_history: [] `web-fetcher` runs declarative fetch specs and returns `Observation` records with the value **and** its provenance. Values are never estimated: each spec -yields a sourced value or an explicit `not_found` / `error` record. +yields a value or an explicit `not_found` / `error` record. Gaps against the +repo's intent (replay, source enforcement, latest-period rule, selective retry) +are tracked in `WEBFETCH-WP-0002`. ## Try it diff --git a/registry/indexes/capabilities.yaml b/registry/indexes/capabilities.yaml index 091a86b..5a37ad8 100644 --- a/registry/indexes/capabilities.yaml +++ b/registry/indexes/capabilities.yaml @@ -7,8 +7,8 @@ capabilities: summary: Standard-library Python fetcher that turns declarative JSON specs into Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing - data is an explicit not_found, never estimated. Optional content-addressed cache replays - a run from the exact bytes it used. + data is an explicit not_found, never estimated. Optional content-addressed cache stores + the exact bytes a run used; replay from it is not implemented yet. domain: infotech status: draft owner: web-fetcher diff --git a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md index f7ab2d2..b5bb553 100644 --- a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md +++ b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md @@ -17,10 +17,11 @@ state_hub_workstream_id: "68acbef4-d706-5cb4-8138-eb3807881f36" # Versioned release and fetch hardening Move `capability.infotech.provenance-fetch` from A2 (source) to A3 (versioned, -installable release) and close the reliability gaps listed in the capability -record, without adding third-party dependencies. +installable release) and close the gaps between `SCOPE.md` (what is) and +`INTENT.md` (what the repo commits to), found in the 2026-10-19 scope-vs-intent +assessment. No third-party dependencies. -## Retry and Backoff in fetch_bytes +## Selective Retry in fetch_bytes ```task id: WEBFETCH-WP-0002-T01 @@ -29,8 +30,10 @@ priority: high state_hub_task_id: "e64ddb81-3d56-5e09-9744-5c0a7c26005b" ``` -Add bounded retry with exponential backoff for transient failures (connection -errors, HTTP 429/5xx), honouring `Retry-After`. Retries must not change +`fetch_bytes` already retries twice with linear backoff, but on every +`URLError` — including permanent HTTP 4xx such as 404. Retry only transient +failures (connection errors, timeouts, HTTP 429/5xx), use exponential backoff, +honour `Retry-After`, and fail fast on other 4xx. Retries must not change provenance semantics: `retrieved_at` and `payload_sha256` describe the attempt that succeeded. Offline tests with a fake getter. @@ -47,6 +50,49 @@ Add a live smoke test (skipped unless `WEB_FETCHER_LIVE=1`) that runs one spec per adapter against the real endpoints, to detect upstream shape changes that fixtures cannot. +## Enforce Source on Every Value + +```task +id: WEBFETCH-WP-0002-T05 +status: todo +priority: high +``` + +INTENT commitment 1: no value without source. Today `source` comes from the +optional spec field, so `http_json`/`http_csv` can emit `status: ok` with +`source: null`. Make a missing source impossible for `ok` observations (reject +the spec as `error`, or derive it from the URL host — decide and document). +Also capture `Last-Modified` as `published` in the HTTP adapters where present. + +## Replay Runs from the Cache + +```task +id: WEBFETCH-WP-0002-T06 +status: todo +priority: high +``` + +INTENT commitment 4. The cache writes `.bin` plus `index.jsonl` but +nothing reads it back. Add a replay mode (`--replay DIR` / `replay_dir=`) that +serves each URL from the cache index without network access and yields an +`error` observation for an uncached URL. Test: a fetch run followed by a replay +run produces identical observations, `retrieved_at` included. + +## Deterministic Latest-Period Selection + +```task +id: WEBFETCH-WP-0002-T07 +status: todo +priority: medium +``` + +INTENT commitment 2. `http_csv` (and `xlsx_wide`) pick the "latest" row by +string comparison of the period, or the last row when no period column is +given; mixed formats (`2024-Q4` vs `2024-10`) can silently select the wrong +value. Normalise periods or require a declared format, and record the chosen +period/row in the observation. Also decide whether a non-numeric value should +stay `ok`. + ## Tagged Release Installable from Forgejo ```task