From 96fe5b022bb7944e02db8bd2f0370a4eb1e93027 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 19 Oct 2026 14:08:35 +0200 Subject: [PATCH] Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) - --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24 --- CHANGELOG.md | 37 ++++++ README.md | 35 ++++- SCOPE.md | 54 ++++---- pyproject.toml | 2 +- .../capability.infotech.provenance-fetch.md | 116 +++++++++------- registry/indexes/capabilities.yaml | 7 +- tests/test_live.py | 44 +++++++ tests/test_web_fetcher.py | 124 +++++++++++++++++- web_fetcher/__init__.py | 2 +- web_fetcher/__main__.py | 11 +- web_fetcher/adapters/http.py | 90 +++++++++---- web_fetcher/core.py | 98 ++++++++++++-- web_fetcher/periods.py | 62 +++++++++ web_fetcher/registry.py | 20 ++- ...TCH-WP-0002-release-and-fetch-hardening.md | 33 +++-- workplans/WEBFETCH-WP-0003-repo-hygiene.md | 94 +++++++++++++ 16 files changed, 696 insertions(+), 133 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 tests/test_live.py create mode 100644 web_fetcher/periods.py create mode 100644 workplans/WEBFETCH-WP-0003-repo-hygiene.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..48c13c7 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,37 @@ +# Changelog + +## 0.2.0 — 2026-10-19 + +Closes the gaps between `SCOPE.md` and `INTENT.md` (WEBFETCH-WP-0002). + +### Added +- Replay: `--replay DIR` / `run_spec(..., replay_dir=)` serves every URL from a + cache written by `--cache`, without network access; payload hashes are + verified and the original retrieval time is kept, so a replayed run + reproduces the original observations. Uncached URLs become `error`. +- HTTP `Last-Modified` is recorded as `published` for the `http_json`, + `http_csv` and `xlsx_wide` adapters, and stored in the cache index. +- `period_format` (default `auto`) and `value_type` spec fields. +- Opt-in live smoke test: `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live`. + +### Changed (may affect existing specs) +- A value is never returned without a source: an `ok` observation from a + spec without `source` becomes `error` (the Eurostat adapter names its own). +- "Latest" period is chosen by parsed period order (year, month, date, + quarter, semester, week, fixed-width digits). Mixed or unrecognised formats + are an `error` unless `period_format` is set (`string` = lexicographic). + The rule used is stated in the observation `note`. +- Non-numeric values are `not_found` unless `value_type: string`; `.` is also + treated as a missing marker. Ambiguous matches (several CSV rows with + different values for the latest period, or several XLSX rows) are `error`. +- Retries only on connection errors, timeouts and HTTP 429/5xx, with + exponential backoff (1.5 s, 3 s) or the server's `Retry-After` (capped at + 120 s); other HTTP errors fail immediately. + +Verified: all 31 assembly-economics checkpoint specs give identical +status/value/period to 0.1.0, and a replay of the cached run is byte-identical. + +## 0.1.0 — 2026-10-12 + +Initial release: Observation records, Eurostat / JSON / CSV / wide-XLSX +adapters, write-only payload cache, CLI. diff --git a/README.md b/README.md index c86fc0a..5441fcf 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,8 @@ Every fetch returns `Observation` records with the value **and** where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit `not_found` or `error` record. An -optional content-addressed cache stores the exact bytes each run used (replay -from the cache is not implemented yet; see `WEBFETCH-WP-0002`). +optional content-addressed cache stores the exact bytes each run used, and a +run can be replayed from it offline. Standard library only. Python 3.10+. @@ -16,15 +16,35 @@ Standard library only. Python 3.10+. | Adapter | Use | Key spec fields | |---|---|---| | `eurostat` | Eurostat dissemination API (JSON-stat 2.0) | `dataset`, `filters`, `split_by`, `last` | -| `http_json` | any JSON endpoint | `url`, `path` (dotted), `period_path` | -| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `filter`, `value_column`, `period_column` | -| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `sheet`, `filter`, `period_regex` | +| `http_json` | any JSON endpoint | `url`, `source`, `path` (dotted), `period_path` | +| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `source`, `filter`, `value_column`, `period_column` | +| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `source`, `sheet`, `filter`, `period_regex` | | `manual` | no machine-readable source; recorded as `not_found` with a note | `source` | +### Provenance rules + +- Every non-Eurostat spec must name its `source`; an `ok` value without one is + turned into an `error`. +- "Latest" period (`http_csv`, `xlsx_wide`) follows parsed period order — + year, month (`2026-09`), date, quarter (`2026-Q3`/`2026Q3`), semester, week, + fixed-width digits (`202620`). Mixed formats are an `error`; pin one with + `period_format`, or `period_format: string` for plain lexicographic order. + The rule applied is written to the observation's `note`. +- Values must be numeric unless `value_type: string`; otherwise `not_found`. + +## Install + +```bash +pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0" +``` + +Or put a checkout on `PYTHONPATH` — there are no dependencies. + ## Usage ```bash python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl +python3 -m web_fetcher fetch specs.json --replay .cache > replayed.jsonl # offline, same bytes ``` ```python @@ -40,7 +60,10 @@ obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai", python3 -B -m unittest discover -s tests ``` -Tests run offline against fixtures in `tests/fixtures/`. +Tests run offline against fixtures in `tests/fixtures/`. Live smoke test, +one spec per adapter: `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v`. + +See `CHANGELOG.md` for release notes. ## Consumers diff --git a/SCOPE.md b/SCOPE.md index a2b168d..36d4f2c 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -18,30 +18,36 @@ failures become `error`; a failing spec never aborts the run. ## What Exists - `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`): - sets `retrieved_at` and `payload_sha256` on every fetch; retries any - `URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this - includes permanent HTTP 4xx such as 404; no `Retry-After` handling. + sets `retrieved_at`, `payload_sha256` and — when the server sends + `Last-Modified` — `published`. Retries only connection errors, timeouts and + HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s); + other HTTP errors fail immediately. +- Cache and replay: `--cache DIR` stores `.bin` and an `index.jsonl` + entry per fetch; `--replay DIR` serves URLs from it without network, + verifying hashes and keeping the original retrieval time. The most recent + entry per URL wins, so exact reproduction needs one cache directory per run. - Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures - and parse errors become `error` observations. + and parse errors become `error` observations; an `ok` value without + `source` becomes `error`. - Adapters: - - `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`. + - `eurostat` (JSON-stat 2.0) — names its own source; `published` from the + dataset's `updated`. - `http_json` — dotted path lookup. - - `http_csv` — row filter; "latest" row chosen by **string** comparison of the - period column, or the last row if none is given; non-numeric values are - returned as strings with `status: ok`. - - `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`). + - `http_csv` — row filter; latest period by parsed order + (`web_fetcher/periods.py`), mixed formats refused unless `period_format` + is set; without `period_column` the last matching row, stated in `note`. + - `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule; + skips newer periods without a value and says so in `note`. - `manual` — always `not_found` with a note. -- `source` is copied from the optional spec field; the HTTP adapters can emit - `status: ok` with `source: null`. `published` is unset outside `eurostat`. -- Cache (`--cache DIR`): **write-only**. Stores `.bin` and appends to - `index.jsonl`; nothing reads it back, so runs cannot yet be replayed. -- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]` + - `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless + `value_type: string`. +- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]` emitting JSONL; exit code 1 if any observation is `error`. Python API - `run_spec`, `run_specs`. -- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake - getters; no live-endpoint tests. + `run_spec`, `run_specs`, `fetch_bytes`. +- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`), + last passed 2026-10-19. - Reuse-surface capability record `capability.infotech.provenance-fetch` - (`registry/`), vector D2 / A2 / C1 / R1. + (`registry/`), vector D2 / A3 / C2 / R2. ## Out of Scope @@ -55,12 +61,14 @@ failures become `error`; a failing spec never aborts the run. ## Current State -- Version 0.1.0, ~400 lines of Python, untagged; consumed from source. -- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`, - via `WEB_FETCHER_PATH`). +- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the + tag; ~600 lines of Python. See `CHANGELOG.md`. +- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`). + All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still + reads a checkout via `WEB_FETCHER_PATH` rather than the tag + (`WEBFETCH-WP-0003-T03`). - Running inside rein-aharness is plausible (stdlib only) but unverified. -- Gaps against `INTENT.md` (source enforcement, replay, latest-period - selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`. +- No CI and no licence yet (`WEBFETCH-WP-0003`). ## Getting Oriented diff --git a/pyproject.toml b/pyproject.toml index 94b5e55..c3ab60f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "web-fetcher" -version = "0.1.0" +version = "0.2.0" description = "Provenance-first fetching of web and API data for research" requires-python = ">=3.10" dependencies = [] diff --git a/registry/capabilities/capability.infotech.provenance-fetch.md b/registry/capabilities/capability.infotech.provenance-fetch.md index 4f0f248..ffdfaef 100644 --- a/registry/capabilities/capability.infotech.provenance-fetch.md +++ b/registry/capabilities/capability.infotech.provenance-fetch.md @@ -6,8 +6,9 @@ summary: >- Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found, - never estimated. Optional content-addressed cache stores the exact bytes a - run used; replay from it is not implemented yet. + never estimated, and values without a source are refused. A + content-addressed cache stores the exact bytes of a run and replays it + offline. Tagged release v0.2.0, pip-installable from Forgejo. owner: web-fetcher status: draft domain: infotech @@ -19,49 +20,48 @@ maturity: confidence: medium rationale: > Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md) - and one consumer (assembly-economics reality checkpoints) drove the - design. No survey of alternative fetching libraries in the estate yet. + and checked against the code; one consumer (assembly-economics reality + checkpoints) drove the design. No survey of alternative fetching + libraries in the estate yet. availability: - current: A2 - target: A3 + current: A3 + target: A4 confidence: medium rationale: > - Installable source package (pyproject, CLI entry point, importable API) - on Forgejo with offline fixture tests. Not published to a package index - or as a versioned release artifact; consumers vendor via path or pip - install from the repository. + Versioned, tagged release v0.2.0 with changelog, installable with + pip from the Forgejo tag, CLI and importable API. Not on a package + index; the consumer still reads a checkout rather than the tag. external_evidence: completeness: - level: C1 + level: C2 confidence: medium basis: scope_vs_intent_and_consumer_expectations satisfied_expectations: - - Every value carries URL, retrieval time and payload hash. + - Every ok value carries source, URL, retrieval time and payload hash; a spec without source yields error. + - Source publication time from Eurostat metadata or HTTP Last-Modified when present. + - Runs replay offline from the cache with identical observations. + - Latest-period choice follows a parsed period order; mixed formats are refused and the rule is stated in the note. - Failures and missing values are explicit not_found/error records, never exceptions per spec. - - Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources. + - Eurostat, JSON, CSV and wide-XLSX adapters cover all 27 machine-readable specs of the first consumer. broken_expectations: - - Runs cannot be replayed; the cache is write-only. - - source is optional in specs, so HTTP adapters can return ok values with source null. - - Source publication time is only captured by the Eurostat adapter. - - Latest-period selection compares period strings and can pick the wrong row for mixed formats. - - No pagination, authentication or rate-limit handling for APIs that need it. + - No pagination or authentication for APIs that need it. - Only four adapters; new statistics APIs require code. out_of_scope_expectations: - Crawling, scraping behind logins, LLM-based extraction. - Scheduling (activity-core) and execution orchestration (rein-aharness). reliability: - level: R1 - confidence: low + level: R2 + confidence: medium basis: consumer_quality_signals known_reliability_risks: - - Retries cover every URLError, including permanent 4xx; 429/Retry-After is not honoured. - - Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time. - - Live endpoints are not exercised by the test suite. + - Upstream API shape changes surface as parse errors at run time; the live smoke test is opt-in, not scheduled. - Single consumer so far. + - Replay picks the most recent cache entry per URL; use one cache directory per run for exact reproduction. evidence: satisfied_signals: - - 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures. - - assembly-economics run_checkpoint.py consumes it for light checkpoint runs. + - 18 offline unit tests (fixtures, fake getters, patched HTTP) pass. + - Opt-in live smoke test passed for all four adapters on 2026-10-19. + - All 31 assembly-economics checkpoint specs gave identical status/value/period under 0.1.0 and 0.2.0 on 2026-10-19; replay of the cached run was byte-identical. discovery: intent: > Give research repositories one small, trustworthy way to pull numbers from @@ -69,9 +69,10 @@ discovery: each repo writing ad-hoc urllib or pandas download code. includes: - Declarative JSON fetch specs and a CLI emitting observations as JSONL. - - Python API (run_spec, run_specs, Observation). + - Python API (run_spec, run_specs, Observation, fetch_bytes, replay). - Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder. - - Optional content-addressed payload cache (write-only; replay planned in WEBFETCH-WP-0002). + - Content-addressed payload cache and offline replay. + - Selective retry (connection errors, timeouts, HTTP 429/5xx, Retry-After). excludes: - Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping. - LLM extraction or estimation of missing values. @@ -81,16 +82,18 @@ discovery: - Sources are public and reachable over plain HTTPS. use_cases: - Reality checkpoints comparing forecasts against official statistics. - - Retaining the exact source bytes behind a research run for later audit. + - Reproducible research runs replayed from cached source bytes. availability: - current_level: A2 - target_level: A3 + current_level: A3 + target_level: A4 current_artifacts: - - https://forgejo.coulomb.social/coulomb/web-fetcher + - https://forgejo.coulomb.social/coulomb/web-fetcher/src/tag/v0.2.0 + - git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0 + - CHANGELOG.md - web_fetcher/__main__.py - web_fetcher/registry.py target_artifacts: - - Tagged, versioned release installable with pip from Forgejo. + - Consumers pinned to release tags; published to a package index if a second consumer needs it. consumption_modes: [Python library, cli, source] relations: depends_on: [] @@ -99,39 +102,64 @@ relations: evidence: documentation: - README.md + - CHANGELOG.md - INTENT.md - SCOPE.md tests: - tests/test_web_fetcher.py + - tests/test_live.py consumer_feedback: [] consumer_guidance: recommended_for: - - Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos. + - Fetching official statistics (Eurostat, FRED CSV, World Bank/WHO JSON, Census XLSX) with provenance for research repos. - Runs inside constrained executors where installing dependencies is undesirable. + - Audit or reproduction of a past run from its cached bytes. not_recommended_for: - JavaScript-rendered pages or sites requiring login. - High-volume crawling or bulk mirroring. known_limitations: - - No auth or pagination; retries are two fixed linear attempts on any network/HTTP error. - - Cache is write-only; replay is planned in WEBFETCH-WP-0002. - - Supply "source" in every spec; it is not enforced yet. -promotion_history: [] + - No auth or pagination. + - Every non-Eurostat spec must set "source"; otherwise values become error. + - Mixed period label formats need an explicit period_format. +promotion_history: + - date: "2026-10-19" + dimension: availability + from: A2 + to: A3 + rationale: >- + Tagged release v0.2.0 with changelog and pip install from the Forgejo tag + (WEBFETCH-WP-0002-T03). + author: claude-code + - date: "2026-10-19" + dimension: completeness + from: C1 + to: C2 + rationale: >- + Replay, source enforcement, Last-Modified capture and a stated + latest-period rule close the INTENT gaps found in the scope review. + author: claude-code + - date: "2026-10-19" + dimension: reliability + from: R1 + to: R2 + rationale: >- + Live smoke test across all adapters and a full consumer regression run + (31 specs, identical results, byte-identical replay). + author: claude-code --- # Provenance-first Web and API Data Fetching `web-fetcher` runs declarative fetch specs and returns `Observation` records with the value **and** its provenance. Values are never estimated: each spec -yields a value or an explicit `not_found` / `error` record. Gaps against the -repo's intent (replay, source enforcement, latest-period rule, selective retry) -are tracked in `WEBFETCH-WP-0002`. +yields a sourced value or an explicit `not_found` / `error` record. ## Try it ```bash -git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher -python3 -B -m unittest discover -s tests # offline fixtures -python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl +pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0" +python3 -m web_fetcher fetch specs.json --cache run-01 > observations.jsonl +python3 -m web_fetcher fetch specs.json --replay run-01 > replayed.jsonl # offline ``` ```python @@ -143,4 +171,4 @@ obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai", New sources are added as adapters in `web_fetcher/adapters/` with offline fixture tests. Consumer reference: assembly-economics -`checkpoints/run_checkpoint.py`. +`checkpoints/run_checkpoint.py`. Release notes: `CHANGELOG.md`. diff --git a/registry/indexes/capabilities.yaml b/registry/indexes/capabilities.yaml index 5a37ad8..8559d57 100644 --- a/registry/indexes/capabilities.yaml +++ b/registry/indexes/capabilities.yaml @@ -7,8 +7,9 @@ capabilities: summary: Standard-library Python fetcher that turns declarative JSON specs into Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing - data is an explicit not_found, never estimated. Optional content-addressed cache stores - the exact bytes a run used; replay from it is not implemented yet. + data is an explicit not_found, never estimated, and values without a source are refused. + A content-addressed cache stores the exact bytes of a run and replays it offline. Tagged + release v0.2.0, pip-installable from Forgejo. domain: infotech status: draft owner: web-fetcher @@ -27,7 +28,7 @@ capabilities: - replay - data-ingestion - stdlib - vector: D2 / A2 / C1 / R1 + vector: D2 / A3 / C2 / R2 path: registry/capabilities/capability.infotech.provenance-fetch.md consumption_modes: - Python library diff --git a/tests/test_live.py b/tests/test_live.py new file mode 100644 index 0000000..208d5f9 --- /dev/null +++ b/tests/test_live.py @@ -0,0 +1,44 @@ +"""Live smoke tests against real endpoints, one spec per adapter. + +Skipped unless WEB_FETCHER_LIVE=1. They detect upstream shape changes that the +recorded fixtures cannot; a failure here means a source changed, not +necessarily that web-fetcher is broken. + + WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v +""" + +import os +import unittest + +from web_fetcher.registry import run_spec + +SPECS = [ + {"id": "live-eurostat", "adapter": "eurostat", "dataset": "isoc_eb_ai", + "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10", "indic_is": "E_AI_TANY"}, + "split_by": ["geo"]}, + {"id": "live-json", "adapter": "http_json", + "url": "https://api.worldbank.org/v2/country/DEU/indicator/SE.XPD.TOTL.GD.ZS?format=json&mrnev=1", + "path": "1.0.value", "period_path": "1.0.date", "source": "World Bank WDI (SE.XPD.TOTL.GD.ZS)"}, + {"id": "live-csv", "adapter": "http_csv", "url": "https://fred.stlouisfed.org/graph/fredgraph.csv?id=OPHNFB", + "value_column": "OPHNFB", "period_column": "observation_date", "source": "BLS via FRED (OPHNFB)"}, + {"id": "live-xlsx", "adapter": "xlsx_wide", "url": "https://www.census.gov/hfp/btos/downloads/National.xlsx", + "sheet": "Response Estimates", "filter": {"Question ID": "7", "Answer ID": "1"}, + "source": "U.S. Census Bureau BTOS"}, +] + + +@unittest.skipUnless(os.environ.get("WEB_FETCHER_LIVE") == "1", "set WEB_FETCHER_LIVE=1 for live tests") +class TestLive(unittest.TestCase): + def test_each_adapter_returns_a_sourced_value(self): + for spec in SPECS: + with self.subTest(spec["id"]): + obs = run_spec(spec) + self.assertTrue(obs, "no observations") + o = obs[0] + self.assertEqual(o.status, "ok", o.note) + self.assertIsNotNone(o.value) + self.assertTrue(o.source and o.url and o.retrieved_at and o.payload_sha256) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_web_fetcher.py b/tests/test_web_fetcher.py index 16a96d8..c4a6369 100644 --- a/tests/test_web_fetcher.py +++ b/tests/test_web_fetcher.py @@ -41,7 +41,7 @@ class TestXlsx(unittest.TestCase): body = (FIX / "btos_national.xlsx").read_bytes() obs = run_spec({"id": "us_ai_use", "adapter": "xlsx_wide", "url": "https://x.test/National.xlsx", "sheet": "Response Estimates", "filter": {"Question ID": "7", "Answer ID": "1"}, - "unit": "% of firms"}, get=fake(body)) + "unit": "% of firms", "source": "Census BTOS"}, get=fake(body)) self.assertEqual(obs[0].status, "ok") self.assertEqual(obs[0].period, "202620") self.assertAlmostEqual(obs[0].value, 23.8) @@ -51,7 +51,7 @@ class TestHttp(unittest.TestCase): def test_json_path(self): body = json.dumps({"data": {"rows": [{"v": 7.5, "p": "2026-Q3"}]}}).encode() obs = run_spec({"id": "j", "adapter": "http_json", "url": "https://x.test", "path": "data.rows.0.v", - "period_path": "data.rows.0.p", "unit": "%"}, get=fake(body)) + "period_path": "data.rows.0.p", "unit": "%", "source": "test"}, get=fake(body)) self.assertEqual((obs[0].status, obs[0].value, obs[0].period), ("ok", 7.5, "2026-Q3")) def test_json_missing_path_is_not_found(self): @@ -62,7 +62,8 @@ class TestHttp(unittest.TestCase): def test_csv_latest_matching_row(self): body = b"geo,period,value\nUS,2026-08,8.1\nUS,2026-09,8.7\nDE,2026-09,9.9\n" obs = run_spec({"id": "c", "adapter": "http_csv", "url": "https://x.test", "filter": {"geo": "US"}, - "value_column": "value", "period_column": "period"}, get=fake(body)) + "value_column": "value", "period_column": "period", "source": "test"}, + get=fake(body)) self.assertEqual((obs[0].value, obs[0].period), (8.7, "2026-09")) def test_manual_and_unknown_adapters_never_raise(self): @@ -80,3 +81,120 @@ class TestHttp(unittest.TestCase): if __name__ == "__main__": unittest.main() + + +class FakeResponse: + def __init__(self, body: bytes, headers: dict[str, str] | None = None): + self.body, self.headers = body, headers or {} + + def read(self) -> bytes: + return self.body + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + +def http_error(code: int, headers: dict[str, str] | None = None): + import email.message + import urllib.error + msg = email.message.Message() + for k, v in (headers or {}).items(): + msg[k] = v + return urllib.error.HTTPError("https://x.test", code, "err", msg, None) + + +class TestFetchBytes(unittest.TestCase): + def run_with(self, responses, **kw): + from unittest import mock + + from web_fetcher import core + calls, sleeps = [], [] + + def opener(req, timeout): + calls.append(req.full_url) + r = responses.pop(0) + if isinstance(r, Exception): + raise r + return r + with mock.patch.object(core, "_urlopen", opener), mock.patch.object(core, "_sleep", sleeps.append): + try: + return core.fetch_bytes("https://x.test/a", **kw), calls, sleeps + except core.FetchError as exc: + return exc, calls, sleeps + + def test_404_fails_fast(self): + res, calls, sleeps = self.run_with([http_error(404)]) + self.assertIn("HTTP 404", str(res)) + self.assertEqual((len(calls), sleeps), (1, [])) + + def test_503_honours_retry_after(self): + res, calls, sleeps = self.run_with([http_error(503, {"Retry-After": "7"}), FakeResponse(b"ok")]) + self.assertEqual((res.body, len(calls), sleeps), (b"ok", 2, [7.0])) + + def test_connection_errors_back_off_exponentially(self): + import urllib.error + res, calls, sleeps = self.run_with([urllib.error.URLError("down")] * 3) + self.assertEqual((len(calls), sleeps), (3, [1.5, 3.0])) + self.assertIn("down", str(res)) + + def test_last_modified_becomes_published(self): + res, _, _ = self.run_with([FakeResponse(b"{}", {"Last-Modified": "Wed, 01 Oct 2026 08:00:00 GMT"})]) + self.assertEqual(res.published, "2026-10-01T08:00:00+00:00") + + +class TestReplay(unittest.TestCase): + def test_replay_reproduces_cached_run(self): + import tempfile + from unittest import mock + + from web_fetcher import core + body = json.dumps({"v": 3.2}).encode() + spec = {"id": "r", "adapter": "http_json", "url": "https://x.test/r", "path": "v", "source": "test"} + with tempfile.TemporaryDirectory() as cache: + with mock.patch.object(core, "_urlopen", lambda req, timeout: FakeResponse(body)): + first = run_spec(spec, cache_dir=cache) + + def no_network(req, timeout): + raise AssertionError("network used during replay") + with mock.patch.object(core, "_urlopen", no_network): + again = run_spec(spec, replay_dir=cache) + missing = run_spec({**spec, "url": "https://x.test/other"}, replay_dir=cache) + self.assertEqual([o.to_dict() for o in again], [o.to_dict() for o in first]) + self.assertEqual(first[0].value, 3.2) + self.assertEqual(missing[0].status, "error") + self.assertIn("not in replay cache", missing[0].note) + + +class TestProvenanceRules(unittest.TestCase): + def test_value_without_source_is_error(self): + obs = run_spec({"id": "s", "adapter": "http_json", "url": "https://x.test", "path": "v"}, + get=fake(b'{"v": 1}')) + self.assertEqual(obs[0].status, "error") + self.assertIn("source", obs[0].note) + + def csv(self, body: bytes, **extra): + spec = {"id": "c", "adapter": "http_csv", "url": "https://x.test", "value_column": "value", + "period_column": "period", "source": "test", **extra} + return run_spec(spec, get=fake(body))[0] + + def test_quarters_ordered_by_value_not_string(self): + o = self.csv(b"period,value\n2025-Q4,1\n2026Q1,2\n") + self.assertEqual((o.status, o.period, o.value), ("ok", "2026Q1", 2.0)) + self.assertEqual(o.note, "latest of 2 periods by quarter order") + + def test_mixed_period_formats_are_error(self): + o = self.csv(b"period,value\n2026-Q1,1\n2026-02,2\n") + self.assertEqual(o.status, "error") + self.assertIn("mixed period formats", o.note) + + def test_explicit_string_order_is_allowed(self): + o = self.csv(b"period,value\nb,1\na,2\n", period_format="string") + self.assertEqual((o.status, o.period), ("ok", "b")) + + def test_non_numeric_value_is_not_found_unless_string(self): + body = b"period,value\n2026-01,(D)\n" + self.assertEqual(self.csv(body).status, "not_found") + self.assertEqual(self.csv(body, value_type="string").value, "(D)") diff --git a/web_fetcher/__init__.py b/web_fetcher/__init__.py index 03a6eb5..3f1a42f 100644 --- a/web_fetcher/__init__.py +++ b/web_fetcher/__init__.py @@ -10,4 +10,4 @@ from web_fetcher.core import Observation, fetch_bytes from web_fetcher.registry import ADAPTERS, run_spec, run_specs __all__ = ["Observation", "fetch_bytes", "ADAPTERS", "run_spec", "run_specs"] -__version__ = "0.1.0" +__version__ = "0.2.0" diff --git a/web_fetcher/__main__.py b/web_fetcher/__main__.py index 00ee670..424ef3e 100644 --- a/web_fetcher/__main__.py +++ b/web_fetcher/__main__.py @@ -1,7 +1,8 @@ -"""CLI: python -m web_fetcher fetch SPECS.json [--cache DIR] [--out FILE] +"""CLI: python -m web_fetcher fetch SPECS.json [--cache DIR | --replay DIR] [--out FILE] SPECS.json is a JSON list of spec objects (or {"specs": [...]}). Output is -JSON Lines, one Observation per line. +JSON Lines, one Observation per line. --replay serves every URL from a cache +written by an earlier --cache run, without network access. """ from __future__ import annotations @@ -18,7 +19,9 @@ def main(argv: list[str] | None = None) -> int: sub = ap.add_subparsers(dest="cmd", required=True) f = sub.add_parser("fetch", help="run fetch specs") f.add_argument("specs") - f.add_argument("--cache", default=None, help="content-addressed cache directory") + src = f.add_mutually_exclusive_group() + src.add_argument("--cache", default=None, help="content-addressed cache directory") + src.add_argument("--replay", default=None, help="replay from a cache directory; no network") f.add_argument("--out", default="-", help="output JSONL path (default stdout)") f.add_argument("--only", action="append", default=[], help="run only these spec ids") args = ap.parse_args(argv) @@ -27,7 +30,7 @@ def main(argv: list[str] | None = None) -> int: specs = data["specs"] if isinstance(data, dict) else data if args.only: specs = [s for s in specs if s.get("id") in set(args.only)] - obs = run_specs(specs, cache_dir=args.cache) + obs = run_specs(specs, cache_dir=args.cache, replay_dir=args.replay) lines = "\n".join(o.to_json() for o in obs) + "\n" if args.out == "-": sys.stdout.write(lines) diff --git a/web_fetcher/adapters/http.py b/web_fetcher/adapters/http.py index bf296ac..3467fef 100644 --- a/web_fetcher/adapters/http.py +++ b/web_fetcher/adapters/http.py @@ -2,10 +2,17 @@ r"""Generic HTTP adapters: JSON (dotted path) and CSV (filter + pick latest). json spec fields: url, path ("a.b.0.c"), unit?, period_path? csv spec fields: url, filter (dict column -> value), value_column, - period_column?, unit?, delimiter? -> latest row by period + period_column?, period_format?, value_type?, unit?, delimiter? + -> latest row by period xlsx_wide fields: url, sheet, filter (dict header -> value), period_regex - (header pattern of period columns, default ^\d{6}$), unit? + (header pattern of period columns, default ^\d{6}$), + period_format?, value_type?, unit? -> value of the matching row in the latest period column + +All three take `source`; a value without one is turned into an error by the +registry. "Latest" follows `web_fetcher.periods` (period_format, default auto) +and the chosen rule is stated in the observation's note. Values must be +numeric unless value_type is "string"; otherwise the result is not_found. """ from __future__ import annotations @@ -16,6 +23,20 @@ import json from typing import Any, Callable from web_fetcher.core import Fetched, Observation +from web_fetcher.periods import order + +MISSING = ("", "NA", "N/A", "-", ".", "S") + + +def _value(raw: str, spec: dict[str, Any]) -> tuple[Any, str | None]: + """Parse a cell; returns (value, None) or (None, reason) for non-numeric data.""" + text = raw.strip() + if spec.get("value_type") == "string": + return text, None + try: + return float(text.replace("%", "").replace(",", "")), None + except ValueError: + return None, f"non-numeric value {text!r} (set value_type: string to accept)" def _dig(obj: Any, path: str) -> Any: @@ -40,7 +61,8 @@ def fetch_json(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Obse status = "ok" if value is not None else "not_found" return [Observation(spec_id=spec["id"], status=status, value=value, unit=spec.get("unit"), period=None if period is None else str(period), source=spec.get("source"), - url=f.url, retrieved_at=f.retrieved_at, payload_sha256=f.sha256, + url=f.url, published=f.published, retrieved_at=f.retrieved_at, + payload_sha256=f.sha256, note=None if value is not None else f"path {spec['path']} not found")] @@ -51,20 +73,26 @@ def fetch_csv(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Obser flt = spec.get("filter") or {} rows = [r for r in rows if all(str(r.get(k, "")).strip() == str(v) for k, v in flt.items())] pcol, vcol = spec.get("period_column"), spec["value_column"] - rows = [r for r in rows if str(r.get(vcol, "")).strip() not in ("", "NA", "N/A", "-")] + rows = [r for r in rows if str(r.get(vcol) or "").strip() not in MISSING] + base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at, + payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"), + labels={k: str(v) for k, v in flt.items()}) if not rows: - return [Observation(spec_id=spec["id"], status="not_found", url=f.url, retrieved_at=f.retrieved_at, - payload_sha256=f.sha256, source=spec.get("source"), note="no matching row")] - row = max(rows, key=lambda r: str(r.get(pcol, ""))) if pcol else rows[-1] - raw = str(row[vcol]).strip().replace("%", "") - try: - value: Any = float(raw) - except ValueError: - value = raw - return [Observation(spec_id=spec["id"], status="ok", value=value, unit=spec.get("unit"), - period=str(row.get(pcol)) if pcol else None, source=spec.get("source"), - labels={k: str(v) for k, v in flt.items()}, url=f.url, - retrieved_at=f.retrieved_at, payload_sha256=f.sha256)] + return [Observation(status="not_found", note="no matching row", **base)] + if pcol: + fmt, newest = order(sorted({str(r.get(pcol) or "").strip() for r in rows}), spec.get("period_format", "auto")) + period = newest[0] + latest = [r for r in rows if str(r.get(pcol) or "").strip() == period] + if len({str(r[vcol]).strip() for r in latest}) > 1: + raise ValueError(f"{len(latest)} matching rows with different values for period {period}") + row, rule = latest[-1], f"latest of {len(newest)} periods by {fmt} order" + else: + period, row = None, rows[-1] + rule = f"no period_column: last of {len(rows)} matching rows" + value, why = _value(str(row[vcol]), spec) + if why: + return [Observation(status="not_found", period=period, note=f"{why}; {rule}", **base)] + return [Observation(status="ok", value=value, period=period, note=rule, **base)] def fetch_xlsx_wide(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]: @@ -76,22 +104,30 @@ def fetch_xlsx_wide(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list rows = read_sheet(f.body, spec.get("sheet", 0)) header, body = rows[0], rows[1:] pat = re.compile(spec.get("period_regex", r"^\d{6}$")) - periods = [(i, h) for i, h in enumerate(header) if pat.match(h or "")] + periods = {(h or "").strip(): i for i, h in enumerate(header) if pat.match(h or "")} flt = spec.get("filter") or {} col = {h: i for i, h in enumerate(header)} match = [r for r in body if all(i < len(r) and r[i].strip() == str(v) for h, v in flt.items() for i in [col.get(h, 10**9)])] - base = dict(spec_id=spec["id"], url=f.url, retrieved_at=f.retrieved_at, payload_sha256=f.sha256, - source=spec.get("source"), labels={k: str(v) for k, v in flt.items()}) + base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at, + payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"), + labels={k: str(v) for k, v in flt.items()}) if not match or not periods: return [Observation(status="not_found", note="no matching row or period columns", **base)] + if len(match) > 1: + raise ValueError(f"filter matches {len(match)} rows; expected one") row = match[0] - for i, h in sorted(periods, key=lambda p: p[1], reverse=True): - raw = (row[i] if i < len(row) else "").strip().replace("%", "") - if raw not in ("", "S", "NA", "-"): - try: - value: Any = float(raw) - except ValueError: - value = raw - return [Observation(status="ok", value=value, unit=spec.get("unit"), period=h, **base)] + fmt, newest = order(list(periods), spec.get("period_format", "auto")) + for n, h in enumerate(newest): + i = periods[h] + raw = (row[i] if i < len(row) else "").strip() + if raw in MISSING: + continue + rule = f"latest of {len(newest)} periods by {fmt} order" + if n: + rule += f", skipping {n} newer period(s) without a value" + value, why = _value(raw, spec) + if why: + return [Observation(status="not_found", period=h, note=f"{why}; {rule}", **base)] + return [Observation(status="ok", value=value, period=h, note=rule, **base)] return [Observation(status="not_found", note="matching row has no value", **base)] diff --git a/web_fetcher/core.py b/web_fetcher/core.py index 9fc59a2..9d47c16 100644 --- a/web_fetcher/core.py +++ b/web_fetcher/core.py @@ -10,10 +10,11 @@ import urllib.error import urllib.request from dataclasses import asdict, dataclass, field from datetime import datetime, timezone +from email.utils import parsedate_to_datetime from pathlib import Path from typing import Any -USER_AGENT = "web-fetcher/0.1 (+https://forgejo.coulomb.social/coulomb/web-fetcher)" +USER_AGENT = "web-fetcher/0.2 (+https://forgejo.coulomb.social/coulomb/web-fetcher)" @dataclass @@ -47,47 +48,128 @@ class Fetched: retrieved_at: str sha256: str content_type: str | None + published: str | None = None # HTTP Last-Modified, ISO 8601, when the server sends it class FetchError(RuntimeError): pass +RETRY_STATUS = {429, 500, 502, 503, 504} +MAX_RETRY_AFTER = 120.0 + +# Indirection points for offline tests. +_urlopen = urllib.request.urlopen +_sleep = time.sleep + + def _now() -> str: return datetime.now(timezone.utc).isoformat(timespec="seconds") +def _http_date(value: str | None) -> str | None: + if not value: + return None + try: + return parsedate_to_datetime(value).astimezone(timezone.utc).isoformat(timespec="seconds") + except (TypeError, ValueError): + return None + + +def _retry_after(value: str | None) -> float | None: + """Seconds to wait from a Retry-After header (delta-seconds or HTTP date).""" + if not value: + return None + try: + return max(0.0, float(value)) + except ValueError: + when = _http_date(value) + if when is None: + return None + return max(0.0, (datetime.fromisoformat(when) - datetime.now(timezone.utc)).total_seconds()) + + def fetch_bytes(url: str, *, timeout: float = 60.0, retries: int = 2, cache_dir: str | os.PathLike | None = None, + replay_dir: str | os.PathLike | None = None, headers: dict[str, str] | None = None) -> Fetched: """GET `url` and return body plus provenance. Optional content-addressed cache. - The cache stores each payload under its SHA-256 and an index entry per URL, - so a research run can be replayed offline from the exact bytes it used. + The cache stores each payload under its SHA-256 and an index entry per URL. + With `replay_dir`, no network request is made: the payload and its original + provenance are served from such a cache (see `replay_bytes`). + + Only transient failures are retried (connection errors, timeouts, HTTP + 429/5xx), with exponential backoff or the server's Retry-After; other HTTP + errors fail immediately. Provenance describes the attempt that succeeded. """ + if replay_dir: + return replay_bytes(url, replay_dir) hdrs = {"User-Agent": USER_AGENT, **(headers or {})} last: Exception | None = None for attempt in range(retries + 1): + wait = 1.5 * 2 ** attempt try: req = urllib.request.Request(url, headers=hdrs) - with urllib.request.urlopen(req, timeout=timeout) as resp: + with _urlopen(req, timeout=timeout) as resp: body = resp.read() ctype = resp.headers.get("Content-Type") + modified = _http_date(resp.headers.get("Last-Modified")) digest = hashlib.sha256(body).hexdigest() - fetched = Fetched(url=url, body=body, retrieved_at=_now(), sha256=digest, content_type=ctype) + fetched = Fetched(url=url, body=body, retrieved_at=_now(), sha256=digest, + content_type=ctype, published=modified) if cache_dir: _cache_store(Path(cache_dir), fetched) return fetched + except urllib.error.HTTPError as exc: + if exc.code not in RETRY_STATUS: + raise FetchError(f"fetch failed for {url}: HTTP {exc.code} {exc.reason}") from exc + last = exc + hinted = _retry_after(exc.headers.get("Retry-After") if exc.headers else None) + if hinted is not None: + wait = min(hinted, MAX_RETRY_AFTER) except (urllib.error.URLError, TimeoutError, OSError) as exc: last = exc - if attempt < retries: - time.sleep(1.5 * (attempt + 1)) + if attempt < retries: + _sleep(wait) raise FetchError(f"fetch failed for {url}: {last}") +def replay_bytes(url: str, cache_dir: str | os.PathLike) -> Fetched: + """Serve `url` from a cache written by `fetch_bytes(cache_dir=...)`, without network. + + Uses the most recent index entry for the URL and verifies the payload hash. + Returns the original retrieval time, so a replayed run reproduces the + observations of the run that filled the cache. + """ + cache = Path(cache_dir) + entry: dict[str, Any] | None = None + try: + with (cache / "index.jsonl").open(encoding="utf-8") as fh: + for line in fh: + if line.strip(): + rec = json.loads(line) + if rec.get("url") == url: + entry = rec + except FileNotFoundError: + raise FetchError(f"replay cache {cache} has no index.jsonl") from None + if entry is None: + raise FetchError(f"not in replay cache: {url}") + try: + body = (cache / f"{entry['sha256']}.bin").read_bytes() + except FileNotFoundError: + raise FetchError(f"replay payload missing for {url}: {entry['sha256']}") from None + if hashlib.sha256(body).hexdigest() != entry["sha256"]: + raise FetchError(f"replay payload hash mismatch for {url}") + return Fetched(url=url, body=body, retrieved_at=entry["retrieved_at"], sha256=entry["sha256"], + content_type=entry.get("content_type"), published=entry.get("published")) + + def _cache_store(cache: Path, fetched: Fetched) -> None: cache.mkdir(parents=True, exist_ok=True) (cache / f"{fetched.sha256}.bin").write_bytes(fetched.body) with (cache / "index.jsonl").open("a", encoding="utf-8") as fh: fh.write(json.dumps({"url": fetched.url, "sha256": fetched.sha256, - "retrieved_at": fetched.retrieved_at}) + "\n") + "retrieved_at": fetched.retrieved_at, + "content_type": fetched.content_type, + "published": fetched.published}) + "\n") diff --git a/web_fetcher/periods.py b/web_fetcher/periods.py new file mode 100644 index 0000000..e4d338d --- /dev/null +++ b/web_fetcher/periods.py @@ -0,0 +1,62 @@ +"""Period ordering for "latest value" selection. + +A source's period labels are only compared when they share one recognised +format; mixed or unrecognised formats are an error, never a silent string +comparison. Spec field `period_format` pins the format ("auto" by default); +"string" opts in to plain lexicographic order. +""" + +from __future__ import annotations + +import re +from typing import Callable + +_FORMATS: dict[str, tuple[re.Pattern[str], Callable[[re.Match[str]], tuple[int, ...]]]] = { + "year": (re.compile(r"^(\d{4})$"), lambda m: (int(m[1]),)), + "month": (re.compile(r"^(\d{4})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]))), + "date": (re.compile(r"^(\d{4})-(\d{2})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]), int(m[3]))), + "quarter": (re.compile(r"^(\d{4})-?Q([1-4])$"), lambda m: (int(m[1]), int(m[2]))), + "semester": (re.compile(r"^(\d{4})-?S([12])$"), lambda m: (int(m[1]), int(m[2]))), + "week": (re.compile(r"^(\d{4})-?W(\d{2})$"), lambda m: (int(m[1]), int(m[2]))), + # Fixed-width digit labels such as YYYYMM or Census BTOS YYYYPP. + "digits": (re.compile(r"^(\d+)$"), lambda m: (len(m[1]), int(m[1]))), +} + + +class PeriodError(ValueError): + pass + + +def detect(label: str) -> str | None: + label = label.strip() + for name, (pat, _) in _FORMATS.items(): + if pat.match(label): + return name + return None + + +def sort_key(label: str, fmt: str) -> tuple[int, ...] | str: + if fmt == "string": + return label + pat, key = _FORMATS[fmt] + m = pat.match(label.strip()) + if m is None: + raise PeriodError(f"period {label!r} does not match format {fmt!r}") + return key(m) + + +def order(labels: list[str], fmt: str = "auto") -> tuple[str, list[str]]: + """Return (format, labels newest first). Raises PeriodError for mixed formats.""" + if fmt == "auto": + found = {detect(lbl) for lbl in labels} + if None in found: + bad = [lbl for lbl in labels if detect(lbl) is None][:3] + raise PeriodError(f"unrecognised period labels {bad}; set period_format") + if len(found) > 1: + raise PeriodError(f"mixed period formats {sorted(found)}; set period_format") + fmt = found.pop() if found else "string" + if fmt == "digits" and len({len(lbl.strip()) for lbl in labels}) > 1: + raise PeriodError("digit period labels of different widths; set period_format") + elif fmt != "string" and fmt not in _FORMATS: + raise PeriodError(f"unknown period_format {fmt!r}") + return fmt, sorted(labels, key=lambda lbl: sort_key(lbl, fmt), reverse=True) diff --git a/web_fetcher/registry.py b/web_fetcher/registry.py index de64a7e..4db6fd5 100644 --- a/web_fetcher/registry.py +++ b/web_fetcher/registry.py @@ -16,8 +16,12 @@ ADAPTERS: dict[str, Callable[[dict[str, Any], Callable[[str], Fetched]], list[Ob def run_spec(spec: dict[str, Any], get: Callable[[str], Fetched] | None = None, - cache_dir: str | None = None) -> list[Observation]: - """Run one spec. Adapter errors become an `error` observation, not an exception.""" + cache_dir: str | None = None, replay_dir: str | None = None) -> list[Observation]: + """Run one spec. Adapter errors become an `error` observation, not an exception. + + A value is never returned without a source: an `ok` observation whose + adapter could not name one (the spec lacks `source`) becomes an `error`. + """ sid = spec.get("id", "?") kind = spec.get("adapter") if kind == "manual": @@ -26,15 +30,23 @@ def run_spec(spec: dict[str, Any], get: Callable[[str], Fetched] | None = None, adapter = ADAPTERS.get(kind or "") if adapter is None: return [Observation(spec_id=sid, status="error", note=f"unknown adapter {kind!r}")] - getter = get or (lambda url: fetch_bytes(url, cache_dir=cache_dir)) + getter = get or (lambda url: fetch_bytes(url, cache_dir=cache_dir, replay_dir=replay_dir)) try: - return adapter(spec, getter) + return [_require_source(o) for o in adapter(spec, getter)] except FetchError as exc: return [Observation(spec_id=sid, status="error", note=str(exc)[:300])] except (KeyError, ValueError, TypeError) as exc: return [Observation(spec_id=sid, status="error", note=f"parse error: {exc!s}"[:300])] +def _require_source(obs: Observation) -> Observation: + if obs.status == "ok" and not obs.source: + return Observation(spec_id=obs.spec_id, status="error", url=obs.url, retrieved_at=obs.retrieved_at, + payload_sha256=obs.payload_sha256, + note="spec has no 'source'; values are never returned without one") + return obs + + def run_specs(specs: list[dict[str, Any]], **kw: Any) -> list[Observation]: out: list[Observation] = [] for spec in specs: diff --git a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md index 98f70ab..2b34a62 100644 --- a/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md +++ b/workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md @@ -4,7 +4,7 @@ type: workplan title: "Versioned release and fetch hardening" domain: infotech repo: web-fetcher -status: proposed +status: finished flavor: implementation depends_on: [WEBFETCH-WP-0001] owner: codex @@ -25,7 +25,7 @@ assessment. No third-party dependencies. ```task id: WEBFETCH-WP-0002-T01 -status: todo +status: done priority: high state_hub_task_id: "e64ddb81-3d56-5e09-9744-5c0a7c26005b" ``` @@ -41,7 +41,7 @@ that succeeded. Offline tests with a fake getter. ```task id: WEBFETCH-WP-0002-T02 -status: todo +status: done priority: medium state_hub_task_id: "4cd456c7-6bd3-50a5-b267-ef9223d85e2b" ``` @@ -54,7 +54,7 @@ fixtures cannot. ```task id: WEBFETCH-WP-0002-T05 -status: todo +status: done priority: high state_hub_task_id: "80e43999-0884-52c1-a49d-b6bd494f898d" ``` @@ -69,7 +69,7 @@ Also capture `Last-Modified` as `published` in the HTTP adapters where present. ```task id: WEBFETCH-WP-0002-T06 -status: todo +status: done priority: high state_hub_task_id: "7716b978-995e-58cc-8375-40700e28da96" ``` @@ -84,7 +84,7 @@ run produces identical observations, `retrieved_at` included. ```task id: WEBFETCH-WP-0002-T07 -status: todo +status: done priority: medium state_hub_task_id: "193ee646-ecd8-52ca-b924-2d0502039d9d" ``` @@ -100,23 +100,38 @@ stay `ok`. ```task id: WEBFETCH-WP-0002-T03 -status: todo +status: done priority: high state_hub_task_id: "54db7c59-b27a-59f9-b7a3-04042e3087a2" ``` Add a CHANGELOG, tag `v0.1.0` (or `v0.2.0` with T01), and document `pip install git+https://forgejo.coulomb.social/coulomb/web-fetcher@`. -Point assembly-economics at the tag instead of `WEB_FETCHER_PATH`. +Repointing assembly-economics at the tag is handed off to +`WEBFETCH-WP-0003-T03` and the assembly-economics agent (cross-repo change). ## Update Capability Record ```task id: WEBFETCH-WP-0002-T04 -status: todo +status: done priority: medium state_hub_task_id: "35f54169-764e-54a7-a1c2-7513dbafea37" ``` Promote availability A2 → A3 with evidence citations and a `promotion_history` entry; update `SCOPE.md` current state. + +## Outcome (2026-10-19) + +Released v0.2.0. Replay (`--replay`), source enforcement, `Last-Modified` as +`published`, parsed latest-period rule with stated note, numeric-value check, +selective retry with `Retry-After`, opt-in live smoke test, CHANGELOG, README +install docs, capability record promoted to D2 / A3 / C2 / R2. + +Verification: 18 offline tests pass; live smoke test passes for all four +adapters; all 31 assembly-economics specs give identical status/value/period +under 0.1.0 and 0.2.0, and the replay of the cached run is byte-identical. + +Residuals: consumer pinning to the tag → `WEBFETCH-WP-0003-T03` (and a hub +message to assembly-economics); CI → `WEBFETCH-WP-0003-T01`. diff --git a/workplans/WEBFETCH-WP-0003-repo-hygiene.md b/workplans/WEBFETCH-WP-0003-repo-hygiene.md new file mode 100644 index 0000000..72fafc4 --- /dev/null +++ b/workplans/WEBFETCH-WP-0003-repo-hygiene.md @@ -0,0 +1,94 @@ +--- +id: WEBFETCH-WP-0003 +type: workplan +title: "Repository hygiene: CI, licence, deployment pinning, records" +domain: infotech +repo: web-fetcher +status: ready +flavor: implementation +depends_on: [] +owner: codex +topic_slug: stack +created: "2026-10-19" +updated: "2026-10-19" +--- + +# Repository hygiene: CI, licence, deployment pinning, records + +Smaller gaps found in the 2026-10-19 repo review, outside the +scope-vs-intent work of `WEBFETCH-WP-0002`. + +## Continuous Test Run on Forgejo + +```task +id: WEBFETCH-WP-0003-T01 +status: todo +priority: medium +``` + +Tests only run when someone remembers. Add a minimal Forgejo Actions workflow +(`.forgejo/workflows/test.yml`) running +`python3 -B -m unittest discover -s tests` on Python 3.10 and the newest +supported version, plus `reuse-surface validate` if the runner can reach it. +First confirm that the Forgejo instance has runners available for +`coulomb/web-fetcher`; if not, record that and set this task to `wait`. + +## Choose and Add a Licence + +```task +id: WEBFETCH-WP-0003-T02 +status: wait +priority: medium +``` + +The repo is offered for reuse via reuse-surface and pip installs from Forgejo +but has no `LICENSE`. Licence choice is the founder's decision; once made, add +`LICENSE` and the `license` field in `pyproject.toml`. + +## Pin the railiance01 Deployment to a Release + +```task +id: WEBFETCH-WP-0003-T03 +status: todo +priority: medium +``` + +rein-aharness reads a railiance01 checkout that tracks `main` at whatever +commit it was last pulled (`5fd7ec7` on 2026-10-13). Release `v0.2.0` exists +(`WEBFETCH-WP-0002-T03`); coordinate with assembly-economics so the checkpoint +run consumes a pinned tag instead of a moving checkout. + +## Refresh the Custodian Brief + +```task +id: WEBFETCH-WP-0003-T04 +status: todo +priority: low +``` + +`.custodian-brief.md` still lists the WP-0001 tasks as open. It is generated; +regenerate it with `statehub fix-consistency` rather than editing it by hand. + +## Federate web-fetcher in the reuse-surface Checkout + +```task +id: WEBFETCH-WP-0003-T05 +status: wait +priority: low +``` + +reuse-surface commit `be5c006` adds web-fetcher to +`registry/federation/sources.yaml`. It is committed but not pushed; pushing to +reuse-surface `main` needs founder approval. The hosted service already lists +the capability. + +## Acknowledge Inbox Notices + +```task +id: WEBFETCH-WP-0003-T06 +status: todo +priority: low +``` + +Mark the two news items read and acknowledge the two standing notices +(build-mode security layer, safe secret-check helper) for agent `web-fetcher`.