tegwick 7d6d5a04c8 Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002

SCOPE.md now states only what exists (write-only cache, retry on any
URLError, optional source, string-compared latest period). INTENT.md
sharpens commitments 1 and 2. README and the capability record drop the
replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05
(enforce source), T06 (replay from cache), T07 (latest-period rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24

2026-10-19 13:58:27 +02:00

1.7 KiB

Raw Blame History

web-fetcher

Reusable, provenance-first fetching of web and API data for research.

Every fetch returns Observation records with the value and where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit not_found or error record. An optional content-addressed cache stores the exact bytes each run used (replay from the cache is not implemented yet; see WEBFETCH-WP-0002).

Standard library only. Python 3.10+.

Adapters

Adapter	Use	Key spec fields
`eurostat`	Eurostat dissemination API (JSON-stat 2.0)	`dataset`, `filters`, `split_by`, `last`
`http_json`	any JSON endpoint	`url`, `path` (dotted), `period_path`
`http_csv`	any CSV endpoint (e.g. FRED `fredgraph.csv`)	`url`, `filter`, `value_column`, `period_column`
`xlsx_wide`	spreadsheets with one column per period (e.g. Census BTOS)	`url`, `sheet`, `filter`, `period_regex`
`manual`	no machine-readable source; recorded as `not_found` with a note	`source`

Usage

python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

Tests

python3 -B -m unittest discover -s tests

Tests run offline against fixtures in tests/fixtures/.

Consumers

assembly-economics: reality checkpoints (checkpoints/sources.json, checkpoints/run_checkpoint.py).