web-fetcher/README.md

web-fetcher

Reusable, provenance-first fetching of web and API data for research.

Every fetch returns Observation records with the value and where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit not_found or error record. An optional content-addressed cache lets a research run be replayed from the exact bytes it used.

Standard library only. Python 3.10+.

Adapters

Adapter	Use	Key spec fields
eurostat	Eurostat dissemination API (JSON-stat 2.0)	dataset, filters, split_by, last
http_json	any JSON endpoint	url, path (dotted), period_path
http_csv	any CSV endpoint (e.g. FRED fredgraph.csv)	url, filter, value_column, period_column
xlsx_wide	spreadsheets with one column per period (e.g. Census BTOS)	url, sheet, filter, period_regex
manual	no machine-readable source; recorded as not_found with a note	source

Usage

python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

Tests

python3 -B -m unittest discover -s tests

Tests run offline against fixtures in tests/fixtures/.

Consumers

• assembly-economics: reality checkpoints (checkpoints/sources.json, checkpoints/run_checkpoint.py).