web-fetcher/registry/capabilities/capability.infotech.provenance-fetch.md

id	name	summary	owner	status	domain	tags	maturity	external_evidence	discovery	availability	relations	evidence	consumer_guidance	promotion_history
capability.infotech.provenance-fetch	Provenance-first Web and API Data Fetching	Standard-library Python fetcher that turns declarative JSON specs into Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found, never estimated, and values without a source are refused. A content-addressed cache stores the exact bytes of a run and replays it offline. Tagged release v0.2.0, pip-installable from Forgejo.	web-fetcher	draft	infotech	fetch http api provenance research statistics eurostat csv json xlsx cache replay data-ingestion stdlib	discovery availability current target confidence rationale D2 D3 medium Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md) and checked against the code; one consumer (assembly-economics reality checkpoints) drove the design. No survey of alternative fetching libraries in the estate yet. current target confidence rationale A3 A4 medium Versioned, tagged release v0.2.0 with changelog, installable with pip from the Forgejo tag, CLI and importable API. Not on a package index; the consumer still reads a checkout rather than the tag.	completeness reliability level confidence basis satisfied_expectations broken_expectations out_of_scope_expectations C2 medium scope_vs_intent_and_consumer_expectations Every ok value carries source, URL, retrieval time and payload hash; a spec without source yields error. Source publication time from Eurostat metadata or HTTP Last-Modified when present. Runs replay offline from the cache with identical observations. Latest-period choice follows a parsed period order; mixed formats are refused and the rule is stated in the note. Failures and missing values are explicit not_found/error records, never exceptions per spec. Eurostat, JSON, CSV and wide-XLSX adapters cover all 27 machine-readable specs of the first consumer. No pagination or authentication for APIs that need it. Only four adapters; new statistics APIs require code. Crawling, scraping behind logins, LLM-based extraction. Scheduling (activity-core) and execution orchestration (rein-aharness). level confidence basis known_reliability_risks evidence R2 medium consumer_quality_signals Upstream API shape changes surface as parse errors at run time; the live smoke test is opt-in, not scheduled. Single consumer so far. Replay picks the most recent cache entry per URL; use one cache directory per run for exact reproduction. satisfied_signals 18 offline unit tests (fixtures, fake getters, patched HTTP) pass. Opt-in live smoke test passed for all four adapters on 2026-10-19. All 31 assembly-economics checkpoint specs gave identical status/value/period under 0.1.0 and 0.2.0 on 2026-10-19; replay of the cached run was byte-identical.	intent includes excludes assumptions use_cases Give research repositories one small, trustworthy way to pull numbers from public statistics APIs and web data files with full provenance, instead of each repo writing ad-hoc urllib or pandas download code. Declarative JSON fetch specs and a CLI emitting observations as JSONL. Python API (run_spec, run_specs, Observation, fetch_bytes, replay). Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder. Content-addressed payload cache and offline replay. Selective retry (connection errors, timeouts, HTTP 429/5xx, Retry-After). Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping. LLM extraction or estimation of missing values. Scheduling and execution orchestration. Consumers run Python 3.10+; no third-party dependencies are needed. Sources are public and reachable over plain HTTPS. Reality checkpoints comparing forecasts against official statistics. Reproducible research runs replayed from cached source bytes.	current_level target_level current_artifacts target_artifacts consumption_modes A3 A4 https://forgejo.coulomb.social/coulomb/web-fetcher/src/tag/v0.2.0 git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0 CHANGELOG.md web_fetcher/__main__.py web_fetcher/registry.py Consumers pinned to release tags; published to a package index if a second consumer needs it. Python library cli source	depends_on supports related_to capability.infotech.headless-browser	documentation tests consumer_feedback README.md CHANGELOG.md INTENT.md SCOPE.md tests/test_web_fetcher.py tests/test_live.py	recommended_for not_recommended_for known_limitations Fetching official statistics (Eurostat, FRED CSV, World Bank/WHO JSON, Census XLSX) with provenance for research repos. Runs inside constrained executors where installing dependencies is undesirable. Audit or reproduction of a past run from its cached bytes. JavaScript-rendered pages or sites requiring login. High-volume crawling or bulk mirroring. No auth or pagination. Every non-Eurostat spec must set "source"; otherwise values become error. Mixed period label formats need an explicit period_format.	date dimension from to rationale author 2026-10-19 availability A2 A3 Tagged release v0.2.0 with changelog and pip install from the Forgejo tag (WEBFETCH-WP-0002-T03). claude-code date dimension from to rationale author 2026-10-19 completeness C1 C2 Replay, source enforcement, Last-Modified capture and a stated latest-period rule close the INTENT gaps found in the scope review. claude-code date dimension from to rationale author 2026-10-19 reliability R1 R2 Live smoke test across all adapters and a full consumer regression run (31 specs, identical results, byte-identical replay). claude-code

Provenance-first Web and API Data Fetching

web-fetcher runs declarative fetch specs and returns Observation records with the value and its provenance. Values are never estimated: each spec yields a sourced value or an explicit not_found / error record.

Try it

pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"
python3 -m web_fetcher fetch specs.json --cache run-01 > observations.jsonl
python3 -m web_fetcher fetch specs.json --replay run-01 > replayed.jsonl   # offline

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

New sources are added as adapters in web_fetcher/adapters/ with offline fixture tests. Consumer reference: assembly-economics checkpoints/run_checkpoint.py. Release notes: CHANGELOG.md.