web-fetcher/registry/capabilities/capability.infotech.provenance-fetch.md

id	name	summary	owner	status	domain	tags	maturity	external_evidence	discovery	availability	relations	evidence	consumer_guidance	promotion_history
capability.infotech.provenance-fetch	Provenance-first Web and API Data Fetching	Standard-library Python fetcher that turns declarative JSON specs into Observation records carrying value, source, URL, source publication time, retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found, never estimated. Optional content-addressed cache replays a run from the exact bytes it used.	web-fetcher	draft	infotech	fetch http api provenance research statistics eurostat csv json xlsx cache replay data-ingestion stdlib	discovery availability current target confidence rationale D2 D3 medium Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md) and one consumer (assembly-economics reality checkpoints) drove the design. No survey of alternative fetching libraries in the estate yet. current target confidence rationale A2 A3 medium Installable source package (pyproject, CLI entry point, importable API) on Forgejo with offline fixture tests. Not published to a package index or as a versioned release artifact; consumers vendor via path or pip install from the repository.	completeness reliability level confidence basis satisfied_expectations broken_expectations out_of_scope_expectations C1 medium scope_vs_intent_and_consumer_expectations Every value carries source, URL, retrieval time and payload hash. Failures and missing values are explicit not_found/error records, never exceptions per spec. Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources. No pagination, authentication or rate-limit handling for APIs that need it. Only four adapters; new statistics APIs require code. Crawling, scraping behind logins, LLM-based extraction. Scheduling (activity-core) and execution orchestration (rein-aharness). level confidence basis known_reliability_risks evidence R1 low consumer_quality_signals Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time. Live endpoints are not exercised by the test suite. Single consumer so far. satisfied_signals 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures. assembly-economics run_checkpoint.py consumes it for light checkpoint runs.	intent includes excludes assumptions use_cases Give research repositories one small, trustworthy way to pull numbers from public statistics APIs and web data files with full provenance, instead of each repo writing ad-hoc urllib or pandas download code. Declarative JSON fetch specs and a CLI emitting observations as JSONL. Python API (run_spec, run_specs, Observation). Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder. Optional content-addressed payload cache for replayable runs. Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping. LLM extraction or estimation of missing values. Scheduling and execution orchestration. Consumers run Python 3.10+; no third-party dependencies are needed. Sources are public and reachable over plain HTTPS. Reality checkpoints comparing forecasts against official statistics. Reproducible research runs replayed from cached source bytes.	current_level target_level current_artifacts target_artifacts consumption_modes A2 A3 https://forgejo.coulomb.social/coulomb/web-fetcher web_fetcher/__main__.py web_fetcher/registry.py Tagged, versioned release installable with pip from Forgejo. Python library cli source	depends_on supports related_to capability.infotech.headless-browser	documentation tests consumer_feedback README.md INTENT.md SCOPE.md tests/test_web_fetcher.py	recommended_for not_recommended_for known_limitations Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos. Runs inside constrained executors where installing dependencies is undesirable. JavaScript-rendered pages or sites requiring login. High-volume crawling or bulk mirroring. No auth, pagination or retry/backoff policy beyond a single request.

Provenance-first Web and API Data Fetching

web-fetcher runs declarative fetch specs and returns Observation records with the value and its provenance. Values are never estimated: each spec yields a sourced value or an explicit not_found / error record.

Try it

git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher
python3 -B -m unittest discover -s tests          # offline fixtures
python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

New sources are added as adapters in web_fetcher/adapters/ with offline fixture tests. Consumer reference: assembly-economics checkpoints/run_checkpoint.py.