coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Standard-library fetcher to pull information from trusted public data sources like Eurostat, FRED, the World Bank, the Census xlsx files and the like.
5 commits 1 branch 1 tag 244 KiB
Find a file
tegwick 2ea8c1337a Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 
- SCOPE.md rewritten with repo-specific boundaries and current state
- INTENT.md: consumer contract and capability id
- AGENTS.md: developer workflow (test/run/validate)
- registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1)
- workplans: WP-0001 finished, WP-0002 proposed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 13:51:40 +02:00
registry Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 2026-10-19 13:51:40 +02:00
tests Seeded repo 2026-10-13 14:04:02 +02:00
web_fetcher Seeded repo 2026-10-13 14:04:02 +02:00
workplans Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 2026-10-19 13:51:40 +02:00
.custodian-brief.md Register with State Hub (infotech/stack): AGENTS.md, SCOPE.md, brief, bootstrap workplan 2026-10-13 14:08:19 +02:00
.gitignore Register with State Hub (infotech/stack): AGENTS.md, SCOPE.md, brief, bootstrap workplan 2026-10-13 14:08:19 +02:00
AGENTS.md Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 2026-10-19 13:51:40 +02:00
INTENT.md Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 2026-10-19 13:51:40 +02:00
pyproject.toml Seeded repo 2026-10-13 14:04:02 +02:00
README.md Seeded repo 2026-10-13 14:04:02 +02:00
SCOPE.md Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 2026-10-19 13:51:40 +02:00

README.md

web-fetcher

Reusable, provenance-first fetching of web and API data for research.

Every fetch returns Observation records with the value and where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit not_found or error record. An optional content-addressed cache lets a research run be replayed from the exact bytes it used.

Standard library only. Python 3.10+.

Adapters

Adapter Use Key spec fields
eurostat Eurostat dissemination API (JSON-stat 2.0) dataset, filters, split_by, last
http_json any JSON endpoint url, path (dotted), period_path
http_csv any CSV endpoint (e.g. FRED fredgraph.csv) url, filter, value_column, period_column
xlsx_wide spreadsheets with one column per period (e.g. Census BTOS) url, sheet, filter, period_regex
manual no machine-readable source; recorded as not_found with a note source

Usage

python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

Tests

python3 -B -m unittest discover -s tests

Tests run offline against fixtures in tests/fixtures/.

Consumers

  • assembly-economics: reality checkpoints (checkpoints/sources.json, checkpoints/run_checkpoint.py).