coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Standard-library fetcher to pull information from trusted public data sources like Eurostat, FRED, the World Bank, the Census xlsx files and the like.
18 commits 1 branch 1 tag 244 KiB
Find a file
Exact
tegwick d1f7cbb6a4
All checks were successful
Tests / unittest-py310 (push) Successful in 4s
Details
Tests / unittest-py313 (push) Successful in 4s
Details
Review agent-proposed repo classification (add adaptation); record review in WEBFETCH-IN-0002 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 15:11:22 +02:00
.forgejo/workflows WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
build/lib/web_fetcher WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
decisions Finish WEBFETCH-WP-0003: DEC-2026-001 approved, residual intakes IN-0001/IN-0002 2026-10-19 14:59:45 +02:00
intakes Review agent-proposed repo classification (add adaptation); record review in WEBFETCH-IN-0002 2026-10-19 15:11:22 +02:00
registry Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) 2026-10-19 14:08:35 +02:00
tests Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) 2026-10-19 14:08:35 +02:00
web_fetcher Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) 2026-10-19 14:08:35 +02:00
web_fetcher.egg-info WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
workplans/archived Archive finished workplans WEBFETCH-WP-0001..0003 (261019) 2026-10-19 15:02:28 +02:00
.custodian-brief.md Index residual intakes; regenerate brief and work records 2026-10-19 15:01:25 +02:00
.gitignore Register with State Hub (infotech/stack): AGENTS.md, SCOPE.md, brief, bootstrap workplan 2026-10-13 14:08:19 +02:00
.repo-classification.yaml Review agent-proposed repo classification (add adaptation); record review in WEBFETCH-IN-0002 2026-10-19 15:11:22 +02:00
AGENTS.md WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
CHANGELOG.md Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) 2026-10-19 14:08:35 +02:00
INTENT.md Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002 2026-10-19 13:58:27 +02:00
LICENSE WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
pyproject.toml WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress 2026-10-19 14:27:11 +02:00
README.md Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002) 2026-10-19 14:08:35 +02:00
SCOPE.md WEBFETCH-WP-0003-T03 done: assembly-economics pinned to v0.2.0 2026-10-19 14:52:19 +02:00
WORK-RECORDS.md Review agent-proposed repo classification (add adaptation); record review in WEBFETCH-IN-0002 2026-10-19 15:11:22 +02:00

README.md

web-fetcher

Reusable, provenance-first fetching of web and API data for research.

Every fetch returns Observation records with the value and where it came from: source, URL, the source's own publication or update time, retrieval time, and the SHA-256 of the raw payload. Values are never estimated: a spec yields a sourced value, or an explicit not_found or error record. An optional content-addressed cache stores the exact bytes each run used, and a run can be replayed from it offline.

Standard library only. Python 3.10+.

Adapters

Adapter Use Key spec fields
eurostat Eurostat dissemination API (JSON-stat 2.0) dataset, filters, split_by, last
http_json any JSON endpoint url, source, path (dotted), period_path
http_csv any CSV endpoint (e.g. FRED fredgraph.csv) url, source, filter, value_column, period_column
xlsx_wide spreadsheets with one column per period (e.g. Census BTOS) url, source, sheet, filter, period_regex
manual no machine-readable source; recorded as not_found with a note source

Provenance rules

  • Every non-Eurostat spec must name its source; an ok value without one is turned into an error.
  • "Latest" period (http_csv, xlsx_wide) follows parsed period order — year, month (2026-09), date, quarter (2026-Q3/2026Q3), semester, week, fixed-width digits (202620). Mixed formats are an error; pin one with period_format, or period_format: string for plain lexicographic order. The rule applied is written to the observation's note.
  • Values must be numeric unless value_type: string; otherwise not_found.

Install

pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"

Or put a checkout on PYTHONPATH — there are no dependencies.

Usage

python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl
python3 -m web_fetcher fetch specs.json --replay .cache > replayed.jsonl   # offline, same bytes

from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})

Tests

python3 -B -m unittest discover -s tests

Tests run offline against fixtures in tests/fixtures/. Live smoke test, one spec per adapter: WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v.

See CHANGELOG.md for release notes.

Consumers

  • assembly-economics: reality checkpoints (checkpoints/sources.json, checkpoints/run_checkpoint.py).