web-fetcher/SCOPE.md

SCOPE

What the repository is today. What it aspires to be is in INTENT.md; the gap between the two is tracked in workplans/.

One-liner

Standard-library fetching of data from public APIs and web data files into provenance-carrying Observation records, for research repositories.

Core Idea

Research repos declare what they need as JSON fetch specs; web-fetcher fetches each URL and returns Observation records with the value plus URL, retrieval time and the SHA-256 of the raw payload. Missing data becomes not_found, failures become error; a failing spec never aborts the run.

What Exists

• Observation record and fetch_bytes HTTP GET (web_fetcher/core.py): sets retrieved_at, payload_sha256 and — when the server sends Last-Modified — published. Retries only connection errors, timeouts and HTTP 429/5xx (exponential 1.5 s, 3 s, or Retry-After capped at 120 s); other HTTP errors fail immediately.
• Cache and replay: --cache DIR stores <sha256>.bin and an index.jsonl entry per fetch; --replay DIR serves URLs from it without network, verifying hashes and keeping the original retrieval time. The most recent entry per URL wins, so exact reproduction needs one cache directory per run.
• Spec dispatch (web_fetcher/registry.py): unknown adapters, fetch failures and parse errors become error observations; an ok value without source becomes error.
• Adapters:
  • eurostat (JSON-stat 2.0) — names its own source; published from the dataset's updated.
  • http_json — dotted path lookup.
  • http_csv — row filter; latest period by parsed order (web_fetcher/periods.py), mixed formats refused unless period_format is set; without period_column the last matching row, stated in note.
  • xlsx_wide — one column per period, stdlib XLSX reader, same period rule; skips newer periods without a value and says so in note.
  • manual — always not_found with a note.
  • http_csv/xlsx_wide return non-numeric values as not_found unless value_type: string.
• CLI python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only] emitting JSONL; exit code 1 if any observation is error. Python API run_spec, run_specs, fetch_bytes.
• 18 offline unit tests plus an opt-in live smoke test (WEB_FETCHER_LIVE=1), last passed 2026-10-19.
• Reuse-surface capability record capability.infotech.provenance-fetch (registry/), vector D2 / A3 / C2 / R2.

Out of Scope

• Crawling, scraping behind logins, JavaScript-rendered pages (use capability.infotech.headless-browser).
• LLM-based extraction or estimation of missing values.
• Scheduling — activity-core owns when.
• Execution orchestration — rein-aharness owns how it runs.
• Storing or analysing fetched data; that belongs to the consuming research repo.
• Authentication, pagination, third-party runtime dependencies.

Current State

• Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the tag; ~600 lines of Python. See CHANGELOG.md.
• Consumer: assembly-economics reality checkpoints (checkpoints/run_checkpoint.py). All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still reads a checkout via WEB_FETCHER_PATH rather than the tag (WEBFETCH-WP-0003-T03).
• Running inside rein-aharness is plausible (stdlib only) but unverified.
• Licence: Target Revenue Source License V1C1 (org-wide preliminary licence).
• CI: Forgejo Actions runs the offline tests on Python 3.10 and 3.13.

Getting Oriented

• Start with: INTENT.md, then README.md for spec fields and usage.
• Agent instructions and dev commands: AGENTS.md
• Capability record: registry/capabilities/
• Workplans: workplans/