coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
web-fetcher/SCOPE.md
tegwick 7d6d5a04c8 Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002 
SCOPE.md now states only what exists (write-only cache, retry on any
URLError, optional source, string-compared latest period). INTENT.md
sharpens commitments 1 and 2. README and the capability record drop the
replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05
(enforce source), T06 (replay from cache), T07 (latest-period rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 13:58:27 +02:00

3.2 KiB
Raw Blame History

SCOPE

What the repository is today. What it aspires to be is in INTENT.md; the gap between the two is tracked in workplans/.

One-liner

Standard-library fetching of data from public APIs and web data files into provenance-carrying Observation records, for research repositories.

Core Idea

Research repos declare what they need as JSON fetch specs; web-fetcher fetches each URL and returns Observation records with the value plus URL, retrieval time and the SHA-256 of the raw payload. Missing data becomes not_found, failures become error; a failing spec never aborts the run.

What Exists

  • Observation record and fetch_bytes HTTP GET (web_fetcher/core.py): sets retrieved_at and payload_sha256 on every fetch; retries any URLError/timeout/OSError twice with linear backoff (1.5 s, 3 s) — this includes permanent HTTP 4xx such as 404; no Retry-After handling.
  • Spec dispatch (web_fetcher/registry.py): unknown adapters, fetch failures and parse errors become error observations.
  • Adapters:
    • eurostat (JSON-stat 2.0) — also sets published from the dataset's updated.
    • http_json — dotted path lookup.
    • http_csv — row filter; "latest" row chosen by string comparison of the period column, or the last row if none is given; non-numeric values are returned as strings with status: ok.
    • xlsx_wide — one column per period, stdlib XLSX reader (web_fetcher/xlsx.py).
    • manual — always not_found with a note.
  • source is copied from the optional spec field; the HTTP adapters can emit status: ok with source: null. published is unset outside eurostat.
  • Cache (--cache DIR): write-only. Stores <sha256>.bin and appends to index.jsonl; nothing reads it back, so runs cannot yet be replayed.
  • CLI python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only] emitting JSONL; exit code 1 if any observation is error. Python API run_spec, run_specs.
  • 8 offline unit tests against Eurostat and Census BTOS fixtures and fake getters; no live-endpoint tests.
  • Reuse-surface capability record capability.infotech.provenance-fetch (registry/), vector D2 / A2 / C1 / R1.

Out of Scope

  • Crawling, scraping behind logins, JavaScript-rendered pages (use capability.infotech.headless-browser).
  • LLM-based extraction or estimation of missing values.
  • Scheduling — activity-core owns when.
  • Execution orchestration — rein-aharness owns how it runs.
  • Storing or analysing fetched data; that belongs to the consuming research repo.
  • Authentication, pagination, third-party runtime dependencies.

Current State

  • Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
  • Consumer: assembly-economics reality checkpoints (checkpoints/run_checkpoint.py, via WEB_FETCHER_PATH).
  • Running inside rein-aharness is plausible (stdlib only) but unverified.
  • Gaps against INTENT.md (source enforcement, replay, latest-period selection, selective retry, release) are tracked in WEBFETCH-WP-0002.

Getting Oriented

  • Start with: INTENT.md, then README.md for spec fields and usage.
  • Agent instructions and dev commands: AGENTS.md
  • Capability record: registry/capabilities/
  • Workplans: workplans/