coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
web-fetcher/SCOPE.md
tegwick 2ea8c1337a Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002 
- SCOPE.md rewritten with repo-specific boundaries and current state
- INTENT.md: consumer contract and capability id
- AGENTS.md: developer workflow (test/run/validate)
- registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1)
- workplans: WP-0001 finished, WP-0002 proposed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 13:51:40 +02:00

2 KiB
Raw Blame History

SCOPE

One-liner

Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.

Core Idea

Research repos declare what they need as JSON fetch specs; web-fetcher returns Observation records that carry the value together with source, URL, the source's own publication time, retrieval time and the SHA-256 of the raw payload. Missing data is an explicit not_found, failures an explicit error — never a guess. See INTENT.md for the commitments.

In Scope

  • Observation record, provenance-tracked HTTP fetching (web_fetcher/core.py).
  • Spec dispatch that never raises for a single failing spec (web_fetcher/registry.py).
  • Adapters: eurostat (JSON-stat 2.0), http_json, http_csv, xlsx_wide, and the manual not_found placeholder. New public sources as adapters with offline fixture tests.
  • Optional content-addressed payload cache for replayable runs.
  • CLI (python3 -m web_fetcher fetch specs.json) emitting JSONL, and the Python API.
  • Reuse-surface capability record capability.infotech.provenance-fetch (registry/).

Out of Scope

  • Crawling, scraping behind logins, JavaScript-rendered pages (use capability.infotech.headless-browser).
  • LLM-based extraction or estimation of missing values.
  • Scheduling — activity-core owns when.
  • Execution orchestration — rein-aharness owns how it runs.
  • Storing or analysing fetched data; that belongs to the consuming research repo.
  • Third-party runtime dependencies.

Current State

  • Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
  • Consumer: assembly-economics reality checkpoints (checkpoints/run_checkpoint.py).
  • Known gaps: no retry/backoff, auth or pagination; no tagged release yet (tracked in WEBFETCH-WP-0002).
  • Reuse vector: D2 / A2 / C1 / R1.

Getting Oriented

  • Start with: INTENT.md, then README.md for spec fields and usage.
  • Agent instructions and dev commands: AGENTS.md
  • Capability record: registry/capabilities/
  • Workplans: workplans/