web-fetcher/SCOPE.md

# SCOPE

## One-liner

Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.

## Core Idea

Research repos declare *what* they need as JSON fetch specs; web-fetcher returns
`Observation` records that carry the value together with source, URL, the
source's own publication time, retrieval time and the SHA-256 of the raw
payload. Missing data is an explicit `not_found`, failures an explicit `error` —
never a guess. See `INTENT.md` for the commitments.

## In Scope

- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`).
- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`).
- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`,
  and the `manual` not_found placeholder. New public sources as adapters with
  offline fixture tests.
- Optional content-addressed payload cache for replayable runs.
- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API.
- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`).

## Out of Scope

- Crawling, scraping behind logins, JavaScript-rendered pages
  (use `capability.infotech.headless-browser`).
- LLM-based extraction or estimation of missing values.
- Scheduling — activity-core owns *when*.
- Execution orchestration — rein-aharness owns *how it runs*.
- Storing or analysing fetched data; that belongs to the consuming research repo.
- Third-party runtime dependencies.

## Current State

- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
- Known gaps: no retry/backoff, auth or pagination; no tagged release yet
  (tracked in `WEBFETCH-WP-0002`).
- Reuse vector: D2 / A2 / C1 / R1.

## Getting Oriented

- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
- Agent instructions and dev commands: `AGENTS.md`
- Capability record: `registry/capabilities/`
- Workplans: `workplans/`