# SCOPE

> What the repository **is** today. What it aspires to be is in `INTENT.md`;
> the gap between the two is tracked in `workplans/`.

## One-liner

Standard-library fetching of data from public APIs and web data files into
provenance-carrying `Observation` records, for research repositories.

## Core Idea

Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches
each URL and returns `Observation` records with the value plus URL, retrieval
time and the SHA-256 of the raw payload. Missing data becomes `not_found`,
failures become `error`; a failing spec never aborts the run.

## What Exists

- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
  sets `retrieved_at` and `payload_sha256` on every fetch; retries any
  `URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
  includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
  and parse errors become `error` observations.
- Adapters:
  - `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
  - `http_json` — dotted path lookup.
  - `http_csv` — row filter; "latest" row chosen by **string** comparison of the
    period column, or the last row if none is given; non-numeric values are
    returned as strings with `status: ok`.
  - `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
  - `manual` — always `not_found` with a note.
- `source` is copied from the optional spec field; the HTTP adapters can emit
  `status: ok` with `source: null`. `published` is unset outside `eurostat`.
- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
  `index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
  emitting JSONL; exit code 1 if any observation is `error`. Python API
  `run_spec`, `run_specs`.
- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
  getters; no live-endpoint tests.
- Reuse-surface capability record `capability.infotech.provenance-fetch`
  (`registry/`), vector D2 / A2 / C1 / R1.

## Out of Scope

- Crawling, scraping behind logins, JavaScript-rendered pages
  (use `capability.infotech.headless-browser`).
- LLM-based extraction or estimation of missing values.
- Scheduling — activity-core owns *when*.
- Execution orchestration — rein-aharness owns *how it runs*.
- Storing or analysing fetched data; that belongs to the consuming research repo.
- Authentication, pagination, third-party runtime dependencies.

## Current State

- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
  via `WEB_FETCHER_PATH`).
- Running inside rein-aharness is plausible (stdlib only) but unverified.
- Gaps against `INTENT.md` (source enforcement, replay, latest-period
  selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.

## Getting Oriented

- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
- Agent instructions and dev commands: `AGENTS.md`
- Capability record: `registry/capabilities/`
- Workplans: `workplans/`