# SCOPE

> What the repository **is** today. What it aspires to be is in `INTENT.md`;
> the gap between the two is tracked in `workplans/`.

## One-liner

Standard-library fetching of data from public APIs and web data files into
provenance-carrying `Observation` records, for research repositories.

## Core Idea

Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches
each URL and returns `Observation` records with the value plus URL, retrieval
time and the SHA-256 of the raw payload. Missing data becomes `not_found`,
failures become `error`; a failing spec never aborts the run.

## What Exists

- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
  sets `retrieved_at`, `payload_sha256` and — when the server sends
  `Last-Modified` — `published`. Retries only connection errors, timeouts and
  HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s);
  other HTTP errors fail immediately.
- Cache and replay: `--cache DIR` stores `<sha256>.bin` and an `index.jsonl`
  entry per fetch; `--replay DIR` serves URLs from it without network,
  verifying hashes and keeping the original retrieval time. The most recent
  entry per URL wins, so exact reproduction needs one cache directory per run.
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
  and parse errors become `error` observations; an `ok` value without
  `source` becomes `error`.
- Adapters:
  - `eurostat` (JSON-stat 2.0) — names its own source; `published` from the
    dataset's `updated`.
  - `http_json` — dotted path lookup.
  - `http_csv` — row filter; latest period by parsed order
    (`web_fetcher/periods.py`), mixed formats refused unless `period_format`
    is set; without `period_column` the last matching row, stated in `note`.
  - `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule;
    skips newer periods without a value and says so in `note`.
  - `manual` — always `not_found` with a note.
  - `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless
    `value_type: string`.
- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]`
  emitting JSONL; exit code 1 if any observation is `error`. Python API
  `run_spec`, `run_specs`, `fetch_bytes`.
- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`),
  last passed 2026-10-19.
- Reuse-surface capability record `capability.infotech.provenance-fetch`
  (`registry/`), vector D2 / A3 / C2 / R2.

## Out of Scope

- Crawling, scraping behind logins, JavaScript-rendered pages
  (use `capability.infotech.headless-browser`).
- LLM-based extraction or estimation of missing values.
- Scheduling — activity-core owns *when*.
- Execution orchestration — rein-aharness owns *how it runs*.
- Storing or analysing fetched data; that belongs to the consuming research repo.
- Authentication, pagination, third-party runtime dependencies.

## Current State

- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the
  tag; ~600 lines of Python. See `CHANGELOG.md`.
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
  All 31 of its specs give the same results under 0.1.0 and 0.2.0; it requires
  0.2.0 and supports `--cache`/`--replay`. The railiance01 runner still uses a
  checkout rather than the tag (`ASSEMBLY-WP-0014-T01`).
- Running inside rein-aharness is plausible (stdlib only) but unverified.
- Licence: Target Revenue Source License V1C1 (org-wide preliminary licence).
- CI: Forgejo Actions runs the offline tests on Python 3.10 and 3.13.

## Getting Oriented

- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
- Agent instructions and dev commands: `AGENTS.md`
- Capability record: `registry/capabilities/`
- Workplans: `workplans/`