tegwick
96fe5b022b
- --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
78 lines
3.6 KiB
Markdown
78 lines
3.6 KiB
Markdown
# SCOPE
|
|
|
|
> What the repository **is** today. What it aspires to be is in `INTENT.md`;
|
|
> the gap between the two is tracked in `workplans/`.
|
|
|
|
## One-liner
|
|
|
|
Standard-library fetching of data from public APIs and web data files into
|
|
provenance-carrying `Observation` records, for research repositories.
|
|
|
|
## Core Idea
|
|
|
|
Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches
|
|
each URL and returns `Observation` records with the value plus URL, retrieval
|
|
time and the SHA-256 of the raw payload. Missing data becomes `not_found`,
|
|
failures become `error`; a failing spec never aborts the run.
|
|
|
|
## What Exists
|
|
|
|
- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
|
|
sets `retrieved_at`, `payload_sha256` and — when the server sends
|
|
`Last-Modified` — `published`. Retries only connection errors, timeouts and
|
|
HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s);
|
|
other HTTP errors fail immediately.
|
|
- Cache and replay: `--cache DIR` stores `<sha256>.bin` and an `index.jsonl`
|
|
entry per fetch; `--replay DIR` serves URLs from it without network,
|
|
verifying hashes and keeping the original retrieval time. The most recent
|
|
entry per URL wins, so exact reproduction needs one cache directory per run.
|
|
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
|
|
and parse errors become `error` observations; an `ok` value without
|
|
`source` becomes `error`.
|
|
- Adapters:
|
|
- `eurostat` (JSON-stat 2.0) — names its own source; `published` from the
|
|
dataset's `updated`.
|
|
- `http_json` — dotted path lookup.
|
|
- `http_csv` — row filter; latest period by parsed order
|
|
(`web_fetcher/periods.py`), mixed formats refused unless `period_format`
|
|
is set; without `period_column` the last matching row, stated in `note`.
|
|
- `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule;
|
|
skips newer periods without a value and says so in `note`.
|
|
- `manual` — always `not_found` with a note.
|
|
- `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless
|
|
`value_type: string`.
|
|
- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]`
|
|
emitting JSONL; exit code 1 if any observation is `error`. Python API
|
|
`run_spec`, `run_specs`, `fetch_bytes`.
|
|
- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`),
|
|
last passed 2026-10-19.
|
|
- Reuse-surface capability record `capability.infotech.provenance-fetch`
|
|
(`registry/`), vector D2 / A3 / C2 / R2.
|
|
|
|
## Out of Scope
|
|
|
|
- Crawling, scraping behind logins, JavaScript-rendered pages
|
|
(use `capability.infotech.headless-browser`).
|
|
- LLM-based extraction or estimation of missing values.
|
|
- Scheduling — activity-core owns *when*.
|
|
- Execution orchestration — rein-aharness owns *how it runs*.
|
|
- Storing or analysing fetched data; that belongs to the consuming research repo.
|
|
- Authentication, pagination, third-party runtime dependencies.
|
|
|
|
## Current State
|
|
|
|
- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the
|
|
tag; ~600 lines of Python. See `CHANGELOG.md`.
|
|
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
|
|
All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still
|
|
reads a checkout via `WEB_FETCHER_PATH` rather than the tag
|
|
(`WEBFETCH-WP-0003-T03`).
|
|
- Running inside rein-aharness is plausible (stdlib only) but unverified.
|
|
- No CI and no licence yet (`WEBFETCH-WP-0003`).
|
|
|
|
## Getting Oriented
|
|
|
|
- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
|
|
- Agent instructions and dev commands: `AGENTS.md`
|
|
- Capability record: `registry/capabilities/`
|
|
- Workplans: `workplans/`
|