Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002)

- --replay DIR / replay_dir: offline replay from the cache, hash-verified
- ok values without source become error; Last-Modified recorded as published
- latest period by parsed order (web_fetcher/periods.py); mixed formats refused
- non-numeric values not_found unless value_type: string
- retry only connection errors, timeouts, 429/5xx; honour Retry-After
- opt-in live smoke test; 18 offline tests
- CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2
- WEBFETCH-WP-0003 (repo hygiene) added

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
tegwick 2026-10-19 14:08:35 +02:00
parent 5f4467dad8
commit 96fe5b022b
16 changed files with 696 additions and 133 deletions

View file

@ -18,30 +18,36 @@ failures become `error`; a failing spec never aborts the run.
## What Exists
- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
sets `retrieved_at` and `payload_sha256` on every fetch; retries any
`URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
sets `retrieved_at`, `payload_sha256` and — when the server sends
`Last-Modified` — `published`. Retries only connection errors, timeouts and
HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s);
other HTTP errors fail immediately.
- Cache and replay: `--cache DIR` stores `<sha256>.bin` and an `index.jsonl`
entry per fetch; `--replay DIR` serves URLs from it without network,
verifying hashes and keeping the original retrieval time. The most recent
entry per URL wins, so exact reproduction needs one cache directory per run.
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
and parse errors become `error` observations.
and parse errors become `error` observations; an `ok` value without
`source` becomes `error`.
- Adapters:
- `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
- `eurostat` (JSON-stat 2.0) — names its own source; `published` from the
dataset's `updated`.
- `http_json` — dotted path lookup.
- `http_csv` — row filter; "latest" row chosen by **string** comparison of the
period column, or the last row if none is given; non-numeric values are
returned as strings with `status: ok`.
- `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
- `http_csv` — row filter; latest period by parsed order
(`web_fetcher/periods.py`), mixed formats refused unless `period_format`
is set; without `period_column` the last matching row, stated in `note`.
- `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule;
skips newer periods without a value and says so in `note`.
- `manual` — always `not_found` with a note.
- `source` is copied from the optional spec field; the HTTP adapters can emit
`status: ok` with `source: null`. `published` is unset outside `eurostat`.
- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
`index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
- `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless
`value_type: string`.
- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]`
emitting JSONL; exit code 1 if any observation is `error`. Python API
`run_spec`, `run_specs`.
- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
getters; no live-endpoint tests.
`run_spec`, `run_specs`, `fetch_bytes`.
- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`),
last passed 2026-10-19.
- Reuse-surface capability record `capability.infotech.provenance-fetch`
(`registry/`), vector D2 / A2 / C1 / R1.
(`registry/`), vector D2 / A3 / C2 / R2.
## Out of Scope
@ -55,12 +61,14 @@ failures become `error`; a failing spec never aborts the run.
## Current State
- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
via `WEB_FETCHER_PATH`).
- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the
tag; ~600 lines of Python. See `CHANGELOG.md`.
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still
reads a checkout via `WEB_FETCHER_PATH` rather than the tag
(`WEBFETCH-WP-0003-T03`).
- Running inside rein-aharness is plausible (stdlib only) but unverified.
- Gaps against `INTENT.md` (source enforcement, replay, latest-period
selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.
- No CI and no licence yet (`WEBFETCH-WP-0003`).
## Getting Oriented