Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002)
- --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
parent
5f4467dad8
commit
96fe5b022b
16 changed files with 696 additions and 133 deletions
54
SCOPE.md
54
SCOPE.md
|
|
@ -18,30 +18,36 @@ failures become `error`; a failing spec never aborts the run.
|
|||
## What Exists
|
||||
|
||||
- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
|
||||
sets `retrieved_at` and `payload_sha256` on every fetch; retries any
|
||||
`URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
|
||||
includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
|
||||
sets `retrieved_at`, `payload_sha256` and — when the server sends
|
||||
`Last-Modified` — `published`. Retries only connection errors, timeouts and
|
||||
HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s);
|
||||
other HTTP errors fail immediately.
|
||||
- Cache and replay: `--cache DIR` stores `<sha256>.bin` and an `index.jsonl`
|
||||
entry per fetch; `--replay DIR` serves URLs from it without network,
|
||||
verifying hashes and keeping the original retrieval time. The most recent
|
||||
entry per URL wins, so exact reproduction needs one cache directory per run.
|
||||
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
|
||||
and parse errors become `error` observations.
|
||||
and parse errors become `error` observations; an `ok` value without
|
||||
`source` becomes `error`.
|
||||
- Adapters:
|
||||
- `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
|
||||
- `eurostat` (JSON-stat 2.0) — names its own source; `published` from the
|
||||
dataset's `updated`.
|
||||
- `http_json` — dotted path lookup.
|
||||
- `http_csv` — row filter; "latest" row chosen by **string** comparison of the
|
||||
period column, or the last row if none is given; non-numeric values are
|
||||
returned as strings with `status: ok`.
|
||||
- `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
|
||||
- `http_csv` — row filter; latest period by parsed order
|
||||
(`web_fetcher/periods.py`), mixed formats refused unless `period_format`
|
||||
is set; without `period_column` the last matching row, stated in `note`.
|
||||
- `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule;
|
||||
skips newer periods without a value and says so in `note`.
|
||||
- `manual` — always `not_found` with a note.
|
||||
- `source` is copied from the optional spec field; the HTTP adapters can emit
|
||||
`status: ok` with `source: null`. `published` is unset outside `eurostat`.
|
||||
- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
|
||||
`index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
|
||||
- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
|
||||
- `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless
|
||||
`value_type: string`.
|
||||
- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]`
|
||||
emitting JSONL; exit code 1 if any observation is `error`. Python API
|
||||
`run_spec`, `run_specs`.
|
||||
- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
|
||||
getters; no live-endpoint tests.
|
||||
`run_spec`, `run_specs`, `fetch_bytes`.
|
||||
- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`),
|
||||
last passed 2026-10-19.
|
||||
- Reuse-surface capability record `capability.infotech.provenance-fetch`
|
||||
(`registry/`), vector D2 / A2 / C1 / R1.
|
||||
(`registry/`), vector D2 / A3 / C2 / R2.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
|
|
@ -55,12 +61,14 @@ failures become `error`; a failing spec never aborts the run.
|
|||
|
||||
## Current State
|
||||
|
||||
- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
|
||||
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
|
||||
via `WEB_FETCHER_PATH`).
|
||||
- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the
|
||||
tag; ~600 lines of Python. See `CHANGELOG.md`.
|
||||
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
|
||||
All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still
|
||||
reads a checkout via `WEB_FETCHER_PATH` rather than the tag
|
||||
(`WEBFETCH-WP-0003-T03`).
|
||||
- Running inside rein-aharness is plausible (stdlib only) but unverified.
|
||||
- Gaps against `INTENT.md` (source enforcement, replay, latest-period
|
||||
selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.
|
||||
- No CI and no licence yet (`WEBFETCH-WP-0003`).
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue