Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002)
- --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
parent
5f4467dad8
commit
96fe5b022b
16 changed files with 696 additions and 133 deletions
|
|
@ -6,8 +6,9 @@ summary: >-
|
|||
Observation records carrying value, source, URL, source publication time,
|
||||
retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic
|
||||
JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found,
|
||||
never estimated. Optional content-addressed cache stores the exact bytes a
|
||||
run used; replay from it is not implemented yet.
|
||||
never estimated, and values without a source are refused. A
|
||||
content-addressed cache stores the exact bytes of a run and replays it
|
||||
offline. Tagged release v0.2.0, pip-installable from Forgejo.
|
||||
owner: web-fetcher
|
||||
status: draft
|
||||
domain: infotech
|
||||
|
|
@ -19,49 +20,48 @@ maturity:
|
|||
confidence: medium
|
||||
rationale: >
|
||||
Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md)
|
||||
and one consumer (assembly-economics reality checkpoints) drove the
|
||||
design. No survey of alternative fetching libraries in the estate yet.
|
||||
and checked against the code; one consumer (assembly-economics reality
|
||||
checkpoints) drove the design. No survey of alternative fetching
|
||||
libraries in the estate yet.
|
||||
availability:
|
||||
current: A2
|
||||
target: A3
|
||||
current: A3
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Installable source package (pyproject, CLI entry point, importable API)
|
||||
on Forgejo with offline fixture tests. Not published to a package index
|
||||
or as a versioned release artifact; consumers vendor via path or pip
|
||||
install from the repository.
|
||||
Versioned, tagged release v0.2.0 with changelog, installable with
|
||||
pip from the Forgejo tag, CLI and importable API. Not on a package
|
||||
index; the consumer still reads a checkout rather than the tag.
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
level: C2
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- Every value carries URL, retrieval time and payload hash.
|
||||
- Every ok value carries source, URL, retrieval time and payload hash; a spec without source yields error.
|
||||
- Source publication time from Eurostat metadata or HTTP Last-Modified when present.
|
||||
- Runs replay offline from the cache with identical observations.
|
||||
- Latest-period choice follows a parsed period order; mixed formats are refused and the rule is stated in the note.
|
||||
- Failures and missing values are explicit not_found/error records, never exceptions per spec.
|
||||
- Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources.
|
||||
- Eurostat, JSON, CSV and wide-XLSX adapters cover all 27 machine-readable specs of the first consumer.
|
||||
broken_expectations:
|
||||
- Runs cannot be replayed; the cache is write-only.
|
||||
- source is optional in specs, so HTTP adapters can return ok values with source null.
|
||||
- Source publication time is only captured by the Eurostat adapter.
|
||||
- Latest-period selection compares period strings and can pick the wrong row for mixed formats.
|
||||
- No pagination, authentication or rate-limit handling for APIs that need it.
|
||||
- No pagination or authentication for APIs that need it.
|
||||
- Only four adapters; new statistics APIs require code.
|
||||
out_of_scope_expectations:
|
||||
- Crawling, scraping behind logins, LLM-based extraction.
|
||||
- Scheduling (activity-core) and execution orchestration (rein-aharness).
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
level: R2
|
||||
confidence: medium
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- Retries cover every URLError, including permanent 4xx; 429/Retry-After is not honoured.
|
||||
- Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time.
|
||||
- Live endpoints are not exercised by the test suite.
|
||||
- Upstream API shape changes surface as parse errors at run time; the live smoke test is opt-in, not scheduled.
|
||||
- Single consumer so far.
|
||||
- Replay picks the most recent cache entry per URL; use one cache directory per run for exact reproduction.
|
||||
evidence:
|
||||
satisfied_signals:
|
||||
- 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures.
|
||||
- assembly-economics run_checkpoint.py consumes it for light checkpoint runs.
|
||||
- 18 offline unit tests (fixtures, fake getters, patched HTTP) pass.
|
||||
- Opt-in live smoke test passed for all four adapters on 2026-10-19.
|
||||
- All 31 assembly-economics checkpoint specs gave identical status/value/period under 0.1.0 and 0.2.0 on 2026-10-19; replay of the cached run was byte-identical.
|
||||
discovery:
|
||||
intent: >
|
||||
Give research repositories one small, trustworthy way to pull numbers from
|
||||
|
|
@ -69,9 +69,10 @@ discovery:
|
|||
each repo writing ad-hoc urllib or pandas download code.
|
||||
includes:
|
||||
- Declarative JSON fetch specs and a CLI emitting observations as JSONL.
|
||||
- Python API (run_spec, run_specs, Observation).
|
||||
- Python API (run_spec, run_specs, Observation, fetch_bytes, replay).
|
||||
- Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder.
|
||||
- Optional content-addressed payload cache (write-only; replay planned in WEBFETCH-WP-0002).
|
||||
- Content-addressed payload cache and offline replay.
|
||||
- Selective retry (connection errors, timeouts, HTTP 429/5xx, Retry-After).
|
||||
excludes:
|
||||
- Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping.
|
||||
- LLM extraction or estimation of missing values.
|
||||
|
|
@ -81,16 +82,18 @@ discovery:
|
|||
- Sources are public and reachable over plain HTTPS.
|
||||
use_cases:
|
||||
- Reality checkpoints comparing forecasts against official statistics.
|
||||
- Retaining the exact source bytes behind a research run for later audit.
|
||||
- Reproducible research runs replayed from cached source bytes.
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A3
|
||||
current_level: A3
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- https://forgejo.coulomb.social/coulomb/web-fetcher
|
||||
- https://forgejo.coulomb.social/coulomb/web-fetcher/src/tag/v0.2.0
|
||||
- git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0
|
||||
- CHANGELOG.md
|
||||
- web_fetcher/__main__.py
|
||||
- web_fetcher/registry.py
|
||||
target_artifacts:
|
||||
- Tagged, versioned release installable with pip from Forgejo.
|
||||
- Consumers pinned to release tags; published to a package index if a second consumer needs it.
|
||||
consumption_modes: [Python library, cli, source]
|
||||
relations:
|
||||
depends_on: []
|
||||
|
|
@ -99,39 +102,64 @@ relations:
|
|||
evidence:
|
||||
documentation:
|
||||
- README.md
|
||||
- CHANGELOG.md
|
||||
- INTENT.md
|
||||
- SCOPE.md
|
||||
tests:
|
||||
- tests/test_web_fetcher.py
|
||||
- tests/test_live.py
|
||||
consumer_feedback: []
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos.
|
||||
- Fetching official statistics (Eurostat, FRED CSV, World Bank/WHO JSON, Census XLSX) with provenance for research repos.
|
||||
- Runs inside constrained executors where installing dependencies is undesirable.
|
||||
- Audit or reproduction of a past run from its cached bytes.
|
||||
not_recommended_for:
|
||||
- JavaScript-rendered pages or sites requiring login.
|
||||
- High-volume crawling or bulk mirroring.
|
||||
known_limitations:
|
||||
- No auth or pagination; retries are two fixed linear attempts on any network/HTTP error.
|
||||
- Cache is write-only; replay is planned in WEBFETCH-WP-0002.
|
||||
- Supply "source" in every spec; it is not enforced yet.
|
||||
promotion_history: []
|
||||
- No auth or pagination.
|
||||
- Every non-Eurostat spec must set "source"; otherwise values become error.
|
||||
- Mixed period label formats need an explicit period_format.
|
||||
promotion_history:
|
||||
- date: "2026-10-19"
|
||||
dimension: availability
|
||||
from: A2
|
||||
to: A3
|
||||
rationale: >-
|
||||
Tagged release v0.2.0 with changelog and pip install from the Forgejo tag
|
||||
(WEBFETCH-WP-0002-T03).
|
||||
author: claude-code
|
||||
- date: "2026-10-19"
|
||||
dimension: completeness
|
||||
from: C1
|
||||
to: C2
|
||||
rationale: >-
|
||||
Replay, source enforcement, Last-Modified capture and a stated
|
||||
latest-period rule close the INTENT gaps found in the scope review.
|
||||
author: claude-code
|
||||
- date: "2026-10-19"
|
||||
dimension: reliability
|
||||
from: R1
|
||||
to: R2
|
||||
rationale: >-
|
||||
Live smoke test across all adapters and a full consumer regression run
|
||||
(31 specs, identical results, byte-identical replay).
|
||||
author: claude-code
|
||||
---
|
||||
|
||||
# Provenance-first Web and API Data Fetching
|
||||
|
||||
`web-fetcher` runs declarative fetch specs and returns `Observation` records
|
||||
with the value **and** its provenance. Values are never estimated: each spec
|
||||
yields a value or an explicit `not_found` / `error` record. Gaps against the
|
||||
repo's intent (replay, source enforcement, latest-period rule, selective retry)
|
||||
are tracked in `WEBFETCH-WP-0002`.
|
||||
yields a sourced value or an explicit `not_found` / `error` record.
|
||||
|
||||
## Try it
|
||||
|
||||
```bash
|
||||
git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher
|
||||
python3 -B -m unittest discover -s tests # offline fixtures
|
||||
python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl
|
||||
pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"
|
||||
python3 -m web_fetcher fetch specs.json --cache run-01 > observations.jsonl
|
||||
python3 -m web_fetcher fetch specs.json --replay run-01 > replayed.jsonl # offline
|
||||
```
|
||||
|
||||
```python
|
||||
|
|
@ -143,4 +171,4 @@ obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
|||
|
||||
New sources are added as adapters in `web_fetcher/adapters/` with offline
|
||||
fixture tests. Consumer reference: assembly-economics
|
||||
`checkpoints/run_checkpoint.py`.
|
||||
`checkpoints/run_checkpoint.py`. Release notes: `CHANGELOG.md`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue