Release 0.2.0: replay, source enforcement, period rule, selective retry (WEBFETCH-WP-0002)
- --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
parent
5f4467dad8
commit
96fe5b022b
16 changed files with 696 additions and 133 deletions
37
CHANGELOG.md
Normal file
37
CHANGELOG.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Changelog
|
||||
|
||||
## 0.2.0 — 2026-10-19
|
||||
|
||||
Closes the gaps between `SCOPE.md` and `INTENT.md` (WEBFETCH-WP-0002).
|
||||
|
||||
### Added
|
||||
- Replay: `--replay DIR` / `run_spec(..., replay_dir=)` serves every URL from a
|
||||
cache written by `--cache`, without network access; payload hashes are
|
||||
verified and the original retrieval time is kept, so a replayed run
|
||||
reproduces the original observations. Uncached URLs become `error`.
|
||||
- HTTP `Last-Modified` is recorded as `published` for the `http_json`,
|
||||
`http_csv` and `xlsx_wide` adapters, and stored in the cache index.
|
||||
- `period_format` (default `auto`) and `value_type` spec fields.
|
||||
- Opt-in live smoke test: `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live`.
|
||||
|
||||
### Changed (may affect existing specs)
|
||||
- A value is never returned without a source: an `ok` observation from a
|
||||
spec without `source` becomes `error` (the Eurostat adapter names its own).
|
||||
- "Latest" period is chosen by parsed period order (year, month, date,
|
||||
quarter, semester, week, fixed-width digits). Mixed or unrecognised formats
|
||||
are an `error` unless `period_format` is set (`string` = lexicographic).
|
||||
The rule used is stated in the observation `note`.
|
||||
- Non-numeric values are `not_found` unless `value_type: string`; `.` is also
|
||||
treated as a missing marker. Ambiguous matches (several CSV rows with
|
||||
different values for the latest period, or several XLSX rows) are `error`.
|
||||
- Retries only on connection errors, timeouts and HTTP 429/5xx, with
|
||||
exponential backoff (1.5 s, 3 s) or the server's `Retry-After` (capped at
|
||||
120 s); other HTTP errors fail immediately.
|
||||
|
||||
Verified: all 31 assembly-economics checkpoint specs give identical
|
||||
status/value/period to 0.1.0, and a replay of the cached run is byte-identical.
|
||||
|
||||
## 0.1.0 — 2026-10-12
|
||||
|
||||
Initial release: Observation records, Eurostat / JSON / CSV / wide-XLSX
|
||||
adapters, write-only payload cache, CLI.
|
||||
35
README.md
35
README.md
|
|
@ -6,8 +6,8 @@ Every fetch returns `Observation` records with the value **and** where it came
|
|||
from: source, URL, the source's own publication or update time, retrieval
|
||||
time, and the SHA-256 of the raw payload. Values are never estimated: a spec
|
||||
yields a sourced value, or an explicit `not_found` or `error` record. An
|
||||
optional content-addressed cache stores the exact bytes each run used (replay
|
||||
from the cache is not implemented yet; see `WEBFETCH-WP-0002`).
|
||||
optional content-addressed cache stores the exact bytes each run used, and a
|
||||
run can be replayed from it offline.
|
||||
|
||||
Standard library only. Python 3.10+.
|
||||
|
||||
|
|
@ -16,15 +16,35 @@ Standard library only. Python 3.10+.
|
|||
| Adapter | Use | Key spec fields |
|
||||
|---|---|---|
|
||||
| `eurostat` | Eurostat dissemination API (JSON-stat 2.0) | `dataset`, `filters`, `split_by`, `last` |
|
||||
| `http_json` | any JSON endpoint | `url`, `path` (dotted), `period_path` |
|
||||
| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `filter`, `value_column`, `period_column` |
|
||||
| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `sheet`, `filter`, `period_regex` |
|
||||
| `http_json` | any JSON endpoint | `url`, `source`, `path` (dotted), `period_path` |
|
||||
| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `source`, `filter`, `value_column`, `period_column` |
|
||||
| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `source`, `sheet`, `filter`, `period_regex` |
|
||||
| `manual` | no machine-readable source; recorded as `not_found` with a note | `source` |
|
||||
|
||||
### Provenance rules
|
||||
|
||||
- Every non-Eurostat spec must name its `source`; an `ok` value without one is
|
||||
turned into an `error`.
|
||||
- "Latest" period (`http_csv`, `xlsx_wide`) follows parsed period order —
|
||||
year, month (`2026-09`), date, quarter (`2026-Q3`/`2026Q3`), semester, week,
|
||||
fixed-width digits (`202620`). Mixed formats are an `error`; pin one with
|
||||
`period_format`, or `period_format: string` for plain lexicographic order.
|
||||
The rule applied is written to the observation's `note`.
|
||||
- Values must be numeric unless `value_type: string`; otherwise `not_found`.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"
|
||||
```
|
||||
|
||||
Or put a checkout on `PYTHONPATH` — there are no dependencies.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl
|
||||
python3 -m web_fetcher fetch specs.json --replay .cache > replayed.jsonl # offline, same bytes
|
||||
```
|
||||
|
||||
```python
|
||||
|
|
@ -40,7 +60,10 @@ obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
|||
python3 -B -m unittest discover -s tests
|
||||
```
|
||||
|
||||
Tests run offline against fixtures in `tests/fixtures/`.
|
||||
Tests run offline against fixtures in `tests/fixtures/`. Live smoke test,
|
||||
one spec per adapter: `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v`.
|
||||
|
||||
See `CHANGELOG.md` for release notes.
|
||||
|
||||
## Consumers
|
||||
|
||||
|
|
|
|||
54
SCOPE.md
54
SCOPE.md
|
|
@ -18,30 +18,36 @@ failures become `error`; a failing spec never aborts the run.
|
|||
## What Exists
|
||||
|
||||
- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
|
||||
sets `retrieved_at` and `payload_sha256` on every fetch; retries any
|
||||
`URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
|
||||
includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
|
||||
sets `retrieved_at`, `payload_sha256` and — when the server sends
|
||||
`Last-Modified` — `published`. Retries only connection errors, timeouts and
|
||||
HTTP 429/5xx (exponential 1.5 s, 3 s, or `Retry-After` capped at 120 s);
|
||||
other HTTP errors fail immediately.
|
||||
- Cache and replay: `--cache DIR` stores `<sha256>.bin` and an `index.jsonl`
|
||||
entry per fetch; `--replay DIR` serves URLs from it without network,
|
||||
verifying hashes and keeping the original retrieval time. The most recent
|
||||
entry per URL wins, so exact reproduction needs one cache directory per run.
|
||||
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
|
||||
and parse errors become `error` observations.
|
||||
and parse errors become `error` observations; an `ok` value without
|
||||
`source` becomes `error`.
|
||||
- Adapters:
|
||||
- `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
|
||||
- `eurostat` (JSON-stat 2.0) — names its own source; `published` from the
|
||||
dataset's `updated`.
|
||||
- `http_json` — dotted path lookup.
|
||||
- `http_csv` — row filter; "latest" row chosen by **string** comparison of the
|
||||
period column, or the last row if none is given; non-numeric values are
|
||||
returned as strings with `status: ok`.
|
||||
- `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
|
||||
- `http_csv` — row filter; latest period by parsed order
|
||||
(`web_fetcher/periods.py`), mixed formats refused unless `period_format`
|
||||
is set; without `period_column` the last matching row, stated in `note`.
|
||||
- `xlsx_wide` — one column per period, stdlib XLSX reader, same period rule;
|
||||
skips newer periods without a value and says so in `note`.
|
||||
- `manual` — always `not_found` with a note.
|
||||
- `source` is copied from the optional spec field; the HTTP adapters can emit
|
||||
`status: ok` with `source: null`. `published` is unset outside `eurostat`.
|
||||
- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
|
||||
`index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
|
||||
- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
|
||||
- `http_csv`/`xlsx_wide` return non-numeric values as `not_found` unless
|
||||
`value_type: string`.
|
||||
- CLI `python3 -m web_fetcher fetch specs.json [--cache | --replay] [--out] [--only]`
|
||||
emitting JSONL; exit code 1 if any observation is `error`. Python API
|
||||
`run_spec`, `run_specs`.
|
||||
- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
|
||||
getters; no live-endpoint tests.
|
||||
`run_spec`, `run_specs`, `fetch_bytes`.
|
||||
- 18 offline unit tests plus an opt-in live smoke test (`WEB_FETCHER_LIVE=1`),
|
||||
last passed 2026-10-19.
|
||||
- Reuse-surface capability record `capability.infotech.provenance-fetch`
|
||||
(`registry/`), vector D2 / A2 / C1 / R1.
|
||||
(`registry/`), vector D2 / A3 / C2 / R2.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
|
|
@ -55,12 +61,14 @@ failures become `error`; a failing spec never aborts the run.
|
|||
|
||||
## Current State
|
||||
|
||||
- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
|
||||
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
|
||||
via `WEB_FETCHER_PATH`).
|
||||
- Release v0.2.0 (2026-10-19), tagged on Forgejo and pip-installable from the
|
||||
tag; ~600 lines of Python. See `CHANGELOG.md`.
|
||||
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
|
||||
All 31 of its specs give the same results under 0.1.0 and 0.2.0. It still
|
||||
reads a checkout via `WEB_FETCHER_PATH` rather than the tag
|
||||
(`WEBFETCH-WP-0003-T03`).
|
||||
- Running inside rein-aharness is plausible (stdlib only) but unverified.
|
||||
- Gaps against `INTENT.md` (source enforcement, replay, latest-period
|
||||
selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.
|
||||
- No CI and no licence yet (`WEBFETCH-WP-0003`).
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[project]
|
||||
name = "web-fetcher"
|
||||
version = "0.1.0"
|
||||
version = "0.2.0"
|
||||
description = "Provenance-first fetching of web and API data for research"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = []
|
||||
|
|
|
|||
|
|
@ -6,8 +6,9 @@ summary: >-
|
|||
Observation records carrying value, source, URL, source publication time,
|
||||
retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic
|
||||
JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found,
|
||||
never estimated. Optional content-addressed cache stores the exact bytes a
|
||||
run used; replay from it is not implemented yet.
|
||||
never estimated, and values without a source are refused. A
|
||||
content-addressed cache stores the exact bytes of a run and replays it
|
||||
offline. Tagged release v0.2.0, pip-installable from Forgejo.
|
||||
owner: web-fetcher
|
||||
status: draft
|
||||
domain: infotech
|
||||
|
|
@ -19,49 +20,48 @@ maturity:
|
|||
confidence: medium
|
||||
rationale: >
|
||||
Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md)
|
||||
and one consumer (assembly-economics reality checkpoints) drove the
|
||||
design. No survey of alternative fetching libraries in the estate yet.
|
||||
and checked against the code; one consumer (assembly-economics reality
|
||||
checkpoints) drove the design. No survey of alternative fetching
|
||||
libraries in the estate yet.
|
||||
availability:
|
||||
current: A2
|
||||
target: A3
|
||||
current: A3
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Installable source package (pyproject, CLI entry point, importable API)
|
||||
on Forgejo with offline fixture tests. Not published to a package index
|
||||
or as a versioned release artifact; consumers vendor via path or pip
|
||||
install from the repository.
|
||||
Versioned, tagged release v0.2.0 with changelog, installable with
|
||||
pip from the Forgejo tag, CLI and importable API. Not on a package
|
||||
index; the consumer still reads a checkout rather than the tag.
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
level: C2
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- Every value carries URL, retrieval time and payload hash.
|
||||
- Every ok value carries source, URL, retrieval time and payload hash; a spec without source yields error.
|
||||
- Source publication time from Eurostat metadata or HTTP Last-Modified when present.
|
||||
- Runs replay offline from the cache with identical observations.
|
||||
- Latest-period choice follows a parsed period order; mixed formats are refused and the rule is stated in the note.
|
||||
- Failures and missing values are explicit not_found/error records, never exceptions per spec.
|
||||
- Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources.
|
||||
- Eurostat, JSON, CSV and wide-XLSX adapters cover all 27 machine-readable specs of the first consumer.
|
||||
broken_expectations:
|
||||
- Runs cannot be replayed; the cache is write-only.
|
||||
- source is optional in specs, so HTTP adapters can return ok values with source null.
|
||||
- Source publication time is only captured by the Eurostat adapter.
|
||||
- Latest-period selection compares period strings and can pick the wrong row for mixed formats.
|
||||
- No pagination, authentication or rate-limit handling for APIs that need it.
|
||||
- No pagination or authentication for APIs that need it.
|
||||
- Only four adapters; new statistics APIs require code.
|
||||
out_of_scope_expectations:
|
||||
- Crawling, scraping behind logins, LLM-based extraction.
|
||||
- Scheduling (activity-core) and execution orchestration (rein-aharness).
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
level: R2
|
||||
confidence: medium
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- Retries cover every URLError, including permanent 4xx; 429/Retry-After is not honoured.
|
||||
- Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time.
|
||||
- Live endpoints are not exercised by the test suite.
|
||||
- Upstream API shape changes surface as parse errors at run time; the live smoke test is opt-in, not scheduled.
|
||||
- Single consumer so far.
|
||||
- Replay picks the most recent cache entry per URL; use one cache directory per run for exact reproduction.
|
||||
evidence:
|
||||
satisfied_signals:
|
||||
- 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures.
|
||||
- assembly-economics run_checkpoint.py consumes it for light checkpoint runs.
|
||||
- 18 offline unit tests (fixtures, fake getters, patched HTTP) pass.
|
||||
- Opt-in live smoke test passed for all four adapters on 2026-10-19.
|
||||
- All 31 assembly-economics checkpoint specs gave identical status/value/period under 0.1.0 and 0.2.0 on 2026-10-19; replay of the cached run was byte-identical.
|
||||
discovery:
|
||||
intent: >
|
||||
Give research repositories one small, trustworthy way to pull numbers from
|
||||
|
|
@ -69,9 +69,10 @@ discovery:
|
|||
each repo writing ad-hoc urllib or pandas download code.
|
||||
includes:
|
||||
- Declarative JSON fetch specs and a CLI emitting observations as JSONL.
|
||||
- Python API (run_spec, run_specs, Observation).
|
||||
- Python API (run_spec, run_specs, Observation, fetch_bytes, replay).
|
||||
- Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder.
|
||||
- Optional content-addressed payload cache (write-only; replay planned in WEBFETCH-WP-0002).
|
||||
- Content-addressed payload cache and offline replay.
|
||||
- Selective retry (connection errors, timeouts, HTTP 429/5xx, Retry-After).
|
||||
excludes:
|
||||
- Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping.
|
||||
- LLM extraction or estimation of missing values.
|
||||
|
|
@ -81,16 +82,18 @@ discovery:
|
|||
- Sources are public and reachable over plain HTTPS.
|
||||
use_cases:
|
||||
- Reality checkpoints comparing forecasts against official statistics.
|
||||
- Retaining the exact source bytes behind a research run for later audit.
|
||||
- Reproducible research runs replayed from cached source bytes.
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A3
|
||||
current_level: A3
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- https://forgejo.coulomb.social/coulomb/web-fetcher
|
||||
- https://forgejo.coulomb.social/coulomb/web-fetcher/src/tag/v0.2.0
|
||||
- git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0
|
||||
- CHANGELOG.md
|
||||
- web_fetcher/__main__.py
|
||||
- web_fetcher/registry.py
|
||||
target_artifacts:
|
||||
- Tagged, versioned release installable with pip from Forgejo.
|
||||
- Consumers pinned to release tags; published to a package index if a second consumer needs it.
|
||||
consumption_modes: [Python library, cli, source]
|
||||
relations:
|
||||
depends_on: []
|
||||
|
|
@ -99,39 +102,64 @@ relations:
|
|||
evidence:
|
||||
documentation:
|
||||
- README.md
|
||||
- CHANGELOG.md
|
||||
- INTENT.md
|
||||
- SCOPE.md
|
||||
tests:
|
||||
- tests/test_web_fetcher.py
|
||||
- tests/test_live.py
|
||||
consumer_feedback: []
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos.
|
||||
- Fetching official statistics (Eurostat, FRED CSV, World Bank/WHO JSON, Census XLSX) with provenance for research repos.
|
||||
- Runs inside constrained executors where installing dependencies is undesirable.
|
||||
- Audit or reproduction of a past run from its cached bytes.
|
||||
not_recommended_for:
|
||||
- JavaScript-rendered pages or sites requiring login.
|
||||
- High-volume crawling or bulk mirroring.
|
||||
known_limitations:
|
||||
- No auth or pagination; retries are two fixed linear attempts on any network/HTTP error.
|
||||
- Cache is write-only; replay is planned in WEBFETCH-WP-0002.
|
||||
- Supply "source" in every spec; it is not enforced yet.
|
||||
promotion_history: []
|
||||
- No auth or pagination.
|
||||
- Every non-Eurostat spec must set "source"; otherwise values become error.
|
||||
- Mixed period label formats need an explicit period_format.
|
||||
promotion_history:
|
||||
- date: "2026-10-19"
|
||||
dimension: availability
|
||||
from: A2
|
||||
to: A3
|
||||
rationale: >-
|
||||
Tagged release v0.2.0 with changelog and pip install from the Forgejo tag
|
||||
(WEBFETCH-WP-0002-T03).
|
||||
author: claude-code
|
||||
- date: "2026-10-19"
|
||||
dimension: completeness
|
||||
from: C1
|
||||
to: C2
|
||||
rationale: >-
|
||||
Replay, source enforcement, Last-Modified capture and a stated
|
||||
latest-period rule close the INTENT gaps found in the scope review.
|
||||
author: claude-code
|
||||
- date: "2026-10-19"
|
||||
dimension: reliability
|
||||
from: R1
|
||||
to: R2
|
||||
rationale: >-
|
||||
Live smoke test across all adapters and a full consumer regression run
|
||||
(31 specs, identical results, byte-identical replay).
|
||||
author: claude-code
|
||||
---
|
||||
|
||||
# Provenance-first Web and API Data Fetching
|
||||
|
||||
`web-fetcher` runs declarative fetch specs and returns `Observation` records
|
||||
with the value **and** its provenance. Values are never estimated: each spec
|
||||
yields a value or an explicit `not_found` / `error` record. Gaps against the
|
||||
repo's intent (replay, source enforcement, latest-period rule, selective retry)
|
||||
are tracked in `WEBFETCH-WP-0002`.
|
||||
yields a sourced value or an explicit `not_found` / `error` record.
|
||||
|
||||
## Try it
|
||||
|
||||
```bash
|
||||
git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher
|
||||
python3 -B -m unittest discover -s tests # offline fixtures
|
||||
python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl
|
||||
pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"
|
||||
python3 -m web_fetcher fetch specs.json --cache run-01 > observations.jsonl
|
||||
python3 -m web_fetcher fetch specs.json --replay run-01 > replayed.jsonl # offline
|
||||
```
|
||||
|
||||
```python
|
||||
|
|
@ -143,4 +171,4 @@ obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
|||
|
||||
New sources are added as adapters in `web_fetcher/adapters/` with offline
|
||||
fixture tests. Consumer reference: assembly-economics
|
||||
`checkpoints/run_checkpoint.py`.
|
||||
`checkpoints/run_checkpoint.py`. Release notes: `CHANGELOG.md`.
|
||||
|
|
|
|||
|
|
@ -7,8 +7,9 @@ capabilities:
|
|||
summary: Standard-library Python fetcher that turns declarative JSON specs into Observation
|
||||
records carrying value, source, URL, source publication time, retrieval time and payload
|
||||
SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing
|
||||
data is an explicit not_found, never estimated. Optional content-addressed cache stores
|
||||
the exact bytes a run used; replay from it is not implemented yet.
|
||||
data is an explicit not_found, never estimated, and values without a source are refused.
|
||||
A content-addressed cache stores the exact bytes of a run and replays it offline. Tagged
|
||||
release v0.2.0, pip-installable from Forgejo.
|
||||
domain: infotech
|
||||
status: draft
|
||||
owner: web-fetcher
|
||||
|
|
@ -27,7 +28,7 @@ capabilities:
|
|||
- replay
|
||||
- data-ingestion
|
||||
- stdlib
|
||||
vector: D2 / A2 / C1 / R1
|
||||
vector: D2 / A3 / C2 / R2
|
||||
path: registry/capabilities/capability.infotech.provenance-fetch.md
|
||||
consumption_modes:
|
||||
- Python library
|
||||
|
|
|
|||
44
tests/test_live.py
Normal file
44
tests/test_live.py
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
"""Live smoke tests against real endpoints, one spec per adapter.
|
||||
|
||||
Skipped unless WEB_FETCHER_LIVE=1. They detect upstream shape changes that the
|
||||
recorded fixtures cannot; a failure here means a source changed, not
|
||||
necessarily that web-fetcher is broken.
|
||||
|
||||
WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v
|
||||
"""
|
||||
|
||||
import os
|
||||
import unittest
|
||||
|
||||
from web_fetcher.registry import run_spec
|
||||
|
||||
SPECS = [
|
||||
{"id": "live-eurostat", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
||||
"filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10", "indic_is": "E_AI_TANY"},
|
||||
"split_by": ["geo"]},
|
||||
{"id": "live-json", "adapter": "http_json",
|
||||
"url": "https://api.worldbank.org/v2/country/DEU/indicator/SE.XPD.TOTL.GD.ZS?format=json&mrnev=1",
|
||||
"path": "1.0.value", "period_path": "1.0.date", "source": "World Bank WDI (SE.XPD.TOTL.GD.ZS)"},
|
||||
{"id": "live-csv", "adapter": "http_csv", "url": "https://fred.stlouisfed.org/graph/fredgraph.csv?id=OPHNFB",
|
||||
"value_column": "OPHNFB", "period_column": "observation_date", "source": "BLS via FRED (OPHNFB)"},
|
||||
{"id": "live-xlsx", "adapter": "xlsx_wide", "url": "https://www.census.gov/hfp/btos/downloads/National.xlsx",
|
||||
"sheet": "Response Estimates", "filter": {"Question ID": "7", "Answer ID": "1"},
|
||||
"source": "U.S. Census Bureau BTOS"},
|
||||
]
|
||||
|
||||
|
||||
@unittest.skipUnless(os.environ.get("WEB_FETCHER_LIVE") == "1", "set WEB_FETCHER_LIVE=1 for live tests")
|
||||
class TestLive(unittest.TestCase):
|
||||
def test_each_adapter_returns_a_sourced_value(self):
|
||||
for spec in SPECS:
|
||||
with self.subTest(spec["id"]):
|
||||
obs = run_spec(spec)
|
||||
self.assertTrue(obs, "no observations")
|
||||
o = obs[0]
|
||||
self.assertEqual(o.status, "ok", o.note)
|
||||
self.assertIsNotNone(o.value)
|
||||
self.assertTrue(o.source and o.url and o.retrieved_at and o.payload_sha256)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -41,7 +41,7 @@ class TestXlsx(unittest.TestCase):
|
|||
body = (FIX / "btos_national.xlsx").read_bytes()
|
||||
obs = run_spec({"id": "us_ai_use", "adapter": "xlsx_wide", "url": "https://x.test/National.xlsx",
|
||||
"sheet": "Response Estimates", "filter": {"Question ID": "7", "Answer ID": "1"},
|
||||
"unit": "% of firms"}, get=fake(body))
|
||||
"unit": "% of firms", "source": "Census BTOS"}, get=fake(body))
|
||||
self.assertEqual(obs[0].status, "ok")
|
||||
self.assertEqual(obs[0].period, "202620")
|
||||
self.assertAlmostEqual(obs[0].value, 23.8)
|
||||
|
|
@ -51,7 +51,7 @@ class TestHttp(unittest.TestCase):
|
|||
def test_json_path(self):
|
||||
body = json.dumps({"data": {"rows": [{"v": 7.5, "p": "2026-Q3"}]}}).encode()
|
||||
obs = run_spec({"id": "j", "adapter": "http_json", "url": "https://x.test", "path": "data.rows.0.v",
|
||||
"period_path": "data.rows.0.p", "unit": "%"}, get=fake(body))
|
||||
"period_path": "data.rows.0.p", "unit": "%", "source": "test"}, get=fake(body))
|
||||
self.assertEqual((obs[0].status, obs[0].value, obs[0].period), ("ok", 7.5, "2026-Q3"))
|
||||
|
||||
def test_json_missing_path_is_not_found(self):
|
||||
|
|
@ -62,7 +62,8 @@ class TestHttp(unittest.TestCase):
|
|||
def test_csv_latest_matching_row(self):
|
||||
body = b"geo,period,value\nUS,2026-08,8.1\nUS,2026-09,8.7\nDE,2026-09,9.9\n"
|
||||
obs = run_spec({"id": "c", "adapter": "http_csv", "url": "https://x.test", "filter": {"geo": "US"},
|
||||
"value_column": "value", "period_column": "period"}, get=fake(body))
|
||||
"value_column": "value", "period_column": "period", "source": "test"},
|
||||
get=fake(body))
|
||||
self.assertEqual((obs[0].value, obs[0].period), (8.7, "2026-09"))
|
||||
|
||||
def test_manual_and_unknown_adapters_never_raise(self):
|
||||
|
|
@ -80,3 +81,120 @@ class TestHttp(unittest.TestCase):
|
|||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, body: bytes, headers: dict[str, str] | None = None):
|
||||
self.body, self.headers = body, headers or {}
|
||||
|
||||
def read(self) -> bytes:
|
||||
return self.body
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
def http_error(code: int, headers: dict[str, str] | None = None):
|
||||
import email.message
|
||||
import urllib.error
|
||||
msg = email.message.Message()
|
||||
for k, v in (headers or {}).items():
|
||||
msg[k] = v
|
||||
return urllib.error.HTTPError("https://x.test", code, "err", msg, None)
|
||||
|
||||
|
||||
class TestFetchBytes(unittest.TestCase):
|
||||
def run_with(self, responses, **kw):
|
||||
from unittest import mock
|
||||
|
||||
from web_fetcher import core
|
||||
calls, sleeps = [], []
|
||||
|
||||
def opener(req, timeout):
|
||||
calls.append(req.full_url)
|
||||
r = responses.pop(0)
|
||||
if isinstance(r, Exception):
|
||||
raise r
|
||||
return r
|
||||
with mock.patch.object(core, "_urlopen", opener), mock.patch.object(core, "_sleep", sleeps.append):
|
||||
try:
|
||||
return core.fetch_bytes("https://x.test/a", **kw), calls, sleeps
|
||||
except core.FetchError as exc:
|
||||
return exc, calls, sleeps
|
||||
|
||||
def test_404_fails_fast(self):
|
||||
res, calls, sleeps = self.run_with([http_error(404)])
|
||||
self.assertIn("HTTP 404", str(res))
|
||||
self.assertEqual((len(calls), sleeps), (1, []))
|
||||
|
||||
def test_503_honours_retry_after(self):
|
||||
res, calls, sleeps = self.run_with([http_error(503, {"Retry-After": "7"}), FakeResponse(b"ok")])
|
||||
self.assertEqual((res.body, len(calls), sleeps), (b"ok", 2, [7.0]))
|
||||
|
||||
def test_connection_errors_back_off_exponentially(self):
|
||||
import urllib.error
|
||||
res, calls, sleeps = self.run_with([urllib.error.URLError("down")] * 3)
|
||||
self.assertEqual((len(calls), sleeps), (3, [1.5, 3.0]))
|
||||
self.assertIn("down", str(res))
|
||||
|
||||
def test_last_modified_becomes_published(self):
|
||||
res, _, _ = self.run_with([FakeResponse(b"{}", {"Last-Modified": "Wed, 01 Oct 2026 08:00:00 GMT"})])
|
||||
self.assertEqual(res.published, "2026-10-01T08:00:00+00:00")
|
||||
|
||||
|
||||
class TestReplay(unittest.TestCase):
|
||||
def test_replay_reproduces_cached_run(self):
|
||||
import tempfile
|
||||
from unittest import mock
|
||||
|
||||
from web_fetcher import core
|
||||
body = json.dumps({"v": 3.2}).encode()
|
||||
spec = {"id": "r", "adapter": "http_json", "url": "https://x.test/r", "path": "v", "source": "test"}
|
||||
with tempfile.TemporaryDirectory() as cache:
|
||||
with mock.patch.object(core, "_urlopen", lambda req, timeout: FakeResponse(body)):
|
||||
first = run_spec(spec, cache_dir=cache)
|
||||
|
||||
def no_network(req, timeout):
|
||||
raise AssertionError("network used during replay")
|
||||
with mock.patch.object(core, "_urlopen", no_network):
|
||||
again = run_spec(spec, replay_dir=cache)
|
||||
missing = run_spec({**spec, "url": "https://x.test/other"}, replay_dir=cache)
|
||||
self.assertEqual([o.to_dict() for o in again], [o.to_dict() for o in first])
|
||||
self.assertEqual(first[0].value, 3.2)
|
||||
self.assertEqual(missing[0].status, "error")
|
||||
self.assertIn("not in replay cache", missing[0].note)
|
||||
|
||||
|
||||
class TestProvenanceRules(unittest.TestCase):
|
||||
def test_value_without_source_is_error(self):
|
||||
obs = run_spec({"id": "s", "adapter": "http_json", "url": "https://x.test", "path": "v"},
|
||||
get=fake(b'{"v": 1}'))
|
||||
self.assertEqual(obs[0].status, "error")
|
||||
self.assertIn("source", obs[0].note)
|
||||
|
||||
def csv(self, body: bytes, **extra):
|
||||
spec = {"id": "c", "adapter": "http_csv", "url": "https://x.test", "value_column": "value",
|
||||
"period_column": "period", "source": "test", **extra}
|
||||
return run_spec(spec, get=fake(body))[0]
|
||||
|
||||
def test_quarters_ordered_by_value_not_string(self):
|
||||
o = self.csv(b"period,value\n2025-Q4,1\n2026Q1,2\n")
|
||||
self.assertEqual((o.status, o.period, o.value), ("ok", "2026Q1", 2.0))
|
||||
self.assertEqual(o.note, "latest of 2 periods by quarter order")
|
||||
|
||||
def test_mixed_period_formats_are_error(self):
|
||||
o = self.csv(b"period,value\n2026-Q1,1\n2026-02,2\n")
|
||||
self.assertEqual(o.status, "error")
|
||||
self.assertIn("mixed period formats", o.note)
|
||||
|
||||
def test_explicit_string_order_is_allowed(self):
|
||||
o = self.csv(b"period,value\nb,1\na,2\n", period_format="string")
|
||||
self.assertEqual((o.status, o.period), ("ok", "b"))
|
||||
|
||||
def test_non_numeric_value_is_not_found_unless_string(self):
|
||||
body = b"period,value\n2026-01,(D)\n"
|
||||
self.assertEqual(self.csv(body).status, "not_found")
|
||||
self.assertEqual(self.csv(body, value_type="string").value, "(D)")
|
||||
|
|
|
|||
|
|
@ -10,4 +10,4 @@ from web_fetcher.core import Observation, fetch_bytes
|
|||
from web_fetcher.registry import ADAPTERS, run_spec, run_specs
|
||||
|
||||
__all__ = ["Observation", "fetch_bytes", "ADAPTERS", "run_spec", "run_specs"]
|
||||
__version__ = "0.1.0"
|
||||
__version__ = "0.2.0"
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
"""CLI: python -m web_fetcher fetch SPECS.json [--cache DIR] [--out FILE]
|
||||
"""CLI: python -m web_fetcher fetch SPECS.json [--cache DIR | --replay DIR] [--out FILE]
|
||||
|
||||
SPECS.json is a JSON list of spec objects (or {"specs": [...]}). Output is
|
||||
JSON Lines, one Observation per line.
|
||||
JSON Lines, one Observation per line. --replay serves every URL from a cache
|
||||
written by an earlier --cache run, without network access.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -18,7 +19,9 @@ def main(argv: list[str] | None = None) -> int:
|
|||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
f = sub.add_parser("fetch", help="run fetch specs")
|
||||
f.add_argument("specs")
|
||||
f.add_argument("--cache", default=None, help="content-addressed cache directory")
|
||||
src = f.add_mutually_exclusive_group()
|
||||
src.add_argument("--cache", default=None, help="content-addressed cache directory")
|
||||
src.add_argument("--replay", default=None, help="replay from a cache directory; no network")
|
||||
f.add_argument("--out", default="-", help="output JSONL path (default stdout)")
|
||||
f.add_argument("--only", action="append", default=[], help="run only these spec ids")
|
||||
args = ap.parse_args(argv)
|
||||
|
|
@ -27,7 +30,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
specs = data["specs"] if isinstance(data, dict) else data
|
||||
if args.only:
|
||||
specs = [s for s in specs if s.get("id") in set(args.only)]
|
||||
obs = run_specs(specs, cache_dir=args.cache)
|
||||
obs = run_specs(specs, cache_dir=args.cache, replay_dir=args.replay)
|
||||
lines = "\n".join(o.to_json() for o in obs) + "\n"
|
||||
if args.out == "-":
|
||||
sys.stdout.write(lines)
|
||||
|
|
|
|||
|
|
@ -2,10 +2,17 @@ r"""Generic HTTP adapters: JSON (dotted path) and CSV (filter + pick latest).
|
|||
|
||||
json spec fields: url, path ("a.b.0.c"), unit?, period_path?
|
||||
csv spec fields: url, filter (dict column -> value), value_column,
|
||||
period_column?, unit?, delimiter? -> latest row by period
|
||||
period_column?, period_format?, value_type?, unit?, delimiter?
|
||||
-> latest row by period
|
||||
xlsx_wide fields: url, sheet, filter (dict header -> value), period_regex
|
||||
(header pattern of period columns, default ^\d{6}$), unit?
|
||||
(header pattern of period columns, default ^\d{6}$),
|
||||
period_format?, value_type?, unit?
|
||||
-> value of the matching row in the latest period column
|
||||
|
||||
All three take `source`; a value without one is turned into an error by the
|
||||
registry. "Latest" follows `web_fetcher.periods` (period_format, default auto)
|
||||
and the chosen rule is stated in the observation's note. Values must be
|
||||
numeric unless value_type is "string"; otherwise the result is not_found.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -16,6 +23,20 @@ import json
|
|||
from typing import Any, Callable
|
||||
|
||||
from web_fetcher.core import Fetched, Observation
|
||||
from web_fetcher.periods import order
|
||||
|
||||
MISSING = ("", "NA", "N/A", "-", ".", "S")
|
||||
|
||||
|
||||
def _value(raw: str, spec: dict[str, Any]) -> tuple[Any, str | None]:
|
||||
"""Parse a cell; returns (value, None) or (None, reason) for non-numeric data."""
|
||||
text = raw.strip()
|
||||
if spec.get("value_type") == "string":
|
||||
return text, None
|
||||
try:
|
||||
return float(text.replace("%", "").replace(",", "")), None
|
||||
except ValueError:
|
||||
return None, f"non-numeric value {text!r} (set value_type: string to accept)"
|
||||
|
||||
|
||||
def _dig(obj: Any, path: str) -> Any:
|
||||
|
|
@ -40,7 +61,8 @@ def fetch_json(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Obse
|
|||
status = "ok" if value is not None else "not_found"
|
||||
return [Observation(spec_id=spec["id"], status=status, value=value, unit=spec.get("unit"),
|
||||
period=None if period is None else str(period), source=spec.get("source"),
|
||||
url=f.url, retrieved_at=f.retrieved_at, payload_sha256=f.sha256,
|
||||
url=f.url, published=f.published, retrieved_at=f.retrieved_at,
|
||||
payload_sha256=f.sha256,
|
||||
note=None if value is not None else f"path {spec['path']} not found")]
|
||||
|
||||
|
||||
|
|
@ -51,20 +73,26 @@ def fetch_csv(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Obser
|
|||
flt = spec.get("filter") or {}
|
||||
rows = [r for r in rows if all(str(r.get(k, "")).strip() == str(v) for k, v in flt.items())]
|
||||
pcol, vcol = spec.get("period_column"), spec["value_column"]
|
||||
rows = [r for r in rows if str(r.get(vcol, "")).strip() not in ("", "NA", "N/A", "-")]
|
||||
rows = [r for r in rows if str(r.get(vcol) or "").strip() not in MISSING]
|
||||
base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at,
|
||||
payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"),
|
||||
labels={k: str(v) for k, v in flt.items()})
|
||||
if not rows:
|
||||
return [Observation(spec_id=spec["id"], status="not_found", url=f.url, retrieved_at=f.retrieved_at,
|
||||
payload_sha256=f.sha256, source=spec.get("source"), note="no matching row")]
|
||||
row = max(rows, key=lambda r: str(r.get(pcol, ""))) if pcol else rows[-1]
|
||||
raw = str(row[vcol]).strip().replace("%", "")
|
||||
try:
|
||||
value: Any = float(raw)
|
||||
except ValueError:
|
||||
value = raw
|
||||
return [Observation(spec_id=spec["id"], status="ok", value=value, unit=spec.get("unit"),
|
||||
period=str(row.get(pcol)) if pcol else None, source=spec.get("source"),
|
||||
labels={k: str(v) for k, v in flt.items()}, url=f.url,
|
||||
retrieved_at=f.retrieved_at, payload_sha256=f.sha256)]
|
||||
return [Observation(status="not_found", note="no matching row", **base)]
|
||||
if pcol:
|
||||
fmt, newest = order(sorted({str(r.get(pcol) or "").strip() for r in rows}), spec.get("period_format", "auto"))
|
||||
period = newest[0]
|
||||
latest = [r for r in rows if str(r.get(pcol) or "").strip() == period]
|
||||
if len({str(r[vcol]).strip() for r in latest}) > 1:
|
||||
raise ValueError(f"{len(latest)} matching rows with different values for period {period}")
|
||||
row, rule = latest[-1], f"latest of {len(newest)} periods by {fmt} order"
|
||||
else:
|
||||
period, row = None, rows[-1]
|
||||
rule = f"no period_column: last of {len(rows)} matching rows"
|
||||
value, why = _value(str(row[vcol]), spec)
|
||||
if why:
|
||||
return [Observation(status="not_found", period=period, note=f"{why}; {rule}", **base)]
|
||||
return [Observation(status="ok", value=value, period=period, note=rule, **base)]
|
||||
|
||||
|
||||
def fetch_xlsx_wide(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]:
|
||||
|
|
@ -76,22 +104,30 @@ def fetch_xlsx_wide(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list
|
|||
rows = read_sheet(f.body, spec.get("sheet", 0))
|
||||
header, body = rows[0], rows[1:]
|
||||
pat = re.compile(spec.get("period_regex", r"^\d{6}$"))
|
||||
periods = [(i, h) for i, h in enumerate(header) if pat.match(h or "")]
|
||||
periods = {(h or "").strip(): i for i, h in enumerate(header) if pat.match(h or "")}
|
||||
flt = spec.get("filter") or {}
|
||||
col = {h: i for i, h in enumerate(header)}
|
||||
match = [r for r in body if all(i < len(r) and r[i].strip() == str(v) for h, v in flt.items()
|
||||
for i in [col.get(h, 10**9)])]
|
||||
base = dict(spec_id=spec["id"], url=f.url, retrieved_at=f.retrieved_at, payload_sha256=f.sha256,
|
||||
source=spec.get("source"), labels={k: str(v) for k, v in flt.items()})
|
||||
base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at,
|
||||
payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"),
|
||||
labels={k: str(v) for k, v in flt.items()})
|
||||
if not match or not periods:
|
||||
return [Observation(status="not_found", note="no matching row or period columns", **base)]
|
||||
if len(match) > 1:
|
||||
raise ValueError(f"filter matches {len(match)} rows; expected one")
|
||||
row = match[0]
|
||||
for i, h in sorted(periods, key=lambda p: p[1], reverse=True):
|
||||
raw = (row[i] if i < len(row) else "").strip().replace("%", "")
|
||||
if raw not in ("", "S", "NA", "-"):
|
||||
try:
|
||||
value: Any = float(raw)
|
||||
except ValueError:
|
||||
value = raw
|
||||
return [Observation(status="ok", value=value, unit=spec.get("unit"), period=h, **base)]
|
||||
fmt, newest = order(list(periods), spec.get("period_format", "auto"))
|
||||
for n, h in enumerate(newest):
|
||||
i = periods[h]
|
||||
raw = (row[i] if i < len(row) else "").strip()
|
||||
if raw in MISSING:
|
||||
continue
|
||||
rule = f"latest of {len(newest)} periods by {fmt} order"
|
||||
if n:
|
||||
rule += f", skipping {n} newer period(s) without a value"
|
||||
value, why = _value(raw, spec)
|
||||
if why:
|
||||
return [Observation(status="not_found", period=h, note=f"{why}; {rule}", **base)]
|
||||
return [Observation(status="ok", value=value, period=h, note=rule, **base)]
|
||||
return [Observation(status="not_found", note="matching row has no value", **base)]
|
||||
|
|
|
|||
|
|
@ -10,10 +10,11 @@ import urllib.error
|
|||
import urllib.request
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from email.utils import parsedate_to_datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
USER_AGENT = "web-fetcher/0.1 (+https://forgejo.coulomb.social/coulomb/web-fetcher)"
|
||||
USER_AGENT = "web-fetcher/0.2 (+https://forgejo.coulomb.social/coulomb/web-fetcher)"
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -47,47 +48,128 @@ class Fetched:
|
|||
retrieved_at: str
|
||||
sha256: str
|
||||
content_type: str | None
|
||||
published: str | None = None # HTTP Last-Modified, ISO 8601, when the server sends it
|
||||
|
||||
|
||||
class FetchError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
RETRY_STATUS = {429, 500, 502, 503, 504}
|
||||
MAX_RETRY_AFTER = 120.0
|
||||
|
||||
# Indirection points for offline tests.
|
||||
_urlopen = urllib.request.urlopen
|
||||
_sleep = time.sleep
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _http_date(value: str | None) -> str | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return parsedate_to_datetime(value).astimezone(timezone.utc).isoformat(timespec="seconds")
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _retry_after(value: str | None) -> float | None:
|
||||
"""Seconds to wait from a Retry-After header (delta-seconds or HTTP date)."""
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return max(0.0, float(value))
|
||||
except ValueError:
|
||||
when = _http_date(value)
|
||||
if when is None:
|
||||
return None
|
||||
return max(0.0, (datetime.fromisoformat(when) - datetime.now(timezone.utc)).total_seconds())
|
||||
|
||||
|
||||
def fetch_bytes(url: str, *, timeout: float = 60.0, retries: int = 2,
|
||||
cache_dir: str | os.PathLike | None = None,
|
||||
replay_dir: str | os.PathLike | None = None,
|
||||
headers: dict[str, str] | None = None) -> Fetched:
|
||||
"""GET `url` and return body plus provenance. Optional content-addressed cache.
|
||||
|
||||
The cache stores each payload under its SHA-256 and an index entry per URL,
|
||||
so a research run can be replayed offline from the exact bytes it used.
|
||||
The cache stores each payload under its SHA-256 and an index entry per URL.
|
||||
With `replay_dir`, no network request is made: the payload and its original
|
||||
provenance are served from such a cache (see `replay_bytes`).
|
||||
|
||||
Only transient failures are retried (connection errors, timeouts, HTTP
|
||||
429/5xx), with exponential backoff or the server's Retry-After; other HTTP
|
||||
errors fail immediately. Provenance describes the attempt that succeeded.
|
||||
"""
|
||||
if replay_dir:
|
||||
return replay_bytes(url, replay_dir)
|
||||
hdrs = {"User-Agent": USER_AGENT, **(headers or {})}
|
||||
last: Exception | None = None
|
||||
for attempt in range(retries + 1):
|
||||
wait = 1.5 * 2 ** attempt
|
||||
try:
|
||||
req = urllib.request.Request(url, headers=hdrs)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
with _urlopen(req, timeout=timeout) as resp:
|
||||
body = resp.read()
|
||||
ctype = resp.headers.get("Content-Type")
|
||||
modified = _http_date(resp.headers.get("Last-Modified"))
|
||||
digest = hashlib.sha256(body).hexdigest()
|
||||
fetched = Fetched(url=url, body=body, retrieved_at=_now(), sha256=digest, content_type=ctype)
|
||||
fetched = Fetched(url=url, body=body, retrieved_at=_now(), sha256=digest,
|
||||
content_type=ctype, published=modified)
|
||||
if cache_dir:
|
||||
_cache_store(Path(cache_dir), fetched)
|
||||
return fetched
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code not in RETRY_STATUS:
|
||||
raise FetchError(f"fetch failed for {url}: HTTP {exc.code} {exc.reason}") from exc
|
||||
last = exc
|
||||
hinted = _retry_after(exc.headers.get("Retry-After") if exc.headers else None)
|
||||
if hinted is not None:
|
||||
wait = min(hinted, MAX_RETRY_AFTER)
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||
last = exc
|
||||
if attempt < retries:
|
||||
time.sleep(1.5 * (attempt + 1))
|
||||
if attempt < retries:
|
||||
_sleep(wait)
|
||||
raise FetchError(f"fetch failed for {url}: {last}")
|
||||
|
||||
|
||||
def replay_bytes(url: str, cache_dir: str | os.PathLike) -> Fetched:
|
||||
"""Serve `url` from a cache written by `fetch_bytes(cache_dir=...)`, without network.
|
||||
|
||||
Uses the most recent index entry for the URL and verifies the payload hash.
|
||||
Returns the original retrieval time, so a replayed run reproduces the
|
||||
observations of the run that filled the cache.
|
||||
"""
|
||||
cache = Path(cache_dir)
|
||||
entry: dict[str, Any] | None = None
|
||||
try:
|
||||
with (cache / "index.jsonl").open(encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
if line.strip():
|
||||
rec = json.loads(line)
|
||||
if rec.get("url") == url:
|
||||
entry = rec
|
||||
except FileNotFoundError:
|
||||
raise FetchError(f"replay cache {cache} has no index.jsonl") from None
|
||||
if entry is None:
|
||||
raise FetchError(f"not in replay cache: {url}")
|
||||
try:
|
||||
body = (cache / f"{entry['sha256']}.bin").read_bytes()
|
||||
except FileNotFoundError:
|
||||
raise FetchError(f"replay payload missing for {url}: {entry['sha256']}") from None
|
||||
if hashlib.sha256(body).hexdigest() != entry["sha256"]:
|
||||
raise FetchError(f"replay payload hash mismatch for {url}")
|
||||
return Fetched(url=url, body=body, retrieved_at=entry["retrieved_at"], sha256=entry["sha256"],
|
||||
content_type=entry.get("content_type"), published=entry.get("published"))
|
||||
|
||||
|
||||
def _cache_store(cache: Path, fetched: Fetched) -> None:
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
(cache / f"{fetched.sha256}.bin").write_bytes(fetched.body)
|
||||
with (cache / "index.jsonl").open("a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps({"url": fetched.url, "sha256": fetched.sha256,
|
||||
"retrieved_at": fetched.retrieved_at}) + "\n")
|
||||
"retrieved_at": fetched.retrieved_at,
|
||||
"content_type": fetched.content_type,
|
||||
"published": fetched.published}) + "\n")
|
||||
|
|
|
|||
62
web_fetcher/periods.py
Normal file
62
web_fetcher/periods.py
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
"""Period ordering for "latest value" selection.
|
||||
|
||||
A source's period labels are only compared when they share one recognised
|
||||
format; mixed or unrecognised formats are an error, never a silent string
|
||||
comparison. Spec field `period_format` pins the format ("auto" by default);
|
||||
"string" opts in to plain lexicographic order.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Callable
|
||||
|
||||
_FORMATS: dict[str, tuple[re.Pattern[str], Callable[[re.Match[str]], tuple[int, ...]]]] = {
|
||||
"year": (re.compile(r"^(\d{4})$"), lambda m: (int(m[1]),)),
|
||||
"month": (re.compile(r"^(\d{4})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]))),
|
||||
"date": (re.compile(r"^(\d{4})-(\d{2})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]), int(m[3]))),
|
||||
"quarter": (re.compile(r"^(\d{4})-?Q([1-4])$"), lambda m: (int(m[1]), int(m[2]))),
|
||||
"semester": (re.compile(r"^(\d{4})-?S([12])$"), lambda m: (int(m[1]), int(m[2]))),
|
||||
"week": (re.compile(r"^(\d{4})-?W(\d{2})$"), lambda m: (int(m[1]), int(m[2]))),
|
||||
# Fixed-width digit labels such as YYYYMM or Census BTOS YYYYPP.
|
||||
"digits": (re.compile(r"^(\d+)$"), lambda m: (len(m[1]), int(m[1]))),
|
||||
}
|
||||
|
||||
|
||||
class PeriodError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def detect(label: str) -> str | None:
|
||||
label = label.strip()
|
||||
for name, (pat, _) in _FORMATS.items():
|
||||
if pat.match(label):
|
||||
return name
|
||||
return None
|
||||
|
||||
|
||||
def sort_key(label: str, fmt: str) -> tuple[int, ...] | str:
|
||||
if fmt == "string":
|
||||
return label
|
||||
pat, key = _FORMATS[fmt]
|
||||
m = pat.match(label.strip())
|
||||
if m is None:
|
||||
raise PeriodError(f"period {label!r} does not match format {fmt!r}")
|
||||
return key(m)
|
||||
|
||||
|
||||
def order(labels: list[str], fmt: str = "auto") -> tuple[str, list[str]]:
|
||||
"""Return (format, labels newest first). Raises PeriodError for mixed formats."""
|
||||
if fmt == "auto":
|
||||
found = {detect(lbl) for lbl in labels}
|
||||
if None in found:
|
||||
bad = [lbl for lbl in labels if detect(lbl) is None][:3]
|
||||
raise PeriodError(f"unrecognised period labels {bad}; set period_format")
|
||||
if len(found) > 1:
|
||||
raise PeriodError(f"mixed period formats {sorted(found)}; set period_format")
|
||||
fmt = found.pop() if found else "string"
|
||||
if fmt == "digits" and len({len(lbl.strip()) for lbl in labels}) > 1:
|
||||
raise PeriodError("digit period labels of different widths; set period_format")
|
||||
elif fmt != "string" and fmt not in _FORMATS:
|
||||
raise PeriodError(f"unknown period_format {fmt!r}")
|
||||
return fmt, sorted(labels, key=lambda lbl: sort_key(lbl, fmt), reverse=True)
|
||||
|
|
@ -16,8 +16,12 @@ ADAPTERS: dict[str, Callable[[dict[str, Any], Callable[[str], Fetched]], list[Ob
|
|||
|
||||
|
||||
def run_spec(spec: dict[str, Any], get: Callable[[str], Fetched] | None = None,
|
||||
cache_dir: str | None = None) -> list[Observation]:
|
||||
"""Run one spec. Adapter errors become an `error` observation, not an exception."""
|
||||
cache_dir: str | None = None, replay_dir: str | None = None) -> list[Observation]:
|
||||
"""Run one spec. Adapter errors become an `error` observation, not an exception.
|
||||
|
||||
A value is never returned without a source: an `ok` observation whose
|
||||
adapter could not name one (the spec lacks `source`) becomes an `error`.
|
||||
"""
|
||||
sid = spec.get("id", "?")
|
||||
kind = spec.get("adapter")
|
||||
if kind == "manual":
|
||||
|
|
@ -26,15 +30,23 @@ def run_spec(spec: dict[str, Any], get: Callable[[str], Fetched] | None = None,
|
|||
adapter = ADAPTERS.get(kind or "")
|
||||
if adapter is None:
|
||||
return [Observation(spec_id=sid, status="error", note=f"unknown adapter {kind!r}")]
|
||||
getter = get or (lambda url: fetch_bytes(url, cache_dir=cache_dir))
|
||||
getter = get or (lambda url: fetch_bytes(url, cache_dir=cache_dir, replay_dir=replay_dir))
|
||||
try:
|
||||
return adapter(spec, getter)
|
||||
return [_require_source(o) for o in adapter(spec, getter)]
|
||||
except FetchError as exc:
|
||||
return [Observation(spec_id=sid, status="error", note=str(exc)[:300])]
|
||||
except (KeyError, ValueError, TypeError) as exc:
|
||||
return [Observation(spec_id=sid, status="error", note=f"parse error: {exc!s}"[:300])]
|
||||
|
||||
|
||||
def _require_source(obs: Observation) -> Observation:
|
||||
if obs.status == "ok" and not obs.source:
|
||||
return Observation(spec_id=obs.spec_id, status="error", url=obs.url, retrieved_at=obs.retrieved_at,
|
||||
payload_sha256=obs.payload_sha256,
|
||||
note="spec has no 'source'; values are never returned without one")
|
||||
return obs
|
||||
|
||||
|
||||
def run_specs(specs: list[dict[str, Any]], **kw: Any) -> list[Observation]:
|
||||
out: list[Observation] = []
|
||||
for spec in specs:
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Versioned release and fetch hardening"
|
||||
domain: infotech
|
||||
repo: web-fetcher
|
||||
status: proposed
|
||||
status: finished
|
||||
flavor: implementation
|
||||
depends_on: [WEBFETCH-WP-0001]
|
||||
owner: codex
|
||||
|
|
@ -25,7 +25,7 @@ assessment. No third-party dependencies.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "e64ddb81-3d56-5e09-9744-5c0a7c26005b"
|
||||
```
|
||||
|
|
@ -41,7 +41,7 @@ that succeeded. Offline tests with a fake getter.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "4cd456c7-6bd3-50a5-b267-ef9223d85e2b"
|
||||
```
|
||||
|
|
@ -54,7 +54,7 @@ fixtures cannot.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "80e43999-0884-52c1-a49d-b6bd494f898d"
|
||||
```
|
||||
|
|
@ -69,7 +69,7 @@ Also capture `Last-Modified` as `published` in the HTTP adapters where present.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7716b978-995e-58cc-8375-40700e28da96"
|
||||
```
|
||||
|
|
@ -84,7 +84,7 @@ run produces identical observations, `retrieved_at` included.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "193ee646-ecd8-52ca-b924-2d0502039d9d"
|
||||
```
|
||||
|
|
@ -100,23 +100,38 @@ stay `ok`.
|
|||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "54db7c59-b27a-59f9-b7a3-04042e3087a2"
|
||||
```
|
||||
|
||||
Add a CHANGELOG, tag `v0.1.0` (or `v0.2.0` with T01), and document
|
||||
`pip install git+https://forgejo.coulomb.social/coulomb/web-fetcher@<tag>`.
|
||||
Point assembly-economics at the tag instead of `WEB_FETCHER_PATH`.
|
||||
Repointing assembly-economics at the tag is handed off to
|
||||
`WEBFETCH-WP-0003-T03` and the assembly-economics agent (cross-repo change).
|
||||
|
||||
## Update Capability Record
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0002-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "35f54169-764e-54a7-a1c2-7513dbafea37"
|
||||
```
|
||||
|
||||
Promote availability A2 → A3 with evidence citations and a `promotion_history`
|
||||
entry; update `SCOPE.md` current state.
|
||||
|
||||
## Outcome (2026-10-19)
|
||||
|
||||
Released v0.2.0. Replay (`--replay`), source enforcement, `Last-Modified` as
|
||||
`published`, parsed latest-period rule with stated note, numeric-value check,
|
||||
selective retry with `Retry-After`, opt-in live smoke test, CHANGELOG, README
|
||||
install docs, capability record promoted to D2 / A3 / C2 / R2.
|
||||
|
||||
Verification: 18 offline tests pass; live smoke test passes for all four
|
||||
adapters; all 31 assembly-economics specs give identical status/value/period
|
||||
under 0.1.0 and 0.2.0, and the replay of the cached run is byte-identical.
|
||||
|
||||
Residuals: consumer pinning to the tag → `WEBFETCH-WP-0003-T03` (and a hub
|
||||
message to assembly-economics); CI → `WEBFETCH-WP-0003-T01`.
|
||||
|
|
|
|||
94
workplans/WEBFETCH-WP-0003-repo-hygiene.md
Normal file
94
workplans/WEBFETCH-WP-0003-repo-hygiene.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
---
|
||||
id: WEBFETCH-WP-0003
|
||||
type: workplan
|
||||
title: "Repository hygiene: CI, licence, deployment pinning, records"
|
||||
domain: infotech
|
||||
repo: web-fetcher
|
||||
status: ready
|
||||
flavor: implementation
|
||||
depends_on: []
|
||||
owner: codex
|
||||
topic_slug: stack
|
||||
created: "2026-10-19"
|
||||
updated: "2026-10-19"
|
||||
---
|
||||
|
||||
# Repository hygiene: CI, licence, deployment pinning, records
|
||||
|
||||
Smaller gaps found in the 2026-10-19 repo review, outside the
|
||||
scope-vs-intent work of `WEBFETCH-WP-0002`.
|
||||
|
||||
## Continuous Test Run on Forgejo
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T01
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tests only run when someone remembers. Add a minimal Forgejo Actions workflow
|
||||
(`.forgejo/workflows/test.yml`) running
|
||||
`python3 -B -m unittest discover -s tests` on Python 3.10 and the newest
|
||||
supported version, plus `reuse-surface validate` if the runner can reach it.
|
||||
First confirm that the Forgejo instance has runners available for
|
||||
`coulomb/web-fetcher`; if not, record that and set this task to `wait`.
|
||||
|
||||
## Choose and Add a Licence
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T02
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
The repo is offered for reuse via reuse-surface and pip installs from Forgejo
|
||||
but has no `LICENSE`. Licence choice is the founder's decision; once made, add
|
||||
`LICENSE` and the `license` field in `pyproject.toml`.
|
||||
|
||||
## Pin the railiance01 Deployment to a Release
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
rein-aharness reads a railiance01 checkout that tracks `main` at whatever
|
||||
commit it was last pulled (`5fd7ec7` on 2026-10-13). Release `v0.2.0` exists
|
||||
(`WEBFETCH-WP-0002-T03`); coordinate with assembly-economics so the checkpoint
|
||||
run consumes a pinned tag instead of a moving checkout.
|
||||
|
||||
## Refresh the Custodian Brief
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T04
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
`.custodian-brief.md` still lists the WP-0001 tasks as open. It is generated;
|
||||
regenerate it with `statehub fix-consistency` rather than editing it by hand.
|
||||
|
||||
## Federate web-fetcher in the reuse-surface Checkout
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T05
|
||||
status: wait
|
||||
priority: low
|
||||
```
|
||||
|
||||
reuse-surface commit `be5c006` adds web-fetcher to
|
||||
`registry/federation/sources.yaml`. It is committed but not pushed; pushing to
|
||||
reuse-surface `main` needs founder approval. The hosted service already lists
|
||||
the capability.
|
||||
|
||||
## Acknowledge Inbox Notices
|
||||
|
||||
```task
|
||||
id: WEBFETCH-WP-0003-T06
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
Mark the two news items read and acknowledge the two standing notices
|
||||
(build-mode security layer, safe secret-check helper) for agent `web-fetcher`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue