tegwick
96fe5b022b
- --replay DIR / replay_dir: offline replay from the cache, hash-verified - ok values without source become error; Last-Modified recorded as published - latest period by parsed order (web_fetcher/periods.py); mixed formats refused - non-numeric values not_found unless value_type: string - retry only connection errors, timeouts, 429/5xx; honour Retry-After - opt-in live smoke test; 18 offline tests - CHANGELOG, README install docs, SCOPE update, capability D2/A3/C2/R2 - WEBFETCH-WP-0003 (repo hygiene) added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
71 lines
2.7 KiB
Markdown
71 lines
2.7 KiB
Markdown
# web-fetcher
|
|
|
|
Reusable, provenance-first fetching of web and API data for research.
|
|
|
|
Every fetch returns `Observation` records with the value **and** where it came
|
|
from: source, URL, the source's own publication or update time, retrieval
|
|
time, and the SHA-256 of the raw payload. Values are never estimated: a spec
|
|
yields a sourced value, or an explicit `not_found` or `error` record. An
|
|
optional content-addressed cache stores the exact bytes each run used, and a
|
|
run can be replayed from it offline.
|
|
|
|
Standard library only. Python 3.10+.
|
|
|
|
## Adapters
|
|
|
|
| Adapter | Use | Key spec fields |
|
|
|---|---|---|
|
|
| `eurostat` | Eurostat dissemination API (JSON-stat 2.0) | `dataset`, `filters`, `split_by`, `last` |
|
|
| `http_json` | any JSON endpoint | `url`, `source`, `path` (dotted), `period_path` |
|
|
| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `source`, `filter`, `value_column`, `period_column` |
|
|
| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `source`, `sheet`, `filter`, `period_regex` |
|
|
| `manual` | no machine-readable source; recorded as `not_found` with a note | `source` |
|
|
|
|
### Provenance rules
|
|
|
|
- Every non-Eurostat spec must name its `source`; an `ok` value without one is
|
|
turned into an `error`.
|
|
- "Latest" period (`http_csv`, `xlsx_wide`) follows parsed period order —
|
|
year, month (`2026-09`), date, quarter (`2026-Q3`/`2026Q3`), semester, week,
|
|
fixed-width digits (`202620`). Mixed formats are an `error`; pin one with
|
|
`period_format`, or `period_format: string` for plain lexicographic order.
|
|
The rule applied is written to the observation's `note`.
|
|
- Values must be numeric unless `value_type: string`; otherwise `not_found`.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install "git+https://forgejo.coulomb.social/coulomb/web-fetcher@v0.2.0"
|
|
```
|
|
|
|
Or put a checkout on `PYTHONPATH` — there are no dependencies.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl
|
|
python3 -m web_fetcher fetch specs.json --replay .cache > replayed.jsonl # offline, same bytes
|
|
```
|
|
|
|
```python
|
|
from web_fetcher import run_spec
|
|
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
|
"filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
|
|
"indic_is": "E_AI_TANY"}, "split_by": ["geo"]})
|
|
```
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
python3 -B -m unittest discover -s tests
|
|
```
|
|
|
|
Tests run offline against fixtures in `tests/fixtures/`. Live smoke test,
|
|
one spec per adapter: `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v`.
|
|
|
|
See `CHANGELOG.md` for release notes.
|
|
|
|
## Consumers
|
|
|
|
- `assembly-economics`: reality checkpoints (`checkpoints/sources.json`,
|
|
`checkpoints/run_checkpoint.py`).
|