tegwick
7d6d5a04c8
SCOPE.md now states only what exists (write-only cache, retry on any URLError, optional source, string-compared latest period). INTENT.md sharpens commitments 1 and 2. README and the capability record drop the replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05 (enforce source), T06 (replay from cache), T07 (latest-period rule). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
70 lines
3.2 KiB
Markdown
70 lines
3.2 KiB
Markdown
# SCOPE
|
|
|
|
> What the repository **is** today. What it aspires to be is in `INTENT.md`;
|
|
> the gap between the two is tracked in `workplans/`.
|
|
|
|
## One-liner
|
|
|
|
Standard-library fetching of data from public APIs and web data files into
|
|
provenance-carrying `Observation` records, for research repositories.
|
|
|
|
## Core Idea
|
|
|
|
Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches
|
|
each URL and returns `Observation` records with the value plus URL, retrieval
|
|
time and the SHA-256 of the raw payload. Missing data becomes `not_found`,
|
|
failures become `error`; a failing spec never aborts the run.
|
|
|
|
## What Exists
|
|
|
|
- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
|
|
sets `retrieved_at` and `payload_sha256` on every fetch; retries any
|
|
`URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
|
|
includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
|
|
- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
|
|
and parse errors become `error` observations.
|
|
- Adapters:
|
|
- `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
|
|
- `http_json` — dotted path lookup.
|
|
- `http_csv` — row filter; "latest" row chosen by **string** comparison of the
|
|
period column, or the last row if none is given; non-numeric values are
|
|
returned as strings with `status: ok`.
|
|
- `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
|
|
- `manual` — always `not_found` with a note.
|
|
- `source` is copied from the optional spec field; the HTTP adapters can emit
|
|
`status: ok` with `source: null`. `published` is unset outside `eurostat`.
|
|
- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
|
|
`index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
|
|
- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
|
|
emitting JSONL; exit code 1 if any observation is `error`. Python API
|
|
`run_spec`, `run_specs`.
|
|
- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
|
|
getters; no live-endpoint tests.
|
|
- Reuse-surface capability record `capability.infotech.provenance-fetch`
|
|
(`registry/`), vector D2 / A2 / C1 / R1.
|
|
|
|
## Out of Scope
|
|
|
|
- Crawling, scraping behind logins, JavaScript-rendered pages
|
|
(use `capability.infotech.headless-browser`).
|
|
- LLM-based extraction or estimation of missing values.
|
|
- Scheduling — activity-core owns *when*.
|
|
- Execution orchestration — rein-aharness owns *how it runs*.
|
|
- Storing or analysing fetched data; that belongs to the consuming research repo.
|
|
- Authentication, pagination, third-party runtime dependencies.
|
|
|
|
## Current State
|
|
|
|
- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
|
|
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
|
|
via `WEB_FETCHER_PATH`).
|
|
- Running inside rein-aharness is plausible (stdlib only) but unverified.
|
|
- Gaps against `INTENT.md` (source enforcement, replay, latest-period
|
|
selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.
|
|
|
|
## Getting Oriented
|
|
|
|
- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
|
|
- Agent instructions and dev commands: `AGENTS.md`
|
|
- Capability record: `registry/capabilities/`
|
|
- Workplans: `workplans/`
|