Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002

SCOPE.md now states only what exists (write-only cache, retry on any
URLError, optional source, string-compared latest period). INTENT.md
sharpens commitments 1 and 2. README and the capability record drop the
replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05
(enforce source), T06 (replay from cache), T07 (latest-period rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24

INTENT.md

@@ -6,9 +6,12 @@ to pull data from the web and from public APIs.
 **Core commitments**
 
 1. **Provenance first.** No value without source, URL, retrieval time and
-   payload hash.
-2. **Never invent.** Missing data is an explicit `not_found`, never a guess.
-   Language models are not a data source here.
+   payload hash — plus the source's own publication time wherever the source
+   states one.
+2. **Never invent.** Missing data is an explicit `not_found`, never a guess,
+   and when a source offers several candidates (for example the "latest"
+   period) the choice follows a stated rule visible in the record. Language
+   models are not a data source here.
 3. **Reusable adapters.** Statistics APIs, CSV/JSON endpoints and
    spreadsheets behind one small interface. New sources are added as adapters
    with offline fixture tests.
@@ -19,8 +22,8 @@ to pull data from the web and from public APIs.
 
 **Contract.** Consumers depend on the `Observation` fields and the spec
 format documented in `README.md`; adapters may be added freely, but changing
-those fields is a breaking change. Discoverable through reuse-surface as
-`capability.infotech.provenance-fetch`.
+those fields is a breaking change. The capability is discoverable through
+reuse-surface so research repos reuse it instead of writing their own fetchers.
 
 Out of scope: crawling, scraping behind logins, LLM-based extraction,
 scheduling (activity-core owns *when*), and execution orchestration

README.md

@@ -6,8 +6,8 @@ Every fetch returns `Observation` records with the value **and** where it came
 from: source, URL, the source's own publication or update time, retrieval
 time, and the SHA-256 of the raw payload. Values are never estimated: a spec
 yields a sourced value, or an explicit `not_found` or `error` record. An
-optional content-addressed cache lets a research run be replayed from the
-exact bytes it used.
+optional content-addressed cache stores the exact bytes each run used (replay
+from the cache is not implemented yet; see `WEBFETCH-WP-0002`).
 
 Standard library only. Python 3.10+.
 

SCOPE.md

@@ -1,27 +1,47 @@
 # SCOPE
 
+> What the repository **is** today. What it aspires to be is in `INTENT.md`;
+> the gap between the two is tracked in `workplans/`.
+
 ## One-liner
 
-Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.
+Standard-library fetching of data from public APIs and web data files into
+provenance-carrying `Observation` records, for research repositories.
 
 ## Core Idea
 
-Research repos declare *what* they need as JSON fetch specs; web-fetcher returns
-`Observation` records that carry the value together with source, URL, the
-source's own publication time, retrieval time and the SHA-256 of the raw
-payload. Missing data is an explicit `not_found`, failures an explicit `error` —
-never a guess. See `INTENT.md` for the commitments.
+Research repos declare *what* they need as JSON fetch specs; web-fetcher fetches
+each URL and returns `Observation` records with the value plus URL, retrieval
+time and the SHA-256 of the raw payload. Missing data becomes `not_found`,
+failures become `error`; a failing spec never aborts the run.
 
-## In Scope
+## What Exists
 
-- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`).
-- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`).
-- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`,
-  and the `manual` not_found placeholder. New public sources as adapters with
-  offline fixture tests.
-- Optional content-addressed payload cache for replayable runs.
-- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API.
-- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`).
+- `Observation` record and `fetch_bytes` HTTP GET (`web_fetcher/core.py`):
+  sets `retrieved_at` and `payload_sha256` on every fetch; retries any
+  `URLError`/timeout/`OSError` twice with linear backoff (1.5 s, 3 s) — this
+  includes permanent HTTP 4xx such as 404; no `Retry-After` handling.
+- Spec dispatch (`web_fetcher/registry.py`): unknown adapters, fetch failures
+  and parse errors become `error` observations.
+- Adapters:
+  - `eurostat` (JSON-stat 2.0) — also sets `published` from the dataset's `updated`.
+  - `http_json` — dotted path lookup.
+  - `http_csv` — row filter; "latest" row chosen by **string** comparison of the
+    period column, or the last row if none is given; non-numeric values are
+    returned as strings with `status: ok`.
+  - `xlsx_wide` — one column per period, stdlib XLSX reader (`web_fetcher/xlsx.py`).
+  - `manual` — always `not_found` with a note.
+- `source` is copied from the optional spec field; the HTTP adapters can emit
+  `status: ok` with `source: null`. `published` is unset outside `eurostat`.
+- Cache (`--cache DIR`): **write-only**. Stores `<sha256>.bin` and appends to
+  `index.jsonl`; nothing reads it back, so runs cannot yet be replayed.
+- CLI `python3 -m web_fetcher fetch specs.json [--cache] [--out] [--only]`
+  emitting JSONL; exit code 1 if any observation is `error`. Python API
+  `run_spec`, `run_specs`.
+- 8 offline unit tests against Eurostat and Census BTOS fixtures and fake
+  getters; no live-endpoint tests.
+- Reuse-surface capability record `capability.infotech.provenance-fetch`
+  (`registry/`), vector D2 / A2 / C1 / R1.
 
 ## Out of Scope
 
@@ -31,15 +51,16 @@ never a guess. See `INTENT.md` for the commitments.
 - Scheduling — activity-core owns *when*.
 - Execution orchestration — rein-aharness owns *how it runs*.
 - Storing or analysing fetched data; that belongs to the consuming research repo.
-- Third-party runtime dependencies.
+- Authentication, pagination, third-party runtime dependencies.
 
 ## Current State
 
-- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
-- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
-- Known gaps: no retry/backoff, auth or pagination; no tagged release yet
-  (tracked in `WEBFETCH-WP-0002`).
-- Reuse vector: D2 / A2 / C1 / R1.
+- Version 0.1.0, ~400 lines of Python, untagged; consumed from source.
+- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`,
+  via `WEB_FETCHER_PATH`).
+- Running inside rein-aharness is plausible (stdlib only) but unverified.
+- Gaps against `INTENT.md` (source enforcement, replay, latest-period
+  selection, selective retry, release) are tracked in `WEBFETCH-WP-0002`.
 
 ## Getting Oriented
 

registry/capabilities/capability.infotech.provenance-fetch.md

@@ -6,8 +6,8 @@ summary: >-
   Observation records carrying value, source, URL, source publication time,
   retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic
   JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found,
-  never estimated. Optional content-addressed cache replays a run from the
-  exact bytes it used.
+  never estimated. Optional content-addressed cache stores the exact bytes a
+  run used; replay from it is not implemented yet.
 owner: web-fetcher
 status: draft
 domain: infotech
@@ -36,10 +36,14 @@ external_evidence:
     confidence: medium
     basis: scope_vs_intent_and_consumer_expectations
     satisfied_expectations:
-      - Every value carries source, URL, retrieval time and payload hash.
+      - Every value carries URL, retrieval time and payload hash.
       - Failures and missing values are explicit not_found/error records, never exceptions per spec.
       - Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources.
     broken_expectations:
+      - Runs cannot be replayed; the cache is write-only.
+      - source is optional in specs, so HTTP adapters can return ok values with source null.
+      - Source publication time is only captured by the Eurostat adapter.
+      - Latest-period selection compares period strings and can pick the wrong row for mixed formats.
       - No pagination, authentication or rate-limit handling for APIs that need it.
       - Only four adapters; new statistics APIs require code.
     out_of_scope_expectations:
@@ -50,6 +54,7 @@ external_evidence:
     confidence: low
     basis: consumer_quality_signals
     known_reliability_risks:
+      - Retries cover every URLError, including permanent 4xx; 429/Retry-After is not honoured.
       - Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time.
       - Live endpoints are not exercised by the test suite.
       - Single consumer so far.
@@ -66,7 +71,7 @@ discovery:
     - Declarative JSON fetch specs and a CLI emitting observations as JSONL.
     - Python API (run_spec, run_specs, Observation).
     - Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder.
-    - Optional content-addressed payload cache for replayable runs.
+    - Optional content-addressed payload cache (write-only; replay planned in WEBFETCH-WP-0002).
   excludes:
     - Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping.
     - LLM extraction or estimation of missing values.
@@ -76,7 +81,7 @@ discovery:
     - Sources are public and reachable over plain HTTPS.
   use_cases:
     - Reality checkpoints comparing forecasts against official statistics.
-    - Reproducible research runs replayed from cached source bytes.
+    - Retaining the exact source bytes behind a research run for later audit.
 availability:
   current_level: A2
   target_level: A3
@@ -107,7 +112,9 @@ consumer_guidance:
     - JavaScript-rendered pages or sites requiring login.
     - High-volume crawling or bulk mirroring.
   known_limitations:
-    - No auth, pagination or retry/backoff policy beyond a single request.
+    - No auth or pagination; retries are two fixed linear attempts on any network/HTTP error.
+    - Cache is write-only; replay is planned in WEBFETCH-WP-0002.
+    - Supply "source" in every spec; it is not enforced yet.
 promotion_history: []
 ---
 
@@ -115,7 +122,9 @@ promotion_history: []
 
 `web-fetcher` runs declarative fetch specs and returns `Observation` records
 with the value **and** its provenance. Values are never estimated: each spec
-yields a sourced value or an explicit `not_found` / `error` record.
+yields a value or an explicit `not_found` / `error` record. Gaps against the
+repo's intent (replay, source enforcement, latest-period rule, selective retry)
+are tracked in `WEBFETCH-WP-0002`.
 
 ## Try it
 

registry/indexes/capabilities.yaml

@@ -7,8 +7,8 @@ capabilities:
   summary: Standard-library Python fetcher that turns declarative JSON specs into Observation
     records carrying value, source, URL, source publication time, retrieval time and payload
     SHA-256. Adapters for Eurostat JSON-stat, generic JSON, CSV and wide XLSX endpoints; missing
-    data is an explicit not_found, never estimated. Optional content-addressed cache replays
-    a run from the exact bytes it used.
+    data is an explicit not_found, never estimated. Optional content-addressed cache stores
+    the exact bytes a run used; replay from it is not implemented yet.
   domain: infotech
   status: draft
   owner: web-fetcher

workplans/WEBFETCH-WP-0002-release-and-fetch-hardening.md

@@ -17,10 +17,11 @@ state_hub_workstream_id: "68acbef4-d706-5cb4-8138-eb3807881f36"
 # Versioned release and fetch hardening
 
 Move `capability.infotech.provenance-fetch` from A2 (source) to A3 (versioned,
-installable release) and close the reliability gaps listed in the capability
-record, without adding third-party dependencies.
+installable release) and close the gaps between `SCOPE.md` (what is) and
+`INTENT.md` (what the repo commits to), found in the 2026-10-19 scope-vs-intent
+assessment. No third-party dependencies.
 
-## Retry and Backoff in fetch_bytes
+## Selective Retry in fetch_bytes
 
 ```task
 id: WEBFETCH-WP-0002-T01
@@ -29,8 +30,10 @@ priority: high
 state_hub_task_id: "e64ddb81-3d56-5e09-9744-5c0a7c26005b"
 ```
 
-Add bounded retry with exponential backoff for transient failures (connection
-errors, HTTP 429/5xx), honouring `Retry-After`. Retries must not change
+`fetch_bytes` already retries twice with linear backoff, but on every
+`URLError` — including permanent HTTP 4xx such as 404. Retry only transient
+failures (connection errors, timeouts, HTTP 429/5xx), use exponential backoff,
+honour `Retry-After`, and fail fast on other 4xx. Retries must not change
 provenance semantics: `retrieved_at` and `payload_sha256` describe the attempt
 that succeeded. Offline tests with a fake getter.
 
@@ -47,6 +50,49 @@ Add a live smoke test (skipped unless `WEB_FETCHER_LIVE=1`) that runs one spec
 per adapter against the real endpoints, to detect upstream shape changes that
 fixtures cannot.
 
+## Enforce Source on Every Value
+
+```task
+id: WEBFETCH-WP-0002-T05
+status: todo
+priority: high
+```
+
+INTENT commitment 1: no value without source. Today `source` comes from the
+optional spec field, so `http_json`/`http_csv` can emit `status: ok` with
+`source: null`. Make a missing source impossible for `ok` observations (reject
+the spec as `error`, or derive it from the URL host — decide and document).
+Also capture `Last-Modified` as `published` in the HTTP adapters where present.
+
+## Replay Runs from the Cache
+
+```task
+id: WEBFETCH-WP-0002-T06
+status: todo
+priority: high
+```
+
+INTENT commitment 4. The cache writes `<sha256>.bin` plus `index.jsonl` but
+nothing reads it back. Add a replay mode (`--replay DIR` / `replay_dir=`) that
+serves each URL from the cache index without network access and yields an
+`error` observation for an uncached URL. Test: a fetch run followed by a replay
+run produces identical observations, `retrieved_at` included.
+
+## Deterministic Latest-Period Selection
+
+```task
+id: WEBFETCH-WP-0002-T07
+status: todo
+priority: medium
+```
+
+INTENT commitment 2. `http_csv` (and `xlsx_wide`) pick the "latest" row by
+string comparison of the period, or the last row when no period column is
+given; mixed formats (`2024-Q4` vs `2024-10`) can silently select the wrong
+value. Normalise periods or require a declared format, and record the chosen
+period/row in the observation. Also decide whether a non-numeric value should
+stay `ok`.
+
 ## Tagged Release Installable from Forgejo
 
 ```task