tegwick
2ea8c1337a
- SCOPE.md rewritten with repo-specific boundaries and current state - INTENT.md: consumer contract and capability id - AGENTS.md: developer workflow (test/run/validate) - registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1) - workplans: WP-0001 finished, WP-0002 proposed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
49 lines
2 KiB
Markdown
49 lines
2 KiB
Markdown
# SCOPE
|
|
|
|
## One-liner
|
|
|
|
Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.
|
|
|
|
## Core Idea
|
|
|
|
Research repos declare *what* they need as JSON fetch specs; web-fetcher returns
|
|
`Observation` records that carry the value together with source, URL, the
|
|
source's own publication time, retrieval time and the SHA-256 of the raw
|
|
payload. Missing data is an explicit `not_found`, failures an explicit `error` —
|
|
never a guess. See `INTENT.md` for the commitments.
|
|
|
|
## In Scope
|
|
|
|
- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`).
|
|
- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`).
|
|
- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`,
|
|
and the `manual` not_found placeholder. New public sources as adapters with
|
|
offline fixture tests.
|
|
- Optional content-addressed payload cache for replayable runs.
|
|
- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API.
|
|
- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`).
|
|
|
|
## Out of Scope
|
|
|
|
- Crawling, scraping behind logins, JavaScript-rendered pages
|
|
(use `capability.infotech.headless-browser`).
|
|
- LLM-based extraction or estimation of missing values.
|
|
- Scheduling — activity-core owns *when*.
|
|
- Execution orchestration — rein-aharness owns *how it runs*.
|
|
- Storing or analysing fetched data; that belongs to the consuming research repo.
|
|
- Third-party runtime dependencies.
|
|
|
|
## Current State
|
|
|
|
- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
|
|
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
|
|
- Known gaps: no retry/backoff, auth or pagination; no tagged release yet
|
|
(tracked in `WEBFETCH-WP-0002`).
|
|
- Reuse vector: D2 / A2 / C1 / R1.
|
|
|
|
## Getting Oriented
|
|
|
|
- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
|
|
- Agent instructions and dev commands: `AGENTS.md`
|
|
- Capability record: `registry/capabilities/`
|
|
- Workplans: `workplans/`
|