Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002
- SCOPE.md rewritten with repo-specific boundaries and current state - INTENT.md: consumer contract and capability id - AGENTS.md: developer workflow (test/run/validate) - registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1) - workplans: WP-0001 finished, WP-0002 proposed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
parent
5fd7ec7257
commit
2ea8c1337a
8 changed files with 321 additions and 17 deletions
41
SCOPE.md
41
SCOPE.md
|
|
@ -1,32 +1,49 @@
|
|||
# SCOPE
|
||||
|
||||
> This file was generated by `statehub register`. Refine it as the repository
|
||||
> boundaries become clearer.
|
||||
|
||||
## One-liner
|
||||
|
||||
Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.
|
||||
|
||||
## Core Idea
|
||||
|
||||
web-fetcher exists to provide the capability described in INTENT.md.
|
||||
Research repos declare *what* they need as JSON fetch specs; web-fetcher returns
|
||||
`Observation` records that carry the value together with source, URL, the
|
||||
source's own publication time, retrieval time and the SHA-256 of the raw
|
||||
payload. Missing data is an explicit `not_found`, failures an explicit `error` —
|
||||
never a guess. See `INTENT.md` for the commitments.
|
||||
|
||||
## In Scope
|
||||
|
||||
- Maintain the repository's primary implementation.
|
||||
- Keep docs, tests, and operational metadata current.
|
||||
- `Observation` record, provenance-tracked HTTP fetching (`web_fetcher/core.py`).
|
||||
- Spec dispatch that never raises for a single failing spec (`web_fetcher/registry.py`).
|
||||
- Adapters: `eurostat` (JSON-stat 2.0), `http_json`, `http_csv`, `xlsx_wide`,
|
||||
and the `manual` not_found placeholder. New public sources as adapters with
|
||||
offline fixture tests.
|
||||
- Optional content-addressed payload cache for replayable runs.
|
||||
- CLI (`python3 -m web_fetcher fetch specs.json`) emitting JSONL, and the Python API.
|
||||
- Reuse-surface capability record `capability.infotech.provenance-fetch` (`registry/`).
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Own unrelated adjacent systems.
|
||||
- Make irreversible operational decisions without human approval.
|
||||
- Crawling, scraping behind logins, JavaScript-rendered pages
|
||||
(use `capability.infotech.headless-browser`).
|
||||
- LLM-based extraction or estimation of missing values.
|
||||
- Scheduling — activity-core owns *when*.
|
||||
- Execution orchestration — rein-aharness owns *how it runs*.
|
||||
- Storing or analysing fetched data; that belongs to the consuming research repo.
|
||||
- Third-party runtime dependencies.
|
||||
|
||||
## Current State
|
||||
|
||||
- Status: active; implementation and stability should be verified by the repo agent.
|
||||
- Version 0.1.0, ~400 lines of stdlib Python, 8 offline tests passing (2026-10-19).
|
||||
- Consumer: assembly-economics reality checkpoints (`checkpoints/run_checkpoint.py`).
|
||||
- Known gaps: no retry/backoff, auth or pagination; no tagged release yet
|
||||
(tracked in `WEBFETCH-WP-0002`).
|
||||
- Reuse vector: D2 / A2 / C1 / R1.
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
- Start with: INTENT.md
|
||||
- Agent instructions: AGENTS.md
|
||||
- Workplans: workplans/
|
||||
- Start with: `INTENT.md`, then `README.md` for spec fields and usage.
|
||||
- Agent instructions and dev commands: `AGENTS.md`
|
||||
- Capability record: `registry/capabilities/`
|
||||
- Workplans: `workplans/`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue