Finish WEBFETCH-WP-0001: refine SCOPE/INTENT, dev workflow, reuse-surface registry, seed WP-0002
- SCOPE.md rewritten with repo-specific boundaries and current state - INTENT.md: consumer contract and capability id - AGENTS.md: developer workflow (test/run/validate) - registry/: capability.infotech.provenance-fetch (D2/A2/C1/R1) - workplans: WP-0001 finished, WP-0002 proposed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 153021@bnt-lap001 Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
parent
5fd7ec7257
commit
2ea8c1337a
8 changed files with 321 additions and 17 deletions
137
registry/capabilities/capability.infotech.provenance-fetch.md
Normal file
137
registry/capabilities/capability.infotech.provenance-fetch.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
---
|
||||
id: capability.infotech.provenance-fetch
|
||||
name: Provenance-first Web and API Data Fetching
|
||||
summary: >-
|
||||
Standard-library Python fetcher that turns declarative JSON specs into
|
||||
Observation records carrying value, source, URL, source publication time,
|
||||
retrieval time and payload SHA-256. Adapters for Eurostat JSON-stat, generic
|
||||
JSON, CSV and wide XLSX endpoints; missing data is an explicit not_found,
|
||||
never estimated. Optional content-addressed cache replays a run from the
|
||||
exact bytes it used.
|
||||
owner: web-fetcher
|
||||
status: draft
|
||||
domain: infotech
|
||||
tags: [fetch, http, api, provenance, research, statistics, eurostat, csv, json, xlsx, cache, replay, data-ingestion, stdlib]
|
||||
maturity:
|
||||
discovery:
|
||||
current: D2
|
||||
target: D3
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Scope, commitments and boundaries are written down (INTENT.md, SCOPE.md)
|
||||
and one consumer (assembly-economics reality checkpoints) drove the
|
||||
design. No survey of alternative fetching libraries in the estate yet.
|
||||
availability:
|
||||
current: A2
|
||||
target: A3
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Installable source package (pyproject, CLI entry point, importable API)
|
||||
on Forgejo with offline fixture tests. Not published to a package index
|
||||
or as a versioned release artifact; consumers vendor via path or pip
|
||||
install from the repository.
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- Every value carries source, URL, retrieval time and payload hash.
|
||||
- Failures and missing values are explicit not_found/error records, never exceptions per spec.
|
||||
- Eurostat, JSON, CSV and wide-XLSX adapters cover the first consumer's sources.
|
||||
broken_expectations:
|
||||
- No pagination, authentication or rate-limit handling for APIs that need it.
|
||||
- Only four adapters; new statistics APIs require code.
|
||||
out_of_scope_expectations:
|
||||
- Crawling, scraping behind logins, LLM-based extraction.
|
||||
- Scheduling (activity-core) and execution orchestration (rein-aharness).
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- Upstream API shape changes surface as parse errors at run time; fixtures are point-in-time.
|
||||
- Live endpoints are not exercised by the test suite.
|
||||
- Single consumer so far.
|
||||
evidence:
|
||||
satisfied_signals:
|
||||
- 8 offline unit tests pass against recorded Eurostat and Census BTOS fixtures.
|
||||
- assembly-economics run_checkpoint.py consumes it for light checkpoint runs.
|
||||
discovery:
|
||||
intent: >
|
||||
Give research repositories one small, trustworthy way to pull numbers from
|
||||
public statistics APIs and web data files with full provenance, instead of
|
||||
each repo writing ad-hoc urllib or pandas download code.
|
||||
includes:
|
||||
- Declarative JSON fetch specs and a CLI emitting observations as JSONL.
|
||||
- Python API (run_spec, run_specs, Observation).
|
||||
- Adapters eurostat, http_json, http_csv, xlsx_wide and the manual not_found placeholder.
|
||||
- Optional content-addressed payload cache for replayable runs.
|
||||
excludes:
|
||||
- Crawling, headless-browser rendering (see capability.infotech.headless-browser), authenticated scraping.
|
||||
- LLM extraction or estimation of missing values.
|
||||
- Scheduling and execution orchestration.
|
||||
assumptions:
|
||||
- Consumers run Python 3.10+; no third-party dependencies are needed.
|
||||
- Sources are public and reachable over plain HTTPS.
|
||||
use_cases:
|
||||
- Reality checkpoints comparing forecasts against official statistics.
|
||||
- Reproducible research runs replayed from cached source bytes.
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A3
|
||||
current_artifacts:
|
||||
- https://forgejo.coulomb.social/coulomb/web-fetcher
|
||||
- web_fetcher/__main__.py
|
||||
- web_fetcher/registry.py
|
||||
target_artifacts:
|
||||
- Tagged, versioned release installable with pip from Forgejo.
|
||||
consumption_modes: [Python library, cli, source]
|
||||
relations:
|
||||
depends_on: []
|
||||
supports: []
|
||||
related_to: [capability.infotech.headless-browser]
|
||||
evidence:
|
||||
documentation:
|
||||
- README.md
|
||||
- INTENT.md
|
||||
- SCOPE.md
|
||||
tests:
|
||||
- tests/test_web_fetcher.py
|
||||
consumer_feedback: []
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- Fetching official statistics (Eurostat, FRED CSV, Census XLSX) with provenance for research repos.
|
||||
- Runs inside constrained executors where installing dependencies is undesirable.
|
||||
not_recommended_for:
|
||||
- JavaScript-rendered pages or sites requiring login.
|
||||
- High-volume crawling or bulk mirroring.
|
||||
known_limitations:
|
||||
- No auth, pagination or retry/backoff policy beyond a single request.
|
||||
promotion_history: []
|
||||
---
|
||||
|
||||
# Provenance-first Web and API Data Fetching
|
||||
|
||||
`web-fetcher` runs declarative fetch specs and returns `Observation` records
|
||||
with the value **and** its provenance. Values are never estimated: each spec
|
||||
yields a sourced value or an explicit `not_found` / `error` record.
|
||||
|
||||
## Try it
|
||||
|
||||
```bash
|
||||
git clone https://forgejo.coulomb.social/coulomb/web-fetcher && cd web-fetcher
|
||||
python3 -B -m unittest discover -s tests # offline fixtures
|
||||
python3 -m web_fetcher fetch specs.json --cache .cache > observations.jsonl
|
||||
```
|
||||
|
||||
```python
|
||||
from web_fetcher import run_spec
|
||||
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
|
||||
"filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
|
||||
"indic_is": "E_AI_TANY"}, "split_by": ["geo"]})
|
||||
```
|
||||
|
||||
New sources are added as adapters in `web_fetcher/adapters/` with offline
|
||||
fixture tests. Consumer reference: assembly-economics
|
||||
`checkpoints/run_checkpoint.py`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue