coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
web-fetcher/INTENT.md
tegwick 7d6d5a04c8 Align SCOPE with actual code; INTENT as aspiration; plan the gaps in WP-0002 
SCOPE.md now states only what exists (write-only cache, retry on any
URLError, optional source, string-compared latest period). INTENT.md
sharpens commitments 1 and 2. README and the capability record drop the
replay overclaim. WP-0002 reworks T01 (selective retry) and adds T05
(enforce source), T06 (replay from cache), T07 (latest-period rule).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
2026-10-19 13:58:27 +02:00

1.5 KiB
Raw Permalink Blame History

INTENT

web-fetcher exists to give research repositories a shared, trustworthy way to pull data from the web and from public APIs.

Core commitments

  1. Provenance first. No value without source, URL, retrieval time and payload hash — plus the source's own publication time wherever the source states one.
  2. Never invent. Missing data is an explicit not_found, never a guess, and when a source offers several candidates (for example the "latest" period) the choice follows a stated rule visible in the record. Language models are not a data source here.
  3. Reusable adapters. Statistics APIs, CSV/JSON endpoints and spreadsheets behind one small interface. New sources are added as adapters with offline fixture tests.
  4. Replayable. An optional content-addressed cache reproduces a run from the exact bytes it used.
  5. Minimal footprint. Standard library only, so it runs inside executors (for example rein-aharness on railiance01) without dependency management.

Contract. Consumers depend on the Observation fields and the spec format documented in README.md; adapters may be added freely, but changing those fields is a breaking change. The capability is discoverable through reuse-surface so research repos reuse it instead of writing their own fetchers.

Out of scope: crawling, scraping behind logins, LLM-based extraction, scheduling (activity-core owns when), and execution orchestration (rein-aharness owns how it runs).