23 lines
982 B
Markdown
23 lines
982 B
Markdown
|
# INTENT
|
||
|
|
|
||
|
|
`web-fetcher` exists to give research repositories a shared, trustworthy way
|
||
|
|
to pull data from the web and from public APIs.
|
||
|
|
|
||
|
|
**Core commitments**
|
||
|
|
|
||
|
|
1. **Provenance first.** No value without source, URL, retrieval time and
|
||
|
|
payload hash.
|
||
|
|
2. **Never invent.** Missing data is an explicit `not_found`, never a guess.
|
||
|
|
Language models are not a data source here.
|
||
|
|
3. **Reusable adapters.** Statistics APIs, CSV/JSON endpoints and
|
||
|
|
spreadsheets behind one small interface. New sources are added as adapters
|
||
|
|
with offline fixture tests.
|
||
|
|
4. **Replayable.** An optional content-addressed cache reproduces a run from
|
||
|
|
the exact bytes it used.
|
||
|
|
5. **Minimal footprint.** Standard library only, so it runs inside executors
|
||
|
|
(for example rein-aharness on railiance01) without dependency management.
|
||
|
|
|
||
|
|
Out of scope: crawling, scraping behind logins, LLM-based extraction,
|
||
|
|
scheduling (activity-core owns *when*), and execution orchestration
|
||
|
|
(rein-aharness owns *how it runs*).
|