# web-fetcher

Reusable, provenance-first fetching of web and API data for research.

Every fetch returns `Observation` records with the value **and** where it came
from: source, URL, the source's own publication or update time, retrieval
time, and the SHA-256 of the raw payload. Values are never estimated: a spec
yields a sourced value, or an explicit `not_found` or `error` record. An
optional content-addressed cache lets a research run be replayed from the
exact bytes it used.

Standard library only. Python 3.10+.

## Adapters

| Adapter | Use | Key spec fields |
|---|---|---|
| `eurostat` | Eurostat dissemination API (JSON-stat 2.0) | `dataset`, `filters`, `split_by`, `last` |
| `http_json` | any JSON endpoint | `url`, `path` (dotted), `period_path` |
| `http_csv` | any CSV endpoint (e.g. FRED `fredgraph.csv`) | `url`, `filter`, `value_column`, `period_column` |
| `xlsx_wide` | spreadsheets with one column per period (e.g. Census BTOS) | `url`, `sheet`, `filter`, `period_regex` |
| `manual` | no machine-readable source; recorded as `not_found` with a note | `source` |

## Usage

```bash
python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl
```

```python
from web_fetcher import run_spec
obs = run_spec({"id": "eu_ai", "adapter": "eurostat", "dataset": "isoc_eb_ai",
                "filters": {"geo": ["EU27_2020"], "unit": "PC_ENT", "size_emp": "GE10",
                            "indic_is": "E_AI_TANY"}, "split_by": ["geo"]})
```

## Tests

```bash
python3 -B -m unittest discover -s tests
```

Tests run offline against fixtures in `tests/fixtures/`.

## Consumers

- `assembly-economics`: reality checkpoints (`checkpoints/sources.json`,
  `checkpoints/run_checkpoint.py`).