coulomb/web-fetcher
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

WEBFETCH-WP-0003: Forgejo CI, org licence, inbox acks, progress
All checks were successful
Tests / unittest-py310 (push) Successful in 9s
Details
Tests / unittest-py313 (push) Successful in 13s
Details

Browse source 
- .forgejo/workflows/test.yaml: offline tests on Python 3.10 and 3.13
- LICENSE: Target Revenue Source License V1C1 (org-wide preliminary licence)
- pyproject: license field; AGENTS/SCOPE: CI and licence facts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 153021@bnt-lap001
Assistant-Session: 46917be4-6400-4106-b8b7-61953a0caf24
This commit is contained in:
tegwick 2026-10-19 14:27:11 +02:00
parent 361218fe40
commit 49de8d15b6
20 changed files with 1000 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

43
.forgejo/workflows/test.yaml Normal file
View file

@ -0,0 +1,43 @@
# Offline unit tests on the oldest and newest supported Python.
# Standard library only: no install step; the exact pushed revision is fetched
# as an archive (the repository is public), following key-cape's acceptance job.
name: Tests
on:
push:
branches: [main]
tags: ["v*"]
workflow_dispatch:
jobs:
unittest-py310:
runs-on: ubuntu-latest
container:
image: python:3.10-alpine
steps:
- name: Offline unit tests (Python 3.10)
run: |
set -eu
mkdir -p /tmp/src
wget -qO /tmp/src.tar.gz "https://forgejo.coulomb.social/${GITHUB_REPOSITORY}/archive/${GITHUB_SHA}.tar.gz"
tar xzf /tmp/src.tar.gz -C /tmp/src --strip-components=1
cd /tmp/src
python3 --version
python3 -m compileall -q web_fetcher
python3 -B -m unittest discover -s tests -v
unittest-py313:
runs-on: ubuntu-latest
container:
image: python:3.13-alpine
steps:
- name: Offline unit tests (Python 3.13)
run: |
set -eu
mkdir -p /tmp/src
wget -qO /tmp/src.tar.gz "https://forgejo.coulomb.social/${GITHUB_REPOSITORY}/archive/${GITHUB_SHA}.tar.gz"
tar xzf /tmp/src.tar.gz -C /tmp/src --strip-components=1
cd /tmp/src
python3 --version
python3 -m compileall -q web_fetcher
python3 -B -m unittest discover -s tests -v

2
AGENTS.md
View file

@ -272,6 +272,8 @@ no dependency install step.
| Run | `python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl` |
| Byte-compile check | `python3 -m compileall -q web_fetcher` |
| Validate capability registry | `uv run --project ~/reuse-surface reuse-surface validate --root .` |
| Live smoke test (network) | `WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v` |
| CI | `.forgejo/workflows/test.yaml` runs the offline tests on Python 3.10 and 3.13 for every push to `main` and every `v*` tag |
Rules: every new adapter ships with a recorded fixture in `tests/fixtures/` and
an offline test; tests must not hit the network. Do not add third-party

151
LICENSE Normal file
View file

@ -0,0 +1,151 @@
# Target Revenue Source License
**Version 1.0, Candidate 1 (V1C1)**
---
> **PRELIMINARY CANDIDATE — SUBJECT TO CHANGE — NOT FINAL**
>
> This repository is governed by the Target Revenue Source License
> (TRSL), Version 1, Candidate 1. This is the framework's first working
> candidate, adopted as the org's preliminary operating license across
> all repos (maintainer decision, 2026-07-29) during the build/alpha
> stage — see `SCOPE.md` §1 and Appendix A of the canonical text (link
> below) for the alpha/beta risk-acceptance decision this reflects and
> what it does and does not mean. Full specialist legal review is
> explicitly deferred until the framework moves out of beta.
>
> **Canonical source, full candidate-status banner, and Appendix A
> (non-normative candidate notes tracking every open item):** the
> `coulomb` org's `target-revenue` repository,
> `specs/TargetRevenueSourceLicense-V1C1.md` — this file is the operative
> legal text (Preamble through Section 11) only; the canonical document
> is authoritative if this copy and that document ever diverge.
---
## Preamble
This Target Revenue Source License ("**License**") governs the Software identified in the applicable Phase Manifest. It implements the Target Revenue Framework: a defined development Phase accumulates Development Credit and Remission Credit against an immutable Initial Target until the Milestone Release automatically and irrevocably converts to a declared permissive Future License.
Commercial beneficiaries fund the creation and early availability of a software improvement; once the declared target is satisfied, the governed release becomes permissively open source.
## 1. Definitions
Capitalized terms used in this License have the meanings given below. Where a term is also defined in the Phase Manifest or Target Ledger for a specific Phase, the Phase Manifest and Target Ledger govern the *values* (amounts, dates, identifiers) and this License governs the *legal effect* of those values — the two must not be read as conflicting definitions of the same concept.
**"Commercial Entitlement"** means a right, purchased or otherwise granted under a Commercial Use Agreement, to make Commercial Use of the Software during a Phase.
**"Commercial Use"** means billing, invoicing, or otherwise charging any customer a fee, subscription, license fee, or other consideration for or in connection with use of the Software, at any time before the Conversion Event for the applicable Phase, regardless of whether the person or organization billed would otherwise qualify for Noncommercial Use. Commercial Use occurs by virtue of such billing alone, whether or not the resulting payment is registered with the Trust Service; in particular, billing a customer for pre-conversion use of the Software without recording the corresponding payment in the applicable Phase's Target Ledger is Commercial Use without a valid Commercial Entitlement — a violation of Section 3, addressed under Section 7 and, where applicable, the Enforcement Network described in the canonical repository's `specs/EnforcementNetworkConcept.md`.
**"Commercial Use Agreement"** means the separate agreement, referenced by the applicable Phase Manifest, under which a Commercial Entitlement is purchased or granted. This License does not itself set pricing, metering, or payment terms — those are governed by the Commercial Use Agreement.
**"Conversion Event"** means the moment the Outstanding Target for a Phase reaches zero, as computed from the Phase Manifest and Target Ledger per the Target Ledger Specification. The Conversion Event occurs automatically and is not conditioned on any declaration, attestation, or other act by the Licensor or any Trust Service.
**"Development Credit"** means the portion of a collected and settled payment explicitly allocated toward satisfying the Initial Target of a specific Phase, as recorded in that Phase's Target Ledger.
**"Future License"** means the permissive license identified in the applicable Phase Manifest, being either the MIT License or the Apache License, Version 2.0, which applies to the Milestone Release upon the Conversion Event.
**"Initial Target"** means the immutable monetary target declared for a Phase in its Phase Manifest.
**"Licensor"** means **Binky Hedgehog GmbH**, the party that publishes the Phase Manifest and holds the rights necessary to grant this License and the Future License for the Milestone Release.
**"Milestone Release"** means the precisely identified software release designated in the applicable Phase Manifest, identified by an immutable source revision, release artifact, or cryptographic digest.
**"Noncommercial Use"** means use of the Software for personal purposes, private study, hobby or amateur projects; use by any charitable organization, educational institution, public research organization, or government institution acting in a non-revenue-generating capacity; or other use of a materially similar character.
**"Outstanding Target"** means, at any time, `max(0, Initial Target − cumulative Development Credit − cumulative Remission Credit)` for a Phase, as computed from that Phase's Target Ledger.
**"Phase"** means a bounded development undertaking governed by one Initial Target, one Milestone Release, one degeneration policy, and one Future License declaration, as declared in a Phase Manifest.
**"Phase Manifest"** means the published, immutable declaration identifying a Phase, its Milestone Release, Initial Target, Future License, degeneration policy, and Target Ledger location, as specified in the Phase Manifest Specification.
**"Remission Credit"** means a transparent, non-revenue reduction of a Phase's Outstanding Target, generated under that Phase's published degeneration policy and recorded in the Target Ledger.
**"Settled Payment"** means a payment that has cleared through its payment processor and is no longer subject to reversal in the ordinary course (chargeback, dispute, or equivalent), as further specified by the applicable Commercial Use Agreement or monetization extension.
**"Software"** means the source code, object code, and associated documentation of the Milestone Release identified in the applicable Phase Manifest.
**"Target Ledger"** means the append-only record of Development Credit, Remission Credit, and correction entries for a Phase, as specified in the Target Ledger Specification.
**"You"** or **"Licensee"** means the individual or entity exercising rights under this License.
## 2. Grant of Rights for Noncommercial Use
Subject to the terms of this License, the Licensor grants You a worldwide, royalty-free, non-exclusive license, during the applicable Phase, to:
(a) use, reproduce, and study the Software for any Noncommercial Use;
(b) modify the Software and create derivative works of it for any Noncommercial Use; and
(c) redistribute the Software and Your modifications, in source or object form, for any Noncommercial Use, provided that You include this License, unmodified, with any such redistribution, and that You do not remove or alter any copyright, patent, trademark, or attribution notices contained in the Software.
This grant does not extend to Commercial Use. Commercial Use requires a Commercial Entitlement under Section 3.
## 3. Commercial Use
You may not make Commercial Use of the Software during the applicable Phase unless You hold a valid, current Commercial Entitlement under a Commercial Use Agreement with the Licensor covering the applicable Phase. A Commercial Entitlement granted under one Phase's Commercial Use Agreement does not extend to a later Phase's Milestone Release unless the Commercial Use Agreement expressly says so.
This Section 3 states the existence and boundary of the commercial-use restriction. It does not itself set pricing, invoicing, metering, audit rights, or payment terms — those are governed exclusively by the applicable Commercial Use Agreement.
## 4. Patent License
Subject to the terms of this License, each contributor to the Software grants You, during the applicable Phase and solely to the extent of rights granted under Sections 2 and 3, a perpetual (subject to the termination below), worldwide, non-exclusive, no-charge, royalty-free patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Software, limited to those patent claims licensable by that contributor that are necessarily infringed by their contribution(s) alone or by combination of their contribution(s) with the Software.
If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Software or a contribution incorporated within it constitutes direct or contributory patent infringement, then any patent licenses granted to You under this Section 4 for the Software shall terminate as of the date such litigation is filed.
## 5. Automatic Conversion to the Future License
**5.1 Automatic effect.** Upon the Conversion Event for a Phase, the rights and restrictions in Sections 3 (Commercial Use) of this License, as they apply to that Phase's Milestone Release, terminate automatically. In their place, the Milestone Release is licensed under the Future License identified in that Phase's Phase Manifest, effective as of the Conversion Event, without any further act, declaration, or attestation required by the Licensor, any Trust Service, or any other party.
**5.2 Irrevocability.** Once a valid Conversion Event has occurred for a Phase, no subsequent refund, chargeback, accounting correction, dispute, or termination of this License for an unrelated breach shall revoke, suspend, or otherwise impair the Future License grant for that Phase's Milestone Release. Any shortfall or dispute arising after a Conversion Event is a commercial or accounting matter between the relevant parties and does not reinstate a commercial-use restriction over already-converted Software.
**5.3 Prior freedom preserved.** A later Phase covering subsequent improvements to the Software does not restrict, withdraw, or otherwise affect the rights granted under the Future License for an earlier Phase's already-converted Milestone Release.
**5.4 Evidence, not cause.** A Trust Service may publish a Conversion Attestation documenting a Conversion Event. Such an attestation is evidence that the Conversion Event occurred; it is not a condition of, and its absence or delay does not postpone, the automatic effect described in Section 5.1. Any person may independently verify whether a Conversion Event has occurred directly from the Phase Manifest and Target Ledger.
## 6. Successive Phases
The Licensor may declare a new Phase covering subsequent improvements to the Software following a Milestone Release's Conversion Event. Each Phase is independently governed by its own Phase Manifest, Initial Target, degeneration policy, and Target Ledger. Nothing in a later Phase's Phase Manifest may be construed to reduce or withdraw rights already granted under Section 5 for an earlier Phase's Milestone Release.
## 7. Term and Termination
**7.1 Term.** This License applies to the Software for the duration of the applicable Phase, and, for the Milestone Release, indefinitely following that Phase's Conversion Event under the Future License.
**7.2 Termination for breach.** If You breach Section 3 (Commercial Use) or Section 2(c) (redistribution notice requirement), the Licensor may terminate this License as to You. Before such termination becomes effective, the Licensor shall provide You written notice of the breach; if You cure the breach within thirty (30) days of that notice, this License continues in effect. A second breach of the same provision within twelve (12) months may be terminated immediately without a further cure opportunity.
**7.3 Effect of termination.** Termination under this Section 7 affects only Your rights under Sections 2 and 3 for the Phase in which the breach occurred. It does not affect any rights already vested under Section 5 (Automatic Conversion) for a Milestone Release whose Conversion Event has already occurred, per Section 5.2.
**7.4 Public record of breach and resolution.** The Licensor shall cause the Trust Service to publish, as part of the public record for the affected Phase, notice of: (a) any breach notice issued under Section 7.2, stating the general nature of the breach and the date of notice; (b) whether the breach was cured within the applicable cure period, and the date of cure; and (c) any termination determination made under this Section 7, including its effective date and scope. This public record exists to give the ecosystem a transparent, verifiable conformity signal for the Phase, distinct from and in addition to the Development Credit and Remission Credit facts already published under Section 5.4 and the Target Ledger Specification.
A breach that You dispute, and that has not been finally determined, shall be recorded as **alleged**; it shall be recorded as **determined** only once the cure period has run without cure, or the dispute has been resolved against You under the applicable Commercial Use Agreement's dispute process, if any. The Trust Service shall update the record promptly upon resolution in either direction. Recording an alleged or determined breach under this Section 7.4 is a ministerial act of publishing the Licensor's determination (or a dispute process's outcome); it does not give the Trust Service discretionary authority to decide whether a breach occurred, consistent with Section 5.4's evidence-not-cause principle.
Whether, and under what conditions, the public record identifies a Commercial Entitlement holder by name is governed exclusively by the applicable Commercial Use Agreement, which the Licensor and that Commercial Entitlement holder negotiate and agree to directly. This License does not itself set a naming default. Where no Commercial Use Agreement addresses the question, or where the affected party has no Commercial Use Agreement at all (for example, a Section 2(c) breach by a Noncommercial Use licensee), the public record states the Phase and breach category only, without naming the party.
## 8. Disclaimer of Warranty
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. THE LICENSOR DOES NOT WARRANT THAT THE SOFTWARE WILL BE ERROR-FREE OR THAT ANY PHASE WILL REACH ITS CONVERSION EVENT.
## 9. Limitation of Liability
IN NO EVENT SHALL THE LICENSOR OR ANY CONTRIBUTOR BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE, EXCEPT TO THE EXTENT SUCH LIMITATION IS PROHIBITED BY APPLICABLE LAW.
## 10. Trademarks
This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary attribution.
## 11. General Provisions
**11.1 Governing law and venue.** Adopted for alpha/beta 2026-07-29: disputes arising under this License shall be resolved by binding arbitration, seated at a neutral, arbitration-mature venue (Singapore or London are the two candidates concretely supported by current research), rather than by litigation in a national court. The specific arbitral institution and substantive governing law remain a per-deployment blank pending final selection; they are not fixed by this candidate. See the canonical repository's `history/260729-TRSL-Jurisdiction-Synthesis.md` §2.
**11.2 Severability.** If any provision of this License is held unenforceable, the remaining provisions remain in full force, and the unenforceable provision shall be reformed to the minimum extent necessary to make it enforceable.
**11.3 No waiver.** Failure to enforce any provision of this License is not a waiver of future enforcement of that or any other provision.
**11.4 Entire agreement (as to licensing).** This License, together with the applicable Phase Manifest and, where applicable, the Commercial Use Agreement, constitutes the entire agreement between You and the Licensor regarding the Software's licensing terms. Operations, service, and consulting arrangements are governed by separate agreements, if any, and are not part of this License.
**11.5 Definitions control.** Marketing materials, documentation, or other non-normative communications about the Software must not describe pre-Conversion-Event Software as "Open Source," "free software," or "open core." Pre-conversion Noncommercial Use is **source-available**; pre-conversion Commercial Use requires a **Commercial Entitlement**; only post-conversion Software may be described as Open Source, under the Future License.
---
**No Phase is currently declared for this repository under this License.** Until a Phase Manifest is published and registered with the Trust Service for a Milestone Release in this repository, Sections 2–7 above have no operative subject matter here — this License establishes the governing framework in advance of that declaration, consistent with the org-wide rollout decision recorded in `target-revenue`'s `workplans/TREV-WP-0008-governance-and-pilot-rollout.md`.

3
SCOPE.md
View file

@ -68,7 +68,8 @@ failures become `error`; a failing spec never aborts the run.
reads a checkout via `WEB_FETCHER_PATH` rather than the tag
(`WEBFETCH-WP-0003-T03`).
- Running inside rein-aharness is plausible (stdlib only) but unverified.
- No CI and no licence yet (`WEBFETCH-WP-0003`).
- Licence: Target Revenue Source License V1C1 (org-wide preliminary licence).
- CI: Forgejo Actions runs the offline tests on Python 3.10 and 3.13.
## Getting Oriented

13
build/lib/web_fetcher/__init__.py Normal file
View file

@ -0,0 +1,13 @@
"""web-fetcher: reusable, provenance-first fetching of web and API data for research.
Every fetch returns Observation records that say where a value came from
(URL), when it was retrieved, and a hash of the raw payload, so downstream
research can cite and re-verify it. Values are never estimated: a fetch either
yields a sourced value or an explicit `not_found` / `error` record.
"""
from web_fetcher.core import Observation, fetch_bytes
from web_fetcher.registry import ADAPTERS, run_spec, run_specs
__all__ = ["Observation", "fetch_bytes", "ADAPTERS", "run_spec", "run_specs"]
__version__ = "0.2.0"

44
build/lib/web_fetcher/__main__.py Normal file
View file

@ -0,0 +1,44 @@
"""CLI: python -m web_fetcher fetch SPECS.json [--cache DIR | --replay DIR] [--out FILE]
SPECS.json is a JSON list of spec objects (or {"specs": [...]}). Output is
JSON Lines, one Observation per line. --replay serves every URL from a cache
written by an earlier --cache run, without network access.
"""
from __future__ import annotations
import argparse
import json
import sys
from web_fetcher.registry import run_specs
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(prog="web_fetcher")
sub = ap.add_subparsers(dest="cmd", required=True)
f = sub.add_parser("fetch", help="run fetch specs")
f.add_argument("specs")
src = f.add_mutually_exclusive_group()
src.add_argument("--cache", default=None, help="content-addressed cache directory")
src.add_argument("--replay", default=None, help="replay from a cache directory; no network")
f.add_argument("--out", default="-", help="output JSONL path (default stdout)")
f.add_argument("--only", action="append", default=[], help="run only these spec ids")
args = ap.parse_args(argv)
with open(args.specs, encoding="utf-8") as fh:
data = json.load(fh)
specs = data["specs"] if isinstance(data, dict) else data
if args.only:
specs = [s for s in specs if s.get("id") in set(args.only)]
obs = run_specs(specs, cache_dir=args.cache, replay_dir=args.replay)
lines = "\n".join(o.to_json() for o in obs) + "\n"
if args.out == "-":
sys.stdout.write(lines)
else:
with open(args.out, "w", encoding="utf-8") as fh:
fh.write(lines)
return 0 if all(o.status != "error" for o in obs) else 1
if __name__ == "__main__":
raise SystemExit(main())

1
build/lib/web_fetcher/adapters/__init__.py Normal file
View file

@ -0,0 +1 @@
"""Source adapters. Each exposes `fetch(spec, fetcher) -> list[Observation]`."""

61
build/lib/web_fetcher/adapters/eurostat.py Normal file
View file

@ -0,0 +1,61 @@
"""Eurostat dissemination API (JSON-stat 2.0).
Spec fields:
dataset Eurostat dataset code, e.g. "isoc_eb_ai"
filters dict of dimension -> code or list of codes (all non-time dimensions
should be pinned to one code except those listed in `split_by`)
split_by dimensions that may vary; one observation per combination (e.g. ["geo"])
last number of latest time periods to return (default 1)
"""
from __future__ import annotations
import itertools
import json
import urllib.parse
from typing import Any, Callable
from web_fetcher.core import Fetched, Observation
BASE = "https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/"
def build_url(spec: dict[str, Any]) -> str:
params: list[tuple[str, str]] = [("format", "JSON"), ("lang", "EN")]
for dim, val in (spec.get("filters") or {}).items():
for v in (val if isinstance(val, list) else [val]):
params.append((dim, str(v)))
params.append(("lastTimePeriod", str(int(spec.get("last", 1)))))
return BASE + spec["dataset"] + "?" + urllib.parse.urlencode(params)
def parse(doc: dict[str, Any], spec: dict[str, Any], fetched: Fetched) -> list[Observation]:
ids = doc["id"]
dims = doc["dimension"]
cats = [sorted(dims[i]["category"]["index"], key=lambda k: dims[i]["category"]["index"][k]) for i in ids]
values = doc.get("value") or {}
unit_dim = "unit" if "unit" in ids else None
out: list[Observation] = []
for n, combo in enumerate(itertools.product(*cats)):
v = values.get(str(n)) if isinstance(values, dict) else (values[n] if n < len(values) else None)
if v is None:
continue
coord = dict(zip(ids, combo))
labels = {d: coord[d] for d in spec.get("split_by", [])}
out.append(Observation(
spec_id=spec["id"], status="ok", value=v,
unit=(dims[unit_dim]["category"]["label"].get(coord[unit_dim]) if unit_dim else None),
period=coord.get("time"), labels=labels, source=f"Eurostat {spec['dataset']}: {doc.get('label')}",
url=fetched.url, published=doc.get("updated"), retrieved_at=fetched.retrieved_at,
payload_sha256=fetched.sha256,
))
if not out:
out.append(Observation(spec_id=spec["id"], status="not_found", url=fetched.url,
retrieved_at=fetched.retrieved_at, payload_sha256=fetched.sha256,
note="no values for the given filters"))
return out
def fetch(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]:
fetched = get(build_url(spec))
return parse(json.loads(fetched.body), spec, fetched)

133
build/lib/web_fetcher/adapters/http.py Normal file
View file

@ -0,0 +1,133 @@
r"""Generic HTTP adapters: JSON (dotted path) and CSV (filter + pick latest).
json spec fields: url, path ("a.b.0.c"), unit?, period_path?
csv spec fields: url, filter (dict column -> value), value_column,
period_column?, period_format?, value_type?, unit?, delimiter?
-> latest row by period
xlsx_wide fields: url, sheet, filter (dict header -> value), period_regex
(header pattern of period columns, default ^\d{6}$),
period_format?, value_type?, unit?
-> value of the matching row in the latest period column
All three take `source`; a value without one is turned into an error by the
registry. "Latest" follows `web_fetcher.periods` (period_format, default auto)
and the chosen rule is stated in the observation's note. Values must be
numeric unless value_type is "string"; otherwise the result is not_found.
"""
from __future__ import annotations
import csv
import io
import json
from typing import Any, Callable
from web_fetcher.core import Fetched, Observation
from web_fetcher.periods import order
MISSING = ("", "NA", "N/A", "-", ".", "S")
def _value(raw: str, spec: dict[str, Any]) -> tuple[Any, str | None]:
"""Parse a cell; returns (value, None) or (None, reason) for non-numeric data."""
text = raw.strip()
if spec.get("value_type") == "string":
return text, None
try:
return float(text.replace("%", "").replace(",", "")), None
except ValueError:
return None, f"non-numeric value {text!r} (set value_type: string to accept)"
def _dig(obj: Any, path: str) -> Any:
for part in path.split("."):
if obj is None:
return None
if isinstance(obj, list):
try:
obj = obj[int(part)]
except (ValueError, IndexError):
return None
else:
obj = obj.get(part) if isinstance(obj, dict) else None
return obj
def fetch_json(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]:
f = get(spec["url"])
doc = json.loads(f.body)
value = _dig(doc, spec["path"])
period = _dig(doc, spec["period_path"]) if spec.get("period_path") else spec.get("period")
status = "ok" if value is not None else "not_found"
return [Observation(spec_id=spec["id"], status=status, value=value, unit=spec.get("unit"),
period=None if period is None else str(period), source=spec.get("source"),
url=f.url, published=f.published, retrieved_at=f.retrieved_at,
payload_sha256=f.sha256,
note=None if value is not None else f"path {spec['path']} not found")]
def fetch_csv(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]:
f = get(spec["url"])
text = f.body.decode(spec.get("encoding", "utf-8-sig"), errors="replace")
rows = list(csv.DictReader(io.StringIO(text), delimiter=spec.get("delimiter", ",")))
flt = spec.get("filter") or {}
rows = [r for r in rows if all(str(r.get(k, "")).strip() == str(v) for k, v in flt.items())]
pcol, vcol = spec.get("period_column"), spec["value_column"]
rows = [r for r in rows if str(r.get(vcol) or "").strip() not in MISSING]
base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at,
payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"),
labels={k: str(v) for k, v in flt.items()})
if not rows:
return [Observation(status="not_found", note="no matching row", **base)]
if pcol:
fmt, newest = order(sorted({str(r.get(pcol) or "").strip() for r in rows}), spec.get("period_format", "auto"))
period = newest[0]
latest = [r for r in rows if str(r.get(pcol) or "").strip() == period]
if len({str(r[vcol]).strip() for r in latest}) > 1:
raise ValueError(f"{len(latest)} matching rows with different values for period {period}")
row, rule = latest[-1], f"latest of {len(newest)} periods by {fmt} order"
else:
period, row = None, rows[-1]
rule = f"no period_column: last of {len(rows)} matching rows"
value, why = _value(str(row[vcol]), spec)
if why:
return [Observation(status="not_found", period=period, note=f"{why}; {rule}", **base)]
return [Observation(status="ok", value=value, period=period, note=rule, **base)]
def fetch_xlsx_wide(spec: dict[str, Any], get: Callable[[str], Fetched]) -> list[Observation]:
import re
from web_fetcher.xlsx import read_sheet
f = get(spec["url"])
rows = read_sheet(f.body, spec.get("sheet", 0))
header, body = rows[0], rows[1:]
pat = re.compile(spec.get("period_regex", r"^\d{6}$"))
periods = {(h or "").strip(): i for i, h in enumerate(header) if pat.match(h or "")}
flt = spec.get("filter") or {}
col = {h: i for i, h in enumerate(header)}
match = [r for r in body if all(i < len(r) and r[i].strip() == str(v) for h, v in flt.items()
for i in [col.get(h, 10**9)])]
base = dict(spec_id=spec["id"], url=f.url, published=f.published, retrieved_at=f.retrieved_at,
payload_sha256=f.sha256, source=spec.get("source"), unit=spec.get("unit"),
labels={k: str(v) for k, v in flt.items()})
if not match or not periods:
return [Observation(status="not_found", note="no matching row or period columns", **base)]
if len(match) > 1:
raise ValueError(f"filter matches {len(match)} rows; expected one")
row = match[0]
fmt, newest = order(list(periods), spec.get("period_format", "auto"))
for n, h in enumerate(newest):
i = periods[h]
raw = (row[i] if i < len(row) else "").strip()
if raw in MISSING:
continue
rule = f"latest of {len(newest)} periods by {fmt} order"
if n:
rule += f", skipping {n} newer period(s) without a value"
value, why = _value(raw, spec)
if why:
return [Observation(status="not_found", period=h, note=f"{why}; {rule}", **base)]
return [Observation(status="ok", value=value, period=h, note=rule, **base)]
return [Observation(status="not_found", note="matching row has no value", **base)]

175
build/lib/web_fetcher/core.py Normal file
View file

@ -0,0 +1,175 @@
"""Core types: Observation record and provenance-tracked HTTP fetching."""
from __future__ import annotations
import hashlib
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
from typing import Any
USER_AGENT = "web-fetcher/0.2 (+https://forgejo.coulomb.social/coulomb/web-fetcher)"
@dataclass
class Observation:
"""One fetched value with its provenance. status: ok | not_found | error."""
spec_id: str
status: str
value: Any = None
unit: str | None = None
period: str | None = None
labels: dict[str, str] = field(default_factory=dict)
source: str | None = None # dataset or source name
url: str | None = None
published: str | None = None # source-side update/publication timestamp
retrieved_at: str | None = None
payload_sha256: str | None = None
note: str | None = None
def to_dict(self) -> dict[str, Any]:
return asdict(self)
def to_json(self) -> str:
return json.dumps(self.to_dict(), ensure_ascii=False, sort_keys=True)
@dataclass
class Fetched:
url: str
body: bytes
retrieved_at: str
sha256: str
content_type: str | None
published: str | None = None # HTTP Last-Modified, ISO 8601, when the server sends it
class FetchError(RuntimeError):
pass
RETRY_STATUS = {429, 500, 502, 503, 504}
MAX_RETRY_AFTER = 120.0
# Indirection points for offline tests.
_urlopen = urllib.request.urlopen
_sleep = time.sleep
def _now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
def _http_date(value: str | None) -> str | None:
if not value:
return None
try:
return parsedate_to_datetime(value).astimezone(timezone.utc).isoformat(timespec="seconds")
except (TypeError, ValueError):
return None
def _retry_after(value: str | None) -> float | None:
"""Seconds to wait from a Retry-After header (delta-seconds or HTTP date)."""
if not value:
return None
try:
return max(0.0, float(value))
except ValueError:
when = _http_date(value)
if when is None:
return None
return max(0.0, (datetime.fromisoformat(when) - datetime.now(timezone.utc)).total_seconds())
def fetch_bytes(url: str, *, timeout: float = 60.0, retries: int = 2,
cache_dir: str | os.PathLike | None = None,
replay_dir: str | os.PathLike | None = None,
headers: dict[str, str] | None = None) -> Fetched:
"""GET `url` and return body plus provenance. Optional content-addressed cache.
The cache stores each payload under its SHA-256 and an index entry per URL.
With `replay_dir`, no network request is made: the payload and its original
provenance are served from such a cache (see `replay_bytes`).
Only transient failures are retried (connection errors, timeouts, HTTP
429/5xx), with exponential backoff or the server's Retry-After; other HTTP
errors fail immediately. Provenance describes the attempt that succeeded.
"""
if replay_dir:
return replay_bytes(url, replay_dir)
hdrs = {"User-Agent": USER_AGENT, **(headers or {})}
last: Exception | None = None
for attempt in range(retries + 1):
wait = 1.5 * 2 ** attempt
try:
req = urllib.request.Request(url, headers=hdrs)
with _urlopen(req, timeout=timeout) as resp:
body = resp.read()
ctype = resp.headers.get("Content-Type")
modified = _http_date(resp.headers.get("Last-Modified"))
digest = hashlib.sha256(body).hexdigest()
fetched = Fetched(url=url, body=body, retrieved_at=_now(), sha256=digest,
content_type=ctype, published=modified)
if cache_dir:
_cache_store(Path(cache_dir), fetched)
return fetched
except urllib.error.HTTPError as exc:
if exc.code not in RETRY_STATUS:
raise FetchError(f"fetch failed for {url}: HTTP {exc.code} {exc.reason}") from exc
last = exc
hinted = _retry_after(exc.headers.get("Retry-After") if exc.headers else None)
if hinted is not None:
wait = min(hinted, MAX_RETRY_AFTER)
except (urllib.error.URLError, TimeoutError, OSError) as exc:
last = exc
if attempt < retries:
_sleep(wait)
raise FetchError(f"fetch failed for {url}: {last}")
def replay_bytes(url: str, cache_dir: str | os.PathLike) -> Fetched:
"""Serve `url` from a cache written by `fetch_bytes(cache_dir=...)`, without network.
Uses the most recent index entry for the URL and verifies the payload hash.
Returns the original retrieval time, so a replayed run reproduces the
observations of the run that filled the cache.
"""
cache = Path(cache_dir)
entry: dict[str, Any] | None = None
try:
with (cache / "index.jsonl").open(encoding="utf-8") as fh:
for line in fh:
if line.strip():
rec = json.loads(line)
if rec.get("url") == url:
entry = rec
except FileNotFoundError:
raise FetchError(f"replay cache {cache} has no index.jsonl") from None
if entry is None:
raise FetchError(f"not in replay cache: {url}")
try:
body = (cache / f"{entry['sha256']}.bin").read_bytes()
except FileNotFoundError:
raise FetchError(f"replay payload missing for {url}: {entry['sha256']}") from None
if hashlib.sha256(body).hexdigest() != entry["sha256"]:
raise FetchError(f"replay payload hash mismatch for {url}")
return Fetched(url=url, body=body, retrieved_at=entry["retrieved_at"], sha256=entry["sha256"],
content_type=entry.get("content_type"), published=entry.get("published"))
def _cache_store(cache: Path, fetched: Fetched) -> None:
cache.mkdir(parents=True, exist_ok=True)
(cache / f"{fetched.sha256}.bin").write_bytes(fetched.body)
with (cache / "index.jsonl").open("a", encoding="utf-8") as fh:
fh.write(json.dumps({"url": fetched.url, "sha256": fetched.sha256,
"retrieved_at": fetched.retrieved_at,
"content_type": fetched.content_type,
"published": fetched.published}) + "\n")

62
build/lib/web_fetcher/periods.py Normal file
View file

@ -0,0 +1,62 @@
"""Period ordering for "latest value" selection.
A source's period labels are only compared when they share one recognised
format; mixed or unrecognised formats are an error, never a silent string
comparison. Spec field `period_format` pins the format ("auto" by default);
"string" opts in to plain lexicographic order.
"""
from __future__ import annotations
import re
from typing import Callable
_FORMATS: dict[str, tuple[re.Pattern[str], Callable[[re.Match[str]], tuple[int, ...]]]] = {
"year": (re.compile(r"^(\d{4})$"), lambda m: (int(m[1]),)),
"month": (re.compile(r"^(\d{4})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]))),
"date": (re.compile(r"^(\d{4})-(\d{2})-(\d{2})$"), lambda m: (int(m[1]), int(m[2]), int(m[3]))),
"quarter": (re.compile(r"^(\d{4})-?Q([1-4])$"), lambda m: (int(m[1]), int(m[2]))),
"semester": (re.compile(r"^(\d{4})-?S([12])$"), lambda m: (int(m[1]), int(m[2]))),
"week": (re.compile(r"^(\d{4})-?W(\d{2})$"), lambda m: (int(m[1]), int(m[2]))),
# Fixed-width digit labels such as YYYYMM or Census BTOS YYYYPP.
"digits": (re.compile(r"^(\d+)$"), lambda m: (len(m[1]), int(m[1]))),
}
class PeriodError(ValueError):
pass
def detect(label: str) -> str | None:
label = label.strip()
for name, (pat, _) in _FORMATS.items():
if pat.match(label):
return name
return None
def sort_key(label: str, fmt: str) -> tuple[int, ...] | str:
if fmt == "string":
return label
pat, key = _FORMATS[fmt]
m = pat.match(label.strip())
if m is None:
raise PeriodError(f"period {label!r} does not match format {fmt!r}")
return key(m)
def order(labels: list[str], fmt: str = "auto") -> tuple[str, list[str]]:
"""Return (format, labels newest first). Raises PeriodError for mixed formats."""
if fmt == "auto":
found = {detect(lbl) for lbl in labels}
if None in found:
bad = [lbl for lbl in labels if detect(lbl) is None][:3]
raise PeriodError(f"unrecognised period labels {bad}; set period_format")
if len(found) > 1:
raise PeriodError(f"mixed period formats {sorted(found)}; set period_format")
fmt = found.pop() if found else "string"
if fmt == "digits" and len({len(lbl.strip()) for lbl in labels}) > 1:
raise PeriodError("digit period labels of different widths; set period_format")
elif fmt != "string" and fmt not in _FORMATS:
raise PeriodError(f"unknown period_format {fmt!r}")
return fmt, sorted(labels, key=lambda lbl: sort_key(lbl, fmt), reverse=True)

54
build/lib/web_fetcher/registry.py Normal file
View file

@ -0,0 +1,54 @@
"""Spec dispatch: run one or many fetch specs, never raising for a single failure."""
from __future__ import annotations
from typing import Any, Callable
from web_fetcher.adapters import eurostat, http
from web_fetcher.core import Fetched, FetchError, Observation, fetch_bytes
ADAPTERS: dict[str, Callable[[dict[str, Any], Callable[[str], Fetched]], list[Observation]]] = {
"eurostat": eurostat.fetch,
"http_json": http.fetch_json,
"http_csv": http.fetch_csv,
"xlsx_wide": http.fetch_xlsx_wide,
}
def run_spec(spec: dict[str, Any], get: Callable[[str], Fetched] | None = None,
cache_dir: str | None = None, replay_dir: str | None = None) -> list[Observation]:
"""Run one spec. Adapter errors become an `error` observation, not an exception.
A value is never returned without a source: an `ok` observation whose
adapter could not name one (the spec lacks `source`) becomes an `error`.
"""
sid = spec.get("id", "?")
kind = spec.get("adapter")
if kind == "manual":
return [Observation(spec_id=sid, status="not_found", source=spec.get("source"),
note="manual: no machine-readable source; collect at review")]
adapter = ADAPTERS.get(kind or "")
if adapter is None:
return [Observation(spec_id=sid, status="error", note=f"unknown adapter {kind!r}")]
getter = get or (lambda url: fetch_bytes(url, cache_dir=cache_dir, replay_dir=replay_dir))
try:
return [_require_source(o) for o in adapter(spec, getter)]
except FetchError as exc:
return [Observation(spec_id=sid, status="error", note=str(exc)[:300])]
except (KeyError, ValueError, TypeError) as exc:
return [Observation(spec_id=sid, status="error", note=f"parse error: {exc!s}"[:300])]
def _require_source(obs: Observation) -> Observation:
if obs.status == "ok" and not obs.source:
return Observation(spec_id=obs.spec_id, status="error", url=obs.url, retrieved_at=obs.retrieved_at,
payload_sha256=obs.payload_sha256,
note="spec has no 'source'; values are never returned without one")
return obs
def run_specs(specs: list[dict[str, Any]], **kw: Any) -> list[Observation]:
out: list[Observation] = []
for spec in specs:
out.extend(run_spec(spec, **kw))
return out

53
build/lib/web_fetcher/xlsx.py Normal file
View file

@ -0,0 +1,53 @@
"""Minimal read-only XLSX reader (stdlib only): sheet -> list of rows of strings."""
from __future__ import annotations
import io
import re
import zipfile
import xml.etree.ElementTree as ET
_NS = {"m": "http://schemas.openxmlformats.org/spreadsheetml/2006/main",
"r": "http://schemas.openxmlformats.org/officeDocument/2006/relationships"}
_T = "{%s}t" % _NS["m"]
def _col_index(ref: str) -> int:
letters = re.match(r"[A-Z]+", ref).group(0)
n = 0
for ch in letters:
n = n * 26 + (ord(ch) - 64)
return n - 1
def read_sheet(data: bytes, sheet: str | int = 0) -> list[list[str]]:
z = zipfile.ZipFile(io.BytesIO(data))
shared: list[str] = []
if "xl/sharedStrings.xml" in z.namelist():
for si in ET.fromstring(z.read("xl/sharedStrings.xml")).findall("m:si", _NS):
shared.append("".join(t.text or "" for t in si.iter(_T)))
wb = ET.fromstring(z.read("xl/workbook.xml"))
sheets = [s for s in wb.find("m:sheets", _NS)]
if isinstance(sheet, str):
idx = next(i for i, s in enumerate(sheets) if s.get("name") == sheet)
else:
idx = sheet
rid = sheets[idx].get("{%s}id" % _NS["r"])
rels = ET.fromstring(z.read("xl/_rels/workbook.xml.rels"))
target = next(r.get("Target") for r in rels if r.get("Id") == rid)
path = "xl/" + target.lstrip("/").removeprefix("xl/")
root = ET.fromstring(z.read(path))
rows: list[list[str]] = []
for r in root.find("m:sheetData", _NS).findall("m:row", _NS):
cells: dict[int, str] = {}
for c in r.findall("m:c", _NS):
v = c.find("m:v", _NS)
if v is None:
inline = c.find("m:is", _NS)
text = "".join(t.text or "" for t in inline.iter(_T)) if inline is not None else ""
else:
text = shared[int(v.text)] if c.get("t") == "s" else (v.text or "")
cells[_col_index(c.get("r"))] = text
width = max(cells) + 1 if cells else 0
rows.append([cells.get(i, "") for i in range(width)])
return rows

1
pyproject.toml
View file

@ -3,6 +3,7 @@ name = "web-fetcher"
version = "0.2.0"
description = "Provenance-first fetching of web and API data for research"
requires-python = ">=3.10"
license = { file = "LICENSE" }
dependencies = []
[project.scripts]

159
web_fetcher.egg-info/PKG-INFO Normal file
View file

@ -0,0 +1,159 @@
Metadata-Version: 2.4
Name: web-fetcher
Version: 0.2.0
Summary: Provenance-first fetching of web and API data for research
License: # Target Revenue Source License
**Version 1.0, Candidate 1 (V1C1)**
---
> **PRELIMINARY CANDIDATE — SUBJECT TO CHANGE — NOT FINAL**
>
> This repository is governed by the Target Revenue Source License
> (TRSL), Version 1, Candidate 1. This is the framework's first working
> candidate, adopted as the org's preliminary operating license across
> all repos (maintainer decision, 2026-07-29) during the build/alpha
> stage — see `SCOPE.md` §1 and Appendix A of the canonical text (link
> below) for the alpha/beta risk-acceptance decision this reflects and
> what it does and does not mean. Full specialist legal review is
> explicitly deferred until the framework moves out of beta.
>
> **Canonical source, full candidate-status banner, and Appendix A
> (non-normative candidate notes tracking every open item):** the
> `coulomb` org's `target-revenue` repository,
> `specs/TargetRevenueSourceLicense-V1C1.md` — this file is the operative
> legal text (Preamble through Section 11) only; the canonical document
> is authoritative if this copy and that document ever diverge.
---
## Preamble
This Target Revenue Source License ("**License**") governs the Software identified in the applicable Phase Manifest. It implements the Target Revenue Framework: a defined development Phase accumulates Development Credit and Remission Credit against an immutable Initial Target until the Milestone Release automatically and irrevocably converts to a declared permissive Future License.
Commercial beneficiaries fund the creation and early availability of a software improvement; once the declared target is satisfied, the governed release becomes permissively open source.
## 1. Definitions
Capitalized terms used in this License have the meanings given below. Where a term is also defined in the Phase Manifest or Target Ledger for a specific Phase, the Phase Manifest and Target Ledger govern the *values* (amounts, dates, identifiers) and this License governs the *legal effect* of those values — the two must not be read as conflicting definitions of the same concept.
**"Commercial Entitlement"** means a right, purchased or otherwise granted under a Commercial Use Agreement, to make Commercial Use of the Software during a Phase.
**"Commercial Use"** means billing, invoicing, or otherwise charging any customer a fee, subscription, license fee, or other consideration for or in connection with use of the Software, at any time before the Conversion Event for the applicable Phase, regardless of whether the person or organization billed would otherwise qualify for Noncommercial Use. Commercial Use occurs by virtue of such billing alone, whether or not the resulting payment is registered with the Trust Service; in particular, billing a customer for pre-conversion use of the Software without recording the corresponding payment in the applicable Phase's Target Ledger is Commercial Use without a valid Commercial Entitlement — a violation of Section 3, addressed under Section 7 and, where applicable, the Enforcement Network described in the canonical repository's `specs/EnforcementNetworkConcept.md`.
**"Commercial Use Agreement"** means the separate agreement, referenced by the applicable Phase Manifest, under which a Commercial Entitlement is purchased or granted. This License does not itself set pricing, metering, or payment terms — those are governed by the Commercial Use Agreement.
**"Conversion Event"** means the moment the Outstanding Target for a Phase reaches zero, as computed from the Phase Manifest and Target Ledger per the Target Ledger Specification. The Conversion Event occurs automatically and is not conditioned on any declaration, attestation, or other act by the Licensor or any Trust Service.
**"Development Credit"** means the portion of a collected and settled payment explicitly allocated toward satisfying the Initial Target of a specific Phase, as recorded in that Phase's Target Ledger.
**"Future License"** means the permissive license identified in the applicable Phase Manifest, being either the MIT License or the Apache License, Version 2.0, which applies to the Milestone Release upon the Conversion Event.
**"Initial Target"** means the immutable monetary target declared for a Phase in its Phase Manifest.
**"Licensor"** means **Binky Hedgehog GmbH**, the party that publishes the Phase Manifest and holds the rights necessary to grant this License and the Future License for the Milestone Release.
**"Milestone Release"** means the precisely identified software release designated in the applicable Phase Manifest, identified by an immutable source revision, release artifact, or cryptographic digest.
**"Noncommercial Use"** means use of the Software for personal purposes, private study, hobby or amateur projects; use by any charitable organization, educational institution, public research organization, or government institution acting in a non-revenue-generating capacity; or other use of a materially similar character.
**"Outstanding Target"** means, at any time, `max(0, Initial Target − cumulative Development Credit − cumulative Remission Credit)` for a Phase, as computed from that Phase's Target Ledger.
**"Phase"** means a bounded development undertaking governed by one Initial Target, one Milestone Release, one degeneration policy, and one Future License declaration, as declared in a Phase Manifest.
**"Phase Manifest"** means the published, immutable declaration identifying a Phase, its Milestone Release, Initial Target, Future License, degeneration policy, and Target Ledger location, as specified in the Phase Manifest Specification.
**"Remission Credit"** means a transparent, non-revenue reduction of a Phase's Outstanding Target, generated under that Phase's published degeneration policy and recorded in the Target Ledger.
**"Settled Payment"** means a payment that has cleared through its payment processor and is no longer subject to reversal in the ordinary course (chargeback, dispute, or equivalent), as further specified by the applicable Commercial Use Agreement or monetization extension.
**"Software"** means the source code, object code, and associated documentation of the Milestone Release identified in the applicable Phase Manifest.
**"Target Ledger"** means the append-only record of Development Credit, Remission Credit, and correction entries for a Phase, as specified in the Target Ledger Specification.
**"You"** or **"Licensee"** means the individual or entity exercising rights under this License.
## 2. Grant of Rights for Noncommercial Use
Subject to the terms of this License, the Licensor grants You a worldwide, royalty-free, non-exclusive license, during the applicable Phase, to:
(a) use, reproduce, and study the Software for any Noncommercial Use;
(b) modify the Software and create derivative works of it for any Noncommercial Use; and
(c) redistribute the Software and Your modifications, in source or object form, for any Noncommercial Use, provided that You include this License, unmodified, with any such redistribution, and that You do not remove or alter any copyright, patent, trademark, or attribution notices contained in the Software.
This grant does not extend to Commercial Use. Commercial Use requires a Commercial Entitlement under Section 3.
## 3. Commercial Use
You may not make Commercial Use of the Software during the applicable Phase unless You hold a valid, current Commercial Entitlement under a Commercial Use Agreement with the Licensor covering the applicable Phase. A Commercial Entitlement granted under one Phase's Commercial Use Agreement does not extend to a later Phase's Milestone Release unless the Commercial Use Agreement expressly says so.
This Section 3 states the existence and boundary of the commercial-use restriction. It does not itself set pricing, invoicing, metering, audit rights, or payment terms — those are governed exclusively by the applicable Commercial Use Agreement.
## 4. Patent License
Subject to the terms of this License, each contributor to the Software grants You, during the applicable Phase and solely to the extent of rights granted under Sections 2 and 3, a perpetual (subject to the termination below), worldwide, non-exclusive, no-charge, royalty-free patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Software, limited to those patent claims licensable by that contributor that are necessarily infringed by their contribution(s) alone or by combination of their contribution(s) with the Software.
If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Software or a contribution incorporated within it constitutes direct or contributory patent infringement, then any patent licenses granted to You under this Section 4 for the Software shall terminate as of the date such litigation is filed.
## 5. Automatic Conversion to the Future License
**5.1 Automatic effect.** Upon the Conversion Event for a Phase, the rights and restrictions in Sections 3 (Commercial Use) of this License, as they apply to that Phase's Milestone Release, terminate automatically. In their place, the Milestone Release is licensed under the Future License identified in that Phase's Phase Manifest, effective as of the Conversion Event, without any further act, declaration, or attestation required by the Licensor, any Trust Service, or any other party.
**5.2 Irrevocability.** Once a valid Conversion Event has occurred for a Phase, no subsequent refund, chargeback, accounting correction, dispute, or termination of this License for an unrelated breach shall revoke, suspend, or otherwise impair the Future License grant for that Phase's Milestone Release. Any shortfall or dispute arising after a Conversion Event is a commercial or accounting matter between the relevant parties and does not reinstate a commercial-use restriction over already-converted Software.
**5.3 Prior freedom preserved.** A later Phase covering subsequent improvements to the Software does not restrict, withdraw, or otherwise affect the rights granted under the Future License for an earlier Phase's already-converted Milestone Release.
**5.4 Evidence, not cause.** A Trust Service may publish a Conversion Attestation documenting a Conversion Event. Such an attestation is evidence that the Conversion Event occurred; it is not a condition of, and its absence or delay does not postpone, the automatic effect described in Section 5.1. Any person may independently verify whether a Conversion Event has occurred directly from the Phase Manifest and Target Ledger.
## 6. Successive Phases
The Licensor may declare a new Phase covering subsequent improvements to the Software following a Milestone Release's Conversion Event. Each Phase is independently governed by its own Phase Manifest, Initial Target, degeneration policy, and Target Ledger. Nothing in a later Phase's Phase Manifest may be construed to reduce or withdraw rights already granted under Section 5 for an earlier Phase's Milestone Release.
## 7. Term and Termination
**7.1 Term.** This License applies to the Software for the duration of the applicable Phase, and, for the Milestone Release, indefinitely following that Phase's Conversion Event under the Future License.
**7.2 Termination for breach.** If You breach Section 3 (Commercial Use) or Section 2(c) (redistribution notice requirement), the Licensor may terminate this License as to You. Before such termination becomes effective, the Licensor shall provide You written notice of the breach; if You cure the breach within thirty (30) days of that notice, this License continues in effect. A second breach of the same provision within twelve (12) months may be terminated immediately without a further cure opportunity.
**7.3 Effect of termination.** Termination under this Section 7 affects only Your rights under Sections 2 and 3 for the Phase in which the breach occurred. It does not affect any rights already vested under Section 5 (Automatic Conversion) for a Milestone Release whose Conversion Event has already occurred, per Section 5.2.
**7.4 Public record of breach and resolution.** The Licensor shall cause the Trust Service to publish, as part of the public record for the affected Phase, notice of: (a) any breach notice issued under Section 7.2, stating the general nature of the breach and the date of notice; (b) whether the breach was cured within the applicable cure period, and the date of cure; and (c) any termination determination made under this Section 7, including its effective date and scope. This public record exists to give the ecosystem a transparent, verifiable conformity signal for the Phase, distinct from and in addition to the Development Credit and Remission Credit facts already published under Section 5.4 and the Target Ledger Specification.
A breach that You dispute, and that has not been finally determined, shall be recorded as **alleged**; it shall be recorded as **determined** only once the cure period has run without cure, or the dispute has been resolved against You under the applicable Commercial Use Agreement's dispute process, if any. The Trust Service shall update the record promptly upon resolution in either direction. Recording an alleged or determined breach under this Section 7.4 is a ministerial act of publishing the Licensor's determination (or a dispute process's outcome); it does not give the Trust Service discretionary authority to decide whether a breach occurred, consistent with Section 5.4's evidence-not-cause principle.
Whether, and under what conditions, the public record identifies a Commercial Entitlement holder by name is governed exclusively by the applicable Commercial Use Agreement, which the Licensor and that Commercial Entitlement holder negotiate and agree to directly. This License does not itself set a naming default. Where no Commercial Use Agreement addresses the question, or where the affected party has no Commercial Use Agreement at all (for example, a Section 2(c) breach by a Noncommercial Use licensee), the public record states the Phase and breach category only, without naming the party.
## 8. Disclaimer of Warranty
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. THE LICENSOR DOES NOT WARRANT THAT THE SOFTWARE WILL BE ERROR-FREE OR THAT ANY PHASE WILL REACH ITS CONVERSION EVENT.
## 9. Limitation of Liability
IN NO EVENT SHALL THE LICENSOR OR ANY CONTRIBUTOR BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE, EXCEPT TO THE EXTENT SUCH LIMITATION IS PROHIBITED BY APPLICABLE LAW.
## 10. Trademarks
This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary attribution.
## 11. General Provisions
**11.1 Governing law and venue.** Adopted for alpha/beta 2026-07-29: disputes arising under this License shall be resolved by binding arbitration, seated at a neutral, arbitration-mature venue (Singapore or London are the two candidates concretely supported by current research), rather than by litigation in a national court. The specific arbitral institution and substantive governing law remain a per-deployment blank pending final selection; they are not fixed by this candidate. See the canonical repository's `history/260729-TRSL-Jurisdiction-Synthesis.md` §2.
**11.2 Severability.** If any provision of this License is held unenforceable, the remaining provisions remain in full force, and the unenforceable provision shall be reformed to the minimum extent necessary to make it enforceable.
**11.3 No waiver.** Failure to enforce any provision of this License is not a waiver of future enforcement of that or any other provision.
**11.4 Entire agreement (as to licensing).** This License, together with the applicable Phase Manifest and, where applicable, the Commercial Use Agreement, constitutes the entire agreement between You and the Licensor regarding the Software's licensing terms. Operations, service, and consulting arrangements are governed by separate agreements, if any, and are not part of this License.
**11.5 Definitions control.** Marketing materials, documentation, or other non-normative communications about the Software must not describe pre-Conversion-Event Software as "Open Source," "free software," or "open core." Pre-conversion Noncommercial Use is **source-available**; pre-conversion Commercial Use requires a **Commercial Entitlement**; only post-conversion Software may be described as Open Source, under the Future License.
---
**No Phase is currently declared for this repository under this License.** Until a Phase Manifest is published and registered with the Trust Service for a Milestone Release in this repository, Sections 2–7 above have no operative subject matter here — this License establishes the governing framework in advance of that declaration, consistent with the org-wide rollout decision recorded in `target-revenue`'s `workplans/TREV-WP-0008-governance-and-pilot-rollout.md`.
Requires-Python: >=3.10
License-File: LICENSE
Dynamic: license-file

19
web_fetcher.egg-info/SOURCES.txt Normal file
View file

@ -0,0 +1,19 @@
LICENSE
README.md
pyproject.toml
tests/test_live.py
tests/test_web_fetcher.py
web_fetcher/__init__.py
web_fetcher/__main__.py
web_fetcher/core.py
web_fetcher/periods.py
web_fetcher/registry.py
web_fetcher/xlsx.py
web_fetcher.egg-info/PKG-INFO
web_fetcher.egg-info/SOURCES.txt
web_fetcher.egg-info/dependency_links.txt
web_fetcher.egg-info/entry_points.txt
web_fetcher.egg-info/top_level.txt
web_fetcher/adapters/__init__.py
web_fetcher/adapters/eurostat.py
web_fetcher/adapters/http.py

1
web_fetcher.egg-info/dependency_links.txt Normal file
View file

@ -0,0 +1 @@

2
web_fetcher.egg-info/entry_points.txt Normal file
View file

@ -0,0 +1,2 @@
[console_scripts]
web-fetcher = web_fetcher.__main__:main

1
web_fetcher.egg-info/top_level.txt Normal file
View file

@ -0,0 +1 @@
web_fetcher

29
workplans/WEBFETCH-WP-0003-repo-hygiene.md
View file

@ -4,7 +4,7 @@ type: workplan
title: "Repository hygiene: CI, licence, deployment pinning, records"
domain: infotech
repo: web-fetcher
status: ready
status: active
flavor: implementation
depends_on: []
owner: codex
@ -23,7 +23,7 @@ scope-vs-intent work of `WEBFETCH-WP-0002`.
```task
id: WEBFETCH-WP-0003-T01
status: todo
status: progress
priority: medium
state_hub_task_id: "e0a1bf54-1446-50d7-bd4d-b43b66541c2e"
```
@ -39,7 +39,7 @@ First confirm that the Forgejo instance has runners available for
```task
id: WEBFETCH-WP-0003-T02
status: wait
status: done
priority: medium
state_hub_task_id: "f09e4b10-a955-5f61-b7bd-2ccfd610bc69"
```
@ -52,7 +52,7 @@ but has no `LICENSE`. Licence choice is the founder's decision; once made, add
```task
id: WEBFETCH-WP-0003-T03
status: todo
status: wait
priority: medium
state_hub_task_id: "cbd3081d-e544-5bdc-af32-b287e11211dd"
```
@ -66,7 +66,7 @@ run consumes a pinned tag instead of a moving checkout.
```task
id: WEBFETCH-WP-0003-T04
status: todo
status: progress
priority: low
state_hub_task_id: "d97884b7-042f-55bb-bd4c-ec6b01371f54"
```
@ -92,10 +92,27 @@ the capability.
```task
id: WEBFETCH-WP-0003-T06
status: todo
status: done
priority: low
state_hub_task_id: "9ad723d5-4548-5f28-b824-d8c888758e94"
```
Mark the two news items read and acknowledge the two standing notices
(build-mode security layer, safe secret-check helper) for agent `web-fetcher`.
## Progress (2026-10-19)
- T01: `.forgejo/workflows/test.yaml` — offline tests on `python:3.10-alpine`
and `python:3.13-alpine` containers (`ubuntu-latest` label, archive fetch as in
key-cape's acceptance job). Runners confirmed active (audit-core, key-cape
runs in October). Registry validation stays local (needs a reuse-surface
install).
- T02: Org-wide preliminary licence applied (Target Revenue Source License
V1C1; maintainer decision 2026-07-29 "across all repos"), byte-identical to
the 111 other repos; `license = { file = "LICENSE" }` in `pyproject.toml`.
- T03: Waiting on assembly-economics — handoff message `f276e74a` (2026-10-19)
asks them to pin `v0.2.0`. Agents do not touch the railiance01 checkout directly.
- T05: reuse-surface commits `be5c006` and `307fcaf` (federated refresh to
v0.2.0) are ready locally; the push to reuse-surface `main` was blocked by
the permission guard and needs the founder to run it or allow it.
- T06: Both standing notices acknowledged; inbox empty. Neither affects this repo.