Checkpoint executor: deterministic runner, sources, tests; definitions route to assembly-checkpoint

- checkpoints/run_checkpoint.py: window guard, web-fetcher fetch with
  provenance, record + raw JSONL, optional llm-connect assessment rejected if
  it contains numbers not in the fetched data
- checkpoints/sources.json: BTOS (US), Eurostat isoc_eb_ai (EU, DE), FRED
  info-processing investment (US); manual entries for AR, usage indices,
  sentiment, legislation
- activity definitions: label assembly-checkpoint (rein-aharness approach)
- PLAYBOOK: automated path + manual/agent path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1494246@bnt-lap001
Assistant-Session: caf6de6f-f9e4-47ce-a584-49baa271fa4b
This commit is contained in:
tegwick 2026-10-12 18:45:12 +02:00
parent 72feabf573
commit c0254aa2da
8 changed files with 287 additions and 48 deletions

View file

@ -21,9 +21,9 @@ projection: `activity-core/k8s/railiance/20-runtime.yaml`
Steady state: full checkpoint on 20 Jan from 2027.
Execution: rein-aharness claims the ops_run and runs the `agent-session`
approach in the `assembly-economics` checkout. The session follows
`checkpoints/PLAYBOOK.md`; the window guard in `checkpoints/schedule.md` is
Execution: rein-aharness claims the ops_run and runs the `assembly-checkpoint`
approach in the `assembly-economics` checkout: it runs
`checkpoints/run_checkpoint.py` (data via web-fetcher); the window guard in `checkpoints/schedule.md` is
authoritative for dates. Completion: State Hub progress event
`assembly_checkpoint` with `detail.repo`, `detail.date`, `detail.kind`,
`detail.path`.
@ -33,15 +33,12 @@ id: emit-assembly-checkpoint-annual
action:
task_template: "Assembly Economics reality checkpoint (full) — steady phase"
description: >
Run a full reality checkpoint for assembly-economics. Follow
checkpoints/PLAYBOOK.md exactly: window check against
checkpoints/schedule.md first; collect the full indicators from
checkpoints/IndicatorCatalog.md with sources (never invent values);
compare against checkpoints/predictions.md; write
checkpoints/YYYY/YYYY-MM-DD-full.md; append observations; commit and
push; post progress event_type=assembly_checkpoint with detail.repo,
detail.date, detail.kind, detail.path.
Run the reality checkpoint runner for assembly-economics
(checkpoints/run_checkpoint.py via the rein-aharness assembly-checkpoint
approach). Window guard: checkpoints/schedule.md. Data via web-fetcher
with provenance; never invented values. Writes checkpoints/YYYY/ and
checkpoints/data/, pushes, posts event_type=assembly_checkpoint.
target_repo: assembly-economics
priority: medium
labels: ["agent-session", "assembly-checkpoint", "research", "automated"]
labels: ["assembly-checkpoint", "research", "automated"]
```

View file

@ -21,9 +21,9 @@ projection: `activity-core/k8s/railiance/20-runtime.yaml`
Transition phase: light checkpoints on 15 Nov and 15 Dec 2026. Fires every year; the window guard makes later years no-ops, and the definition is disabled after 2026.
Execution: rein-aharness claims the ops_run and runs the `agent-session`
approach in the `assembly-economics` checkout. The session follows
`checkpoints/PLAYBOOK.md`; the window guard in `checkpoints/schedule.md` is
Execution: rein-aharness claims the ops_run and runs the `assembly-checkpoint`
approach in the `assembly-economics` checkout: it runs
`checkpoints/run_checkpoint.py` (data via web-fetcher); the window guard in `checkpoints/schedule.md` is
authoritative for dates. Completion: State Hub progress event
`assembly_checkpoint` with `detail.repo`, `detail.date`, `detail.kind`,
`detail.path`.
@ -33,15 +33,12 @@ id: emit-assembly-checkpoint-monthly-2026
action:
task_template: "Assembly Economics reality checkpoint (light) — transition phase"
description: >
Run a light reality checkpoint for assembly-economics. Follow
checkpoints/PLAYBOOK.md exactly: window check against
checkpoints/schedule.md first; collect the light indicators from
checkpoints/IndicatorCatalog.md with sources (never invent values);
compare against checkpoints/predictions.md; write
checkpoints/YYYY/YYYY-MM-DD-light.md; append observations; commit and
push; post progress event_type=assembly_checkpoint with detail.repo,
detail.date, detail.kind, detail.path.
Run the reality checkpoint runner for assembly-economics
(checkpoints/run_checkpoint.py via the rein-aharness assembly-checkpoint
approach). Window guard: checkpoints/schedule.md. Data via web-fetcher
with provenance; never invented values. Writes checkpoints/YYYY/ and
checkpoints/data/, pushes, posts event_type=assembly_checkpoint.
target_repo: assembly-economics
priority: medium
labels: ["agent-session", "assembly-checkpoint", "research", "automated"]
labels: ["assembly-checkpoint", "research", "automated"]
```

View file

@ -21,9 +21,9 @@ projection: `activity-core/k8s/railiance/20-runtime.yaml`
Steady state: light checkpoints on 15 Jan, Apr, Jul and Oct from 2027.
Execution: rein-aharness claims the ops_run and runs the `agent-session`
approach in the `assembly-economics` checkout. The session follows
`checkpoints/PLAYBOOK.md`; the window guard in `checkpoints/schedule.md` is
Execution: rein-aharness claims the ops_run and runs the `assembly-checkpoint`
approach in the `assembly-economics` checkout: it runs
`checkpoints/run_checkpoint.py` (data via web-fetcher); the window guard in `checkpoints/schedule.md` is
authoritative for dates. Completion: State Hub progress event
`assembly_checkpoint` with `detail.repo`, `detail.date`, `detail.kind`,
`detail.path`.
@ -33,15 +33,12 @@ id: emit-assembly-checkpoint-quarterly
action:
task_template: "Assembly Economics reality checkpoint (light) — steady phase"
description: >
Run a light reality checkpoint for assembly-economics. Follow
checkpoints/PLAYBOOK.md exactly: window check against
checkpoints/schedule.md first; collect the light indicators from
checkpoints/IndicatorCatalog.md with sources (never invent values);
compare against checkpoints/predictions.md; write
checkpoints/YYYY/YYYY-MM-DD-light.md; append observations; commit and
push; post progress event_type=assembly_checkpoint with detail.repo,
detail.date, detail.kind, detail.path.
Run the reality checkpoint runner for assembly-economics
(checkpoints/run_checkpoint.py via the rein-aharness assembly-checkpoint
approach). Window guard: checkpoints/schedule.md. Data via web-fetcher
with provenance; never invented values. Writes checkpoints/YYYY/ and
checkpoints/data/, pushes, posts event_type=assembly_checkpoint.
target_repo: assembly-economics
priority: medium
labels: ["agent-session", "assembly-checkpoint", "research", "automated"]
labels: ["assembly-checkpoint", "research", "automated"]
```

View file

@ -21,9 +21,9 @@ projection: `activity-core/k8s/railiance/20-runtime.yaml`
Setup phase: daily light checkpoints on 13-15 Oct 2026 to verify the pipeline end to end. Fires on 13-15 Oct of every year; the executor's window guard (checkpoints/schedule.md) turns later years into no-ops, and the definition is disabled after 2026 (ASSEMBLY-WP-0010-T04).
Execution: rein-aharness claims the ops_run and runs the `agent-session`
approach in the `assembly-economics` checkout. The session follows
`checkpoints/PLAYBOOK.md`; the window guard in `checkpoints/schedule.md` is
Execution: rein-aharness claims the ops_run and runs the `assembly-checkpoint`
approach in the `assembly-economics` checkout: it runs
`checkpoints/run_checkpoint.py` (data via web-fetcher); the window guard in `checkpoints/schedule.md` is
authoritative for dates. Completion: State Hub progress event
`assembly_checkpoint` with `detail.repo`, `detail.date`, `detail.kind`,
`detail.path`.
@ -33,15 +33,12 @@ id: emit-assembly-checkpoint-setup-daily
action:
task_template: "Assembly Economics reality checkpoint (light) — setup phase"
description: >
Run a light reality checkpoint for assembly-economics. Follow
checkpoints/PLAYBOOK.md exactly: window check against
checkpoints/schedule.md first; collect the light indicators from
checkpoints/IndicatorCatalog.md with sources (never invent values);
compare against checkpoints/predictions.md; write
checkpoints/YYYY/YYYY-MM-DD-light.md; append observations; commit and
push; post progress event_type=assembly_checkpoint with detail.repo,
detail.date, detail.kind, detail.path.
Run the reality checkpoint runner for assembly-economics
(checkpoints/run_checkpoint.py via the rein-aharness assembly-checkpoint
approach). Window guard: checkpoints/schedule.md. Data via web-fetcher
with provenance; never invented values. Writes checkpoints/YYYY/ and
checkpoints/data/, pushes, posts event_type=assembly_checkpoint.
target_repo: assembly-economics
priority: medium
labels: ["agent-session", "assembly-checkpoint", "research", "automated"]
labels: ["assembly-checkpoint", "research", "automated"]
```

View file

@ -1,5 +1,18 @@
# Checkpoint playbook (executor instructions)
**Automated path (default).** Scheduled checkpoints run without an agent:
activity-core fires an `assembly-checkpoint-*` definition, rein-aharness
claims it (approach `assembly-checkpoint`), runs
`checkpoints/run_checkpoint.py` (data via
[web-fetcher](https://forgejo.coulomb.social/coulomb/web-fetcher), optional
llm-connect assessment with a number guard), commits only `checkpoints/`,
pushes and posts `assembly_checkpoint`. Machine-readable sources are mapped in
`checkpoints/sources.json`. Indicators marked `manual` are collected by an
agent or human at reviews, following the steps below.
**Manual / agent path.** Use the steps below for manual indicators, annual
reviews, or when the automated path fails.
You are running a **reality checkpoint** for the `assembly-economics`
repository. Work only inside this repository. Keep the run within budget:
a `light` checkpoint should take well under 15 minutes.

View file

@ -0,0 +1,165 @@
"""Deterministic reality-checkpoint runner (ASSEMBLY-WP-0010).
1. Window check against checkpoints/schedule.md (authoritative dates).
2. Fetch the indicators of this kind from checkpoints/sources.json with
web-fetcher (provenance: URL, retrieval time, payload hash).
3. Write checkpoints/data/YYYY-MM-DD-<kind>.jsonl (raw observations) and
checkpoints/YYYY/YYYY-MM-DD-<kind>.md (record).
4. Optional assessment via llm-connect (LLM_CONNECT_URL). The model only sees
fetched observations; any number in its text that is not among them makes
the assessment rejected (no invented values).
Usage:
python3 checkpoints/run_checkpoint.py --kind light [--date YYYY-MM-DD] [--no-llm]
Requires web-fetcher importable (pip install, or WEB_FETCHER_PATH=/path/to/web-fetcher).
Exit codes: 0 ok, 3 out of window, 1 failure.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import urllib.request
from datetime import date, datetime
from pathlib import Path
from zoneinfo import ZoneInfo
HERE = Path(__file__).resolve().parent
REPO = HERE.parent
if os.environ.get("WEB_FETCHER_PATH"):
sys.path.insert(0, os.environ["WEB_FETCHER_PATH"])
WINDOW_RE = re.compile(r"(\d{4}-\d{2}-\d{2})\s*[–-]\s*(\d{4}-\d{2}-\d{2})|from\s+(\d{4}-\d{2}-\d{2})")
def berlin_today() -> date:
return datetime.now(ZoneInfo("Europe/Berlin")).date()
def active_phase(day: date, kind: str, schedule: str) -> str | None:
"""Return the phase name whose window for `kind` contains `day`, else None."""
for line in schedule.splitlines():
cells = [c.strip().strip("`") for c in line.strip().strip("|").split("|")]
if len(cells) < 3 or cells[1] != kind:
continue
m = WINDOW_RE.search(cells[2])
if not m:
continue
if m.group(3):
if day >= date.fromisoformat(m.group(3)):
return cells[0]
elif date.fromisoformat(m.group(1)) <= day <= date.fromisoformat(m.group(2)):
return cells[0]
return None
def fetch(kind: str) -> list[dict]:
from web_fetcher.registry import run_spec
specs = json.loads((HERE / "sources.json").read_text())["specs"]
out = []
for spec in specs:
if kind == "light" and spec.get("kind") != "light":
continue
for obs in run_spec(spec):
d = obs.to_dict()
d.update(indicator=spec.get("indicator"), jurisdiction=spec.get("jurisdiction"),
label=spec.get("label"), source=d.get("source") or spec.get("source"))
out.append(d)
return out
def model_version() -> str:
readme = (REPO / "models" / "README.md").read_text()
return "GADM-0.4" if "GADM-0.4" in readme else "unknown"
def _numbers(text: str) -> set[str]:
return {n.rstrip(".") for n in re.findall(r"\d+(?:[.,]\d+)?", text)}
def assessment(observations: list[dict]) -> tuple[str | None, str | None]:
url = os.environ.get("LLM_CONNECT_URL", "").rstrip("/")
if not url:
return None, "no LLM_CONNECT_URL; assessment skipped"
ok = [o for o in observations if o["status"] == "ok"]
facts = "\n".join(f"- {o['indicator']} {o['jurisdiction']} {o.get('labels') or ''}: {o['label']} = "
f"{o['value']} {o.get('unit') or ''} (period {o['period']})" for o in ok)
prompt = (
"You assess AI adoption observations for a research checkpoint. Use ONLY the facts below. "
"Do not add any number, date or statistic that is not literally in the facts. "
"Write 2-4 short bullet points: what the observations say about the pace of AI adoption per "
"jurisdiction (US, EU, DE), and which differences stand out. No forecasts.\n\nFacts:\n" + facts)
body = json.dumps({"prompt": prompt, "config": {"temperature": 0.1, "max_tokens": 500}}).encode()
try:
req = urllib.request.Request(url + "/execute", data=body, headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=180) as resp:
text = (json.loads(resp.read()).get("content") or "").strip()
except Exception as exc: # network / server errors must not fail the checkpoint
return None, f"assessment failed: {exc}"[:200]
allowed = _numbers(facts)
extra = sorted(n for n in _numbers(text) if n not in allowed and n.replace(",", ".") not in allowed)
if extra:
return None, f"assessment rejected: numbers not in fetched data: {extra[:8]}"
return text, None
def render(day: date, kind: str, phase: str, obs: list[dict], assess: str | None, assess_note: str | None,
data_rel: str) -> str:
rows, missing = [], []
for o in obs:
if o["status"] == "ok":
geo = (o.get("labels") or {}).get("geo")
jur = geo or o["jurisdiction"]
pub = (o.get("published") or "")[:10] or "—"
rows.append(f"| {o['indicator']} {o['label']} | {jur} | {o['value']} | {o.get('unit') or ''} | "
f"{o['period']} | {pub} | [source]({o['url']}) | no prediction registered |")
else:
missing.append(f"- {o['indicator']} {o['label']} ({o['jurisdiction']}): {o['status']}"
f" — {o.get('note') or ''} Source: {o.get('source') or '—'}")
lines = [f"# Checkpoint {day.isoformat()} ({kind})", "",
f"- Model version: {model_version()}", f"- Phase: {phase}",
"- Executor: checkpoints/run_checkpoint.py (web-fetcher; llm-connect assessment)",
f"- Raw observations with provenance: `{data_rel}`", "",
"## Observations", "",
"| Indicator | Jurisdiction | Value | Unit | Reference period | Published | Source | Band position |",
"|---|---|---|---|---|---|---|---|", *rows, "", "## Assessment", ""]
lines += [assess] if assess else [f"_No automated assessment: {assess_note}_"]
lines += ["", "## Not found / manual", "", *(missing or ["- none"]), "",
"## Follow-ups", "", "- Predictions are not yet registered (ASSEMBLY-WP-0009-T06); band positions",
" are reported once the register has entries.", ""]
return "\n".join(lines)
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--kind", choices=["light", "full"], required=True)
ap.add_argument("--date", default=None)
ap.add_argument("--no-llm", action="store_true")
args = ap.parse_args(argv)
day = date.fromisoformat(args.date) if args.date else berlin_today()
phase = active_phase(day, args.kind, (HERE / "schedule.md").read_text())
if phase is None:
print(json.dumps({"status": "out-of-window", "date": day.isoformat(), "kind": args.kind}))
return 3
obs = fetch(args.kind)
data_rel = f"checkpoints/data/{day.isoformat()}-{args.kind}.jsonl"
(REPO / data_rel).parent.mkdir(parents=True, exist_ok=True)
(REPO / data_rel).write_text("\n".join(json.dumps(o, ensure_ascii=False, sort_keys=True) for o in obs) + "\n")
assess, note = (None, "disabled (--no-llm)") if args.no_llm else assessment(obs)
rec_rel = f"checkpoints/{day.year}/{day.isoformat()}-{args.kind}.md"
(REPO / rec_rel).parent.mkdir(parents=True, exist_ok=True)
(REPO / rec_rel).write_text(render(day, args.kind, phase, obs, assess, note, data_rel))
summary = {"status": "ok", "date": day.isoformat(), "kind": args.kind, "phase": phase, "path": rec_rel,
"data": data_rel, "ok": sum(o["status"] == "ok" for o in obs),
"not_found": sum(o["status"] == "not_found" for o in obs),
"error": sum(o["status"] == "error" for o in obs), "assessment": bool(assess)}
print(json.dumps(summary))
return 0
if __name__ == "__main__":
raise SystemExit(main())

37
checkpoints/sources.json Normal file
View file

@ -0,0 +1,37 @@
{
"note": "Machine-readable sources for checkpoint indicators (checkpoints/IndicatorCatalog.md). Fetched with web-fetcher. 'manual' = no machine-readable source yet; collected at reviews.",
"specs": [
{"id": "L1-US", "indicator": "L1", "kind": "light", "jurisdiction": "US",
"label": "Firms using AI in any business function (last two weeks)",
"adapter": "xlsx_wide", "url": "https://www.census.gov/hfp/btos/downloads/National.xlsx",
"sheet": "Response Estimates", "filter": {"Question ID": "7", "Answer ID": "1"},
"unit": "% of employer firms", "source": "U.S. Census Bureau, Business Trends and Outlook Survey (BTOS)"},
{"id": "L1-US-next6m", "indicator": "L1", "kind": "light", "jurisdiction": "US",
"label": "Firms expecting to use AI in the next six months",
"adapter": "xlsx_wide", "url": "https://www.census.gov/hfp/btos/downloads/National.xlsx",
"sheet": "Response Estimates", "filter": {"Question ID": "24", "Answer ID": "1"},
"unit": "% of employer firms", "source": "U.S. Census Bureau, Business Trends and Outlook Survey (BTOS)"},
{"id": "L2-EU-DE", "indicator": "L2", "kind": "light", "jurisdiction": "EU, DE",
"label": "Enterprises (10+ employees) using at least one AI technology",
"adapter": "eurostat", "dataset": "isoc_eb_ai",
"filters": {"geo": ["EU27_2020", "DE"], "unit": "PC_ENT", "size_emp": "GE10", "indic_is": "E_AI_TANY"},
"split_by": ["geo"], "last": 1},
{"id": "L3-AR", "indicator": "L3", "kind": "light", "jurisdiction": "AR",
"label": "AI use by firms or workers", "adapter": "manual",
"source": "INDEC / national ICT surveys; OECD / IDB regional data"},
{"id": "L4", "indicator": "L4", "kind": "light", "jurisdiction": "global",
"label": "AI task usage by occupation; automation vs. augmentation", "adapter": "manual",
"source": "Anthropic Economic Index (dataset releases)"},
{"id": "L5-US", "indicator": "L5", "kind": "light", "jurisdiction": "US",
"label": "Private fixed investment in information processing equipment (SAAR)",
"adapter": "http_csv", "url": "https://fred.stlouisfed.org/graph/fredgraph.csv?id=Y033RC1Q027SBEA",
"value_column": "Y033RC1Q027SBEA", "period_column": "observation_date",
"unit": "billions of USD, SAAR", "source": "BEA via FRED (Y033RC1Q027SBEA)"},
{"id": "L6", "indicator": "L6", "kind": "light", "jurisdiction": "US, EU, AR",
"label": "Public sentiment toward AI", "adapter": "manual",
"source": "Pew Research; Eurobarometer; Edelman Trust Barometer; Latinobarómetro"},
{"id": "L7", "indicator": "L7", "kind": "light", "jurisdiction": "US, EU, AR",
"label": "AI legislation milestones since last checkpoint", "adapter": "manual",
"source": "official journals; legislative trackers"}
]
}

View file

@ -0,0 +1,36 @@
"""Tests for the checkpoint runner (offline). Usage:
python3 -B -m unittest discover -s checkpoints -p 'test_*.py'
"""
import unittest
from datetime import date
from pathlib import Path
import run_checkpoint as rc
SCHEDULE = (Path(__file__).parent / "schedule.md").read_text()
class TestWindow(unittest.TestCase):
def test_monthly_and_steady_windows(self):
self.assertEqual(rc.active_phase(date(2026, 11, 15), "light", SCHEDULE), "transition")
self.assertEqual(rc.active_phase(date(2027, 4, 15), "light", SCHEDULE), "steady")
self.assertEqual(rc.active_phase(date(2027, 1, 20), "full", SCHEDULE), "steady")
def test_out_of_window(self):
self.assertIsNone(rc.active_phase(date(2026, 10, 1), "light", SCHEDULE))
self.assertIsNone(rc.active_phase(date(2026, 11, 15), "full", SCHEDULE))
def test_parser_handles_ranges(self):
sched = "| setup | `light` | 2026-10-14 – 2026-10-16 | daily | x |\n"
self.assertEqual(rc.active_phase(date(2026, 10, 15), "light", sched), "setup")
self.assertIsNone(rc.active_phase(date(2026, 10, 17), "light", sched))
class TestNumberGuard(unittest.TestCase):
def test_numbers_extraction(self):
self.assertEqual(rc._numbers("EU at 19.95 % in 2025, US 23.8."), {"19.95", "2025", "23.8"})
if __name__ == "__main__":
unittest.main()