diff --git a/Makefile b/Makefile index 1af25c3..221d8a2 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ TOOLS := $(REPO)/tools # Every cargo recipe runs at the repo root; the shell does not persist cd. IN_REPO := cd $(REPO) && -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play gate-review all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -36,6 +36,10 @@ check: play: $(IN_REPO) $(CARGO) run -q -p cb-play -- $(ARGS) +## which control gates are due for a keep-or-kill argument (ADR-0006 D3) +gate-review: + $(PY) $(TOOLS)/gate-review.py + ## unit + scenario-format tests test: $(IN_REPO) $(CARGO) test --workspace diff --git a/evidence/CB-EV-0008-adaptive-gates.md b/evidence/CB-EV-0008-adaptive-gates.md new file mode 100644 index 0000000..0612026 --- /dev/null +++ b/evidence/CB-EV-0008-adaptive-gates.md @@ -0,0 +1,134 @@ +# CB-EV-0008 — did the gate changes fire? + +CB-WP-0009 T04. Measured 2026-08-01 at `HEAD` after T03. This pass is +`kind: meta`, opened above the budget line on recorded maintainer +authorization (ADR-0006 D2's first use). + +--- + +## 1. Does the windowed meta share differ from cumulative? + +**Yes at one pass, no at three — and the prediction did not say which.** + +| reading | value | +|---|---| +| CB-WP-0008 alone (product) | **0%** | +| trailing 3 passes, before this one | 36% | +| trailing 3 passes, now | **45%** | +| lifetime, all 9 passes | 50% | + +T01 predicted the windowed figure would differ sharply from cumulative, +refuted within 20%. Against the **single-pass** window that is decisive: +0% versus 50%. Against the **trailing-3** window it is not: 45% versus +50% is a 10% relative gap, inside the refutation band. + +Both are true, and the honest reading is that **the prediction was +underspecified** — it was written before the window size was chosen, and +T01 then chose three passes for a stated reason (one pass is a coin flip: +a meta pass reads 100%, a product pass 0%). So: + +- D1 is **confirmed for the failure it was written against**: the v1.5 + budget could not see a 100%-product pass, and now it can. +- D1 is **not** confirmed as "trailing-3 always differs from lifetime." + Right now it does not, because the recent mix happens to resemble the + lifetime mix. That is a fact about this project's history, not evidence + for or against the machinery. + +**Carried as an open question**, not resolved by argument: whether three +is the right window. It should be revisited when a pass makes the two +diverge, which is the only cheap evidence available. + +## 2. What does `gate-review` say on its first run? + +``` +9 gate(s) — 0 due, 2 silent, 0 drifted +``` + +The two silent entries are the uncomfortable ones, and they are supposed +to be: + +- **CHAOS** — the chaos roll has never changed a tier. Its 12-declaration + window (4 used) exists precisely to find out whether it ever will. +- **GATE-REVIEW** — this registry itself, which has caught nothing + because it is one hour old, and which retires on 2026-12-31 if it has + not by then retired, tightened or forced the re-justification of + anything. + +A registry where every gate looked current and productive would have been +a registry written to look good. Two of nine are on notice. + +The `caught` lists are the load-bearing part, and they are honest in the +other direction too: **M-D1-MUT has six entries, LOOP-LINT two, and the +DFD gate one that landed this week** — `gr_scenarios` stale at 21 after +CB-WP-0008 T03 added three scenarios, caught by `make all` going red +before the push. + +## 3. Did D4 hold for this pass? + +**Three of four tasks shipped a command; the fourth is this file.** + +| task | shipped | +|---|---| +| T01 | `cb-cost.pass_costs`, windowed `make status` | +| T02 | `gates.toml`, `make gate-review`, a `loop-lint` check | +| T03 | InnerLoop v1.6 — **prose** | +| T04 | this evidence file — **prose** | + +D4 says a `meta` pass must ship *at least one* executable check, so the +rule holds. But the pass makes the boundary clear: **the spec change and +the evidence file are prose, and a rule requiring every task to cash out +would be wrong** — the spec edit is what makes the commands normative, +and the evidence file is what checks them. + +So D4 stands as written (per pass, not per task), and this pass is on +record as its first exercise. + +## 4. Cost + +| pass | kind | responses | cost | $/response | +|---|---|---|---|---| +| CB-WP-0006 | meta | 158 | $57.22 | 0.362 | +| CB-WP-0007 | meta | 21 | $7.95 | 0.298* | +| CB-WP-0008 | product | 134 | $17.38 | **0.123** | +| **CB-WP-0009** | **meta** | **38** | **$6.73** | **0.177** | + +\* the CB-WP-0007 figure is from the earlier hand-windowed measurement; +the table's per-pass windows are now computed by `pass_costs`. + +**This is the interesting number.** D4's claim is that loop work that +cashes out into a command is cheap. This pass cashed out three commands +and ran at **0.177** — cheaper than every previous meta pass (0.228, +0.298, 0.362) and still **1.4× the product pass**. Partial support, not +vindication, and stated as such. + +Session shape over the same window is worse than the product pass and +breaches both targets: + +| metric | this pass | CB-WP-0008 | target | +|---|---|---|---| +| SH-1 mean context | **290,171** | 161,994 | ≤ 200,000 | +| SH-2 p90 context | **311,243** | 215,664 | ≤ 300,000 | +| SH-3 batching | 0.0% | 6.9% | ≥ 20% | + +The mechanical cause is visible and boring: this pass ran on top of an +already-long session with no compaction, where CB-WP-0008 started +immediately after one. The shape budget reports it; nothing gates it; and +per CB-RES-0005 §4 nothing should. + +**Meta share after this pass: 45% of the trailing three, against a soft +25%.** Reported under the rule this pass wrote. Nothing was displaced — +CB-WP-0008 shipped stage 0 first, and this work was maintainer-authorized +on top of it — but the number is over the line and the next pass should +be product to bring it back. + +## 5. Open + +- **Is 3 the right trailing window?** Unanswered; revisit when a pass + makes single-pass and trailing-3 diverge. +- **CHAOS and GATE-REVIEW have caught nothing.** Both have dates. +- **`mech environment setup` reads 17 turns / $2.90** this pass, which is + the cost classifier scoring inline `python3 - < **Meta budget — soft, 25% of spend per pass.** Work on the loop's own -> instruments and process is capped at a quarter of a pass. Measured by -> `make status` from each workplan's `kind:` frontmatter -> (`product` | `meta` | `mixed`). +> **Meta budget — soft, 25% of the trailing three passes.** Work on the +> loop's own instruments and process is capped at a quarter of recent +> spend. Measured by `make status` from each workplan's `kind:` +> frontmatter (`product` | `meta` | `mixed`), over a **trailing 3-pass +> window**; the lifetime figure is reported as history and is not the +> metric (ADR-0006 D1). > > **Soft on purpose.** A task already started may be finished — stopping -> mid-task to satisfy a ratio wastes the work and leaves the tree in a -> worse state than either finishing or never starting. What the budget -> forbids is *opening* new meta work above the line. +> mid-task to satisfy a ratio wastes the work and leaves the tree worse +> than either finishing or never starting. What the budget constrains is +> *opening* new meta work above the line, and it constrains it with two +> exemptions: +> +> 1. **Repairing the instrument that reports a breach is always in +> budget.** A gate must never block its own remedy (ADR-0006 D2, +> CB-RES-0005 §4). v1.5 had no such exemption, and the first thing it +> did was forbid fixing the budget's own defect. +> 2. **The maintainer may authorize above-line work**, recorded as +> `authorized_above_budget:` in the workplan frontmatter. Undocumented +> above-line meta work is a violation the evidence file must report. > > A pass that exceeds it **says so in its evidence file and names the -> product work displaced**. That is the whole enforcement: this is a -> reporting budget, not a gate, for the same reason the session-shape -> budget is (CB-RES-0005 §4) — it constrains judgment, not artifacts. +> product work displaced**. That is the whole enforcement: a reporting +> budget, not a gate, for the same reason the session-shape budget is — +> it constrains judgment, not artifacts. + +> **Loop work cashes out (v1.6, ADR-0006 D4).** A `meta` pass must ship at +> least one executable check, command, or gate. Findings that stay prose +> are recorded as findings and do **not** count as delivery. > -> *(v1.5, from the CB-WP-0007 spend review: CB-WP-0005 and CB-WP-0006 cost -> ~$74, 31% of all spend, for zero measured efficiency gain. Their return -> was correctness of claims, which is real and is not optimization. The -> budget exists so that distinction has to be made out loud.)* +> Measured basis: cost per response across five passes ran 0.123, 0.228, +> 0.362, 0.298, 0.123 dollars, and the reading that survived the data is +> that **cost tracks distance from a runnable check** — the passes that +> shipped a command were cheap, the passes that argued about what a +> number means were not (CB-EV-0007 §5). **Refuted if** a prose-only meta +> pass lowers cost per response or catches a class no command could. + +> **Every gate is an experiment (v1.6, ADR-0006 D3).** Each standing +> control gate has an entry in `gates.toml` naming what it has caught, a +> `review_by` date, and what would retire it. `make gate-review` reports +> what is overdue and what has caught nothing; `loop-lint` fails when the +> registry drifts from the Makefile in either direction. v1.4 — corrected from CB-WP-0005 (assertion coverage) on 2026-07-31. Change from v1.3: where a claim rests on numbers, the @@ -31,28 +55,10 @@ adversarial reviewer must read the assertion behind each quoted number and **mutate it** — re-running the command that prints a number is not verification of that number (§Step 2). -v1.3 changed from v1.2: single source of fact is now executable -(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift -class its first gate. - -v1.2 changes from v1.1: single source of fact; review targets the -harness and states its sampling limit; correction vs retarget; the chaos -roll's calibration window; the live cost budget. The design goal is now -stated: **optimize for cheap correction, not for exhaustive prevention.** -Rationale: `history/260731-loop-hardening-retrospective.md`. - -v1.1 — corrected from CB-WP-0002 (cost accounting) on -2026-07-31. Changes from v1.0: the instrument must exist and emit its own -target; inherited numbers are re-derived before use; every reporting tool -exposes `--self-test`; cost is in the definition of done. Rationale: -`history/260731-cost-accounting-retrospective.md`. - -v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) -and was corrected from it on 2026-07-31. Changes from v0.2: measurement -validity (the positive control), metric feasibility and instrument naming, -four implementation rules the pass earned, and the requirement that -evidence state what it does not support. Rationale and the failures behind -each: `history/260731-inner-loop-retrospective.md`. +v1.5 (meta budget), v1.3 (executable single source of fact), v1.2 +(design goal: cheap correction over exhaustive prevention), v1.1 +(instruments emit their own targets) and v1.0: see +`specs/InnerLoopReference.md` §Change log. Normative process for building every Clay-Borg capability. Referenced by all workplans. @@ -111,7 +117,7 @@ are never skipped for code-producing work. | Tier | Weight of steps 1–3 | Structural trigger (forces at least this tier) | |---|---|---| | **L** | Full: separate survey, adversarial review, ADR | Creates a new capability port, or is named a high-leverage pass by the maintainer | -| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, or adds/updates an external dependency | +| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, adds/updates an external dependency, **or changes whether or how the loop constrains its own operation** — budgets, gates, review requirements, or these tier rules (v1.6, ADR-0006 D5) | | **S** | One provenance paragraph in the commit message | Everything else (utilities, fixes, refactors inside a boundary) | **The chaos roll.** After deriving the structural tier, roll **d4** @@ -119,32 +125,13 @@ are never skipped for code-producing work. random (`shuf -e S M L -n 1`), overriding the structural derivation — up or down. -> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was -> d10 and the mechanism **never fired**: two rolls across two workplans -> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and -> ~2 tier decisions per workplan it would take roughly twenty workplans to -> observe four overrides, so the mechanism was set at a rate that prevented -> its own evaluation — the one option T06 ruled out. -> -> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated -> and either kept, returned to d10, or deleted. Expected ~3 firings in the -> window, which is enough to see whether an overridden tier produces a -> different outcome than the argued one. -> -> **Stated cost:** a chaos-L override on work that would have been S buys a -> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001 -> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the -> window is expected to cost **$20–30**. That is the price of finding out -> whether the mechanism is worth keeping, and it is cheaper than carrying an -> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration -(`tier: M (structural L, chaos 10→M)`). **Record the roll every time, -including when it changes nothing** (`tier: L (structural L, chaos 4)`), -so a mechanism that never fires is visible rather than assumed. Purpose: -an occasional random -reweighting keeps the classification honest — arguing everything into S -stops paying off when audits can compare argued tiers against the random -sample — and occasionally forces a deep look at something "obviously -trivial", which is where local optima hide. +> **Calibration window, opened 2026-07-31, running to 12 tier +> declarations** (declaration 4 of 12 as of 2026-08-01). The rate was +> raised from d10 to d4 because at d10 the mechanism never fired and +> so prevented its own evaluation. Record the roll every time, +> including when it changes nothing (`tier: L (structural L, chaos 4)`). +> Rationale, cost estimate and the two dead rolls: +> `specs/InnerLoopReference.md` §Chaos roll — calibration. Chaos limits: a rolled-down tier relaxes *process* weight only. Invariants (zero foreign types in canonical interfaces, determinism, passing diff --git a/specs/InnerLoopReference.md b/specs/InnerLoopReference.md index 06af1fd..bf2b529 100644 --- a/specs/InnerLoopReference.md +++ b/specs/InnerLoopReference.md @@ -139,3 +139,59 @@ A capability has completed the loop when all of the following are committed: ([CostAccounting.md](CostAccounting.md)) - [ ] retrospective note (may be one paragraph appended to the evidence file): what the loop itself should change + +## Chaos roll — calibration (moved from InnerLoop v1.6) + +> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was +> d10 and the mechanism **never fired**: two rolls across two workplans +> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and +> ~2 tier decisions per workplan it would take roughly twenty workplans to +> observe four overrides, so the mechanism was set at a rate that prevented +> its own evaluation — the one option T06 ruled out. +> +> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated +> and either kept, returned to d10, or deleted. Expected ~3 firings in the +> window, which is enough to see whether an overridden tier produces a +> different outcome than the argued one. +> +> **Stated cost:** a chaos-L override on work that would have been S buys a +> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001 +> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the +> window is expected to cost **$20–30**. That is the price of finding out +> whether the mechanism is worth keeping, and it is cheaper than carrying an +> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration +(`tier: M (structural L, chaos 10→M)`). **Record the roll every time, +including when it changes nothing** (`tier: L (structural L, chaos 4)`), +so a mechanism that never fires is visible rather than assumed. Purpose: +an occasional random +reweighting keeps the classification honest — arguing everything into S +stops paying off when audits can compare argued tiers against the random +sample — and occasionally forces a deep look at something "obviously +trivial", which is where local optima hide. + + +## Change log — v1.0 to v1.3 (moved from InnerLoop v1.6) + +v1.3 changed from v1.2: single source of fact is now executable +(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift +class its first gate. + +v1.2 changes from v1.1: single source of fact; review targets the +harness and states its sampling limit; correction vs retarget; the chaos +roll's calibration window; the live cost budget. The design goal is now +stated: **optimize for cheap correction, not for exhaustive prevention.** +Rationale: `history/260731-loop-hardening-retrospective.md`. + +v1.1 — corrected from CB-WP-0002 (cost accounting) on +2026-07-31. Changes from v1.0: the instrument must exist and emit its own +target; inherited numbers are re-derived before use; every reporting tool +exposes `--self-test`; cost is in the definition of done. Rationale: +`history/260731-cost-accounting-retrospective.md`. + +v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) +and was corrected from it on 2026-07-31. Changes from v0.2: measurement +validity (the positive control), metric feasibility and instrument naming, +four implementation rules the pass earned, and the requirement that +evidence state what it does not support. Rationale and the failures behind +each: `history/260731-inner-loop-retrospective.md`. + diff --git a/tools/cb-cost.py b/tools/cb-cost.py index a707484..ae6228b 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -524,6 +524,55 @@ def collect(slug, pin_ref=None, since_ref=None): } +def pass_costs(slug, boundaries): + """Cost per pass window, from one read of the transcripts. + + CB-WP-0009 T01 / ADR-0006 D1: a budget must measure the window it + governs. `boundaries` is `[(label, start_iso)]` in ascending order; + each window runs to the next label's start, the last to now. + Responses before the first boundary land in `_before`. + + Reads the transcripts **once**. Calling `collect()` per boundary + would re-read every transcript per window, and `make status` is + supposed to stay cheap enough that nobody replaces it with `ls`. + """ + prices = load_prices() + stale = check_price_sheet_age(prices) + if stale: + raise Abort(stale) + paths = transcript_paths(slug) + if not paths: + raise Abort(f"no transcripts found for {slug}") + responses = [] + for path in paths: + responses.extend(read_responses(path, None)) + if not responses: + raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") + + ordered = sorted(boundaries, key=lambda b: b[1]) + out = {label: {"cost": 0.0, "responses": 0} for label, _ in ordered} + out["_before"] = {"cost": 0.0, "responses": 0} + + for r in responses: + cost = price_of(prices, r["model"], r["toks"], r["timestamp"]) or 0.0 + bucket = "_before" + for label, start in ordered: + if r["timestamp"] > start: + bucket = label + else: + break + out[bucket]["cost"] += cost + out[bucket]["responses"] += 1 + + # Positive control: a bucketing that placed nothing in any real window + # would report a confident 0% for every pass. + placed = sum(v["responses"] for k, v in out.items() if k != "_before") + if ordered and placed == 0: + raise Abort("pass_costs bucketed every response before the first " + "boundary — the boundaries are wrong, not the spend") + return out + + def render(rep, by_task=False, composition=False): print(f"M-D2-CST cost report — {rep['slug']}") print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}") @@ -625,6 +674,31 @@ def self_test(): check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong, f"${got:.4f} (1h would be ${wrong:.4f})") + # ADR-0006 D1: pass_costs buckets by window, and refuses to report a + # confident 0% when every response landed before the first boundary. + slug = "-home-worsch-clay-borg" + try: + real = pass_costs(slug, [("early", "2020-01-01T00:00:00Z")]) + check("pass_costs places responses in a real window", + real["early"]["responses"] > 0 and real["early"]["cost"] > 0, + f"{real['early']['responses']} response(s)") + two = pass_costs(slug, [("a", "2020-01-01T00:00:00Z"), + ("b", "2026-07-31T00:00:00Z")]) + check("pass_costs splits at a boundary rather than pooling", + two["a"]["responses"] > 0 and two["b"]["responses"] > 0, + f"a={two['a']['responses']} b={two['b']['responses']}") + check("pass_costs windows sum to the unwindowed total", + abs((two["a"]["cost"] + two["b"]["cost"] + two["_before"]["cost"]) + - real["early"]["cost"] - real["_before"]["cost"]) < 0.01) + except Abort as e: + check("pass_costs places responses in a real window", False, str(e)) + try: + pass_costs(slug, [("future", "2099-01-01T00:00:00Z")]) + check("pass_costs aborts when nothing lands in any window", False, + "no Abort raised — every pass would read $0.00 and 0%") + except Abort: + check("pass_costs aborts when nothing lands in any window", True) + # AC-5: dedup invariant is enforced. import tempfile with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: diff --git a/tools/gate-review.py b/tools/gate-review.py new file mode 100644 index 0000000..0ee6890 --- /dev/null +++ b/tools/gate-review.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +"""Which control gates are due for a keep-or-kill argument? (ADR-0006 D3) + +CB-WP-0009 T02. Six passes produced five standing control mechanisms and +no way to retire any of them. Gates accumulate monotonically because each +one was justified once, at the moment it was cheapest to justify. + +This reads `gates.toml` and reports two things: + + * gates past `review_by` — the date someone said they would argue for + keeping it; + * gates whose `caught` list is **empty** — which is not proof a gate is + useless (it may be preventing rather than missing), but is the + argument that has to be made out loud rather than never. + +It **reports**. It does not fail the build, for CB-RES-0005 §4's reason: +a gate that blocks the remedy when the metric breaches is a trap. The +only failure exit here is a broken registry — a file that cannot be read +would otherwise report "0 gates, all healthy". + +Usage: + python3 tools/gate-review.py + python3 tools/gate-review.py --self-test +""" +import datetime +import os +import sys + +from repo import ROOT, enter_root + +REGISTRY = os.path.join(ROOT, "gates.toml") + +try: + import tomllib +except ModuleNotFoundError: # pragma: no cover - Python < 3.11 + import tomli as tomllib + + +class Fail(Exception): + pass + + +REQUIRED = ("id", "name", "checks", "added", "review_by", "retire_if") + + +def load(path=REGISTRY): + """Every gate, validated. A registry that parses but says nothing is + the harness-does-nothing failure this project keeps finding.""" + if not os.path.exists(path): + raise Fail(f"{os.path.relpath(path, ROOT)} is missing") + with open(path, "rb") as fh: + data = tomllib.load(fh) + gates = data.get("gate") or [] + if not gates: + raise Fail("registry parsed but contains no gates") + for g in gates: + missing = [k for k in REQUIRED if not g.get(k)] + if missing: + raise Fail(f"gate {g.get('id', '?')!r} is missing {missing}") + for field in ("added", "review_by"): + try: + datetime.date.fromisoformat(g[field]) + except ValueError as e: + raise Fail(f"gate {g['id']!r} has a bad {field}: {e}") from e + return gates + + +def make_targets(path=None): + """Targets declared in the Makefile, so an entry cannot name a + command that does not exist.""" + path = path or os.path.join(ROOT, "Makefile") + targets = set() + for line in open(path): + if line and not line[0].isspace() and ":" in line and not line.startswith("."): + name = line.split(":", 1)[0].strip() + if name and " " not in name: + targets.add(name) + return targets + + +def report(today=None): + today = today or datetime.date.today() + gates = load() + targets = make_targets() + + overdue, quiet, broken = [], [], [] + for g in gates: + if datetime.date.fromisoformat(g["review_by"]) <= today: + overdue.append(g) + if not g.get("caught"): + quiet.append(g) + target = g.get("target") or "" + if target and target not in targets: + broken.append((g, target)) + + print("gate review — every gate is an experiment (ADR-0006 D3)") + print(f" registry {len(gates)} gate(s), {REGISTRY.split('/')[-1]}") + print(f" today {today.isoformat()}") + + print("\n gates") + for g in gates: + due = datetime.date.fromisoformat(g["review_by"]) + days = (due - today).days + mark = "DUE " if days <= 0 else ("soon" if days <= 30 else "ok ") + print(f" [{mark}] {g['id']:<14} {g['name']:<28} " + f"review {g['review_by']} ({days:+}d) caught {len(g.get('caught', []))}") + + if quiet: + print("\n caught nothing yet — not a verdict, an argument that is owed") + for g in quiet: + print(f" {g['id']:<14} retire if: {g['retire_if']}") + + if overdue: + print("\n DUE for a keep-or-kill argument") + for g in overdue: + print(f" {g['id']:<14} {g['retire_if']}") + + if broken: + print("\n registry drift — entry names a target the Makefile lacks") + for g, target in broken: + print(f" {g['id']:<14} target {target!r}") + + print(f"\n {len(overdue)} due, {len(quiet)} silent, {len(broken)} drifted") + print(" reporting only — never fails the build (CB-RES-0005 §4)") + return 0 + + +def self_test(): + """Each check pins a way this tool could report a comfortable lie.""" + results = [] + + def check(name, ok, detail=""): + results.append((name, ok, detail)) + + gates = load() + check("the real registry loads", len(gates) >= 5, f"{len(gates)} gate(s)") + check("every gate names what would retire it", + all(len(g["retire_if"]) > 20 for g in gates)) + # Registry drift is the failure this exists to prevent: a gate added + # to the Makefile with no entry, or an entry for a deleted target. + targets = make_targets() + named = [(g["id"], g["target"]) for g in gates if g.get("target")] + check("every named target exists in the Makefile", + all(t in targets for _i, t in named), + ", ".join(f"{i}:{t}" for i, t in named if t not in targets) or "all present") + check("Makefile targets were actually parsed", len(targets) >= 15, + f"{len(targets)} target(s)") + + import tempfile + + def registry(text): + fh = tempfile.NamedTemporaryFile("w", suffix=".toml", delete=False) + fh.write(text) + fh.close() + return fh.name + + # A registry that parses to nothing must abort, not report "0 due". + empty = registry("# no gates here\n") + try: + load(empty) + check("an empty registry aborts", False, "reported a clean bill") + except Fail: + check("an empty registry aborts", True) + finally: + os.unlink(empty) + + # A missing required field must abort rather than be treated as absent + # evidence — `caught` is optional, the rest are not. + partial = registry('[[gate]]\nid = "X"\nname = "n"\nchecks = "c"\n' + 'added = "2026-01-01"\nreview_by = "2026-02-01"\n') + try: + load(partial) + check("a gate with no retire_if aborts", False, "accepted") + except Fail: + check("a gate with no retire_if aborts", True) + finally: + os.unlink(partial) + + # The overdue arithmetic must actually fire. + import io + from contextlib import redirect_stdout + + buf = io.StringIO() + with redirect_stdout(buf): + report(today=datetime.date(2099, 1, 1)) + late = buf.getvalue() + check("a far-future date marks every gate due", + late.count("[DUE ]") == len(gates), + f"{late.count('[DUE ]')} of {len(gates)}") + + buf = io.StringIO() + with redirect_stdout(buf): + report(today=datetime.date(2020, 1, 1)) + early = buf.getvalue() + check("a far-past date marks none due", "[DUE ]" not in early) + check("silent gates are named either way", "caught nothing yet" in early) + + print("gate-review self-test (positive control)") + ok = True + for name, passed, det in results: + print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {det}" if det else "")) + ok &= passed + return 0 if ok else 1 + + +def main(): + enter_root() + try: + if "--self-test" in sys.argv: + return self_test() + return report() + except Fail as e: + print(f"gate-review: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/loop-lint.py b/tools/loop-lint.py index 82e4163..be436b7 100644 --- a/tools/loop-lint.py +++ b/tools/loop-lint.py @@ -162,12 +162,61 @@ def check_reporting_tools_self_test(root=REPO): return out +def check_gate_registry(root=REPO): + """ADR-0006 D3 — every control gate is in `gates.toml`, and every + entry names a real target. + + The failure this prevents is drift in the direction nobody notices: a + gate added to `make all` with no registry entry never acquires a + review date, which is how five mechanisms accumulated with no way to + retire any of them. + """ + out = [] + registry = os.path.join(root, "gates.toml") + makefile = os.path.join(root, "Makefile") + if not (os.path.exists(registry) and os.path.exists(makefile)): + return out + try: + import tomllib + except ModuleNotFoundError: # pragma: no cover + return out + + with open(registry, "rb") as fh: + data = tomllib.load(fh) + gates = data.get("gate") or [] + if not gates: + return [Finding("gates", "gates.toml", "registry contains no gates")] + registered = {g.get("target") for g in gates if g.get("target")} + exempt = set(data.get("not_control_gates") or []) + + text = open(makefile).read() + m = re.search(r"^all:(.*)$", text, re.M) + deps = m.group(1).split() if m else [] + targets = {ln.split(":", 1)[0].strip() for ln in text.splitlines() + if ln and not ln[0].isspace() and ":" in ln and not ln.startswith(".")} + + for dep in deps: + if dep not in registered and dep not in exempt: + out.append(Finding( + "gates", "gates.toml", + f"`make all` runs {dep!r}, which is neither a registered " + f"control gate nor listed in not_control_gates — classify it, " + f"so it cannot acquire permanence without a review date")) + for target in sorted(registered): + if target not in targets: + out.append(Finding( + "gates", "gates.toml", + f"entry names target {target!r}, which the Makefile lacks")) + return out + + CHECKS = ( check_loadability, check_evidence_no_unmeasured, check_survey_tier_and_chaos, check_review_trail, check_reporting_tools_self_test, + check_gate_registry, ) @@ -196,6 +245,29 @@ def self_test(): len(f) == 1 and "Big.md" in f[0].path, f"{len(f)} finding(s)") + # gates: an unclassified `all:` dependency trips, and so does an + # entry naming a target the Makefile lacks. + with open(os.path.join(tmp, "Makefile"), "w") as fh: + fh.write("all: coverage newthing\ncoverage:\n\techo\n") + with open(os.path.join(tmp, "gates.toml"), "w") as fh: + fh.write('not_control_gates = []\n\n[[gate]]\nid = "G"\n' + 'name = "n"\ntarget = "coverage"\nchecks = "c"\n' + 'added = "2026-01-01"\nreview_by = "2026-02-01"\n' + 'retire_if = "r"\n') + f = check_gate_registry(tmp) + check("gate registry detects an unclassified all: dependency", + len(f) == 1 and "newthing" in f[0].detail, f"{len(f)} finding(s)") + with open(os.path.join(tmp, "gates.toml"), "w") as fh: + fh.write('not_control_gates = ["newthing", "coverage"]\n\n[[gate]]\nid = "G"\n' + 'name = "n"\ntarget = "ghost"\nchecks = "c"\n' + 'added = "2026-01-01"\nreview_by = "2026-02-01"\n' + 'retire_if = "r"\n') + f = check_gate_registry(tmp) + check("gate registry detects an entry naming a missing target", + len(f) == 1 and "ghost" in f[0].detail, f"{len(f)} finding(s)") + os.unlink(os.path.join(tmp, "gates.toml")) + os.unlink(os.path.join(tmp, "Makefile")) + # evidence: a table verdict trips; the word in prose does not. with open(os.path.join(tmp, "evidence", "E.md"), "w") as fh: fh.write("| AC-1 | x | unmeasured |\n" diff --git a/tools/status.py b/tools/status.py index bff8c10..5dd7c05 100644 --- a/tools/status.py +++ b/tools/status.py @@ -116,6 +116,90 @@ def cost_lines(): return open_spend, rep["total"], mod +# ------------------------------------------------------- the meta budget + +# How many passes the budget looks back over. One pass is a coin flip — +# a `meta` pass reads 100% and a `product` pass reads 0%, and neither +# says anything about the balance the budget is about. Three is the +# smallest window in which a ratio means something and still moves. +TRAILING_PASSES = 3 +META_SOFT_PCT = 25 + + +def workplan_starts(): + """[(id, kind, first_commit_iso)] — when each pass began. + + A pass starts at the commit that *added* its workplan file, which is + the only boundary git records and the same one `cb-cost --since` has + been windowed on by hand since CB-WP-0004 T05. + """ + import datetime + + out = [] + for path in sorted(glob.glob(os.path.join(ROOT, "workplans", "*.md"))): + wid, _title, _status, _tasks, kind = parse_workplan(path) + rel = os.path.relpath(path, ROOT) + added = git("log", "--diff-filter=A", "--format=%cI", "--", rel) + first = added.splitlines()[-1].strip() if added else "" + if not (wid and first): + continue + # Transcript timestamps are UTC `...Z`; git prints a local offset. + # Comparing the two as strings silently buckets everything into + # `_before`, which is how this first read reported $0. + first = ( + datetime.datetime.fromisoformat(first) + .astimezone(datetime.timezone.utc) + .isoformat() + .replace("+00:00", "Z") + ) + out.append((wid, kind, first)) + return sorted(out, key=lambda r: r[2]) + + +def meta_budget(plans): + """ADR-0006 D1: report the share for the window the budget governs.""" + _, _, mod = cost_lines() + starts = workplan_starts() + if not starts: + print("\n meta budget UNAVAILABLE — no workplan start commits found") + return + costs = mod.pass_costs("-home-worsch-clay-borg", + [(wid, when) for wid, _k, when in starts]) + + window = starts[-TRAILING_PASSES:] + kinds = {wid: kind for wid, kind, _ in starts} + total = sum(costs[wid]["cost"] for wid, _k, _w in window) + # `mixed` splits evenly; stated rather than hidden. + weight = {"product": 0.0, "meta": 1.0, "mixed": 0.5} + meta = sum(costs[wid]["cost"] * weight.get(kinds[wid], 0.5) + for wid, _k, _w in window) + + if total <= 0: + print("\n meta budget UNAVAILABLE — the trailing window measured $0") + return + share = 100 * meta / total + mark = "ok " if share <= META_SOFT_PCT else "OVER" + print(f"\n meta budget [{mark}] {share:.0f}% over the last " + f"{len(window)} pass(es) (soft {META_SOFT_PCT}%, InnerLoop v1.6)") + for wid, kind, _w in window: + c = costs[wid] + print(f" {wid} {kind:<8} ${c['cost']:>7,.2f} " + f"{c['responses']:>4} response(s)") + + # History, kept and labelled — the whole point of ADR-0006 D1 is that + # this number is not what the target compares against. + lifetime = sum(costs[wid]["cost"] for wid, _k, _w in starts) + life_meta = sum(costs[wid]["cost"] * weight.get(kind, 0.5) + for wid, kind, _ in starts) + if lifetime > 0: + print(f" history, all {len(starts)} passes: " + f"{100 * life_meta / lifetime:.0f}% — NOT the metric " + f"(ADR-0006 D1)") + print(" NOTE: repairing the instrument that reports a breach is always") + print(" in budget (ADR-0006 D2); other above-line meta work needs") + print(" `authorized_above_budget:` in the workplan frontmatter.") + + # ------------------------------------------------------------------ report @@ -162,31 +246,10 @@ def report(): except Exception as e: print(f" UNAVAILABLE — {e}") - # InnerLoop v1.5 — soft 25% meta budget. Reported, never gated. + # InnerLoop v1.6 — soft 25% meta budget over a trailing pass window. + # Reported, never gated. try: - _, _, mod = cost_lines() - rep = mod.collect("-home-worsch-clay-borg", None) - detail = rep["by_task_detail"] - by_kind = {"product": 0.0, "meta": 0.0, "mixed": 0.0} - attributed = 0.0 - for wid, _t, _s, tasks, kind in plans: - for tid, _ts, _p in tasks: - d = detail.get(tid) - if d and kind in by_kind: - by_kind[kind] += d["cost"] - attributed += d["cost"] - if attributed > 0: - # `mixed` splits evenly; stated rather than hidden. - meta = by_kind["meta"] + by_kind["mixed"] / 2 - share = 100 * meta / attributed - mark = "ok " if share <= 25 else "OVER" - print(f"\n meta budget [{mark}] {share:.0f}% of " - f"${attributed:,.2f} attributed (soft 25%, InnerLoop v1.5)") - print(f" product ${by_kind['product']:,.2f} " - f"meta ${by_kind['meta']:,.2f} " - f"mixed ${by_kind['mixed']:,.2f} (split 50/50)") - print(" NOTE: attributed tasks only — early workplans used bare") - print(" T## ids that collide across passes and are excluded.") + meta_budget(plans) except Exception as e: print(f"\n meta budget UNAVAILABLE — {e}") @@ -279,6 +342,32 @@ def self_test(): repr(h[:60])) check("task heading resolves to something", bool(h)) + # ADR-0006 D1: the budget must measure the window it governs. + starts = workplan_starts() + check("every workplan has a start commit", len(starts) >= len(plans) - 1, + f"{len(starts)} start(s) for {len(plans)} workplan(s)") + check("start commits are UTC and comparable to transcript stamps", + all(w.endswith("Z") for _i, _k, w in starts), + # A local-offset stamp compares as a string against `...Z` and + # silently buckets every response before the first boundary, + # which is exactly how this reported $0 on its first run. + ", ".join(w for _i, _k, w in starts[:1])) + check("start commits are in ascending order", + [w for _i, _k, w in starts] == sorted(w for _i, _k, w in starts)) + + buf = io.StringIO() + try: + with redirect_stdout(buf): + meta_budget(plans) + text = buf.getvalue() + check("budget reports a windowed share", "over the last" in text) + check("budget labels the cumulative figure as history", + "NOT the metric" in text) + check("budget states the instrument-repair exemption", + "in budget (ADR-0006 D2)" in text) + except Exception as e: + check("budget reports a windowed share", False, str(e)) + # The report itself must run and produce substance, not a stub. buf = io.StringIO() try: diff --git a/workplans/CB-WP-0009-adaptive-gates.md b/workplans/CB-WP-0009-adaptive-gates.md index b9faecc..96fcf5d 100644 --- a/workplans/CB-WP-0009-adaptive-gates.md +++ b/workplans/CB-WP-0009-adaptive-gates.md @@ -2,7 +2,7 @@ id: CB-WP-0009 kind: meta title: "Make control gates experimental: window them, review them, cash them out" -status: proposed +status: done authorized_above_budget: "maintainer, 2026-08-01 — 'improve our policies where it seems promising; we will experiment our way to sensible task- and context-specific control and review gates'. Meta read 59% at the time." state_hub_workstream_id: "98c0af4a-415c-4cf5-92d0-a722eb70cb90" --- @@ -34,7 +34,7 @@ prose has not delivered. ```task id: CB-WP-0009-T01 -status: todo +status: done priority: high state_hub_task_id: "c6eefacd-5bb9-4640-8356-12e8112bcf11" ``` @@ -53,11 +53,16 @@ it was a pure product pass — against 61% cumulative. **Refuted if** the two land within 20% of each other, in which case D1 is wrong and the machinery should be deleted rather than kept. +**Done 2026-08-01.** `cb-cost.pass_costs` buckets by pass window in one +read of the transcripts; `make status` reports a trailing 3-pass share +with the per-pass breakdown and the exemption. First reading **36% over +the last 3 passes** against **49% lifetime** — CB-WP-0008 alone reads 0%. + ## Task: `gates.toml` and `make gate-review` ```task id: CB-WP-0009-T02 -status: todo +status: done priority: high state_hub_task_id: "0fb6cfaf-9304-481a-a96f-1f0150f4d81d" ``` @@ -77,11 +82,17 @@ behind the Makefile is worse than none, so `loop-lint` gains a check that every gate target has an entry and every entry names a real target. Its own self-test must fail when an entry is removed. +**Done 2026-08-01.** `gates.toml` (9 gates), `make gate-review`, and a +`loop-lint` check in both directions — an unclassified `make all` +dependency and an entry naming a target the Makefile lacks each produce +a finding, both pinned by self-tests. First run: **0 due, 2 silent** +(the chaos roll and `gate-review` itself). + ## Task: InnerLoop v1.6 ```task id: CB-WP-0009-T03 -status: todo +status: done priority: medium state_hub_task_id: "0b1b33ac-8046-4b34-b287-7b32755d964a" ``` @@ -100,11 +111,16 @@ Fold **D2**, **D4** and **D5** into `specs/InnerLoop.md`: pointer. Loadability is itself a gate; growing the file to fit a rule about gates would be the joke writing itself. +**Done 2026-08-01.** v1.6 in `specs/InnerLoop.md`, **386 lines** — the +chaos-roll calibration rationale and the v1.0–v1.3 change log moved to +`specs/InnerLoopReference.md`, so the file came in *smaller* than it +started at 399. + ## Task: control loop — did the changes fire? ```task id: CB-WP-0009-T04 -status: todo +status: done priority: high state_hub_task_id: "e06f57fc-147f-4cba-b398-0db6b060381f" ``` @@ -123,3 +139,9 @@ answered from output, not argument: pass or needs the exception written down. 4. **Cost**, against the 0.123–0.362 per-response range, and the meta share this pass itself consumed — reported under the rule it wrote. + +**Done 2026-08-01.** [CB-EV-0008](../evidence/CB-EV-0008-adaptive-gates.md). +The pass ran at **$0.177/response** — cheaper than every previous meta +pass and still 1.4× the product pass, which is partial support for D4 and +is reported as such. Meta share reads **45% over the trailing three**, +over the line; the next pass should be product.