diff --git a/Makefile b/Makefile index 221d8a2..1af25c3 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ TOOLS := $(REPO)/tools # Every cargo recipe runs at the repo root; the shell does not persist cd. IN_REPO := cd $(REPO) && -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play gate-review all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -36,10 +36,6 @@ check: play: $(IN_REPO) $(CARGO) run -q -p cb-play -- $(ARGS) -## which control gates are due for a keep-or-kill argument (ADR-0006 D3) -gate-review: - $(PY) $(TOOLS)/gate-review.py - ## unit + scenario-format tests test: $(IN_REPO) $(CARGO) test --workspace diff --git a/evidence/CB-EV-0008-adaptive-gates.md b/evidence/CB-EV-0008-adaptive-gates.md deleted file mode 100644 index 0612026..0000000 --- a/evidence/CB-EV-0008-adaptive-gates.md +++ /dev/null @@ -1,134 +0,0 @@ -# CB-EV-0008 — did the gate changes fire? - -CB-WP-0009 T04. Measured 2026-08-01 at `HEAD` after T03. This pass is -`kind: meta`, opened above the budget line on recorded maintainer -authorization (ADR-0006 D2's first use). - ---- - -## 1. Does the windowed meta share differ from cumulative? - -**Yes at one pass, no at three — and the prediction did not say which.** - -| reading | value | -|---|---| -| CB-WP-0008 alone (product) | **0%** | -| trailing 3 passes, before this one | 36% | -| trailing 3 passes, now | **45%** | -| lifetime, all 9 passes | 50% | - -T01 predicted the windowed figure would differ sharply from cumulative, -refuted within 20%. Against the **single-pass** window that is decisive: -0% versus 50%. Against the **trailing-3** window it is not: 45% versus -50% is a 10% relative gap, inside the refutation band. - -Both are true, and the honest reading is that **the prediction was -underspecified** — it was written before the window size was chosen, and -T01 then chose three passes for a stated reason (one pass is a coin flip: -a meta pass reads 100%, a product pass 0%). So: - -- D1 is **confirmed for the failure it was written against**: the v1.5 - budget could not see a 100%-product pass, and now it can. -- D1 is **not** confirmed as "trailing-3 always differs from lifetime." - Right now it does not, because the recent mix happens to resemble the - lifetime mix. That is a fact about this project's history, not evidence - for or against the machinery. - -**Carried as an open question**, not resolved by argument: whether three -is the right window. It should be revisited when a pass makes the two -diverge, which is the only cheap evidence available. - -## 2. What does `gate-review` say on its first run? - -``` -9 gate(s) — 0 due, 2 silent, 0 drifted -``` - -The two silent entries are the uncomfortable ones, and they are supposed -to be: - -- **CHAOS** — the chaos roll has never changed a tier. Its 12-declaration - window (4 used) exists precisely to find out whether it ever will. -- **GATE-REVIEW** — this registry itself, which has caught nothing - because it is one hour old, and which retires on 2026-12-31 if it has - not by then retired, tightened or forced the re-justification of - anything. - -A registry where every gate looked current and productive would have been -a registry written to look good. Two of nine are on notice. - -The `caught` lists are the load-bearing part, and they are honest in the -other direction too: **M-D1-MUT has six entries, LOOP-LINT two, and the -DFD gate one that landed this week** — `gr_scenarios` stale at 21 after -CB-WP-0008 T03 added three scenarios, caught by `make all` going red -before the push. - -## 3. Did D4 hold for this pass? - -**Three of four tasks shipped a command; the fourth is this file.** - -| task | shipped | -|---|---| -| T01 | `cb-cost.pass_costs`, windowed `make status` | -| T02 | `gates.toml`, `make gate-review`, a `loop-lint` check | -| T03 | InnerLoop v1.6 — **prose** | -| T04 | this evidence file — **prose** | - -D4 says a `meta` pass must ship *at least one* executable check, so the -rule holds. But the pass makes the boundary clear: **the spec change and -the evidence file are prose, and a rule requiring every task to cash out -would be wrong** — the spec edit is what makes the commands normative, -and the evidence file is what checks them. - -So D4 stands as written (per pass, not per task), and this pass is on -record as its first exercise. - -## 4. Cost - -| pass | kind | responses | cost | $/response | -|---|---|---|---|---| -| CB-WP-0006 | meta | 158 | $57.22 | 0.362 | -| CB-WP-0007 | meta | 21 | $7.95 | 0.298* | -| CB-WP-0008 | product | 134 | $17.38 | **0.123** | -| **CB-WP-0009** | **meta** | **38** | **$6.73** | **0.177** | - -\* the CB-WP-0007 figure is from the earlier hand-windowed measurement; -the table's per-pass windows are now computed by `pass_costs`. - -**This is the interesting number.** D4's claim is that loop work that -cashes out into a command is cheap. This pass cashed out three commands -and ran at **0.177** — cheaper than every previous meta pass (0.228, -0.298, 0.362) and still **1.4× the product pass**. Partial support, not -vindication, and stated as such. - -Session shape over the same window is worse than the product pass and -breaches both targets: - -| metric | this pass | CB-WP-0008 | target | -|---|---|---|---| -| SH-1 mean context | **290,171** | 161,994 | ≤ 200,000 | -| SH-2 p90 context | **311,243** | 215,664 | ≤ 300,000 | -| SH-3 batching | 0.0% | 6.9% | ≥ 20% | - -The mechanical cause is visible and boring: this pass ran on top of an -already-long session with no compaction, where CB-WP-0008 started -immediately after one. The shape budget reports it; nothing gates it; and -per CB-RES-0005 §4 nothing should. - -**Meta share after this pass: 45% of the trailing three, against a soft -25%.** Reported under the rule this pass wrote. Nothing was displaced — -CB-WP-0008 shipped stage 0 first, and this work was maintainer-authorized -on top of it — but the number is over the line and the next pass should -be product to bring it back. - -## 5. Open - -- **Is 3 the right trailing window?** Unanswered; revisit when a pass - makes single-pass and trailing-3 diverge. -- **CHAOS and GATE-REVIEW have caught nothing.** Both have dates. -- **`mech environment setup` reads 17 turns / $2.90** this pass, which is - the cost classifier scoring inline `python3 - < **Meta budget — soft, 25% of the trailing three passes.** Work on the -> loop's own instruments and process is capped at a quarter of recent -> spend. Measured by `make status` from each workplan's `kind:` -> frontmatter (`product` | `meta` | `mixed`), over a **trailing 3-pass -> window**; the lifetime figure is reported as history and is not the -> metric (ADR-0006 D1). +> **Meta budget — soft, 25% of spend per pass.** Work on the loop's own +> instruments and process is capped at a quarter of a pass. Measured by +> `make status` from each workplan's `kind:` frontmatter +> (`product` | `meta` | `mixed`). > > **Soft on purpose.** A task already started may be finished — stopping -> mid-task to satisfy a ratio wastes the work and leaves the tree worse -> than either finishing or never starting. What the budget constrains is -> *opening* new meta work above the line, and it constrains it with two -> exemptions: -> -> 1. **Repairing the instrument that reports a breach is always in -> budget.** A gate must never block its own remedy (ADR-0006 D2, -> CB-RES-0005 §4). v1.5 had no such exemption, and the first thing it -> did was forbid fixing the budget's own defect. -> 2. **The maintainer may authorize above-line work**, recorded as -> `authorized_above_budget:` in the workplan frontmatter. Undocumented -> above-line meta work is a violation the evidence file must report. +> mid-task to satisfy a ratio wastes the work and leaves the tree in a +> worse state than either finishing or never starting. What the budget +> forbids is *opening* new meta work above the line. > > A pass that exceeds it **says so in its evidence file and names the -> product work displaced**. That is the whole enforcement: a reporting -> budget, not a gate, for the same reason the session-shape budget is — -> it constrains judgment, not artifacts. - -> **Loop work cashes out (v1.6, ADR-0006 D4).** A `meta` pass must ship at -> least one executable check, command, or gate. Findings that stay prose -> are recorded as findings and do **not** count as delivery. +> product work displaced**. That is the whole enforcement: this is a +> reporting budget, not a gate, for the same reason the session-shape +> budget is (CB-RES-0005 §4) — it constrains judgment, not artifacts. > -> Measured basis: cost per response across five passes ran 0.123, 0.228, -> 0.362, 0.298, 0.123 dollars, and the reading that survived the data is -> that **cost tracks distance from a runnable check** — the passes that -> shipped a command were cheap, the passes that argued about what a -> number means were not (CB-EV-0007 §5). **Refuted if** a prose-only meta -> pass lowers cost per response or catches a class no command could. - -> **Every gate is an experiment (v1.6, ADR-0006 D3).** Each standing -> control gate has an entry in `gates.toml` naming what it has caught, a -> `review_by` date, and what would retire it. `make gate-review` reports -> what is overdue and what has caught nothing; `loop-lint` fails when the -> registry drifts from the Makefile in either direction. +> *(v1.5, from the CB-WP-0007 spend review: CB-WP-0005 and CB-WP-0006 cost +> ~$74, 31% of all spend, for zero measured efficiency gain. Their return +> was correctness of claims, which is real and is not optimization. The +> budget exists so that distinction has to be made out loud.)* v1.4 — corrected from CB-WP-0005 (assertion coverage) on 2026-07-31. Change from v1.3: where a claim rests on numbers, the @@ -55,10 +31,28 @@ adversarial reviewer must read the assertion behind each quoted number and **mutate it** — re-running the command that prints a number is not verification of that number (§Step 2). -v1.5 (meta budget), v1.3 (executable single source of fact), v1.2 -(design goal: cheap correction over exhaustive prevention), v1.1 -(instruments emit their own targets) and v1.0: see -`specs/InnerLoopReference.md` §Change log. +v1.3 changed from v1.2: single source of fact is now executable +(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift +class its first gate. + +v1.2 changes from v1.1: single source of fact; review targets the +harness and states its sampling limit; correction vs retarget; the chaos +roll's calibration window; the live cost budget. The design goal is now +stated: **optimize for cheap correction, not for exhaustive prevention.** +Rationale: `history/260731-loop-hardening-retrospective.md`. + +v1.1 — corrected from CB-WP-0002 (cost accounting) on +2026-07-31. Changes from v1.0: the instrument must exist and emit its own +target; inherited numbers are re-derived before use; every reporting tool +exposes `--self-test`; cost is in the definition of done. Rationale: +`history/260731-cost-accounting-retrospective.md`. + +v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) +and was corrected from it on 2026-07-31. Changes from v0.2: measurement +validity (the positive control), metric feasibility and instrument naming, +four implementation rules the pass earned, and the requirement that +evidence state what it does not support. Rationale and the failures behind +each: `history/260731-inner-loop-retrospective.md`. Normative process for building every Clay-Borg capability. Referenced by all workplans. @@ -117,7 +111,7 @@ are never skipped for code-producing work. | Tier | Weight of steps 1–3 | Structural trigger (forces at least this tier) | |---|---|---| | **L** | Full: separate survey, adversarial review, ADR | Creates a new capability port, or is named a high-leverage pass by the maintainer | -| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, adds/updates an external dependency, **or changes whether or how the loop constrains its own operation** — budgets, gates, review requirements, or these tier rules (v1.6, ADR-0006 D5) | +| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, or adds/updates an external dependency | | **S** | One provenance paragraph in the commit message | Everything else (utilities, fixes, refactors inside a boundary) | **The chaos roll.** After deriving the structural tier, roll **d4** @@ -125,13 +119,32 @@ are never skipped for code-producing work. random (`shuf -e S M L -n 1`), overriding the structural derivation — up or down. -> **Calibration window, opened 2026-07-31, running to 12 tier -> declarations** (declaration 4 of 12 as of 2026-08-01). The rate was -> raised from d10 to d4 because at d10 the mechanism never fired and -> so prevented its own evaluation. Record the roll every time, -> including when it changes nothing (`tier: L (structural L, chaos 4)`). -> Rationale, cost estimate and the two dead rolls: -> `specs/InnerLoopReference.md` §Chaos roll — calibration. +> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was +> d10 and the mechanism **never fired**: two rolls across two workplans +> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and +> ~2 tier decisions per workplan it would take roughly twenty workplans to +> observe four overrides, so the mechanism was set at a rate that prevented +> its own evaluation — the one option T06 ruled out. +> +> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated +> and either kept, returned to d10, or deleted. Expected ~3 firings in the +> window, which is enough to see whether an overridden tier produces a +> different outcome than the argued one. +> +> **Stated cost:** a chaos-L override on work that would have been S buys a +> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001 +> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the +> window is expected to cost **$20–30**. That is the price of finding out +> whether the mechanism is worth keeping, and it is cheaper than carrying an +> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration +(`tier: M (structural L, chaos 10→M)`). **Record the roll every time, +including when it changes nothing** (`tier: L (structural L, chaos 4)`), +so a mechanism that never fires is visible rather than assumed. Purpose: +an occasional random +reweighting keeps the classification honest — arguing everything into S +stops paying off when audits can compare argued tiers against the random +sample — and occasionally forces a deep look at something "obviously +trivial", which is where local optima hide. Chaos limits: a rolled-down tier relaxes *process* weight only. Invariants (zero foreign types in canonical interfaces, determinism, passing diff --git a/specs/InnerLoopReference.md b/specs/InnerLoopReference.md index bf2b529..06af1fd 100644 --- a/specs/InnerLoopReference.md +++ b/specs/InnerLoopReference.md @@ -139,59 +139,3 @@ A capability has completed the loop when all of the following are committed: ([CostAccounting.md](CostAccounting.md)) - [ ] retrospective note (may be one paragraph appended to the evidence file): what the loop itself should change - -## Chaos roll — calibration (moved from InnerLoop v1.6) - -> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was -> d10 and the mechanism **never fired**: two rolls across two workplans -> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and -> ~2 tier decisions per workplan it would take roughly twenty workplans to -> observe four overrides, so the mechanism was set at a rate that prevented -> its own evaluation — the one option T06 ruled out. -> -> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated -> and either kept, returned to d10, or deleted. Expected ~3 firings in the -> window, which is enough to see whether an overridden tier produces a -> different outcome than the argued one. -> -> **Stated cost:** a chaos-L override on work that would have been S buys a -> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001 -> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the -> window is expected to cost **$20–30**. That is the price of finding out -> whether the mechanism is worth keeping, and it is cheaper than carrying an -> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration -(`tier: M (structural L, chaos 10→M)`). **Record the roll every time, -including when it changes nothing** (`tier: L (structural L, chaos 4)`), -so a mechanism that never fires is visible rather than assumed. Purpose: -an occasional random -reweighting keeps the classification honest — arguing everything into S -stops paying off when audits can compare argued tiers against the random -sample — and occasionally forces a deep look at something "obviously -trivial", which is where local optima hide. - - -## Change log — v1.0 to v1.3 (moved from InnerLoop v1.6) - -v1.3 changed from v1.2: single source of fact is now executable -(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift -class its first gate. - -v1.2 changes from v1.1: single source of fact; review targets the -harness and states its sampling limit; correction vs retarget; the chaos -roll's calibration window; the live cost budget. The design goal is now -stated: **optimize for cheap correction, not for exhaustive prevention.** -Rationale: `history/260731-loop-hardening-retrospective.md`. - -v1.1 — corrected from CB-WP-0002 (cost accounting) on -2026-07-31. Changes from v1.0: the instrument must exist and emit its own -target; inherited numbers are re-derived before use; every reporting tool -exposes `--self-test`; cost is in the definition of done. Rationale: -`history/260731-cost-accounting-retrospective.md`. - -v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) -and was corrected from it on 2026-07-31. Changes from v0.2: measurement -validity (the positive control), metric feasibility and instrument naming, -four implementation rules the pass earned, and the requirement that -evidence state what it does not support. Rationale and the failures behind -each: `history/260731-inner-loop-retrospective.md`. - diff --git a/tools/cb-cost.py b/tools/cb-cost.py index ae6228b..a707484 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -524,55 +524,6 @@ def collect(slug, pin_ref=None, since_ref=None): } -def pass_costs(slug, boundaries): - """Cost per pass window, from one read of the transcripts. - - CB-WP-0009 T01 / ADR-0006 D1: a budget must measure the window it - governs. `boundaries` is `[(label, start_iso)]` in ascending order; - each window runs to the next label's start, the last to now. - Responses before the first boundary land in `_before`. - - Reads the transcripts **once**. Calling `collect()` per boundary - would re-read every transcript per window, and `make status` is - supposed to stay cheap enough that nobody replaces it with `ls`. - """ - prices = load_prices() - stale = check_price_sheet_age(prices) - if stale: - raise Abort(stale) - paths = transcript_paths(slug) - if not paths: - raise Abort(f"no transcripts found for {slug}") - responses = [] - for path in paths: - responses.extend(read_responses(path, None)) - if not responses: - raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") - - ordered = sorted(boundaries, key=lambda b: b[1]) - out = {label: {"cost": 0.0, "responses": 0} for label, _ in ordered} - out["_before"] = {"cost": 0.0, "responses": 0} - - for r in responses: - cost = price_of(prices, r["model"], r["toks"], r["timestamp"]) or 0.0 - bucket = "_before" - for label, start in ordered: - if r["timestamp"] > start: - bucket = label - else: - break - out[bucket]["cost"] += cost - out[bucket]["responses"] += 1 - - # Positive control: a bucketing that placed nothing in any real window - # would report a confident 0% for every pass. - placed = sum(v["responses"] for k, v in out.items() if k != "_before") - if ordered and placed == 0: - raise Abort("pass_costs bucketed every response before the first " - "boundary — the boundaries are wrong, not the spend") - return out - - def render(rep, by_task=False, composition=False): print(f"M-D2-CST cost report — {rep['slug']}") print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}") @@ -674,31 +625,6 @@ def self_test(): check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong, f"${got:.4f} (1h would be ${wrong:.4f})") - # ADR-0006 D1: pass_costs buckets by window, and refuses to report a - # confident 0% when every response landed before the first boundary. - slug = "-home-worsch-clay-borg" - try: - real = pass_costs(slug, [("early", "2020-01-01T00:00:00Z")]) - check("pass_costs places responses in a real window", - real["early"]["responses"] > 0 and real["early"]["cost"] > 0, - f"{real['early']['responses']} response(s)") - two = pass_costs(slug, [("a", "2020-01-01T00:00:00Z"), - ("b", "2026-07-31T00:00:00Z")]) - check("pass_costs splits at a boundary rather than pooling", - two["a"]["responses"] > 0 and two["b"]["responses"] > 0, - f"a={two['a']['responses']} b={two['b']['responses']}") - check("pass_costs windows sum to the unwindowed total", - abs((two["a"]["cost"] + two["b"]["cost"] + two["_before"]["cost"]) - - real["early"]["cost"] - real["_before"]["cost"]) < 0.01) - except Abort as e: - check("pass_costs places responses in a real window", False, str(e)) - try: - pass_costs(slug, [("future", "2099-01-01T00:00:00Z")]) - check("pass_costs aborts when nothing lands in any window", False, - "no Abort raised — every pass would read $0.00 and 0%") - except Abort: - check("pass_costs aborts when nothing lands in any window", True) - # AC-5: dedup invariant is enforced. import tempfile with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: diff --git a/tools/gate-review.py b/tools/gate-review.py deleted file mode 100644 index 0ee6890..0000000 --- a/tools/gate-review.py +++ /dev/null @@ -1,218 +0,0 @@ -#!/usr/bin/env python3 -"""Which control gates are due for a keep-or-kill argument? (ADR-0006 D3) - -CB-WP-0009 T02. Six passes produced five standing control mechanisms and -no way to retire any of them. Gates accumulate monotonically because each -one was justified once, at the moment it was cheapest to justify. - -This reads `gates.toml` and reports two things: - - * gates past `review_by` — the date someone said they would argue for - keeping it; - * gates whose `caught` list is **empty** — which is not proof a gate is - useless (it may be preventing rather than missing), but is the - argument that has to be made out loud rather than never. - -It **reports**. It does not fail the build, for CB-RES-0005 §4's reason: -a gate that blocks the remedy when the metric breaches is a trap. The -only failure exit here is a broken registry — a file that cannot be read -would otherwise report "0 gates, all healthy". - -Usage: - python3 tools/gate-review.py - python3 tools/gate-review.py --self-test -""" -import datetime -import os -import sys - -from repo import ROOT, enter_root - -REGISTRY = os.path.join(ROOT, "gates.toml") - -try: - import tomllib -except ModuleNotFoundError: # pragma: no cover - Python < 3.11 - import tomli as tomllib - - -class Fail(Exception): - pass - - -REQUIRED = ("id", "name", "checks", "added", "review_by", "retire_if") - - -def load(path=REGISTRY): - """Every gate, validated. A registry that parses but says nothing is - the harness-does-nothing failure this project keeps finding.""" - if not os.path.exists(path): - raise Fail(f"{os.path.relpath(path, ROOT)} is missing") - with open(path, "rb") as fh: - data = tomllib.load(fh) - gates = data.get("gate") or [] - if not gates: - raise Fail("registry parsed but contains no gates") - for g in gates: - missing = [k for k in REQUIRED if not g.get(k)] - if missing: - raise Fail(f"gate {g.get('id', '?')!r} is missing {missing}") - for field in ("added", "review_by"): - try: - datetime.date.fromisoformat(g[field]) - except ValueError as e: - raise Fail(f"gate {g['id']!r} has a bad {field}: {e}") from e - return gates - - -def make_targets(path=None): - """Targets declared in the Makefile, so an entry cannot name a - command that does not exist.""" - path = path or os.path.join(ROOT, "Makefile") - targets = set() - for line in open(path): - if line and not line[0].isspace() and ":" in line and not line.startswith("."): - name = line.split(":", 1)[0].strip() - if name and " " not in name: - targets.add(name) - return targets - - -def report(today=None): - today = today or datetime.date.today() - gates = load() - targets = make_targets() - - overdue, quiet, broken = [], [], [] - for g in gates: - if datetime.date.fromisoformat(g["review_by"]) <= today: - overdue.append(g) - if not g.get("caught"): - quiet.append(g) - target = g.get("target") or "" - if target and target not in targets: - broken.append((g, target)) - - print("gate review — every gate is an experiment (ADR-0006 D3)") - print(f" registry {len(gates)} gate(s), {REGISTRY.split('/')[-1]}") - print(f" today {today.isoformat()}") - - print("\n gates") - for g in gates: - due = datetime.date.fromisoformat(g["review_by"]) - days = (due - today).days - mark = "DUE " if days <= 0 else ("soon" if days <= 30 else "ok ") - print(f" [{mark}] {g['id']:<14} {g['name']:<28} " - f"review {g['review_by']} ({days:+}d) caught {len(g.get('caught', []))}") - - if quiet: - print("\n caught nothing yet — not a verdict, an argument that is owed") - for g in quiet: - print(f" {g['id']:<14} retire if: {g['retire_if']}") - - if overdue: - print("\n DUE for a keep-or-kill argument") - for g in overdue: - print(f" {g['id']:<14} {g['retire_if']}") - - if broken: - print("\n registry drift — entry names a target the Makefile lacks") - for g, target in broken: - print(f" {g['id']:<14} target {target!r}") - - print(f"\n {len(overdue)} due, {len(quiet)} silent, {len(broken)} drifted") - print(" reporting only — never fails the build (CB-RES-0005 §4)") - return 0 - - -def self_test(): - """Each check pins a way this tool could report a comfortable lie.""" - results = [] - - def check(name, ok, detail=""): - results.append((name, ok, detail)) - - gates = load() - check("the real registry loads", len(gates) >= 5, f"{len(gates)} gate(s)") - check("every gate names what would retire it", - all(len(g["retire_if"]) > 20 for g in gates)) - # Registry drift is the failure this exists to prevent: a gate added - # to the Makefile with no entry, or an entry for a deleted target. - targets = make_targets() - named = [(g["id"], g["target"]) for g in gates if g.get("target")] - check("every named target exists in the Makefile", - all(t in targets for _i, t in named), - ", ".join(f"{i}:{t}" for i, t in named if t not in targets) or "all present") - check("Makefile targets were actually parsed", len(targets) >= 15, - f"{len(targets)} target(s)") - - import tempfile - - def registry(text): - fh = tempfile.NamedTemporaryFile("w", suffix=".toml", delete=False) - fh.write(text) - fh.close() - return fh.name - - # A registry that parses to nothing must abort, not report "0 due". - empty = registry("# no gates here\n") - try: - load(empty) - check("an empty registry aborts", False, "reported a clean bill") - except Fail: - check("an empty registry aborts", True) - finally: - os.unlink(empty) - - # A missing required field must abort rather than be treated as absent - # evidence — `caught` is optional, the rest are not. - partial = registry('[[gate]]\nid = "X"\nname = "n"\nchecks = "c"\n' - 'added = "2026-01-01"\nreview_by = "2026-02-01"\n') - try: - load(partial) - check("a gate with no retire_if aborts", False, "accepted") - except Fail: - check("a gate with no retire_if aborts", True) - finally: - os.unlink(partial) - - # The overdue arithmetic must actually fire. - import io - from contextlib import redirect_stdout - - buf = io.StringIO() - with redirect_stdout(buf): - report(today=datetime.date(2099, 1, 1)) - late = buf.getvalue() - check("a far-future date marks every gate due", - late.count("[DUE ]") == len(gates), - f"{late.count('[DUE ]')} of {len(gates)}") - - buf = io.StringIO() - with redirect_stdout(buf): - report(today=datetime.date(2020, 1, 1)) - early = buf.getvalue() - check("a far-past date marks none due", "[DUE ]" not in early) - check("silent gates are named either way", "caught nothing yet" in early) - - print("gate-review self-test (positive control)") - ok = True - for name, passed, det in results: - print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {det}" if det else "")) - ok &= passed - return 0 if ok else 1 - - -def main(): - enter_root() - try: - if "--self-test" in sys.argv: - return self_test() - return report() - except Fail as e: - print(f"gate-review: {e}", file=sys.stderr) - return 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tools/loop-lint.py b/tools/loop-lint.py index be436b7..82e4163 100644 --- a/tools/loop-lint.py +++ b/tools/loop-lint.py @@ -162,61 +162,12 @@ def check_reporting_tools_self_test(root=REPO): return out -def check_gate_registry(root=REPO): - """ADR-0006 D3 — every control gate is in `gates.toml`, and every - entry names a real target. - - The failure this prevents is drift in the direction nobody notices: a - gate added to `make all` with no registry entry never acquires a - review date, which is how five mechanisms accumulated with no way to - retire any of them. - """ - out = [] - registry = os.path.join(root, "gates.toml") - makefile = os.path.join(root, "Makefile") - if not (os.path.exists(registry) and os.path.exists(makefile)): - return out - try: - import tomllib - except ModuleNotFoundError: # pragma: no cover - return out - - with open(registry, "rb") as fh: - data = tomllib.load(fh) - gates = data.get("gate") or [] - if not gates: - return [Finding("gates", "gates.toml", "registry contains no gates")] - registered = {g.get("target") for g in gates if g.get("target")} - exempt = set(data.get("not_control_gates") or []) - - text = open(makefile).read() - m = re.search(r"^all:(.*)$", text, re.M) - deps = m.group(1).split() if m else [] - targets = {ln.split(":", 1)[0].strip() for ln in text.splitlines() - if ln and not ln[0].isspace() and ":" in ln and not ln.startswith(".")} - - for dep in deps: - if dep not in registered and dep not in exempt: - out.append(Finding( - "gates", "gates.toml", - f"`make all` runs {dep!r}, which is neither a registered " - f"control gate nor listed in not_control_gates — classify it, " - f"so it cannot acquire permanence without a review date")) - for target in sorted(registered): - if target not in targets: - out.append(Finding( - "gates", "gates.toml", - f"entry names target {target!r}, which the Makefile lacks")) - return out - - CHECKS = ( check_loadability, check_evidence_no_unmeasured, check_survey_tier_and_chaos, check_review_trail, check_reporting_tools_self_test, - check_gate_registry, ) @@ -245,29 +196,6 @@ def self_test(): len(f) == 1 and "Big.md" in f[0].path, f"{len(f)} finding(s)") - # gates: an unclassified `all:` dependency trips, and so does an - # entry naming a target the Makefile lacks. - with open(os.path.join(tmp, "Makefile"), "w") as fh: - fh.write("all: coverage newthing\ncoverage:\n\techo\n") - with open(os.path.join(tmp, "gates.toml"), "w") as fh: - fh.write('not_control_gates = []\n\n[[gate]]\nid = "G"\n' - 'name = "n"\ntarget = "coverage"\nchecks = "c"\n' - 'added = "2026-01-01"\nreview_by = "2026-02-01"\n' - 'retire_if = "r"\n') - f = check_gate_registry(tmp) - check("gate registry detects an unclassified all: dependency", - len(f) == 1 and "newthing" in f[0].detail, f"{len(f)} finding(s)") - with open(os.path.join(tmp, "gates.toml"), "w") as fh: - fh.write('not_control_gates = ["newthing", "coverage"]\n\n[[gate]]\nid = "G"\n' - 'name = "n"\ntarget = "ghost"\nchecks = "c"\n' - 'added = "2026-01-01"\nreview_by = "2026-02-01"\n' - 'retire_if = "r"\n') - f = check_gate_registry(tmp) - check("gate registry detects an entry naming a missing target", - len(f) == 1 and "ghost" in f[0].detail, f"{len(f)} finding(s)") - os.unlink(os.path.join(tmp, "gates.toml")) - os.unlink(os.path.join(tmp, "Makefile")) - # evidence: a table verdict trips; the word in prose does not. with open(os.path.join(tmp, "evidence", "E.md"), "w") as fh: fh.write("| AC-1 | x | unmeasured |\n" diff --git a/tools/status.py b/tools/status.py index 5dd7c05..bff8c10 100644 --- a/tools/status.py +++ b/tools/status.py @@ -116,90 +116,6 @@ def cost_lines(): return open_spend, rep["total"], mod -# ------------------------------------------------------- the meta budget - -# How many passes the budget looks back over. One pass is a coin flip — -# a `meta` pass reads 100% and a `product` pass reads 0%, and neither -# says anything about the balance the budget is about. Three is the -# smallest window in which a ratio means something and still moves. -TRAILING_PASSES = 3 -META_SOFT_PCT = 25 - - -def workplan_starts(): - """[(id, kind, first_commit_iso)] — when each pass began. - - A pass starts at the commit that *added* its workplan file, which is - the only boundary git records and the same one `cb-cost --since` has - been windowed on by hand since CB-WP-0004 T05. - """ - import datetime - - out = [] - for path in sorted(glob.glob(os.path.join(ROOT, "workplans", "*.md"))): - wid, _title, _status, _tasks, kind = parse_workplan(path) - rel = os.path.relpath(path, ROOT) - added = git("log", "--diff-filter=A", "--format=%cI", "--", rel) - first = added.splitlines()[-1].strip() if added else "" - if not (wid and first): - continue - # Transcript timestamps are UTC `...Z`; git prints a local offset. - # Comparing the two as strings silently buckets everything into - # `_before`, which is how this first read reported $0. - first = ( - datetime.datetime.fromisoformat(first) - .astimezone(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - out.append((wid, kind, first)) - return sorted(out, key=lambda r: r[2]) - - -def meta_budget(plans): - """ADR-0006 D1: report the share for the window the budget governs.""" - _, _, mod = cost_lines() - starts = workplan_starts() - if not starts: - print("\n meta budget UNAVAILABLE — no workplan start commits found") - return - costs = mod.pass_costs("-home-worsch-clay-borg", - [(wid, when) for wid, _k, when in starts]) - - window = starts[-TRAILING_PASSES:] - kinds = {wid: kind for wid, kind, _ in starts} - total = sum(costs[wid]["cost"] for wid, _k, _w in window) - # `mixed` splits evenly; stated rather than hidden. - weight = {"product": 0.0, "meta": 1.0, "mixed": 0.5} - meta = sum(costs[wid]["cost"] * weight.get(kinds[wid], 0.5) - for wid, _k, _w in window) - - if total <= 0: - print("\n meta budget UNAVAILABLE — the trailing window measured $0") - return - share = 100 * meta / total - mark = "ok " if share <= META_SOFT_PCT else "OVER" - print(f"\n meta budget [{mark}] {share:.0f}% over the last " - f"{len(window)} pass(es) (soft {META_SOFT_PCT}%, InnerLoop v1.6)") - for wid, kind, _w in window: - c = costs[wid] - print(f" {wid} {kind:<8} ${c['cost']:>7,.2f} " - f"{c['responses']:>4} response(s)") - - # History, kept and labelled — the whole point of ADR-0006 D1 is that - # this number is not what the target compares against. - lifetime = sum(costs[wid]["cost"] for wid, _k, _w in starts) - life_meta = sum(costs[wid]["cost"] * weight.get(kind, 0.5) - for wid, kind, _ in starts) - if lifetime > 0: - print(f" history, all {len(starts)} passes: " - f"{100 * life_meta / lifetime:.0f}% — NOT the metric " - f"(ADR-0006 D1)") - print(" NOTE: repairing the instrument that reports a breach is always") - print(" in budget (ADR-0006 D2); other above-line meta work needs") - print(" `authorized_above_budget:` in the workplan frontmatter.") - - # ------------------------------------------------------------------ report @@ -246,10 +162,31 @@ def report(): except Exception as e: print(f" UNAVAILABLE — {e}") - # InnerLoop v1.6 — soft 25% meta budget over a trailing pass window. - # Reported, never gated. + # InnerLoop v1.5 — soft 25% meta budget. Reported, never gated. try: - meta_budget(plans) + _, _, mod = cost_lines() + rep = mod.collect("-home-worsch-clay-borg", None) + detail = rep["by_task_detail"] + by_kind = {"product": 0.0, "meta": 0.0, "mixed": 0.0} + attributed = 0.0 + for wid, _t, _s, tasks, kind in plans: + for tid, _ts, _p in tasks: + d = detail.get(tid) + if d and kind in by_kind: + by_kind[kind] += d["cost"] + attributed += d["cost"] + if attributed > 0: + # `mixed` splits evenly; stated rather than hidden. + meta = by_kind["meta"] + by_kind["mixed"] / 2 + share = 100 * meta / attributed + mark = "ok " if share <= 25 else "OVER" + print(f"\n meta budget [{mark}] {share:.0f}% of " + f"${attributed:,.2f} attributed (soft 25%, InnerLoop v1.5)") + print(f" product ${by_kind['product']:,.2f} " + f"meta ${by_kind['meta']:,.2f} " + f"mixed ${by_kind['mixed']:,.2f} (split 50/50)") + print(" NOTE: attributed tasks only — early workplans used bare") + print(" T## ids that collide across passes and are excluded.") except Exception as e: print(f"\n meta budget UNAVAILABLE — {e}") @@ -342,32 +279,6 @@ def self_test(): repr(h[:60])) check("task heading resolves to something", bool(h)) - # ADR-0006 D1: the budget must measure the window it governs. - starts = workplan_starts() - check("every workplan has a start commit", len(starts) >= len(plans) - 1, - f"{len(starts)} start(s) for {len(plans)} workplan(s)") - check("start commits are UTC and comparable to transcript stamps", - all(w.endswith("Z") for _i, _k, w in starts), - # A local-offset stamp compares as a string against `...Z` and - # silently buckets every response before the first boundary, - # which is exactly how this reported $0 on its first run. - ", ".join(w for _i, _k, w in starts[:1])) - check("start commits are in ascending order", - [w for _i, _k, w in starts] == sorted(w for _i, _k, w in starts)) - - buf = io.StringIO() - try: - with redirect_stdout(buf): - meta_budget(plans) - text = buf.getvalue() - check("budget reports a windowed share", "over the last" in text) - check("budget labels the cumulative figure as history", - "NOT the metric" in text) - check("budget states the instrument-repair exemption", - "in budget (ADR-0006 D2)" in text) - except Exception as e: - check("budget reports a windowed share", False, str(e)) - # The report itself must run and produce substance, not a stub. buf = io.StringIO() try: diff --git a/workplans/CB-WP-0009-adaptive-gates.md b/workplans/CB-WP-0009-adaptive-gates.md index 96fcf5d..b9faecc 100644 --- a/workplans/CB-WP-0009-adaptive-gates.md +++ b/workplans/CB-WP-0009-adaptive-gates.md @@ -2,7 +2,7 @@ id: CB-WP-0009 kind: meta title: "Make control gates experimental: window them, review them, cash them out" -status: done +status: proposed authorized_above_budget: "maintainer, 2026-08-01 — 'improve our policies where it seems promising; we will experiment our way to sensible task- and context-specific control and review gates'. Meta read 59% at the time." state_hub_workstream_id: "98c0af4a-415c-4cf5-92d0-a722eb70cb90" --- @@ -34,7 +34,7 @@ prose has not delivered. ```task id: CB-WP-0009-T01 -status: done +status: todo priority: high state_hub_task_id: "c6eefacd-5bb9-4640-8356-12e8112bcf11" ``` @@ -53,16 +53,11 @@ it was a pure product pass — against 61% cumulative. **Refuted if** the two land within 20% of each other, in which case D1 is wrong and the machinery should be deleted rather than kept. -**Done 2026-08-01.** `cb-cost.pass_costs` buckets by pass window in one -read of the transcripts; `make status` reports a trailing 3-pass share -with the per-pass breakdown and the exemption. First reading **36% over -the last 3 passes** against **49% lifetime** — CB-WP-0008 alone reads 0%. - ## Task: `gates.toml` and `make gate-review` ```task id: CB-WP-0009-T02 -status: done +status: todo priority: high state_hub_task_id: "0fb6cfaf-9304-481a-a96f-1f0150f4d81d" ``` @@ -82,17 +77,11 @@ behind the Makefile is worse than none, so `loop-lint` gains a check that every gate target has an entry and every entry names a real target. Its own self-test must fail when an entry is removed. -**Done 2026-08-01.** `gates.toml` (9 gates), `make gate-review`, and a -`loop-lint` check in both directions — an unclassified `make all` -dependency and an entry naming a target the Makefile lacks each produce -a finding, both pinned by self-tests. First run: **0 due, 2 silent** -(the chaos roll and `gate-review` itself). - ## Task: InnerLoop v1.6 ```task id: CB-WP-0009-T03 -status: done +status: todo priority: medium state_hub_task_id: "0b1b33ac-8046-4b34-b287-7b32755d964a" ``` @@ -111,16 +100,11 @@ Fold **D2**, **D4** and **D5** into `specs/InnerLoop.md`: pointer. Loadability is itself a gate; growing the file to fit a rule about gates would be the joke writing itself. -**Done 2026-08-01.** v1.6 in `specs/InnerLoop.md`, **386 lines** — the -chaos-roll calibration rationale and the v1.0–v1.3 change log moved to -`specs/InnerLoopReference.md`, so the file came in *smaller* than it -started at 399. - ## Task: control loop — did the changes fire? ```task id: CB-WP-0009-T04 -status: done +status: todo priority: high state_hub_task_id: "e06f57fc-147f-4cba-b398-0db6b060381f" ``` @@ -139,9 +123,3 @@ answered from output, not argument: pass or needs the exception written down. 4. **Cost**, against the 0.123–0.362 per-response range, and the meta share this pass itself consumed — reported under the rule it wrote. - -**Done 2026-08-01.** [CB-EV-0008](../evidence/CB-EV-0008-adaptive-gates.md). -The pass ran at **$0.177/response** — cheaper than every previous meta -pass and still 1.4× the product pass, which is partial support for D4 and -is reported as such. Meta share reads **45% over the trailing three**, -over the line; the next pass should be product.