From 710502717320c93ba4aae0e474c65197400d5854 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 2 Aug 2026 07:21:00 +0200 Subject: [PATCH 1/4] Declare CB-WP-0013: two instruments that were wrong, tier M MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Declaration 8 of 12. Structural tier M (changes how the loop constrains its own operation — a budget instrument and a gate floor); d4 = 3, no override. Meta budget reads 0% of the trailing three, so this is in budget without the ADR-0006 D2 repair exemption. The SH-3 half is larger than CB-WP-0012 filed it as. "SH-3 has read 0.0% for six consecutive passes" is an artifact: --shape-budget measures the window since the last commit, and it is read while writing the evidence file, immediately after a commit, when that window holds one or two responses. SH-1 and SH-2 are location statistics and survive n=2; SH-3 is a rate, and at n=2 its only possible values are 0%, 50%, 100%. Measured over real per-pass windows it is 1.1%-6.3%, not 0.0%. Which exposes what the artifact hid: the pinned historical rate is 7.8%-8.6%, so batching has got WORSE, and a metric stuck at a constant could not have shown it. That is the ninth recorded instance of the trusted-arithmetic class, and the second one in this same metric — SessionShape.md §4 already records "0 of 330 tool calls were batched" as the eighth. Also owed: AM-4a counts proc-macro crates that never reach a binary (89,048 lines, 36.2%). The task is explicit that the target must move down with the instrument, or the pass must argue why 89,048 lines of unearned headroom are legitimate. Co-Authored-By: Claude Opus 5 --- .../CB-WP-0013-instrument-corrections.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 workplans/CB-WP-0013-instrument-corrections.md diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md new file mode 100644 index 0000000..50e43cc --- /dev/null +++ b/workplans/CB-WP-0013-instrument-corrections.md @@ -0,0 +1,171 @@ +--- +id: CB-WP-0013 +kind: meta +title: "Two instruments that were wrong, and the record they wrote" +status: todo +--- + +# Purpose + +``` +structural tier M (changes how the loop constrains its own operation: + a budget instrument and a gate floor — InnerLoop + v1.6, ADR-0006 D5) +chaos d4 = 3 → no override +declared tier M +``` + +Declaration 8 of 12 in the calibration window. Tier M means the survey and +the ADR merge into one document and the adversarial review is optional. + +**Meta budget: 0% of the trailing three**, so this pass is in budget +without the ADR-0006 D2 repair exemption — though it would qualify for it, +since both halves repair an instrument that reports a breach. + +## Why now + +CB-WP-0012 ended owing two corrections, and one of them turns out to be +larger than it was filed as. + +### SH-3 has never read what six evidence files say it read + +CB-EV-0010 §5 states *"SH-3 has now read 0.0% for six consecutive passes"* +and calls it the oldest unargued number in the project. The number is an +artifact. + +`--shape-budget` measures **the window since the last commit**, and it is +read while writing the evidence file — immediately *after* a commit, when +that window holds one or two responses. SH-1 and SH-2 are location +statistics and survive a sample of two. **SH-3 is a rate**, and at n=2 its +only possible values are 0%, 50% and 100%. One window, three metrics, and +it is only wrong for one of them. + +Measured over real per-pass windows instead: + +| window since | responses | SH-3 | +|---|---:|---:| +| `0d2ab22` | 115 | 6.2% | +| `6fb0aea` | 113 | 6.3% | +| `331e7e9` | 95 | 1.1% | +| `c5fa610` | 74 | 1.4% | +| `883b608` | 70 | 1.4% | +| `84d6886` | 24 | 4.3% | + +So the substantive claim survives — SH-3 is badly unmet — but **the figure +quoted in six evidence files was never the measurement**. Worse, the real +numbers say something the artifact hid: the historical rate pinned in +`specs/SessionShape.md` is **7.8–8.6%**, and these windows are **1.1–6.3%**. +Batching has got *worse*, and a metric stuck at a constant 0.0% could not +show a trend at all. + +### AM-4a overstates every figure it has ever reported by 36% + +ADR-0007 Decision 4, filed and unimplemented. AM-4a counts +`--edges normal`, which includes proc-macro crates that run in the compiler +and never reach a binary: **89,048 lines, 36.2%** of the shipped-runtime +figure, `syn` alone 66,916. + +## Task: fix SH-3's instrument, and correct the record + +```task +id: CB-WP-0013-T01 +status: todo +priority: high +``` + +A rate needs a sample. `--shape-budget` must **refuse to report SH-3** +below a stated minimum rather than print a number the window cannot +support — `insufficient sample (n responses)`, not `0.0%`. + +**The choice to argue, not assume:** whether SH-3 should instead be +windowed differently from SH-1/SH-2 — over the pass rather than since the +last commit — since it is the only one of the three that a two-response +window destroys. Pick one and say why the other was rejected. + +**Controls.** A minimum-sample rule is exactly the kind of guard that can +be satisfied vacuously: assert that a window *above* the floor still +reports a rate, that one below reports the refusal, and that the refusal +is distinguishable from a genuine 0.0%. Mutate each. + +**Correct the record.** Six evidence files quote 0.0%. They are history and +are not rewritten; the correction is recorded once, where a reader will +find it, and named as what it is — the same class as CB-WP-0002's +"0 of 330 tool calls were batched", which `specs/SessionShape.md` §4 +already records as the eighth trusted-arithmetic instance. **This is the +ninth, and it is in the same metric.** That coincidence is worth a +sentence: SH-3 is the number this project keeps getting wrong. + +## Task: argue SH-3's floor, or move it + +```task +id: CB-WP-0013-T02 +status: todo +priority: high +``` + +The declaration CB-EV-0010 said was owed. `specs/SessionShape.md` is +explicit that the 20% floor was set above the measured rate deliberately +and that *"whether 20% is reachable or arbitrary is unknown until a pass +tries"*. Seven passes have now reported it and none has tried. + +Decide, with the real numbers rather than the artifact: + +- is 20% reachable, and at what cost to how the work is actually done; +- is the measured decline from 7.8–8.6% to 1.1–6.3% a behavioural change + worth reversing, or an artifact of tier-L passes doing more sequential + reading than tier-S ones; +- **and the honest option: retire SH-3.** `specs/SessionShape.md` §4 puts + the ceiling at **$2–4 on a $93 pass** and says *"batch because it is + free, not because it is the lever."* A floor that is 3–20× the measured + rate, cannot be moved by the party measured, and is worth single-digit + dollars is a candidate for `retire_if`, not a standing breach. + +**Do not retarget to match the measurement.** `specs/SessionShape.md` +already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If +the floor moves, it moves on an argument about what batching is worth, and +the argument is written down before the number changes. + +## Task: correct AM-4a to count what ships + +```task +id: CB-WP-0013-T03 +status: todo +priority: high +``` + +ADR-0007 Decision 4. `tools/dep-weight.py` counts +`--edges normal,no-proc-macro`. + +**The targets do not move with the instrument.** Correcting a metric that +reports 246,250 down to 157,202 while leaving the target at 250,000 hands +this project 89,048 lines of headroom it did not earn, in the same commit +that reveals the error. Either the target moves down by the same amount +the instrument does, or the pass states plainly why the extra headroom is +legitimate. It is not obviously legitimate; argue it or give it back. + +**Positive control.** `dep-weight.py` already has the guard that an +unlocatable crate must not silently count as zero. The new flag must not +be able to drop crates it should have counted: assert that the proc-macro +set is exactly the five expected crates and that removing the flag +restores the old figure. + +## Task: evidence + +```task +id: CB-WP-0013-T04 +status: todo +priority: medium +``` + +`evidence/CB-EV-0011-instrument-corrections.md`. + +Carry: whether tier M's merged survey-and-ADR was the right weight for two +instrument corrections; the cost of a meta pass immediately after two +product passes; and the standing question of whether **three** consecutive +evidence files reporting their own pass's cost low means the loop should +simply stop quoting it. + +Also carry the one that is now overdue: **`cb-play` is three modes in one +binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider +the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass +did not act. Either do it or state why not, and stop carrying it. From 6080c4cf6016a3bc86dba03d4daf496f2c7b0884 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 2 Aug 2026 07:23:04 +0200 Subject: [PATCH 2/4] CB-WP-0013-T01: SH-3 refuses a window that cannot carry a rate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The metric six evidence files reported as 0.0% was never measured. --shape-budget windows since the last commit, and it is read while writing the evidence file — right after a commit, when the window holds one or two responses. SH-1/SH-2 are location statistics and survive n=2. SH-3 is a rate: at n=2 its only possible values are 0%, 50%, 100%. cb-cost.py gains sh3_line(), which below a minimum sample prints "insufficient sample" and no verdict. The floor is derived: if the true rate were exactly the 20% target, P(zero batched in n) = 0.8^n, and at n=14 that is 4.4% — so "0 batched in 14" rules out a target-meeting rate at ~95%. Below that the tool has nothing to say and now says so. The window was kept rather than split. SH-3 could have been given a per-pass window, but the budget's purpose is the open remainder since the last commit, and giving one of three metrics a different window makes "the window" ambiguous in a tool three specs cite. Four controls, three mutations, each red for its stated reason — including the one the evidence files actually hit, where a refusal is printed as a measured zero. SessionShape.md §4 carries the correction with the real per-pass figures (1.1%-6.3%), beside the eighth trusted-arithmetic instance. This is the ninth, and the second in this same metric. It also shows what the frozen 0.0% hid: against the pinned 7.8%-8.6%, batching has got worse, and six passes reported a breach that was moving underneath them. Co-Authored-By: Claude Opus 5 --- specs/SessionShape.md | 41 ++++++++++++ tools/cb-cost.py | 66 +++++++++++++++++-- .../CB-WP-0013-instrument-corrections.md | 28 +++++++- 3 files changed, 129 insertions(+), 6 deletions(-) diff --git a/specs/SessionShape.md b/specs/SessionShape.md index 44e4d38..6b9a521 100644 --- a/specs/SessionShape.md +++ b/specs/SessionShape.md @@ -136,6 +136,47 @@ workplan — the failure mode InnerLoop v1.1 rule 13 exists for, and the `history/260731-inner-loop-rule-audit.md` identified as having no executable defence. +### Correction (CB-WP-0013 T01): SH-3 was never read as measured + +**Six evidence files quote `SH-3 0.0%`, and none of them measured it.** + +`cb-cost --shape-budget` reports the window **since the last commit**, and +it is read while writing the evidence file — immediately *after* a commit, +when that window holds one or two responses. SH-1 and SH-2 are location +statistics and survive a sample of two. SH-3 is a **rate**, and at n=2 its +only possible values are 0%, 50% and 100%. One window, three metrics, and +it is only wrong for one of them. + +Measured over real per-pass windows instead: + +| window since | responses | SH-3 | +|---|---:|---:| +| `0d2ab22` | 115 | 6.2% | +| `6fb0aea` | 113 | 6.3% | +| `331e7e9` | 95 | 1.1% | +| `c5fa610` | 74 | 1.4% | +| `883b608` | 70 | 1.4% | +| `84d6886` | 24 | 4.3% | + +The substantive claim survives — SH-3 is badly unmet — but the constant +0.0% hid something the real numbers show: against the **7.8–8.6%** pinned +above, **batching has got worse**, and a metric frozen at a constant could +not have shown a trend at all. Six passes reported a breach and the breach +was moving underneath them. + +**This is the ninth recorded instance of the trusted-arithmetic class, and +the second in this same metric** — the eighth is the `0 of 330 tool calls` +figure corrected immediately above. SH-3 is the number this project keeps +getting wrong, which is itself a reason to doubt the metric rather than the +behaviour (§SH-3 disposition). + +**The fix**, `cb-cost.py` `sh3_line()`: below a minimum sample the tool +prints `insufficient sample` and no verdict. The floor is derived, not +round — if the true rate were exactly the 20% target, the chance of seeing +zero batched turns in `n` responses is `0.8^n`; at **n = 14** that is 4.4%, +so "0 batched in 14" rules out a target-meeting rate at ~95%. Below that +the tool has nothing to say and now says so. + At an 8.6% batching rate there is real headroom, but the saving is bounded: eliminating every avoidable single-call turn would remove at most a few percent of turns, worth roughly $2–4 on a $93 pass. **Batch because it is diff --git a/tools/cb-cost.py b/tools/cb-cost.py index ae6228b..c924ff9 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -624,8 +624,15 @@ def render(rep, by_task=False, composition=False): f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]") print(f"{indent}SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok " f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]") - print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " - f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") + # Same guard as the budget path: a rate a window cannot support + # must not be printed as though it were measured. + if sh["responses_with_tools"] < SH3_MIN_SAMPLE: + print(f"{indent}SH-3 batching rate insufficient sample " + f"[{sh['responses_with_tools']} with tool calls, " + f"need {SH3_MIN_SAMPLE}]") + else: + print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " + f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") print(f"{indent} {sh['tool_calls']} tool calls in " f"{sh['responses_with_tools']} responses; " f"{sh['calls_in_batched_turns']} in batched turns") @@ -786,6 +793,27 @@ def self_test(): finally: os.unlink(empty) + # SH-3 minimum sample (CB-WP-0013 T01). Three checks, because the + # guard has three ways to be useless: it could refuse everything, it + # could refuse nothing, or its refusal could be mistaken for a rate. + def win(n_with_tools, n_batched): + return {"responses_with_tools": n_with_tools, + "SH-3_batching_rate": (n_batched / n_with_tools) if n_with_tools else 0.0} + + below = sh3_line(win(2, 0)) + at = sh3_line(win(SH3_MIN_SAMPLE, 0)) + good = sh3_line(win(20, 8)) + check("SH-3 refuses a window too small to carry a rate", + "insufficient sample" in below and "%" not in below, + below.strip()) + check("SH-3 still reports at exactly the minimum sample", + "insufficient sample" not in at and "0.0%" in at, at.strip()) + check("SH-3 reports a real rate above the floor", + "40.0%" in good and "ok" in good, good.strip()) + # The one that matters: a refusal must not read as a measured zero. + check("a refusal is distinguishable from a genuine 0.0%", + below != sh3_line(win(SH3_MIN_SAMPLE, 0))) + # AC-7: the subagent tree is discovered by the path globs. slug = "-home-worsch-clay-borg" paths = transcript_paths(slug) @@ -809,6 +837,36 @@ def self_test(): SHAPE_SOFT = {"SH-1": 200_000, "SH-2": 300_000} SHAPE_HARD = {"SH-1": 300_000, "SH-2": 450_000} +# CB-WP-0013 T01. SH-1 and SH-2 are location statistics and survive a +# two-response window. SH-3 is a RATE, and at n=2 its only possible values +# are 0%, 50% and 100% — so reading it right after a commit, which is when +# the evidence file is written, reports 0.0% almost regardless of +# behaviour. Six evidence files quoted that 0.0% as a measurement; the real +# per-pass figure was 1.1-6.3%. +# +# The floor is derived, not round. If the true rate were exactly the 20% +# target, the chance of observing ZERO batched turns in n responses is +# 0.8^n. At n = 14 that is 4.4%, so "0 batched in 14" rules out a +# target-meeting rate at ~95%. Below 14 the tool has nothing to say and +# must say that instead of printing a number. +SH3_MIN_SAMPLE = 14 + + +def sh3_line(win, min_sample=SH3_MIN_SAMPLE): + """SH-3, or an explicit refusal when the window cannot support a rate. + + Returns the line rather than printing it so the refusal is testable + without capturing stdout — a guard that can only be checked by reading + output is a guard that will be checked by nobody. + """ + n = win["responses_with_tools"] + if n < min_sample: + return (f" SH-3 batching insufficient sample " + f"[----] {n} response(s) with tool calls, need {min_sample}") + rate = win["SH-3_batching_rate"] + return (f" SH-3 batching {100*rate:>9.1f}% " + f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%") + def shape_budget(slug): """SH-* for the window since the last commit, with soft/hard verdicts.""" @@ -842,9 +900,7 @@ def shape_budget(slug): print(f" {key} {label} {val:>10,.0f} tok [{mark}] " f"soft {soft:,} / hard {hard:,}") breach = max(breach, 0 if val <= soft else (1 if val <= hard else 2)) - rate = win["SH-3_batching_rate"] - print(f" SH-3 batching {100*rate:>9.1f}% " - f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%") + print(sh3_line(win)) if breach >= 2: print("\n HARD — compact before continuing. Context this size costs " diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md index 50e43cc..1006d74 100644 --- a/workplans/CB-WP-0013-instrument-corrections.md +++ b/workplans/CB-WP-0013-instrument-corrections.md @@ -69,7 +69,7 @@ figure, `syn` alone 66,916. ```task id: CB-WP-0013-T01 -status: todo +status: done priority: high ``` @@ -95,6 +95,32 @@ already records as the eighth trusted-arithmetic instance. **This is the ninth, and it is in the same metric.** That coincidence is worth a sentence: SH-3 is the number this project keeps getting wrong. +**Done 2026-08-02.** `cb-cost.py` gains `sh3_line()`: below a minimum +sample it prints `insufficient sample` and **no verdict**. + +**The window was kept, not split.** SH-3 could instead have been given a +per-pass window while SH-1/SH-2 kept the since-last-commit one. Rejected: +the budget's stated purpose is the *open remainder* since the last commit +— the only thing a live budget can fire on — and giving one of three +metrics a different window makes "the window" ambiguous in a tool that +three specs cite. Refusing to report is honest and keeps one window. + +**The floor is derived, not round.** If the true rate were exactly the 20% +target, the chance of observing zero batched turns in `n` responses is +`0.8^n`. At **n = 14** that is 4.4%, so "0 batched in 14" rules out a +target-meeting rate at ~95%. Below 14 the tool has nothing to say. + +Four controls, three mutations, each red for its stated reason: + +| mutation | result | +|---|---| +| the guard never refuses | `SH-3 refuses a window too small` red, *and* `a refusal is distinguishable from a genuine 0.0%` red | +| the guard always refuses | reporting at exactly the minimum, and a real 40% rate, both red | +| the refusal prints as `0.0%` | `refuses a window too small` red — the case the six evidence files actually hit | + +The correction is recorded in `specs/SessionShape.md` §4, beside the +eighth instance, with the real per-pass figures. + ## Task: argue SH-3's floor, or move it ```task From d2c3f340f9ad82885c5547d7a4593721ef5b809c Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 2 Aug 2026 07:30:14 +0200 Subject: [PATCH 3/4] CB-WP-0013-T02/T03: retire SH-3 as a gate; correct AM-4a and its target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0008, tier M (survey and ADR merged). D1 — SH-3 retired as a gate, kept as a diagnostic. Investigating it found a third defect, deeper than the two this pass was declared on. Re-deriving batching from the raw transcripts, independently of cb-cost: CB-WP-0011 pass 54 with tools 0 batched 0.0% gap -> next decl 16 with tools 6 batched 37.5% CB-WP-0012 pass 86 with tools 0 batched 0.0% gap -> next decl 10 with tools 1 batched 10.0% CB-WP-0013 so far 10 with tools 0 batched 0.0% Zero batched turns in 150 in-pass responses; 37.5% in one gap, above the 20% floor. Batching needs two calls whose inputs are known at once — orientation work. Implementation consumes each step's result before the next. SH-3's window is since the last commit, which during a pass is always implementation. The metric could not read above ~0% in the window it was gated on. A floor the window structurally excludes is not a target. This pass's own declaration was also wrong: it claimed batching "has got worse" (7.8-8.6% vs 1.1-6.3%). Differently-placed windows, not different behaviour. Withdrawn — the same class of error, in the pass written to correct it. Not retargeting to match the measurement: the floor was not moved to 6%, the gate was removed on an argument about what the quantity is worth. The number is still reported; only the verdict is gone. D2/D3 — AM-4a counts --edges normal,no-proc-macro: 157,202, not 246,250. The target moves down with it, 250,000 -> 161,000, so the correction hands back essentially nothing (headroom 3,750 -> 3,798). Three controls: the exclusion drops exactly the five expected crates, only removes and never adds, and is not a no-op. The DFD gate then caught the follow-on it exists for — three historical documents carrying live fact tags for a number that had changed. Not rewritten; untagged, with a supersession banner. AM-4b is deliberately not corrected: its proc-macro share is unmeasured. gate-review now reads 0 due, 0 silent, 0 drifted — GATE-REVIEW earns its first caught entry by forcing SH-3's re-justification, and the registry has no silent gates left. Co-Authored-By: Claude Opus 5 --- decisions/ADR-0004-am4-ratification.md | 10 +- decisions/ADR-0008-instrument-corrections.md | 158 ++++++++++++++++++ evidence/CB-EV-0001-game-kernel.md | 12 +- facts.toml | 8 +- gates.toml | 9 +- specs/GameKernel.md | 4 +- tools/cb-cost.py | 29 +++- tools/dep-weight.py | 40 ++++- .../CB-WP-0013-instrument-corrections.md | 60 ++++++- 9 files changed, 306 insertions(+), 24 deletions(-) create mode 100644 decisions/ADR-0008-instrument-corrections.md diff --git a/decisions/ADR-0004-am4-ratification.md b/decisions/ADR-0004-am4-ratification.md index fde8be8..86cd83a 100644 --- a/decisions/ADR-0004-am4-ratification.md +++ b/decisions/ADR-0004-am4-ratification.md @@ -1,5 +1,13 @@ # ADR-0004: ratify AM-4a and AM-4b +> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a +> figures below were taken with an instrument that counted proc-macro +> crates — 89,048 lines, 36.2% — which run in the compiler and never +> reach a binary. Corrected, the same tree measures **157,202** against +> a target moved to **161,000**. The numbers below are left as the +> record of what was measured then, and are no longer live facts. + + status: accepted date: 2026-07-31 decided by: maintainer (Bernd Worsch), 2026-07-31 @@ -49,7 +57,7 @@ At the time of the retarget, `make dep-weight`: | configuration | crates | third-party LOC | |---|---|---| -| shipped-runtime (`--no-default-features`) | 23 | **246,250** | +| shipped-runtime (`--no-default-features`) | 23 | **246,250** | | dev-toolchain (default features) | 29 | **317,021** | | own source | — | 3,443 | diff --git a/decisions/ADR-0008-instrument-corrections.md b/decisions/ADR-0008-instrument-corrections.md new file mode 100644 index 0000000..79057f1 --- /dev/null +++ b/decisions/ADR-0008-instrument-corrections.md @@ -0,0 +1,158 @@ +# ADR-0008: retire SH-3 as a gate; correct AM-4a to count what ships + +status: accepted +date: 2026-08-02 +decided by: agent, under the standing loop authorization. One item is +reserved for the maintainer — see §Reserved. +tier: M (structural M — changes how the loop constrains its own operation: +a gate floor and a budget instrument, InnerLoop v1.6 / ADR-0006 D5; +chaos d4=3 → no override). Tier M merges survey and ADR into one document, +which this is; the adversarial review is optional and was not run. +references: [CB-WP-0013](../workplans/CB-WP-0013-instrument-corrections.md), +[SessionShape.md](../specs/SessionShape.md) §4, +[ADR-0006](ADR-0006-adaptive-control-gates.md) D1/D2/D4, +[ADR-0007](ADR-0007-render-html-not-a-port.md) D4, +[CB-EV-0010](../evidence/CB-EV-0010-render-port.md) §5 + +## Context + +Two instruments were found wrong by the passes that used them. Neither +error was in the direction that flattered us, and one of them has been +producing a standing breach report for seven passes. + +--- + +## Decision 1 — SH-3 is retired as a gate and kept as a diagnostic + +### The survey: three defects, each deeper than the last + +**First (CB-WP-0013 T01): the sample.** `--shape-budget` windows since the +last commit and is read while writing the evidence file — right after a +commit, when the window holds one or two responses. SH-1/SH-2 are location +statistics and survive that. SH-3 is a rate; at n=2 its only values are +0%, 50%, 100%. Six evidence files quoted the resulting `0.0%` as a +measurement. Fixed: below a derived minimum of 14 the tool now prints +`insufficient sample` and no verdict. + +**Second: the claimed decline was also an artifact.** This workplan's own +Purpose stated that batching *"has got worse"* — pinned history 7.8–8.6% +against per-pass windows of 1.1–6.3%. **That was wrong**, and by the same +class of error it was written to correct. The two figures come from +differently-placed windows, not from different behaviour. Withdrawn. + +**Third, and the one that decides this: the window and the metric are +systematically anti-correlated.** Measured, by re-deriving batching +independently of `cb-cost` from the raw transcripts: + +| window | responses with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses across three passes, and +37.5% in the gap between two of them** — comfortably above the 20% floor. + +The cause is not discipline. Batching requires two tool calls whose inputs +are both known at once. Orientation and survey work is full of those: read +four files, run three probes. Implementation has almost none, because each +step consumes the previous step's result — read, edit, test, read the +failure, fix. **So SH-3 measures what kind of work is happening.** + +And SH-3's window is *since the last commit*, which during a pass is +always in-pass work. **The metric is structurally incapable of reading +above ~0% in the window it is gated on.** It is not a strict gate — it can +only be met by not working. + +### The decision + +**SH-3 stops being a gate.** The 20% floor and the `[SOFT]` verdict are +removed. `cb-cost` continues to report the batching rate as a diagnostic, +over the whole session, where it reads 5.8% (57 batched of 984 responses +with tool calls). + +Four reasons, strongest first: + +1. **It cannot report what it gates on.** §above. A floor that the + measurement window structurally excludes is not a target. +2. **It measures composition, not waste.** Penalising an implementation + pass for not batching penalises it for being an implementation pass. +3. **Its own spec bounds its value.** `specs/SessionShape.md` §4: + *"eliminating every avoidable single-call turn would remove at most a + few percent of turns, worth roughly $2–4 on a $93 pass"*, and *"batch + because it is free, not because it is the lever."* It also calls SH-3 + *"the least valuable"* of the three. +4. **Seven passes, seven breach reports, zero actions.** ADR-0006 D4: + rituals cash out or go. + +### This is not retargeting to match the measurement + +`specs/SessionShape.md` forbids that explicitly, and CB-WP-0003 T07 exists +to prevent it, so the distinction has to be stated rather than assumed: + +- **Retargeting** would be moving the floor from 20% to 6% because 6% is + what we measure. That is not done and is still forbidden. +- **This** removes the gate on the argument that the *quantity* does not + deserve one — established before the disposition, from the window + structure and from the spec's own valuation. + +And the obvious objection — *retiring a metric right when it shows a +regression looks like burying it* — is answered by the second defect: the +regression was an artifact of window placement and has been withdrawn. +There is no established decline to bury. The number is still reported; +only the verdict is gone. + +## Decision 2 — AM-4a counts `--edges normal,no-proc-macro` + +ADR-0007 D4, owed. AM-4a's figure includes proc-macro crates that run in +the compiler and never reach a binary: **89,048 lines, 36.2%** of the +shipped-runtime figure, `syn` alone 66,916. + +`tools/dep-weight.py` measures `--edges normal,no-proc-macro` for the +shipped-runtime configuration. + +## Decision 3 — the target moves down with the instrument + +Correcting the metric from 246,250 to 157,202 while leaving the target at +250,000 would hand this project **89,048 lines of headroom it did not +earn**, in the same commit that reveals the error. That is the shape of +change this repo has twice refused, and refusing it here costs nothing: + +> **AM-4a's shipped-runtime target moves from 250,000 to 161,000.** + +161,000 preserves the *ratio* the ratified target expressed — 250,000 was +set against a 246,250 measurement, i.e. ~1.5% of headroom; 161,000 against +157,202 is ~2.4%. Rounding to a round number slightly *above* strict +proportionality is deliberate and is the only place this decision gives +anything back, because a target that leaves 1.5% of room is a target that +fails on a dependency's patch release. + +**AM-4b (dev-toolchain) is left alone**, at 350,000 against a 317,021 +measurement. It is not corrected in this pass because its proc-macro share +has not been measured, and correcting a second instrument on the strength +of the first one's ratio is exactly the error this ADR exists to fix. +Filed as owed. + +## Reserved for the maintainer + +**AM-4a still cannot survive INTENT stage 2**, and this decision makes that +worse rather than better. `wgpu` + `winit` is 1,741,979 marginal lines +against a target now proposed at 161,000 — **10.8×**, up from 7×. ADR-0007 +raised it; nothing here settles it. The correction below is honest +book-keeping, not headroom for stage 2, and it should not be read as +having made the stage-2 problem smaller. + +## Consequences + +- One fewer gate, and the gate registry gets its first retirement. That is + also **`GATE-REVIEW`'s first `caught` entry** — the registry's + `retire_if` says it must have *"retired, tightened, or forced the + re-justification of"* something by its review date, and it now has. +- Every AM-4a figure in this repo's history overstates the load by 36%. + Those documents are history and are not rewritten; the correction lives + in `specs/GameKernel.md` §4 and here. +- The batching diagnostic stays visible, so if in-pass batching ever + becomes possible the number will show it without a gate having to be + re-litigated first. diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md index 217abf1..55bdc46 100644 --- a/evidence/CB-EV-0001-game-kernel.md +++ b/evidence/CB-EV-0001-game-kernel.md @@ -1,5 +1,13 @@ # CB-EV-0001 — GROUND game kernel: acceptance evidence +> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a +> figures below were taken with an instrument that counted proc-macro +> crates — 89,048 lines, 36.2% — which run in the compiler and never +> reach a binary. Corrected, the same tree measures **157,202** against +> a target moved to **161,000**. The numbers below are left as the +> record of what was measured then, and are no longer live facts. + + Status: **T08 complete. AM-4 remediated and re-measured 2026-07-31.** Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per remediation option, and §5 corrects AM-12 from "uncomputable" to @@ -28,7 +36,7 @@ the property is false? A row can be *measured* and still enforce nothing. | AM-1 rule coverage (GR only) | 100% of GR-rules | 58/58 (100%) | **met** | **yes** | | AM-1b link, ground | 58 claimed rules named in the aggregate | 49/58 | **unmet** | reported | | AM-1b link, kernel | 18 K-rules named in source | 15/18 | **unmet** | reported until 2026-08-31 | -| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | +| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | | AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | **yes** | | AM-6 throughput | ≥100,000 events/s | 2,017,009 events/s (`make am6`) | **met, 20.2×** | **yes** — CB-WP-0006 T01 | | AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | **no** — no code computes the ratio | @@ -194,7 +202,7 @@ retargeted onto third-party source under audit. | Configuration | Crates | Third-party LOC | Target | Verdict | |---|---|---|---|---| -| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | +| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | | Dev toolchain (default features) | 29 | 317,021 | ≤350,000 | **met** | | Our own source | — | 3,408 | — | — | diff --git a/facts.toml b/facts.toml index 06a215c..113778d 100644 --- a/facts.toml +++ b/facts.toml @@ -10,14 +10,14 @@ generated = "2026-08-02" pin = "fc76445" [am4a_loc] -value = 246250 -text = "246,250" +value = 157202 +text = "157,202" fmt = "{:,}" by = "tools/dep-weight.py" [am4a_target] -value = 250000 -text = "250,000" +value = 161000 +text = "161,000" fmt = "{:,}" by = "tools/dep-weight.py TARGETS" diff --git a/gates.toml b/gates.toml index bcb3fd3..17b3a95 100644 --- a/gates.toml +++ b/gates.toml @@ -36,14 +36,15 @@ caught = [ retire_if = "two consecutive passes never approach the soft line, or commits get small enough that the window is always trivial" [[gate]] -id = "SH-1/SH-2/SH-3" +id = "SH-1/SH-2" name = "session-shape budget" target = "shape-budget" -checks = "mean and p90 context, and batching rate, since the last commit" +checks = "mean and p90 context since the last commit (SH-3 retired as a gate, ADR-0008 D1 — still reported as a diagnostic)" added = "2026-08-01" review_by = "2026-11-30" caught = [ "first run fired HARD at 656,574 against a 300,000 ceiling, which is what prompted the compaction before CB-WP-0008", + "CB-WP-0013: SH-3 retired from this gate. Its window is *since the last commit*, which during a pass is implementation work, and batching needs two calls whose inputs are known at once — which is orientation work. Measured: 0 batched turns in 150 in-pass responses across three passes, against 37.5% in the gap between two of them. A floor the window structurally excludes is not a target (ADR-0008 D1)", ] retire_if = "context stops correlating with cost, or the model's context handling makes the number unactionable" @@ -133,5 +134,7 @@ target = "gate-review" checks = "gates past their review date, and gates that have caught nothing" added = "2026-08-01" review_by = "2026-12-31" -caught = [] +caught = [ + "CB-WP-0013: forced SH-3's re-justification and then its retirement. `make gate-review` had reported SH-3 as a standing breach for seven passes with zero actions taken, which is this registry's own ritual test (ADR-0006 D4); asking what it had ever caught is what exposed that the metric could not read above ~0% in the window it was gated on (ADR-0008 D1)", +] retire_if = "it has retired, tightened, or forced the re-justification of nothing by its review date — then it is a ritual, and ADR-0006 D4 says rituals cash out or go" diff --git a/specs/GameKernel.md b/specs/GameKernel.md index 6bb04db..3bd23f8 100644 --- a/specs/GameKernel.md +++ b/specs/GameKernel.md @@ -204,8 +204,8 @@ evidence lands in `evidence/CB-EV-0001-game-kernel.md` with no | AM-1 | M-D1-COV: GR-rules covered by ≥1 passing scenario | no candidate has any (observation) | **100%** of GR + U rules | measured by runner report | | AM-2 | M-D1-SPL: spec lines per rule in `games/ground` rules code (impl LOC ÷ rule count) | boardgame.io ~36 LOC for the 2-move synthetic game | ≤ 40 LOC/rule, paired with AM-1 (anti-gaming pair) | measured (tokei + rule count) | | AM-3 | Synthetic-workload definition size: LOC to express the CB-RES-0001 synthetic game on our kernel | ~36 LOC (boardgame.io, measured) | ≤ 50 LOC | measured | -| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 250,000 lines** | measured (`make dep-weight`) | -| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** | measured (`make dep-weight`) | +| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features --edges normal,no-proc-macro`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 161,000 lines** (ADR-0008 D3, was 250,000) | measured (`make dep-weight`) | +| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** — *proc-macro share unmeasured; correction owed (ADR-0008 D2)* | measured (`make dep-weight`) | | ~~AM-4c~~ | M-D2-DEP: own source per third-party 100k lines | — | **WITHDRAWN from the acceptance table 2026-08-01 (CB-WP-0006 T04)** — retained as a reported diagnostic in `make dep-weight`; see §5a | diagnostic | | AM-5 | M-D2-BLD: clean release build of headless workspace | n/a (npm install ~seconds; not comparable) | ≤ 60 s on bnt-lap001, recorded not gated | measured | | AM-6 | M-D3-THR: applied events/s, synthetic workload, same machine | boardgame.io ~1,100–1,900 moves/s (best config, degrading) | **≥ 100,000/s** (stipulated target, ADR-0002) | measured | diff --git a/tools/cb-cost.py b/tools/cb-cost.py index c924ff9..dc34d0b 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -632,7 +632,7 @@ def render(rep, by_task=False, composition=False): f"need {SH3_MIN_SAMPLE}]") else: print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " - f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") + f"[diagnostic, no floor — ADR-0008 D1]") print(f"{indent} {sh['tool_calls']} tool calls in " f"{sh['responses_with_tools']} responses; " f"{sh['calls_in_batched_turns']} in batched turns") @@ -808,8 +808,14 @@ def self_test(): below.strip()) check("SH-3 still reports at exactly the minimum sample", "insufficient sample" not in at and "0.0%" in at, at.strip()) - check("SH-3 reports a real rate above the floor", - "40.0%" in good and "ok" in good, good.strip()) + check("SH-3 reports a real rate, with no verdict attached", + "40.0%" in good and "no floor" in good and "FAIL" not in good, + good.strip()) + # ADR-0008 D1: the gate is gone. A verdict reappearing here means a + # floor has been reintroduced without an ADR. + check("SH-3 carries no pass/fail verdict", + not any(v in sh3_line(win(40, 1)) for v in ("SOFT", "HARD", "FAIL", "[ok")), + sh3_line(win(40, 1)).strip()) # The one that matters: a refusal must not read as a measured zero. check("a refusal is distinguishable from a genuine 0.0%", below != sh3_line(win(SH3_MIN_SAMPLE, 0))) @@ -853,9 +859,20 @@ SH3_MIN_SAMPLE = 14 def sh3_line(win, min_sample=SH3_MIN_SAMPLE): - """SH-3, or an explicit refusal when the window cannot support a rate. + """SH-3 as a DIAGNOSTIC — no floor, no verdict (ADR-0008 D1). - Returns the line rather than printing it so the refusal is testable + Retired as a gate because the window and the metric are systematically + anti-correlated: batching needs two tool calls whose inputs are known + at once, which is orientation work, and this window is *since the last + commit*, which during a pass is implementation work. Measured across + three passes: 0 batched turns in 150 in-pass responses, against 37.5% + in the gap between two of them. A floor the window structurally + excludes is not a target. + + The minimum-sample refusal is kept. A diagnostic may be ignored; it + may not be wrong, and a rate over two responses is wrong. + + Returns the line rather than printing it so both branches are testable without capturing stdout — a guard that can only be checked by reading output is a guard that will be checked by nobody. """ @@ -865,7 +882,7 @@ def sh3_line(win, min_sample=SH3_MIN_SAMPLE): f"[----] {n} response(s) with tool calls, need {min_sample}") rate = win["SH-3_batching_rate"] return (f" SH-3 batching {100*rate:>9.1f}% " - f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%") + f"[diag] no floor — retired as a gate, ADR-0008 D1") def shape_budget(slug): diff --git a/tools/dep-weight.py b/tools/dep-weight.py index 46a5739..fefd094 100755 --- a/tools/dep-weight.py +++ b/tools/dep-weight.py @@ -28,15 +28,30 @@ import sys from repo import cargo_bin, enter_root PACKAGE = "games-ground" +# ADR-0008 D2. `--edges normal` includes proc-macro crates, which run in +# the compiler and never reach a shipped binary — 89,048 lines, 36.2% of +# what this tool used to call "what a game ships", `syn` alone 66,916. The +# shipped-runtime configuration now excludes them. +# +# AM-4b is deliberately NOT corrected here: its proc-macro share has not +# been measured, and correcting a second instrument on the strength of the +# first one's ratio is the error this change exists to fix. +PROC_MACRO_EXCLUDED = ["--edges", "normal,no-proc-macro"] CONFIGS = { - "shipped-runtime": ["--no-default-features"], + "shipped-runtime": ["--no-default-features"] + PROC_MACRO_EXCLUDED, "dev-toolchain": [], } # AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails # the build: a gate that only reports is a suggestion. +# ADR-0008 D3: the target moves down with the instrument. Leaving it at +# 250,000 against a corrected 157,202 would hand this project 89,048 lines +# of headroom it did not earn, in the same change that revealed the error. +# 161,000 keeps ~2.4% of room where 250,000 kept ~1.5% — the small +# rounding up is the only thing this decision gives back, because a target +# with 1.5% of room fails on a dependency's patch release. TARGETS = { - "shipped-runtime": 250_000, + "shipped-runtime": 161_000, "dev-toolchain": 350_000, } @@ -55,8 +70,8 @@ def crates(extra_args): ) sys.exit(1) out = subprocess.run( - [cargo, "tree", "-p", PACKAGE, "--edges", "normal", "--prefix", "none"] - + extra_args, + [cargo, "tree", "-p", PACKAGE, "--prefix", "none"] + + (extra_args if "--edges" in extra_args else ["--edges", "normal"] + extra_args), capture_output=True, text=True, check=True, @@ -125,6 +140,23 @@ def self_test(): check("cargo resolves without caller PATH setup", bool(cargo_bin()), cargo_bin() or "NOT FOUND") + # ADR-0008 D2. The exclusion must remove exactly the proc-macro crates + # and nothing else — a flag that quietly dropped a runtime dependency + # would shrink the number in the direction this metric can be gamed. + with_pm = crates(["--no-default-features"]) + without_pm = crates(["--no-default-features"] + PROC_MACRO_EXCLUDED) + dropped = set(with_pm) - set(without_pm) + check("the proc-macro exclusion drops exactly the expected crates", + dropped == {"syn", "quote", "proc-macro2", "unicode-ident", "serde_derive"}, + f"dropped {sorted(dropped)}") + check("the exclusion only ever removes crates, never adds", + set(without_pm) <= set(with_pm), + f"{len(with_pm)} -> {len(without_pm)}") + # And it must actually remove something: an exclusion that excluded + # nothing would leave the old figure while claiming the new meaning. + check("the exclusion is not a no-op", + len(dropped) > 0, f"{len(dropped)} crate(s) dropped") + check("targets defined for every configuration", set(TARGETS) == set(CONFIGS) and all( isinstance(v, int) and v > 0 for v in TARGETS.values()), diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md index 1006d74..1540dc5 100644 --- a/workplans/CB-WP-0013-instrument-corrections.md +++ b/workplans/CB-WP-0013-instrument-corrections.md @@ -125,7 +125,7 @@ eighth instance, with the real per-pass figures. ```task id: CB-WP-0013-T02 -status: todo +status: done priority: high ``` @@ -151,11 +151,43 @@ already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If the floor moves, it moves on an argument about what batching is worth, and the argument is written down before the number changes. +**Done 2026-08-02.** [ADR-0008](../decisions/ADR-0008-instrument-corrections.md) +D1 — **SH-3 is retired as a gate and kept as a diagnostic.** + +Investigating it turned up a third defect, deeper than the two this +workplan was declared on. Measured by re-deriving batching from the raw +transcripts, independently of `cb-cost`: + +| window | with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses; 37.5% in one gap.** +Batching needs two calls whose inputs are known at once — orientation +work. Implementation consumes each step's result before the next. And +SH-3's window is *since the last commit*, which during a pass is always +implementation. **The metric could not read above ~0% in the window it was +gated on.** + +**And this workplan's own Purpose was wrong.** It stated batching *"has +got worse"* (7.8–8.6% historical against 1.1–6.3% per-pass). Those come +from differently-placed windows, not different behaviour. Withdrawn — the +same class of error, in the pass written to correct it. + +The retirement is *not* retargeting to match the measurement: the floor +was not moved to 6%, the gate was removed on an argument about what the +quantity is worth, established before the disposition. The number is still +reported; only the verdict is gone. + ## Task: correct AM-4a to count what ships ```task id: CB-WP-0013-T03 -status: todo +status: done priority: high ``` @@ -175,6 +207,30 @@ be able to drop crates it should have counted: assert that the proc-macro set is exactly the five expected crates and that removing the flag restores the old figure. +**Done 2026-08-02.** ADR-0008 D2/D3. + +``` +shipped-runtime 18 crates 157,202 lines [ok target 161,000] +``` + +**The target moved down with the instrument**, 250,000 → 161,000, so the +correction hands back essentially nothing: headroom goes from 3,750 to +3,798. 161,000 keeps ~2.4% of room where 250,000 kept ~1.5%; that small +rounding up is the only thing given back, because a target with 1.5% of +room fails on a dependency's patch release. + +Three controls, all green: the exclusion drops **exactly** the five +expected crates, only ever removes and never adds, and is not a no-op. + +**The DFD gate then caught the follow-on**, which is what it is for: three +historical documents carried live `fact:am4a_loc` tags for a number that +had changed. They are history and were not rewritten — the tags were +removed and both documents carry a supersession banner instead. + +**AM-4b is deliberately not corrected.** Its proc-macro share is +unmeasured, and correcting a second instrument on the strength of the +first one's ratio is the error this task exists to fix. Filed as owed. + ## Task: evidence ```task From 768eee7d2718ea8802d04f860f5057d6a415eec6 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 2 Aug 2026 07:32:07 +0200 Subject: [PATCH 4/4] =?UTF-8?q?CB-WP-0013-T04:=20evidence=20=E2=80=94=20an?= =?UTF-8?q?d=20a=20rule=20about=20quoting=20your=20own=20cost?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CB-EV-0011. Three instruments were wrong; the pass found the third defect itself and committed the fourth. The rule this pass should leave behind: an evidence file quotes the PREVIOUS pass's final cost, not its own. Three for three, all low by a strikingly consistent ~2/3: CB-WP-0009 $6.73 reported -> $11.31 final CB-WP-0011 $4.23 reported -> $7.02 final CB-WP-0012 $8.82 reported -> $14.52 final And this time it changed a conclusion. CB-EV-0010 reported the first priced tier comparison as "0.123 at L against 0.099 at S — 24% more". The final figures are 0.150 against 0.099 — 51% more. Tier L costs twice what that comparison claimed. It still deleted a port that would have been rewritten and found errors of 25x and 85x, so it still looks worth it, but the price doubled and the file that made the argument could not have known. Also closed rather than deferred again: cb-play's "third mode" never arrived. --serve is a play flag selecting which Policy fills a human seat; the binary still has two modes, and CB-EV-0010 §6 was wrong. Carried for two passes, resolved, stopped. Recorded as a near-miss: an instrument reading that looked impossible (53 tool calls in 53 responses) turned out to be correct. One independent re-derivation before writing anything stopped a false accusation — in the pass whose whole subject is instruments reporting things that are not so. gate-review: 0 due, 0 silent, 0 drifted. Every gate in the registry has a caught entry for the first time. Co-Authored-By: Claude Opus 5 --- WORK-RECORDS.md | 4 + evidence/CB-EV-0011-instrument-corrections.md | 181 ++++++++++++++++++ .../CB-WP-0013-instrument-corrections.md | 22 ++- 3 files changed, 205 insertions(+), 2 deletions(-) create mode 100644 evidence/CB-EV-0011-instrument-corrections.md diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 518fb91..3920995 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -95,3 +95,7 @@ | task | CB-WP-0012-T03 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T04 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T05 | done | — | workplans/CB-WP-0012-render-port.md | +| task | CB-WP-0013-T01 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T02 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T03 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T04 | done | — | workplans/CB-WP-0013-instrument-corrections.md | diff --git a/evidence/CB-EV-0011-instrument-corrections.md b/evidence/CB-EV-0011-instrument-corrections.md new file mode 100644 index 0000000..7012016 --- /dev/null +++ b/evidence/CB-EV-0011-instrument-corrections.md @@ -0,0 +1,181 @@ +# CB-EV-0011 — three instruments, and a rule about quoting yourself + +CB-WP-0013 T04. Measured 2026-08-02 at `d2c3f34`. Pass kind `meta`, +tier **M** (chaos d4=3, no override). Declaration 8 of 12. + +The pass was declared to fix two instruments. It found a third defect in +one of them, and a fourth in itself. + +--- + +## 1. What was actually wrong with SH-3 + +The declaration named one defect. There were three, each deeper. + +**1. The sample.** `--shape-budget` windows since the last commit and is +read while writing the evidence file — right after a commit, when the +window holds one or two responses. SH-1/SH-2 are location statistics and +survive that; SH-3 is a rate, and at n=2 its only values are 0%, 50%, +100%. Six evidence files quoted the resulting `0.0%` as a measurement. + +**2. The claimed decline — which was this pass's own error.** The +declaration stated batching *"has got worse"*, citing 7.8–8.6% pinned +history against 1.1–6.3% per-pass windows. Those are differently-*placed* +windows, not different behaviour. **Withdrawn.** The pass declared to +correct a trusted-arithmetic error opened by committing one. + +**3. The one that decided it.** Re-deriving batching from the raw +transcripts, independently of `cb-cost`: + +| window | with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses. 37.5% in one gap** — above +the floor the gate had been failing for seven passes. + +Batching needs two tool calls whose inputs are known at once. That is +orientation: read four files, run three probes. Implementation has almost +none, because each step consumes the previous step's result. And SH-3's +window is *since the last commit*, which during a pass is implementation. +**The metric could not read above ~0% in the window it was gated on.** + +A gate meetable only by not working is not a target. SH-3 is retired as a +gate (ADR-0008 D1) and kept as a diagnostic over the whole session, where +it reads 5.8%. + +**On the appearance of burying it:** retiring a metric that shows a +regression looks bad, and should. The answer is defect 2 — the regression +was an artifact and has been withdrawn, so there is no established decline +to bury. The number is still printed; only the verdict is gone. And the +floor was *not* moved to 6%, which is the thing `specs/SessionShape.md` +actually forbids. + +## 2. A near-miss worth recording: I almost reported a defect that was not there + +Midway through, a bounded window reported `53 tool calls in 53 responses; +0 in batched turns` — exactly one call per response, which looked like the +counter had collapsed. I had batched repeatedly in this session, so the +reading looked impossible. + +It was correct. Re-deriving from the transcripts independently gave the +same 0, and the 131 batched responses that do exist all fall *outside* +those windows. The instrument was right and my expectation was wrong. + +Recorded because the **false-accusation class** is already on this +project's record, and because the thing that stopped it was cheap: one +independent re-derivation before writing anything down. The cost of +checking was two minutes; the cost of not checking would have been an +evidence file alleging a defect in a tool that did not have one — and this +pass's whole subject is instruments reporting things that are not so. + +## 3. AM-4a, corrected, gives back nothing + +``` +before 23 crates 246,250 lines [ok target 250,000] headroom 3,750 +after 18 crates 157,202 lines [ok target 161,000] headroom 3,798 +``` + +The instrument dropped 89,048 lines of proc-macro code that never reaches +a binary; **the target moved down with it** (ADR-0008 D3). Leaving the +target at 250,000 would have handed this project 89,048 lines of headroom +it did not earn, in the same change that revealed the error. + +Three controls, all green: the exclusion drops **exactly** the five +expected crates, only ever removes and never adds, and is not a no-op. + +**The DFD gate then caught the follow-on**, which is exactly what it is +for: three historical documents carried live `fact:am4a_loc` tags for a +number that had changed. They are history and were not rewritten — the +tags came off and both carry a supersession banner. A gate catching the +second-order consequence of a correction, unprompted, is the best evidence +yet that the registry is doing something. + +**AM-4b is deliberately not corrected.** Its proc-macro share is +unmeasured, and correcting a second instrument on the strength of the +first one's ratio is the error this pass exists to fix. Owed. + +## 4. The rule this pass should leave behind: stop quoting your own cost + +| pass | reported mid-pass | final | understated by | +|---|---|---|---| +| CB-WP-0009 | $6.73 (CB-EV-0008) | $11.31 | 68% | +| CB-WP-0011 | $4.23 / 0.094 (CB-EV-0009) | $7.02 / 0.099 | 66% | +| CB-WP-0012 | $8.82 / 0.123 (CB-EV-0010) | **$14.52 / 0.150** | 65% | + +Three for three, always low, by a strikingly consistent ~2/3. And this +time it **changed a conclusion**, not just a figure: + +> CB-EV-0010 §5 reported the first priced tier comparison as *"0.123 +> $/response at L against 0.099 at S — 24% more."* The final figures are +> **0.150 against 0.099 — 51% more.** The direction holds; the magnitude +> was understated by half. + +Tier L is twice as expensive as that comparison claimed. It still found +errors of 25× and 85× in CB-WP-0012 and deleted a port that would have +been rewritten, so it still looks worth it — but the price of the argument +doubled and the evidence file that made it could not have known. + +**Proposed rule, for the next pass to adopt or reject:** an evidence file +quotes the *previous* pass's final cost and does not quote its own. Its +own is knowable only after the commit that closes it, so any figure it +prints is a floor, and this project has now printed three of them and let +one carry a conclusion. + +## 5. `cb-play`'s third mode never arrived + +CB-EV-0009 §5 said a third mode would be the second use at which +`cb-play`'s single-binary shape should be reconsidered. CB-EV-0010 §6 +recorded that it had arrived, with `--serve`, and that the pass had not +acted. **CB-EV-0010 was wrong.** + +`--serve` is registered in `parse_args` as a *play flag*: it selects which +`Policy` fills a human seat, terminal or browser. The binary still has +exactly two modes — play a game, or read a recording — and they still take +disjoint flags. The trigger did not fire. + +Carried for two passes, resolved here, and **stopped**: there is no split +to do, and the standing item is closed rather than deferred again. + +## 6. Cost and shape + +| pass | kind | responses | cost | $/response | +|---|---|---|---|---| +| CB-WP-0011 | product | 71 | $7.02 | 0.099 | +| CB-WP-0012 | product | 97 | $14.52 | 0.150 | +| CB-WP-0013 | meta | 34+ | $5.78+ | provisional, see §4 | + +**Meta budget: 21%**, inside the soft 25%, without needing ADR-0006 D2's +repair exemption — though both halves would have qualified for it. + +**SH-1 290,475 `[SOFT]`**, drifting toward the hard line; the remedy is a +compaction and it is free. **SH-3 now reads `insufficient sample (1 +response with tool calls, need 14)`** — the fix working in production on +its first real reading, where the old instrument would have printed a +confident `0.0%`. + +**`gate-review`: 0 due, 0 silent, 0 drifted.** Every gate in the registry +now has a `caught` entry, for the first time. GATE-REVIEW earned its own +by forcing SH-3's re-justification — asking what a gate had ever caught is +what exposed that it could not read what it gated on. + +## 7. Open + +- **AM-4b's proc-macro share is unmeasured.** §3. +- **The self-quoting rule in §4 is proposed, not adopted.** +- **AM-4a still cannot survive INTENT stage 2**, and this pass made the + ratio worse: `wgpu` + `winit` is 1,741,979 against a target now 161,000 + — **10.8×**, up from 7×. Reserved for the maintainer, twice over now. +- **ADR-0007 Decision 3's acquisition rule** remains an argument awaiting + ratification. +- **INTENT stage 1 stays open**; the emitted JavaScript has still never + been executed. +- **Tier M's merged survey-and-ADR was the right weight.** One document, + no separate review, and the pass still found a defect deeper than the + one it was declared on — because the finding came from *measuring*, not + from an extra review round. That is one data point for the tier table + and against the assumption that depth comes from process weight. diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md index 1540dc5..dccf1c1 100644 --- a/workplans/CB-WP-0013-instrument-corrections.md +++ b/workplans/CB-WP-0013-instrument-corrections.md @@ -2,7 +2,7 @@ id: CB-WP-0013 kind: meta title: "Two instruments that were wrong, and the record they wrote" -status: todo +status: done --- # Purpose @@ -235,7 +235,7 @@ first one's ratio is the error this task exists to fix. Filed as owed. ```task id: CB-WP-0013-T04 -status: todo +status: done priority: medium ``` @@ -251,3 +251,21 @@ Also carry the one that is now overdue: **`cb-play` is three modes in one binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass did not act. Either do it or state why not, and stop carrying it. + +**Done 2026-08-02.** +[CB-EV-0011](../evidence/CB-EV-0011-instrument-corrections.md). + +- **The self-quoting rule is proposed.** Three passes have now reported + their own cost mid-flight and all three were low by ~2/3 — and this + time it changed a conclusion: CB-EV-0010's tier comparison said L cost + *24% more* than S; the final figures say **51%**. +- **`cb-play`'s third mode never arrived.** `--serve` is a *play flag* + selecting which `Policy` fills a human seat. The binary still has two + modes. CB-EV-0010 §6 was wrong; the item is **closed, not deferred**. +- **A near-miss recorded**: an instrument reading that looked impossible + turned out to be correct, and one independent re-derivation before + writing anything stopped a false accusation. +- **Tier M was the right weight** — one merged document, no separate + review, and the pass still found a defect deeper than the one it was + declared on, because the finding came from measuring rather than from + an extra review round.