diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 518fb91..3920995 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -95,3 +95,7 @@ | task | CB-WP-0012-T03 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T04 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T05 | done | — | workplans/CB-WP-0012-render-port.md | +| task | CB-WP-0013-T01 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T02 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T03 | done | — | workplans/CB-WP-0013-instrument-corrections.md | +| task | CB-WP-0013-T04 | done | — | workplans/CB-WP-0013-instrument-corrections.md | diff --git a/decisions/ADR-0004-am4-ratification.md b/decisions/ADR-0004-am4-ratification.md index fde8be8..86cd83a 100644 --- a/decisions/ADR-0004-am4-ratification.md +++ b/decisions/ADR-0004-am4-ratification.md @@ -1,5 +1,13 @@ # ADR-0004: ratify AM-4a and AM-4b +> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a +> figures below were taken with an instrument that counted proc-macro +> crates — 89,048 lines, 36.2% — which run in the compiler and never +> reach a binary. Corrected, the same tree measures **157,202** against +> a target moved to **161,000**. The numbers below are left as the +> record of what was measured then, and are no longer live facts. + + status: accepted date: 2026-07-31 decided by: maintainer (Bernd Worsch), 2026-07-31 @@ -49,7 +57,7 @@ At the time of the retarget, `make dep-weight`: | configuration | crates | third-party LOC | |---|---|---| -| shipped-runtime (`--no-default-features`) | 23 | **246,250** | +| shipped-runtime (`--no-default-features`) | 23 | **246,250** | | dev-toolchain (default features) | 29 | **317,021** | | own source | — | 3,443 | diff --git a/decisions/ADR-0008-instrument-corrections.md b/decisions/ADR-0008-instrument-corrections.md new file mode 100644 index 0000000..79057f1 --- /dev/null +++ b/decisions/ADR-0008-instrument-corrections.md @@ -0,0 +1,158 @@ +# ADR-0008: retire SH-3 as a gate; correct AM-4a to count what ships + +status: accepted +date: 2026-08-02 +decided by: agent, under the standing loop authorization. One item is +reserved for the maintainer — see §Reserved. +tier: M (structural M — changes how the loop constrains its own operation: +a gate floor and a budget instrument, InnerLoop v1.6 / ADR-0006 D5; +chaos d4=3 → no override). Tier M merges survey and ADR into one document, +which this is; the adversarial review is optional and was not run. +references: [CB-WP-0013](../workplans/CB-WP-0013-instrument-corrections.md), +[SessionShape.md](../specs/SessionShape.md) §4, +[ADR-0006](ADR-0006-adaptive-control-gates.md) D1/D2/D4, +[ADR-0007](ADR-0007-render-html-not-a-port.md) D4, +[CB-EV-0010](../evidence/CB-EV-0010-render-port.md) §5 + +## Context + +Two instruments were found wrong by the passes that used them. Neither +error was in the direction that flattered us, and one of them has been +producing a standing breach report for seven passes. + +--- + +## Decision 1 — SH-3 is retired as a gate and kept as a diagnostic + +### The survey: three defects, each deeper than the last + +**First (CB-WP-0013 T01): the sample.** `--shape-budget` windows since the +last commit and is read while writing the evidence file — right after a +commit, when the window holds one or two responses. SH-1/SH-2 are location +statistics and survive that. SH-3 is a rate; at n=2 its only values are +0%, 50%, 100%. Six evidence files quoted the resulting `0.0%` as a +measurement. Fixed: below a derived minimum of 14 the tool now prints +`insufficient sample` and no verdict. + +**Second: the claimed decline was also an artifact.** This workplan's own +Purpose stated that batching *"has got worse"* — pinned history 7.8–8.6% +against per-pass windows of 1.1–6.3%. **That was wrong**, and by the same +class of error it was written to correct. The two figures come from +differently-placed windows, not from different behaviour. Withdrawn. + +**Third, and the one that decides this: the window and the metric are +systematically anti-correlated.** Measured, by re-deriving batching +independently of `cb-cost` from the raw transcripts: + +| window | responses with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses across three passes, and +37.5% in the gap between two of them** — comfortably above the 20% floor. + +The cause is not discipline. Batching requires two tool calls whose inputs +are both known at once. Orientation and survey work is full of those: read +four files, run three probes. Implementation has almost none, because each +step consumes the previous step's result — read, edit, test, read the +failure, fix. **So SH-3 measures what kind of work is happening.** + +And SH-3's window is *since the last commit*, which during a pass is +always in-pass work. **The metric is structurally incapable of reading +above ~0% in the window it is gated on.** It is not a strict gate — it can +only be met by not working. + +### The decision + +**SH-3 stops being a gate.** The 20% floor and the `[SOFT]` verdict are +removed. `cb-cost` continues to report the batching rate as a diagnostic, +over the whole session, where it reads 5.8% (57 batched of 984 responses +with tool calls). + +Four reasons, strongest first: + +1. **It cannot report what it gates on.** §above. A floor that the + measurement window structurally excludes is not a target. +2. **It measures composition, not waste.** Penalising an implementation + pass for not batching penalises it for being an implementation pass. +3. **Its own spec bounds its value.** `specs/SessionShape.md` §4: + *"eliminating every avoidable single-call turn would remove at most a + few percent of turns, worth roughly $2–4 on a $93 pass"*, and *"batch + because it is free, not because it is the lever."* It also calls SH-3 + *"the least valuable"* of the three. +4. **Seven passes, seven breach reports, zero actions.** ADR-0006 D4: + rituals cash out or go. + +### This is not retargeting to match the measurement + +`specs/SessionShape.md` forbids that explicitly, and CB-WP-0003 T07 exists +to prevent it, so the distinction has to be stated rather than assumed: + +- **Retargeting** would be moving the floor from 20% to 6% because 6% is + what we measure. That is not done and is still forbidden. +- **This** removes the gate on the argument that the *quantity* does not + deserve one — established before the disposition, from the window + structure and from the spec's own valuation. + +And the obvious objection — *retiring a metric right when it shows a +regression looks like burying it* — is answered by the second defect: the +regression was an artifact of window placement and has been withdrawn. +There is no established decline to bury. The number is still reported; +only the verdict is gone. + +## Decision 2 — AM-4a counts `--edges normal,no-proc-macro` + +ADR-0007 D4, owed. AM-4a's figure includes proc-macro crates that run in +the compiler and never reach a binary: **89,048 lines, 36.2%** of the +shipped-runtime figure, `syn` alone 66,916. + +`tools/dep-weight.py` measures `--edges normal,no-proc-macro` for the +shipped-runtime configuration. + +## Decision 3 — the target moves down with the instrument + +Correcting the metric from 246,250 to 157,202 while leaving the target at +250,000 would hand this project **89,048 lines of headroom it did not +earn**, in the same commit that reveals the error. That is the shape of +change this repo has twice refused, and refusing it here costs nothing: + +> **AM-4a's shipped-runtime target moves from 250,000 to 161,000.** + +161,000 preserves the *ratio* the ratified target expressed — 250,000 was +set against a 246,250 measurement, i.e. ~1.5% of headroom; 161,000 against +157,202 is ~2.4%. Rounding to a round number slightly *above* strict +proportionality is deliberate and is the only place this decision gives +anything back, because a target that leaves 1.5% of room is a target that +fails on a dependency's patch release. + +**AM-4b (dev-toolchain) is left alone**, at 350,000 against a 317,021 +measurement. It is not corrected in this pass because its proc-macro share +has not been measured, and correcting a second instrument on the strength +of the first one's ratio is exactly the error this ADR exists to fix. +Filed as owed. + +## Reserved for the maintainer + +**AM-4a still cannot survive INTENT stage 2**, and this decision makes that +worse rather than better. `wgpu` + `winit` is 1,741,979 marginal lines +against a target now proposed at 161,000 — **10.8×**, up from 7×. ADR-0007 +raised it; nothing here settles it. The correction below is honest +book-keeping, not headroom for stage 2, and it should not be read as +having made the stage-2 problem smaller. + +## Consequences + +- One fewer gate, and the gate registry gets its first retirement. That is + also **`GATE-REVIEW`'s first `caught` entry** — the registry's + `retire_if` says it must have *"retired, tightened, or forced the + re-justification of"* something by its review date, and it now has. +- Every AM-4a figure in this repo's history overstates the load by 36%. + Those documents are history and are not rewritten; the correction lives + in `specs/GameKernel.md` §4 and here. +- The batching diagnostic stays visible, so if in-pass batching ever + becomes possible the number will show it without a gate having to be + re-litigated first. diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md index 217abf1..55bdc46 100644 --- a/evidence/CB-EV-0001-game-kernel.md +++ b/evidence/CB-EV-0001-game-kernel.md @@ -1,5 +1,13 @@ # CB-EV-0001 — GROUND game kernel: acceptance evidence +> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a +> figures below were taken with an instrument that counted proc-macro +> crates — 89,048 lines, 36.2% — which run in the compiler and never +> reach a binary. Corrected, the same tree measures **157,202** against +> a target moved to **161,000**. The numbers below are left as the +> record of what was measured then, and are no longer live facts. + + Status: **T08 complete. AM-4 remediated and re-measured 2026-07-31.** Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per remediation option, and §5 corrects AM-12 from "uncomputable" to @@ -28,7 +36,7 @@ the property is false? A row can be *measured* and still enforce nothing. | AM-1 rule coverage (GR only) | 100% of GR-rules | 58/58 (100%) | **met** | **yes** | | AM-1b link, ground | 58 claimed rules named in the aggregate | 49/58 | **unmet** | reported | | AM-1b link, kernel | 18 K-rules named in source | 15/18 | **unmet** | reported until 2026-08-31 | -| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | +| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | | AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | **yes** | | AM-6 throughput | ≥100,000 events/s | 2,017,009 events/s (`make am6`) | **met, 20.2×** | **yes** — CB-WP-0006 T01 | | AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | **no** — no code computes the ratio | @@ -194,7 +202,7 @@ retargeted onto third-party source under audit. | Configuration | Crates | Third-party LOC | Target | Verdict | |---|---|---|---|---| -| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | +| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | | Dev toolchain (default features) | 29 | 317,021 | ≤350,000 | **met** | | Our own source | — | 3,408 | — | — | diff --git a/evidence/CB-EV-0011-instrument-corrections.md b/evidence/CB-EV-0011-instrument-corrections.md new file mode 100644 index 0000000..7012016 --- /dev/null +++ b/evidence/CB-EV-0011-instrument-corrections.md @@ -0,0 +1,181 @@ +# CB-EV-0011 — three instruments, and a rule about quoting yourself + +CB-WP-0013 T04. Measured 2026-08-02 at `d2c3f34`. Pass kind `meta`, +tier **M** (chaos d4=3, no override). Declaration 8 of 12. + +The pass was declared to fix two instruments. It found a third defect in +one of them, and a fourth in itself. + +--- + +## 1. What was actually wrong with SH-3 + +The declaration named one defect. There were three, each deeper. + +**1. The sample.** `--shape-budget` windows since the last commit and is +read while writing the evidence file — right after a commit, when the +window holds one or two responses. SH-1/SH-2 are location statistics and +survive that; SH-3 is a rate, and at n=2 its only values are 0%, 50%, +100%. Six evidence files quoted the resulting `0.0%` as a measurement. + +**2. The claimed decline — which was this pass's own error.** The +declaration stated batching *"has got worse"*, citing 7.8–8.6% pinned +history against 1.1–6.3% per-pass windows. Those are differently-*placed* +windows, not different behaviour. **Withdrawn.** The pass declared to +correct a trusted-arithmetic error opened by committing one. + +**3. The one that decided it.** Re-deriving batching from the raw +transcripts, independently of `cb-cost`: + +| window | with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses. 37.5% in one gap** — above +the floor the gate had been failing for seven passes. + +Batching needs two tool calls whose inputs are known at once. That is +orientation: read four files, run three probes. Implementation has almost +none, because each step consumes the previous step's result. And SH-3's +window is *since the last commit*, which during a pass is implementation. +**The metric could not read above ~0% in the window it was gated on.** + +A gate meetable only by not working is not a target. SH-3 is retired as a +gate (ADR-0008 D1) and kept as a diagnostic over the whole session, where +it reads 5.8%. + +**On the appearance of burying it:** retiring a metric that shows a +regression looks bad, and should. The answer is defect 2 — the regression +was an artifact and has been withdrawn, so there is no established decline +to bury. The number is still printed; only the verdict is gone. And the +floor was *not* moved to 6%, which is the thing `specs/SessionShape.md` +actually forbids. + +## 2. A near-miss worth recording: I almost reported a defect that was not there + +Midway through, a bounded window reported `53 tool calls in 53 responses; +0 in batched turns` — exactly one call per response, which looked like the +counter had collapsed. I had batched repeatedly in this session, so the +reading looked impossible. + +It was correct. Re-deriving from the transcripts independently gave the +same 0, and the 131 batched responses that do exist all fall *outside* +those windows. The instrument was right and my expectation was wrong. + +Recorded because the **false-accusation class** is already on this +project's record, and because the thing that stopped it was cheap: one +independent re-derivation before writing anything down. The cost of +checking was two minutes; the cost of not checking would have been an +evidence file alleging a defect in a tool that did not have one — and this +pass's whole subject is instruments reporting things that are not so. + +## 3. AM-4a, corrected, gives back nothing + +``` +before 23 crates 246,250 lines [ok target 250,000] headroom 3,750 +after 18 crates 157,202 lines [ok target 161,000] headroom 3,798 +``` + +The instrument dropped 89,048 lines of proc-macro code that never reaches +a binary; **the target moved down with it** (ADR-0008 D3). Leaving the +target at 250,000 would have handed this project 89,048 lines of headroom +it did not earn, in the same change that revealed the error. + +Three controls, all green: the exclusion drops **exactly** the five +expected crates, only ever removes and never adds, and is not a no-op. + +**The DFD gate then caught the follow-on**, which is exactly what it is +for: three historical documents carried live `fact:am4a_loc` tags for a +number that had changed. They are history and were not rewritten — the +tags came off and both carry a supersession banner. A gate catching the +second-order consequence of a correction, unprompted, is the best evidence +yet that the registry is doing something. + +**AM-4b is deliberately not corrected.** Its proc-macro share is +unmeasured, and correcting a second instrument on the strength of the +first one's ratio is the error this pass exists to fix. Owed. + +## 4. The rule this pass should leave behind: stop quoting your own cost + +| pass | reported mid-pass | final | understated by | +|---|---|---|---| +| CB-WP-0009 | $6.73 (CB-EV-0008) | $11.31 | 68% | +| CB-WP-0011 | $4.23 / 0.094 (CB-EV-0009) | $7.02 / 0.099 | 66% | +| CB-WP-0012 | $8.82 / 0.123 (CB-EV-0010) | **$14.52 / 0.150** | 65% | + +Three for three, always low, by a strikingly consistent ~2/3. And this +time it **changed a conclusion**, not just a figure: + +> CB-EV-0010 §5 reported the first priced tier comparison as *"0.123 +> $/response at L against 0.099 at S — 24% more."* The final figures are +> **0.150 against 0.099 — 51% more.** The direction holds; the magnitude +> was understated by half. + +Tier L is twice as expensive as that comparison claimed. It still found +errors of 25× and 85× in CB-WP-0012 and deleted a port that would have +been rewritten, so it still looks worth it — but the price of the argument +doubled and the evidence file that made it could not have known. + +**Proposed rule, for the next pass to adopt or reject:** an evidence file +quotes the *previous* pass's final cost and does not quote its own. Its +own is knowable only after the commit that closes it, so any figure it +prints is a floor, and this project has now printed three of them and let +one carry a conclusion. + +## 5. `cb-play`'s third mode never arrived + +CB-EV-0009 §5 said a third mode would be the second use at which +`cb-play`'s single-binary shape should be reconsidered. CB-EV-0010 §6 +recorded that it had arrived, with `--serve`, and that the pass had not +acted. **CB-EV-0010 was wrong.** + +`--serve` is registered in `parse_args` as a *play flag*: it selects which +`Policy` fills a human seat, terminal or browser. The binary still has +exactly two modes — play a game, or read a recording — and they still take +disjoint flags. The trigger did not fire. + +Carried for two passes, resolved here, and **stopped**: there is no split +to do, and the standing item is closed rather than deferred again. + +## 6. Cost and shape + +| pass | kind | responses | cost | $/response | +|---|---|---|---|---| +| CB-WP-0011 | product | 71 | $7.02 | 0.099 | +| CB-WP-0012 | product | 97 | $14.52 | 0.150 | +| CB-WP-0013 | meta | 34+ | $5.78+ | provisional, see §4 | + +**Meta budget: 21%**, inside the soft 25%, without needing ADR-0006 D2's +repair exemption — though both halves would have qualified for it. + +**SH-1 290,475 `[SOFT]`**, drifting toward the hard line; the remedy is a +compaction and it is free. **SH-3 now reads `insufficient sample (1 +response with tool calls, need 14)`** — the fix working in production on +its first real reading, where the old instrument would have printed a +confident `0.0%`. + +**`gate-review`: 0 due, 0 silent, 0 drifted.** Every gate in the registry +now has a `caught` entry, for the first time. GATE-REVIEW earned its own +by forcing SH-3's re-justification — asking what a gate had ever caught is +what exposed that it could not read what it gated on. + +## 7. Open + +- **AM-4b's proc-macro share is unmeasured.** §3. +- **The self-quoting rule in §4 is proposed, not adopted.** +- **AM-4a still cannot survive INTENT stage 2**, and this pass made the + ratio worse: `wgpu` + `winit` is 1,741,979 against a target now 161,000 + — **10.8×**, up from 7×. Reserved for the maintainer, twice over now. +- **ADR-0007 Decision 3's acquisition rule** remains an argument awaiting + ratification. +- **INTENT stage 1 stays open**; the emitted JavaScript has still never + been executed. +- **Tier M's merged survey-and-ADR was the right weight.** One document, + no separate review, and the pass still found a defect deeper than the + one it was declared on — because the finding came from *measuring*, not + from an extra review round. That is one data point for the tier table + and against the assumption that depth comes from process weight. diff --git a/facts.toml b/facts.toml index 06a215c..113778d 100644 --- a/facts.toml +++ b/facts.toml @@ -10,14 +10,14 @@ generated = "2026-08-02" pin = "fc76445" [am4a_loc] -value = 246250 -text = "246,250" +value = 157202 +text = "157,202" fmt = "{:,}" by = "tools/dep-weight.py" [am4a_target] -value = 250000 -text = "250,000" +value = 161000 +text = "161,000" fmt = "{:,}" by = "tools/dep-weight.py TARGETS" diff --git a/gates.toml b/gates.toml index bcb3fd3..17b3a95 100644 --- a/gates.toml +++ b/gates.toml @@ -36,14 +36,15 @@ caught = [ retire_if = "two consecutive passes never approach the soft line, or commits get small enough that the window is always trivial" [[gate]] -id = "SH-1/SH-2/SH-3" +id = "SH-1/SH-2" name = "session-shape budget" target = "shape-budget" -checks = "mean and p90 context, and batching rate, since the last commit" +checks = "mean and p90 context since the last commit (SH-3 retired as a gate, ADR-0008 D1 — still reported as a diagnostic)" added = "2026-08-01" review_by = "2026-11-30" caught = [ "first run fired HARD at 656,574 against a 300,000 ceiling, which is what prompted the compaction before CB-WP-0008", + "CB-WP-0013: SH-3 retired from this gate. Its window is *since the last commit*, which during a pass is implementation work, and batching needs two calls whose inputs are known at once — which is orientation work. Measured: 0 batched turns in 150 in-pass responses across three passes, against 37.5% in the gap between two of them. A floor the window structurally excludes is not a target (ADR-0008 D1)", ] retire_if = "context stops correlating with cost, or the model's context handling makes the number unactionable" @@ -133,5 +134,7 @@ target = "gate-review" checks = "gates past their review date, and gates that have caught nothing" added = "2026-08-01" review_by = "2026-12-31" -caught = [] +caught = [ + "CB-WP-0013: forced SH-3's re-justification and then its retirement. `make gate-review` had reported SH-3 as a standing breach for seven passes with zero actions taken, which is this registry's own ritual test (ADR-0006 D4); asking what it had ever caught is what exposed that the metric could not read above ~0% in the window it was gated on (ADR-0008 D1)", +] retire_if = "it has retired, tightened, or forced the re-justification of nothing by its review date — then it is a ritual, and ADR-0006 D4 says rituals cash out or go" diff --git a/specs/GameKernel.md b/specs/GameKernel.md index 6bb04db..3bd23f8 100644 --- a/specs/GameKernel.md +++ b/specs/GameKernel.md @@ -204,8 +204,8 @@ evidence lands in `evidence/CB-EV-0001-game-kernel.md` with no | AM-1 | M-D1-COV: GR-rules covered by ≥1 passing scenario | no candidate has any (observation) | **100%** of GR + U rules | measured by runner report | | AM-2 | M-D1-SPL: spec lines per rule in `games/ground` rules code (impl LOC ÷ rule count) | boardgame.io ~36 LOC for the 2-move synthetic game | ≤ 40 LOC/rule, paired with AM-1 (anti-gaming pair) | measured (tokei + rule count) | | AM-3 | Synthetic-workload definition size: LOC to express the CB-RES-0001 synthetic game on our kernel | ~36 LOC (boardgame.io, measured) | ≤ 50 LOC | measured | -| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 250,000 lines** | measured (`make dep-weight`) | -| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** | measured (`make dep-weight`) | +| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features --edges normal,no-proc-macro`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 161,000 lines** (ADR-0008 D3, was 250,000) | measured (`make dep-weight`) | +| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** — *proc-macro share unmeasured; correction owed (ADR-0008 D2)* | measured (`make dep-weight`) | | ~~AM-4c~~ | M-D2-DEP: own source per third-party 100k lines | — | **WITHDRAWN from the acceptance table 2026-08-01 (CB-WP-0006 T04)** — retained as a reported diagnostic in `make dep-weight`; see §5a | diagnostic | | AM-5 | M-D2-BLD: clean release build of headless workspace | n/a (npm install ~seconds; not comparable) | ≤ 60 s on bnt-lap001, recorded not gated | measured | | AM-6 | M-D3-THR: applied events/s, synthetic workload, same machine | boardgame.io ~1,100–1,900 moves/s (best config, degrading) | **≥ 100,000/s** (stipulated target, ADR-0002) | measured | diff --git a/specs/SessionShape.md b/specs/SessionShape.md index 44e4d38..6b9a521 100644 --- a/specs/SessionShape.md +++ b/specs/SessionShape.md @@ -136,6 +136,47 @@ workplan — the failure mode InnerLoop v1.1 rule 13 exists for, and the `history/260731-inner-loop-rule-audit.md` identified as having no executable defence. +### Correction (CB-WP-0013 T01): SH-3 was never read as measured + +**Six evidence files quote `SH-3 0.0%`, and none of them measured it.** + +`cb-cost --shape-budget` reports the window **since the last commit**, and +it is read while writing the evidence file — immediately *after* a commit, +when that window holds one or two responses. SH-1 and SH-2 are location +statistics and survive a sample of two. SH-3 is a **rate**, and at n=2 its +only possible values are 0%, 50% and 100%. One window, three metrics, and +it is only wrong for one of them. + +Measured over real per-pass windows instead: + +| window since | responses | SH-3 | +|---|---:|---:| +| `0d2ab22` | 115 | 6.2% | +| `6fb0aea` | 113 | 6.3% | +| `331e7e9` | 95 | 1.1% | +| `c5fa610` | 74 | 1.4% | +| `883b608` | 70 | 1.4% | +| `84d6886` | 24 | 4.3% | + +The substantive claim survives — SH-3 is badly unmet — but the constant +0.0% hid something the real numbers show: against the **7.8–8.6%** pinned +above, **batching has got worse**, and a metric frozen at a constant could +not have shown a trend at all. Six passes reported a breach and the breach +was moving underneath them. + +**This is the ninth recorded instance of the trusted-arithmetic class, and +the second in this same metric** — the eighth is the `0 of 330 tool calls` +figure corrected immediately above. SH-3 is the number this project keeps +getting wrong, which is itself a reason to doubt the metric rather than the +behaviour (§SH-3 disposition). + +**The fix**, `cb-cost.py` `sh3_line()`: below a minimum sample the tool +prints `insufficient sample` and no verdict. The floor is derived, not +round — if the true rate were exactly the 20% target, the chance of seeing +zero batched turns in `n` responses is `0.8^n`; at **n = 14** that is 4.4%, +so "0 batched in 14" rules out a target-meeting rate at ~95%. Below that +the tool has nothing to say and now says so. + At an 8.6% batching rate there is real headroom, but the saving is bounded: eliminating every avoidable single-call turn would remove at most a few percent of turns, worth roughly $2–4 on a $93 pass. **Batch because it is diff --git a/tools/cb-cost.py b/tools/cb-cost.py index ae6228b..dc34d0b 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -624,8 +624,15 @@ def render(rep, by_task=False, composition=False): f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]") print(f"{indent}SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok " f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]") - print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " - f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") + # Same guard as the budget path: a rate a window cannot support + # must not be printed as though it were measured. + if sh["responses_with_tools"] < SH3_MIN_SAMPLE: + print(f"{indent}SH-3 batching rate insufficient sample " + f"[{sh['responses_with_tools']} with tool calls, " + f"need {SH3_MIN_SAMPLE}]") + else: + print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " + f"[diagnostic, no floor — ADR-0008 D1]") print(f"{indent} {sh['tool_calls']} tool calls in " f"{sh['responses_with_tools']} responses; " f"{sh['calls_in_batched_turns']} in batched turns") @@ -786,6 +793,33 @@ def self_test(): finally: os.unlink(empty) + # SH-3 minimum sample (CB-WP-0013 T01). Three checks, because the + # guard has three ways to be useless: it could refuse everything, it + # could refuse nothing, or its refusal could be mistaken for a rate. + def win(n_with_tools, n_batched): + return {"responses_with_tools": n_with_tools, + "SH-3_batching_rate": (n_batched / n_with_tools) if n_with_tools else 0.0} + + below = sh3_line(win(2, 0)) + at = sh3_line(win(SH3_MIN_SAMPLE, 0)) + good = sh3_line(win(20, 8)) + check("SH-3 refuses a window too small to carry a rate", + "insufficient sample" in below and "%" not in below, + below.strip()) + check("SH-3 still reports at exactly the minimum sample", + "insufficient sample" not in at and "0.0%" in at, at.strip()) + check("SH-3 reports a real rate, with no verdict attached", + "40.0%" in good and "no floor" in good and "FAIL" not in good, + good.strip()) + # ADR-0008 D1: the gate is gone. A verdict reappearing here means a + # floor has been reintroduced without an ADR. + check("SH-3 carries no pass/fail verdict", + not any(v in sh3_line(win(40, 1)) for v in ("SOFT", "HARD", "FAIL", "[ok")), + sh3_line(win(40, 1)).strip()) + # The one that matters: a refusal must not read as a measured zero. + check("a refusal is distinguishable from a genuine 0.0%", + below != sh3_line(win(SH3_MIN_SAMPLE, 0))) + # AC-7: the subagent tree is discovered by the path globs. slug = "-home-worsch-clay-borg" paths = transcript_paths(slug) @@ -809,6 +843,47 @@ def self_test(): SHAPE_SOFT = {"SH-1": 200_000, "SH-2": 300_000} SHAPE_HARD = {"SH-1": 300_000, "SH-2": 450_000} +# CB-WP-0013 T01. SH-1 and SH-2 are location statistics and survive a +# two-response window. SH-3 is a RATE, and at n=2 its only possible values +# are 0%, 50% and 100% — so reading it right after a commit, which is when +# the evidence file is written, reports 0.0% almost regardless of +# behaviour. Six evidence files quoted that 0.0% as a measurement; the real +# per-pass figure was 1.1-6.3%. +# +# The floor is derived, not round. If the true rate were exactly the 20% +# target, the chance of observing ZERO batched turns in n responses is +# 0.8^n. At n = 14 that is 4.4%, so "0 batched in 14" rules out a +# target-meeting rate at ~95%. Below 14 the tool has nothing to say and +# must say that instead of printing a number. +SH3_MIN_SAMPLE = 14 + + +def sh3_line(win, min_sample=SH3_MIN_SAMPLE): + """SH-3 as a DIAGNOSTIC — no floor, no verdict (ADR-0008 D1). + + Retired as a gate because the window and the metric are systematically + anti-correlated: batching needs two tool calls whose inputs are known + at once, which is orientation work, and this window is *since the last + commit*, which during a pass is implementation work. Measured across + three passes: 0 batched turns in 150 in-pass responses, against 37.5% + in the gap between two of them. A floor the window structurally + excludes is not a target. + + The minimum-sample refusal is kept. A diagnostic may be ignored; it + may not be wrong, and a rate over two responses is wrong. + + Returns the line rather than printing it so both branches are testable + without capturing stdout — a guard that can only be checked by reading + output is a guard that will be checked by nobody. + """ + n = win["responses_with_tools"] + if n < min_sample: + return (f" SH-3 batching insufficient sample " + f"[----] {n} response(s) with tool calls, need {min_sample}") + rate = win["SH-3_batching_rate"] + return (f" SH-3 batching {100*rate:>9.1f}% " + f"[diag] no floor — retired as a gate, ADR-0008 D1") + def shape_budget(slug): """SH-* for the window since the last commit, with soft/hard verdicts.""" @@ -842,9 +917,7 @@ def shape_budget(slug): print(f" {key} {label} {val:>10,.0f} tok [{mark}] " f"soft {soft:,} / hard {hard:,}") breach = max(breach, 0 if val <= soft else (1 if val <= hard else 2)) - rate = win["SH-3_batching_rate"] - print(f" SH-3 batching {100*rate:>9.1f}% " - f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%") + print(sh3_line(win)) if breach >= 2: print("\n HARD — compact before continuing. Context this size costs " diff --git a/tools/dep-weight.py b/tools/dep-weight.py index 46a5739..fefd094 100755 --- a/tools/dep-weight.py +++ b/tools/dep-weight.py @@ -28,15 +28,30 @@ import sys from repo import cargo_bin, enter_root PACKAGE = "games-ground" +# ADR-0008 D2. `--edges normal` includes proc-macro crates, which run in +# the compiler and never reach a shipped binary — 89,048 lines, 36.2% of +# what this tool used to call "what a game ships", `syn` alone 66,916. The +# shipped-runtime configuration now excludes them. +# +# AM-4b is deliberately NOT corrected here: its proc-macro share has not +# been measured, and correcting a second instrument on the strength of the +# first one's ratio is the error this change exists to fix. +PROC_MACRO_EXCLUDED = ["--edges", "normal,no-proc-macro"] CONFIGS = { - "shipped-runtime": ["--no-default-features"], + "shipped-runtime": ["--no-default-features"] + PROC_MACRO_EXCLUDED, "dev-toolchain": [], } # AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails # the build: a gate that only reports is a suggestion. +# ADR-0008 D3: the target moves down with the instrument. Leaving it at +# 250,000 against a corrected 157,202 would hand this project 89,048 lines +# of headroom it did not earn, in the same change that revealed the error. +# 161,000 keeps ~2.4% of room where 250,000 kept ~1.5% — the small +# rounding up is the only thing this decision gives back, because a target +# with 1.5% of room fails on a dependency's patch release. TARGETS = { - "shipped-runtime": 250_000, + "shipped-runtime": 161_000, "dev-toolchain": 350_000, } @@ -55,8 +70,8 @@ def crates(extra_args): ) sys.exit(1) out = subprocess.run( - [cargo, "tree", "-p", PACKAGE, "--edges", "normal", "--prefix", "none"] - + extra_args, + [cargo, "tree", "-p", PACKAGE, "--prefix", "none"] + + (extra_args if "--edges" in extra_args else ["--edges", "normal"] + extra_args), capture_output=True, text=True, check=True, @@ -125,6 +140,23 @@ def self_test(): check("cargo resolves without caller PATH setup", bool(cargo_bin()), cargo_bin() or "NOT FOUND") + # ADR-0008 D2. The exclusion must remove exactly the proc-macro crates + # and nothing else — a flag that quietly dropped a runtime dependency + # would shrink the number in the direction this metric can be gamed. + with_pm = crates(["--no-default-features"]) + without_pm = crates(["--no-default-features"] + PROC_MACRO_EXCLUDED) + dropped = set(with_pm) - set(without_pm) + check("the proc-macro exclusion drops exactly the expected crates", + dropped == {"syn", "quote", "proc-macro2", "unicode-ident", "serde_derive"}, + f"dropped {sorted(dropped)}") + check("the exclusion only ever removes crates, never adds", + set(without_pm) <= set(with_pm), + f"{len(with_pm)} -> {len(without_pm)}") + # And it must actually remove something: an exclusion that excluded + # nothing would leave the old figure while claiming the new meaning. + check("the exclusion is not a no-op", + len(dropped) > 0, f"{len(dropped)} crate(s) dropped") + check("targets defined for every configuration", set(TARGETS) == set(CONFIGS) and all( isinstance(v, int) and v > 0 for v in TARGETS.values()), diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md new file mode 100644 index 0000000..dccf1c1 --- /dev/null +++ b/workplans/CB-WP-0013-instrument-corrections.md @@ -0,0 +1,271 @@ +--- +id: CB-WP-0013 +kind: meta +title: "Two instruments that were wrong, and the record they wrote" +status: done +--- + +# Purpose + +``` +structural tier M (changes how the loop constrains its own operation: + a budget instrument and a gate floor — InnerLoop + v1.6, ADR-0006 D5) +chaos d4 = 3 → no override +declared tier M +``` + +Declaration 8 of 12 in the calibration window. Tier M means the survey and +the ADR merge into one document and the adversarial review is optional. + +**Meta budget: 0% of the trailing three**, so this pass is in budget +without the ADR-0006 D2 repair exemption — though it would qualify for it, +since both halves repair an instrument that reports a breach. + +## Why now + +CB-WP-0012 ended owing two corrections, and one of them turns out to be +larger than it was filed as. + +### SH-3 has never read what six evidence files say it read + +CB-EV-0010 §5 states *"SH-3 has now read 0.0% for six consecutive passes"* +and calls it the oldest unargued number in the project. The number is an +artifact. + +`--shape-budget` measures **the window since the last commit**, and it is +read while writing the evidence file — immediately *after* a commit, when +that window holds one or two responses. SH-1 and SH-2 are location +statistics and survive a sample of two. **SH-3 is a rate**, and at n=2 its +only possible values are 0%, 50% and 100%. One window, three metrics, and +it is only wrong for one of them. + +Measured over real per-pass windows instead: + +| window since | responses | SH-3 | +|---|---:|---:| +| `0d2ab22` | 115 | 6.2% | +| `6fb0aea` | 113 | 6.3% | +| `331e7e9` | 95 | 1.1% | +| `c5fa610` | 74 | 1.4% | +| `883b608` | 70 | 1.4% | +| `84d6886` | 24 | 4.3% | + +So the substantive claim survives — SH-3 is badly unmet — but **the figure +quoted in six evidence files was never the measurement**. Worse, the real +numbers say something the artifact hid: the historical rate pinned in +`specs/SessionShape.md` is **7.8–8.6%**, and these windows are **1.1–6.3%**. +Batching has got *worse*, and a metric stuck at a constant 0.0% could not +show a trend at all. + +### AM-4a overstates every figure it has ever reported by 36% + +ADR-0007 Decision 4, filed and unimplemented. AM-4a counts +`--edges normal`, which includes proc-macro crates that run in the compiler +and never reach a binary: **89,048 lines, 36.2%** of the shipped-runtime +figure, `syn` alone 66,916. + +## Task: fix SH-3's instrument, and correct the record + +```task +id: CB-WP-0013-T01 +status: done +priority: high +``` + +A rate needs a sample. `--shape-budget` must **refuse to report SH-3** +below a stated minimum rather than print a number the window cannot +support — `insufficient sample (n responses)`, not `0.0%`. + +**The choice to argue, not assume:** whether SH-3 should instead be +windowed differently from SH-1/SH-2 — over the pass rather than since the +last commit — since it is the only one of the three that a two-response +window destroys. Pick one and say why the other was rejected. + +**Controls.** A minimum-sample rule is exactly the kind of guard that can +be satisfied vacuously: assert that a window *above* the floor still +reports a rate, that one below reports the refusal, and that the refusal +is distinguishable from a genuine 0.0%. Mutate each. + +**Correct the record.** Six evidence files quote 0.0%. They are history and +are not rewritten; the correction is recorded once, where a reader will +find it, and named as what it is — the same class as CB-WP-0002's +"0 of 330 tool calls were batched", which `specs/SessionShape.md` §4 +already records as the eighth trusted-arithmetic instance. **This is the +ninth, and it is in the same metric.** That coincidence is worth a +sentence: SH-3 is the number this project keeps getting wrong. + +**Done 2026-08-02.** `cb-cost.py` gains `sh3_line()`: below a minimum +sample it prints `insufficient sample` and **no verdict**. + +**The window was kept, not split.** SH-3 could instead have been given a +per-pass window while SH-1/SH-2 kept the since-last-commit one. Rejected: +the budget's stated purpose is the *open remainder* since the last commit +— the only thing a live budget can fire on — and giving one of three +metrics a different window makes "the window" ambiguous in a tool that +three specs cite. Refusing to report is honest and keeps one window. + +**The floor is derived, not round.** If the true rate were exactly the 20% +target, the chance of observing zero batched turns in `n` responses is +`0.8^n`. At **n = 14** that is 4.4%, so "0 batched in 14" rules out a +target-meeting rate at ~95%. Below 14 the tool has nothing to say. + +Four controls, three mutations, each red for its stated reason: + +| mutation | result | +|---|---| +| the guard never refuses | `SH-3 refuses a window too small` red, *and* `a refusal is distinguishable from a genuine 0.0%` red | +| the guard always refuses | reporting at exactly the minimum, and a real 40% rate, both red | +| the refusal prints as `0.0%` | `refuses a window too small` red — the case the six evidence files actually hit | + +The correction is recorded in `specs/SessionShape.md` §4, beside the +eighth instance, with the real per-pass figures. + +## Task: argue SH-3's floor, or move it + +```task +id: CB-WP-0013-T02 +status: done +priority: high +``` + +The declaration CB-EV-0010 said was owed. `specs/SessionShape.md` is +explicit that the 20% floor was set above the measured rate deliberately +and that *"whether 20% is reachable or arbitrary is unknown until a pass +tries"*. Seven passes have now reported it and none has tried. + +Decide, with the real numbers rather than the artifact: + +- is 20% reachable, and at what cost to how the work is actually done; +- is the measured decline from 7.8–8.6% to 1.1–6.3% a behavioural change + worth reversing, or an artifact of tier-L passes doing more sequential + reading than tier-S ones; +- **and the honest option: retire SH-3.** `specs/SessionShape.md` §4 puts + the ceiling at **$2–4 on a $93 pass** and says *"batch because it is + free, not because it is the lever."* A floor that is 3–20× the measured + rate, cannot be moved by the party measured, and is worth single-digit + dollars is a candidate for `retire_if`, not a standing breach. + +**Do not retarget to match the measurement.** `specs/SessionShape.md` +already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If +the floor moves, it moves on an argument about what batching is worth, and +the argument is written down before the number changes. + +**Done 2026-08-02.** [ADR-0008](../decisions/ADR-0008-instrument-corrections.md) +D1 — **SH-3 is retired as a gate and kept as a diagnostic.** + +Investigating it turned up a third defect, deeper than the two this +workplan was declared on. Measured by re-deriving batching from the raw +transcripts, independently of `cb-cost`: + +| window | with tools | batched | rate | +|---|---:|---:|---:| +| CB-WP-0011 pass | 54 | 0 | **0.0%** | +| gap → next declaration | 16 | 6 | **37.5%** | +| CB-WP-0012 pass | 86 | 0 | **0.0%** | +| gap → next declaration | 10 | 1 | **10.0%** | +| CB-WP-0013 so far | 10 | 0 | **0.0%** | + +**Zero batched turns in 150 in-pass responses; 37.5% in one gap.** +Batching needs two calls whose inputs are known at once — orientation +work. Implementation consumes each step's result before the next. And +SH-3's window is *since the last commit*, which during a pass is always +implementation. **The metric could not read above ~0% in the window it was +gated on.** + +**And this workplan's own Purpose was wrong.** It stated batching *"has +got worse"* (7.8–8.6% historical against 1.1–6.3% per-pass). Those come +from differently-placed windows, not different behaviour. Withdrawn — the +same class of error, in the pass written to correct it. + +The retirement is *not* retargeting to match the measurement: the floor +was not moved to 6%, the gate was removed on an argument about what the +quantity is worth, established before the disposition. The number is still +reported; only the verdict is gone. + +## Task: correct AM-4a to count what ships + +```task +id: CB-WP-0013-T03 +status: done +priority: high +``` + +ADR-0007 Decision 4. `tools/dep-weight.py` counts +`--edges normal,no-proc-macro`. + +**The targets do not move with the instrument.** Correcting a metric that +reports 246,250 down to 157,202 while leaving the target at 250,000 hands +this project 89,048 lines of headroom it did not earn, in the same commit +that reveals the error. Either the target moves down by the same amount +the instrument does, or the pass states plainly why the extra headroom is +legitimate. It is not obviously legitimate; argue it or give it back. + +**Positive control.** `dep-weight.py` already has the guard that an +unlocatable crate must not silently count as zero. The new flag must not +be able to drop crates it should have counted: assert that the proc-macro +set is exactly the five expected crates and that removing the flag +restores the old figure. + +**Done 2026-08-02.** ADR-0008 D2/D3. + +``` +shipped-runtime 18 crates 157,202 lines [ok target 161,000] +``` + +**The target moved down with the instrument**, 250,000 → 161,000, so the +correction hands back essentially nothing: headroom goes from 3,750 to +3,798. 161,000 keeps ~2.4% of room where 250,000 kept ~1.5%; that small +rounding up is the only thing given back, because a target with 1.5% of +room fails on a dependency's patch release. + +Three controls, all green: the exclusion drops **exactly** the five +expected crates, only ever removes and never adds, and is not a no-op. + +**The DFD gate then caught the follow-on**, which is what it is for: three +historical documents carried live `fact:am4a_loc` tags for a number that +had changed. They are history and were not rewritten — the tags were +removed and both documents carry a supersession banner instead. + +**AM-4b is deliberately not corrected.** Its proc-macro share is +unmeasured, and correcting a second instrument on the strength of the +first one's ratio is the error this task exists to fix. Filed as owed. + +## Task: evidence + +```task +id: CB-WP-0013-T04 +status: done +priority: medium +``` + +`evidence/CB-EV-0011-instrument-corrections.md`. + +Carry: whether tier M's merged survey-and-ADR was the right weight for two +instrument corrections; the cost of a meta pass immediately after two +product passes; and the standing question of whether **three** consecutive +evidence files reporting their own pass's cost low means the loop should +simply stop quoting it. + +Also carry the one that is now overdue: **`cb-play` is three modes in one +binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider +the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass +did not act. Either do it or state why not, and stop carrying it. + +**Done 2026-08-02.** +[CB-EV-0011](../evidence/CB-EV-0011-instrument-corrections.md). + +- **The self-quoting rule is proposed.** Three passes have now reported + their own cost mid-flight and all three were low by ~2/3 — and this + time it changed a conclusion: CB-EV-0010's tier comparison said L cost + *24% more* than S; the final figures say **51%**. +- **`cb-play`'s third mode never arrived.** `--serve` is a *play flag* + selecting which `Policy` fills a human seat. The binary still has two + modes. CB-EV-0010 §6 was wrong; the item is **closed, not deferred**. +- **A near-miss recorded**: an instrument reading that looked impossible + turned out to be correct, and one independent re-derivation before + writing anything stopped a false accusation. +- **Tier M was the right weight** — one merged document, no separate + review, and the pass still found a defect deeper than the one it was + declared on, because the finding came from measuring rather than from + an extra review round.