diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 3920995..518fb91 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -95,7 +95,3 @@ | task | CB-WP-0012-T03 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T04 | done | — | workplans/CB-WP-0012-render-port.md | | task | CB-WP-0012-T05 | done | — | workplans/CB-WP-0012-render-port.md | -| task | CB-WP-0013-T01 | done | — | workplans/CB-WP-0013-instrument-corrections.md | -| task | CB-WP-0013-T02 | done | — | workplans/CB-WP-0013-instrument-corrections.md | -| task | CB-WP-0013-T03 | done | — | workplans/CB-WP-0013-instrument-corrections.md | -| task | CB-WP-0013-T04 | done | — | workplans/CB-WP-0013-instrument-corrections.md | diff --git a/decisions/ADR-0004-am4-ratification.md b/decisions/ADR-0004-am4-ratification.md index 86cd83a..fde8be8 100644 --- a/decisions/ADR-0004-am4-ratification.md +++ b/decisions/ADR-0004-am4-ratification.md @@ -1,13 +1,5 @@ # ADR-0004: ratify AM-4a and AM-4b -> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a -> figures below were taken with an instrument that counted proc-macro -> crates — 89,048 lines, 36.2% — which run in the compiler and never -> reach a binary. Corrected, the same tree measures **157,202** against -> a target moved to **161,000**. The numbers below are left as the -> record of what was measured then, and are no longer live facts. - - status: accepted date: 2026-07-31 decided by: maintainer (Bernd Worsch), 2026-07-31 @@ -57,7 +49,7 @@ At the time of the retarget, `make dep-weight`: | configuration | crates | third-party LOC | |---|---|---| -| shipped-runtime (`--no-default-features`) | 23 | **246,250** | +| shipped-runtime (`--no-default-features`) | 23 | **246,250** | | dev-toolchain (default features) | 29 | **317,021** | | own source | — | 3,443 | diff --git a/decisions/ADR-0008-instrument-corrections.md b/decisions/ADR-0008-instrument-corrections.md deleted file mode 100644 index 79057f1..0000000 --- a/decisions/ADR-0008-instrument-corrections.md +++ /dev/null @@ -1,158 +0,0 @@ -# ADR-0008: retire SH-3 as a gate; correct AM-4a to count what ships - -status: accepted -date: 2026-08-02 -decided by: agent, under the standing loop authorization. One item is -reserved for the maintainer — see §Reserved. -tier: M (structural M — changes how the loop constrains its own operation: -a gate floor and a budget instrument, InnerLoop v1.6 / ADR-0006 D5; -chaos d4=3 → no override). Tier M merges survey and ADR into one document, -which this is; the adversarial review is optional and was not run. -references: [CB-WP-0013](../workplans/CB-WP-0013-instrument-corrections.md), -[SessionShape.md](../specs/SessionShape.md) §4, -[ADR-0006](ADR-0006-adaptive-control-gates.md) D1/D2/D4, -[ADR-0007](ADR-0007-render-html-not-a-port.md) D4, -[CB-EV-0010](../evidence/CB-EV-0010-render-port.md) §5 - -## Context - -Two instruments were found wrong by the passes that used them. Neither -error was in the direction that flattered us, and one of them has been -producing a standing breach report for seven passes. - ---- - -## Decision 1 — SH-3 is retired as a gate and kept as a diagnostic - -### The survey: three defects, each deeper than the last - -**First (CB-WP-0013 T01): the sample.** `--shape-budget` windows since the -last commit and is read while writing the evidence file — right after a -commit, when the window holds one or two responses. SH-1/SH-2 are location -statistics and survive that. SH-3 is a rate; at n=2 its only values are -0%, 50%, 100%. Six evidence files quoted the resulting `0.0%` as a -measurement. Fixed: below a derived minimum of 14 the tool now prints -`insufficient sample` and no verdict. - -**Second: the claimed decline was also an artifact.** This workplan's own -Purpose stated that batching *"has got worse"* — pinned history 7.8–8.6% -against per-pass windows of 1.1–6.3%. **That was wrong**, and by the same -class of error it was written to correct. The two figures come from -differently-placed windows, not from different behaviour. Withdrawn. - -**Third, and the one that decides this: the window and the metric are -systematically anti-correlated.** Measured, by re-deriving batching -independently of `cb-cost` from the raw transcripts: - -| window | responses with tools | batched | rate | -|---|---:|---:|---:| -| CB-WP-0011 pass | 54 | 0 | **0.0%** | -| gap → next declaration | 16 | 6 | **37.5%** | -| CB-WP-0012 pass | 86 | 0 | **0.0%** | -| gap → next declaration | 10 | 1 | **10.0%** | -| CB-WP-0013 so far | 10 | 0 | **0.0%** | - -**Zero batched turns in 150 in-pass responses across three passes, and -37.5% in the gap between two of them** — comfortably above the 20% floor. - -The cause is not discipline. Batching requires two tool calls whose inputs -are both known at once. Orientation and survey work is full of those: read -four files, run three probes. Implementation has almost none, because each -step consumes the previous step's result — read, edit, test, read the -failure, fix. **So SH-3 measures what kind of work is happening.** - -And SH-3's window is *since the last commit*, which during a pass is -always in-pass work. **The metric is structurally incapable of reading -above ~0% in the window it is gated on.** It is not a strict gate — it can -only be met by not working. - -### The decision - -**SH-3 stops being a gate.** The 20% floor and the `[SOFT]` verdict are -removed. `cb-cost` continues to report the batching rate as a diagnostic, -over the whole session, where it reads 5.8% (57 batched of 984 responses -with tool calls). - -Four reasons, strongest first: - -1. **It cannot report what it gates on.** §above. A floor that the - measurement window structurally excludes is not a target. -2. **It measures composition, not waste.** Penalising an implementation - pass for not batching penalises it for being an implementation pass. -3. **Its own spec bounds its value.** `specs/SessionShape.md` §4: - *"eliminating every avoidable single-call turn would remove at most a - few percent of turns, worth roughly $2–4 on a $93 pass"*, and *"batch - because it is free, not because it is the lever."* It also calls SH-3 - *"the least valuable"* of the three. -4. **Seven passes, seven breach reports, zero actions.** ADR-0006 D4: - rituals cash out or go. - -### This is not retargeting to match the measurement - -`specs/SessionShape.md` forbids that explicitly, and CB-WP-0003 T07 exists -to prevent it, so the distinction has to be stated rather than assumed: - -- **Retargeting** would be moving the floor from 20% to 6% because 6% is - what we measure. That is not done and is still forbidden. -- **This** removes the gate on the argument that the *quantity* does not - deserve one — established before the disposition, from the window - structure and from the spec's own valuation. - -And the obvious objection — *retiring a metric right when it shows a -regression looks like burying it* — is answered by the second defect: the -regression was an artifact of window placement and has been withdrawn. -There is no established decline to bury. The number is still reported; -only the verdict is gone. - -## Decision 2 — AM-4a counts `--edges normal,no-proc-macro` - -ADR-0007 D4, owed. AM-4a's figure includes proc-macro crates that run in -the compiler and never reach a binary: **89,048 lines, 36.2%** of the -shipped-runtime figure, `syn` alone 66,916. - -`tools/dep-weight.py` measures `--edges normal,no-proc-macro` for the -shipped-runtime configuration. - -## Decision 3 — the target moves down with the instrument - -Correcting the metric from 246,250 to 157,202 while leaving the target at -250,000 would hand this project **89,048 lines of headroom it did not -earn**, in the same commit that reveals the error. That is the shape of -change this repo has twice refused, and refusing it here costs nothing: - -> **AM-4a's shipped-runtime target moves from 250,000 to 161,000.** - -161,000 preserves the *ratio* the ratified target expressed — 250,000 was -set against a 246,250 measurement, i.e. ~1.5% of headroom; 161,000 against -157,202 is ~2.4%. Rounding to a round number slightly *above* strict -proportionality is deliberate and is the only place this decision gives -anything back, because a target that leaves 1.5% of room is a target that -fails on a dependency's patch release. - -**AM-4b (dev-toolchain) is left alone**, at 350,000 against a 317,021 -measurement. It is not corrected in this pass because its proc-macro share -has not been measured, and correcting a second instrument on the strength -of the first one's ratio is exactly the error this ADR exists to fix. -Filed as owed. - -## Reserved for the maintainer - -**AM-4a still cannot survive INTENT stage 2**, and this decision makes that -worse rather than better. `wgpu` + `winit` is 1,741,979 marginal lines -against a target now proposed at 161,000 — **10.8×**, up from 7×. ADR-0007 -raised it; nothing here settles it. The correction below is honest -book-keeping, not headroom for stage 2, and it should not be read as -having made the stage-2 problem smaller. - -## Consequences - -- One fewer gate, and the gate registry gets its first retirement. That is - also **`GATE-REVIEW`'s first `caught` entry** — the registry's - `retire_if` says it must have *"retired, tightened, or forced the - re-justification of"* something by its review date, and it now has. -- Every AM-4a figure in this repo's history overstates the load by 36%. - Those documents are history and are not rewritten; the correction lives - in `specs/GameKernel.md` §4 and here. -- The batching diagnostic stays visible, so if in-pass batching ever - becomes possible the number will show it without a gate having to be - re-litigated first. diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md index 55bdc46..217abf1 100644 --- a/evidence/CB-EV-0001-game-kernel.md +++ b/evidence/CB-EV-0001-game-kernel.md @@ -1,13 +1,5 @@ # CB-EV-0001 — GROUND game kernel: acceptance evidence -> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a -> figures below were taken with an instrument that counted proc-macro -> crates — 89,048 lines, 36.2% — which run in the compiler and never -> reach a binary. Corrected, the same tree measures **157,202** against -> a target moved to **161,000**. The numbers below are left as the -> record of what was measured then, and are no longer live facts. - - Status: **T08 complete. AM-4 remediated and re-measured 2026-07-31.** Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per remediation option, and §5 corrects AM-12 from "uncomputable" to @@ -36,7 +28,7 @@ the property is false? A row can be *measured* and still enforce nothing. | AM-1 rule coverage (GR only) | 100% of GR-rules | 58/58 (100%) | **met** | **yes** | | AM-1b link, ground | 58 claimed rules named in the aggregate | 49/58 | **unmet** | reported | | AM-1b link, kernel | 18 K-rules named in source | 15/18 | **unmet** | reported until 2026-08-31 | -| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | +| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | | AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | **yes** | | AM-6 throughput | ≥100,000 events/s | 2,017,009 events/s (`make am6`) | **met, 20.2×** | **yes** — CB-WP-0006 T01 | | AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | **no** — no code computes the ratio | @@ -202,7 +194,7 @@ retargeted onto third-party source under audit. | Configuration | Crates | Third-party LOC | Target | Verdict | |---|---|---|---|---| -| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | +| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | | Dev toolchain (default features) | 29 | 317,021 | ≤350,000 | **met** | | Our own source | — | 3,408 | — | — | diff --git a/evidence/CB-EV-0011-instrument-corrections.md b/evidence/CB-EV-0011-instrument-corrections.md deleted file mode 100644 index 7012016..0000000 --- a/evidence/CB-EV-0011-instrument-corrections.md +++ /dev/null @@ -1,181 +0,0 @@ -# CB-EV-0011 — three instruments, and a rule about quoting yourself - -CB-WP-0013 T04. Measured 2026-08-02 at `d2c3f34`. Pass kind `meta`, -tier **M** (chaos d4=3, no override). Declaration 8 of 12. - -The pass was declared to fix two instruments. It found a third defect in -one of them, and a fourth in itself. - ---- - -## 1. What was actually wrong with SH-3 - -The declaration named one defect. There were three, each deeper. - -**1. The sample.** `--shape-budget` windows since the last commit and is -read while writing the evidence file — right after a commit, when the -window holds one or two responses. SH-1/SH-2 are location statistics and -survive that; SH-3 is a rate, and at n=2 its only values are 0%, 50%, -100%. Six evidence files quoted the resulting `0.0%` as a measurement. - -**2. The claimed decline — which was this pass's own error.** The -declaration stated batching *"has got worse"*, citing 7.8–8.6% pinned -history against 1.1–6.3% per-pass windows. Those are differently-*placed* -windows, not different behaviour. **Withdrawn.** The pass declared to -correct a trusted-arithmetic error opened by committing one. - -**3. The one that decided it.** Re-deriving batching from the raw -transcripts, independently of `cb-cost`: - -| window | with tools | batched | rate | -|---|---:|---:|---:| -| CB-WP-0011 pass | 54 | 0 | **0.0%** | -| gap → next declaration | 16 | 6 | **37.5%** | -| CB-WP-0012 pass | 86 | 0 | **0.0%** | -| gap → next declaration | 10 | 1 | **10.0%** | -| CB-WP-0013 so far | 10 | 0 | **0.0%** | - -**Zero batched turns in 150 in-pass responses. 37.5% in one gap** — above -the floor the gate had been failing for seven passes. - -Batching needs two tool calls whose inputs are known at once. That is -orientation: read four files, run three probes. Implementation has almost -none, because each step consumes the previous step's result. And SH-3's -window is *since the last commit*, which during a pass is implementation. -**The metric could not read above ~0% in the window it was gated on.** - -A gate meetable only by not working is not a target. SH-3 is retired as a -gate (ADR-0008 D1) and kept as a diagnostic over the whole session, where -it reads 5.8%. - -**On the appearance of burying it:** retiring a metric that shows a -regression looks bad, and should. The answer is defect 2 — the regression -was an artifact and has been withdrawn, so there is no established decline -to bury. The number is still printed; only the verdict is gone. And the -floor was *not* moved to 6%, which is the thing `specs/SessionShape.md` -actually forbids. - -## 2. A near-miss worth recording: I almost reported a defect that was not there - -Midway through, a bounded window reported `53 tool calls in 53 responses; -0 in batched turns` — exactly one call per response, which looked like the -counter had collapsed. I had batched repeatedly in this session, so the -reading looked impossible. - -It was correct. Re-deriving from the transcripts independently gave the -same 0, and the 131 batched responses that do exist all fall *outside* -those windows. The instrument was right and my expectation was wrong. - -Recorded because the **false-accusation class** is already on this -project's record, and because the thing that stopped it was cheap: one -independent re-derivation before writing anything down. The cost of -checking was two minutes; the cost of not checking would have been an -evidence file alleging a defect in a tool that did not have one — and this -pass's whole subject is instruments reporting things that are not so. - -## 3. AM-4a, corrected, gives back nothing - -``` -before 23 crates 246,250 lines [ok target 250,000] headroom 3,750 -after 18 crates 157,202 lines [ok target 161,000] headroom 3,798 -``` - -The instrument dropped 89,048 lines of proc-macro code that never reaches -a binary; **the target moved down with it** (ADR-0008 D3). Leaving the -target at 250,000 would have handed this project 89,048 lines of headroom -it did not earn, in the same change that revealed the error. - -Three controls, all green: the exclusion drops **exactly** the five -expected crates, only ever removes and never adds, and is not a no-op. - -**The DFD gate then caught the follow-on**, which is exactly what it is -for: three historical documents carried live `fact:am4a_loc` tags for a -number that had changed. They are history and were not rewritten — the -tags came off and both carry a supersession banner. A gate catching the -second-order consequence of a correction, unprompted, is the best evidence -yet that the registry is doing something. - -**AM-4b is deliberately not corrected.** Its proc-macro share is -unmeasured, and correcting a second instrument on the strength of the -first one's ratio is the error this pass exists to fix. Owed. - -## 4. The rule this pass should leave behind: stop quoting your own cost - -| pass | reported mid-pass | final | understated by | -|---|---|---|---| -| CB-WP-0009 | $6.73 (CB-EV-0008) | $11.31 | 68% | -| CB-WP-0011 | $4.23 / 0.094 (CB-EV-0009) | $7.02 / 0.099 | 66% | -| CB-WP-0012 | $8.82 / 0.123 (CB-EV-0010) | **$14.52 / 0.150** | 65% | - -Three for three, always low, by a strikingly consistent ~2/3. And this -time it **changed a conclusion**, not just a figure: - -> CB-EV-0010 §5 reported the first priced tier comparison as *"0.123 -> $/response at L against 0.099 at S — 24% more."* The final figures are -> **0.150 against 0.099 — 51% more.** The direction holds; the magnitude -> was understated by half. - -Tier L is twice as expensive as that comparison claimed. It still found -errors of 25× and 85× in CB-WP-0012 and deleted a port that would have -been rewritten, so it still looks worth it — but the price of the argument -doubled and the evidence file that made it could not have known. - -**Proposed rule, for the next pass to adopt or reject:** an evidence file -quotes the *previous* pass's final cost and does not quote its own. Its -own is knowable only after the commit that closes it, so any figure it -prints is a floor, and this project has now printed three of them and let -one carry a conclusion. - -## 5. `cb-play`'s third mode never arrived - -CB-EV-0009 §5 said a third mode would be the second use at which -`cb-play`'s single-binary shape should be reconsidered. CB-EV-0010 §6 -recorded that it had arrived, with `--serve`, and that the pass had not -acted. **CB-EV-0010 was wrong.** - -`--serve` is registered in `parse_args` as a *play flag*: it selects which -`Policy` fills a human seat, terminal or browser. The binary still has -exactly two modes — play a game, or read a recording — and they still take -disjoint flags. The trigger did not fire. - -Carried for two passes, resolved here, and **stopped**: there is no split -to do, and the standing item is closed rather than deferred again. - -## 6. Cost and shape - -| pass | kind | responses | cost | $/response | -|---|---|---|---|---| -| CB-WP-0011 | product | 71 | $7.02 | 0.099 | -| CB-WP-0012 | product | 97 | $14.52 | 0.150 | -| CB-WP-0013 | meta | 34+ | $5.78+ | provisional, see §4 | - -**Meta budget: 21%**, inside the soft 25%, without needing ADR-0006 D2's -repair exemption — though both halves would have qualified for it. - -**SH-1 290,475 `[SOFT]`**, drifting toward the hard line; the remedy is a -compaction and it is free. **SH-3 now reads `insufficient sample (1 -response with tool calls, need 14)`** — the fix working in production on -its first real reading, where the old instrument would have printed a -confident `0.0%`. - -**`gate-review`: 0 due, 0 silent, 0 drifted.** Every gate in the registry -now has a `caught` entry, for the first time. GATE-REVIEW earned its own -by forcing SH-3's re-justification — asking what a gate had ever caught is -what exposed that it could not read what it gated on. - -## 7. Open - -- **AM-4b's proc-macro share is unmeasured.** §3. -- **The self-quoting rule in §4 is proposed, not adopted.** -- **AM-4a still cannot survive INTENT stage 2**, and this pass made the - ratio worse: `wgpu` + `winit` is 1,741,979 against a target now 161,000 - — **10.8×**, up from 7×. Reserved for the maintainer, twice over now. -- **ADR-0007 Decision 3's acquisition rule** remains an argument awaiting - ratification. -- **INTENT stage 1 stays open**; the emitted JavaScript has still never - been executed. -- **Tier M's merged survey-and-ADR was the right weight.** One document, - no separate review, and the pass still found a defect deeper than the - one it was declared on — because the finding came from *measuring*, not - from an extra review round. That is one data point for the tier table - and against the assumption that depth comes from process weight. diff --git a/facts.toml b/facts.toml index 113778d..06a215c 100644 --- a/facts.toml +++ b/facts.toml @@ -10,14 +10,14 @@ generated = "2026-08-02" pin = "fc76445" [am4a_loc] -value = 157202 -text = "157,202" +value = 246250 +text = "246,250" fmt = "{:,}" by = "tools/dep-weight.py" [am4a_target] -value = 161000 -text = "161,000" +value = 250000 +text = "250,000" fmt = "{:,}" by = "tools/dep-weight.py TARGETS" diff --git a/gates.toml b/gates.toml index 17b3a95..bcb3fd3 100644 --- a/gates.toml +++ b/gates.toml @@ -36,15 +36,14 @@ caught = [ retire_if = "two consecutive passes never approach the soft line, or commits get small enough that the window is always trivial" [[gate]] -id = "SH-1/SH-2" +id = "SH-1/SH-2/SH-3" name = "session-shape budget" target = "shape-budget" -checks = "mean and p90 context since the last commit (SH-3 retired as a gate, ADR-0008 D1 — still reported as a diagnostic)" +checks = "mean and p90 context, and batching rate, since the last commit" added = "2026-08-01" review_by = "2026-11-30" caught = [ "first run fired HARD at 656,574 against a 300,000 ceiling, which is what prompted the compaction before CB-WP-0008", - "CB-WP-0013: SH-3 retired from this gate. Its window is *since the last commit*, which during a pass is implementation work, and batching needs two calls whose inputs are known at once — which is orientation work. Measured: 0 batched turns in 150 in-pass responses across three passes, against 37.5% in the gap between two of them. A floor the window structurally excludes is not a target (ADR-0008 D1)", ] retire_if = "context stops correlating with cost, or the model's context handling makes the number unactionable" @@ -134,7 +133,5 @@ target = "gate-review" checks = "gates past their review date, and gates that have caught nothing" added = "2026-08-01" review_by = "2026-12-31" -caught = [ - "CB-WP-0013: forced SH-3's re-justification and then its retirement. `make gate-review` had reported SH-3 as a standing breach for seven passes with zero actions taken, which is this registry's own ritual test (ADR-0006 D4); asking what it had ever caught is what exposed that the metric could not read above ~0% in the window it was gated on (ADR-0008 D1)", -] +caught = [] retire_if = "it has retired, tightened, or forced the re-justification of nothing by its review date — then it is a ritual, and ADR-0006 D4 says rituals cash out or go" diff --git a/specs/GameKernel.md b/specs/GameKernel.md index 3bd23f8..6bb04db 100644 --- a/specs/GameKernel.md +++ b/specs/GameKernel.md @@ -204,8 +204,8 @@ evidence lands in `evidence/CB-EV-0001-game-kernel.md` with no | AM-1 | M-D1-COV: GR-rules covered by ≥1 passing scenario | no candidate has any (observation) | **100%** of GR + U rules | measured by runner report | | AM-2 | M-D1-SPL: spec lines per rule in `games/ground` rules code (impl LOC ÷ rule count) | boardgame.io ~36 LOC for the 2-move synthetic game | ≤ 40 LOC/rule, paired with AM-1 (anti-gaming pair) | measured (tokei + rule count) | | AM-3 | Synthetic-workload definition size: LOC to express the CB-RES-0001 synthetic game on our kernel | ~36 LOC (boardgame.io, measured) | ≤ 50 LOC | measured | -| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features --edges normal,no-proc-macro`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 161,000 lines** (ADR-0008 D3, was 250,000) | measured (`make dep-weight`) | -| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** — *proc-macro share unmeasured; correction owed (ADR-0008 D2)* | measured (`make dep-weight`) | +| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 250,000 lines** | measured (`make dep-weight`) | +| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** | measured (`make dep-weight`) | | ~~AM-4c~~ | M-D2-DEP: own source per third-party 100k lines | — | **WITHDRAWN from the acceptance table 2026-08-01 (CB-WP-0006 T04)** — retained as a reported diagnostic in `make dep-weight`; see §5a | diagnostic | | AM-5 | M-D2-BLD: clean release build of headless workspace | n/a (npm install ~seconds; not comparable) | ≤ 60 s on bnt-lap001, recorded not gated | measured | | AM-6 | M-D3-THR: applied events/s, synthetic workload, same machine | boardgame.io ~1,100–1,900 moves/s (best config, degrading) | **≥ 100,000/s** (stipulated target, ADR-0002) | measured | diff --git a/specs/SessionShape.md b/specs/SessionShape.md index 6b9a521..44e4d38 100644 --- a/specs/SessionShape.md +++ b/specs/SessionShape.md @@ -136,47 +136,6 @@ workplan — the failure mode InnerLoop v1.1 rule 13 exists for, and the `history/260731-inner-loop-rule-audit.md` identified as having no executable defence. -### Correction (CB-WP-0013 T01): SH-3 was never read as measured - -**Six evidence files quote `SH-3 0.0%`, and none of them measured it.** - -`cb-cost --shape-budget` reports the window **since the last commit**, and -it is read while writing the evidence file — immediately *after* a commit, -when that window holds one or two responses. SH-1 and SH-2 are location -statistics and survive a sample of two. SH-3 is a **rate**, and at n=2 its -only possible values are 0%, 50% and 100%. One window, three metrics, and -it is only wrong for one of them. - -Measured over real per-pass windows instead: - -| window since | responses | SH-3 | -|---|---:|---:| -| `0d2ab22` | 115 | 6.2% | -| `6fb0aea` | 113 | 6.3% | -| `331e7e9` | 95 | 1.1% | -| `c5fa610` | 74 | 1.4% | -| `883b608` | 70 | 1.4% | -| `84d6886` | 24 | 4.3% | - -The substantive claim survives — SH-3 is badly unmet — but the constant -0.0% hid something the real numbers show: against the **7.8–8.6%** pinned -above, **batching has got worse**, and a metric frozen at a constant could -not have shown a trend at all. Six passes reported a breach and the breach -was moving underneath them. - -**This is the ninth recorded instance of the trusted-arithmetic class, and -the second in this same metric** — the eighth is the `0 of 330 tool calls` -figure corrected immediately above. SH-3 is the number this project keeps -getting wrong, which is itself a reason to doubt the metric rather than the -behaviour (§SH-3 disposition). - -**The fix**, `cb-cost.py` `sh3_line()`: below a minimum sample the tool -prints `insufficient sample` and no verdict. The floor is derived, not -round — if the true rate were exactly the 20% target, the chance of seeing -zero batched turns in `n` responses is `0.8^n`; at **n = 14** that is 4.4%, -so "0 batched in 14" rules out a target-meeting rate at ~95%. Below that -the tool has nothing to say and now says so. - At an 8.6% batching rate there is real headroom, but the saving is bounded: eliminating every avoidable single-call turn would remove at most a few percent of turns, worth roughly $2–4 on a $93 pass. **Batch because it is diff --git a/tools/cb-cost.py b/tools/cb-cost.py index dc34d0b..ae6228b 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -624,15 +624,8 @@ def render(rep, by_task=False, composition=False): f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]") print(f"{indent}SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok " f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]") - # Same guard as the budget path: a rate a window cannot support - # must not be printed as though it were measured. - if sh["responses_with_tools"] < SH3_MIN_SAMPLE: - print(f"{indent}SH-3 batching rate insufficient sample " - f"[{sh['responses_with_tools']} with tool calls, " - f"need {SH3_MIN_SAMPLE}]") - else: - print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " - f"[diagnostic, no floor — ADR-0008 D1]") + print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " + f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") print(f"{indent} {sh['tool_calls']} tool calls in " f"{sh['responses_with_tools']} responses; " f"{sh['calls_in_batched_turns']} in batched turns") @@ -793,33 +786,6 @@ def self_test(): finally: os.unlink(empty) - # SH-3 minimum sample (CB-WP-0013 T01). Three checks, because the - # guard has three ways to be useless: it could refuse everything, it - # could refuse nothing, or its refusal could be mistaken for a rate. - def win(n_with_tools, n_batched): - return {"responses_with_tools": n_with_tools, - "SH-3_batching_rate": (n_batched / n_with_tools) if n_with_tools else 0.0} - - below = sh3_line(win(2, 0)) - at = sh3_line(win(SH3_MIN_SAMPLE, 0)) - good = sh3_line(win(20, 8)) - check("SH-3 refuses a window too small to carry a rate", - "insufficient sample" in below and "%" not in below, - below.strip()) - check("SH-3 still reports at exactly the minimum sample", - "insufficient sample" not in at and "0.0%" in at, at.strip()) - check("SH-3 reports a real rate, with no verdict attached", - "40.0%" in good and "no floor" in good and "FAIL" not in good, - good.strip()) - # ADR-0008 D1: the gate is gone. A verdict reappearing here means a - # floor has been reintroduced without an ADR. - check("SH-3 carries no pass/fail verdict", - not any(v in sh3_line(win(40, 1)) for v in ("SOFT", "HARD", "FAIL", "[ok")), - sh3_line(win(40, 1)).strip()) - # The one that matters: a refusal must not read as a measured zero. - check("a refusal is distinguishable from a genuine 0.0%", - below != sh3_line(win(SH3_MIN_SAMPLE, 0))) - # AC-7: the subagent tree is discovered by the path globs. slug = "-home-worsch-clay-borg" paths = transcript_paths(slug) @@ -843,47 +809,6 @@ def self_test(): SHAPE_SOFT = {"SH-1": 200_000, "SH-2": 300_000} SHAPE_HARD = {"SH-1": 300_000, "SH-2": 450_000} -# CB-WP-0013 T01. SH-1 and SH-2 are location statistics and survive a -# two-response window. SH-3 is a RATE, and at n=2 its only possible values -# are 0%, 50% and 100% — so reading it right after a commit, which is when -# the evidence file is written, reports 0.0% almost regardless of -# behaviour. Six evidence files quoted that 0.0% as a measurement; the real -# per-pass figure was 1.1-6.3%. -# -# The floor is derived, not round. If the true rate were exactly the 20% -# target, the chance of observing ZERO batched turns in n responses is -# 0.8^n. At n = 14 that is 4.4%, so "0 batched in 14" rules out a -# target-meeting rate at ~95%. Below 14 the tool has nothing to say and -# must say that instead of printing a number. -SH3_MIN_SAMPLE = 14 - - -def sh3_line(win, min_sample=SH3_MIN_SAMPLE): - """SH-3 as a DIAGNOSTIC — no floor, no verdict (ADR-0008 D1). - - Retired as a gate because the window and the metric are systematically - anti-correlated: batching needs two tool calls whose inputs are known - at once, which is orientation work, and this window is *since the last - commit*, which during a pass is implementation work. Measured across - three passes: 0 batched turns in 150 in-pass responses, against 37.5% - in the gap between two of them. A floor the window structurally - excludes is not a target. - - The minimum-sample refusal is kept. A diagnostic may be ignored; it - may not be wrong, and a rate over two responses is wrong. - - Returns the line rather than printing it so both branches are testable - without capturing stdout — a guard that can only be checked by reading - output is a guard that will be checked by nobody. - """ - n = win["responses_with_tools"] - if n < min_sample: - return (f" SH-3 batching insufficient sample " - f"[----] {n} response(s) with tool calls, need {min_sample}") - rate = win["SH-3_batching_rate"] - return (f" SH-3 batching {100*rate:>9.1f}% " - f"[diag] no floor — retired as a gate, ADR-0008 D1") - def shape_budget(slug): """SH-* for the window since the last commit, with soft/hard verdicts.""" @@ -917,7 +842,9 @@ def shape_budget(slug): print(f" {key} {label} {val:>10,.0f} tok [{mark}] " f"soft {soft:,} / hard {hard:,}") breach = max(breach, 0 if val <= soft else (1 if val <= hard else 2)) - print(sh3_line(win)) + rate = win["SH-3_batching_rate"] + print(f" SH-3 batching {100*rate:>9.1f}% " + f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%") if breach >= 2: print("\n HARD — compact before continuing. Context this size costs " diff --git a/tools/dep-weight.py b/tools/dep-weight.py index fefd094..46a5739 100755 --- a/tools/dep-weight.py +++ b/tools/dep-weight.py @@ -28,30 +28,15 @@ import sys from repo import cargo_bin, enter_root PACKAGE = "games-ground" -# ADR-0008 D2. `--edges normal` includes proc-macro crates, which run in -# the compiler and never reach a shipped binary — 89,048 lines, 36.2% of -# what this tool used to call "what a game ships", `syn` alone 66,916. The -# shipped-runtime configuration now excludes them. -# -# AM-4b is deliberately NOT corrected here: its proc-macro share has not -# been measured, and correcting a second instrument on the strength of the -# first one's ratio is the error this change exists to fix. -PROC_MACRO_EXCLUDED = ["--edges", "normal,no-proc-macro"] CONFIGS = { - "shipped-runtime": ["--no-default-features"] + PROC_MACRO_EXCLUDED, + "shipped-runtime": ["--no-default-features"], "dev-toolchain": [], } # AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails # the build: a gate that only reports is a suggestion. -# ADR-0008 D3: the target moves down with the instrument. Leaving it at -# 250,000 against a corrected 157,202 would hand this project 89,048 lines -# of headroom it did not earn, in the same change that revealed the error. -# 161,000 keeps ~2.4% of room where 250,000 kept ~1.5% — the small -# rounding up is the only thing this decision gives back, because a target -# with 1.5% of room fails on a dependency's patch release. TARGETS = { - "shipped-runtime": 161_000, + "shipped-runtime": 250_000, "dev-toolchain": 350_000, } @@ -70,8 +55,8 @@ def crates(extra_args): ) sys.exit(1) out = subprocess.run( - [cargo, "tree", "-p", PACKAGE, "--prefix", "none"] - + (extra_args if "--edges" in extra_args else ["--edges", "normal"] + extra_args), + [cargo, "tree", "-p", PACKAGE, "--edges", "normal", "--prefix", "none"] + + extra_args, capture_output=True, text=True, check=True, @@ -140,23 +125,6 @@ def self_test(): check("cargo resolves without caller PATH setup", bool(cargo_bin()), cargo_bin() or "NOT FOUND") - # ADR-0008 D2. The exclusion must remove exactly the proc-macro crates - # and nothing else — a flag that quietly dropped a runtime dependency - # would shrink the number in the direction this metric can be gamed. - with_pm = crates(["--no-default-features"]) - without_pm = crates(["--no-default-features"] + PROC_MACRO_EXCLUDED) - dropped = set(with_pm) - set(without_pm) - check("the proc-macro exclusion drops exactly the expected crates", - dropped == {"syn", "quote", "proc-macro2", "unicode-ident", "serde_derive"}, - f"dropped {sorted(dropped)}") - check("the exclusion only ever removes crates, never adds", - set(without_pm) <= set(with_pm), - f"{len(with_pm)} -> {len(without_pm)}") - # And it must actually remove something: an exclusion that excluded - # nothing would leave the old figure while claiming the new meaning. - check("the exclusion is not a no-op", - len(dropped) > 0, f"{len(dropped)} crate(s) dropped") - check("targets defined for every configuration", set(TARGETS) == set(CONFIGS) and all( isinstance(v, int) and v > 0 for v in TARGETS.values()), diff --git a/workplans/CB-WP-0013-instrument-corrections.md b/workplans/CB-WP-0013-instrument-corrections.md deleted file mode 100644 index dccf1c1..0000000 --- a/workplans/CB-WP-0013-instrument-corrections.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -id: CB-WP-0013 -kind: meta -title: "Two instruments that were wrong, and the record they wrote" -status: done ---- - -# Purpose - -``` -structural tier M (changes how the loop constrains its own operation: - a budget instrument and a gate floor — InnerLoop - v1.6, ADR-0006 D5) -chaos d4 = 3 → no override -declared tier M -``` - -Declaration 8 of 12 in the calibration window. Tier M means the survey and -the ADR merge into one document and the adversarial review is optional. - -**Meta budget: 0% of the trailing three**, so this pass is in budget -without the ADR-0006 D2 repair exemption — though it would qualify for it, -since both halves repair an instrument that reports a breach. - -## Why now - -CB-WP-0012 ended owing two corrections, and one of them turns out to be -larger than it was filed as. - -### SH-3 has never read what six evidence files say it read - -CB-EV-0010 §5 states *"SH-3 has now read 0.0% for six consecutive passes"* -and calls it the oldest unargued number in the project. The number is an -artifact. - -`--shape-budget` measures **the window since the last commit**, and it is -read while writing the evidence file — immediately *after* a commit, when -that window holds one or two responses. SH-1 and SH-2 are location -statistics and survive a sample of two. **SH-3 is a rate**, and at n=2 its -only possible values are 0%, 50% and 100%. One window, three metrics, and -it is only wrong for one of them. - -Measured over real per-pass windows instead: - -| window since | responses | SH-3 | -|---|---:|---:| -| `0d2ab22` | 115 | 6.2% | -| `6fb0aea` | 113 | 6.3% | -| `331e7e9` | 95 | 1.1% | -| `c5fa610` | 74 | 1.4% | -| `883b608` | 70 | 1.4% | -| `84d6886` | 24 | 4.3% | - -So the substantive claim survives — SH-3 is badly unmet — but **the figure -quoted in six evidence files was never the measurement**. Worse, the real -numbers say something the artifact hid: the historical rate pinned in -`specs/SessionShape.md` is **7.8–8.6%**, and these windows are **1.1–6.3%**. -Batching has got *worse*, and a metric stuck at a constant 0.0% could not -show a trend at all. - -### AM-4a overstates every figure it has ever reported by 36% - -ADR-0007 Decision 4, filed and unimplemented. AM-4a counts -`--edges normal`, which includes proc-macro crates that run in the compiler -and never reach a binary: **89,048 lines, 36.2%** of the shipped-runtime -figure, `syn` alone 66,916. - -## Task: fix SH-3's instrument, and correct the record - -```task -id: CB-WP-0013-T01 -status: done -priority: high -``` - -A rate needs a sample. `--shape-budget` must **refuse to report SH-3** -below a stated minimum rather than print a number the window cannot -support — `insufficient sample (n responses)`, not `0.0%`. - -**The choice to argue, not assume:** whether SH-3 should instead be -windowed differently from SH-1/SH-2 — over the pass rather than since the -last commit — since it is the only one of the three that a two-response -window destroys. Pick one and say why the other was rejected. - -**Controls.** A minimum-sample rule is exactly the kind of guard that can -be satisfied vacuously: assert that a window *above* the floor still -reports a rate, that one below reports the refusal, and that the refusal -is distinguishable from a genuine 0.0%. Mutate each. - -**Correct the record.** Six evidence files quote 0.0%. They are history and -are not rewritten; the correction is recorded once, where a reader will -find it, and named as what it is — the same class as CB-WP-0002's -"0 of 330 tool calls were batched", which `specs/SessionShape.md` §4 -already records as the eighth trusted-arithmetic instance. **This is the -ninth, and it is in the same metric.** That coincidence is worth a -sentence: SH-3 is the number this project keeps getting wrong. - -**Done 2026-08-02.** `cb-cost.py` gains `sh3_line()`: below a minimum -sample it prints `insufficient sample` and **no verdict**. - -**The window was kept, not split.** SH-3 could instead have been given a -per-pass window while SH-1/SH-2 kept the since-last-commit one. Rejected: -the budget's stated purpose is the *open remainder* since the last commit -— the only thing a live budget can fire on — and giving one of three -metrics a different window makes "the window" ambiguous in a tool that -three specs cite. Refusing to report is honest and keeps one window. - -**The floor is derived, not round.** If the true rate were exactly the 20% -target, the chance of observing zero batched turns in `n` responses is -`0.8^n`. At **n = 14** that is 4.4%, so "0 batched in 14" rules out a -target-meeting rate at ~95%. Below 14 the tool has nothing to say. - -Four controls, three mutations, each red for its stated reason: - -| mutation | result | -|---|---| -| the guard never refuses | `SH-3 refuses a window too small` red, *and* `a refusal is distinguishable from a genuine 0.0%` red | -| the guard always refuses | reporting at exactly the minimum, and a real 40% rate, both red | -| the refusal prints as `0.0%` | `refuses a window too small` red — the case the six evidence files actually hit | - -The correction is recorded in `specs/SessionShape.md` §4, beside the -eighth instance, with the real per-pass figures. - -## Task: argue SH-3's floor, or move it - -```task -id: CB-WP-0013-T02 -status: done -priority: high -``` - -The declaration CB-EV-0010 said was owed. `specs/SessionShape.md` is -explicit that the 20% floor was set above the measured rate deliberately -and that *"whether 20% is reachable or arbitrary is unknown until a pass -tries"*. Seven passes have now reported it and none has tried. - -Decide, with the real numbers rather than the artifact: - -- is 20% reachable, and at what cost to how the work is actually done; -- is the measured decline from 7.8–8.6% to 1.1–6.3% a behavioural change - worth reversing, or an artifact of tier-L passes doing more sequential - reading than tier-S ones; -- **and the honest option: retire SH-3.** `specs/SessionShape.md` §4 puts - the ceiling at **$2–4 on a $93 pass** and says *"batch because it is - free, not because it is the lever."* A floor that is 3–20× the measured - rate, cannot be moved by the party measured, and is worth single-digit - dollars is a candidate for `retire_if`, not a standing breach. - -**Do not retarget to match the measurement.** `specs/SessionShape.md` -already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If -the floor moves, it moves on an argument about what batching is worth, and -the argument is written down before the number changes. - -**Done 2026-08-02.** [ADR-0008](../decisions/ADR-0008-instrument-corrections.md) -D1 — **SH-3 is retired as a gate and kept as a diagnostic.** - -Investigating it turned up a third defect, deeper than the two this -workplan was declared on. Measured by re-deriving batching from the raw -transcripts, independently of `cb-cost`: - -| window | with tools | batched | rate | -|---|---:|---:|---:| -| CB-WP-0011 pass | 54 | 0 | **0.0%** | -| gap → next declaration | 16 | 6 | **37.5%** | -| CB-WP-0012 pass | 86 | 0 | **0.0%** | -| gap → next declaration | 10 | 1 | **10.0%** | -| CB-WP-0013 so far | 10 | 0 | **0.0%** | - -**Zero batched turns in 150 in-pass responses; 37.5% in one gap.** -Batching needs two calls whose inputs are known at once — orientation -work. Implementation consumes each step's result before the next. And -SH-3's window is *since the last commit*, which during a pass is always -implementation. **The metric could not read above ~0% in the window it was -gated on.** - -**And this workplan's own Purpose was wrong.** It stated batching *"has -got worse"* (7.8–8.6% historical against 1.1–6.3% per-pass). Those come -from differently-placed windows, not different behaviour. Withdrawn — the -same class of error, in the pass written to correct it. - -The retirement is *not* retargeting to match the measurement: the floor -was not moved to 6%, the gate was removed on an argument about what the -quantity is worth, established before the disposition. The number is still -reported; only the verdict is gone. - -## Task: correct AM-4a to count what ships - -```task -id: CB-WP-0013-T03 -status: done -priority: high -``` - -ADR-0007 Decision 4. `tools/dep-weight.py` counts -`--edges normal,no-proc-macro`. - -**The targets do not move with the instrument.** Correcting a metric that -reports 246,250 down to 157,202 while leaving the target at 250,000 hands -this project 89,048 lines of headroom it did not earn, in the same commit -that reveals the error. Either the target moves down by the same amount -the instrument does, or the pass states plainly why the extra headroom is -legitimate. It is not obviously legitimate; argue it or give it back. - -**Positive control.** `dep-weight.py` already has the guard that an -unlocatable crate must not silently count as zero. The new flag must not -be able to drop crates it should have counted: assert that the proc-macro -set is exactly the five expected crates and that removing the flag -restores the old figure. - -**Done 2026-08-02.** ADR-0008 D2/D3. - -``` -shipped-runtime 18 crates 157,202 lines [ok target 161,000] -``` - -**The target moved down with the instrument**, 250,000 → 161,000, so the -correction hands back essentially nothing: headroom goes from 3,750 to -3,798. 161,000 keeps ~2.4% of room where 250,000 kept ~1.5%; that small -rounding up is the only thing given back, because a target with 1.5% of -room fails on a dependency's patch release. - -Three controls, all green: the exclusion drops **exactly** the five -expected crates, only ever removes and never adds, and is not a no-op. - -**The DFD gate then caught the follow-on**, which is what it is for: three -historical documents carried live `fact:am4a_loc` tags for a number that -had changed. They are history and were not rewritten — the tags were -removed and both documents carry a supersession banner instead. - -**AM-4b is deliberately not corrected.** Its proc-macro share is -unmeasured, and correcting a second instrument on the strength of the -first one's ratio is the error this task exists to fix. Filed as owed. - -## Task: evidence - -```task -id: CB-WP-0013-T04 -status: done -priority: medium -``` - -`evidence/CB-EV-0011-instrument-corrections.md`. - -Carry: whether tier M's merged survey-and-ADR was the right weight for two -instrument corrections; the cost of a meta pass immediately after two -product passes; and the standing question of whether **three** consecutive -evidence files reporting their own pass's cost low means the loop should -simply stop quoting it. - -Also carry the one that is now overdue: **`cb-play` is three modes in one -binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider -the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass -did not act. Either do it or state why not, and stop carrying it. - -**Done 2026-08-02.** -[CB-EV-0011](../evidence/CB-EV-0011-instrument-corrections.md). - -- **The self-quoting rule is proposed.** Three passes have now reported - their own cost mid-flight and all three were low by ~2/3 — and this - time it changed a conclusion: CB-EV-0010's tier comparison said L cost - *24% more* than S; the final figures say **51%**. -- **`cb-play`'s third mode never arrived.** `--serve` is a *play flag* - selecting which `Policy` fills a human seat. The binary still has two - modes. CB-EV-0010 §6 was wrong; the item is **closed, not deferred**. -- **A near-miss recorded**: an instrument reading that looked impossible - turned out to be correct, and one independent re-derivation before - writing anything stopped a false accusation. -- **Tier M was the right weight** — one merged document, no separate - review, and the pass still found a defect deeper than the one it was - declared on, because the finding came from measuring rather than from - an extra review round.