Compare commits
4 commits
e482ab9f8f
...
768eee7d27
| Author | SHA1 | Date | |
|---|---|---|---|
| 768eee7d27 | |||
| d2c3f340f9 | |||
| 6080c4cf60 | |||
| 7105027173 |
12 changed files with 800 additions and 21 deletions
|
|
@ -95,3 +95,7 @@
|
|||
| task | CB-WP-0012-T03 | done | — | workplans/CB-WP-0012-render-port.md |
|
||||
| task | CB-WP-0012-T04 | done | — | workplans/CB-WP-0012-render-port.md |
|
||||
| task | CB-WP-0012-T05 | done | — | workplans/CB-WP-0012-render-port.md |
|
||||
| task | CB-WP-0013-T01 | done | — | workplans/CB-WP-0013-instrument-corrections.md |
|
||||
| task | CB-WP-0013-T02 | done | — | workplans/CB-WP-0013-instrument-corrections.md |
|
||||
| task | CB-WP-0013-T03 | done | — | workplans/CB-WP-0013-instrument-corrections.md |
|
||||
| task | CB-WP-0013-T04 | done | — | workplans/CB-WP-0013-instrument-corrections.md |
|
||||
|
|
|
|||
|
|
@ -1,5 +1,13 @@
|
|||
# ADR-0004: ratify AM-4a and AM-4b
|
||||
|
||||
> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a
|
||||
> figures below were taken with an instrument that counted proc-macro
|
||||
> crates — 89,048 lines, 36.2% — which run in the compiler and never
|
||||
> reach a binary. Corrected, the same tree measures **157,202** against
|
||||
> a target moved to **161,000**. The numbers below are left as the
|
||||
> record of what was measured then, and are no longer live facts.
|
||||
|
||||
|
||||
status: accepted
|
||||
date: 2026-07-31
|
||||
decided by: maintainer (Bernd Worsch), 2026-07-31
|
||||
|
|
@ -49,7 +57,7 @@ At the time of the retarget, `make dep-weight`:
|
|||
|
||||
| configuration | crates | third-party LOC |
|
||||
|---|---|---|
|
||||
| shipped-runtime (`--no-default-features`) | 23 | **246,250** | <!-- fact:am4a_loc -->
|
||||
| shipped-runtime (`--no-default-features`) | 23 | **246,250** | <!-- historical: measured under the pre-ADR-0008 instrument; not a live fact -->
|
||||
| dev-toolchain (default features) | 29 | **317,021** | <!-- fact:am4b_loc -->
|
||||
| own source | — | 3,443 |
|
||||
|
||||
|
|
|
|||
158
decisions/ADR-0008-instrument-corrections.md
Normal file
158
decisions/ADR-0008-instrument-corrections.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
# ADR-0008: retire SH-3 as a gate; correct AM-4a to count what ships
|
||||
|
||||
status: accepted
|
||||
date: 2026-08-02
|
||||
decided by: agent, under the standing loop authorization. One item is
|
||||
reserved for the maintainer — see §Reserved.
|
||||
tier: M (structural M — changes how the loop constrains its own operation:
|
||||
a gate floor and a budget instrument, InnerLoop v1.6 / ADR-0006 D5;
|
||||
chaos d4=3 → no override). Tier M merges survey and ADR into one document,
|
||||
which this is; the adversarial review is optional and was not run.
|
||||
references: [CB-WP-0013](../workplans/CB-WP-0013-instrument-corrections.md),
|
||||
[SessionShape.md](../specs/SessionShape.md) §4,
|
||||
[ADR-0006](ADR-0006-adaptive-control-gates.md) D1/D2/D4,
|
||||
[ADR-0007](ADR-0007-render-html-not-a-port.md) D4,
|
||||
[CB-EV-0010](../evidence/CB-EV-0010-render-port.md) §5
|
||||
|
||||
## Context
|
||||
|
||||
Two instruments were found wrong by the passes that used them. Neither
|
||||
error was in the direction that flattered us, and one of them has been
|
||||
producing a standing breach report for seven passes.
|
||||
|
||||
---
|
||||
|
||||
## Decision 1 — SH-3 is retired as a gate and kept as a diagnostic
|
||||
|
||||
### The survey: three defects, each deeper than the last
|
||||
|
||||
**First (CB-WP-0013 T01): the sample.** `--shape-budget` windows since the
|
||||
last commit and is read while writing the evidence file — right after a
|
||||
commit, when the window holds one or two responses. SH-1/SH-2 are location
|
||||
statistics and survive that. SH-3 is a rate; at n=2 its only values are
|
||||
0%, 50%, 100%. Six evidence files quoted the resulting `0.0%` as a
|
||||
measurement. Fixed: below a derived minimum of 14 the tool now prints
|
||||
`insufficient sample` and no verdict.
|
||||
|
||||
**Second: the claimed decline was also an artifact.** This workplan's own
|
||||
Purpose stated that batching *"has got worse"* — pinned history 7.8–8.6%
|
||||
against per-pass windows of 1.1–6.3%. **That was wrong**, and by the same
|
||||
class of error it was written to correct. The two figures come from
|
||||
differently-placed windows, not from different behaviour. Withdrawn.
|
||||
|
||||
**Third, and the one that decides this: the window and the metric are
|
||||
systematically anti-correlated.** Measured, by re-deriving batching
|
||||
independently of `cb-cost` from the raw transcripts:
|
||||
|
||||
| window | responses with tools | batched | rate |
|
||||
|---|---:|---:|---:|
|
||||
| CB-WP-0011 pass | 54 | 0 | **0.0%** |
|
||||
| gap → next declaration | 16 | 6 | **37.5%** |
|
||||
| CB-WP-0012 pass | 86 | 0 | **0.0%** |
|
||||
| gap → next declaration | 10 | 1 | **10.0%** |
|
||||
| CB-WP-0013 so far | 10 | 0 | **0.0%** |
|
||||
|
||||
**Zero batched turns in 150 in-pass responses across three passes, and
|
||||
37.5% in the gap between two of them** — comfortably above the 20% floor.
|
||||
|
||||
The cause is not discipline. Batching requires two tool calls whose inputs
|
||||
are both known at once. Orientation and survey work is full of those: read
|
||||
four files, run three probes. Implementation has almost none, because each
|
||||
step consumes the previous step's result — read, edit, test, read the
|
||||
failure, fix. **So SH-3 measures what kind of work is happening.**
|
||||
|
||||
And SH-3's window is *since the last commit*, which during a pass is
|
||||
always in-pass work. **The metric is structurally incapable of reading
|
||||
above ~0% in the window it is gated on.** It is not a strict gate — it can
|
||||
only be met by not working.
|
||||
|
||||
### The decision
|
||||
|
||||
**SH-3 stops being a gate.** The 20% floor and the `[SOFT]` verdict are
|
||||
removed. `cb-cost` continues to report the batching rate as a diagnostic,
|
||||
over the whole session, where it reads 5.8% (57 batched of 984 responses
|
||||
with tool calls).
|
||||
|
||||
Four reasons, strongest first:
|
||||
|
||||
1. **It cannot report what it gates on.** §above. A floor that the
|
||||
measurement window structurally excludes is not a target.
|
||||
2. **It measures composition, not waste.** Penalising an implementation
|
||||
pass for not batching penalises it for being an implementation pass.
|
||||
3. **Its own spec bounds its value.** `specs/SessionShape.md` §4:
|
||||
*"eliminating every avoidable single-call turn would remove at most a
|
||||
few percent of turns, worth roughly $2–4 on a $93 pass"*, and *"batch
|
||||
because it is free, not because it is the lever."* It also calls SH-3
|
||||
*"the least valuable"* of the three.
|
||||
4. **Seven passes, seven breach reports, zero actions.** ADR-0006 D4:
|
||||
rituals cash out or go.
|
||||
|
||||
### This is not retargeting to match the measurement
|
||||
|
||||
`specs/SessionShape.md` forbids that explicitly, and CB-WP-0003 T07 exists
|
||||
to prevent it, so the distinction has to be stated rather than assumed:
|
||||
|
||||
- **Retargeting** would be moving the floor from 20% to 6% because 6% is
|
||||
what we measure. That is not done and is still forbidden.
|
||||
- **This** removes the gate on the argument that the *quantity* does not
|
||||
deserve one — established before the disposition, from the window
|
||||
structure and from the spec's own valuation.
|
||||
|
||||
And the obvious objection — *retiring a metric right when it shows a
|
||||
regression looks like burying it* — is answered by the second defect: the
|
||||
regression was an artifact of window placement and has been withdrawn.
|
||||
There is no established decline to bury. The number is still reported;
|
||||
only the verdict is gone.
|
||||
|
||||
## Decision 2 — AM-4a counts `--edges normal,no-proc-macro`
|
||||
|
||||
ADR-0007 D4, owed. AM-4a's figure includes proc-macro crates that run in
|
||||
the compiler and never reach a binary: **89,048 lines, 36.2%** of the
|
||||
shipped-runtime figure, `syn` alone 66,916.
|
||||
|
||||
`tools/dep-weight.py` measures `--edges normal,no-proc-macro` for the
|
||||
shipped-runtime configuration.
|
||||
|
||||
## Decision 3 — the target moves down with the instrument
|
||||
|
||||
Correcting the metric from 246,250 to 157,202 while leaving the target at
|
||||
250,000 would hand this project **89,048 lines of headroom it did not
|
||||
earn**, in the same commit that reveals the error. That is the shape of
|
||||
change this repo has twice refused, and refusing it here costs nothing:
|
||||
|
||||
> **AM-4a's shipped-runtime target moves from 250,000 to 161,000.**
|
||||
|
||||
161,000 preserves the *ratio* the ratified target expressed — 250,000 was
|
||||
set against a 246,250 measurement, i.e. ~1.5% of headroom; 161,000 against
|
||||
157,202 is ~2.4%. Rounding to a round number slightly *above* strict
|
||||
proportionality is deliberate and is the only place this decision gives
|
||||
anything back, because a target that leaves 1.5% of room is a target that
|
||||
fails on a dependency's patch release.
|
||||
|
||||
**AM-4b (dev-toolchain) is left alone**, at 350,000 against a 317,021
|
||||
measurement. It is not corrected in this pass because its proc-macro share
|
||||
has not been measured, and correcting a second instrument on the strength
|
||||
of the first one's ratio is exactly the error this ADR exists to fix.
|
||||
Filed as owed.
|
||||
|
||||
## Reserved for the maintainer
|
||||
|
||||
**AM-4a still cannot survive INTENT stage 2**, and this decision makes that
|
||||
worse rather than better. `wgpu` + `winit` is 1,741,979 marginal lines
|
||||
against a target now proposed at 161,000 — **10.8×**, up from 7×. ADR-0007
|
||||
raised it; nothing here settles it. The correction below is honest
|
||||
book-keeping, not headroom for stage 2, and it should not be read as
|
||||
having made the stage-2 problem smaller.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One fewer gate, and the gate registry gets its first retirement. That is
|
||||
also **`GATE-REVIEW`'s first `caught` entry** — the registry's
|
||||
`retire_if` says it must have *"retired, tightened, or forced the
|
||||
re-justification of"* something by its review date, and it now has.
|
||||
- Every AM-4a figure in this repo's history overstates the load by 36%.
|
||||
Those documents are history and are not rewritten; the correction lives
|
||||
in `specs/GameKernel.md` §4 and here.
|
||||
- The batching diagnostic stays visible, so if in-pass batching ever
|
||||
becomes possible the number will show it without a gate having to be
|
||||
re-litigated first.
|
||||
|
|
@ -1,5 +1,13 @@
|
|||
# CB-EV-0001 — GROUND game kernel: acceptance evidence
|
||||
|
||||
> **Superseded measurement (ADR-0008 D2/D3, 2026-08-02).** The AM-4a
|
||||
> figures below were taken with an instrument that counted proc-macro
|
||||
> crates — 89,048 lines, 36.2% — which run in the compiler and never
|
||||
> reach a binary. Corrected, the same tree measures **157,202** against
|
||||
> a target moved to **161,000**. The numbers below are left as the
|
||||
> record of what was measured then, and are no longer live facts.
|
||||
|
||||
|
||||
Status: **T08 complete. AM-4 remediated and re-measured 2026-07-31.**
|
||||
Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per
|
||||
remediation option, and §5 corrects AM-12 from "uncomputable" to
|
||||
|
|
@ -28,7 +36,7 @@ the property is false? A row can be *measured* and still enforce nothing.
|
|||
| AM-1 rule coverage (GR only) | 100% of GR-rules | 58/58 (100%) | **met** | **yes** |
|
||||
| AM-1b link, ground | 58 claimed rules named in the aggregate | 49/58 | **unmet** | reported |
|
||||
| AM-1b link, kernel | 18 K-rules named in source | 15/18 | **unmet** | reported until 2026-08-31 |
|
||||
| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | <!-- fact:am4a_loc -->
|
||||
| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | **yes** | <!-- historical: measured under the pre-ADR-0008 instrument; not a live fact -->
|
||||
| AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | **yes** | <!-- fact:am4b_loc -->
|
||||
| AM-6 throughput | ≥100,000 events/s | 2,017,009 events/s (`make am6`) | **met, 20.2×** | **yes** — CB-WP-0006 T01 |
|
||||
| AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | **no** — no code computes the ratio |
|
||||
|
|
@ -194,7 +202,7 @@ retargeted onto third-party source under audit.
|
|||
|
||||
| Configuration | Crates | Third-party LOC | Target | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | <!-- fact:am4a_loc --><!-- fact:am4a_target -->
|
||||
| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | <!-- historical: measured under the pre-ADR-0008 instrument; not a live fact -->
|
||||
| Dev toolchain (default features) | 29 | 317,021 | ≤350,000 | **met** | <!-- fact:am4b_loc --><!-- fact:am4b_target -->
|
||||
| Our own source | — | 3,408 | — | — |
|
||||
|
||||
|
|
|
|||
181
evidence/CB-EV-0011-instrument-corrections.md
Normal file
181
evidence/CB-EV-0011-instrument-corrections.md
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
# CB-EV-0011 — three instruments, and a rule about quoting yourself
|
||||
|
||||
CB-WP-0013 T04. Measured 2026-08-02 at `d2c3f34`. Pass kind `meta`,
|
||||
tier **M** (chaos d4=3, no override). Declaration 8 of 12.
|
||||
|
||||
The pass was declared to fix two instruments. It found a third defect in
|
||||
one of them, and a fourth in itself.
|
||||
|
||||
---
|
||||
|
||||
## 1. What was actually wrong with SH-3
|
||||
|
||||
The declaration named one defect. There were three, each deeper.
|
||||
|
||||
**1. The sample.** `--shape-budget` windows since the last commit and is
|
||||
read while writing the evidence file — right after a commit, when the
|
||||
window holds one or two responses. SH-1/SH-2 are location statistics and
|
||||
survive that; SH-3 is a rate, and at n=2 its only values are 0%, 50%,
|
||||
100%. Six evidence files quoted the resulting `0.0%` as a measurement.
|
||||
|
||||
**2. The claimed decline — which was this pass's own error.** The
|
||||
declaration stated batching *"has got worse"*, citing 7.8–8.6% pinned
|
||||
history against 1.1–6.3% per-pass windows. Those are differently-*placed*
|
||||
windows, not different behaviour. **Withdrawn.** The pass declared to
|
||||
correct a trusted-arithmetic error opened by committing one.
|
||||
|
||||
**3. The one that decided it.** Re-deriving batching from the raw
|
||||
transcripts, independently of `cb-cost`:
|
||||
|
||||
| window | with tools | batched | rate |
|
||||
|---|---:|---:|---:|
|
||||
| CB-WP-0011 pass | 54 | 0 | **0.0%** |
|
||||
| gap → next declaration | 16 | 6 | **37.5%** |
|
||||
| CB-WP-0012 pass | 86 | 0 | **0.0%** |
|
||||
| gap → next declaration | 10 | 1 | **10.0%** |
|
||||
| CB-WP-0013 so far | 10 | 0 | **0.0%** |
|
||||
|
||||
**Zero batched turns in 150 in-pass responses. 37.5% in one gap** — above
|
||||
the floor the gate had been failing for seven passes.
|
||||
|
||||
Batching needs two tool calls whose inputs are known at once. That is
|
||||
orientation: read four files, run three probes. Implementation has almost
|
||||
none, because each step consumes the previous step's result. And SH-3's
|
||||
window is *since the last commit*, which during a pass is implementation.
|
||||
**The metric could not read above ~0% in the window it was gated on.**
|
||||
|
||||
A gate meetable only by not working is not a target. SH-3 is retired as a
|
||||
gate (ADR-0008 D1) and kept as a diagnostic over the whole session, where
|
||||
it reads 5.8%.
|
||||
|
||||
**On the appearance of burying it:** retiring a metric that shows a
|
||||
regression looks bad, and should. The answer is defect 2 — the regression
|
||||
was an artifact and has been withdrawn, so there is no established decline
|
||||
to bury. The number is still printed; only the verdict is gone. And the
|
||||
floor was *not* moved to 6%, which is the thing `specs/SessionShape.md`
|
||||
actually forbids.
|
||||
|
||||
## 2. A near-miss worth recording: I almost reported a defect that was not there
|
||||
|
||||
Midway through, a bounded window reported `53 tool calls in 53 responses;
|
||||
0 in batched turns` — exactly one call per response, which looked like the
|
||||
counter had collapsed. I had batched repeatedly in this session, so the
|
||||
reading looked impossible.
|
||||
|
||||
It was correct. Re-deriving from the transcripts independently gave the
|
||||
same 0, and the 131 batched responses that do exist all fall *outside*
|
||||
those windows. The instrument was right and my expectation was wrong.
|
||||
|
||||
Recorded because the **false-accusation class** is already on this
|
||||
project's record, and because the thing that stopped it was cheap: one
|
||||
independent re-derivation before writing anything down. The cost of
|
||||
checking was two minutes; the cost of not checking would have been an
|
||||
evidence file alleging a defect in a tool that did not have one — and this
|
||||
pass's whole subject is instruments reporting things that are not so.
|
||||
|
||||
## 3. AM-4a, corrected, gives back nothing
|
||||
|
||||
```
|
||||
before 23 crates 246,250 lines [ok target 250,000] headroom 3,750
|
||||
after 18 crates 157,202 lines [ok target 161,000] headroom 3,798
|
||||
```
|
||||
|
||||
The instrument dropped 89,048 lines of proc-macro code that never reaches
|
||||
a binary; **the target moved down with it** (ADR-0008 D3). Leaving the
|
||||
target at 250,000 would have handed this project 89,048 lines of headroom
|
||||
it did not earn, in the same change that revealed the error.
|
||||
|
||||
Three controls, all green: the exclusion drops **exactly** the five
|
||||
expected crates, only ever removes and never adds, and is not a no-op.
|
||||
|
||||
**The DFD gate then caught the follow-on**, which is exactly what it is
|
||||
for: three historical documents carried live `fact:am4a_loc` tags for a
|
||||
number that had changed. They are history and were not rewritten — the
|
||||
tags came off and both carry a supersession banner. A gate catching the
|
||||
second-order consequence of a correction, unprompted, is the best evidence
|
||||
yet that the registry is doing something.
|
||||
|
||||
**AM-4b is deliberately not corrected.** Its proc-macro share is
|
||||
unmeasured, and correcting a second instrument on the strength of the
|
||||
first one's ratio is the error this pass exists to fix. Owed.
|
||||
|
||||
## 4. The rule this pass should leave behind: stop quoting your own cost
|
||||
|
||||
| pass | reported mid-pass | final | understated by |
|
||||
|---|---|---|---|
|
||||
| CB-WP-0009 | $6.73 (CB-EV-0008) | $11.31 | 68% |
|
||||
| CB-WP-0011 | $4.23 / 0.094 (CB-EV-0009) | $7.02 / 0.099 | 66% |
|
||||
| CB-WP-0012 | $8.82 / 0.123 (CB-EV-0010) | **$14.52 / 0.150** | 65% |
|
||||
|
||||
Three for three, always low, by a strikingly consistent ~2/3. And this
|
||||
time it **changed a conclusion**, not just a figure:
|
||||
|
||||
> CB-EV-0010 §5 reported the first priced tier comparison as *"0.123
|
||||
> $/response at L against 0.099 at S — 24% more."* The final figures are
|
||||
> **0.150 against 0.099 — 51% more.** The direction holds; the magnitude
|
||||
> was understated by half.
|
||||
|
||||
Tier L is twice as expensive as that comparison claimed. It still found
|
||||
errors of 25× and 85× in CB-WP-0012 and deleted a port that would have
|
||||
been rewritten, so it still looks worth it — but the price of the argument
|
||||
doubled and the evidence file that made it could not have known.
|
||||
|
||||
**Proposed rule, for the next pass to adopt or reject:** an evidence file
|
||||
quotes the *previous* pass's final cost and does not quote its own. Its
|
||||
own is knowable only after the commit that closes it, so any figure it
|
||||
prints is a floor, and this project has now printed three of them and let
|
||||
one carry a conclusion.
|
||||
|
||||
## 5. `cb-play`'s third mode never arrived
|
||||
|
||||
CB-EV-0009 §5 said a third mode would be the second use at which
|
||||
`cb-play`'s single-binary shape should be reconsidered. CB-EV-0010 §6
|
||||
recorded that it had arrived, with `--serve`, and that the pass had not
|
||||
acted. **CB-EV-0010 was wrong.**
|
||||
|
||||
`--serve` is registered in `parse_args` as a *play flag*: it selects which
|
||||
`Policy` fills a human seat, terminal or browser. The binary still has
|
||||
exactly two modes — play a game, or read a recording — and they still take
|
||||
disjoint flags. The trigger did not fire.
|
||||
|
||||
Carried for two passes, resolved here, and **stopped**: there is no split
|
||||
to do, and the standing item is closed rather than deferred again.
|
||||
|
||||
## 6. Cost and shape
|
||||
|
||||
| pass | kind | responses | cost | $/response |
|
||||
|---|---|---|---|---|
|
||||
| CB-WP-0011 | product | 71 | $7.02 | 0.099 |
|
||||
| CB-WP-0012 | product | 97 | $14.52 | 0.150 |
|
||||
| CB-WP-0013 | meta | 34+ | $5.78+ | provisional, see §4 |
|
||||
|
||||
**Meta budget: 21%**, inside the soft 25%, without needing ADR-0006 D2's
|
||||
repair exemption — though both halves would have qualified for it.
|
||||
|
||||
**SH-1 290,475 `[SOFT]`**, drifting toward the hard line; the remedy is a
|
||||
compaction and it is free. **SH-3 now reads `insufficient sample (1
|
||||
response with tool calls, need 14)`** — the fix working in production on
|
||||
its first real reading, where the old instrument would have printed a
|
||||
confident `0.0%`.
|
||||
|
||||
**`gate-review`: 0 due, 0 silent, 0 drifted.** Every gate in the registry
|
||||
now has a `caught` entry, for the first time. GATE-REVIEW earned its own
|
||||
by forcing SH-3's re-justification — asking what a gate had ever caught is
|
||||
what exposed that it could not read what it gated on.
|
||||
|
||||
## 7. Open
|
||||
|
||||
- **AM-4b's proc-macro share is unmeasured.** §3.
|
||||
- **The self-quoting rule in §4 is proposed, not adopted.**
|
||||
- **AM-4a still cannot survive INTENT stage 2**, and this pass made the
|
||||
ratio worse: `wgpu` + `winit` is 1,741,979 against a target now 161,000
|
||||
— **10.8×**, up from 7×. Reserved for the maintainer, twice over now.
|
||||
- **ADR-0007 Decision 3's acquisition rule** remains an argument awaiting
|
||||
ratification.
|
||||
- **INTENT stage 1 stays open**; the emitted JavaScript has still never
|
||||
been executed.
|
||||
- **Tier M's merged survey-and-ADR was the right weight.** One document,
|
||||
no separate review, and the pass still found a defect deeper than the
|
||||
one it was declared on — because the finding came from *measuring*, not
|
||||
from an extra review round. That is one data point for the tier table
|
||||
and against the assumption that depth comes from process weight.
|
||||
|
|
@ -10,14 +10,14 @@ generated = "2026-08-02"
|
|||
pin = "fc76445"
|
||||
|
||||
[am4a_loc]
|
||||
value = 246250
|
||||
text = "246,250"
|
||||
value = 157202
|
||||
text = "157,202"
|
||||
fmt = "{:,}"
|
||||
by = "tools/dep-weight.py"
|
||||
|
||||
[am4a_target]
|
||||
value = 250000
|
||||
text = "250,000"
|
||||
value = 161000
|
||||
text = "161,000"
|
||||
fmt = "{:,}"
|
||||
by = "tools/dep-weight.py TARGETS"
|
||||
|
||||
|
|
|
|||
|
|
@ -36,14 +36,15 @@ caught = [
|
|||
retire_if = "two consecutive passes never approach the soft line, or commits get small enough that the window is always trivial"
|
||||
|
||||
[[gate]]
|
||||
id = "SH-1/SH-2/SH-3"
|
||||
id = "SH-1/SH-2"
|
||||
name = "session-shape budget"
|
||||
target = "shape-budget"
|
||||
checks = "mean and p90 context, and batching rate, since the last commit"
|
||||
checks = "mean and p90 context since the last commit (SH-3 retired as a gate, ADR-0008 D1 — still reported as a diagnostic)"
|
||||
added = "2026-08-01"
|
||||
review_by = "2026-11-30"
|
||||
caught = [
|
||||
"first run fired HARD at 656,574 against a 300,000 ceiling, which is what prompted the compaction before CB-WP-0008",
|
||||
"CB-WP-0013: SH-3 retired from this gate. Its window is *since the last commit*, which during a pass is implementation work, and batching needs two calls whose inputs are known at once — which is orientation work. Measured: 0 batched turns in 150 in-pass responses across three passes, against 37.5% in the gap between two of them. A floor the window structurally excludes is not a target (ADR-0008 D1)",
|
||||
]
|
||||
retire_if = "context stops correlating with cost, or the model's context handling makes the number unactionable"
|
||||
|
||||
|
|
@ -133,5 +134,7 @@ target = "gate-review"
|
|||
checks = "gates past their review date, and gates that have caught nothing"
|
||||
added = "2026-08-01"
|
||||
review_by = "2026-12-31"
|
||||
caught = []
|
||||
caught = [
|
||||
"CB-WP-0013: forced SH-3's re-justification and then its retirement. `make gate-review` had reported SH-3 as a standing breach for seven passes with zero actions taken, which is this registry's own ritual test (ADR-0006 D4); asking what it had ever caught is what exposed that the metric could not read above ~0% in the window it was gated on (ADR-0008 D1)",
|
||||
]
|
||||
retire_if = "it has retired, tightened, or forced the re-justification of nothing by its review date — then it is a ritual, and ADR-0006 D4 says rituals cash out or go"
|
||||
|
|
|
|||
|
|
@ -204,8 +204,8 @@ evidence lands in `evidence/CB-EV-0001-game-kernel.md` with no
|
|||
| AM-1 | M-D1-COV: GR-rules covered by ≥1 passing scenario | no candidate has any (observation) | **100%** of GR + U rules | measured by runner report |
|
||||
| AM-2 | M-D1-SPL: spec lines per rule in `games/ground` rules code (impl LOC ÷ rule count) | boardgame.io ~36 LOC for the 2-move synthetic game | ≤ 40 LOC/rule, paired with AM-1 (anti-gaming pair) | measured (tokei + rule count) |
|
||||
| AM-3 | Synthetic-workload definition size: LOC to express the CB-RES-0001 synthetic game on our kernel | ~36 LOC (boardgame.io, measured) | ≤ 50 LOC | measured |
|
||||
| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 250,000 lines** | measured (`make dep-weight`) |
|
||||
| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** | measured (`make dep-weight`) |
|
||||
| AM-4a | M-D2-DEP: third-party LOC, **shipped runtime** (`--no-default-features --edges normal,no-proc-macro`) | boardgame.io: 120 npm packages / 3.9M LOC | **≤ 161,000 lines** (ADR-0008 D3, was 250,000) | measured (`make dep-weight`) |
|
||||
| AM-4b | M-D2-DEP: third-party LOC, **dev toolchain** (default features) | as above | **≤ 350,000 lines** — *proc-macro share unmeasured; correction owed (ADR-0008 D2)* | measured (`make dep-weight`) |
|
||||
| ~~AM-4c~~ | M-D2-DEP: own source per third-party 100k lines | — | **WITHDRAWN from the acceptance table 2026-08-01 (CB-WP-0006 T04)** — retained as a reported diagnostic in `make dep-weight`; see §5a | diagnostic |
|
||||
| AM-5 | M-D2-BLD: clean release build of headless workspace | n/a (npm install ~seconds; not comparable) | ≤ 60 s on bnt-lap001, recorded not gated | measured |
|
||||
| AM-6 | M-D3-THR: applied events/s, synthetic workload, same machine | boardgame.io ~1,100–1,900 moves/s (best config, degrading) | **≥ 100,000/s** (stipulated target, ADR-0002) | measured |
|
||||
|
|
|
|||
|
|
@ -136,6 +136,47 @@ workplan — the failure mode InnerLoop v1.1 rule 13 exists for, and the
|
|||
`history/260731-inner-loop-rule-audit.md` identified as having no
|
||||
executable defence.
|
||||
|
||||
### Correction (CB-WP-0013 T01): SH-3 was never read as measured
|
||||
|
||||
**Six evidence files quote `SH-3 0.0%`, and none of them measured it.**
|
||||
|
||||
`cb-cost --shape-budget` reports the window **since the last commit**, and
|
||||
it is read while writing the evidence file — immediately *after* a commit,
|
||||
when that window holds one or two responses. SH-1 and SH-2 are location
|
||||
statistics and survive a sample of two. SH-3 is a **rate**, and at n=2 its
|
||||
only possible values are 0%, 50% and 100%. One window, three metrics, and
|
||||
it is only wrong for one of them.
|
||||
|
||||
Measured over real per-pass windows instead:
|
||||
|
||||
| window since | responses | SH-3 |
|
||||
|---|---:|---:|
|
||||
| `0d2ab22` | 115 | 6.2% |
|
||||
| `6fb0aea` | 113 | 6.3% |
|
||||
| `331e7e9` | 95 | 1.1% |
|
||||
| `c5fa610` | 74 | 1.4% |
|
||||
| `883b608` | 70 | 1.4% |
|
||||
| `84d6886` | 24 | 4.3% |
|
||||
|
||||
The substantive claim survives — SH-3 is badly unmet — but the constant
|
||||
0.0% hid something the real numbers show: against the **7.8–8.6%** pinned
|
||||
above, **batching has got worse**, and a metric frozen at a constant could
|
||||
not have shown a trend at all. Six passes reported a breach and the breach
|
||||
was moving underneath them.
|
||||
|
||||
**This is the ninth recorded instance of the trusted-arithmetic class, and
|
||||
the second in this same metric** — the eighth is the `0 of 330 tool calls`
|
||||
figure corrected immediately above. SH-3 is the number this project keeps
|
||||
getting wrong, which is itself a reason to doubt the metric rather than the
|
||||
behaviour (§SH-3 disposition).
|
||||
|
||||
**The fix**, `cb-cost.py` `sh3_line()`: below a minimum sample the tool
|
||||
prints `insufficient sample` and no verdict. The floor is derived, not
|
||||
round — if the true rate were exactly the 20% target, the chance of seeing
|
||||
zero batched turns in `n` responses is `0.8^n`; at **n = 14** that is 4.4%,
|
||||
so "0 batched in 14" rules out a target-meeting rate at ~95%. Below that
|
||||
the tool has nothing to say and now says so.
|
||||
|
||||
At an 8.6% batching rate there is real headroom, but the saving is bounded:
|
||||
eliminating every avoidable single-call turn would remove at most a few
|
||||
percent of turns, worth roughly $2–4 on a $93 pass. **Batch because it is
|
||||
|
|
|
|||
|
|
@ -624,8 +624,15 @@ def render(rep, by_task=False, composition=False):
|
|||
f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]")
|
||||
print(f"{indent}SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok "
|
||||
f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]")
|
||||
print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% "
|
||||
f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]")
|
||||
# Same guard as the budget path: a rate a window cannot support
|
||||
# must not be printed as though it were measured.
|
||||
if sh["responses_with_tools"] < SH3_MIN_SAMPLE:
|
||||
print(f"{indent}SH-3 batching rate insufficient sample "
|
||||
f"[{sh['responses_with_tools']} with tool calls, "
|
||||
f"need {SH3_MIN_SAMPLE}]")
|
||||
else:
|
||||
print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% "
|
||||
f"[diagnostic, no floor — ADR-0008 D1]")
|
||||
print(f"{indent} {sh['tool_calls']} tool calls in "
|
||||
f"{sh['responses_with_tools']} responses; "
|
||||
f"{sh['calls_in_batched_turns']} in batched turns")
|
||||
|
|
@ -786,6 +793,33 @@ def self_test():
|
|||
finally:
|
||||
os.unlink(empty)
|
||||
|
||||
# SH-3 minimum sample (CB-WP-0013 T01). Three checks, because the
|
||||
# guard has three ways to be useless: it could refuse everything, it
|
||||
# could refuse nothing, or its refusal could be mistaken for a rate.
|
||||
def win(n_with_tools, n_batched):
|
||||
return {"responses_with_tools": n_with_tools,
|
||||
"SH-3_batching_rate": (n_batched / n_with_tools) if n_with_tools else 0.0}
|
||||
|
||||
below = sh3_line(win(2, 0))
|
||||
at = sh3_line(win(SH3_MIN_SAMPLE, 0))
|
||||
good = sh3_line(win(20, 8))
|
||||
check("SH-3 refuses a window too small to carry a rate",
|
||||
"insufficient sample" in below and "%" not in below,
|
||||
below.strip())
|
||||
check("SH-3 still reports at exactly the minimum sample",
|
||||
"insufficient sample" not in at and "0.0%" in at, at.strip())
|
||||
check("SH-3 reports a real rate, with no verdict attached",
|
||||
"40.0%" in good and "no floor" in good and "FAIL" not in good,
|
||||
good.strip())
|
||||
# ADR-0008 D1: the gate is gone. A verdict reappearing here means a
|
||||
# floor has been reintroduced without an ADR.
|
||||
check("SH-3 carries no pass/fail verdict",
|
||||
not any(v in sh3_line(win(40, 1)) for v in ("SOFT", "HARD", "FAIL", "[ok")),
|
||||
sh3_line(win(40, 1)).strip())
|
||||
# The one that matters: a refusal must not read as a measured zero.
|
||||
check("a refusal is distinguishable from a genuine 0.0%",
|
||||
below != sh3_line(win(SH3_MIN_SAMPLE, 0)))
|
||||
|
||||
# AC-7: the subagent tree is discovered by the path globs.
|
||||
slug = "-home-worsch-clay-borg"
|
||||
paths = transcript_paths(slug)
|
||||
|
|
@ -809,6 +843,47 @@ def self_test():
|
|||
SHAPE_SOFT = {"SH-1": 200_000, "SH-2": 300_000}
|
||||
SHAPE_HARD = {"SH-1": 300_000, "SH-2": 450_000}
|
||||
|
||||
# CB-WP-0013 T01. SH-1 and SH-2 are location statistics and survive a
|
||||
# two-response window. SH-3 is a RATE, and at n=2 its only possible values
|
||||
# are 0%, 50% and 100% — so reading it right after a commit, which is when
|
||||
# the evidence file is written, reports 0.0% almost regardless of
|
||||
# behaviour. Six evidence files quoted that 0.0% as a measurement; the real
|
||||
# per-pass figure was 1.1-6.3%.
|
||||
#
|
||||
# The floor is derived, not round. If the true rate were exactly the 20%
|
||||
# target, the chance of observing ZERO batched turns in n responses is
|
||||
# 0.8^n. At n = 14 that is 4.4%, so "0 batched in 14" rules out a
|
||||
# target-meeting rate at ~95%. Below 14 the tool has nothing to say and
|
||||
# must say that instead of printing a number.
|
||||
SH3_MIN_SAMPLE = 14
|
||||
|
||||
|
||||
def sh3_line(win, min_sample=SH3_MIN_SAMPLE):
|
||||
"""SH-3 as a DIAGNOSTIC — no floor, no verdict (ADR-0008 D1).
|
||||
|
||||
Retired as a gate because the window and the metric are systematically
|
||||
anti-correlated: batching needs two tool calls whose inputs are known
|
||||
at once, which is orientation work, and this window is *since the last
|
||||
commit*, which during a pass is implementation work. Measured across
|
||||
three passes: 0 batched turns in 150 in-pass responses, against 37.5%
|
||||
in the gap between two of them. A floor the window structurally
|
||||
excludes is not a target.
|
||||
|
||||
The minimum-sample refusal is kept. A diagnostic may be ignored; it
|
||||
may not be wrong, and a rate over two responses is wrong.
|
||||
|
||||
Returns the line rather than printing it so both branches are testable
|
||||
without capturing stdout — a guard that can only be checked by reading
|
||||
output is a guard that will be checked by nobody.
|
||||
"""
|
||||
n = win["responses_with_tools"]
|
||||
if n < min_sample:
|
||||
return (f" SH-3 batching insufficient sample "
|
||||
f"[----] {n} response(s) with tool calls, need {min_sample}")
|
||||
rate = win["SH-3_batching_rate"]
|
||||
return (f" SH-3 batching {100*rate:>9.1f}% "
|
||||
f"[diag] no floor — retired as a gate, ADR-0008 D1")
|
||||
|
||||
|
||||
def shape_budget(slug):
|
||||
"""SH-* for the window since the last commit, with soft/hard verdicts."""
|
||||
|
|
@ -842,9 +917,7 @@ def shape_budget(slug):
|
|||
print(f" {key} {label} {val:>10,.0f} tok [{mark}] "
|
||||
f"soft {soft:,} / hard {hard:,}")
|
||||
breach = max(breach, 0 if val <= soft else (1 if val <= hard else 2))
|
||||
rate = win["SH-3_batching_rate"]
|
||||
print(f" SH-3 batching {100*rate:>9.1f}% "
|
||||
f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%")
|
||||
print(sh3_line(win))
|
||||
|
||||
if breach >= 2:
|
||||
print("\n HARD — compact before continuing. Context this size costs "
|
||||
|
|
|
|||
|
|
@ -28,15 +28,30 @@ import sys
|
|||
from repo import cargo_bin, enter_root
|
||||
|
||||
PACKAGE = "games-ground"
|
||||
# ADR-0008 D2. `--edges normal` includes proc-macro crates, which run in
|
||||
# the compiler and never reach a shipped binary — 89,048 lines, 36.2% of
|
||||
# what this tool used to call "what a game ships", `syn` alone 66,916. The
|
||||
# shipped-runtime configuration now excludes them.
|
||||
#
|
||||
# AM-4b is deliberately NOT corrected here: its proc-macro share has not
|
||||
# been measured, and correcting a second instrument on the strength of the
|
||||
# first one's ratio is the error this change exists to fix.
|
||||
PROC_MACRO_EXCLUDED = ["--edges", "normal,no-proc-macro"]
|
||||
CONFIGS = {
|
||||
"shipped-runtime": ["--no-default-features"],
|
||||
"shipped-runtime": ["--no-default-features"] + PROC_MACRO_EXCLUDED,
|
||||
"dev-toolchain": [],
|
||||
}
|
||||
|
||||
# AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails
|
||||
# the build: a gate that only reports is a suggestion.
|
||||
# ADR-0008 D3: the target moves down with the instrument. Leaving it at
|
||||
# 250,000 against a corrected 157,202 would hand this project 89,048 lines
|
||||
# of headroom it did not earn, in the same change that revealed the error.
|
||||
# 161,000 keeps ~2.4% of room where 250,000 kept ~1.5% — the small
|
||||
# rounding up is the only thing this decision gives back, because a target
|
||||
# with 1.5% of room fails on a dependency's patch release.
|
||||
TARGETS = {
|
||||
"shipped-runtime": 250_000,
|
||||
"shipped-runtime": 161_000,
|
||||
"dev-toolchain": 350_000,
|
||||
}
|
||||
|
||||
|
|
@ -55,8 +70,8 @@ def crates(extra_args):
|
|||
)
|
||||
sys.exit(1)
|
||||
out = subprocess.run(
|
||||
[cargo, "tree", "-p", PACKAGE, "--edges", "normal", "--prefix", "none"]
|
||||
+ extra_args,
|
||||
[cargo, "tree", "-p", PACKAGE, "--prefix", "none"]
|
||||
+ (extra_args if "--edges" in extra_args else ["--edges", "normal"] + extra_args),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
|
|
@ -125,6 +140,23 @@ def self_test():
|
|||
check("cargo resolves without caller PATH setup", bool(cargo_bin()),
|
||||
cargo_bin() or "NOT FOUND")
|
||||
|
||||
# ADR-0008 D2. The exclusion must remove exactly the proc-macro crates
|
||||
# and nothing else — a flag that quietly dropped a runtime dependency
|
||||
# would shrink the number in the direction this metric can be gamed.
|
||||
with_pm = crates(["--no-default-features"])
|
||||
without_pm = crates(["--no-default-features"] + PROC_MACRO_EXCLUDED)
|
||||
dropped = set(with_pm) - set(without_pm)
|
||||
check("the proc-macro exclusion drops exactly the expected crates",
|
||||
dropped == {"syn", "quote", "proc-macro2", "unicode-ident", "serde_derive"},
|
||||
f"dropped {sorted(dropped)}")
|
||||
check("the exclusion only ever removes crates, never adds",
|
||||
set(without_pm) <= set(with_pm),
|
||||
f"{len(with_pm)} -> {len(without_pm)}")
|
||||
# And it must actually remove something: an exclusion that excluded
|
||||
# nothing would leave the old figure while claiming the new meaning.
|
||||
check("the exclusion is not a no-op",
|
||||
len(dropped) > 0, f"{len(dropped)} crate(s) dropped")
|
||||
|
||||
check("targets defined for every configuration",
|
||||
set(TARGETS) == set(CONFIGS) and all(
|
||||
isinstance(v, int) and v > 0 for v in TARGETS.values()),
|
||||
|
|
|
|||
271
workplans/CB-WP-0013-instrument-corrections.md
Normal file
271
workplans/CB-WP-0013-instrument-corrections.md
Normal file
|
|
@ -0,0 +1,271 @@
|
|||
---
|
||||
id: CB-WP-0013
|
||||
kind: meta
|
||||
title: "Two instruments that were wrong, and the record they wrote"
|
||||
status: done
|
||||
---
|
||||
|
||||
# Purpose
|
||||
|
||||
```
|
||||
structural tier M (changes how the loop constrains its own operation:
|
||||
a budget instrument and a gate floor — InnerLoop
|
||||
v1.6, ADR-0006 D5)
|
||||
chaos d4 = 3 → no override
|
||||
declared tier M
|
||||
```
|
||||
|
||||
Declaration 8 of 12 in the calibration window. Tier M means the survey and
|
||||
the ADR merge into one document and the adversarial review is optional.
|
||||
|
||||
**Meta budget: 0% of the trailing three**, so this pass is in budget
|
||||
without the ADR-0006 D2 repair exemption — though it would qualify for it,
|
||||
since both halves repair an instrument that reports a breach.
|
||||
|
||||
## Why now
|
||||
|
||||
CB-WP-0012 ended owing two corrections, and one of them turns out to be
|
||||
larger than it was filed as.
|
||||
|
||||
### SH-3 has never read what six evidence files say it read
|
||||
|
||||
CB-EV-0010 §5 states *"SH-3 has now read 0.0% for six consecutive passes"*
|
||||
and calls it the oldest unargued number in the project. The number is an
|
||||
artifact.
|
||||
|
||||
`--shape-budget` measures **the window since the last commit**, and it is
|
||||
read while writing the evidence file — immediately *after* a commit, when
|
||||
that window holds one or two responses. SH-1 and SH-2 are location
|
||||
statistics and survive a sample of two. **SH-3 is a rate**, and at n=2 its
|
||||
only possible values are 0%, 50% and 100%. One window, three metrics, and
|
||||
it is only wrong for one of them.
|
||||
|
||||
Measured over real per-pass windows instead:
|
||||
|
||||
| window since | responses | SH-3 |
|
||||
|---|---:|---:|
|
||||
| `0d2ab22` | 115 | 6.2% |
|
||||
| `6fb0aea` | 113 | 6.3% |
|
||||
| `331e7e9` | 95 | 1.1% |
|
||||
| `c5fa610` | 74 | 1.4% |
|
||||
| `883b608` | 70 | 1.4% |
|
||||
| `84d6886` | 24 | 4.3% |
|
||||
|
||||
So the substantive claim survives — SH-3 is badly unmet — but **the figure
|
||||
quoted in six evidence files was never the measurement**. Worse, the real
|
||||
numbers say something the artifact hid: the historical rate pinned in
|
||||
`specs/SessionShape.md` is **7.8–8.6%**, and these windows are **1.1–6.3%**.
|
||||
Batching has got *worse*, and a metric stuck at a constant 0.0% could not
|
||||
show a trend at all.
|
||||
|
||||
### AM-4a overstates every figure it has ever reported by 36%
|
||||
|
||||
ADR-0007 Decision 4, filed and unimplemented. AM-4a counts
|
||||
`--edges normal`, which includes proc-macro crates that run in the compiler
|
||||
and never reach a binary: **89,048 lines, 36.2%** of the shipped-runtime
|
||||
figure, `syn` alone 66,916.
|
||||
|
||||
## Task: fix SH-3's instrument, and correct the record
|
||||
|
||||
```task
|
||||
id: CB-WP-0013-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
A rate needs a sample. `--shape-budget` must **refuse to report SH-3**
|
||||
below a stated minimum rather than print a number the window cannot
|
||||
support — `insufficient sample (n responses)`, not `0.0%`.
|
||||
|
||||
**The choice to argue, not assume:** whether SH-3 should instead be
|
||||
windowed differently from SH-1/SH-2 — over the pass rather than since the
|
||||
last commit — since it is the only one of the three that a two-response
|
||||
window destroys. Pick one and say why the other was rejected.
|
||||
|
||||
**Controls.** A minimum-sample rule is exactly the kind of guard that can
|
||||
be satisfied vacuously: assert that a window *above* the floor still
|
||||
reports a rate, that one below reports the refusal, and that the refusal
|
||||
is distinguishable from a genuine 0.0%. Mutate each.
|
||||
|
||||
**Correct the record.** Six evidence files quote 0.0%. They are history and
|
||||
are not rewritten; the correction is recorded once, where a reader will
|
||||
find it, and named as what it is — the same class as CB-WP-0002's
|
||||
"0 of 330 tool calls were batched", which `specs/SessionShape.md` §4
|
||||
already records as the eighth trusted-arithmetic instance. **This is the
|
||||
ninth, and it is in the same metric.** That coincidence is worth a
|
||||
sentence: SH-3 is the number this project keeps getting wrong.
|
||||
|
||||
**Done 2026-08-02.** `cb-cost.py` gains `sh3_line()`: below a minimum
|
||||
sample it prints `insufficient sample` and **no verdict**.
|
||||
|
||||
**The window was kept, not split.** SH-3 could instead have been given a
|
||||
per-pass window while SH-1/SH-2 kept the since-last-commit one. Rejected:
|
||||
the budget's stated purpose is the *open remainder* since the last commit
|
||||
— the only thing a live budget can fire on — and giving one of three
|
||||
metrics a different window makes "the window" ambiguous in a tool that
|
||||
three specs cite. Refusing to report is honest and keeps one window.
|
||||
|
||||
**The floor is derived, not round.** If the true rate were exactly the 20%
|
||||
target, the chance of observing zero batched turns in `n` responses is
|
||||
`0.8^n`. At **n = 14** that is 4.4%, so "0 batched in 14" rules out a
|
||||
target-meeting rate at ~95%. Below 14 the tool has nothing to say.
|
||||
|
||||
Four controls, three mutations, each red for its stated reason:
|
||||
|
||||
| mutation | result |
|
||||
|---|---|
|
||||
| the guard never refuses | `SH-3 refuses a window too small` red, *and* `a refusal is distinguishable from a genuine 0.0%` red |
|
||||
| the guard always refuses | reporting at exactly the minimum, and a real 40% rate, both red |
|
||||
| the refusal prints as `0.0%` | `refuses a window too small` red — the case the six evidence files actually hit |
|
||||
|
||||
The correction is recorded in `specs/SessionShape.md` §4, beside the
|
||||
eighth instance, with the real per-pass figures.
|
||||
|
||||
## Task: argue SH-3's floor, or move it
|
||||
|
||||
```task
|
||||
id: CB-WP-0013-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
The declaration CB-EV-0010 said was owed. `specs/SessionShape.md` is
|
||||
explicit that the 20% floor was set above the measured rate deliberately
|
||||
and that *"whether 20% is reachable or arbitrary is unknown until a pass
|
||||
tries"*. Seven passes have now reported it and none has tried.
|
||||
|
||||
Decide, with the real numbers rather than the artifact:
|
||||
|
||||
- is 20% reachable, and at what cost to how the work is actually done;
|
||||
- is the measured decline from 7.8–8.6% to 1.1–6.3% a behavioural change
|
||||
worth reversing, or an artifact of tier-L passes doing more sequential
|
||||
reading than tier-S ones;
|
||||
- **and the honest option: retire SH-3.** `specs/SessionShape.md` §4 puts
|
||||
the ceiling at **$2–4 on a $93 pass** and says *"batch because it is
|
||||
free, not because it is the lever."* A floor that is 3–20× the measured
|
||||
rate, cannot be moved by the party measured, and is worth single-digit
|
||||
dollars is a candidate for `retire_if`, not a standing breach.
|
||||
|
||||
**Do not retarget to match the measurement.** `specs/SessionShape.md`
|
||||
already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If
|
||||
the floor moves, it moves on an argument about what batching is worth, and
|
||||
the argument is written down before the number changes.
|
||||
|
||||
**Done 2026-08-02.** [ADR-0008](../decisions/ADR-0008-instrument-corrections.md)
|
||||
D1 — **SH-3 is retired as a gate and kept as a diagnostic.**
|
||||
|
||||
Investigating it turned up a third defect, deeper than the two this
|
||||
workplan was declared on. Measured by re-deriving batching from the raw
|
||||
transcripts, independently of `cb-cost`:
|
||||
|
||||
| window | with tools | batched | rate |
|
||||
|---|---:|---:|---:|
|
||||
| CB-WP-0011 pass | 54 | 0 | **0.0%** |
|
||||
| gap → next declaration | 16 | 6 | **37.5%** |
|
||||
| CB-WP-0012 pass | 86 | 0 | **0.0%** |
|
||||
| gap → next declaration | 10 | 1 | **10.0%** |
|
||||
| CB-WP-0013 so far | 10 | 0 | **0.0%** |
|
||||
|
||||
**Zero batched turns in 150 in-pass responses; 37.5% in one gap.**
|
||||
Batching needs two calls whose inputs are known at once — orientation
|
||||
work. Implementation consumes each step's result before the next. And
|
||||
SH-3's window is *since the last commit*, which during a pass is always
|
||||
implementation. **The metric could not read above ~0% in the window it was
|
||||
gated on.**
|
||||
|
||||
**And this workplan's own Purpose was wrong.** It stated batching *"has
|
||||
got worse"* (7.8–8.6% historical against 1.1–6.3% per-pass). Those come
|
||||
from differently-placed windows, not different behaviour. Withdrawn — the
|
||||
same class of error, in the pass written to correct it.
|
||||
|
||||
The retirement is *not* retargeting to match the measurement: the floor
|
||||
was not moved to 6%, the gate was removed on an argument about what the
|
||||
quantity is worth, established before the disposition. The number is still
|
||||
reported; only the verdict is gone.
|
||||
|
||||
## Task: correct AM-4a to count what ships
|
||||
|
||||
```task
|
||||
id: CB-WP-0013-T03
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
ADR-0007 Decision 4. `tools/dep-weight.py` counts
|
||||
`--edges normal,no-proc-macro`.
|
||||
|
||||
**The targets do not move with the instrument.** Correcting a metric that
|
||||
reports 246,250 down to 157,202 while leaving the target at 250,000 hands
|
||||
this project 89,048 lines of headroom it did not earn, in the same commit
|
||||
that reveals the error. Either the target moves down by the same amount
|
||||
the instrument does, or the pass states plainly why the extra headroom is
|
||||
legitimate. It is not obviously legitimate; argue it or give it back.
|
||||
|
||||
**Positive control.** `dep-weight.py` already has the guard that an
|
||||
unlocatable crate must not silently count as zero. The new flag must not
|
||||
be able to drop crates it should have counted: assert that the proc-macro
|
||||
set is exactly the five expected crates and that removing the flag
|
||||
restores the old figure.
|
||||
|
||||
**Done 2026-08-02.** ADR-0008 D2/D3.
|
||||
|
||||
```
|
||||
shipped-runtime 18 crates 157,202 lines [ok target 161,000]
|
||||
```
|
||||
|
||||
**The target moved down with the instrument**, 250,000 → 161,000, so the
|
||||
correction hands back essentially nothing: headroom goes from 3,750 to
|
||||
3,798. 161,000 keeps ~2.4% of room where 250,000 kept ~1.5%; that small
|
||||
rounding up is the only thing given back, because a target with 1.5% of
|
||||
room fails on a dependency's patch release.
|
||||
|
||||
Three controls, all green: the exclusion drops **exactly** the five
|
||||
expected crates, only ever removes and never adds, and is not a no-op.
|
||||
|
||||
**The DFD gate then caught the follow-on**, which is what it is for: three
|
||||
historical documents carried live `fact:am4a_loc` tags for a number that
|
||||
had changed. They are history and were not rewritten — the tags were
|
||||
removed and both documents carry a supersession banner instead.
|
||||
|
||||
**AM-4b is deliberately not corrected.** Its proc-macro share is
|
||||
unmeasured, and correcting a second instrument on the strength of the
|
||||
first one's ratio is the error this task exists to fix. Filed as owed.
|
||||
|
||||
## Task: evidence
|
||||
|
||||
```task
|
||||
id: CB-WP-0013-T04
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
`evidence/CB-EV-0011-instrument-corrections.md`.
|
||||
|
||||
Carry: whether tier M's merged survey-and-ADR was the right weight for two
|
||||
instrument corrections; the cost of a meta pass immediately after two
|
||||
product passes; and the standing question of whether **three** consecutive
|
||||
evidence files reporting their own pass's cost low means the loop should
|
||||
simply stop quoting it.
|
||||
|
||||
Also carry the one that is now overdue: **`cb-play` is three modes in one
|
||||
binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider
|
||||
the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass
|
||||
did not act. Either do it or state why not, and stop carrying it.
|
||||
|
||||
**Done 2026-08-02.**
|
||||
[CB-EV-0011](../evidence/CB-EV-0011-instrument-corrections.md).
|
||||
|
||||
- **The self-quoting rule is proposed.** Three passes have now reported
|
||||
their own cost mid-flight and all three were low by ~2/3 — and this
|
||||
time it changed a conclusion: CB-EV-0010's tier comparison said L cost
|
||||
*24% more* than S; the final figures say **51%**.
|
||||
- **`cb-play`'s third mode never arrived.** `--serve` is a *play flag*
|
||||
selecting which `Policy` fills a human seat. The binary still has two
|
||||
modes. CB-EV-0010 §6 was wrong; the item is **closed, not deferred**.
|
||||
- **A near-miss recorded**: an instrument reading that looked impossible
|
||||
turned out to be correct, and one independent re-derivation before
|
||||
writing anything stopped a false accusation.
|
||||
- **Tier M was the right weight** — one merged document, no separate
|
||||
review, and the pass still found a defect deeper than the one it was
|
||||
declared on, because the finding came from measuring rather than from
|
||||
an extra review round.
|
||||
Loading…
Add table
Add a link
Reference in a new issue