clay-borg/workplans/CB-WP-0013-instrument-corrections.md

---
id: CB-WP-0013
kind: meta
title: "Two instruments that were wrong, and the record they wrote"
status: done
state_hub_workstream_id: "b5adc29b-1a83-4958-8ca6-7be072dccacf"
---

# Purpose

```
structural tier  M   (changes how the loop constrains its own operation:
                      a budget instrument and a gate floor — InnerLoop
                      v1.6, ADR-0006 D5)
chaos            d4 = 3  → no override
declared tier    M
```

Declaration 8 of 12 in the calibration window. Tier M means the survey and
the ADR merge into one document and the adversarial review is optional.

**Meta budget: 0% of the trailing three**, so this pass is in budget
without the ADR-0006 D2 repair exemption — though it would qualify for it,
since both halves repair an instrument that reports a breach.

## Why now

CB-WP-0012 ended owing two corrections, and one of them turns out to be
larger than it was filed as.

### SH-3 has never read what six evidence files say it read

CB-EV-0010 §5 states *"SH-3 has now read 0.0% for six consecutive passes"*
and calls it the oldest unargued number in the project. The number is an
artifact.

`--shape-budget` measures **the window since the last commit**, and it is
read while writing the evidence file — immediately *after* a commit, when
that window holds one or two responses. SH-1 and SH-2 are location
statistics and survive a sample of two. **SH-3 is a rate**, and at n=2 its
only possible values are 0%, 50% and 100%. One window, three metrics, and
it is only wrong for one of them.

Measured over real per-pass windows instead:

| window since | responses | SH-3 |
|---|---:|---:|
| `0d2ab22` | 115 | 6.2% |
| `6fb0aea` | 113 | 6.3% |
| `331e7e9` | 95 | 1.1% |
| `c5fa610` | 74 | 1.4% |
| `883b608` | 70 | 1.4% |
| `84d6886` | 24 | 4.3% |

So the substantive claim survives — SH-3 is badly unmet — but **the figure
quoted in six evidence files was never the measurement**. Worse, the real
numbers say something the artifact hid: the historical rate pinned in
`specs/SessionShape.md` is **7.8–8.6%**, and these windows are **1.1–6.3%**.
Batching has got *worse*, and a metric stuck at a constant 0.0% could not
show a trend at all.

### AM-4a overstates every figure it has ever reported by 36%

ADR-0007 Decision 4, filed and unimplemented. AM-4a counts
`--edges normal`, which includes proc-macro crates that run in the compiler
and never reach a binary: **89,048 lines, 36.2%** of the shipped-runtime
figure, `syn` alone 66,916.

## Task: fix SH-3's instrument, and correct the record

```task
id: CB-WP-0013-T01
status: done
priority: high
state_hub_task_id: "0e2b9dc1-d3e7-4833-b03c-24ff987e5b95"
```

A rate needs a sample. `--shape-budget` must **refuse to report SH-3**
below a stated minimum rather than print a number the window cannot
support — `insufficient sample (n responses)`, not `0.0%`.

**The choice to argue, not assume:** whether SH-3 should instead be
windowed differently from SH-1/SH-2 — over the pass rather than since the
last commit — since it is the only one of the three that a two-response
window destroys. Pick one and say why the other was rejected.

**Controls.** A minimum-sample rule is exactly the kind of guard that can
be satisfied vacuously: assert that a window *above* the floor still
reports a rate, that one below reports the refusal, and that the refusal
is distinguishable from a genuine 0.0%. Mutate each.

**Correct the record.** Six evidence files quote 0.0%. They are history and
are not rewritten; the correction is recorded once, where a reader will
find it, and named as what it is — the same class as CB-WP-0002's
"0 of 330 tool calls were batched", which `specs/SessionShape.md` §4
already records as the eighth trusted-arithmetic instance. **This is the
ninth, and it is in the same metric.** That coincidence is worth a
sentence: SH-3 is the number this project keeps getting wrong.

**Done 2026-08-02.** `cb-cost.py` gains `sh3_line()`: below a minimum
sample it prints `insufficient sample` and **no verdict**.

**The window was kept, not split.** SH-3 could instead have been given a
per-pass window while SH-1/SH-2 kept the since-last-commit one. Rejected:
the budget's stated purpose is the *open remainder* since the last commit
— the only thing a live budget can fire on — and giving one of three
metrics a different window makes "the window" ambiguous in a tool that
three specs cite. Refusing to report is honest and keeps one window.

**The floor is derived, not round.** If the true rate were exactly the 20%
target, the chance of observing zero batched turns in `n` responses is
`0.8^n`. At **n = 14** that is 4.4%, so "0 batched in 14" rules out a
target-meeting rate at ~95%. Below 14 the tool has nothing to say.

Four controls, three mutations, each red for its stated reason:

| mutation | result |
|---|---|
| the guard never refuses | `SH-3 refuses a window too small` red, *and* `a refusal is distinguishable from a genuine 0.0%` red |
| the guard always refuses | reporting at exactly the minimum, and a real 40% rate, both red |
| the refusal prints as `0.0%` | `refuses a window too small` red — the case the six evidence files actually hit |

The correction is recorded in `specs/SessionShape.md` §4, beside the
eighth instance, with the real per-pass figures.

## Task: argue SH-3's floor, or move it

```task
id: CB-WP-0013-T02
status: done
priority: high
state_hub_task_id: "c926c370-26fd-4450-aeee-bc2dff69cdbc"
```

The declaration CB-EV-0010 said was owed. `specs/SessionShape.md` is
explicit that the 20% floor was set above the measured rate deliberately
and that *"whether 20% is reachable or arbitrary is unknown until a pass
tries"*. Seven passes have now reported it and none has tried.

Decide, with the real numbers rather than the artifact:

- is 20% reachable, and at what cost to how the work is actually done;
- is the measured decline from 7.8–8.6% to 1.1–6.3% a behavioural change
  worth reversing, or an artifact of tier-L passes doing more sequential
  reading than tier-S ones;
- **and the honest option: retire SH-3.** `specs/SessionShape.md` §4 puts
  the ceiling at **$2–4 on a $93 pass** and says *"batch because it is
  free, not because it is the lever."* A floor that is 3–20× the measured
  rate, cannot be moved by the party measured, and is worth single-digit
  dollars is a candidate for `retire_if`, not a standing breach.

**Do not retarget to match the measurement.** `specs/SessionShape.md`
already forbids exactly that, and CB-WP-0003 T07 exists to prevent it. If
the floor moves, it moves on an argument about what batching is worth, and
the argument is written down before the number changes.

**Done 2026-08-02.** [ADR-0008](../decisions/ADR-0008-instrument-corrections.md)
D1 — **SH-3 is retired as a gate and kept as a diagnostic.**

Investigating it turned up a third defect, deeper than the two this
workplan was declared on. Measured by re-deriving batching from the raw
transcripts, independently of `cb-cost`:

| window | with tools | batched | rate |
|---|---:|---:|---:|
| CB-WP-0011 pass | 54 | 0 | **0.0%** |
| gap → next declaration | 16 | 6 | **37.5%** |
| CB-WP-0012 pass | 86 | 0 | **0.0%** |
| gap → next declaration | 10 | 1 | **10.0%** |
| CB-WP-0013 so far | 10 | 0 | **0.0%** |

**Zero batched turns in 150 in-pass responses; 37.5% in one gap.**
Batching needs two calls whose inputs are known at once — orientation
work. Implementation consumes each step's result before the next. And
SH-3's window is *since the last commit*, which during a pass is always
implementation. **The metric could not read above ~0% in the window it was
gated on.**

**And this workplan's own Purpose was wrong.** It stated batching *"has
got worse"* (7.8–8.6% historical against 1.1–6.3% per-pass). Those come
from differently-placed windows, not different behaviour. Withdrawn — the
same class of error, in the pass written to correct it.

The retirement is *not* retargeting to match the measurement: the floor
was not moved to 6%, the gate was removed on an argument about what the
quantity is worth, established before the disposition. The number is still
reported; only the verdict is gone.

## Task: correct AM-4a to count what ships

```task
id: CB-WP-0013-T03
status: done
priority: high
state_hub_task_id: "51265ba6-2517-49d0-b842-d9a73016124d"
```

ADR-0007 Decision 4. `tools/dep-weight.py` counts
`--edges normal,no-proc-macro`.

**The targets do not move with the instrument.** Correcting a metric that
reports 246,250 down to 157,202 while leaving the target at 250,000 hands
this project 89,048 lines of headroom it did not earn, in the same commit
that reveals the error. Either the target moves down by the same amount
the instrument does, or the pass states plainly why the extra headroom is
legitimate. It is not obviously legitimate; argue it or give it back.

**Positive control.** `dep-weight.py` already has the guard that an
unlocatable crate must not silently count as zero. The new flag must not
be able to drop crates it should have counted: assert that the proc-macro
set is exactly the five expected crates and that removing the flag
restores the old figure.

**Done 2026-08-02.** ADR-0008 D2/D3.

```
shipped-runtime   18 crates   157,202 lines   [ok  target 161,000]
```

**The target moved down with the instrument**, 250,000 → 161,000, so the
correction hands back essentially nothing: headroom goes from 3,750 to
3,798. 161,000 keeps ~2.4% of room where 250,000 kept ~1.5%; that small
rounding up is the only thing given back, because a target with 1.5% of
room fails on a dependency's patch release.

Three controls, all green: the exclusion drops **exactly** the five
expected crates, only ever removes and never adds, and is not a no-op.

**The DFD gate then caught the follow-on**, which is what it is for: three
historical documents carried live `fact:am4a_loc` tags for a number that
had changed. They are history and were not rewritten — the tags were
removed and both documents carry a supersession banner instead.

**AM-4b is deliberately not corrected.** Its proc-macro share is
unmeasured, and correcting a second instrument on the strength of the
first one's ratio is the error this task exists to fix. Filed as owed.

## Task: evidence

```task
id: CB-WP-0013-T04
status: done
priority: medium
state_hub_task_id: "e9b20874-692a-48df-b3ee-c2929d7fe5a8"
```

`evidence/CB-EV-0011-instrument-corrections.md`.

Carry: whether tier M's merged survey-and-ADR was the right weight for two
instrument corrections; the cost of a meta pass immediately after two
product passes; and the standing question of whether **three** consecutive
evidence files reporting their own pass's cost low means the loop should
simply stop quoting it.

Also carry the one that is now overdue: **`cb-play` is three modes in one
binary.** CB-EV-0009 §5 named the third mode as the trigger to reconsider
the shape, CB-EV-0010 §6 recorded that it had arrived and that the pass
did not act. Either do it or state why not, and stop carrying it.

**Done 2026-08-02.**
[CB-EV-0011](../evidence/CB-EV-0011-instrument-corrections.md).

- **The self-quoting rule is proposed.** Three passes have now reported
  their own cost mid-flight and all three were low by ~2/3 — and this
  time it changed a conclusion: CB-EV-0010's tier comparison said L cost
  *24% more* than S; the final figures say **51%**.
- **`cb-play`'s third mode never arrived.** `--serve` is a *play flag*
  selecting which `Policy` fills a human seat. The binary still has two
  modes. CB-EV-0010 §6 was wrong; the item is **closed, not deferred**.
- **A near-miss recorded**: an instrument reading that looked impossible
  turned out to be correct, and one independent re-derivation before
  writing anything stopped a false accusation.
- **Tier M was the right weight** — one merged document, no separate
  review, and the pass still found a defect deeper than the one it was
  declared on, because the finding came from measuring rather than from
  an extra review round.