tegwick
0e394ccb3f
Tier M — structural S by the letter of InnerLoop's table (no capability
port, no canonical interface, no new dependency), declared M because the
table's triggers only cover code capabilities and this decides whether the
loop constrains its own operation. Chaos d4=2, no override. Survey and ADR
merged per tier M.
The gap is not the one CB-WP-0006 assumed. SH-1/2/3 ARE measured —
cb-cost emits all three every run. The defect is that SH-1 and SH-2 are
cumulative means over every response ever recorded, so they cannot detect
a worsening trend: the history outvotes the present.
cumulative -> dfd0d6d 923 resp mean 206,952 p90 389,733 9.6%
CB-WP-0006 alone 156 resp mean 503,464 p90 607,105 0.0%
Read cumulatively, session shape looks stable and improving. Read by
window, the last pass ran at 2.4x the mean of all preceding history and
batched nothing at all — 146 tool calls in 146 responses.
This is the third shape of one disease. CB-WP-0005 found numbers that were
unmeasured; CB-WP-0006 found numbers measured but unasserted; these are
measured AND asserted and still cannot fire, because they are aggregated
over a window in which nothing can move them.
Decisions: window SH-1/SH-2 and keep cumulative as labelled history;
enforce with a live budget modelled on CB-01/CB-02, NOT `make all` —
failing the build on context would block committing, and committing is
what closes the attribution window and is the natural point to compact, so
a gate that blocks the remedy is a trap; targets do not move, since all
three are unmet by wide margins and adjusting them in the pass that first
enforces them is the retarget §Step 4 forbids; and SH-3 is tested by
batching deliberately rather than by argument.
Recorded and not fixed: InnerLoop's tier table triggers on code
properties, so this pass would be tier S despite deciding how the loop
governs itself. Changing the taxonomy on one instance is the
invention-in-isolation INTENT warns against — the same argument used to
amend K14 and to leave v1.4 alone. A second instance should trigger it.
CB-WP-0007 executes in 6 tasks. T04's third outcome is the interesting
one: if batching stays below 5% despite trying, 20% is the number to
challenge with evidence, in an ADR rather than a quiet edit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7.8 KiB
CB-RES-0005: the session-shape metrics measure the wrong window
capability: meta.loop.session-shape
status: draft
tier: M (structural S by the letter of InnerLoop's table — no capability
port, no canonical interface, no new dependency — declared M because
the table's triggers only cover code capabilities and this changes
whether the loop constrains its own operation. The taxonomy gap is itself
a finding, §6. Chaos d4=2 → no override.)
instrument: make cost-mix / cb-cost --since (session shape block)
baseline: our own SH-1/SH-2/SH-3 targets, specs/SessionShape.md §6
Survey and ADR merged, per tier M. The decision is §5.
1. The finding
specs/SessionShape.md has stated SS-01…SS-05 and SH-1/2/3 since
CB-WP-0003. None has ever been enforced. They are the only
acceptance-adjacent numbers in this project with no gate at all.
But the gap is not the one CB-WP-0006 assumed. These metrics are
measured — cb-cost emits all three on every run. The defect is worse and
more interesting:
SH-1 and SH-2 are cumulative means over every response ever recorded. They cannot detect a worsening trend, because the history outvotes the present.
Measured on the same instrument, cumulatively and per window:
| window | responses | SH-1 mean context | SH-2 p90 | SH-3 batching |
|---|---|---|---|---|
cumulative → fc76445 |
346 | 232,982 | 492,042 | 7.8% |
cumulative → 578dcbe |
662 | 206,052 | 437,688 | 8.6% |
cumulative → dfd0d6d |
923 | 206,952 | 389,733 | 9.6% |
| CB-WP-0006 alone | 156 | 503,464 | 607,105 | 0.0% |
Read the cumulative column alone and session shape looks stable and slowly improving — mean context down 232k → 207k, p90 down 492k → 390k. Read the window and the most recent pass ran at 2.4× the mean and 1.6× the p90 of the whole preceding history.
Both numbers come from the same instrument on the same data. The
cumulative one is not wrong; it answers a question nobody is asking.
A ceiling on a running average over 923 responses can absorb 156
responses at 500k and still print 206,952.
This is a new shape of the same disease the last two passes chased: not an unmeasured number, and not an unasserted one, but a number aggregated so that it cannot fire.
2. SH-3 is zero, and that is a fact about behaviour
146 tool calls in 146 responses across CB-WP-0006. Zero batched. The preceding window was also 0.0%. Cumulatively 9.6%, and falling, against a 20% target.
Unlike SH-1/SH-2 this needs no new aggregation to be alarming — it is
already zero. It is also the cheapest of the three to move and, per
SessionShape.md §4, the least valuable: the ceiling is $2–4 on a $93
pass. Worth stating both halves, because a metric that is easy to move and
worth little is exactly the one a pass will optimize to look productive.
3. What the state of the art does
Cited, not measured; caps any comparison at parity per
MetricsAndScenarios §3.
| practice | window | enforcement |
|---|---|---|
| SLO error budgets (Google SRE) | rolling window, explicitly not cumulative | budget burn alerts, not build failure |
| Criterion regression detection | last run vs this run | fails on >N% regression |
cargo build --timings |
per build | reports, never gates |
| Our own CB-01/CB-02 (cost budget) | since the last commit | soft warn / hard stop, surfaced live |
The strongest precedent is the one already in this repo. CB-01/CB-02
measures spend since the last commit, warns at $10, stops at $22, and
is surfaced by make status and make cost-budget — not by make all.
It has fired usefully and has never blocked a commit.
Provenance: adapted:sre-error-budget — a rolling window with soft and
hard thresholds, adapted by making the window since the last commit
rather than time-based, because commits are what this loop can act on.
4. Why a build gate is the wrong instrument here
Stating this because it is the obvious move and it is wrong.
Failing make all on context size would block committing. Committing
is what closes the cost-attribution window and is the natural point to
compact. A gate that blocks the remedy when the metric is breached is not
a gate, it is a trap.
Worse, it punishes the wrong artifact. SH-* measure agent behaviour,
not the code under test. A red make all says "this tree is broken"; a
long session does not break the tree. Every other gate in this project
fails the build because the committed artifact is wrong.
There is also a limit SessionShape.md §5 states and this survey will not
paper over: there is no claim about quality. Compaction discards
detail; every number says it is cheaper, none says the work is as good.
Driving context down under a gate could trade cost for quality silently,
and this loop cannot currently detect that. CB-WP-0006 is weak evidence
against the worry — it ran at 503k, uncompacted, and produced 9/9 tasks
and six gate-caught defects — but "the pass that broke the ceiling went
fine" is not an argument for raising ceilings.
5. Decision
Adopted, merged into this document per tier M.
D1 — SH-1 and SH-2 become windowed. cb-cost reports session shape
for the window under measurement, and the cumulative figure is retained
and labelled as history, not as the metric. A cumulative mean stays
useful for the long view and stops being the thing a target compares to.
D2 — enforcement is a live budget, not a build gate, modelled on
CB-01/CB-02: soft and hard thresholds on the window since the last commit,
surfaced by make status and a new make shape-budget. Not in
make all. The soft threshold is the existing SH-1/SH-2 targets; the hard
threshold is stated in the workplan and set before the next measurement,
not after.
D3 — the targets do not move. SH-1 ≤ 200,000, SH-2 ≤ 300,000, SH-3 ≥ 20% stand as written. Every one is currently unmet, badly, and adjusting them in the pass that first enforces them is precisely the retarget InnerLoop §Step 4 forbids. If 20% batching proves unreachable, that is a finding to publish, not a number to lower.
D4 — SH-3 is tested by doing, not by argument. The pass that implements this should batch deliberately and report what the rate reaches. A target nobody has tried to hit is not evidence of anything.
6. A finding about the loop itself
InnerLoop's tier table triggers on code properties: "creates a new capability port", "touches a canonical interface", "adds a dependency". This pass does none of those and would be tier S — one provenance paragraph in a commit message — despite deciding whether and how the loop constrains its own operation, with a stated and unmeasured quality trade-off.
That is a gap, not a reason to inflate the tier by feel. It is recorded here rather than fixed, because changing the tier taxonomy on the evidence of one pass is the invention-in-isolation INTENT warns against — the same argument used to amend K14 and to leave InnerLoop v1.4 alone. A second instance should trigger the change.
7. Prediction
Stated per task, as a mechanism, with the alternative named — the form that collapsed prediction error in CB-WP-0006.
| claim | mechanism | refuted if |
|---|---|---|
| windowed SH-1/SH-2 differ sharply from cumulative | the window is computed over the same data with a different denominator | the windowed figure lands within 20% of cumulative |
| the budget fires on a real pass | soft threshold at the existing target, checked since last commit | it never reports a breach across a full pass |
| SH-3 is movable by intent | batch deliberately and measure | the rate stays below 5% despite trying, in which case 20% is the number to challenge, with evidence |
| gating does not cost quality | same gates, same findings rate | fewer defects surface, or work is repeated after compaction |