tegwick
0e394ccb3f
Some checks failed
ci / check (push) Failing after 4s
Tier M — structural S by the letter of InnerLoop's table (no capability
port, no canonical interface, no new dependency), declared M because the
table's triggers only cover code capabilities and this decides whether the
loop constrains its own operation. Chaos d4=2, no override. Survey and ADR
merged per tier M.
The gap is not the one CB-WP-0006 assumed. SH-1/2/3 ARE measured —
cb-cost emits all three every run. The defect is that SH-1 and SH-2 are
cumulative means over every response ever recorded, so they cannot detect
a worsening trend: the history outvotes the present.
cumulative -> dfd0d6d 923 resp mean 206,952 p90 389,733 9.6%
CB-WP-0006 alone 156 resp mean 503,464 p90 607,105 0.0%
Read cumulatively, session shape looks stable and improving. Read by
window, the last pass ran at 2.4x the mean of all preceding history and
batched nothing at all — 146 tool calls in 146 responses.
This is the third shape of one disease. CB-WP-0005 found numbers that were
unmeasured; CB-WP-0006 found numbers measured but unasserted; these are
measured AND asserted and still cannot fire, because they are aggregated
over a window in which nothing can move them.
Decisions: window SH-1/SH-2 and keep cumulative as labelled history;
enforce with a live budget modelled on CB-01/CB-02, NOT `make all` —
failing the build on context would block committing, and committing is
what closes the attribution window and is the natural point to compact, so
a gate that blocks the remedy is a trap; targets do not move, since all
three are unmet by wide margins and adjusting them in the pass that first
enforces them is the retarget §Step 4 forbids; and SH-3 is tested by
batching deliberately rather than by argument.
Recorded and not fixed: InnerLoop's tier table triggers on code
properties, so this pass would be tier S despite deciding how the loop
governs itself. Changing the taxonomy on one instance is the
invention-in-isolation INTENT warns against — the same argument used to
amend K14 and to leave v1.4 alone. A second instance should trigger it.
CB-WP-0007 executes in 6 tasks. T04's third outcome is the interesting
one: if batching stays below 5% despite trying, 20% is the number to
challenge with evidence, in an ADR rather than a quiet edit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
161 lines
7.8 KiB
Markdown
161 lines
7.8 KiB
Markdown
# CB-RES-0005: the session-shape metrics measure the wrong window
|
||
|
||
capability: meta.loop.session-shape
|
||
status: draft
|
||
tier: M (structural S by the letter of InnerLoop's table — no capability
|
||
port, no canonical interface, no new dependency — declared **M** because
|
||
the table's triggers only cover *code* capabilities and this changes
|
||
whether the loop constrains its own operation. The taxonomy gap is itself
|
||
a finding, §6. Chaos d4=2 → no override.)
|
||
instrument: `make cost-mix` / `cb-cost --since` (session shape block)
|
||
baseline: our own SH-1/SH-2/SH-3 targets, `specs/SessionShape.md` §6
|
||
|
||
**Survey and ADR merged, per tier M.** The decision is §5.
|
||
|
||
---
|
||
|
||
## 1. The finding
|
||
|
||
`specs/SessionShape.md` has stated SS-01…SS-05 and SH-1/2/3 since
|
||
CB-WP-0003. **None has ever been enforced.** They are the only
|
||
acceptance-adjacent numbers in this project with no gate at all.
|
||
|
||
But the gap is not the one CB-WP-0006 assumed. These metrics *are*
|
||
measured — `cb-cost` emits all three on every run. The defect is worse and
|
||
more interesting:
|
||
|
||
> **SH-1 and SH-2 are cumulative means over every response ever recorded.
|
||
> They cannot detect a worsening trend, because the history outvotes the
|
||
> present.**
|
||
|
||
Measured on the same instrument, cumulatively and per window:
|
||
|
||
| window | responses | SH-1 mean context | SH-2 p90 | SH-3 batching |
|
||
|---|---|---|---|---|
|
||
| cumulative → `fc76445` | 346 | 232,982 | 492,042 | 7.8% |
|
||
| cumulative → `578dcbe` | 662 | 206,052 | 437,688 | 8.6% |
|
||
| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
|
||
| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |
|
||
|
||
Read the cumulative column alone and session shape looks **stable and
|
||
slowly improving** — mean context down 232k → 207k, p90 down 492k → 390k.
|
||
Read the window and the most recent pass ran at **2.4× the mean** and
|
||
**1.6× the p90** of the whole preceding history.
|
||
|
||
Both numbers come from the same instrument on the same data. The
|
||
cumulative one is not wrong; it answers a question nobody is asking.
|
||
A ceiling on a running average over 923 responses can absorb 156
|
||
responses at 500k and still print `206,952`.
|
||
|
||
This is a **new shape of the same disease** the last two passes chased:
|
||
not an unmeasured number, and not an unasserted one, but a number
|
||
**aggregated so that it cannot fire**.
|
||
|
||
## 2. SH-3 is zero, and that is a fact about behaviour
|
||
|
||
**146 tool calls in 146 responses across CB-WP-0006. Zero batched.** The
|
||
preceding window was also 0.0%. Cumulatively 9.6%, and falling, against a
|
||
20% target.
|
||
|
||
Unlike SH-1/SH-2 this needs no new aggregation to be alarming — it is
|
||
already zero. It is also the cheapest of the three to move and, per
|
||
`SessionShape.md` §4, the least valuable: the ceiling is $2–4 on a $93
|
||
pass. Worth stating both halves, because a metric that is easy to move and
|
||
worth little is exactly the one a pass will optimize to look productive.
|
||
|
||
## 3. What the state of the art does
|
||
|
||
Cited, not measured; caps any comparison at `parity` per
|
||
MetricsAndScenarios §3.
|
||
|
||
| practice | window | enforcement |
|
||
|---|---|---|
|
||
| **SLO error budgets** (Google SRE) | rolling window, explicitly *not* cumulative | budget burn alerts, not build failure |
|
||
| **Criterion regression detection** | last run vs this run | fails on >N% regression |
|
||
| **`cargo build --timings`** | per build | reports, never gates |
|
||
| **Our own CB-01/CB-02** (cost budget) | **since the last commit** | soft warn / hard stop, surfaced live |
|
||
|
||
The strongest precedent is the one already in this repo. CB-01/CB-02
|
||
measures spend **since the last commit**, warns at $10, stops at $22, and
|
||
is surfaced by `make status` and `make cost-budget` — not by `make all`.
|
||
It has fired usefully and has never blocked a commit.
|
||
|
||
**Provenance: `adapted:sre-error-budget`** — a rolling window with soft and
|
||
hard thresholds, adapted by making the window *since the last commit*
|
||
rather than time-based, because commits are what this loop can act on.
|
||
|
||
## 4. Why a build gate is the wrong instrument here
|
||
|
||
Stating this because it is the obvious move and it is wrong.
|
||
|
||
Failing `make all` on context size would **block committing**. Committing
|
||
is what closes the cost-attribution window and is the natural point to
|
||
compact. A gate that blocks the remedy when the metric is breached is not
|
||
a gate, it is a trap.
|
||
|
||
Worse, it punishes the wrong artifact. SH-* measure **agent behaviour**,
|
||
not the code under test. A red `make all` says "this tree is broken"; a
|
||
long session does not break the tree. Every other gate in this project
|
||
fails the build because the *committed artifact* is wrong.
|
||
|
||
There is also a limit `SessionShape.md` §5 states and this survey will not
|
||
paper over: **there is no claim about quality.** Compaction discards
|
||
detail; every number says it is cheaper, none says the work is as good.
|
||
Driving context down under a gate could trade cost for quality silently,
|
||
and this loop cannot currently detect that. CB-WP-0006 is weak evidence
|
||
against the worry — it ran at 503k, uncompacted, and produced 9/9 tasks
|
||
and six gate-caught defects — but "the pass that broke the ceiling went
|
||
fine" is not an argument for raising ceilings.
|
||
|
||
## 5. Decision
|
||
|
||
**Adopted, merged into this document per tier M.**
|
||
|
||
**D1 — SH-1 and SH-2 become windowed.** `cb-cost` reports session shape
|
||
for the window under measurement, and the cumulative figure is retained
|
||
and labelled as *history*, not as the metric. A cumulative mean stays
|
||
useful for the long view and stops being the thing a target compares to.
|
||
|
||
**D2 — enforcement is a live budget, not a build gate**, modelled on
|
||
CB-01/CB-02: soft and hard thresholds on the window since the last commit,
|
||
surfaced by `make status` and a new `make shape-budget`. **Not** in
|
||
`make all`. The soft threshold is the existing SH-1/SH-2 targets; the hard
|
||
threshold is stated in the workplan and set *before* the next measurement,
|
||
not after.
|
||
|
||
**D3 — the targets do not move.** SH-1 ≤ 200,000, SH-2 ≤ 300,000,
|
||
SH-3 ≥ 20% stand as written. Every one is currently unmet, badly, and
|
||
adjusting them in the pass that first enforces them is precisely the
|
||
retarget InnerLoop §Step 4 forbids. If 20% batching proves unreachable,
|
||
that is a finding to publish, not a number to lower.
|
||
|
||
**D4 — SH-3 is tested by doing, not by argument.** The pass that
|
||
implements this should batch deliberately and report what the rate
|
||
reaches. A target nobody has tried to hit is not evidence of anything.
|
||
|
||
## 6. A finding about the loop itself
|
||
|
||
InnerLoop's tier table triggers on **code** properties: "creates a new
|
||
capability port", "touches a canonical interface", "adds a dependency".
|
||
This pass does none of those and would be **tier S** — one provenance
|
||
paragraph in a commit message — despite deciding whether and how the loop
|
||
constrains its own operation, with a stated and unmeasured quality
|
||
trade-off.
|
||
|
||
That is a gap, not a reason to inflate the tier by feel. It is recorded
|
||
here rather than fixed, because changing the tier taxonomy on the evidence
|
||
of one pass is the invention-in-isolation INTENT warns against — the same
|
||
argument used to amend K14 and to leave InnerLoop v1.4 alone. **A second
|
||
instance should trigger the change.**
|
||
|
||
## 7. Prediction
|
||
|
||
Stated per task, as a mechanism, with the alternative named — the form
|
||
that collapsed prediction error in CB-WP-0006.
|
||
|
||
| claim | mechanism | refuted if |
|
||
|---|---|---|
|
||
| windowed SH-1/SH-2 differ sharply from cumulative | the window is computed over the same data with a different denominator | the windowed figure lands within 20% of cumulative |
|
||
| the budget fires on a real pass | soft threshold at the existing target, checked since last commit | it never reports a breach across a full pass |
|
||
| SH-3 is movable by intent | batch deliberately and measure | the rate stays below 5% despite trying, in which case **20% is the number to challenge, with evidence** |
|
||
| gating does not cost quality | same gates, same findings rate | fewer defects surface, or work is repeated after compaction |
|