clay-borg/research/CB-RES-0005-session-shape.md

# CB-RES-0005: the session-shape metrics measure the wrong window

capability: meta.loop.session-shape
status: draft
tier: M (structural S by the letter of InnerLoop's table — no capability
port, no canonical interface, no new dependency — declared **M** because
the table's triggers only cover *code* capabilities and this changes
whether the loop constrains its own operation. The taxonomy gap is itself
a finding, §6. Chaos d4=2 → no override.)
instrument: `make cost-mix` / `cb-cost --since` (session shape block)
baseline: our own SH-1/SH-2/SH-3 targets, `specs/SessionShape.md` §6

**Survey and ADR merged, per tier M.** The decision is §5.

---

## 1. The finding

`specs/SessionShape.md` has stated SS-01…SS-05 and SH-1/2/3 since
CB-WP-0003. **None has ever been enforced.** They are the only
acceptance-adjacent numbers in this project with no gate at all.

But the gap is not the one CB-WP-0006 assumed. These metrics *are*
measured — `cb-cost` emits all three on every run. The defect is worse and
more interesting:

> **SH-1 and SH-2 are cumulative means over every response ever recorded.
> They cannot detect a worsening trend, because the history outvotes the
> present.**

Measured on the same instrument, cumulatively and per window:

| window | responses | SH-1 mean context | SH-2 p90 | SH-3 batching |
|---|---|---|---|---|
| cumulative → `fc76445` | 346 | 232,982 | 492,042 | 7.8% |
| cumulative → `578dcbe` | 662 | 206,052 | 437,688 | 8.6% |
| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |

Read the cumulative column alone and session shape looks **stable and
slowly improving** — mean context down 232k → 207k, p90 down 492k → 390k.
Read the window and the most recent pass ran at **2.4× the mean** and
**1.6× the p90** of the whole preceding history.

Both numbers come from the same instrument on the same data. The
cumulative one is not wrong; it answers a question nobody is asking.
A ceiling on a running average over 923 responses can absorb 156
responses at 500k and still print `206,952`.

This is a **new shape of the same disease** the last two passes chased:
not an unmeasured number, and not an unasserted one, but a number
**aggregated so that it cannot fire**.

## 2. SH-3 is zero, and that is a fact about behaviour

**146 tool calls in 146 responses across CB-WP-0006. Zero batched.** The
preceding window was also 0.0%. Cumulatively 9.6%, and falling, against a
20% target.

Unlike SH-1/SH-2 this needs no new aggregation to be alarming — it is
already zero. It is also the cheapest of the three to move and, per
`SessionShape.md` §4, the least valuable: the ceiling is $2–4 on a $93
pass. Worth stating both halves, because a metric that is easy to move and
worth little is exactly the one a pass will optimize to look productive.

## 3. What the state of the art does

Cited, not measured; caps any comparison at `parity` per
MetricsAndScenarios §3.

| practice | window | enforcement |
|---|---|---|
| **SLO error budgets** (Google SRE) | rolling window, explicitly *not* cumulative | budget burn alerts, not build failure |
| **Criterion regression detection** | last run vs this run | fails on >N% regression |
| **`cargo build --timings`** | per build | reports, never gates |
| **Our own CB-01/CB-02** (cost budget) | **since the last commit** | soft warn / hard stop, surfaced live |

The strongest precedent is the one already in this repo. CB-01/CB-02
measures spend **since the last commit**, warns at $10, stops at $22, and
is surfaced by `make status` and `make cost-budget` — not by `make all`.
It has fired usefully and has never blocked a commit.

**Provenance: `adapted:sre-error-budget`** — a rolling window with soft and
hard thresholds, adapted by making the window *since the last commit*
rather than time-based, because commits are what this loop can act on.

## 4. Why a build gate is the wrong instrument here

Stating this because it is the obvious move and it is wrong.

Failing `make all` on context size would **block committing**. Committing
is what closes the cost-attribution window and is the natural point to
compact. A gate that blocks the remedy when the metric is breached is not
a gate, it is a trap.

Worse, it punishes the wrong artifact. SH-* measure **agent behaviour**,
not the code under test. A red `make all` says "this tree is broken"; a
long session does not break the tree. Every other gate in this project
fails the build because the *committed artifact* is wrong.

There is also a limit `SessionShape.md` §5 states and this survey will not
paper over: **there is no claim about quality.** Compaction discards
detail; every number says it is cheaper, none says the work is as good.
Driving context down under a gate could trade cost for quality silently,
and this loop cannot currently detect that. CB-WP-0006 is weak evidence
against the worry — it ran at 503k, uncompacted, and produced 9/9 tasks
and six gate-caught defects — but "the pass that broke the ceiling went
fine" is not an argument for raising ceilings.

## 5. Decision

**Adopted, merged into this document per tier M.**

**D1 — SH-1 and SH-2 become windowed.** `cb-cost` reports session shape
for the window under measurement, and the cumulative figure is retained
and labelled as *history*, not as the metric. A cumulative mean stays
useful for the long view and stops being the thing a target compares to.

**D2 — enforcement is a live budget, not a build gate**, modelled on
CB-01/CB-02: soft and hard thresholds on the window since the last commit,
surfaced by `make status` and a new `make shape-budget`. **Not** in
`make all`. The soft threshold is the existing SH-1/SH-2 targets; the hard
threshold is stated in the workplan and set *before* the next measurement,
not after.

**D3 — the targets do not move.** SH-1 ≤ 200,000, SH-2 ≤ 300,000,
SH-3 ≥ 20% stand as written. Every one is currently unmet, badly, and
adjusting them in the pass that first enforces them is precisely the
retarget InnerLoop §Step 4 forbids. If 20% batching proves unreachable,
that is a finding to publish, not a number to lower.

**D4 — SH-3 is tested by doing, not by argument.** The pass that
implements this should batch deliberately and report what the rate
reaches. A target nobody has tried to hit is not evidence of anything.

## 6. A finding about the loop itself

InnerLoop's tier table triggers on **code** properties: "creates a new
capability port", "touches a canonical interface", "adds a dependency".
This pass does none of those and would be **tier S** — one provenance
paragraph in a commit message — despite deciding whether and how the loop
constrains its own operation, with a stated and unmeasured quality
trade-off.

That is a gap, not a reason to inflate the tier by feel. It is recorded
here rather than fixed, because changing the tier taxonomy on the evidence
of one pass is the invention-in-isolation INTENT warns against — the same
argument used to amend K14 and to leave InnerLoop v1.4 alone. **A second
instance should trigger the change.**

## 7. Prediction

Stated per task, as a mechanism, with the alternative named — the form
that collapsed prediction error in CB-WP-0006.

| claim | mechanism | refuted if |
|---|---|---|
| windowed SH-1/SH-2 differ sharply from cumulative | the window is computed over the same data with a different denominator | the windowed figure lands within 20% of cumulative |
| the budget fires on a real pass | soft threshold at the existing target, checked since last commit | it never reports a breach across a full pass |
| SH-3 is movable by intent | batch deliberately and measure | the rate stays below 5% despite trying, in which case **20% is the number to challenge, with evidence** |
| gating does not cost quality | same gates, same findings rate | fewer defects surface, or work is repeated after compaction |