CB-RES-0005 + CB-WP-0007: session shape measures the wrong window

Tier M — structural S by the letter of InnerLoop's table (no capability
port, no canonical interface, no new dependency), declared M because the
table's triggers only cover code capabilities and this decides whether the
loop constrains its own operation. Chaos d4=2, no override. Survey and ADR
merged per tier M.

The gap is not the one CB-WP-0006 assumed. SH-1/2/3 ARE measured —
cb-cost emits all three every run. The defect is that SH-1 and SH-2 are
cumulative means over every response ever recorded, so they cannot detect
a worsening trend: the history outvotes the present.

  cumulative -> dfd0d6d   923 resp   mean 206,952   p90 389,733   9.6%
  CB-WP-0006 alone        156 resp   mean 503,464   p90 607,105   0.0%

Read cumulatively, session shape looks stable and improving. Read by
window, the last pass ran at 2.4x the mean of all preceding history and
batched nothing at all — 146 tool calls in 146 responses.

This is the third shape of one disease. CB-WP-0005 found numbers that were
unmeasured; CB-WP-0006 found numbers measured but unasserted; these are
measured AND asserted and still cannot fire, because they are aggregated
over a window in which nothing can move them.

Decisions: window SH-1/SH-2 and keep cumulative as labelled history;
enforce with a live budget modelled on CB-01/CB-02, NOT `make all` —
failing the build on context would block committing, and committing is
what closes the attribution window and is the natural point to compact, so
a gate that blocks the remedy is a trap; targets do not move, since all
three are unmet by wide margins and adjusting them in the pass that first
enforces them is the retarget §Step 4 forbids; and SH-3 is tested by
batching deliberately rather than by argument.

Recorded and not fixed: InnerLoop's tier table triggers on code
properties, so this pass would be tier S despite deciding how the loop
governs itself. Changing the taxonomy on one instance is the
invention-in-isolation INTENT warns against — the same argument used to
amend K14 and to leave v1.4 alone. A second instance should trigger it.

CB-WP-0007 executes in 6 tasks. T04's third outcome is the interesting
one: if batching stays below 5% despite trying, 20% is the number to
challenge with evidence, in an ADR rather than a quiet edit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

research/CB-RES-0005-session-shape.md

@@ -0,0 +1,161 @@
+# CB-RES-0005: the session-shape metrics measure the wrong window
+
+capability: meta.loop.session-shape
+status: draft
+tier: M (structural S by the letter of InnerLoop's table — no capability
+port, no canonical interface, no new dependency — declared **M** because
+the table's triggers only cover *code* capabilities and this changes
+whether the loop constrains its own operation. The taxonomy gap is itself
+a finding, §6. Chaos d4=2 → no override.)
+instrument: `make cost-mix` / `cb-cost --since` (session shape block)
+baseline: our own SH-1/SH-2/SH-3 targets, `specs/SessionShape.md` §6
+
+**Survey and ADR merged, per tier M.** The decision is §5.
+
+---
+
+## 1. The finding
+
+`specs/SessionShape.md` has stated SS-01…SS-05 and SH-1/2/3 since
+CB-WP-0003. **None has ever been enforced.** They are the only
+acceptance-adjacent numbers in this project with no gate at all.
+
+But the gap is not the one CB-WP-0006 assumed. These metrics *are*
+measured — `cb-cost` emits all three on every run. The defect is worse and
+more interesting:
+
+> **SH-1 and SH-2 are cumulative means over every response ever recorded.
+> They cannot detect a worsening trend, because the history outvotes the
+> present.**
+
+Measured on the same instrument, cumulatively and per window:
+
+| window | responses | SH-1 mean context | SH-2 p90 | SH-3 batching |
+|---|---|---|---|---|
+| cumulative → `fc76445` | 346 | 232,982 | 492,042 | 7.8% |
+| cumulative → `578dcbe` | 662 | 206,052 | 437,688 | 8.6% |
+| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
+| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |
+
+Read the cumulative column alone and session shape looks **stable and
+slowly improving** — mean context down 232k → 207k, p90 down 492k → 390k.
+Read the window and the most recent pass ran at **2.4× the mean** and
+**1.6× the p90** of the whole preceding history.
+
+Both numbers come from the same instrument on the same data. The
+cumulative one is not wrong; it answers a question nobody is asking.
+A ceiling on a running average over 923 responses can absorb 156
+responses at 500k and still print `206,952`.
+
+This is a **new shape of the same disease** the last two passes chased:
+not an unmeasured number, and not an unasserted one, but a number
+**aggregated so that it cannot fire**.
+
+## 2. SH-3 is zero, and that is a fact about behaviour
+
+**146 tool calls in 146 responses across CB-WP-0006. Zero batched.** The
+preceding window was also 0.0%. Cumulatively 9.6%, and falling, against a
+20% target.
+
+Unlike SH-1/SH-2 this needs no new aggregation to be alarming — it is
+already zero. It is also the cheapest of the three to move and, per
+`SessionShape.md` §4, the least valuable: the ceiling is $2–4 on a $93
+pass. Worth stating both halves, because a metric that is easy to move and
+worth little is exactly the one a pass will optimize to look productive.
+
+## 3. What the state of the art does
+
+Cited, not measured; caps any comparison at `parity` per
+MetricsAndScenarios §3.
+
+| practice | window | enforcement |
+|---|---|---|
+| **SLO error budgets** (Google SRE) | rolling window, explicitly *not* cumulative | budget burn alerts, not build failure |
+| **Criterion regression detection** | last run vs this run | fails on >N% regression |
+| **`cargo build --timings`** | per build | reports, never gates |
+| **Our own CB-01/CB-02** (cost budget) | **since the last commit** | soft warn / hard stop, surfaced live |
+
+The strongest precedent is the one already in this repo. CB-01/CB-02
+measures spend **since the last commit**, warns at $10, stops at $22, and
+is surfaced by `make status` and `make cost-budget` — not by `make all`.
+It has fired usefully and has never blocked a commit.
+
+**Provenance: `adapted:sre-error-budget`** — a rolling window with soft and
+hard thresholds, adapted by making the window *since the last commit*
+rather than time-based, because commits are what this loop can act on.
+
+## 4. Why a build gate is the wrong instrument here
+
+Stating this because it is the obvious move and it is wrong.
+
+Failing `make all` on context size would **block committing**. Committing
+is what closes the cost-attribution window and is the natural point to
+compact. A gate that blocks the remedy when the metric is breached is not
+a gate, it is a trap.
+
+Worse, it punishes the wrong artifact. SH-* measure **agent behaviour**,
+not the code under test. A red `make all` says "this tree is broken"; a
+long session does not break the tree. Every other gate in this project
+fails the build because the *committed artifact* is wrong.
+
+There is also a limit `SessionShape.md` §5 states and this survey will not
+paper over: **there is no claim about quality.** Compaction discards
+detail; every number says it is cheaper, none says the work is as good.
+Driving context down under a gate could trade cost for quality silently,
+and this loop cannot currently detect that. CB-WP-0006 is weak evidence
+against the worry — it ran at 503k, uncompacted, and produced 9/9 tasks
+and six gate-caught defects — but "the pass that broke the ceiling went
+fine" is not an argument for raising ceilings.
+
+## 5. Decision
+
+**Adopted, merged into this document per tier M.**
+
+**D1 — SH-1 and SH-2 become windowed.** `cb-cost` reports session shape
+for the window under measurement, and the cumulative figure is retained
+and labelled as *history*, not as the metric. A cumulative mean stays
+useful for the long view and stops being the thing a target compares to.
+
+**D2 — enforcement is a live budget, not a build gate**, modelled on
+CB-01/CB-02: soft and hard thresholds on the window since the last commit,
+surfaced by `make status` and a new `make shape-budget`. **Not** in
+`make all`. The soft threshold is the existing SH-1/SH-2 targets; the hard
+threshold is stated in the workplan and set *before* the next measurement,
+not after.
+
+**D3 — the targets do not move.** SH-1 ≤ 200,000, SH-2 ≤ 300,000,
+SH-3 ≥ 20% stand as written. Every one is currently unmet, badly, and
+adjusting them in the pass that first enforces them is precisely the
+retarget InnerLoop §Step 4 forbids. If 20% batching proves unreachable,
+that is a finding to publish, not a number to lower.
+
+**D4 — SH-3 is tested by doing, not by argument.** The pass that
+implements this should batch deliberately and report what the rate
+reaches. A target nobody has tried to hit is not evidence of anything.
+
+## 6. A finding about the loop itself
+
+InnerLoop's tier table triggers on **code** properties: "creates a new
+capability port", "touches a canonical interface", "adds a dependency".
+This pass does none of those and would be **tier S** — one provenance
+paragraph in a commit message — despite deciding whether and how the loop
+constrains its own operation, with a stated and unmeasured quality
+trade-off.
+
+That is a gap, not a reason to inflate the tier by feel. It is recorded
+here rather than fixed, because changing the tier taxonomy on the evidence
+of one pass is the invention-in-isolation INTENT warns against — the same
+argument used to amend K14 and to leave InnerLoop v1.4 alone. **A second
+instance should trigger the change.**
+
+## 7. Prediction
+
+Stated per task, as a mechanism, with the alternative named — the form
+that collapsed prediction error in CB-WP-0006.
+
+| claim | mechanism | refuted if |
+|---|---|---|
+| windowed SH-1/SH-2 differ sharply from cumulative | the window is computed over the same data with a different denominator | the windowed figure lands within 20% of cumulative |
+| the budget fires on a real pass | soft threshold at the existing target, checked since last commit | it never reports a breach across a full pass |
+| SH-3 is movable by intent | batch deliberately and measure | the rate stays below 5% despite trying, in which case **20% is the number to challenge, with evidence** |
+| gating does not cost quality | same gates, same findings rate | fewer defects surface, or work is repeated after compaction |

workplans/CB-WP-0007-session-shape.md

@@ -0,0 +1,186 @@
+---
+id: CB-WP-0007
+title: "Make session shape measurable in the window that matters, then enforce it"
+status: proposed
+---
+
+# Purpose
+
+[CB-RES-0005](../research/CB-RES-0005-session-shape.md) (tier M, survey
+and ADR merged; chaos d4=2, no override):
+
+> **SH-1 and SH-2 are cumulative means over every response ever recorded.
+> They cannot detect a worsening trend, because the history outvotes the
+> present.**
+
+| window | responses | SH-1 mean | SH-2 p90 | SH-3 batching |
+|---|---|---|---|---|
+| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
+| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |
+
+Read cumulatively, session shape looks stable and improving. Read by
+window, the last pass ran at **2.4× the mean** of all preceding history
+and batched **nothing at all** — 146 tool calls in 146 responses.
+
+This is the third shape of one disease. CB-WP-0005 found numbers that were
+**unmeasured**; CB-WP-0006 found numbers that were measured but
+**unasserted**; this pass has numbers that are measured *and* asserted and
+still cannot fire, because they are **aggregated over a window in which
+nothing can move them**.
+
+Per InnerLoop §Step 4, no target moves in the commit that measures it —
+and CB-RES-0005 D3 states that up front, because all three targets are
+currently unmet by wide margins and the temptation is obvious.
+
+## Phase A — measure the right window
+
+## Task: window the session-shape metrics
+
+```task
+id: CB-WP-0007-T01
+status: todo
+priority: high
+```
+
+`cb-cost`'s session-shape block reports over every response in scope. With
+`--since` it already windows correctly; without it, it reports a
+cumulative mean that a bad pass cannot move.
+
+Deliver: the session-shape block **states its window explicitly** and, when
+run without `--since`, reports **both** — the window since the last commit
+*and* the cumulative history, labelled so they cannot be confused. The
+cumulative figure stays useful for the long view and stops being what a
+target compares against.
+
+**Predicted:** the windowed figure differs sharply from cumulative.
+**Refuted if** it lands within 20% of the cumulative number, in which case
+the aggregation was not the problem and this pass should re-plan.
+
+## Task: SH-4 — a windowed trend the instrument can see
+
+```task
+id: CB-WP-0007-T02
+status: todo
+priority: medium
+```
+
+A ceiling catches a bad pass. It does not catch a slow slide, which is how
+SH-1 got from 232k to 503k without any single pass looking anomalous.
+
+Deliver a reported (not gated) SH-4: **mean context of the current window
+against the previous window**, so a rising trend is visible before it
+breaches. Provenance `adapted:criterion` — the same shape as the >3%
+merge-blocking regression rule against our own last evidence file, but
+reported rather than blocking, because CB-RES-0005 §4 argues a build gate
+is the wrong instrument for behaviour.
+
+State the limit in the output, as every other gate here does: two windows
+is not a trend, and the first run has nothing to compare against.
+
+## Phase B — enforce it where enforcement belongs
+
+## Task: `make shape-budget` — a live budget, not a build gate
+
+```task
+id: CB-WP-0007-T03
+status: todo
+priority: high
+```
+
+Per CB-RES-0005 D2, modelled on CB-01/CB-02, which measures spend since
+the last commit, warns at $10, stops at $22, and is surfaced by
+`make status` — never by `make all`.
+
+**Why not a build gate, restated so it is not quietly reversed later:**
+failing `make all` on context size would **block committing**, and
+committing is what closes the attribution window and is the natural point
+to compact. A gate that blocks the remedy when the metric breaches is a
+trap, not a gate. It also punishes the wrong artifact — a long session
+does not break the tree.
+
+Deliver `make shape-budget` and a line in `make status`:
+
+| threshold | value | action |
+|---|---|---|
+| SH-1 soft | 200,000 mean context in the window | warn |
+| SH-1 hard | **300,000** | report loudly: compact before continuing |
+| SH-2 soft | 300,000 p90 | warn |
+| SH-3 floor | 20% batching | warn |
+
+**The hard threshold is set here, before the next measurement**, per
+§Step 4. It is 1.5× the soft target, matching CB-02's 2.2× ratio on spend
+scaled down because context is bounded by the model and spend is not.
+
+Carries `--self-test`. Its positive control is the one this project keeps
+needing: a budget that reports `ok` because it measured nothing must
+abort instead.
+
+## Task: batch deliberately, and report what the rate reaches
+
+```task
+id: CB-WP-0007-T04
+status: todo
+priority: medium
+```
+
+SH-3 is **0.0%** across two consecutive passes. The target is 20%. Nobody
+has ever tried to hit it, so its reachability is unknown and the number is
+currently unfalsified rather than wrong.
+
+This task is the trial: batch independent tool calls deliberately through
+the pass and report the rate the instrument measures.
+
+**Three outcomes, all publishable, and the third is the interesting one:**
+
+1. **≥20%** — the target is reachable; it stops being aspirational.
+2. **5–20%** — partially reachable; report the ceiling and what blocked it.
+3. **<5% despite trying** — then **20% is the number to challenge, with
+   evidence**, and the challenge belongs in an ADR rather than a quiet
+   edit. Per §Step 4 the target does not move in this pass regardless.
+
+## Phase C — measure, and say what it cost
+
+## Task: control loop — did windowing change what fires?
+
+```task
+id: CB-WP-0007-T05
+status: todo
+priority: high
+```
+
+Commit `evidence/CB-EV-0006-session-shape.md`. Four tests, all reported:
+
+1. **Does the windowed metric differ from cumulative?** Against T01's
+   prediction; refuted if within 20%.
+2. **Did the budget fire on this pass?** A budget that never reports a
+   breach across a full pass is either miscalibrated or measuring nothing —
+   both are findings.
+3. **What did SH-3 reach?** T04's three outcomes.
+4. **Did quality hold?** `make all` green, and the honest question this
+   pass cannot dodge: **did compacting to satisfy a context budget cause
+   work to be repeated or detail to be lost?** `SessionShape.md` §5 says
+   the loop cannot detect this. Record the judgment explicitly rather than
+   letting the cost number imply it.
+
+## Task: retrospective
+
+```task
+id: CB-WP-0007-T06
+status: todo
+priority: medium
+```
+
+The question: **three passes have now found three different ways a number
+can exist and not bind — unmeasured, unasserted, and aggregated so it
+cannot fire. Is there a fourth, and is the loop discovering a pattern or
+enumerating an infinite set?**
+
+CB-WP-0003 concluded that class-by-class hardening does not converge.
+Three passes later, the *instrument* failures have their own taxonomy and
+it is also not converging. Say plainly whether that changes the design
+goal, or confirms it.
+
+Also due: **the tier-taxonomy gap** CB-RES-0005 §6 recorded but did not
+fix. This pass is the first instance. Per the standing argument used for
+K14 and InnerLoop v1.4, a second instance triggers the change — say
+whether this counts as one or two.