From 0e394ccb3fe26dfeba85db5ba034c7be9deb09fa Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 1 Aug 2026 13:49:19 +0200 Subject: [PATCH] CB-RES-0005 + CB-WP-0007: session shape measures the wrong window MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tier M — structural S by the letter of InnerLoop's table (no capability port, no canonical interface, no new dependency), declared M because the table's triggers only cover code capabilities and this decides whether the loop constrains its own operation. Chaos d4=2, no override. Survey and ADR merged per tier M. The gap is not the one CB-WP-0006 assumed. SH-1/2/3 ARE measured — cb-cost emits all three every run. The defect is that SH-1 and SH-2 are cumulative means over every response ever recorded, so they cannot detect a worsening trend: the history outvotes the present. cumulative -> dfd0d6d 923 resp mean 206,952 p90 389,733 9.6% CB-WP-0006 alone 156 resp mean 503,464 p90 607,105 0.0% Read cumulatively, session shape looks stable and improving. Read by window, the last pass ran at 2.4x the mean of all preceding history and batched nothing at all — 146 tool calls in 146 responses. This is the third shape of one disease. CB-WP-0005 found numbers that were unmeasured; CB-WP-0006 found numbers measured but unasserted; these are measured AND asserted and still cannot fire, because they are aggregated over a window in which nothing can move them. Decisions: window SH-1/SH-2 and keep cumulative as labelled history; enforce with a live budget modelled on CB-01/CB-02, NOT `make all` — failing the build on context would block committing, and committing is what closes the attribution window and is the natural point to compact, so a gate that blocks the remedy is a trap; targets do not move, since all three are unmet by wide margins and adjusting them in the pass that first enforces them is the retarget §Step 4 forbids; and SH-3 is tested by batching deliberately rather than by argument. Recorded and not fixed: InnerLoop's tier table triggers on code properties, so this pass would be tier S despite deciding how the loop governs itself. Changing the taxonomy on one instance is the invention-in-isolation INTENT warns against — the same argument used to amend K14 and to leave v1.4 alone. A second instance should trigger it. CB-WP-0007 executes in 6 tasks. T04's third outcome is the interesting one: if batching stays below 5% despite trying, 20% is the number to challenge with evidence, in an ADR rather than a quiet edit. Co-Authored-By: Claude Opus 5 --- research/CB-RES-0005-session-shape.md | 161 ++++++++++++++++++++++ workplans/CB-WP-0007-session-shape.md | 186 ++++++++++++++++++++++++++ 2 files changed, 347 insertions(+) create mode 100644 research/CB-RES-0005-session-shape.md create mode 100644 workplans/CB-WP-0007-session-shape.md diff --git a/research/CB-RES-0005-session-shape.md b/research/CB-RES-0005-session-shape.md new file mode 100644 index 0000000..f414afd --- /dev/null +++ b/research/CB-RES-0005-session-shape.md @@ -0,0 +1,161 @@ +# CB-RES-0005: the session-shape metrics measure the wrong window + +capability: meta.loop.session-shape +status: draft +tier: M (structural S by the letter of InnerLoop's table — no capability +port, no canonical interface, no new dependency — declared **M** because +the table's triggers only cover *code* capabilities and this changes +whether the loop constrains its own operation. The taxonomy gap is itself +a finding, §6. Chaos d4=2 → no override.) +instrument: `make cost-mix` / `cb-cost --since` (session shape block) +baseline: our own SH-1/SH-2/SH-3 targets, `specs/SessionShape.md` §6 + +**Survey and ADR merged, per tier M.** The decision is §5. + +--- + +## 1. The finding + +`specs/SessionShape.md` has stated SS-01…SS-05 and SH-1/2/3 since +CB-WP-0003. **None has ever been enforced.** They are the only +acceptance-adjacent numbers in this project with no gate at all. + +But the gap is not the one CB-WP-0006 assumed. These metrics *are* +measured — `cb-cost` emits all three on every run. The defect is worse and +more interesting: + +> **SH-1 and SH-2 are cumulative means over every response ever recorded. +> They cannot detect a worsening trend, because the history outvotes the +> present.** + +Measured on the same instrument, cumulatively and per window: + +| window | responses | SH-1 mean context | SH-2 p90 | SH-3 batching | +|---|---|---|---|---| +| cumulative → `fc76445` | 346 | 232,982 | 492,042 | 7.8% | +| cumulative → `578dcbe` | 662 | 206,052 | 437,688 | 8.6% | +| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% | +| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** | + +Read the cumulative column alone and session shape looks **stable and +slowly improving** — mean context down 232k → 207k, p90 down 492k → 390k. +Read the window and the most recent pass ran at **2.4× the mean** and +**1.6× the p90** of the whole preceding history. + +Both numbers come from the same instrument on the same data. The +cumulative one is not wrong; it answers a question nobody is asking. +A ceiling on a running average over 923 responses can absorb 156 +responses at 500k and still print `206,952`. + +This is a **new shape of the same disease** the last two passes chased: +not an unmeasured number, and not an unasserted one, but a number +**aggregated so that it cannot fire**. + +## 2. SH-3 is zero, and that is a fact about behaviour + +**146 tool calls in 146 responses across CB-WP-0006. Zero batched.** The +preceding window was also 0.0%. Cumulatively 9.6%, and falling, against a +20% target. + +Unlike SH-1/SH-2 this needs no new aggregation to be alarming — it is +already zero. It is also the cheapest of the three to move and, per +`SessionShape.md` §4, the least valuable: the ceiling is $2–4 on a $93 +pass. Worth stating both halves, because a metric that is easy to move and +worth little is exactly the one a pass will optimize to look productive. + +## 3. What the state of the art does + +Cited, not measured; caps any comparison at `parity` per +MetricsAndScenarios §3. + +| practice | window | enforcement | +|---|---|---| +| **SLO error budgets** (Google SRE) | rolling window, explicitly *not* cumulative | budget burn alerts, not build failure | +| **Criterion regression detection** | last run vs this run | fails on >N% regression | +| **`cargo build --timings`** | per build | reports, never gates | +| **Our own CB-01/CB-02** (cost budget) | **since the last commit** | soft warn / hard stop, surfaced live | + +The strongest precedent is the one already in this repo. CB-01/CB-02 +measures spend **since the last commit**, warns at $10, stops at $22, and +is surfaced by `make status` and `make cost-budget` — not by `make all`. +It has fired usefully and has never blocked a commit. + +**Provenance: `adapted:sre-error-budget`** — a rolling window with soft and +hard thresholds, adapted by making the window *since the last commit* +rather than time-based, because commits are what this loop can act on. + +## 4. Why a build gate is the wrong instrument here + +Stating this because it is the obvious move and it is wrong. + +Failing `make all` on context size would **block committing**. Committing +is what closes the cost-attribution window and is the natural point to +compact. A gate that blocks the remedy when the metric is breached is not +a gate, it is a trap. + +Worse, it punishes the wrong artifact. SH-* measure **agent behaviour**, +not the code under test. A red `make all` says "this tree is broken"; a +long session does not break the tree. Every other gate in this project +fails the build because the *committed artifact* is wrong. + +There is also a limit `SessionShape.md` §5 states and this survey will not +paper over: **there is no claim about quality.** Compaction discards +detail; every number says it is cheaper, none says the work is as good. +Driving context down under a gate could trade cost for quality silently, +and this loop cannot currently detect that. CB-WP-0006 is weak evidence +against the worry — it ran at 503k, uncompacted, and produced 9/9 tasks +and six gate-caught defects — but "the pass that broke the ceiling went +fine" is not an argument for raising ceilings. + +## 5. Decision + +**Adopted, merged into this document per tier M.** + +**D1 — SH-1 and SH-2 become windowed.** `cb-cost` reports session shape +for the window under measurement, and the cumulative figure is retained +and labelled as *history*, not as the metric. A cumulative mean stays +useful for the long view and stops being the thing a target compares to. + +**D2 — enforcement is a live budget, not a build gate**, modelled on +CB-01/CB-02: soft and hard thresholds on the window since the last commit, +surfaced by `make status` and a new `make shape-budget`. **Not** in +`make all`. The soft threshold is the existing SH-1/SH-2 targets; the hard +threshold is stated in the workplan and set *before* the next measurement, +not after. + +**D3 — the targets do not move.** SH-1 ≤ 200,000, SH-2 ≤ 300,000, +SH-3 ≥ 20% stand as written. Every one is currently unmet, badly, and +adjusting them in the pass that first enforces them is precisely the +retarget InnerLoop §Step 4 forbids. If 20% batching proves unreachable, +that is a finding to publish, not a number to lower. + +**D4 — SH-3 is tested by doing, not by argument.** The pass that +implements this should batch deliberately and report what the rate +reaches. A target nobody has tried to hit is not evidence of anything. + +## 6. A finding about the loop itself + +InnerLoop's tier table triggers on **code** properties: "creates a new +capability port", "touches a canonical interface", "adds a dependency". +This pass does none of those and would be **tier S** — one provenance +paragraph in a commit message — despite deciding whether and how the loop +constrains its own operation, with a stated and unmeasured quality +trade-off. + +That is a gap, not a reason to inflate the tier by feel. It is recorded +here rather than fixed, because changing the tier taxonomy on the evidence +of one pass is the invention-in-isolation INTENT warns against — the same +argument used to amend K14 and to leave InnerLoop v1.4 alone. **A second +instance should trigger the change.** + +## 7. Prediction + +Stated per task, as a mechanism, with the alternative named — the form +that collapsed prediction error in CB-WP-0006. + +| claim | mechanism | refuted if | +|---|---|---| +| windowed SH-1/SH-2 differ sharply from cumulative | the window is computed over the same data with a different denominator | the windowed figure lands within 20% of cumulative | +| the budget fires on a real pass | soft threshold at the existing target, checked since last commit | it never reports a breach across a full pass | +| SH-3 is movable by intent | batch deliberately and measure | the rate stays below 5% despite trying, in which case **20% is the number to challenge, with evidence** | +| gating does not cost quality | same gates, same findings rate | fewer defects surface, or work is repeated after compaction | diff --git a/workplans/CB-WP-0007-session-shape.md b/workplans/CB-WP-0007-session-shape.md new file mode 100644 index 0000000..4523513 --- /dev/null +++ b/workplans/CB-WP-0007-session-shape.md @@ -0,0 +1,186 @@ +--- +id: CB-WP-0007 +title: "Make session shape measurable in the window that matters, then enforce it" +status: proposed +--- + +# Purpose + +[CB-RES-0005](../research/CB-RES-0005-session-shape.md) (tier M, survey +and ADR merged; chaos d4=2, no override): + +> **SH-1 and SH-2 are cumulative means over every response ever recorded. +> They cannot detect a worsening trend, because the history outvotes the +> present.** + +| window | responses | SH-1 mean | SH-2 p90 | SH-3 batching | +|---|---|---|---|---| +| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% | +| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** | + +Read cumulatively, session shape looks stable and improving. Read by +window, the last pass ran at **2.4× the mean** of all preceding history +and batched **nothing at all** — 146 tool calls in 146 responses. + +This is the third shape of one disease. CB-WP-0005 found numbers that were +**unmeasured**; CB-WP-0006 found numbers that were measured but +**unasserted**; this pass has numbers that are measured *and* asserted and +still cannot fire, because they are **aggregated over a window in which +nothing can move them**. + +Per InnerLoop §Step 4, no target moves in the commit that measures it — +and CB-RES-0005 D3 states that up front, because all three targets are +currently unmet by wide margins and the temptation is obvious. + +## Phase A — measure the right window + +## Task: window the session-shape metrics + +```task +id: CB-WP-0007-T01 +status: todo +priority: high +``` + +`cb-cost`'s session-shape block reports over every response in scope. With +`--since` it already windows correctly; without it, it reports a +cumulative mean that a bad pass cannot move. + +Deliver: the session-shape block **states its window explicitly** and, when +run without `--since`, reports **both** — the window since the last commit +*and* the cumulative history, labelled so they cannot be confused. The +cumulative figure stays useful for the long view and stops being what a +target compares against. + +**Predicted:** the windowed figure differs sharply from cumulative. +**Refuted if** it lands within 20% of the cumulative number, in which case +the aggregation was not the problem and this pass should re-plan. + +## Task: SH-4 — a windowed trend the instrument can see + +```task +id: CB-WP-0007-T02 +status: todo +priority: medium +``` + +A ceiling catches a bad pass. It does not catch a slow slide, which is how +SH-1 got from 232k to 503k without any single pass looking anomalous. + +Deliver a reported (not gated) SH-4: **mean context of the current window +against the previous window**, so a rising trend is visible before it +breaches. Provenance `adapted:criterion` — the same shape as the >3% +merge-blocking regression rule against our own last evidence file, but +reported rather than blocking, because CB-RES-0005 §4 argues a build gate +is the wrong instrument for behaviour. + +State the limit in the output, as every other gate here does: two windows +is not a trend, and the first run has nothing to compare against. + +## Phase B — enforce it where enforcement belongs + +## Task: `make shape-budget` — a live budget, not a build gate + +```task +id: CB-WP-0007-T03 +status: todo +priority: high +``` + +Per CB-RES-0005 D2, modelled on CB-01/CB-02, which measures spend since +the last commit, warns at $10, stops at $22, and is surfaced by +`make status` — never by `make all`. + +**Why not a build gate, restated so it is not quietly reversed later:** +failing `make all` on context size would **block committing**, and +committing is what closes the attribution window and is the natural point +to compact. A gate that blocks the remedy when the metric breaches is a +trap, not a gate. It also punishes the wrong artifact — a long session +does not break the tree. + +Deliver `make shape-budget` and a line in `make status`: + +| threshold | value | action | +|---|---|---| +| SH-1 soft | 200,000 mean context in the window | warn | +| SH-1 hard | **300,000** | report loudly: compact before continuing | +| SH-2 soft | 300,000 p90 | warn | +| SH-3 floor | 20% batching | warn | + +**The hard threshold is set here, before the next measurement**, per +§Step 4. It is 1.5× the soft target, matching CB-02's 2.2× ratio on spend +scaled down because context is bounded by the model and spend is not. + +Carries `--self-test`. Its positive control is the one this project keeps +needing: a budget that reports `ok` because it measured nothing must +abort instead. + +## Task: batch deliberately, and report what the rate reaches + +```task +id: CB-WP-0007-T04 +status: todo +priority: medium +``` + +SH-3 is **0.0%** across two consecutive passes. The target is 20%. Nobody +has ever tried to hit it, so its reachability is unknown and the number is +currently unfalsified rather than wrong. + +This task is the trial: batch independent tool calls deliberately through +the pass and report the rate the instrument measures. + +**Three outcomes, all publishable, and the third is the interesting one:** + +1. **≥20%** — the target is reachable; it stops being aspirational. +2. **5–20%** — partially reachable; report the ceiling and what blocked it. +3. **<5% despite trying** — then **20% is the number to challenge, with + evidence**, and the challenge belongs in an ADR rather than a quiet + edit. Per §Step 4 the target does not move in this pass regardless. + +## Phase C — measure, and say what it cost + +## Task: control loop — did windowing change what fires? + +```task +id: CB-WP-0007-T05 +status: todo +priority: high +``` + +Commit `evidence/CB-EV-0006-session-shape.md`. Four tests, all reported: + +1. **Does the windowed metric differ from cumulative?** Against T01's + prediction; refuted if within 20%. +2. **Did the budget fire on this pass?** A budget that never reports a + breach across a full pass is either miscalibrated or measuring nothing — + both are findings. +3. **What did SH-3 reach?** T04's three outcomes. +4. **Did quality hold?** `make all` green, and the honest question this + pass cannot dodge: **did compacting to satisfy a context budget cause + work to be repeated or detail to be lost?** `SessionShape.md` §5 says + the loop cannot detect this. Record the judgment explicitly rather than + letting the cost number imply it. + +## Task: retrospective + +```task +id: CB-WP-0007-T06 +status: todo +priority: medium +``` + +The question: **three passes have now found three different ways a number +can exist and not bind — unmeasured, unasserted, and aggregated so it +cannot fire. Is there a fourth, and is the loop discovering a pattern or +enumerating an infinite set?** + +CB-WP-0003 concluded that class-by-class hardening does not converge. +Three passes later, the *instrument* failures have their own taxonomy and +it is also not converging. Say plainly whether that changes the design +goal, or confirms it. + +Also due: **the tier-taxonomy gap** CB-RES-0005 §6 recorded but did not +fix. This pass is the first instance. Per the standing argument used for +K14 and InnerLoop v1.4, a second instance triggers the change — say +whether this counts as one or two.