CB-RES-0005 + CB-WP-0007: session shape measures the wrong window
Some checks failed
ci / check (push) Failing after 4s
Some checks failed
ci / check (push) Failing after 4s
Tier M — structural S by the letter of InnerLoop's table (no capability
port, no canonical interface, no new dependency), declared M because the
table's triggers only cover code capabilities and this decides whether the
loop constrains its own operation. Chaos d4=2, no override. Survey and ADR
merged per tier M.
The gap is not the one CB-WP-0006 assumed. SH-1/2/3 ARE measured —
cb-cost emits all three every run. The defect is that SH-1 and SH-2 are
cumulative means over every response ever recorded, so they cannot detect
a worsening trend: the history outvotes the present.
cumulative -> dfd0d6d 923 resp mean 206,952 p90 389,733 9.6%
CB-WP-0006 alone 156 resp mean 503,464 p90 607,105 0.0%
Read cumulatively, session shape looks stable and improving. Read by
window, the last pass ran at 2.4x the mean of all preceding history and
batched nothing at all — 146 tool calls in 146 responses.
This is the third shape of one disease. CB-WP-0005 found numbers that were
unmeasured; CB-WP-0006 found numbers measured but unasserted; these are
measured AND asserted and still cannot fire, because they are aggregated
over a window in which nothing can move them.
Decisions: window SH-1/SH-2 and keep cumulative as labelled history;
enforce with a live budget modelled on CB-01/CB-02, NOT `make all` —
failing the build on context would block committing, and committing is
what closes the attribution window and is the natural point to compact, so
a gate that blocks the remedy is a trap; targets do not move, since all
three are unmet by wide margins and adjusting them in the pass that first
enforces them is the retarget §Step 4 forbids; and SH-3 is tested by
batching deliberately rather than by argument.
Recorded and not fixed: InnerLoop's tier table triggers on code
properties, so this pass would be tier S despite deciding how the loop
governs itself. Changing the taxonomy on one instance is the
invention-in-isolation INTENT warns against — the same argument used to
amend K14 and to leave v1.4 alone. A second instance should trigger it.
CB-WP-0007 executes in 6 tasks. T04's third outcome is the interesting
one: if batching stays below 5% despite trying, 20% is the number to
challenge with evidence, in an ADR rather than a quiet edit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
4b899fbfe5
commit
0e394ccb3f
2 changed files with 347 additions and 0 deletions
186
workplans/CB-WP-0007-session-shape.md
Normal file
186
workplans/CB-WP-0007-session-shape.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
---
|
||||
id: CB-WP-0007
|
||||
title: "Make session shape measurable in the window that matters, then enforce it"
|
||||
status: proposed
|
||||
---
|
||||
|
||||
# Purpose
|
||||
|
||||
[CB-RES-0005](../research/CB-RES-0005-session-shape.md) (tier M, survey
|
||||
and ADR merged; chaos d4=2, no override):
|
||||
|
||||
> **SH-1 and SH-2 are cumulative means over every response ever recorded.
|
||||
> They cannot detect a worsening trend, because the history outvotes the
|
||||
> present.**
|
||||
|
||||
| window | responses | SH-1 mean | SH-2 p90 | SH-3 batching |
|
||||
|---|---|---|---|---|
|
||||
| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
|
||||
| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |
|
||||
|
||||
Read cumulatively, session shape looks stable and improving. Read by
|
||||
window, the last pass ran at **2.4× the mean** of all preceding history
|
||||
and batched **nothing at all** — 146 tool calls in 146 responses.
|
||||
|
||||
This is the third shape of one disease. CB-WP-0005 found numbers that were
|
||||
**unmeasured**; CB-WP-0006 found numbers that were measured but
|
||||
**unasserted**; this pass has numbers that are measured *and* asserted and
|
||||
still cannot fire, because they are **aggregated over a window in which
|
||||
nothing can move them**.
|
||||
|
||||
Per InnerLoop §Step 4, no target moves in the commit that measures it —
|
||||
and CB-RES-0005 D3 states that up front, because all three targets are
|
||||
currently unmet by wide margins and the temptation is obvious.
|
||||
|
||||
## Phase A — measure the right window
|
||||
|
||||
## Task: window the session-shape metrics
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
`cb-cost`'s session-shape block reports over every response in scope. With
|
||||
`--since` it already windows correctly; without it, it reports a
|
||||
cumulative mean that a bad pass cannot move.
|
||||
|
||||
Deliver: the session-shape block **states its window explicitly** and, when
|
||||
run without `--since`, reports **both** — the window since the last commit
|
||||
*and* the cumulative history, labelled so they cannot be confused. The
|
||||
cumulative figure stays useful for the long view and stops being what a
|
||||
target compares against.
|
||||
|
||||
**Predicted:** the windowed figure differs sharply from cumulative.
|
||||
**Refuted if** it lands within 20% of the cumulative number, in which case
|
||||
the aggregation was not the problem and this pass should re-plan.
|
||||
|
||||
## Task: SH-4 — a windowed trend the instrument can see
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
A ceiling catches a bad pass. It does not catch a slow slide, which is how
|
||||
SH-1 got from 232k to 503k without any single pass looking anomalous.
|
||||
|
||||
Deliver a reported (not gated) SH-4: **mean context of the current window
|
||||
against the previous window**, so a rising trend is visible before it
|
||||
breaches. Provenance `adapted:criterion` — the same shape as the >3%
|
||||
merge-blocking regression rule against our own last evidence file, but
|
||||
reported rather than blocking, because CB-RES-0005 §4 argues a build gate
|
||||
is the wrong instrument for behaviour.
|
||||
|
||||
State the limit in the output, as every other gate here does: two windows
|
||||
is not a trend, and the first run has nothing to compare against.
|
||||
|
||||
## Phase B — enforce it where enforcement belongs
|
||||
|
||||
## Task: `make shape-budget` — a live budget, not a build gate
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Per CB-RES-0005 D2, modelled on CB-01/CB-02, which measures spend since
|
||||
the last commit, warns at $10, stops at $22, and is surfaced by
|
||||
`make status` — never by `make all`.
|
||||
|
||||
**Why not a build gate, restated so it is not quietly reversed later:**
|
||||
failing `make all` on context size would **block committing**, and
|
||||
committing is what closes the attribution window and is the natural point
|
||||
to compact. A gate that blocks the remedy when the metric breaches is a
|
||||
trap, not a gate. It also punishes the wrong artifact — a long session
|
||||
does not break the tree.
|
||||
|
||||
Deliver `make shape-budget` and a line in `make status`:
|
||||
|
||||
| threshold | value | action |
|
||||
|---|---|---|
|
||||
| SH-1 soft | 200,000 mean context in the window | warn |
|
||||
| SH-1 hard | **300,000** | report loudly: compact before continuing |
|
||||
| SH-2 soft | 300,000 p90 | warn |
|
||||
| SH-3 floor | 20% batching | warn |
|
||||
|
||||
**The hard threshold is set here, before the next measurement**, per
|
||||
§Step 4. It is 1.5× the soft target, matching CB-02's 2.2× ratio on spend
|
||||
scaled down because context is bounded by the model and spend is not.
|
||||
|
||||
Carries `--self-test`. Its positive control is the one this project keeps
|
||||
needing: a budget that reports `ok` because it measured nothing must
|
||||
abort instead.
|
||||
|
||||
## Task: batch deliberately, and report what the rate reaches
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T04
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
SH-3 is **0.0%** across two consecutive passes. The target is 20%. Nobody
|
||||
has ever tried to hit it, so its reachability is unknown and the number is
|
||||
currently unfalsified rather than wrong.
|
||||
|
||||
This task is the trial: batch independent tool calls deliberately through
|
||||
the pass and report the rate the instrument measures.
|
||||
|
||||
**Three outcomes, all publishable, and the third is the interesting one:**
|
||||
|
||||
1. **≥20%** — the target is reachable; it stops being aspirational.
|
||||
2. **5–20%** — partially reachable; report the ceiling and what blocked it.
|
||||
3. **<5% despite trying** — then **20% is the number to challenge, with
|
||||
evidence**, and the challenge belongs in an ADR rather than a quiet
|
||||
edit. Per §Step 4 the target does not move in this pass regardless.
|
||||
|
||||
## Phase C — measure, and say what it cost
|
||||
|
||||
## Task: control loop — did windowing change what fires?
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Commit `evidence/CB-EV-0006-session-shape.md`. Four tests, all reported:
|
||||
|
||||
1. **Does the windowed metric differ from cumulative?** Against T01's
|
||||
prediction; refuted if within 20%.
|
||||
2. **Did the budget fire on this pass?** A budget that never reports a
|
||||
breach across a full pass is either miscalibrated or measuring nothing —
|
||||
both are findings.
|
||||
3. **What did SH-3 reach?** T04's three outcomes.
|
||||
4. **Did quality hold?** `make all` green, and the honest question this
|
||||
pass cannot dodge: **did compacting to satisfy a context budget cause
|
||||
work to be repeated or detail to be lost?** `SessionShape.md` §5 says
|
||||
the loop cannot detect this. Record the judgment explicitly rather than
|
||||
letting the cost number imply it.
|
||||
|
||||
## Task: retrospective
|
||||
|
||||
```task
|
||||
id: CB-WP-0007-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
The question: **three passes have now found three different ways a number
|
||||
can exist and not bind — unmeasured, unasserted, and aggregated so it
|
||||
cannot fire. Is there a fourth, and is the loop discovering a pattern or
|
||||
enumerating an infinite set?**
|
||||
|
||||
CB-WP-0003 concluded that class-by-class hardening does not converge.
|
||||
Three passes later, the *instrument* failures have their own taxonomy and
|
||||
it is also not converging. Say plainly whether that changes the design
|
||||
goal, or confirms it.
|
||||
|
||||
Also due: **the tier-taxonomy gap** CB-RES-0005 §6 recorded but did not
|
||||
fix. This pass is the first instance. Per the standing argument used for
|
||||
K14 and InnerLoop v1.4, a second instance triggers the change — say
|
||||
whether this counts as one or two.
|
||||
Loading…
Add table
Add a link
Reference in a new issue