tegwick
d07caca22b
Both remaining tasks were overtaken and are cancelled with pointers. T05's control loop was answered by CB-EV-0008 §1 and §4, which measured the windowed metric and the budget firing on real passes. T06 asked whether there is a fourth way a number can fail to bind; there is, and CB-EV-0007 §3 recorded it — the meta budget, aggregated over a window that could not move it, two tasks after the same defect was fixed for session shape. The tier-taxonomy gap T06 also carried was closed by ADR-0006 D5. A workplan left in_progress makes `make status` name a stale next task, which it had been doing for two passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
240 lines
9.7 KiB
Markdown
240 lines
9.7 KiB
Markdown
---
|
||
id: CB-WP-0007
|
||
kind: meta
|
||
title: "Make session shape measurable in the window that matters, then enforce it"
|
||
status: done
|
||
state_hub_workstream_id: "bee19b76-fb60-4dcf-94d0-9242b43e0e42"
|
||
---
|
||
|
||
# Purpose
|
||
|
||
[CB-RES-0005](../research/CB-RES-0005-session-shape.md) (tier M, survey
|
||
and ADR merged; chaos d4=2, no override):
|
||
|
||
> **SH-1 and SH-2 are cumulative means over every response ever recorded.
|
||
> They cannot detect a worsening trend, because the history outvotes the
|
||
> present.**
|
||
|
||
| window | responses | SH-1 mean | SH-2 p90 | SH-3 batching |
|
||
|---|---|---|---|---|
|
||
| cumulative → `dfd0d6d` | 923 | **206,952** | **389,733** | 9.6% |
|
||
| **CB-WP-0006 alone** | 156 | **503,464** | **607,105** | **0.0%** |
|
||
|
||
Read cumulatively, session shape looks stable and improving. Read by
|
||
window, the last pass ran at **2.4× the mean** of all preceding history
|
||
and batched **nothing at all** — 146 tool calls in 146 responses.
|
||
|
||
This is the third shape of one disease. CB-WP-0005 found numbers that were
|
||
**unmeasured**; CB-WP-0006 found numbers that were measured but
|
||
**unasserted**; this pass has numbers that are measured *and* asserted and
|
||
still cannot fire, because they are **aggregated over a window in which
|
||
nothing can move them**.
|
||
|
||
Per InnerLoop §Step 4, no target moves in the commit that measures it —
|
||
and CB-RES-0005 D3 states that up front, because all three targets are
|
||
currently unmet by wide margins and the temptation is obvious.
|
||
|
||
## Scope cut, 2026-08-01 (maintainer decision)
|
||
|
||
A spend review before starting found the project **38% product / 62%
|
||
loop-meta**, with cost per response degraded 2.9× from its best window and
|
||
INTENT stage 0 still missing a CLI player and bots. CB-WP-0005 and
|
||
CB-WP-0006 cost **~$74 — 31% of all spend — for zero measured efficiency
|
||
gain** (their return was correctness of claims, which is real but is not
|
||
optimization).
|
||
|
||
So this workplan is **cut to T01 and T03**, the two tasks that attack the
|
||
2.9× regression directly. **T02 and T04 are cancelled**, not deferred —
|
||
SH-4 trend reporting and the batching trial are more instrument work, and
|
||
the instrument-failure taxonomy is not converging. T06 becomes the
|
||
project-level retrospective the maintainer asked for.
|
||
|
||
**A soft 25% meta budget is established here** (T03), because the review's
|
||
finding needs a standing number, not a memory.
|
||
|
||
## Phase A — measure the right window
|
||
|
||
## Task: window the session-shape metrics
|
||
|
||
```task
|
||
id: CB-WP-0007-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "7e036db4-d97a-45bd-a70a-20fe68225402"
|
||
```
|
||
|
||
`cb-cost`'s session-shape block reports over every response in scope. With
|
||
`--since` it already windows correctly; without it, it reports a
|
||
cumulative mean that a bad pass cannot move.
|
||
|
||
Deliver: the session-shape block **states its window explicitly** and, when
|
||
run without `--since`, reports **both** — the window since the last commit
|
||
*and* the cumulative history, labelled so they cannot be confused. The
|
||
cumulative figure stays useful for the long view and stops being what a
|
||
target compares against.
|
||
|
||
**Predicted:** the windowed figure differs sharply from cumulative.
|
||
**Refuted if** it lands within 20% of the cumulative number, in which case
|
||
the aggregation was not the problem and this pass should re-plan.
|
||
|
||
## Task: SH-4 — a windowed trend the instrument can see (CANCELLED)
|
||
|
||
> **Cancelled unstarted 2026-08-01.** More instrument work, against a
|
||
> review finding that instrument work has stopped paying. The ceiling T03
|
||
> adds catches a bad pass; a trend line would catch a slow slide, which is
|
||
> a real gap — recorded as open, not built.
|
||
|
||
```task
|
||
id: CB-WP-0007-T02
|
||
status: cancel
|
||
priority: medium
|
||
state_hub_task_id: "1d5a32ef-4086-4021-b874-24ed2160c4dd"
|
||
```
|
||
|
||
A ceiling catches a bad pass. It does not catch a slow slide, which is how
|
||
SH-1 got from 232k to 503k without any single pass looking anomalous.
|
||
|
||
Deliver a reported (not gated) SH-4: **mean context of the current window
|
||
against the previous window**, so a rising trend is visible before it
|
||
breaches. Provenance `adapted:criterion` — the same shape as the >3%
|
||
merge-blocking regression rule against our own last evidence file, but
|
||
reported rather than blocking, because CB-RES-0005 §4 argues a build gate
|
||
is the wrong instrument for behaviour.
|
||
|
||
State the limit in the output, as every other gate here does: two windows
|
||
is not a trend, and the first run has nothing to compare against.
|
||
|
||
## Phase B — enforce it where enforcement belongs
|
||
|
||
## Task: `make shape-budget` — a live budget, not a build gate
|
||
|
||
```task
|
||
id: CB-WP-0007-T03
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "126a6313-b1f7-462c-8771-5100d38f9429"
|
||
```
|
||
|
||
Per CB-RES-0005 D2, modelled on CB-01/CB-02, which measures spend since
|
||
the last commit, warns at $10, stops at $22, and is surfaced by
|
||
`make status` — never by `make all`.
|
||
|
||
**Why not a build gate, restated so it is not quietly reversed later:**
|
||
failing `make all` on context size would **block committing**, and
|
||
committing is what closes the attribution window and is the natural point
|
||
to compact. A gate that blocks the remedy when the metric breaches is a
|
||
trap, not a gate. It also punishes the wrong artifact — a long session
|
||
does not break the tree.
|
||
|
||
Deliver `make shape-budget` and a line in `make status`:
|
||
|
||
| threshold | value | action |
|
||
|---|---|---|
|
||
| SH-1 soft | 200,000 mean context in the window | warn |
|
||
| SH-1 hard | **300,000** | report loudly: compact before continuing |
|
||
| SH-2 soft | 300,000 p90 | warn |
|
||
| SH-3 floor | 20% batching | warn |
|
||
|
||
**The hard threshold is set here, before the next measurement**, per
|
||
§Step 4. It is 1.5× the soft target, matching CB-02's 2.2× ratio on spend
|
||
scaled down because context is bounded by the model and spend is not.
|
||
|
||
Carries `--self-test`. Its positive control is the one this project keeps
|
||
needing: a budget that reports `ok` because it measured nothing must
|
||
abort instead.
|
||
|
||
## Task: batch deliberately, and report what the rate reaches (CANCELLED)
|
||
|
||
> **Cancelled unstarted 2026-08-01.** `SessionShape.md` §4 already puts
|
||
> the ceiling at **$2–4 on a $93 pass** — the cheapest of the three
|
||
> metrics to move and the least valuable. Spending a task on it while
|
||
> stage 0 lacks a CLI player is the misallocation the review found.
|
||
> SH-3 remains measured, unmet at 0.0%, and unfalsified.
|
||
|
||
```task
|
||
id: CB-WP-0007-T04
|
||
status: cancel
|
||
priority: medium
|
||
state_hub_task_id: "c284db6f-2e16-40fb-be8b-83ae5391c058"
|
||
```
|
||
|
||
SH-3 is **0.0%** across two consecutive passes. The target is 20%. Nobody
|
||
has ever tried to hit it, so its reachability is unknown and the number is
|
||
currently unfalsified rather than wrong.
|
||
|
||
This task is the trial: batch independent tool calls deliberately through
|
||
the pass and report the rate the instrument measures.
|
||
|
||
**Three outcomes, all publishable, and the third is the interesting one:**
|
||
|
||
1. **≥20%** — the target is reachable; it stops being aspirational.
|
||
2. **5–20%** — partially reachable; report the ceiling and what blocked it.
|
||
3. **<5% despite trying** — then **20% is the number to challenge, with
|
||
evidence**, and the challenge belongs in an ADR rather than a quiet
|
||
edit. Per §Step 4 the target does not move in this pass regardless.
|
||
|
||
## Phase C — measure, and say what it cost
|
||
|
||
## Task: control loop — did windowing change what fires? (CANCELLED)
|
||
|
||
> **Cancelled 2026-08-02 (CB-WP-0010 T01), overtaken.** Both questions
|
||
> were answered with measurements two passes later:
|
||
> [CB-EV-0008](../evidence/CB-EV-0008-adaptive-gates.md) §1 compares the
|
||
> windowed metric against cumulative on real passes, and §4 records the
|
||
> shape budget firing. Writing CB-EV-0006 now would re-derive them from
|
||
> the same transcripts.
|
||
|
||
```task
|
||
id: CB-WP-0007-T05
|
||
status: cancel
|
||
priority: high
|
||
state_hub_task_id: "2f78b272-e1f9-4530-bc8c-3a9d82833546"
|
||
```
|
||
|
||
Commit `evidence/CB-EV-0006-session-shape.md`. **Reduced with the scope
|
||
cut** — tests 1, 2 and 4 remain; test 3 (SH-3) is cancelled with T04.
|
||
|
||
Four tests, all reported:
|
||
|
||
1. **Does the windowed metric differ from cumulative?** Against T01's
|
||
prediction; refuted if within 20%.
|
||
2. **Did the budget fire on this pass?** A budget that never reports a
|
||
breach across a full pass is either miscalibrated or measuring nothing —
|
||
both are findings.
|
||
3. **What did SH-3 reach?** T04's three outcomes.
|
||
4. **Did quality hold?** `make all` green, and the honest question this
|
||
pass cannot dodge: **did compacting to satisfy a context budget cause
|
||
work to be repeated or detail to be lost?** `SessionShape.md` §5 says
|
||
the loop cannot detect this. Record the judgment explicitly rather than
|
||
letting the cost number imply it.
|
||
|
||
## Task: retrospective (CANCELLED)
|
||
|
||
> **Cancelled 2026-08-02 (CB-WP-0010 T01), answered elsewhere.** The
|
||
> question was whether there is a *fourth* way a number can fail to bind.
|
||
> There is, and it is on record: CB-EV-0007 §3 found a number **aggregated
|
||
> over a window that cannot move it** — the meta budget, reintroducing the
|
||
> defect two tasks after the same one was fixed for session shape. The
|
||
> tier-taxonomy gap this task also carried was closed by ADR-0006 D5.
|
||
|
||
```task
|
||
id: CB-WP-0007-T06
|
||
status: cancel
|
||
priority: medium
|
||
state_hub_task_id: "93fbd2bb-403b-4be9-a5ff-d6142b916aba"
|
||
```
|
||
|
||
The question: **three passes have now found three different ways a number
|
||
can exist and not bind — unmeasured, unasserted, and aggregated so it
|
||
cannot fire. Is there a fourth, and is the loop discovering a pattern or
|
||
enumerating an infinite set?**
|
||
|
||
CB-WP-0003 concluded that class-by-class hardening does not converge.
|
||
Three passes later, the *instrument* failures have their own taxonomy and
|
||
it is also not converging. Say plainly whether that changes the design
|
||
goal, or confirms it.
|
||
|
||
Also due: **the tier-taxonomy gap** CB-RES-0005 §6 recorded but did not
|
||
fix. This pass is the first instance. Per the standing argument used for
|
||
K14 and InnerLoop v1.4, a second instance triggers the change — say
|
||
whether this counts as one or two.
|