diff --git a/specs/SessionShape.md b/specs/SessionShape.md new file mode 100644 index 0000000..44e4d38 --- /dev/null +++ b/specs/SessionShape.md @@ -0,0 +1,185 @@ +# Session Shape + +Status: **v1.0** — 2026-07-31, from CB-WP-0003 T04. +Instrument: `make cost` ([CostAccounting.md](CostAccounting.md)). + +How to shape an agentic session so it does not cost more than the work is +worth. **Every claim here carries the measurement it rests on.** This spec +exists because the numbers were surprising and because the advice this task +was created to give turned out to be wrong. + +--- + +## 1. The cost model + +```text +cost ≈ Σ over turns ( context_size × cache_read_rate ) + output + cache writes +``` + +Measured on the CB-WP-0001 pass: **88.0% of spend is cache, 11.9% is +output**, at 249:1 cache-read tokens to output tokens. Cost is driven by +**how much context is re-read per turn**, not by how much the model writes. + +Marginal cost of one turn, at the Opus cache-read rate ($0.50/M): + +| context | $/turn | +|---|---| +| 20,000 | $0.010 | +| 50,000 | $0.025 | +| 100,000 | $0.050 | +| 200,000 | $0.100 | +| 400,000 | $0.200 | +| 540,000 | $0.270 | + +> **SS-01.** A turn's cost is a function of context size alone, to within +> ~12%. Optimizing output length is not a cost lever; optimizing context is. + +## 2. Compaction is the primary control + +**The original premise of this spec was wrong and is recorded rather than +deleted.** CB-WP-0003 T04 was written to prescribe *one task per session*, +on the theory that mean context grows with turn count and long sessions are +therefore quadratic. Measured: + +| segment | turns | mean context | total | $/turn | +|---|---|---|---|---| +| start → compact 1 | 136 | 304,178 | $62.19 | **$0.457** | +| compact 1 → compact 2 | 202 | 193,493 | $30.01 | **$0.149** | + +The 202 turns *after* the first compaction cost less than half the 136 +before it. Sessions are **bounded-quadratic**: context grows between +compactions and resets at each one. + +Compaction events, read from `compactMetadata` in the transcript: + +| | pre | post | reduction | duration | +|---|---|---|---|---| +| C1 | 542,991 | 19,974 | **27×** | 151 s | +| C2 | 344,954 | 19,035 | **18×** | 97 s | + +> **SS-02.** The failure mode is a long **uncompacted** session, not a long +> session. Turn count is not the variable to control; context is. + +### 2.1 When compaction pays + +A compaction costs its own summarization call plus one cache +re-establishment. The re-establishment is measurable — the first post-compact +turn wrote 36,149 cache tokens and cost **$0.382**. The summarization call +itself does **not** appear in the transcript's `usage` records, so it is +bounded rather than measured: at worst it is one full read of the +pre-compaction context (543k tokens ≈ $2.72 at the uncached Opus input +rate), at best a cache read of the same (≈ $0.27). + +Savings are immediate and permanent: dropping from 543k to 20k context saves +**$0.262 per subsequent turn**. + +> **SS-03.** Break-even is **2–11 turns** depending on where the +> summarization call falls in that range. Any session expecting more than +> ~11 further turns should compact. In practice this means: **compact +> whenever context exceeds ~300k and work remains.** + +Measured context distribution across the CB-WP-0001 session shows how much +was spent above that line — p75 was 308,325 and p90 was 481,722, so roughly +a quarter of all turns ran at more than 2× the recommended ceiling. + +| percentile | context | +|---|---| +| p50 | 156,771 | +| p75 | 308,325 | +| p90 | 481,722 | +| p99 | 553,252 | + +## 3. Fresh session vs compaction + +A fresh session is **not** free, and this is the measurement that overturned +the original advice. Cold-start context floors, measured over the first 20 +turns of each session: + +| session | start type | mean context, first 20 turns | cost | +|---|---|---|---| +| `8cbd5701` | cold (system + CLAUDE.md + orientation) | 50,872 | $2.17 | +| `f1eb1147` | seeded by a compact summary | 65,819 | $1.90 | + +A cold session starts ~51k and must then **re-read the committed artifacts** +to become productive — the survey, the spec, the ADR — which is exactly the +context a compaction summary already contains. A compact-seeded session +starts ~66k with that reading already done. + +> **SS-04.** Prefer compaction over a fresh session for continuing work. +> A fresh session's lower floor (51k vs 66k) is repaid within the first few +> turns of re-reading artifacts the summary already held. +> +> Prefer a fresh session when the *task changes*, because then the prior +> context is not an asset — it is 66k of irrelevance re-read every turn. + +## 4. Batching + +> **SS-05.** Independent tool calls in one turn cost nothing extra: a turn +> is billed for its context regardless of how many tool calls it carries. +> Two calls in one turn cost half of two calls in two turns. + +Measured batching rate: + +| session | responses w/ tool calls | tool calls | responses batching ≥2 | calls in batched turns | +|---|---|---|---|---| +| `8cbd5701` | 361 | 406 | 31 (8.6%) | 76 (18.7%) | +| `f1eb1147` | 146 | 158 | 12 (8.2%) | 24 (15.2%) | +| pinned at `fc76445`, all transcripts (instrument) | 322 | 362 | 25 (7.8%) | 65 (18.0%) | + +**Correction to a prior claim.** CB-WP-0001's retrospective and the first +draft of CB-WP-0003 T04 both stated *"0 of 330 tool calls were batched"*. +That is wrong: 330 was the count of responses making exactly **one** call, +not the total number of calls. The real figure is 7.8–8.6% of responses and +18.0–18.7% of calls, depending on the window. The claim was never re-derived before being carried into a +workplan — the failure mode InnerLoop v1.1 rule 13 exists for, and the +**eighth** recorded error instance, in the trusted-arithmetic class that +`history/260731-inner-loop-rule-audit.md` identified as having no +executable defence. + +At an 8.6% batching rate there is real headroom, but the saving is bounded: +eliminating every avoidable single-call turn would remove at most a few +percent of turns, worth roughly $2–4 on a $93 pass. **Batch because it is +free, not because it is the lever.** The lever is §2. + +## 5. What this spec does not support + +- **n = 2 sessions, one repo, one model mix.** Every ratio here is from + CB-WP-0001/0002 and should be re-measured before being treated as general. +- **The compaction summarization call is bounded, not measured.** SS-03's + 2–11 turn break-even is a range because of it. Narrowing it requires a + cost source that sees non-transcript calls — the billing API (CB-RES-0002 + C4), which needs an admin key this machine does not have. +- **Context size is read as `cache_read_input_tokens`**, which is the cached + prefix, not the full prompt. On a cache miss the true context is larger + and the turn costs 10× more. Cache misses were not isolated in this + measurement. +- **No claim about quality.** Compaction discards detail. Every number here + says compaction is cheaper; none says the work is as good. A pass that + compacts and then repeats work it forgot has not saved anything, and this + spec cannot detect that. + +## 6. Acceptance metrics + +| ID | Metric | Target | Measured (pinned `fc76445`) | Verdict | Instrument | +|---|---|---|---|---|---| +| SH-1 | mean context per turn | ≤ 200,000 | **232,982** | **unmet** | `make cost-pin` | +| SH-2 | p90 context per turn | ≤ 300,000 | **492,042** | **unmet** | `make cost-pin` | +| SH-3 | batching rate (responses with ≥2 tool calls) | ≥ 20% | **7.8%** | **unmet** | `make cost-pin` | + +`cb-cost` emits all three (`session shape` block), so these are ratified +under InnerLoop v1.1 rather than hand-derived. Supporting counts from the +same run: **362 tool calls in 322 responses, 65 of them in batched turns.** + +**All three targets are unmet, and that is the finding, not a defect in the +targets.** Per the loop's definition of done, an unmet metric is an output: + +- SH-1/SH-2 are unmet because CB-WP-0001 ran 136 turns before its first + compaction. §2 is the remedy and it is now written down. +- SH-3 is set above the measured rate deliberately. Whether 20% is + reachable or arbitrary is unknown until a pass tries; the first pass that + reports it will say. It is the cheapest of the three to move, and the + least valuable — §4 puts the ceiling at $2–4 on a $93 pass. + +Targets are **not** being retargeted to match the measurement. That is the +structure CB-WP-0003 T07 exists to prevent, and doing it here — in the +commit that first measures them — would be the exact defect. diff --git a/tools/cb-cost.py b/tools/cb-cost.py index 14db4ac..f76a6b0 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -147,6 +147,14 @@ def read_responses(path, pin=None): ) toks["output"] = max(t["output"] for t in per_row) head = rows[0] + # Tool calls are spread across the group's lines, so they are counted + # over the whole group — one response may carry several (SS-05). + tool_calls = sum( + 1 + for r in rows + for c in (r["message"].get("content") or []) + if c.get("type") == "tool_use" + ) out.append( { "request_id": rid, @@ -154,12 +162,33 @@ def read_responses(path, pin=None): "timestamp": head.get("timestamp") or "", "session": head.get("sessionId") or os.path.basename(path), "toks": toks, + "tool_calls": tool_calls, "subagent": "/subagents/" in path, } ) return out +def session_shape(responses): + """SH-1..SH-3 from specs/SessionShape.md.""" + import statistics + + ctx = sorted(r["toks"]["cache_read"] for r in responses) + with_tools = [r for r in responses if r["tool_calls"] > 0] + batched = [r for r in with_tools if r["tool_calls"] > 1] + calls = sum(r["tool_calls"] for r in with_tools) + pct = statistics.quantiles(ctx, n=100) if len(ctx) > 1 else [ctx[0]] * 99 + return { + "SH-1_mean_context": statistics.mean(ctx) if ctx else 0, + "SH-2_p90_context": pct[89], + "SH-3_batching_rate": (len(batched) / len(with_tools)) if with_tools else 0.0, + "p50_context": pct[49], + "responses_with_tools": len(with_tools), + "tool_calls": calls, + "calls_in_batched_turns": calls - (len(with_tools) - len(batched)), + } + + # ------------------------------------------------------------- attribution @@ -293,6 +322,7 @@ def collect(slug, pin_ref=None): sub = sum(r["cost"] or 0 for r in responses if r["subagent"]) return { + "session_shape": session_shape(responses), "slug": slug, "pin": pin, "responses": len(responses), @@ -344,6 +374,17 @@ def render(rep, by_task=False, composition=False): f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend." ) + sh = rep["session_shape"] + print("\n session shape (specs/SessionShape.md)") + print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok " + f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]") + print(f" SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok " + f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]") + print(f" SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% " + f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]") + print(f" {sh['tool_calls']} tool calls in {sh['responses_with_tools']} responses; " + f"{sh['calls_in_batched_turns']} in batched turns") + if rep["unpriced"]: print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):") for u in rep["unpriced"]: @@ -446,6 +487,8 @@ def main(): ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)") ap.add_argument("--by-task", action="store_true") ap.add_argument("--composition", action="store_true") + ap.add_argument("--session-shape", action="store_true", + help="SH-1..SH-3 (always shown in the default report)") ap.add_argument("--self-test", action="store_true") ap.add_argument("--json", action="store_true") args = ap.parse_args() diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md index 56660d5..819c0dc 100644 --- a/workplans/CB-WP-0003-loop-hardening.md +++ b/workplans/CB-WP-0003-loop-hardening.md @@ -158,7 +158,7 @@ rather than reporting a clean verify. ```task id: CB-WP-0003-T04 -status: todo +status: done priority: high state_hub_task_id: "382724e6-efd5-4ef8-a438-9d31a65dacdb" ```