coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-WP-0013-T01: SH-3 refuses a window that cannot carry a rate

Browse source 
The metric six evidence files reported as 0.0% was never measured.
--shape-budget windows since the last commit, and it is read while
writing the evidence file — right after a commit, when the window holds
one or two responses. SH-1/SH-2 are location statistics and survive n=2.
SH-3 is a rate: at n=2 its only possible values are 0%, 50%, 100%.

cb-cost.py gains sh3_line(), which below a minimum sample prints
"insufficient sample" and no verdict. The floor is derived: if the true
rate were exactly the 20% target, P(zero batched in n) = 0.8^n, and at
n=14 that is 4.4% — so "0 batched in 14" rules out a target-meeting rate
at ~95%. Below that the tool has nothing to say and now says so.

The window was kept rather than split. SH-3 could have been given a
per-pass window, but the budget's purpose is the open remainder since
the last commit, and giving one of three metrics a different window
makes "the window" ambiguous in a tool three specs cite.

Four controls, three mutations, each red for its stated reason —
including the one the evidence files actually hit, where a refusal is
printed as a measured zero.

SessionShape.md §4 carries the correction with the real per-pass figures
(1.1%-6.3%), beside the eighth trusted-arithmetic instance. This is the
ninth, and the second in this same metric. It also shows what the frozen
0.0% hid: against the pinned 7.8%-8.6%, batching has got worse, and six
passes reported a breach that was moving underneath them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-02 07:23:04 +02:00
parent 7105027173
commit 6080c4cf60
3 changed files with 129 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

66
tools/cb-cost.py
View file

@ -624,8 +624,15 @@ def render(rep, by_task=False, composition=False):
f"[{'ok ' if sh['SH-1_mean_context']<=200_000 else 'FAIL'} target 200,000]")
print(f"{indent}SH-2 p90 context {sh['SH-2_p90_context']:>12,.0f} tok "
f"[{'ok ' if sh['SH-2_p90_context']<=300_000 else 'FAIL'} target 300,000]")
print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% "
f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]")
# Same guard as the budget path: a rate a window cannot support
# must not be printed as though it were measured.
if sh["responses_with_tools"] < SH3_MIN_SAMPLE:
print(f"{indent}SH-3 batching rate insufficient sample "
f"[{sh['responses_with_tools']} with tool calls, "
f"need {SH3_MIN_SAMPLE}]")
else:
print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% "
f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]")
print(f"{indent} {sh['tool_calls']} tool calls in "
f"{sh['responses_with_tools']} responses; "
f"{sh['calls_in_batched_turns']} in batched turns")
@ -786,6 +793,27 @@ def self_test():
finally:
os.unlink(empty)
# SH-3 minimum sample (CB-WP-0013 T01). Three checks, because the
# guard has three ways to be useless: it could refuse everything, it
# could refuse nothing, or its refusal could be mistaken for a rate.
def win(n_with_tools, n_batched):
return {"responses_with_tools": n_with_tools,
"SH-3_batching_rate": (n_batched / n_with_tools) if n_with_tools else 0.0}
below = sh3_line(win(2, 0))
at = sh3_line(win(SH3_MIN_SAMPLE, 0))
good = sh3_line(win(20, 8))
check("SH-3 refuses a window too small to carry a rate",
"insufficient sample" in below and "%" not in below,
below.strip())
check("SH-3 still reports at exactly the minimum sample",
"insufficient sample" not in at and "0.0%" in at, at.strip())
check("SH-3 reports a real rate above the floor",
"40.0%" in good and "ok" in good, good.strip())
# The one that matters: a refusal must not read as a measured zero.
check("a refusal is distinguishable from a genuine 0.0%",
below != sh3_line(win(SH3_MIN_SAMPLE, 0)))
# AC-7: the subagent tree is discovered by the path globs.
slug = "-home-worsch-clay-borg"
paths = transcript_paths(slug)
@ -809,6 +837,36 @@ def self_test():
SHAPE_SOFT = {"SH-1": 200_000, "SH-2": 300_000}
SHAPE_HARD = {"SH-1": 300_000, "SH-2": 450_000}
# CB-WP-0013 T01. SH-1 and SH-2 are location statistics and survive a
# two-response window. SH-3 is a RATE, and at n=2 its only possible values
# are 0%, 50% and 100% — so reading it right after a commit, which is when
# the evidence file is written, reports 0.0% almost regardless of
# behaviour. Six evidence files quoted that 0.0% as a measurement; the real
# per-pass figure was 1.1-6.3%.
#
# The floor is derived, not round. If the true rate were exactly the 20%
# target, the chance of observing ZERO batched turns in n responses is
# 0.8^n. At n = 14 that is 4.4%, so "0 batched in 14" rules out a
# target-meeting rate at ~95%. Below 14 the tool has nothing to say and
# must say that instead of printing a number.
SH3_MIN_SAMPLE = 14
def sh3_line(win, min_sample=SH3_MIN_SAMPLE):
"""SH-3, or an explicit refusal when the window cannot support a rate.
Returns the line rather than printing it so the refusal is testable
without capturing stdout — a guard that can only be checked by reading
output is a guard that will be checked by nobody.
"""
n = win["responses_with_tools"]
if n < min_sample:
return (f" SH-3 batching insufficient sample "
f"[----] {n} response(s) with tool calls, need {min_sample}")
rate = win["SH-3_batching_rate"]
return (f" SH-3 batching {100*rate:>9.1f}% "
f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%")
def shape_budget(slug):
"""SH-* for the window since the last commit, with soft/hard verdicts."""
@ -842,9 +900,7 @@ def shape_budget(slug):
print(f" {key} {label} {val:>10,.0f} tok [{mark}] "
f"soft {soft:,} / hard {hard:,}")
breach = max(breach, 0 if val <= soft else (1 if val <= hard else 2))
rate = win["SH-3_batching_rate"]
print(f" SH-3 batching {100*rate:>9.1f}% "
f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%")
print(sh3_line(win))
if breach >= 2:
print("\n HARD — compact before continuing. Context this size costs "