CB-WP-0004 T05: control loop — 6 points recovered, not 25-30

cb-cost gains --since, so the baseline (--pin 578dcbe, 662 responses,
$135.60) and this pass (--since 578dcbe, 83 responses, $7.82) are two
disjoint windows over the same transcripts. Every verdict is on share of
pass, since the windows differ 17x in size.

Test 1 — did mechanical turns disappear? Partly. Mechanical share fell
38.2% -> 32.4%. Six points against a predicted 25-30; reported unmet, and
no target was moved in this commit.

  environment setup   11.5% -> 0.6%   (85 turns -> 1)   met, decisively
  hub + workplan      8.5%  -> 0.0%   (46 turns -> 0)   met
  text patching       10.4% -> 9.6%                     not met
  orientation         5.1%  -> 21.2%                    not met, worse

The confound is stated before any defence of the numbers: this is the
pass that built the tools, and the two categories that missed are exactly
the two whose tools were under construction. The clean test is the next
pass, and it is carried forward rather than waived.

Test 2 — did the work relocate? Not into prose. Output tokens per
response fell 896 -> 681. Output's rising share of cost is a shrinking
denominator, not more writing. Cost per response halved and the evidence
refuses to claim it: that is compaction (mean context 232,982 ->
117,822), and attributing it to tooling would repeat CB-WP-0002's
original error in a new direction.

Test 3 — did quality hold? Yes, recorded as explicit judgment. make all
green with two gates that did not exist before, and four findings
surfaced this pass, three caught by controls written this pass — one of
them a 5.2x attribution error that would have reached the hub as a
measured number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 10:27:09 +02:00
parent 53c460c992
commit 7a374d37d2
4 changed files with 202 additions and 3 deletions

View file

@ -64,6 +64,8 @@ def task_label(subject):
return f"{m.group(1)}-{m.group(2)}"
m = TASK_RE.search(subject)
return m.group(0) if m else None
UNATTRIBUTED = "UNATTRIBUTED"
OPEN_REMAINDER = "OPEN (uncommitted)"
@ -362,9 +364,17 @@ def attribute(responses, commits):
# ----------------------------------------------------------------- report
def collect(slug, pin_ref=None):
def collect(slug, pin_ref=None, since_ref=None):
"""Report over (since_ref, pin_ref]. Both ends optional.
`since` exists for CB-WP-0004 T05: comparing a pass against the pass
that measured it needs two disjoint windows over the same transcripts,
not two whole-corpus totals. Attribution and reconciliation are
unchanged the window only selects which responses are counted.
"""
prices = load_prices()
pin = resolve_pin(pin_ref)
since = resolve_pin(since_ref)
paths = transcript_paths(slug)
if not paths:
raise Abort(f"no transcripts found for {slug}")
@ -372,6 +382,8 @@ def collect(slug, pin_ref=None):
responses = []
for p in paths:
responses.extend(read_responses(p, pin))
if since:
responses = [r for r in responses if r["timestamp"] > since]
if not responses:
# Positive control: a run that measured nothing must not report $0.00
# as though it were an answer.
@ -453,6 +465,7 @@ def collect(slug, pin_ref=None):
},
"slug": slug,
"pin": pin,
"since": since,
"responses": len(responses),
"transcripts": len(paths),
"total": total,
@ -494,6 +507,8 @@ def collect(slug, pin_ref=None):
def render(rep, by_task=False, composition=False):
print(f"M-D2-CST cost report — {rep['slug']}")
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
if rep.get("since"):
print(f" since {rep['since']} (window is exclusive of this instant)")
print(f" transcripts {rep['transcripts']} responses {rep['responses']}")
print(f" main ${rep['main_total']:>10,.2f}")
print(f" subagent tree ${rep['subagent_total']:>10,.2f}")
@ -714,6 +729,7 @@ def main():
ap = argparse.ArgumentParser()
ap.add_argument("--slug", default="-home-worsch-clay-borg")
ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)")
ap.add_argument("--since", help="window start, exclusive: commit-ish or ISO Z")
ap.add_argument("--by-task", action="store_true")
ap.add_argument("--composition", action="store_true")
ap.add_argument("--session-shape", action="store_true",
@ -733,7 +749,7 @@ def main():
return budget(args.slug, args.soft, args.hard)
try:
rep = collect(args.slug, args.pin)
rep = collect(args.slug, args.pin, args.since)
except Abort as e:
print(f"ABORT — {e}", file=sys.stderr)
return 1