CB-WP-0004 T05: control loop — 6 points recovered, not 25-30
cb-cost gains --since, so the baseline (--pin578dcbe, 662 responses, $135.60) and this pass (--since578dcbe, 83 responses, $7.82) are two disjoint windows over the same transcripts. Every verdict is on share of pass, since the windows differ 17x in size. Test 1 — did mechanical turns disappear? Partly. Mechanical share fell 38.2% -> 32.4%. Six points against a predicted 25-30; reported unmet, and no target was moved in this commit. environment setup 11.5% -> 0.6% (85 turns -> 1) met, decisively hub + workplan 8.5% -> 0.0% (46 turns -> 0) met text patching 10.4% -> 9.6% not met orientation 5.1% -> 21.2% not met, worse The confound is stated before any defence of the numbers: this is the pass that built the tools, and the two categories that missed are exactly the two whose tools were under construction. The clean test is the next pass, and it is carried forward rather than waived. Test 2 — did the work relocate? Not into prose. Output tokens per response fell 896 -> 681. Output's rising share of cost is a shrinking denominator, not more writing. Cost per response halved and the evidence refuses to claim it: that is compaction (mean context 232,982 -> 117,822), and attributing it to tooling would repeat CB-WP-0002's original error in a new direction. Test 3 — did quality hold? Yes, recorded as explicit judgment. make all green with two gates that did not exist before, and four findings surfaced this pass, three caught by controls written this pass — one of them a 5.2x attribution error that would have reached the hub as a measured number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
53c460c992
commit
7a374d37d2
4 changed files with 202 additions and 3 deletions
Binary file not shown.
|
|
@ -64,6 +64,8 @@ def task_label(subject):
|
|||
return f"{m.group(1)}-{m.group(2)}"
|
||||
m = TASK_RE.search(subject)
|
||||
return m.group(0) if m else None
|
||||
|
||||
|
||||
UNATTRIBUTED = "UNATTRIBUTED"
|
||||
OPEN_REMAINDER = "OPEN (uncommitted)"
|
||||
|
||||
|
|
@ -362,9 +364,17 @@ def attribute(responses, commits):
|
|||
# ----------------------------------------------------------------- report
|
||||
|
||||
|
||||
def collect(slug, pin_ref=None):
|
||||
def collect(slug, pin_ref=None, since_ref=None):
|
||||
"""Report over (since_ref, pin_ref]. Both ends optional.
|
||||
|
||||
`since` exists for CB-WP-0004 T05: comparing a pass against the pass
|
||||
that measured it needs two disjoint windows over the same transcripts,
|
||||
not two whole-corpus totals. Attribution and reconciliation are
|
||||
unchanged — the window only selects which responses are counted.
|
||||
"""
|
||||
prices = load_prices()
|
||||
pin = resolve_pin(pin_ref)
|
||||
since = resolve_pin(since_ref)
|
||||
paths = transcript_paths(slug)
|
||||
if not paths:
|
||||
raise Abort(f"no transcripts found for {slug}")
|
||||
|
|
@ -372,6 +382,8 @@ def collect(slug, pin_ref=None):
|
|||
responses = []
|
||||
for p in paths:
|
||||
responses.extend(read_responses(p, pin))
|
||||
if since:
|
||||
responses = [r for r in responses if r["timestamp"] > since]
|
||||
if not responses:
|
||||
# Positive control: a run that measured nothing must not report $0.00
|
||||
# as though it were an answer.
|
||||
|
|
@ -453,6 +465,7 @@ def collect(slug, pin_ref=None):
|
|||
},
|
||||
"slug": slug,
|
||||
"pin": pin,
|
||||
"since": since,
|
||||
"responses": len(responses),
|
||||
"transcripts": len(paths),
|
||||
"total": total,
|
||||
|
|
@ -494,6 +507,8 @@ def collect(slug, pin_ref=None):
|
|||
def render(rep, by_task=False, composition=False):
|
||||
print(f"M-D2-CST cost report — {rep['slug']}")
|
||||
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
|
||||
if rep.get("since"):
|
||||
print(f" since {rep['since']} (window is exclusive of this instant)")
|
||||
print(f" transcripts {rep['transcripts']} responses {rep['responses']}")
|
||||
print(f" main ${rep['main_total']:>10,.2f}")
|
||||
print(f" subagent tree ${rep['subagent_total']:>10,.2f}")
|
||||
|
|
@ -714,6 +729,7 @@ def main():
|
|||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--slug", default="-home-worsch-clay-borg")
|
||||
ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)")
|
||||
ap.add_argument("--since", help="window start, exclusive: commit-ish or ISO Z")
|
||||
ap.add_argument("--by-task", action="store_true")
|
||||
ap.add_argument("--composition", action="store_true")
|
||||
ap.add_argument("--session-shape", action="store_true",
|
||||
|
|
@ -733,7 +749,7 @@ def main():
|
|||
return budget(args.slug, args.soft, args.hard)
|
||||
|
||||
try:
|
||||
rep = collect(args.slug, args.pin)
|
||||
rep = collect(args.slug, args.pin, args.since)
|
||||
except Abort as e:
|
||||
print(f"ABORT — {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue