CB-RES-0003 + CB-WP-0004: 38% of pass cost is mechanical turns

Review of where token-priced turns did work a deterministic tool could
do. Method: classify every turn in both transcripts by the tool calls it
made. The classifier is committed in tools/cb-cost.py and emitted by
`make cost-mix`, so the baseline is reproducible and the same command
can later falsify the predictions.

  mech environment setup       84 turns  $15.33
  mech ad-hoc text patching    75 turns  $13.86
       git                     37 turns  $13.85
  mech hub task status         25 turns  $ 7.46
  mech orientation / inspect   49 turns  $ 6.87
       hub other               32 turns  $ 6.22
  mech ad-hoc transcript       39 turns  $ 4.56
  mech workplan status edit    21 turns  $ 4.06
  MECHANICAL (dedup)          290 turns  $51.26  = 38% of pass

Largest category is `cd` and `export PATH` -- pure friction, and
dep-weight.py already patched it at the leaf, which is evidence it was
noticed and fixed in the wrong place. Second is heredocs string-patching
markdown, which is also the mechanism behind duplicated-fact drift, the
error class InnerLoop v1.2 names and cannot gate.

Explicitly NOT automated: git (37 turns, $13.85) is mostly commit
message authorship -- the highest-output turns in the corpus and the
project's reasoning record. Automating it would save money and destroy
what makes corrections cheap.

CB-WP-0004 implements five candidates and predicts $33-41 recovery
(25-30%), below the 38% measured share on purpose: some inspection and
patching is genuinely exploratory.

The control loop is the deliverable, not a formality. T05 tests three
things and must report all: did mechanical turns disappear, did they
RELOCATE into prose, and did quality hold. If mechanical turns fall and
prose rises by as much, the saving is zero and that is the result to
publish.

Also a self-indictment worth recording: every hub update_task_status in
this project carried hand-typed token estimates, in a repo whose central
finding is that estimated token counts are worthless. T02 fixes it by
reading measured values from cb-cost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tools/cb-cost.py

@@ -186,12 +186,13 @@ def read_responses(path, pin=None):
         head = rows[0]
         # Tool calls are spread across the group's lines, so they are counted
         # over the whole group — one response may carry several (SS-05).
-        tool_calls = sum(
-            1
-            for r in rows
-            for c in (r["message"].get("content") or [])
-            if c.get("type") == "tool_use"
-        )
+        blocks = [c for r in rows for c in (r["message"].get("content") or [])
+                  if c.get("type") == "tool_use"]
+        tool_calls = len(blocks)
+        cats = sorted({
+            c for c in (classify_tool(b.get("name"), b.get("input") or {})
+                        for b in blocks) if c
+        })
         out.append(
             {
                 "request_id": rid,
@@ -200,12 +201,51 @@ def read_responses(path, pin=None):
                 "session": head.get("sessionId") or os.path.basename(path),
                 "toks": toks,
                 "tool_calls": tool_calls,
+                "categories": cats,
                 "subagent": "/subagents/" in path,
             }
         )
     return out
 
 
+# CB-RES-0003: which turns are mechanical (a deterministic tool could do
+# them) versus judgment (only an agent can). The baseline for measuring
+# whether automation actually removes turns rather than relocating them.
+def classify_tool(name, inp):
+    if name == "mcp__dev-hub__update_task_status":
+        return "hub task status"
+    if name.startswith("mcp__dev-hub__"):
+        return "hub other"
+    if name != "Bash":
+        return None
+    cmd = (inp.get("command") or "").strip()
+    if not cmd:
+        return None
+    if "python3 - <<" in cmd:
+        if "status: todo" in cmd or "status: done" in cmd:
+            return "workplan status edit"
+        if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd:
+            return "ad-hoc transcript analysis"
+        return "ad-hoc text patching"
+    if cmd.startswith(("cd ", "export ")):
+        return "environment setup"
+    if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"):
+        return "orientation / inspect"
+    if cmd.startswith("git "):
+        return "git"
+    if "make " in cmd:
+        return "make (gates)"
+    return None
+
+
+# Categories a deterministic tool could plausibly own. Judgment-bearing
+# categories (git commit messages, make gates) are excluded deliberately.
+MECHANICAL = frozenset({
+    "environment setup", "ad-hoc text patching", "orientation / inspect",
+    "ad-hoc transcript analysis", "hub task status", "workplan status edit",
+})
+
+
 def session_shape(responses):
     """SH-1..SH-3 from specs/SessionShape.md."""
     import statistics
@@ -361,9 +401,25 @@ def collect(slug, pin_ref=None):
             f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
         )
 
+    # Tool mix: a turn's whole cost is charged to each category it touched,
+    # so columns may overlap and must not be summed as if disjoint.
+    mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float)
+    for r in responses:
+        for c in r.get("categories") or []:
+            mix_turns[c] += 1
+            mix_cost[c] += r["cost"] or 0.0
+    mech = [r for r in responses
+            if set(r.get("categories") or []) & MECHANICAL]
+
     sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
     return {
         "session_shape": session_shape(responses),
+        "tool_mix": {
+            "turns": dict(mix_turns),
+            "cost": dict(mix_cost),
+            "mechanical_turns": len(mech),
+            "mechanical_cost": sum(r["cost"] or 0 for r in mech),
+        },
         "slug": slug,
         "pin": pin,
         "responses": len(responses),
@@ -415,6 +471,18 @@ def render(rep, by_task=False, composition=False):
             f"    A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
         )
 
+    mix = rep["tool_mix"]
+    if mix["turns"]:
+        print("\n  tool mix — a turn's cost is charged to every category it")
+        print("  touched, so columns overlap and must not be summed")
+        for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]):
+            tag = "mech" if k in MECHANICAL else "    "
+            print(f"    {tag} {k:<28}{mix['turns'][k]:>5} turns  "
+                  f"${mix['cost'][k]:>8,.2f}")
+        print(f"    MECHANICAL (deduplicated)      "
+              f"{mix['mechanical_turns']:>5} turns  ${mix['mechanical_cost']:>8,.2f}"
+              f"  = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass")
+
     sh = rep["session_shape"]
     print("\n  session shape (specs/SessionShape.md)")
     print(f"    SH-1 mean context   {sh['SH-1_mean_context']:>12,.0f} tok  "