Compare commits
2 commits
50f06a4d69
...
7e21df378a
| Author | SHA1 | Date | |
|---|---|---|---|
| 7e21df378a | |||
| 0c1eb9ecba |
8 changed files with 571 additions and 14 deletions
6
Makefile
6
Makefile
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
CARGO := cargo
|
||||
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all
|
||||
|
||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||
check:
|
||||
|
|
@ -44,6 +44,10 @@ self-tests:
|
|||
cost-budget: cost-test
|
||||
python3 tools/cb-cost.py --budget
|
||||
|
||||
# CB-RES-0003 baseline: mechanical vs judgment turns.
|
||||
cost-mix: cost-test
|
||||
python3 tools/cb-cost.py --composition
|
||||
|
||||
cost-pin: cost-test
|
||||
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
||||
|
||||
|
|
|
|||
88
decisions/ADR-0004-am4-ratification.md
Normal file
88
decisions/ADR-0004-am4-ratification.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# ADR-0004: ratify AM-4a and AM-4b
|
||||
|
||||
status: accepted
|
||||
date: 2026-07-31
|
||||
decided by: maintainer (Bernd Worsch), 2026-07-31
|
||||
tier: S (structural S — ratifies an existing target, creates no capability; chaos d4=3)
|
||||
supersedes: nothing; discharges the open item raised by CB-WP-0003 T07
|
||||
references: [GameKernel.md](../specs/GameKernel.md) §4,
|
||||
[InnerLoop.md](../specs/InnerLoop.md) §Step 4 (correction vs retarget),
|
||||
`history/260731-inner-loop-rule-audit.md`
|
||||
|
||||
## Why this ADR exists
|
||||
|
||||
CB-WP-0003 T07 added a mechanical test separating a **correction** (the
|
||||
instrument disproved the target; the implementation did not change) from a
|
||||
**retarget** (the same commit moved both the target and the code it
|
||||
measures). AM-4a/AM-4b failed that test: they were set at 250,000 and
|
||||
350,000 in commit `4be6e02`, by the implementer, after seeing the measured
|
||||
246,250 — and that commit also changed the feature gating the metric
|
||||
measures.
|
||||
|
||||
The reasoning was recorded at the time and is defensible. The *structure*
|
||||
was not, and the audit flagged both targets as unratified: `make
|
||||
dep-weight` has been enforcing thresholds no reviewed decision stood
|
||||
behind.
|
||||
|
||||
## Decision
|
||||
|
||||
**AM-4a (≤ 250,000 lines, shipped runtime) and AM-4b (≤ 350,000 lines,
|
||||
dev toolchain) are ratified as written.** No values change.
|
||||
|
||||
## The old target, and why it was abandoned
|
||||
|
||||
AM-4 originally read **≤ 20 transitive crates**, set against
|
||||
boardgame.io's 120 npm packages. Retired for two measured reasons:
|
||||
|
||||
1. **Unreachable without undoing this spec's own contracts.** K5 (seeded
|
||||
ChaCha) and K7 (SHA-256) cost 12 crates between them. The measured
|
||||
ladder showed nothing reached 20 except reimplementing one of those
|
||||
primitives — trading an audited cryptographic implementation for a
|
||||
scoreboard number.
|
||||
2. **Crate count does not compare across ecosystems.** Rust splits crates
|
||||
far more finely than npm, so the original 33-vs-120 comparison
|
||||
flattered us while the ≤20 target punished us, for the same reason.
|
||||
|
||||
## The measurement that motivated the change
|
||||
|
||||
At the time of the retarget, `make dep-weight`:
|
||||
|
||||
| configuration | crates | third-party LOC |
|
||||
|---|---|---|
|
||||
| shipped-runtime (`--no-default-features`) | 23 | **246,250** |
|
||||
| dev-toolchain (default features) | 29 | **317,021** |
|
||||
| own source | — | 3,443 |
|
||||
|
||||
## Why these targets bind on future work rather than merely passing present work
|
||||
|
||||
This is the question T07 requires an ADR to answer, and it is the reason
|
||||
ratification is defensible rather than a rubber stamp.
|
||||
|
||||
- **AM-4a leaves 3,750 lines of headroom — 1.5%.** That is roughly one
|
||||
small crate. Any dependency added to the shipped runtime breaches it
|
||||
almost immediately, which is the intended behaviour: the shipped runtime
|
||||
is meant to be effectively frozen, and the target enforces that a new
|
||||
runtime dependency is a decision someone must argue for, not a default.
|
||||
- **AM-4b leaves 32,979 lines — 10.4%.** Deliberately looser. The dev
|
||||
toolchain is where scenario YAML, benchmarking, and future tooling land,
|
||||
and it does not ship to a player. It should be able to absorb one
|
||||
moderate dependency without a spec change, and not two.
|
||||
- **Both are ceilings on a quantity that only grows by choice.** Nothing
|
||||
drifts a project across these thresholds; only adding a dependency does.
|
||||
A target that can only be breached deliberately is a target that binds.
|
||||
|
||||
**What would falsify this ratification:** if a future pass finds itself
|
||||
raising AM-4a to accommodate a dependency it wanted, that is the failure
|
||||
mode the ceiling exists to catch, and the answer is a new ADR arguing for
|
||||
the dependency — not a quiet retarget.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The open item in `history/260731-inner-loop-rule-audit.md` is discharged.
|
||||
- `make dep-weight` continues to fail the build on breach, now backed by a
|
||||
reviewed decision.
|
||||
- AM-4c (own source per 100k third-party lines) remains **reported, not
|
||||
targeted**, and is unaffected.
|
||||
- Precedent: this is the first ADR written under the correction/retarget
|
||||
test. Future retargets follow this shape — old target, the measurement,
|
||||
and an argument about *future* binding rather than present passing.
|
||||
|
|
@ -136,12 +136,12 @@ evidence files.
|
|||
|
||||
## Open items raised by this audit
|
||||
|
||||
- **AM-4a / AM-4b are unratified retargets.** Measured at 246,250 and set
|
||||
at 250,000 in the same commit, by the implementer, with the
|
||||
implementation changing in that commit too — a retarget under the test
|
||||
added to InnerLoop §Step 4 by T07, not a correction. They must be
|
||||
ratified by ADR or changed. Until then `make dep-weight` is enforcing a
|
||||
target no reviewed decision stands behind.
|
||||
- ~~**AM-4a / AM-4b are unratified retargets.**~~ **Discharged
|
||||
2026-07-31** by [ADR-0004](../decisions/ADR-0004-am4-ratification.md),
|
||||
ratified by the maintainer. Values unchanged; the ADR supplies the
|
||||
future-binding argument the test requires (AM-4a leaves 1.5% headroom,
|
||||
AM-4b 10.4%, and both measure a quantity that only grows by deliberate
|
||||
choice).
|
||||
|
||||
## Class coverage — where the gaps are
|
||||
|
||||
|
|
|
|||
180
research/CB-RES-0003-agent-vs-deterministic.md
Normal file
180
research/CB-RES-0003-agent-vs-deterministic.md
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
# CB-RES-0003: which agent turns can become deterministic compute
|
||||
|
||||
capability: meta.loop.mechanical-work
|
||||
status: draft
|
||||
tier: M (structural M — no new capability port; touches the one command
|
||||
surface and the cost instrument; chaos d4=2 → no override)
|
||||
instrument: `make cost-mix` (`tools/cb-cost.py`, tool-mix block)
|
||||
|
||||
Review of where token-priced agent turns did work a deterministic tool
|
||||
could have done, so that capacity moves to judgment rather than mechanics.
|
||||
Commissioned 2026-07-31.
|
||||
|
||||
**Method.** Every turn in both clay-borg session transcripts (573 responses,
|
||||
$134 total) was classified by the tool calls it made. The classifier is
|
||||
committed as `classify_tool()` in `tools/cb-cost.py` and its output is
|
||||
emitted by `make cost-mix` — the numbers below are reproducible, and the
|
||||
same command measures whether any fix worked.
|
||||
|
||||
---
|
||||
|
||||
## The measurement
|
||||
|
||||
```text
|
||||
mech environment setup 84 turns $ 15.33
|
||||
mech ad-hoc text patching 75 turns $ 13.86
|
||||
git 37 turns $ 13.85
|
||||
mech hub task status 25 turns $ 7.46
|
||||
mech orientation / inspect 49 turns $ 6.87
|
||||
hub other 32 turns $ 6.22
|
||||
mech ad-hoc transcript analysis 39 turns $ 4.56
|
||||
mech workplan status edit 21 turns $ 4.06
|
||||
make (gates) 3 turns $ 1.82
|
||||
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
|
||||
```
|
||||
|
||||
**38% of spend went through turns whose tool calls were mechanical.**
|
||||
|
||||
A turn's whole cost is charged to every category it touched, so the rows
|
||||
overlap and the deduplicated line is the honest total. `git` and `make`
|
||||
are excluded from *mechanical* deliberately: a commit message is judgment,
|
||||
and running a gate is the point.
|
||||
|
||||
## Candidate 1 — environment setup: 84 turns, $15.33
|
||||
|
||||
The single largest category, and pure friction. `cd` (51) and
|
||||
`export PATH="$HOME/.cargo/bin:$PATH"` (10+) recur because the shell does
|
||||
not persist state between calls and `cargo` is not on the default path.
|
||||
Every `make` invocation that touches Rust had to be prefixed.
|
||||
|
||||
- **Deterministic replacement:** put `~/.cargo/bin` on the path the agent
|
||||
starts with, and make every `make` target self-sufficient (the Makefile
|
||||
already knows where it is). `tools/dep-weight.py` already special-cases
|
||||
this with a helpful error — evidence the friction was noticed and
|
||||
patched at the leaf instead of the root.
|
||||
- **Expected effect:** these turns do not become cheaper, they **stop
|
||||
existing**. Predicted saving: **~$12–15 per two-session pass**, the
|
||||
cleanest win in this review.
|
||||
- **Risk:** near zero. Nothing depends on the current behaviour.
|
||||
|
||||
## Candidate 2 — ad-hoc text patching: 75 turns, $13.86
|
||||
|
||||
Inline `python3 - <<'PY'` heredocs that open a markdown file, string-replace,
|
||||
and write it back. Written fresh each time, unreviewed, and the mechanism
|
||||
behind the **duplicated-fact-drift** class: a number changes and every
|
||||
artifact quoting it must be found and patched by hand. The acceptance
|
||||
figure moved four times and each move cost a sweep across three or four
|
||||
files.
|
||||
|
||||
- **Deterministic replacement:** a *fact registry*. Numbers that appear in
|
||||
more than one artifact are declared once (`facts.toml` or emitted by the
|
||||
tool that measures them) and injected into markdown by a generator, with
|
||||
`make facts-check` failing when a committed artifact disagrees with the
|
||||
registry.
|
||||
- **Expected effect:** removes the sweep, and gives the DFD class its
|
||||
first executable gate — which InnerLoop v1.2 currently states only as
|
||||
prose. Predicted saving: **~$6–9 per pass**, plus the error class.
|
||||
- **Risk:** moderate. A registry that is itself hand-maintained just moves
|
||||
the problem. It must be generated from instruments where possible, and
|
||||
the check must be the enforcement, not the generation.
|
||||
|
||||
## Candidate 3 — hub task status: 25 turns, $7.46
|
||||
|
||||
Every `update_task_status` call was hand-written, including
|
||||
`tokens_in`/`tokens_out` — **which were estimates I typed**, in a session
|
||||
whose entire subject was that estimated token counts are worthless. The
|
||||
hub holds numbers derived from the very habit CB-WP-0002 disproved.
|
||||
|
||||
- **Deterministic replacement:** `make task-done T=T05` — flips the
|
||||
workplan file, reads the measured cost for that task from `cb-cost`, and
|
||||
pushes the hub event with real numbers. One command replaces an edit, a
|
||||
status call, and a fabricated figure.
|
||||
- **Expected effect:** **~$7 per pass**, and the hub stops holding
|
||||
fiction. Combines with candidate 5.
|
||||
- **Risk:** low. `cb-cost --by-task` already produces the figure.
|
||||
|
||||
## Candidate 4 — orientation / inspect: 49 turns, $6.87
|
||||
|
||||
`grep`/`ls`/`wc` to answer "what is the state of this repo" — which
|
||||
workplan is active, which tasks are open, which gates pass, what is
|
||||
uncommitted.
|
||||
|
||||
- **Deterministic replacement:** `make status` printing the loop state in
|
||||
one shot: active workplan, task counts, gate results, open cost since
|
||||
last commit, provisional item ages.
|
||||
- **Expected effect:** turns ~10 orientation turns into 1 at the start of
|
||||
a session, and shrinks cold-start context (SS-04). Predicted saving:
|
||||
**~$4 per pass**.
|
||||
- **Risk:** low, but the saving is softer than it looks — some inspection
|
||||
is genuinely exploratory and will not disappear.
|
||||
|
||||
## Candidate 5 — workplan status edit: 21 turns, $4.06
|
||||
|
||||
Heredocs doing `s.replace("status: todo", "status: done")` on a workplan
|
||||
file. Purely mechanical, and error-prone: it silently does nothing if the
|
||||
task is already done or the ID is mistyped.
|
||||
|
||||
- **Deterministic replacement:** folded into candidate 3's
|
||||
`make task-done`, which can *fail* on an unknown task ID instead of
|
||||
no-op'ing.
|
||||
- **Expected effect:** **~$4 per pass** and one class of silent no-op
|
||||
removed.
|
||||
|
||||
## Candidate 6 — ad-hoc transcript analysis: 39 turns, $4.56
|
||||
|
||||
Already partly solved: `cb-cost` subsumed most of this during CB-WP-0002,
|
||||
and this review's own classifier is now committed rather than ad-hoc. The
|
||||
residue is one-off questions (context percentiles, compaction boundaries,
|
||||
per-model splits) that were each written fresh.
|
||||
|
||||
- **Deterministic replacement:** promote the recurring ones to flags. Most
|
||||
already exist (`--by-task`, `--composition`, session shape, tool mix).
|
||||
- **Expected effect:** **~$2 per pass**, diminishing. Listed for
|
||||
completeness, not priority.
|
||||
|
||||
## What must NOT be automated
|
||||
|
||||
Stating this because a review that only finds savings is not a review.
|
||||
|
||||
- **`git` (37 turns, $13.85)** is the second-most expensive category and
|
||||
is mostly *commit message authorship* — the highest-output-token turns
|
||||
in the corpus. That output is the project's reasoning record. Automating
|
||||
it would save money and destroy the thing that makes corrections cheap.
|
||||
- **`make` gates (3 turns, $1.82)** are already deterministic; the agent
|
||||
merely invokes them. Correctly cheap.
|
||||
- **Judgment work is invisible in this table** — writing a spec, choosing
|
||||
an attribution model, deciding a target is legitimate. That is where the
|
||||
remaining 62% went, and it is what the freed capacity should buy.
|
||||
|
||||
## Verdict and expected total
|
||||
|
||||
| # | candidate | turns | measured | predicted saving/pass | confidence |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | environment setup | 84 | $15.33 | **$12–15** | high |
|
||||
| 2 | fact registry (text patching) | 75 | $13.86 | **$6–9** | medium |
|
||||
| 3+5 | `make task-done` | 46 | $11.52 | **$9–11** | high |
|
||||
| 4 | `make status` | 49 | $6.87 | **$4** | medium |
|
||||
| 6 | cb-cost flags | 39 | $4.56 | **$2** | low |
|
||||
| | **total** | | **$51.26** | **$33–41** | |
|
||||
|
||||
Predicted recovery is **25–30% of a pass**, against a measured 38%
|
||||
mechanical share. The gap is deliberate: some inspection and some patching
|
||||
is genuinely exploratory and will not vanish.
|
||||
|
||||
**The prediction is falsifiable and must be checked.** `make cost-mix`
|
||||
emits the same categories, so the next pass measures whether these turns
|
||||
disappeared or merely relocated — the failure mode being that an agent
|
||||
which no longer writes heredocs simply writes more prose instead. A
|
||||
control loop that does not test for relocation is not a control loop.
|
||||
|
||||
## Risks in this review itself
|
||||
|
||||
- **A turn's cost is charged to every category it touched**, so per-row
|
||||
figures overstate. The deduplicated $51.26 is the defensible number;
|
||||
per-candidate savings are apportioned from it and are estimates.
|
||||
- **n = 2 sessions, one repo, one agent.** The mix is a property of how
|
||||
this project was built, not a general law.
|
||||
- **The classifier is a regex over shell commands.** It cannot see intent:
|
||||
a `cd` that precedes real work is charged the whole turn. This inflates
|
||||
candidate 1, which is why its predicted saving is below its measured
|
||||
cost despite being the most certain fix.
|
||||
|
|
@ -135,7 +135,10 @@ Command (actor-tagged intent)
|
|||
|
||||
## 4. Acceptance metrics
|
||||
|
||||
**On AM-4's retarget (2026-07-31).** AM-4 originally read "≤20
|
||||
**On AM-4's retarget (2026-07-31).** *Ratified by
|
||||
[ADR-0004](../decisions/ADR-0004-am4-ratification.md) on 2026-07-31; the
|
||||
headroom argument for why these ceilings bind on future work lives there.*
|
||||
AM-4 originally read "≤20
|
||||
transitive crates", set against boardgame.io's 120 npm packages. That
|
||||
target was retired for two measured reasons. First, it was unreachable
|
||||
without undoing this spec's own contracts: K5 (seeded ChaCha) and K7
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -186,12 +186,13 @@ def read_responses(path, pin=None):
|
|||
head = rows[0]
|
||||
# Tool calls are spread across the group's lines, so they are counted
|
||||
# over the whole group — one response may carry several (SS-05).
|
||||
tool_calls = sum(
|
||||
1
|
||||
for r in rows
|
||||
for c in (r["message"].get("content") or [])
|
||||
if c.get("type") == "tool_use"
|
||||
)
|
||||
blocks = [c for r in rows for c in (r["message"].get("content") or [])
|
||||
if c.get("type") == "tool_use"]
|
||||
tool_calls = len(blocks)
|
||||
cats = sorted({
|
||||
c for c in (classify_tool(b.get("name"), b.get("input") or {})
|
||||
for b in blocks) if c
|
||||
})
|
||||
out.append(
|
||||
{
|
||||
"request_id": rid,
|
||||
|
|
@ -200,12 +201,51 @@ def read_responses(path, pin=None):
|
|||
"session": head.get("sessionId") or os.path.basename(path),
|
||||
"toks": toks,
|
||||
"tool_calls": tool_calls,
|
||||
"categories": cats,
|
||||
"subagent": "/subagents/" in path,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# CB-RES-0003: which turns are mechanical (a deterministic tool could do
|
||||
# them) versus judgment (only an agent can). The baseline for measuring
|
||||
# whether automation actually removes turns rather than relocating them.
|
||||
def classify_tool(name, inp):
|
||||
if name == "mcp__dev-hub__update_task_status":
|
||||
return "hub task status"
|
||||
if name.startswith("mcp__dev-hub__"):
|
||||
return "hub other"
|
||||
if name != "Bash":
|
||||
return None
|
||||
cmd = (inp.get("command") or "").strip()
|
||||
if not cmd:
|
||||
return None
|
||||
if "python3 - <<" in cmd:
|
||||
if "status: todo" in cmd or "status: done" in cmd:
|
||||
return "workplan status edit"
|
||||
if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd:
|
||||
return "ad-hoc transcript analysis"
|
||||
return "ad-hoc text patching"
|
||||
if cmd.startswith(("cd ", "export ")):
|
||||
return "environment setup"
|
||||
if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"):
|
||||
return "orientation / inspect"
|
||||
if cmd.startswith("git "):
|
||||
return "git"
|
||||
if "make " in cmd:
|
||||
return "make (gates)"
|
||||
return None
|
||||
|
||||
|
||||
# Categories a deterministic tool could plausibly own. Judgment-bearing
|
||||
# categories (git commit messages, make gates) are excluded deliberately.
|
||||
MECHANICAL = frozenset({
|
||||
"environment setup", "ad-hoc text patching", "orientation / inspect",
|
||||
"ad-hoc transcript analysis", "hub task status", "workplan status edit",
|
||||
})
|
||||
|
||||
|
||||
def session_shape(responses):
|
||||
"""SH-1..SH-3 from specs/SessionShape.md."""
|
||||
import statistics
|
||||
|
|
@ -361,9 +401,25 @@ def collect(slug, pin_ref=None):
|
|||
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
|
||||
)
|
||||
|
||||
# Tool mix: a turn's whole cost is charged to each category it touched,
|
||||
# so columns may overlap and must not be summed as if disjoint.
|
||||
mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float)
|
||||
for r in responses:
|
||||
for c in r.get("categories") or []:
|
||||
mix_turns[c] += 1
|
||||
mix_cost[c] += r["cost"] or 0.0
|
||||
mech = [r for r in responses
|
||||
if set(r.get("categories") or []) & MECHANICAL]
|
||||
|
||||
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
|
||||
return {
|
||||
"session_shape": session_shape(responses),
|
||||
"tool_mix": {
|
||||
"turns": dict(mix_turns),
|
||||
"cost": dict(mix_cost),
|
||||
"mechanical_turns": len(mech),
|
||||
"mechanical_cost": sum(r["cost"] or 0 for r in mech),
|
||||
},
|
||||
"slug": slug,
|
||||
"pin": pin,
|
||||
"responses": len(responses),
|
||||
|
|
@ -415,6 +471,18 @@ def render(rep, by_task=False, composition=False):
|
|||
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
|
||||
)
|
||||
|
||||
mix = rep["tool_mix"]
|
||||
if mix["turns"]:
|
||||
print("\n tool mix — a turn's cost is charged to every category it")
|
||||
print(" touched, so columns overlap and must not be summed")
|
||||
for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]):
|
||||
tag = "mech" if k in MECHANICAL else " "
|
||||
print(f" {tag} {k:<28}{mix['turns'][k]:>5} turns "
|
||||
f"${mix['cost'][k]:>8,.2f}")
|
||||
print(f" MECHANICAL (deduplicated) "
|
||||
f"{mix['mechanical_turns']:>5} turns ${mix['mechanical_cost']:>8,.2f}"
|
||||
f" = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass")
|
||||
|
||||
sh = rep["session_shape"]
|
||||
print("\n session shape (specs/SessionShape.md)")
|
||||
print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok "
|
||||
|
|
|
|||
214
workplans/CB-WP-0004-mechanical-work.md
Normal file
214
workplans/CB-WP-0004-mechanical-work.md
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
---
|
||||
id: CB-WP-0004
|
||||
title: "Move mechanical turns off the token budget, and prove it worked"
|
||||
status: proposed
|
||||
state_hub_workstream_id: ""
|
||||
---
|
||||
|
||||
# Purpose
|
||||
|
||||
`research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in
|
||||
both clay-borg sessions by the tool calls it made:
|
||||
|
||||
```text
|
||||
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
|
||||
```
|
||||
|
||||
**38% of spend went through turns doing work a deterministic tool could
|
||||
do.** The largest single category is `cd` and `export PATH` — 84 turns and
|
||||
$15.33 of pure environment friction. The second is inline heredocs
|
||||
string-patching markdown, which is also the mechanism behind the
|
||||
duplicated-fact-drift error class InnerLoop v1.2 named and could not gate.
|
||||
|
||||
This workplan converts the five worthwhile categories to classic compute
|
||||
and **measures whether that actually recovered anything**. The predicted
|
||||
recovery is $33–41 per pass, 25–30%.
|
||||
|
||||
**The control loop is the point, not a formality.** The named failure mode
|
||||
is *relocation*: an agent that can no longer write a heredoc may simply
|
||||
write more prose, and the pass costs the same. `make cost-mix` emits the
|
||||
same categories that produced the baseline, so the claim is falsifiable by
|
||||
the same instrument that made it. A saving that cannot be demonstrated in
|
||||
that table did not happen.
|
||||
|
||||
Per InnerLoop v1.2, targets here are **provisional until the instrument
|
||||
emits them**, and no target may be moved in the commit that measures it
|
||||
unless the instrument disproved it (§Step 4, correction vs retarget).
|
||||
|
||||
## Phase A — The certain wins
|
||||
|
||||
## Task: Remove environment friction
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T01
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
84 turns / $15.33, the largest category and the least interesting work in
|
||||
the corpus. `cargo` is not on the default path, so every Rust-touching
|
||||
command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does
|
||||
not persist `cd`.
|
||||
|
||||
Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases
|
||||
the missing `cargo` with a helpful error, which is evidence the friction
|
||||
was noticed and patched in the wrong place.
|
||||
|
||||
Deliver: every `make` target runs from a clean shell with no prefix, from
|
||||
any directory. Document the one-line environment requirement in
|
||||
`README.md` if one remains. Remove the leaf workaround in `dep-weight.py`
|
||||
only if it becomes unreachable — a positive control that never fires is
|
||||
still cheaper than a regression.
|
||||
|
||||
**Predicted:** environment-setup turns → **< 10** (from 84),
|
||||
**$12–15** recovered. Highest confidence in the review.
|
||||
|
||||
## Task: `make task-done` — one command for a task close
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T02
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
Merges two categories: workplan status edits (21 turns, $4.06) and hub
|
||||
task-status calls (25 turns, $7.46).
|
||||
|
||||
`make task-done T=CB-WP-0004-T02` must:
|
||||
|
||||
1. flip `status: todo` → `done` in the workplan file, **failing loudly on
|
||||
an unknown or already-done task** — the heredocs it replaces silently
|
||||
no-op'd on a typo;
|
||||
2. read that task's **measured** cost and tokens from `cb-cost --by-task`;
|
||||
3. push the hub event with the real numbers.
|
||||
|
||||
The third point is the one that matters beyond cost. Every
|
||||
`update_task_status` in this project so far carried **hand-typed token
|
||||
estimates**, in a repo whose central finding is that estimated token counts
|
||||
are worthless. The hub currently holds fiction produced by the exact habit
|
||||
CB-WP-0002 disproved.
|
||||
|
||||
**Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub
|
||||
stops holding estimates. Add `--self-test` per InnerLoop v1.1.
|
||||
|
||||
## Task: `make status` — one-shot orientation
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this
|
||||
repo". Replace with one command printing: active workplan and task counts,
|
||||
gate results, open spend since the last commit (CB-01), provisional item
|
||||
ages, and any `loop-lint` findings.
|
||||
|
||||
Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures
|
||||
at ~51k for a fresh session — the artifacts a new session reads to orient
|
||||
are exactly what this prints.
|
||||
|
||||
**Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered.
|
||||
Confidence medium: some inspection is genuinely exploratory and will not
|
||||
disappear, and the review says so.
|
||||
|
||||
## Phase B — The one that also closes an error class
|
||||
|
||||
## Task: Fact registry and `make facts-check`
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T04
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
75 turns / $13.86 of heredocs opening a markdown file, string-replacing a
|
||||
number, and writing it back — the mechanism behind **duplicated-fact
|
||||
drift**, the fourth error class, which InnerLoop v1.2 states as prose and
|
||||
cannot currently gate.
|
||||
|
||||
Two instances on record: a price sheet inlined into a spec went stale
|
||||
within an hour of the real sheet changing, and the acceptance figure
|
||||
$92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a
|
||||
workplan, and an evidence file on every move.
|
||||
|
||||
Deliver a registry where a number appearing in more than one artifact is
|
||||
declared once — **generated by the instrument that measures it wherever
|
||||
possible**, not hand-maintained — plus `make facts-check` failing when a
|
||||
committed artifact disagrees with it.
|
||||
|
||||
**The trap to avoid, stated up front:** a hand-maintained registry moves
|
||||
the problem rather than solving it, and would itself become a copy that
|
||||
drifts. If generation from instruments proves impractical, deliver only
|
||||
the *check* (detect the same number stated differently in two artifacts)
|
||||
and say so — a gate with no generator still closes the class.
|
||||
|
||||
**Predicted:** **$6–9** recovered, plus DFD's first executable gate.
|
||||
Confidence medium; this is the hardest task here and the most valuable.
|
||||
|
||||
## Phase C — Prove it, or withdraw the claim
|
||||
|
||||
## Task: Control loop — measure recovery and test for relocation
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T05
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
The task this workplan exists for. Run `make cost-mix` over the sessions
|
||||
that executed T01–T04 and commit
|
||||
`evidence/CB-EV-0003-mechanical-work.md` comparing against the committed
|
||||
baseline:
|
||||
|
||||
| category | baseline turns | baseline $ | predicted | measured | verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| environment setup | 84 | $15.33 | <10 turns | | |
|
||||
| ad-hoc text patching | 75 | $13.86 | $6–9 saved | | |
|
||||
| hub task status + workplan edit | 46 | $11.52 | ~6 turns | | |
|
||||
| orientation / inspect | 49 | $6.87 | $4 saved | | |
|
||||
| **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | |
|
||||
|
||||
**Three tests, all of which must be reported:**
|
||||
|
||||
1. **Did the mechanical turns disappear?** Per-category, against
|
||||
prediction. An unmet prediction is reported unmet, not retargeted.
|
||||
2. **Did they relocate?** Total pass cost and non-mechanical turn counts
|
||||
must be compared too. If mechanical turns fell and prose turns rose by
|
||||
as much, the saving is zero and this workplan failed — that is the
|
||||
result to publish.
|
||||
3. **Did quality hold?** `make all` green, and the same class of findings
|
||||
still surfacing. A cheaper pass that catches fewer errors is worse, and
|
||||
the loop has no metric for this yet — record the judgment explicitly
|
||||
rather than implying the cost number settles it.
|
||||
|
||||
Normalize per unit of work, not per session: passes differ in size, so
|
||||
report **mechanical share of pass cost** (baseline: 38%) alongside
|
||||
absolute dollars.
|
||||
|
||||
## Task: Retrospective
|
||||
|
||||
```task
|
||||
id: CB-WP-0004-T06
|
||||
status: todo
|
||||
priority: low
|
||||
state_hub_task_id: ""
|
||||
```
|
||||
|
||||
The question to answer honestly: **does converting agent work to
|
||||
deterministic tooling actually recover capacity, or does the work
|
||||
reappear elsewhere?**
|
||||
|
||||
This is the first pass in the project to make a *quantitative prediction
|
||||
before acting*. Whether the prediction held is more informative than the
|
||||
saving itself — a loop that can forecast its own economics can plan; one
|
||||
that cannot is guessing with numbers attached.
|
||||
|
||||
Record the prediction error per candidate, and whether the review's stated
|
||||
confidence levels (high/medium/low) tracked reality. If they did not, the
|
||||
next review should stop stating confidence, or state it differently.
|
||||
Loading…
Add table
Add a link
Reference in a new issue