Compare commits
No commits in common. "7e21df378a3ab792906a1c68be40cfc73a023e72" and "50f06a4d69b964b5e01bc512ece1c188fc675291" have entirely different histories.
7e21df378a
...
50f06a4d69
8 changed files with 14 additions and 571 deletions
6
Makefile
6
Makefile
|
|
@ -3,7 +3,7 @@
|
||||||
|
|
||||||
CARGO := cargo
|
CARGO := cargo
|
||||||
|
|
||||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all
|
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all
|
||||||
|
|
||||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||||
check:
|
check:
|
||||||
|
|
@ -44,10 +44,6 @@ self-tests:
|
||||||
cost-budget: cost-test
|
cost-budget: cost-test
|
||||||
python3 tools/cb-cost.py --budget
|
python3 tools/cb-cost.py --budget
|
||||||
|
|
||||||
# CB-RES-0003 baseline: mechanical vs judgment turns.
|
|
||||||
cost-mix: cost-test
|
|
||||||
python3 tools/cb-cost.py --composition
|
|
||||||
|
|
||||||
cost-pin: cost-test
|
cost-pin: cost-test
|
||||||
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,88 +0,0 @@
|
||||||
# ADR-0004: ratify AM-4a and AM-4b
|
|
||||||
|
|
||||||
status: accepted
|
|
||||||
date: 2026-07-31
|
|
||||||
decided by: maintainer (Bernd Worsch), 2026-07-31
|
|
||||||
tier: S (structural S — ratifies an existing target, creates no capability; chaos d4=3)
|
|
||||||
supersedes: nothing; discharges the open item raised by CB-WP-0003 T07
|
|
||||||
references: [GameKernel.md](../specs/GameKernel.md) §4,
|
|
||||||
[InnerLoop.md](../specs/InnerLoop.md) §Step 4 (correction vs retarget),
|
|
||||||
`history/260731-inner-loop-rule-audit.md`
|
|
||||||
|
|
||||||
## Why this ADR exists
|
|
||||||
|
|
||||||
CB-WP-0003 T07 added a mechanical test separating a **correction** (the
|
|
||||||
instrument disproved the target; the implementation did not change) from a
|
|
||||||
**retarget** (the same commit moved both the target and the code it
|
|
||||||
measures). AM-4a/AM-4b failed that test: they were set at 250,000 and
|
|
||||||
350,000 in commit `4be6e02`, by the implementer, after seeing the measured
|
|
||||||
246,250 — and that commit also changed the feature gating the metric
|
|
||||||
measures.
|
|
||||||
|
|
||||||
The reasoning was recorded at the time and is defensible. The *structure*
|
|
||||||
was not, and the audit flagged both targets as unratified: `make
|
|
||||||
dep-weight` has been enforcing thresholds no reviewed decision stood
|
|
||||||
behind.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
**AM-4a (≤ 250,000 lines, shipped runtime) and AM-4b (≤ 350,000 lines,
|
|
||||||
dev toolchain) are ratified as written.** No values change.
|
|
||||||
|
|
||||||
## The old target, and why it was abandoned
|
|
||||||
|
|
||||||
AM-4 originally read **≤ 20 transitive crates**, set against
|
|
||||||
boardgame.io's 120 npm packages. Retired for two measured reasons:
|
|
||||||
|
|
||||||
1. **Unreachable without undoing this spec's own contracts.** K5 (seeded
|
|
||||||
ChaCha) and K7 (SHA-256) cost 12 crates between them. The measured
|
|
||||||
ladder showed nothing reached 20 except reimplementing one of those
|
|
||||||
primitives — trading an audited cryptographic implementation for a
|
|
||||||
scoreboard number.
|
|
||||||
2. **Crate count does not compare across ecosystems.** Rust splits crates
|
|
||||||
far more finely than npm, so the original 33-vs-120 comparison
|
|
||||||
flattered us while the ≤20 target punished us, for the same reason.
|
|
||||||
|
|
||||||
## The measurement that motivated the change
|
|
||||||
|
|
||||||
At the time of the retarget, `make dep-weight`:
|
|
||||||
|
|
||||||
| configuration | crates | third-party LOC |
|
|
||||||
|---|---|---|
|
|
||||||
| shipped-runtime (`--no-default-features`) | 23 | **246,250** |
|
|
||||||
| dev-toolchain (default features) | 29 | **317,021** |
|
|
||||||
| own source | — | 3,443 |
|
|
||||||
|
|
||||||
## Why these targets bind on future work rather than merely passing present work
|
|
||||||
|
|
||||||
This is the question T07 requires an ADR to answer, and it is the reason
|
|
||||||
ratification is defensible rather than a rubber stamp.
|
|
||||||
|
|
||||||
- **AM-4a leaves 3,750 lines of headroom — 1.5%.** That is roughly one
|
|
||||||
small crate. Any dependency added to the shipped runtime breaches it
|
|
||||||
almost immediately, which is the intended behaviour: the shipped runtime
|
|
||||||
is meant to be effectively frozen, and the target enforces that a new
|
|
||||||
runtime dependency is a decision someone must argue for, not a default.
|
|
||||||
- **AM-4b leaves 32,979 lines — 10.4%.** Deliberately looser. The dev
|
|
||||||
toolchain is where scenario YAML, benchmarking, and future tooling land,
|
|
||||||
and it does not ship to a player. It should be able to absorb one
|
|
||||||
moderate dependency without a spec change, and not two.
|
|
||||||
- **Both are ceilings on a quantity that only grows by choice.** Nothing
|
|
||||||
drifts a project across these thresholds; only adding a dependency does.
|
|
||||||
A target that can only be breached deliberately is a target that binds.
|
|
||||||
|
|
||||||
**What would falsify this ratification:** if a future pass finds itself
|
|
||||||
raising AM-4a to accommodate a dependency it wanted, that is the failure
|
|
||||||
mode the ceiling exists to catch, and the answer is a new ADR arguing for
|
|
||||||
the dependency — not a quiet retarget.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- The open item in `history/260731-inner-loop-rule-audit.md` is discharged.
|
|
||||||
- `make dep-weight` continues to fail the build on breach, now backed by a
|
|
||||||
reviewed decision.
|
|
||||||
- AM-4c (own source per 100k third-party lines) remains **reported, not
|
|
||||||
targeted**, and is unaffected.
|
|
||||||
- Precedent: this is the first ADR written under the correction/retarget
|
|
||||||
test. Future retargets follow this shape — old target, the measurement,
|
|
||||||
and an argument about *future* binding rather than present passing.
|
|
||||||
|
|
@ -136,12 +136,12 @@ evidence files.
|
||||||
|
|
||||||
## Open items raised by this audit
|
## Open items raised by this audit
|
||||||
|
|
||||||
- ~~**AM-4a / AM-4b are unratified retargets.**~~ **Discharged
|
- **AM-4a / AM-4b are unratified retargets.** Measured at 246,250 and set
|
||||||
2026-07-31** by [ADR-0004](../decisions/ADR-0004-am4-ratification.md),
|
at 250,000 in the same commit, by the implementer, with the
|
||||||
ratified by the maintainer. Values unchanged; the ADR supplies the
|
implementation changing in that commit too — a retarget under the test
|
||||||
future-binding argument the test requires (AM-4a leaves 1.5% headroom,
|
added to InnerLoop §Step 4 by T07, not a correction. They must be
|
||||||
AM-4b 10.4%, and both measure a quantity that only grows by deliberate
|
ratified by ADR or changed. Until then `make dep-weight` is enforcing a
|
||||||
choice).
|
target no reviewed decision stands behind.
|
||||||
|
|
||||||
## Class coverage — where the gaps are
|
## Class coverage — where the gaps are
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,180 +0,0 @@
|
||||||
# CB-RES-0003: which agent turns can become deterministic compute
|
|
||||||
|
|
||||||
capability: meta.loop.mechanical-work
|
|
||||||
status: draft
|
|
||||||
tier: M (structural M — no new capability port; touches the one command
|
|
||||||
surface and the cost instrument; chaos d4=2 → no override)
|
|
||||||
instrument: `make cost-mix` (`tools/cb-cost.py`, tool-mix block)
|
|
||||||
|
|
||||||
Review of where token-priced agent turns did work a deterministic tool
|
|
||||||
could have done, so that capacity moves to judgment rather than mechanics.
|
|
||||||
Commissioned 2026-07-31.
|
|
||||||
|
|
||||||
**Method.** Every turn in both clay-borg session transcripts (573 responses,
|
|
||||||
$134 total) was classified by the tool calls it made. The classifier is
|
|
||||||
committed as `classify_tool()` in `tools/cb-cost.py` and its output is
|
|
||||||
emitted by `make cost-mix` — the numbers below are reproducible, and the
|
|
||||||
same command measures whether any fix worked.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## The measurement
|
|
||||||
|
|
||||||
```text
|
|
||||||
mech environment setup 84 turns $ 15.33
|
|
||||||
mech ad-hoc text patching 75 turns $ 13.86
|
|
||||||
git 37 turns $ 13.85
|
|
||||||
mech hub task status 25 turns $ 7.46
|
|
||||||
mech orientation / inspect 49 turns $ 6.87
|
|
||||||
hub other 32 turns $ 6.22
|
|
||||||
mech ad-hoc transcript analysis 39 turns $ 4.56
|
|
||||||
mech workplan status edit 21 turns $ 4.06
|
|
||||||
make (gates) 3 turns $ 1.82
|
|
||||||
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
|
|
||||||
```
|
|
||||||
|
|
||||||
**38% of spend went through turns whose tool calls were mechanical.**
|
|
||||||
|
|
||||||
A turn's whole cost is charged to every category it touched, so the rows
|
|
||||||
overlap and the deduplicated line is the honest total. `git` and `make`
|
|
||||||
are excluded from *mechanical* deliberately: a commit message is judgment,
|
|
||||||
and running a gate is the point.
|
|
||||||
|
|
||||||
## Candidate 1 — environment setup: 84 turns, $15.33
|
|
||||||
|
|
||||||
The single largest category, and pure friction. `cd` (51) and
|
|
||||||
`export PATH="$HOME/.cargo/bin:$PATH"` (10+) recur because the shell does
|
|
||||||
not persist state between calls and `cargo` is not on the default path.
|
|
||||||
Every `make` invocation that touches Rust had to be prefixed.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** put `~/.cargo/bin` on the path the agent
|
|
||||||
starts with, and make every `make` target self-sufficient (the Makefile
|
|
||||||
already knows where it is). `tools/dep-weight.py` already special-cases
|
|
||||||
this with a helpful error — evidence the friction was noticed and
|
|
||||||
patched at the leaf instead of the root.
|
|
||||||
- **Expected effect:** these turns do not become cheaper, they **stop
|
|
||||||
existing**. Predicted saving: **~$12–15 per two-session pass**, the
|
|
||||||
cleanest win in this review.
|
|
||||||
- **Risk:** near zero. Nothing depends on the current behaviour.
|
|
||||||
|
|
||||||
## Candidate 2 — ad-hoc text patching: 75 turns, $13.86
|
|
||||||
|
|
||||||
Inline `python3 - <<'PY'` heredocs that open a markdown file, string-replace,
|
|
||||||
and write it back. Written fresh each time, unreviewed, and the mechanism
|
|
||||||
behind the **duplicated-fact-drift** class: a number changes and every
|
|
||||||
artifact quoting it must be found and patched by hand. The acceptance
|
|
||||||
figure moved four times and each move cost a sweep across three or four
|
|
||||||
files.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** a *fact registry*. Numbers that appear in
|
|
||||||
more than one artifact are declared once (`facts.toml` or emitted by the
|
|
||||||
tool that measures them) and injected into markdown by a generator, with
|
|
||||||
`make facts-check` failing when a committed artifact disagrees with the
|
|
||||||
registry.
|
|
||||||
- **Expected effect:** removes the sweep, and gives the DFD class its
|
|
||||||
first executable gate — which InnerLoop v1.2 currently states only as
|
|
||||||
prose. Predicted saving: **~$6–9 per pass**, plus the error class.
|
|
||||||
- **Risk:** moderate. A registry that is itself hand-maintained just moves
|
|
||||||
the problem. It must be generated from instruments where possible, and
|
|
||||||
the check must be the enforcement, not the generation.
|
|
||||||
|
|
||||||
## Candidate 3 — hub task status: 25 turns, $7.46
|
|
||||||
|
|
||||||
Every `update_task_status` call was hand-written, including
|
|
||||||
`tokens_in`/`tokens_out` — **which were estimates I typed**, in a session
|
|
||||||
whose entire subject was that estimated token counts are worthless. The
|
|
||||||
hub holds numbers derived from the very habit CB-WP-0002 disproved.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** `make task-done T=T05` — flips the
|
|
||||||
workplan file, reads the measured cost for that task from `cb-cost`, and
|
|
||||||
pushes the hub event with real numbers. One command replaces an edit, a
|
|
||||||
status call, and a fabricated figure.
|
|
||||||
- **Expected effect:** **~$7 per pass**, and the hub stops holding
|
|
||||||
fiction. Combines with candidate 5.
|
|
||||||
- **Risk:** low. `cb-cost --by-task` already produces the figure.
|
|
||||||
|
|
||||||
## Candidate 4 — orientation / inspect: 49 turns, $6.87
|
|
||||||
|
|
||||||
`grep`/`ls`/`wc` to answer "what is the state of this repo" — which
|
|
||||||
workplan is active, which tasks are open, which gates pass, what is
|
|
||||||
uncommitted.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** `make status` printing the loop state in
|
|
||||||
one shot: active workplan, task counts, gate results, open cost since
|
|
||||||
last commit, provisional item ages.
|
|
||||||
- **Expected effect:** turns ~10 orientation turns into 1 at the start of
|
|
||||||
a session, and shrinks cold-start context (SS-04). Predicted saving:
|
|
||||||
**~$4 per pass**.
|
|
||||||
- **Risk:** low, but the saving is softer than it looks — some inspection
|
|
||||||
is genuinely exploratory and will not disappear.
|
|
||||||
|
|
||||||
## Candidate 5 — workplan status edit: 21 turns, $4.06
|
|
||||||
|
|
||||||
Heredocs doing `s.replace("status: todo", "status: done")` on a workplan
|
|
||||||
file. Purely mechanical, and error-prone: it silently does nothing if the
|
|
||||||
task is already done or the ID is mistyped.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** folded into candidate 3's
|
|
||||||
`make task-done`, which can *fail* on an unknown task ID instead of
|
|
||||||
no-op'ing.
|
|
||||||
- **Expected effect:** **~$4 per pass** and one class of silent no-op
|
|
||||||
removed.
|
|
||||||
|
|
||||||
## Candidate 6 — ad-hoc transcript analysis: 39 turns, $4.56
|
|
||||||
|
|
||||||
Already partly solved: `cb-cost` subsumed most of this during CB-WP-0002,
|
|
||||||
and this review's own classifier is now committed rather than ad-hoc. The
|
|
||||||
residue is one-off questions (context percentiles, compaction boundaries,
|
|
||||||
per-model splits) that were each written fresh.
|
|
||||||
|
|
||||||
- **Deterministic replacement:** promote the recurring ones to flags. Most
|
|
||||||
already exist (`--by-task`, `--composition`, session shape, tool mix).
|
|
||||||
- **Expected effect:** **~$2 per pass**, diminishing. Listed for
|
|
||||||
completeness, not priority.
|
|
||||||
|
|
||||||
## What must NOT be automated
|
|
||||||
|
|
||||||
Stating this because a review that only finds savings is not a review.
|
|
||||||
|
|
||||||
- **`git` (37 turns, $13.85)** is the second-most expensive category and
|
|
||||||
is mostly *commit message authorship* — the highest-output-token turns
|
|
||||||
in the corpus. That output is the project's reasoning record. Automating
|
|
||||||
it would save money and destroy the thing that makes corrections cheap.
|
|
||||||
- **`make` gates (3 turns, $1.82)** are already deterministic; the agent
|
|
||||||
merely invokes them. Correctly cheap.
|
|
||||||
- **Judgment work is invisible in this table** — writing a spec, choosing
|
|
||||||
an attribution model, deciding a target is legitimate. That is where the
|
|
||||||
remaining 62% went, and it is what the freed capacity should buy.
|
|
||||||
|
|
||||||
## Verdict and expected total
|
|
||||||
|
|
||||||
| # | candidate | turns | measured | predicted saving/pass | confidence |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| 1 | environment setup | 84 | $15.33 | **$12–15** | high |
|
|
||||||
| 2 | fact registry (text patching) | 75 | $13.86 | **$6–9** | medium |
|
|
||||||
| 3+5 | `make task-done` | 46 | $11.52 | **$9–11** | high |
|
|
||||||
| 4 | `make status` | 49 | $6.87 | **$4** | medium |
|
|
||||||
| 6 | cb-cost flags | 39 | $4.56 | **$2** | low |
|
|
||||||
| | **total** | | **$51.26** | **$33–41** | |
|
|
||||||
|
|
||||||
Predicted recovery is **25–30% of a pass**, against a measured 38%
|
|
||||||
mechanical share. The gap is deliberate: some inspection and some patching
|
|
||||||
is genuinely exploratory and will not vanish.
|
|
||||||
|
|
||||||
**The prediction is falsifiable and must be checked.** `make cost-mix`
|
|
||||||
emits the same categories, so the next pass measures whether these turns
|
|
||||||
disappeared or merely relocated — the failure mode being that an agent
|
|
||||||
which no longer writes heredocs simply writes more prose instead. A
|
|
||||||
control loop that does not test for relocation is not a control loop.
|
|
||||||
|
|
||||||
## Risks in this review itself
|
|
||||||
|
|
||||||
- **A turn's cost is charged to every category it touched**, so per-row
|
|
||||||
figures overstate. The deduplicated $51.26 is the defensible number;
|
|
||||||
per-candidate savings are apportioned from it and are estimates.
|
|
||||||
- **n = 2 sessions, one repo, one agent.** The mix is a property of how
|
|
||||||
this project was built, not a general law.
|
|
||||||
- **The classifier is a regex over shell commands.** It cannot see intent:
|
|
||||||
a `cd` that precedes real work is charged the whole turn. This inflates
|
|
||||||
candidate 1, which is why its predicted saving is below its measured
|
|
||||||
cost despite being the most certain fix.
|
|
||||||
|
|
@ -135,10 +135,7 @@ Command (actor-tagged intent)
|
||||||
|
|
||||||
## 4. Acceptance metrics
|
## 4. Acceptance metrics
|
||||||
|
|
||||||
**On AM-4's retarget (2026-07-31).** *Ratified by
|
**On AM-4's retarget (2026-07-31).** AM-4 originally read "≤20
|
||||||
[ADR-0004](../decisions/ADR-0004-am4-ratification.md) on 2026-07-31; the
|
|
||||||
headroom argument for why these ceilings bind on future work lives there.*
|
|
||||||
AM-4 originally read "≤20
|
|
||||||
transitive crates", set against boardgame.io's 120 npm packages. That
|
transitive crates", set against boardgame.io's 120 npm packages. That
|
||||||
target was retired for two measured reasons. First, it was unreachable
|
target was retired for two measured reasons. First, it was unreachable
|
||||||
without undoing this spec's own contracts: K5 (seeded ChaCha) and K7
|
without undoing this spec's own contracts: K5 (seeded ChaCha) and K7
|
||||||
|
|
|
||||||
Binary file not shown.
|
|
@ -186,13 +186,12 @@ def read_responses(path, pin=None):
|
||||||
head = rows[0]
|
head = rows[0]
|
||||||
# Tool calls are spread across the group's lines, so they are counted
|
# Tool calls are spread across the group's lines, so they are counted
|
||||||
# over the whole group — one response may carry several (SS-05).
|
# over the whole group — one response may carry several (SS-05).
|
||||||
blocks = [c for r in rows for c in (r["message"].get("content") or [])
|
tool_calls = sum(
|
||||||
if c.get("type") == "tool_use"]
|
1
|
||||||
tool_calls = len(blocks)
|
for r in rows
|
||||||
cats = sorted({
|
for c in (r["message"].get("content") or [])
|
||||||
c for c in (classify_tool(b.get("name"), b.get("input") or {})
|
if c.get("type") == "tool_use"
|
||||||
for b in blocks) if c
|
)
|
||||||
})
|
|
||||||
out.append(
|
out.append(
|
||||||
{
|
{
|
||||||
"request_id": rid,
|
"request_id": rid,
|
||||||
|
|
@ -201,51 +200,12 @@ def read_responses(path, pin=None):
|
||||||
"session": head.get("sessionId") or os.path.basename(path),
|
"session": head.get("sessionId") or os.path.basename(path),
|
||||||
"toks": toks,
|
"toks": toks,
|
||||||
"tool_calls": tool_calls,
|
"tool_calls": tool_calls,
|
||||||
"categories": cats,
|
|
||||||
"subagent": "/subagents/" in path,
|
"subagent": "/subagents/" in path,
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
# CB-RES-0003: which turns are mechanical (a deterministic tool could do
|
|
||||||
# them) versus judgment (only an agent can). The baseline for measuring
|
|
||||||
# whether automation actually removes turns rather than relocating them.
|
|
||||||
def classify_tool(name, inp):
|
|
||||||
if name == "mcp__dev-hub__update_task_status":
|
|
||||||
return "hub task status"
|
|
||||||
if name.startswith("mcp__dev-hub__"):
|
|
||||||
return "hub other"
|
|
||||||
if name != "Bash":
|
|
||||||
return None
|
|
||||||
cmd = (inp.get("command") or "").strip()
|
|
||||||
if not cmd:
|
|
||||||
return None
|
|
||||||
if "python3 - <<" in cmd:
|
|
||||||
if "status: todo" in cmd or "status: done" in cmd:
|
|
||||||
return "workplan status edit"
|
|
||||||
if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd:
|
|
||||||
return "ad-hoc transcript analysis"
|
|
||||||
return "ad-hoc text patching"
|
|
||||||
if cmd.startswith(("cd ", "export ")):
|
|
||||||
return "environment setup"
|
|
||||||
if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"):
|
|
||||||
return "orientation / inspect"
|
|
||||||
if cmd.startswith("git "):
|
|
||||||
return "git"
|
|
||||||
if "make " in cmd:
|
|
||||||
return "make (gates)"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# Categories a deterministic tool could plausibly own. Judgment-bearing
|
|
||||||
# categories (git commit messages, make gates) are excluded deliberately.
|
|
||||||
MECHANICAL = frozenset({
|
|
||||||
"environment setup", "ad-hoc text patching", "orientation / inspect",
|
|
||||||
"ad-hoc transcript analysis", "hub task status", "workplan status edit",
|
|
||||||
})
|
|
||||||
|
|
||||||
|
|
||||||
def session_shape(responses):
|
def session_shape(responses):
|
||||||
"""SH-1..SH-3 from specs/SessionShape.md."""
|
"""SH-1..SH-3 from specs/SessionShape.md."""
|
||||||
import statistics
|
import statistics
|
||||||
|
|
@ -401,25 +361,9 @@ def collect(slug, pin_ref=None):
|
||||||
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
|
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Tool mix: a turn's whole cost is charged to each category it touched,
|
|
||||||
# so columns may overlap and must not be summed as if disjoint.
|
|
||||||
mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float)
|
|
||||||
for r in responses:
|
|
||||||
for c in r.get("categories") or []:
|
|
||||||
mix_turns[c] += 1
|
|
||||||
mix_cost[c] += r["cost"] or 0.0
|
|
||||||
mech = [r for r in responses
|
|
||||||
if set(r.get("categories") or []) & MECHANICAL]
|
|
||||||
|
|
||||||
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
|
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
|
||||||
return {
|
return {
|
||||||
"session_shape": session_shape(responses),
|
"session_shape": session_shape(responses),
|
||||||
"tool_mix": {
|
|
||||||
"turns": dict(mix_turns),
|
|
||||||
"cost": dict(mix_cost),
|
|
||||||
"mechanical_turns": len(mech),
|
|
||||||
"mechanical_cost": sum(r["cost"] or 0 for r in mech),
|
|
||||||
},
|
|
||||||
"slug": slug,
|
"slug": slug,
|
||||||
"pin": pin,
|
"pin": pin,
|
||||||
"responses": len(responses),
|
"responses": len(responses),
|
||||||
|
|
@ -471,18 +415,6 @@ def render(rep, by_task=False, composition=False):
|
||||||
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
|
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
|
||||||
)
|
)
|
||||||
|
|
||||||
mix = rep["tool_mix"]
|
|
||||||
if mix["turns"]:
|
|
||||||
print("\n tool mix — a turn's cost is charged to every category it")
|
|
||||||
print(" touched, so columns overlap and must not be summed")
|
|
||||||
for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]):
|
|
||||||
tag = "mech" if k in MECHANICAL else " "
|
|
||||||
print(f" {tag} {k:<28}{mix['turns'][k]:>5} turns "
|
|
||||||
f"${mix['cost'][k]:>8,.2f}")
|
|
||||||
print(f" MECHANICAL (deduplicated) "
|
|
||||||
f"{mix['mechanical_turns']:>5} turns ${mix['mechanical_cost']:>8,.2f}"
|
|
||||||
f" = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass")
|
|
||||||
|
|
||||||
sh = rep["session_shape"]
|
sh = rep["session_shape"]
|
||||||
print("\n session shape (specs/SessionShape.md)")
|
print("\n session shape (specs/SessionShape.md)")
|
||||||
print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok "
|
print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok "
|
||||||
|
|
|
||||||
|
|
@ -1,214 +0,0 @@
|
||||||
---
|
|
||||||
id: CB-WP-0004
|
|
||||||
title: "Move mechanical turns off the token budget, and prove it worked"
|
|
||||||
status: proposed
|
|
||||||
state_hub_workstream_id: ""
|
|
||||||
---
|
|
||||||
|
|
||||||
# Purpose
|
|
||||||
|
|
||||||
`research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in
|
|
||||||
both clay-borg sessions by the tool calls it made:
|
|
||||||
|
|
||||||
```text
|
|
||||||
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
|
|
||||||
```
|
|
||||||
|
|
||||||
**38% of spend went through turns doing work a deterministic tool could
|
|
||||||
do.** The largest single category is `cd` and `export PATH` — 84 turns and
|
|
||||||
$15.33 of pure environment friction. The second is inline heredocs
|
|
||||||
string-patching markdown, which is also the mechanism behind the
|
|
||||||
duplicated-fact-drift error class InnerLoop v1.2 named and could not gate.
|
|
||||||
|
|
||||||
This workplan converts the five worthwhile categories to classic compute
|
|
||||||
and **measures whether that actually recovered anything**. The predicted
|
|
||||||
recovery is $33–41 per pass, 25–30%.
|
|
||||||
|
|
||||||
**The control loop is the point, not a formality.** The named failure mode
|
|
||||||
is *relocation*: an agent that can no longer write a heredoc may simply
|
|
||||||
write more prose, and the pass costs the same. `make cost-mix` emits the
|
|
||||||
same categories that produced the baseline, so the claim is falsifiable by
|
|
||||||
the same instrument that made it. A saving that cannot be demonstrated in
|
|
||||||
that table did not happen.
|
|
||||||
|
|
||||||
Per InnerLoop v1.2, targets here are **provisional until the instrument
|
|
||||||
emits them**, and no target may be moved in the commit that measures it
|
|
||||||
unless the instrument disproved it (§Step 4, correction vs retarget).
|
|
||||||
|
|
||||||
## Phase A — The certain wins
|
|
||||||
|
|
||||||
## Task: Remove environment friction
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T01
|
|
||||||
status: todo
|
|
||||||
priority: high
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
84 turns / $15.33, the largest category and the least interesting work in
|
|
||||||
the corpus. `cargo` is not on the default path, so every Rust-touching
|
|
||||||
command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does
|
|
||||||
not persist `cd`.
|
|
||||||
|
|
||||||
Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases
|
|
||||||
the missing `cargo` with a helpful error, which is evidence the friction
|
|
||||||
was noticed and patched in the wrong place.
|
|
||||||
|
|
||||||
Deliver: every `make` target runs from a clean shell with no prefix, from
|
|
||||||
any directory. Document the one-line environment requirement in
|
|
||||||
`README.md` if one remains. Remove the leaf workaround in `dep-weight.py`
|
|
||||||
only if it becomes unreachable — a positive control that never fires is
|
|
||||||
still cheaper than a regression.
|
|
||||||
|
|
||||||
**Predicted:** environment-setup turns → **< 10** (from 84),
|
|
||||||
**$12–15** recovered. Highest confidence in the review.
|
|
||||||
|
|
||||||
## Task: `make task-done` — one command for a task close
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T02
|
|
||||||
status: todo
|
|
||||||
priority: high
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
Merges two categories: workplan status edits (21 turns, $4.06) and hub
|
|
||||||
task-status calls (25 turns, $7.46).
|
|
||||||
|
|
||||||
`make task-done T=CB-WP-0004-T02` must:
|
|
||||||
|
|
||||||
1. flip `status: todo` → `done` in the workplan file, **failing loudly on
|
|
||||||
an unknown or already-done task** — the heredocs it replaces silently
|
|
||||||
no-op'd on a typo;
|
|
||||||
2. read that task's **measured** cost and tokens from `cb-cost --by-task`;
|
|
||||||
3. push the hub event with the real numbers.
|
|
||||||
|
|
||||||
The third point is the one that matters beyond cost. Every
|
|
||||||
`update_task_status` in this project so far carried **hand-typed token
|
|
||||||
estimates**, in a repo whose central finding is that estimated token counts
|
|
||||||
are worthless. The hub currently holds fiction produced by the exact habit
|
|
||||||
CB-WP-0002 disproved.
|
|
||||||
|
|
||||||
**Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub
|
|
||||||
stops holding estimates. Add `--self-test` per InnerLoop v1.1.
|
|
||||||
|
|
||||||
## Task: `make status` — one-shot orientation
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T03
|
|
||||||
status: todo
|
|
||||||
priority: medium
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this
|
|
||||||
repo". Replace with one command printing: active workplan and task counts,
|
|
||||||
gate results, open spend since the last commit (CB-01), provisional item
|
|
||||||
ages, and any `loop-lint` findings.
|
|
||||||
|
|
||||||
Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures
|
|
||||||
at ~51k for a fresh session — the artifacts a new session reads to orient
|
|
||||||
are exactly what this prints.
|
|
||||||
|
|
||||||
**Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered.
|
|
||||||
Confidence medium: some inspection is genuinely exploratory and will not
|
|
||||||
disappear, and the review says so.
|
|
||||||
|
|
||||||
## Phase B — The one that also closes an error class
|
|
||||||
|
|
||||||
## Task: Fact registry and `make facts-check`
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T04
|
|
||||||
status: todo
|
|
||||||
priority: high
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
75 turns / $13.86 of heredocs opening a markdown file, string-replacing a
|
|
||||||
number, and writing it back — the mechanism behind **duplicated-fact
|
|
||||||
drift**, the fourth error class, which InnerLoop v1.2 states as prose and
|
|
||||||
cannot currently gate.
|
|
||||||
|
|
||||||
Two instances on record: a price sheet inlined into a spec went stale
|
|
||||||
within an hour of the real sheet changing, and the acceptance figure
|
|
||||||
$92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a
|
|
||||||
workplan, and an evidence file on every move.
|
|
||||||
|
|
||||||
Deliver a registry where a number appearing in more than one artifact is
|
|
||||||
declared once — **generated by the instrument that measures it wherever
|
|
||||||
possible**, not hand-maintained — plus `make facts-check` failing when a
|
|
||||||
committed artifact disagrees with it.
|
|
||||||
|
|
||||||
**The trap to avoid, stated up front:** a hand-maintained registry moves
|
|
||||||
the problem rather than solving it, and would itself become a copy that
|
|
||||||
drifts. If generation from instruments proves impractical, deliver only
|
|
||||||
the *check* (detect the same number stated differently in two artifacts)
|
|
||||||
and say so — a gate with no generator still closes the class.
|
|
||||||
|
|
||||||
**Predicted:** **$6–9** recovered, plus DFD's first executable gate.
|
|
||||||
Confidence medium; this is the hardest task here and the most valuable.
|
|
||||||
|
|
||||||
## Phase C — Prove it, or withdraw the claim
|
|
||||||
|
|
||||||
## Task: Control loop — measure recovery and test for relocation
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T05
|
|
||||||
status: todo
|
|
||||||
priority: high
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
The task this workplan exists for. Run `make cost-mix` over the sessions
|
|
||||||
that executed T01–T04 and commit
|
|
||||||
`evidence/CB-EV-0003-mechanical-work.md` comparing against the committed
|
|
||||||
baseline:
|
|
||||||
|
|
||||||
| category | baseline turns | baseline $ | predicted | measured | verdict |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| environment setup | 84 | $15.33 | <10 turns | | |
|
|
||||||
| ad-hoc text patching | 75 | $13.86 | $6–9 saved | | |
|
|
||||||
| hub task status + workplan edit | 46 | $11.52 | ~6 turns | | |
|
|
||||||
| orientation / inspect | 49 | $6.87 | $4 saved | | |
|
|
||||||
| **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | |
|
|
||||||
|
|
||||||
**Three tests, all of which must be reported:**
|
|
||||||
|
|
||||||
1. **Did the mechanical turns disappear?** Per-category, against
|
|
||||||
prediction. An unmet prediction is reported unmet, not retargeted.
|
|
||||||
2. **Did they relocate?** Total pass cost and non-mechanical turn counts
|
|
||||||
must be compared too. If mechanical turns fell and prose turns rose by
|
|
||||||
as much, the saving is zero and this workplan failed — that is the
|
|
||||||
result to publish.
|
|
||||||
3. **Did quality hold?** `make all` green, and the same class of findings
|
|
||||||
still surfacing. A cheaper pass that catches fewer errors is worse, and
|
|
||||||
the loop has no metric for this yet — record the judgment explicitly
|
|
||||||
rather than implying the cost number settles it.
|
|
||||||
|
|
||||||
Normalize per unit of work, not per session: passes differ in size, so
|
|
||||||
report **mechanical share of pass cost** (baseline: 38%) alongside
|
|
||||||
absolute dollars.
|
|
||||||
|
|
||||||
## Task: Retrospective
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: CB-WP-0004-T06
|
|
||||||
status: todo
|
|
||||||
priority: low
|
|
||||||
state_hub_task_id: ""
|
|
||||||
```
|
|
||||||
|
|
||||||
The question to answer honestly: **does converting agent work to
|
|
||||||
deterministic tooling actually recover capacity, or does the work
|
|
||||||
reappear elsewhere?**
|
|
||||||
|
|
||||||
This is the first pass in the project to make a *quantitative prediction
|
|
||||||
before acting*. Whether the prediction held is more informative than the
|
|
||||||
saving itself — a loop that can forecast its own economics can plan; one
|
|
||||||
that cannot is guessing with numbers attached.
|
|
||||||
|
|
||||||
Record the prediction error per candidate, and whether the review's stated
|
|
||||||
confidence levels (high/medium/low) tracked reality. If they did not, the
|
|
||||||
next review should stop stating confidence, or state it differently.
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue