tegwick
7e21df378a
Some checks failed
ci / check (push) Failing after 4s
Review of where token-priced turns did work a deterministic tool could
do. Method: classify every turn in both transcripts by the tool calls it
made. The classifier is committed in tools/cb-cost.py and emitted by
`make cost-mix`, so the baseline is reproducible and the same command
can later falsify the predictions.
mech environment setup 84 turns $15.33
mech ad-hoc text patching 75 turns $13.86
git 37 turns $13.85
mech hub task status 25 turns $ 7.46
mech orientation / inspect 49 turns $ 6.87
hub other 32 turns $ 6.22
mech ad-hoc transcript 39 turns $ 4.56
mech workplan status edit 21 turns $ 4.06
MECHANICAL (dedup) 290 turns $51.26 = 38% of pass
Largest category is `cd` and `export PATH` -- pure friction, and
dep-weight.py already patched it at the leaf, which is evidence it was
noticed and fixed in the wrong place. Second is heredocs string-patching
markdown, which is also the mechanism behind duplicated-fact drift, the
error class InnerLoop v1.2 names and cannot gate.
Explicitly NOT automated: git (37 turns, $13.85) is mostly commit
message authorship -- the highest-output turns in the corpus and the
project's reasoning record. Automating it would save money and destroy
what makes corrections cheap.
CB-WP-0004 implements five candidates and predicts $33-41 recovery
(25-30%), below the 38% measured share on purpose: some inspection and
patching is genuinely exploratory.
The control loop is the deliverable, not a formality. T05 tests three
things and must report all: did mechanical turns disappear, did they
RELOCATE into prose, and did quality hold. If mechanical turns fall and
prose rises by as much, the saving is zero and that is the result to
publish.
Also a self-indictment worth recording: every hub update_task_status in
this project carried hand-typed token estimates, in a repo whose central
finding is that estimated token counts are worthless. T02 fixes it by
reading measured values from cb-cost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
73 lines
2.2 KiB
Makefile
73 lines
2.2 KiB
Makefile
# One command surface (InnerLoop §agentic-efficiency #3). Deterministic,
|
|
# greppable output; precursor of the `cb` CLI.
|
|
|
|
CARGO := cargo
|
|
|
|
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all
|
|
|
|
## fmt + clippy (deny warnings) + HashMap deny-lint
|
|
check:
|
|
$(CARGO) fmt --all --check
|
|
$(CARGO) clippy --workspace --all-targets -- -D warnings
|
|
|
|
## unit + scenario-format tests
|
|
test:
|
|
$(CARGO) test --workspace
|
|
|
|
## run all GROUND scenarios through cb-sim
|
|
dep-weight:
|
|
python3 tools/dep-weight.py
|
|
|
|
coverage:
|
|
python3 tools/rule-coverage.py
|
|
|
|
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
|
|
# runs first: a cost number from an unverified collector is void.
|
|
cost: cost-test
|
|
python3 tools/cb-cost.py --composition --by-task
|
|
|
|
cost-test:
|
|
python3 tools/cb-cost.py --self-test
|
|
|
|
# InnerLoop rules that are mechanically checkable (CB-WP-0003 T01).
|
|
loop-lint:
|
|
python3 tools/loop-lint.py
|
|
|
|
# Positive control for every reporting tool, per InnerLoop v1.1 Step 5.
|
|
self-tests:
|
|
python3 tools/cb-cost.py --self-test
|
|
python3 tools/loop-lint.py --self-test
|
|
python3 tools/rule-coverage.py --self-test
|
|
python3 tools/dep-weight.py --self-test
|
|
|
|
# CB-01/CB-02: live spend since the last commit.
|
|
cost-budget: cost-test
|
|
python3 tools/cb-cost.py --budget
|
|
|
|
# CB-RES-0003 baseline: mechanical vs judgment turns.
|
|
cost-mix: cost-test
|
|
python3 tools/cb-cost.py --composition
|
|
|
|
cost-pin: cost-test
|
|
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
|
|
|
sim:
|
|
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
|
|
|
|
## Criterion benches (AM-6/AM-7)
|
|
bench:
|
|
$(CARGO) bench -p games-ground
|
|
|
|
## InnerLoop positive control: run every bench once, no measurement.
|
|
## Fails if a workload stalls or produces the wrong event count.
|
|
bench-test:
|
|
$(CARGO) bench -p games-ground --bench synthetic -- --test
|
|
|
|
|
|
## AM-2/AM-3 input: source LOC per crate (excludes tests would need tokei)
|
|
loc:
|
|
@for d in crates/cb-kernel crates/cb-events crates/cb-game-runtime games/ground tools/cb-sim; do \
|
|
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
|
|
done
|
|
|
|
all: check test sim coverage dep-weight self-tests loop-lint bench-test
|