coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
clay-borg/Makefile

128 lines
5 KiB
Makefile
Raw Normal View History

T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
# One command surface (InnerLoop §agentic-efficiency #3). Deterministic,
# greppable output; precursor of the `cb` CLI.
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
#
# CB-WP-0004 T01: every target here runs from a clean shell, from any
# directory, with no prefix. Invoke as `make -C <repo> <target>` from
# elsewhere. No target requires `cd` or `export PATH` — CB-RES-0003
# measured 84 turns and $15.33 spent on exactly those two prefixes.
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
# Absolute path to this Makefile's directory, so recipes never depend on
# the caller's working directory.
REPO := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
# Locate cargo instead of requiring it on the inherited PATH. Mirrors
# tools/repo.py:cargo_bin() — kept in sync by `make env-test`.
CARGO := $(firstword $(shell command -v cargo 2>/dev/null) \
$(wildcard $(HOME)/.cargo/bin/cargo) \
$(wildcard /usr/local/cargo/bin/cargo) \
cargo)
export PATH := $(dir $(CARGO)):$(PATH)
PY := python3
TOOLS := $(REPO)/tools
# Every cargo recipe runs at the repo root; the shell does not persist cd.
IN_REPO := cd $(REPO) &&
CB-WP-0004 T03: make status — one-shot orientation 49 turns and $6.87 per CB-RES-0003 went to grep/ls/wc answering "what is the state of this repo". tools/status.py answers it in 22 lines: commit and working-tree state, every workplan with task counts, the next todo task, spend since the last commit against CB-01/CB-02, and the three fast gates. Two constraints are deliberate. It does not build — a status command that takes two minutes gets replaced by `ls` within a day. And it states its own limit in the output ("slow gates not run here"), so a green status cannot be misread as a green `make all`; the self-test asserts that line is present, and that the whole report stays under 40 lines. The positive control is the refusal to be confidently empty: a parser that found zero workplans and zero tasks would print a clean, wrong picture, which is worse than the greps it replaces. It also caught a dot-all regex that returned twenty paragraphs of the preceding task's prose as the "task heading". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:20:27 +02:00
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests env-test task-done status loc all
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
## fmt + clippy (deny warnings) + HashMap deny-lint
check:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(IN_REPO) $(CARGO) fmt --all --check
$(IN_REPO) $(CARGO) clippy --workspace --all-targets -- -D warnings
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
## unit + scenario-format tests
test:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(IN_REPO) $(CARGO) test --workspace
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
## run all GROUND scenarios through cb-sim
AM-4: gate scenario YAML, retarget on audited source, re-measure Adopts both remediations from CB-EV-0001 §4 (maintainer decision). Option A — serde_yaml is now optional behind cb-game-runtime's `scenarios` feature. The scenario module, the ScenarioGame impl and the string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both configurations compile and lint clean under -D warnings. A trap worth recording: `default-features = false` on a *member* dependency is silently ignored when the workspace dependency does not specify it. The first attempt gated nothing while looking correct — the build succeeded and cargo tree still showed all six YAML crates. Fixed by setting it on the workspace dependency. This is the positive-control failure mode in miniature: success was not evidence the change applied. Retarget — AM-4 now measures third-party source under audit, split by build configuration, replacing a crate count that was unreachable without undoing K5/K7 and that does not compare across ecosystems. Re-measured via the new `make dep-weight`, whose own positive control refuses to report when any crate's source cannot be located: shipped runtime 23 crates 246,250 lines target <=250,000 met dev toolchain 29 crates 317,021 lines target <=350,000 met own source 3,408 lines Scenario tooling costs 70,771 lines a shipped game never compiles — the split the single number was hiding. Targets are set at current measurement plus headroom, so they bind on future growth rather than retroactively passing what had failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 03:35:41 +02:00
dep-weight:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/dep-weight.py
AM-4: gate scenario YAML, retarget on audited source, re-measure Adopts both remediations from CB-EV-0001 §4 (maintainer decision). Option A — serde_yaml is now optional behind cb-game-runtime's `scenarios` feature. The scenario module, the ScenarioGame impl and the string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both configurations compile and lint clean under -D warnings. A trap worth recording: `default-features = false` on a *member* dependency is silently ignored when the workspace dependency does not specify it. The first attempt gated nothing while looking correct — the build succeeded and cargo tree still showed all six YAML crates. Fixed by setting it on the workspace dependency. This is the positive-control failure mode in miniature: success was not evidence the change applied. Retarget — AM-4 now measures third-party source under audit, split by build configuration, replacing a crate count that was unreachable without undoing K5/K7 and that does not compare across ecosystems. Re-measured via the new `make dep-weight`, whose own positive control refuses to report when any crate's source cannot be located: shipped runtime 23 crates 246,250 lines target <=250,000 met dev toolchain 29 crates 317,021 lines target <=350,000 met own source 3,408 lines Scenario tooling costs 70,771 lines a shipped game never compiles — the split the single number was hiding. Targets are set at current measurement plus headroom, so they bind on future growth rather than retroactively passing what had failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 03:35:41 +02:00
Add the AM-1 rule-coverage gate make coverage compares the GR-rule IDs defined in specs/GroundRules.md against the covers: lists in scenarios/ground/*.yaml. It exits 1 if a scenario claims a rule the spec does not define, so coverage cannot be inflated by an invented ID, and exits 2 while rules remain uncovered. Current reading: 34/58 (58%). AM-1 requires 100%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 02:23:38 +02:00
coverage:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/rule-coverage.py
Add the AM-1 rule-coverage gate make coverage compares the GR-rule IDs defined in specs/GroundRules.md against the covers: lists in scenarios/ground/*.yaml. It exits 1 if a scenario claims a rule the spec does not define, so coverage cannot be inflated by an invented ID, and exits 2 while rules remain uncovered. Current reading: 34/58 (58%). AM-1 requires 100%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 02:23:38 +02:00
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
# runs first: a cost number from an unverified collector is void.
cost: cost-test
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --composition --by-task
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
cost-test:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --self-test
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
T01: audit every InnerLoop rule, and make the checkable ones executable 41 rules classified executable / checkable / decorative, each tagged with the failure class it catches. Counts: 11 executable, 22 checkable, 4 decorative (one of them dead policy). Audit: history/260731-inner-loop-rule-audit.md New tools/loop-lint.py makes 7 rules executable (tier declared, chaos roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file loadability, reporting tools expose --self-test). It found three real violations on its first run, none previously visible: - specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit the loop has stated since v0.2 and never measured. Split at its own section boundaries into Blueprint (1-8) + Runtime (9-15). - tools/dep-weight.py and tools/rule-coverage.py had positive-control logic and no --self-test, so nothing verified the control worked. Adding rule-coverage's self-test exposed a latent instance of the exact class this workplan is about: if the spec regex stopped matching, rules was empty, missing was empty, and the tool exited 0 reporting "0/0" -- a silent pass, in the tool that reports our headline AM-1 number. Both tools now assert they found something before reporting. Two demotions applied in the spec rather than left implicit: "structured over prose" is marked guidance (nothing can check it), and the 8k/10k token budget is struck through and marked DEAD POLICY pointing at T05. The audit's uncomfortable finding: rule 13 (re-derive inherited numbers) has no mechanical form, is deliberately left decorative, and caught the LARGEST error in CB-WP-0002. That is a counter-example to this workplan's own hypothesis. "A rule that cannot be executed is not a rule" is wrong as stated; the defensible version is that such a rule cannot be relied on to fire, so it must not be the only defence for a class that matters. Class coverage: harness-does-nothing has five executable rules; trusted-arithmetic has ZERO and produced the largest single error. make loop-lint and make self-tests wired into `make all` and CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:16:00 +02:00
# InnerLoop rules that are mechanically checkable (CB-WP-0003 T01).
loop-lint:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/loop-lint.py
T01: audit every InnerLoop rule, and make the checkable ones executable 41 rules classified executable / checkable / decorative, each tagged with the failure class it catches. Counts: 11 executable, 22 checkable, 4 decorative (one of them dead policy). Audit: history/260731-inner-loop-rule-audit.md New tools/loop-lint.py makes 7 rules executable (tier declared, chaos roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file loadability, reporting tools expose --self-test). It found three real violations on its first run, none previously visible: - specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit the loop has stated since v0.2 and never measured. Split at its own section boundaries into Blueprint (1-8) + Runtime (9-15). - tools/dep-weight.py and tools/rule-coverage.py had positive-control logic and no --self-test, so nothing verified the control worked. Adding rule-coverage's self-test exposed a latent instance of the exact class this workplan is about: if the spec regex stopped matching, rules was empty, missing was empty, and the tool exited 0 reporting "0/0" -- a silent pass, in the tool that reports our headline AM-1 number. Both tools now assert they found something before reporting. Two demotions applied in the spec rather than left implicit: "structured over prose" is marked guidance (nothing can check it), and the 8k/10k token budget is struck through and marked DEAD POLICY pointing at T05. The audit's uncomfortable finding: rule 13 (re-derive inherited numbers) has no mechanical form, is deliberately left decorative, and caught the LARGEST error in CB-WP-0002. That is a counter-example to this workplan's own hypothesis. "A rule that cannot be executed is not a rule" is wrong as stated; the defensible version is that such a rule cannot be relied on to fire, so it must not be the only defence for a class that matters. Class coverage: harness-does-nothing has five executable rules; trusted-arithmetic has ZERO and produced the largest single error. make loop-lint and make self-tests wired into `make all` and CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:16:00 +02:00
# Positive control for every reporting tool, per InnerLoop v1.1 Step 5.
self-tests:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --self-test
$(PY) $(TOOLS)/loop-lint.py --self-test
$(PY) $(TOOLS)/rule-coverage.py --self-test
$(PY) $(TOOLS)/dep-weight.py --self-test
$(PY) $(TOOLS)/repo.py --self-test
CB-WP-0004 T02: make task-done — close a task on measured numbers Replaces the three hand-done steps of a task close (46 turns, $11.52 per CB-RES-0003): the heredoc flipping status in the workplan file, the hand-written hub call, and the hand-typed token counts. The third is the reason this task exists. Every update_task_status this repo produced carried estimated tokens_in/tokens_out — in a project whose central finding is that estimated token counts are worthless. task-done reads the measured figure from the transcripts, or refuses; there is no path through it that emits an estimate. cb-cost gains by_task_detail: cost, response count, model histogram and token components per task. task-done imports cb-cost rather than parsing its printed table, so the hub figure is not a copy that can drift from its source. The positive control found a real defect before the tool ran once. Attribution keyed on a bare T\d\d from the commit subject, so CB-WP-0002 T01, CB-WP-0003 T01 and CB-WP-0004 T01 shared a bucket: the self-test reported $12.10 for "T01" where the qualified figure is $2.33. That 5.2x overstatement would have been pushed to the hub as a *measured* number — the same fiction in a new form. task_label() now keys qualified subjects on the full id and leaves unqualified ones bare rather than retro-assigning them to a workplan. The pinned $93.15 benchmark is unchanged, so historical attribution was not disturbed. Fourth instance of trusted arithmetic: a number believed because a program produced it rather than a hand. Refusals, all exercised by --self-test: unknown id, typo'd id, already-done task, missing state_hub_task_id, no measured spend, and a status flip that produced no change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:17:51 +02:00
$(PY) $(TOOLS)/task-done.py --self-test
CB-WP-0004 T03: make status — one-shot orientation 49 turns and $6.87 per CB-RES-0003 went to grep/ls/wc answering "what is the state of this repo". tools/status.py answers it in 22 lines: commit and working-tree state, every workplan with task counts, the next todo task, spend since the last commit against CB-01/CB-02, and the three fast gates. Two constraints are deliberate. It does not build — a status command that takes two minutes gets replaced by `ls` within a day. And it states its own limit in the output ("slow gates not run here"), so a green status cannot be misread as a green `make all`; the self-test asserts that line is present, and that the whole report stays under 40 lines. The positive control is the refusal to be confidently empty: a parser that found zero workplans and zero tasks would print a clean, wrong picture, which is worse than the greps it replaces. It also caught a dot-all regex that returned twenty paragraphs of the preceding task's prose as the "task heading". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:20:27 +02:00
$(PY) $(TOOLS)/status.py --self-test
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
# T01 positive control: prove the environment fix, do not assume it. Runs
# every tool from a foreign working directory with a PATH that has no
# cargo on it. Before T01 this failed; if it fails again, the friction is
# back and CB-EV-0003's measurement is invalid.
env-test:
@cd / && env PATH=/usr/bin:/bin $(PY) $(TOOLS)/repo.py --self-test
@cd / && env PATH=/usr/bin:/bin $(PY) $(TOOLS)/rule-coverage.py --self-test >/dev/null \
&& echo " [ok ] rule-coverage runs from / with no cargo on PATH"
@cd / && env PATH=/usr/bin:/bin $(PY) $(TOOLS)/dep-weight.py --self-test >/dev/null \
&& echo " [ok ] dep-weight runs from / with no cargo on PATH"
@cd / && env PATH=/usr/bin:/bin $(PY) $(TOOLS)/cb-cost.py --self-test >/dev/null \
&& echo " [ok ] cb-cost runs from / with no cargo on PATH"
@cd / && env PATH=/usr/bin:/bin $(PY) $(TOOLS)/loop-lint.py --self-test >/dev/null \
&& echo " [ok ] loop-lint runs from / with no cargo on PATH"
@$(MAKE) -C $(REPO) coverage >/dev/null \
&& echo " [ok ] make -C <repo> works from any directory"
T01: audit every InnerLoop rule, and make the checkable ones executable 41 rules classified executable / checkable / decorative, each tagged with the failure class it catches. Counts: 11 executable, 22 checkable, 4 decorative (one of them dead policy). Audit: history/260731-inner-loop-rule-audit.md New tools/loop-lint.py makes 7 rules executable (tier declared, chaos roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file loadability, reporting tools expose --self-test). It found three real violations on its first run, none previously visible: - specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit the loop has stated since v0.2 and never measured. Split at its own section boundaries into Blueprint (1-8) + Runtime (9-15). - tools/dep-weight.py and tools/rule-coverage.py had positive-control logic and no --self-test, so nothing verified the control worked. Adding rule-coverage's self-test exposed a latent instance of the exact class this workplan is about: if the spec regex stopped matching, rules was empty, missing was empty, and the tool exited 0 reporting "0/0" -- a silent pass, in the tool that reports our headline AM-1 number. Both tools now assert they found something before reporting. Two demotions applied in the spec rather than left implicit: "structured over prose" is marked guidance (nothing can check it), and the 8k/10k token budget is struck through and marked DEAD POLICY pointing at T05. The audit's uncomfortable finding: rule 13 (re-derive inherited numbers) has no mechanical form, is deliberately left decorative, and caught the LARGEST error in CB-WP-0002. That is a counter-example to this workplan's own hypothesis. "A rule that cannot be executed is not a rule" is wrong as stated; the defensible version is that such a rule cannot be relied on to fire, so it must not be the only defence for a class that matters. Class coverage: harness-does-nothing has five executable rules; trusted-arithmetic has ZERO and produced the largest single error. make loop-lint and make self-tests wired into `make all` and CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:16:00 +02:00
CB-WP-0004 T03: make status — one-shot orientation 49 turns and $6.87 per CB-RES-0003 went to grep/ls/wc answering "what is the state of this repo". tools/status.py answers it in 22 lines: commit and working-tree state, every workplan with task counts, the next todo task, spend since the last commit against CB-01/CB-02, and the three fast gates. Two constraints are deliberate. It does not build — a status command that takes two minutes gets replaced by `ls` within a day. And it states its own limit in the output ("slow gates not run here"), so a green status cannot be misread as a green `make all`; the self-test asserts that line is present, and that the whole report stays under 40 lines. The positive control is the refusal to be confidently empty: a parser that found zero workplans and zero tasks would print a clean, wrong picture, which is worse than the greps it replaces. It also caught a dot-all regex that returned twenty paragraphs of the preceding task's prose as the "task heading". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:20:27 +02:00
# T03: one-shot orientation — workplans, next task, spend, fast gates.
# Cheap by design: no build. Start a session with this instead of grepping.
status:
@$(PY) $(TOOLS)/status.py
CB-WP-0004 T02: make task-done — close a task on measured numbers Replaces the three hand-done steps of a task close (46 turns, $11.52 per CB-RES-0003): the heredoc flipping status in the workplan file, the hand-written hub call, and the hand-typed token counts. The third is the reason this task exists. Every update_task_status this repo produced carried estimated tokens_in/tokens_out — in a project whose central finding is that estimated token counts are worthless. task-done reads the measured figure from the transcripts, or refuses; there is no path through it that emits an estimate. cb-cost gains by_task_detail: cost, response count, model histogram and token components per task. task-done imports cb-cost rather than parsing its printed table, so the hub figure is not a copy that can drift from its source. The positive control found a real defect before the tool ran once. Attribution keyed on a bare T\d\d from the commit subject, so CB-WP-0002 T01, CB-WP-0003 T01 and CB-WP-0004 T01 shared a bucket: the self-test reported $12.10 for "T01" where the qualified figure is $2.33. That 5.2x overstatement would have been pushed to the hub as a *measured* number — the same fiction in a new form. task_label() now keys qualified subjects on the full id and leaves unqualified ones bare rather than retro-assigning them to a workplan. The pinned $93.15 benchmark is unchanged, so historical attribution was not disturbed. Fourth instance of trusted arithmetic: a number believed because a program produced it rather than a hand. Refusals, all exercised by --self-test: unknown id, typo'd id, already-done task, missing state_hub_task_id, no measured spend, and a status flip that produced no change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:17:51 +02:00
# T02: close a task — flip the workplan file, read the *measured* cost
# from the transcripts, push the hub event with real numbers. Refuses on an
# unknown or already-done task, and refuses to report an estimate.
# make task-done T=CB-WP-0004-T02
task-done:
@test -n "$(T)" || { echo "usage: make task-done T=CB-WP-0004-T02" >&2; exit 2; }
$(PY) $(TOOLS)/task-done.py $(T) $(ARGS)
T05: live cost budget replaces the dead token budget The 8k/10k per-task token budget was never referenced or enforced and T08 blew past it silently. Replaced with a budget that can actually fire. The design constraint is the interesting part: per-task cost needs the commit that CLOSES the task, so a per-task budget is unavoidably retrospective -- it can only report a breach after the money is spent, which is the dead-policy failure again. What IS observable mid-task is spend since the last commit, because the transcript is append-live. So the budget binds on the open remainder. CB-01 budget = USD since the last commit, via `make cost-budget` CB-02 soft $10.00 (state progress, decide), hard $22.00 (stop) Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50 $1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum -- it would have fired exactly once on the calibration pass. Hard is ~2x the observed max, a value never reached in 32 intervals, so reaching it means the session is doing something the data has no example of. Both thresholds are set ABOVE every observed value, so they bind on future work rather than ratifying present work -- the distinction T07 is about. Stated limit: it is a command, not a daemon. An agent that never runs it gets no signal, which is the dead-policy failure one level up. Mitigated only by being free to run and on the one command surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:21:30 +02:00
# CB-01/CB-02: live spend since the last commit.
cost-budget: cost-test
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --budget
T05: live cost budget replaces the dead token budget The 8k/10k per-task token budget was never referenced or enforced and T08 blew past it silently. Replaced with a budget that can actually fire. The design constraint is the interesting part: per-task cost needs the commit that CLOSES the task, so a per-task budget is unavoidably retrospective -- it can only report a breach after the money is spent, which is the dead-policy failure again. What IS observable mid-task is spend since the last commit, because the transcript is append-live. So the budget binds on the open remainder. CB-01 budget = USD since the last commit, via `make cost-budget` CB-02 soft $10.00 (state progress, decide), hard $22.00 (stop) Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50 $1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum -- it would have fired exactly once on the calibration pass. Hard is ~2x the observed max, a value never reached in 32 intervals, so reaching it means the session is doing something the data has no example of. Both thresholds are set ABOVE every observed value, so they bind on future work rather than ratifying present work -- the distinction T07 is about. Stated limit: it is a command, not a daemon. An agent that never runs it gets no signal, which is the dead-policy failure one level up. Mitigated only by being free to run and on the one command surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:21:30 +02:00
CB-RES-0003 + CB-WP-0004: 38% of pass cost is mechanical turns Review of where token-priced turns did work a deterministic tool could do. Method: classify every turn in both transcripts by the tool calls it made. The classifier is committed in tools/cb-cost.py and emitted by `make cost-mix`, so the baseline is reproducible and the same command can later falsify the predictions. mech environment setup 84 turns $15.33 mech ad-hoc text patching 75 turns $13.86 git 37 turns $13.85 mech hub task status 25 turns $ 7.46 mech orientation / inspect 49 turns $ 6.87 hub other 32 turns $ 6.22 mech ad-hoc transcript 39 turns $ 4.56 mech workplan status edit 21 turns $ 4.06 MECHANICAL (dedup) 290 turns $51.26 = 38% of pass Largest category is `cd` and `export PATH` -- pure friction, and dep-weight.py already patched it at the leaf, which is evidence it was noticed and fixed in the wrong place. Second is heredocs string-patching markdown, which is also the mechanism behind duplicated-fact drift, the error class InnerLoop v1.2 names and cannot gate. Explicitly NOT automated: git (37 turns, $13.85) is mostly commit message authorship -- the highest-output turns in the corpus and the project's reasoning record. Automating it would save money and destroy what makes corrections cheap. CB-WP-0004 implements five candidates and predicts $33-41 recovery (25-30%), below the 38% measured share on purpose: some inspection and patching is genuinely exploratory. The control loop is the deliverable, not a formality. T05 tests three things and must report all: did mechanical turns disappear, did they RELOCATE into prose, and did quality hold. If mechanical turns fall and prose rises by as much, the saving is zero and that is the result to publish. Also a self-indictment worth recording: every hub update_task_status in this project carried hand-typed token estimates, in a repo whose central finding is that estimated token counts are worthless. T02 fixes it by reading measured values from cb-cost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:49:57 +02:00
# CB-RES-0003 baseline: mechanical vs judgment turns.
cost-mix: cost-test
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --composition
CB-RES-0003 + CB-WP-0004: 38% of pass cost is mechanical turns Review of where token-priced turns did work a deterministic tool could do. Method: classify every turn in both transcripts by the tool calls it made. The classifier is committed in tools/cb-cost.py and emitted by `make cost-mix`, so the baseline is reproducible and the same command can later falsify the predictions. mech environment setup 84 turns $15.33 mech ad-hoc text patching 75 turns $13.86 git 37 turns $13.85 mech hub task status 25 turns $ 7.46 mech orientation / inspect 49 turns $ 6.87 hub other 32 turns $ 6.22 mech ad-hoc transcript 39 turns $ 4.56 mech workplan status edit 21 turns $ 4.06 MECHANICAL (dedup) 290 turns $51.26 = 38% of pass Largest category is `cd` and `export PATH` -- pure friction, and dep-weight.py already patched it at the leaf, which is evidence it was noticed and fixed in the wrong place. Second is heredocs string-patching markdown, which is also the mechanism behind duplicated-fact drift, the error class InnerLoop v1.2 names and cannot gate. Explicitly NOT automated: git (37 turns, $13.85) is mostly commit message authorship -- the highest-output turns in the corpus and the project's reasoning record. Automating it would save money and destroy what makes corrections cheap. CB-WP-0004 implements five candidates and predicts $33-41 recovery (25-30%), below the 38% measured share on purpose: some inspection and patching is genuinely exploratory. The control loop is the deliverable, not a formality. T05 tests three things and must report all: did mechanical turns disappear, did they RELOCATE into prose, and did quality hold. If mechanical turns fall and prose rises by as much, the saving is zero and that is the result to publish. Also a self-indictment worth recording: every hub update_task_status in this project carried hand-typed token estimates, in a repo whose central finding is that estimated token counts are worthless. T02 fixes it by reading measured values from cb-cost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:49:57 +02:00
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
cost-pin: cost-test
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(PY) $(TOOLS)/cb-cost.py --pin fc76445 --composition --by-task
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
sim:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(IN_REPO) $(CARGO) run -q -p cb-sim -- $(REPO)/scenarios/ground/*.yaml
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
## Criterion benches (AM-6/AM-7)
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
bench:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(IN_REPO) $(CARGO) bench -p games-ground
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
## InnerLoop positive control: run every bench once, no measurement.
## Fails if a workload stalls or produces the wrong event count.
bench-test:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
$(IN_REPO) $(CARGO) bench -p games-ground --bench synthetic -- --test
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
## AM-2/AM-3 input: source LOC per crate (excludes tests would need tokei)
loc:
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
@$(IN_REPO) for d in crates/cb-kernel crates/cb-events crates/cb-game-runtime games/ground tools/cb-sim; do \
T07: Cargo workspace scaffold — cb-kernel/cb-events/cb-game-runtime/games-ground/cb-sim, HashMap deny-lint, scenario format + runner stub, Criterion skeleton, Makefile, CI
2026-07-31 01:57:13 +02:00
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
done
CB-WP-0004 T01: fix environment friction at the root CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical category — spent prefixing commands with `cd` and `export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once instead of per-leaf. tools/repo.py resolves the repo root from __file__ and cargo from PATH then the standard rustup locations. Every tool imports ROOT from it, so the repo path is stated once rather than redefined in four files — single source of fact, the rule DFD earned in InnerLoop v1.2. rule-coverage and dep-weight now call enter_root(), which is why their relative paths did not need rewriting one by one. The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same way, so `make -C <repo> <target>` works from any directory with no prefix. make env-test is the positive control, and is in `make all`: every tool runs from / with PATH=/usr/bin:/bin. Without it this fix could regress silently and invalidate T05's measurement — the whole point of the control loop. dep-weight's "cargo not on PATH" error is kept rather than deleted. It should now be unreachable, and --self-test asserts cargo_bin() resolves unaided; a control that never fires is cheaper than a regression. loop-lint failed on repo.py on its first run — a reporting tool with a positive control but no --self-test entry point. Second time the gate has caught work from its own pass within the hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:13:52 +02:00
all: check test sim coverage dep-weight self-tests env-test loop-lint bench-test
Reference in a new issue Copy permalink