2026-07-31 01:57:13 +02:00
|
|
|
# One command surface (InnerLoop §agentic-efficiency #3). Deterministic,
|
|
|
|
|
# greppable output; precursor of the `cb` CLI.
|
|
|
|
|
|
|
|
|
|
CARGO := cargo
|
|
|
|
|
|
T04: tools/cb-cost.py — and its positive control fires on first contact
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.
The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:
CA-02 assumed usage is identical across the lines of one requestId.
True in the main transcript (206/206 groups, verified twice — by the
survey and by the adversarial reviewer). FALSE in the subagents/
tree, where output_tokens is a running count: one response reads
5, 5, 195 across its three lines. First-wins scored it at 5.
So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.
The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.
Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.
make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
|
|
|
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all
|
2026-07-31 01:57:13 +02:00
|
|
|
|
|
|
|
|
## fmt + clippy (deny warnings) + HashMap deny-lint
|
|
|
|
|
check:
|
|
|
|
|
$(CARGO) fmt --all --check
|
|
|
|
|
$(CARGO) clippy --workspace --all-targets -- -D warnings
|
|
|
|
|
|
|
|
|
|
## unit + scenario-format tests
|
|
|
|
|
test:
|
|
|
|
|
$(CARGO) test --workspace
|
|
|
|
|
|
|
|
|
|
## run all GROUND scenarios through cb-sim
|
AM-4: gate scenario YAML, retarget on audited source, re-measure
Adopts both remediations from CB-EV-0001 §4 (maintainer decision).
Option A — serde_yaml is now optional behind cb-game-runtime's
`scenarios` feature. The scenario module, the ScenarioGame impl and the
string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both
configurations compile and lint clean under -D warnings.
A trap worth recording: `default-features = false` on a *member*
dependency is silently ignored when the workspace dependency does not
specify it. The first attempt gated nothing while looking correct — the
build succeeded and cargo tree still showed all six YAML crates. Fixed
by setting it on the workspace dependency. This is the positive-control
failure mode in miniature: success was not evidence the change applied.
Retarget — AM-4 now measures third-party source under audit, split by
build configuration, replacing a crate count that was unreachable
without undoing K5/K7 and that does not compare across ecosystems.
Re-measured via the new `make dep-weight`, whose own positive control
refuses to report when any crate's source cannot be located:
shipped runtime 23 crates 246,250 lines target <=250,000 met
dev toolchain 29 crates 317,021 lines target <=350,000 met
own source 3,408 lines
Scenario tooling costs 70,771 lines a shipped game never compiles —
the split the single number was hiding.
Targets are set at current measurement plus headroom, so they bind on
future growth rather than retroactively passing what had failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 03:35:41 +02:00
|
|
|
dep-weight:
|
|
|
|
|
python3 tools/dep-weight.py
|
|
|
|
|
|
2026-07-31 02:23:38 +02:00
|
|
|
coverage:
|
|
|
|
|
python3 tools/rule-coverage.py
|
|
|
|
|
|
T04: tools/cb-cost.py — and its positive control fires on first contact
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.
The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:
CA-02 assumed usage is identical across the lines of one requestId.
True in the main transcript (206/206 groups, verified twice — by the
survey and by the adversarial reviewer). FALSE in the subagents/
tree, where output_tokens is a running count: one response reads
5, 5, 195 across its three lines. First-wins scored it at 5.
So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.
The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.
Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.
make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
|
|
|
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
|
|
|
|
|
# runs first: a cost number from an unverified collector is void.
|
|
|
|
|
cost: cost-test
|
|
|
|
|
python3 tools/cb-cost.py --composition --by-task
|
|
|
|
|
|
|
|
|
|
cost-test:
|
|
|
|
|
python3 tools/cb-cost.py --self-test
|
|
|
|
|
|
|
|
|
|
cost-pin: cost-test
|
|
|
|
|
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
|
|
|
|
|
2026-07-31 01:57:13 +02:00
|
|
|
sim:
|
|
|
|
|
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
|
|
|
|
|
|
CI: enforce every gate; close the silent-skip holes
The gates existed; CI ran half of them and tolerated the failure case.
- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
prefix is a failure, and a run in which nothing executed is a failure.
Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
would have skipped every scenario while the pipeline stayed green.
Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
crate-count metric.
The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.
That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.
Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.
The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
|
|
|
## Criterion benches (AM-6/AM-7)
|
2026-07-31 01:57:13 +02:00
|
|
|
bench:
|
|
|
|
|
$(CARGO) bench -p games-ground
|
|
|
|
|
|
CI: enforce every gate; close the silent-skip holes
The gates existed; CI ran half of them and tolerated the failure case.
- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
prefix is a failure, and a run in which nothing executed is a failure.
Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
would have skipped every scenario while the pipeline stayed green.
Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
crate-count metric.
The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.
That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.
Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.
The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
|
|
|
## InnerLoop positive control: run every bench once, no measurement.
|
|
|
|
|
## Fails if a workload stalls or produces the wrong event count.
|
|
|
|
|
bench-test:
|
|
|
|
|
$(CARGO) bench -p games-ground --bench synthetic -- --test
|
|
|
|
|
|
2026-07-31 01:57:13 +02:00
|
|
|
|
|
|
|
|
## AM-2/AM-3 input: source LOC per crate (excludes tests would need tokei)
|
|
|
|
|
loc:
|
|
|
|
|
@for d in crates/cb-kernel crates/cb-events crates/cb-game-runtime games/ground tools/cb-sim; do \
|
|
|
|
|
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
|
|
|
|
|
done
|
|
|
|
|
|
T04: tools/cb-cost.py — and its positive control fires on first contact
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.
The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:
CA-02 assumed usage is identical across the lines of one requestId.
True in the main transcript (206/206 groups, verified twice — by the
survey and by the adversarial reviewer). FALSE in the subagents/
tree, where output_tokens is a running count: one response reads
5, 5, 195 across its three lines. First-wins scored it at 5.
So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.
The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.
Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.
make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
|
|
|
all: check test sim coverage dep-weight cost-test bench-test
|