Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.
Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.
Three errors found in this workplan's own Purpose section, all
overstating cost:
- one API response is written as up to six JSONL lines, each
repeating the same usage object (657 lines, 346 responses)
- a multi-model session was priced entirely at Fable 5 rates,
violating a per-model rule MetricsAndScenarios §1a already had
- the transcript is a live file, so an unpinned total is not
repeatable (346 -> 356 responses between two runs minutes apart)
CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.
Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.
Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written back by fix-consistency: workstream and task UUIDs for the ten
CB-WP-0003 tasks, plus the regenerated WORK-RECORDS.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers what the immediate CI fixes did not. Framing finding: writing a
rule into InnerLoop did not prevent the next instance of the failure it
was written for; making it a CI step caught one on its first run. Four
instances of the harness-does-nothing class, zero prevented by prose.
Working hypothesis: a loop rule that cannot be executed is not a rule.
Each item is made executable or deleted, default delete.
Phase A makes rules executable: audit every InnerLoop rule as
executable/checkable/decorative, extend the positive-control gate beyond
benchmarks, and point adversarial review at measurement rather than only
research — all four real errors were in measurement and build config,
which the review never reads.
Phase B is session economics, from measurement: 592 turns, mean context
245k tokens, $0.245/turn in cache reads, zero batched tool calls, cache
reads 58% of cost against output at 12%. Cost is turns x context and
context grows with turns, so a long session is quadratic. Also replaces
the dead 8k/10k token budget, which was never enforced and which T08
exceeded by orders of magnitude with no signal.
Phase C removes or fixes the rest: resolve the chaos roll (never fired,
and at 1-in-10 will stay unevaluated), require retargeting to be an ADR
rather than something the implementer does in the commit that measures
the metric, give provisional items an owner and expiry, and strengthen
the coverage gate beyond tag-counting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adopts both remediations from CB-EV-0001 §4 (maintainer decision).
Option A — serde_yaml is now optional behind cb-game-runtime's
`scenarios` feature. The scenario module, the ScenarioGame impl and the
string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both
configurations compile and lint clean under -D warnings.
A trap worth recording: `default-features = false` on a *member*
dependency is silently ignored when the workspace dependency does not
specify it. The first attempt gated nothing while looking correct — the
build succeeded and cargo tree still showed all six YAML crates. Fixed
by setting it on the workspace dependency. This is the positive-control
failure mode in miniature: success was not evidence the change applied.
Retarget — AM-4 now measures third-party source under audit, split by
build configuration, replacing a crate count that was unreachable
without undoing K5/K7 and that does not compare across ecosystems.
Re-measured via the new `make dep-weight`, whose own positive control
refuses to report when any crate's source cannot be located:
shipped runtime 23 crates 246,250 lines target <=250,000 met
dev toolchain 29 crates 317,021 lines target <=350,000 met
own source 3,408 lines
Scenario tooling costs 70,771 lines a shipped game never compiles —
the split the single number was hiding.
Targets are set at current measurement plus headroom, so they bind on
future growth rather than retroactively passing what had failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AM-4: measured what each remediation option actually buys, rather than
leaving one recommendation unquantified. serde_yaml optional removes 6
crates, not 5 — ryu belongs to that group, since serde_json now uses
zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23,
inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only
reimplementing a primitive gets under 20, so the target is unreachable
without undoing K5/K7.
Also records that crate count compares badly across ecosystems, and
offers the alternative the count is a proxy for: 307,317 lines of
third-party source under audit against 3,398 of our own.
AM-12: corrected from "uncomputable" to measured. The refusal to
estimate was right; the claim that no instrument existed was wrong.
Session transcripts carry exact per-message usage including the cache
breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is
cache reads — cost is driven by context size times turn count, not by
output volume. What is still missing is per-task attribution, since
nothing marks task boundaries in a transcript.
CB-WP-0002 makes cost measurable and attributable: survey the
instruments, decide the attribution model by ADR, spec metrics that
include cost composition rather than a bare total, build a collector
whose positive control refuses to emit unreconciled numbers, and prove
it by answering a question that could not be answered before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop has now survived CB-WP-0001 end to end and is revised from what
that pass actually taught, recorded in
history/260731-inner-loop-retrospective.md.
Held up: the adversarial review (one blocking and three significant
findings on its single use, resolved with new measurements rather than
argument), the parity-cap rule, the provisional U-item mechanism, and
the ADR gate.
The gap: both serious errors in the pass were measurement errors, and
the loop caught neither. Review reads prose; these were claims about
numbers. In both cases a harness ran successfully while doing no work —
a JS harness timing rejected no-ops, and a Rust benchmark computing
throughput for rounds that never completed.
v1.0 adds:
- Measurement validity: a harness must assert it performed the work it
reports. A number from a run that cannot prove it did the work is void.
- Metric feasibility: every metric names its instrument and is checked
reachable against the contracts in its own spec.
- No silently-ignored input; decisions get commands, not defaults;
scaffolds are exercised or marked; coverage gates that count tags say
so.
- Evidence must state what a comparison does not support.
- The chaos roll is recorded even when it changes nothing, so a
mechanism that never fires is visible rather than assumed.
CB-WP-0001 is complete: 9/9 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evidence/CB-EV-0001-game-kernel.md records the acceptance run against
the CB-RES-0001 baseline.
Met: AM-1 rule coverage 58/58; AM-6 throughput 1.65M events/s against a
100k target; AM-7 scaling 1.08x at 20x workload and a 100k-event replay
in 4.13ms against a 5s budget; AM-8 zero divergence over 10 full runs
with fmt and clippy clean; AM-10 zero foreign collection types.
Not met and reported as such: AM-4 at 33 transitive crates against a
<=20 target. Attribution is in the evidence file. The recommended fix
is making serde_yaml optional (-5, a test-only concern), after which
the remainder is sha2 and rand_chacha, which K5 and K7 require. We are
not hand-rolling crypto primitives to win a dependency count.
AM-12 is recorded as uncomputable: per-task token counts were never
instrumented, and inventing a USD figure would defeat the metric.
A measurement error was found and corrected before publication. The
first benchmark reported 9.3M events/s on a flat curve. The workload
had a player selecting SUPPORT while parked at Stress 4, so GR-R03
rejected it, rounds never completed, and throughput was computed for
rounds that never happened. The bench now asserts the per-round event
count and panics rather than measuring a stalled loop. The corrected
figure is 5.6x lower.
The evidence file states plainly what the boardgame.io comparison does
and does not support: the ~450x command-rate ratio is cross-runtime and
cross-feature-set, so it is a direction, not a verdict, per the
InnerLoop parity-cap rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
