Amend CB-EV-0001; add CB-WP-0002 for cost accounting
Some checks failed
ci / check (push) Failing after 4s

AM-4: measured what each remediation option actually buys, rather than
leaving one recommendation unquantified. serde_yaml optional removes 6
crates, not 5 — ryu belongs to that group, since serde_json now uses
zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23,
inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only
reimplementing a primitive gets under 20, so the target is unreachable
without undoing K5/K7.

Also records that crate count compares badly across ecosystems, and
offers the alternative the count is a proxy for: 307,317 lines of
third-party source under audit against 3,398 of our own.

AM-12: corrected from "uncomputable" to measured. The refusal to
estimate was right; the claim that no instrument existed was wrong.
Session transcripts carry exact per-message usage including the cache
breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is
cache reads — cost is driven by context size times turn count, not by
output volume. What is still missing is per-task attribution, since
nothing marks task boundaries in a transcript.

CB-WP-0002 makes cost measurable and attributable: survey the
instruments, decide the attribution model by ADR, spec metrics that
include cost composition rather than a bare total, build a collector
whose positive control refuses to emit unreconciled numbers, and prove
it by answering a question that could not be answered before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 03:26:03 +02:00
parent 5ea1497e47
commit 8e11fc412e
4 changed files with 240 additions and 47 deletions

View file

@ -1,7 +1,9 @@
# CB-EV-0001 — GROUND game kernel: acceptance evidence
Status: **T08 complete, with one acceptance metric not met (AM-4).**
Recorded: 2026-07-31
Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per
remediation option, and §5 corrects AM-12 from "uncomputable" to
measured-at-session-level; see CB-WP-0002.
Workplan: CB-WP-0001, task T08
Spec: `specs/GameKernel.md` §4 (AM-1..AM-12)
Baseline: `research/CB-RES-0001-game-kernel.md`, measurements in
@ -23,7 +25,7 @@ Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1,
| AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** |
| AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** |
| AM-11 impl pairs | null + reference per port | 1 of 1 (`KernelRng`) | **met, narrow** |
| AM-12 cost log | present | §5 | **met** |
| AM-12 cost | per-task USD | $248.46 session; per-task pending | **partial** |
AM-2, AM-3, AM-5 and AM-9 are not reported: see §6.
@ -121,8 +123,8 @@ Attribution:
|---|---|---|
| `sha2` (K7 state hashing) | sha2, digest, block-buffer, crypto-common, generic-array, typenum, cpufeatures, cfg-if | 8 |
| serde derive chain | serde_derive, proc-macro2, quote, syn, unicode-ident | 5 |
| serde runtime + json | serde, serde_core, serde_json, itoa, ryu, memchr, zmij | 7 |
| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent | 5 |
| serde runtime + json | serde, serde_core, serde_json, itoa, memchr, zmij | 6 |
| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent, ryu | 6 |
| `rand_chacha` (K5 seeded RNG) | rand_chacha, rand_core, ppv-lite86, zerocopy | 4 |
| Clay-Borg crates | cb-kernel, cb-events, cb-game-runtime, games-ground | 4 |
@ -130,20 +132,31 @@ The honest options, in order of preference:
1. **Make `serde_yaml` optional** behind a `scenarios` feature. YAML is
a test-and-tooling concern; a shipped game runtime does not need it.
Removes 5 crates from the default build for no loss of capability.
Removes 6 crates from the default build for no loss of capability
(`ryu` belongs to this group, not to serde_json, which uses `zmij`
for floats — corrected after measuring the reverse-dependency graph).
This is the one to do first, and it improves D4 optionality as well
as D2.
2. **Revisit the target.** ≤20 was set before the K5/K7 contracts named
ChaCha and SHA-256. Those two contracts cost 12 crates between them
and are load-bearing for determinism. A target that a spec's own
contracts make unreachable is a bad target.
2. **Revisit the target, and what it measures.** Measured savings per
option: serde_yaml optional 6 (→27); replacing serde_json 4 more
(→23); inlining SHA-256 8 (→19); inlining ChaCha12 4 (→15). **Only
reimplementing SHA-256 or ChaCha gets under 20**, so the target is
unreachable without undoing K5/K7.
Crate count also compares badly across ecosystems: Rust splits
crates far more finely than npm, so "33 vs 120 npm packages" flatters
us. The measurable thing crate count proxies for is third-party
source under audit: **307,317 lines** across all five groups, against
3,398 of our own. Retargeting AM-4 on audited third-party LOC, split
into shipped-runtime and dev-toolchain, measures the real concern and
cannot be gamed by crate granularity.
What we are **not** doing: hand-rolling SHA-256 or ChaCha to win a
dependency count. That trades an auditable, well-tested primitive for a
number on a scoreboard.
This is a T09 input: either the metric moves for a stated reason, or
option 1 lands and the remainder is justified.
Carried forward as an open decision (see the note at the head of this
file): AM-4 is re-measured once the option is chosen.
## 5. Cost log (AM-12)
@ -152,14 +165,30 @@ Per `specs/MetricsAndScenarios.md` §1a. Model: Claude Fable 5, at
| Task | Model | Iterations | Notes |
|---|---|---|---|
| T08 | claude-fable-5 | 6 code iterations + benchmarks | Token counts not captured per iteration; see limitation below |
| CB-WP-0001 (whole session, T01T09) | claude-fable-5 | 6 T08 code iterations + benchmarks | $248.46 measured; per-task split pending CB-WP-0002 |
**Limitation, stated rather than fabricated:** exact per-task token
counts were not instrumented during T08, so the USD figure the metric
asks for cannot be computed honestly from this run. Recording an
estimate here would defeat the purpose of the metric. T09 should either
wire real token accounting into the loop or drop M-D2-CST as
unmeasurable in this setup.
**Correction (2026-07-31).** This section originally recorded AM-12 as
*uncomputable*. That was wrong. The declining to estimate was right; the
conclusion that no instrument existed was not. Every session transcript
(`~/.claude/projects/<slug>/<session>.jsonl`) carries exact per-message
`usage` including the cache breakdown. Read for this session:
| Component | Tokens | Cost (Fable 5) |
|---|---|---|
| Output | 585,528 | $29.28 |
| Cache read | 131,863,164 | $131.86 |
| Cache write (1h) | 4,365,668 | $87.31 |
| Input | 1,090 | $0.01 |
| **Session total** | | **$248.46** (~$124 on Opus 5) |
**53% of the cost is cache reads**, not output. Cost in an agentic loop
is driven by context size × turn count, which no "tokens per task"
metric would have surfaced.
Still missing is *attribution*: this is a whole-session figure, not a
per-task one, because nothing marks task boundaries in the transcript.
That is what CB-WP-0002 is for. The AM-12 row above should be read as
"session-level cost measured; per-task attribution pending CB-WP-0002".
## 6. Metrics not reported