Amend CB-EV-0001; add CB-WP-0002 for cost accounting
Some checks failed
ci / check (push) Failing after 4s
Some checks failed
ci / check (push) Failing after 4s
AM-4: measured what each remediation option actually buys, rather than leaving one recommendation unquantified. serde_yaml optional removes 6 crates, not 5 — ryu belongs to that group, since serde_json now uses zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23, inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only reimplementing a primitive gets under 20, so the target is unreachable without undoing K5/K7. Also records that crate count compares badly across ecosystems, and offers the alternative the count is a proxy for: 307,317 lines of third-party source under audit against 3,398 of our own. AM-12: corrected from "uncomputable" to measured. The refusal to estimate was right; the claim that no instrument existed was wrong. Session transcripts carry exact per-message usage including the cache breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is cache reads — cost is driven by context size times turn count, not by output volume. What is still missing is per-task attribution, since nothing marks task boundaries in a transcript. CB-WP-0002 makes cost measurable and attributable: survey the instruments, decide the attribution model by ADR, spec metrics that include cost composition rather than a bare total, build a collector whose positive control refuses to emit unreconciled numbers, and prove it by answering a question that could not be answered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
5ea1497e47
commit
8e11fc412e
4 changed files with 240 additions and 47 deletions
|
|
@ -1,7 +1,9 @@
|
|||
# CB-EV-0001 — GROUND game kernel: acceptance evidence
|
||||
|
||||
Status: **T08 complete, with one acceptance metric not met (AM-4).**
|
||||
Recorded: 2026-07-31
|
||||
Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per
|
||||
remediation option, and §5 corrects AM-12 from "uncomputable" to
|
||||
measured-at-session-level; see CB-WP-0002.
|
||||
Workplan: CB-WP-0001, task T08
|
||||
Spec: `specs/GameKernel.md` §4 (AM-1..AM-12)
|
||||
Baseline: `research/CB-RES-0001-game-kernel.md`, measurements in
|
||||
|
|
@ -23,7 +25,7 @@ Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1,
|
|||
| AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** |
|
||||
| AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** |
|
||||
| AM-11 impl pairs | null + reference per port | 1 of 1 (`KernelRng`) | **met, narrow** |
|
||||
| AM-12 cost log | present | §5 | **met** |
|
||||
| AM-12 cost | per-task USD | $248.46 session; per-task pending | **partial** |
|
||||
|
||||
AM-2, AM-3, AM-5 and AM-9 are not reported: see §6.
|
||||
|
||||
|
|
@ -121,8 +123,8 @@ Attribution:
|
|||
|---|---|---|
|
||||
| `sha2` (K7 state hashing) | sha2, digest, block-buffer, crypto-common, generic-array, typenum, cpufeatures, cfg-if | 8 |
|
||||
| serde derive chain | serde_derive, proc-macro2, quote, syn, unicode-ident | 5 |
|
||||
| serde runtime + json | serde, serde_core, serde_json, itoa, ryu, memchr, zmij | 7 |
|
||||
| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent | 5 |
|
||||
| serde runtime + json | serde, serde_core, serde_json, itoa, memchr, zmij | 6 |
|
||||
| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent, ryu | 6 |
|
||||
| `rand_chacha` (K5 seeded RNG) | rand_chacha, rand_core, ppv-lite86, zerocopy | 4 |
|
||||
| Clay-Borg crates | cb-kernel, cb-events, cb-game-runtime, games-ground | 4 |
|
||||
|
||||
|
|
@ -130,20 +132,31 @@ The honest options, in order of preference:
|
|||
|
||||
1. **Make `serde_yaml` optional** behind a `scenarios` feature. YAML is
|
||||
a test-and-tooling concern; a shipped game runtime does not need it.
|
||||
Removes 5 crates from the default build for no loss of capability.
|
||||
Removes 6 crates from the default build for no loss of capability
|
||||
(`ryu` belongs to this group, not to serde_json, which uses `zmij`
|
||||
for floats — corrected after measuring the reverse-dependency graph).
|
||||
This is the one to do first, and it improves D4 optionality as well
|
||||
as D2.
|
||||
2. **Revisit the target.** ≤20 was set before the K5/K7 contracts named
|
||||
ChaCha and SHA-256. Those two contracts cost 12 crates between them
|
||||
and are load-bearing for determinism. A target that a spec's own
|
||||
contracts make unreachable is a bad target.
|
||||
2. **Revisit the target, and what it measures.** Measured savings per
|
||||
option: serde_yaml optional −6 (→27); replacing serde_json −4 more
|
||||
(→23); inlining SHA-256 −8 (→19); inlining ChaCha12 −4 (→15). **Only
|
||||
reimplementing SHA-256 or ChaCha gets under 20**, so the target is
|
||||
unreachable without undoing K5/K7.
|
||||
|
||||
Crate count also compares badly across ecosystems: Rust splits
|
||||
crates far more finely than npm, so "33 vs 120 npm packages" flatters
|
||||
us. The measurable thing crate count proxies for is third-party
|
||||
source under audit: **307,317 lines** across all five groups, against
|
||||
3,398 of our own. Retargeting AM-4 on audited third-party LOC, split
|
||||
into shipped-runtime and dev-toolchain, measures the real concern and
|
||||
cannot be gamed by crate granularity.
|
||||
|
||||
What we are **not** doing: hand-rolling SHA-256 or ChaCha to win a
|
||||
dependency count. That trades an auditable, well-tested primitive for a
|
||||
number on a scoreboard.
|
||||
|
||||
This is a T09 input: either the metric moves for a stated reason, or
|
||||
option 1 lands and the remainder is justified.
|
||||
Carried forward as an open decision (see the note at the head of this
|
||||
file): AM-4 is re-measured once the option is chosen.
|
||||
|
||||
## 5. Cost log (AM-12)
|
||||
|
||||
|
|
@ -152,14 +165,30 @@ Per `specs/MetricsAndScenarios.md` §1a. Model: Claude Fable 5, at
|
|||
|
||||
| Task | Model | Iterations | Notes |
|
||||
|---|---|---|---|
|
||||
| T08 | claude-fable-5 | 6 code iterations + benchmarks | Token counts not captured per iteration; see limitation below |
|
||||
| CB-WP-0001 (whole session, T01–T09) | claude-fable-5 | 6 T08 code iterations + benchmarks | $248.46 measured; per-task split pending CB-WP-0002 |
|
||||
|
||||
**Limitation, stated rather than fabricated:** exact per-task token
|
||||
counts were not instrumented during T08, so the USD figure the metric
|
||||
asks for cannot be computed honestly from this run. Recording an
|
||||
estimate here would defeat the purpose of the metric. T09 should either
|
||||
wire real token accounting into the loop or drop M-D2-CST as
|
||||
unmeasurable in this setup.
|
||||
**Correction (2026-07-31).** This section originally recorded AM-12 as
|
||||
*uncomputable*. That was wrong. The declining to estimate was right; the
|
||||
conclusion that no instrument existed was not. Every session transcript
|
||||
(`~/.claude/projects/<slug>/<session>.jsonl`) carries exact per-message
|
||||
`usage` including the cache breakdown. Read for this session:
|
||||
|
||||
| Component | Tokens | Cost (Fable 5) |
|
||||
|---|---|---|
|
||||
| Output | 585,528 | $29.28 |
|
||||
| Cache read | 131,863,164 | $131.86 |
|
||||
| Cache write (1h) | 4,365,668 | $87.31 |
|
||||
| Input | 1,090 | $0.01 |
|
||||
| **Session total** | | **$248.46** (~$124 on Opus 5) |
|
||||
|
||||
**53% of the cost is cache reads**, not output. Cost in an agentic loop
|
||||
is driven by context size × turn count, which no "tokens per task"
|
||||
metric would have surfaced.
|
||||
|
||||
Still missing is *attribution*: this is a whole-session figure, not a
|
||||
per-task one, because nothing marks task boundaries in the transcript.
|
||||
That is what CB-WP-0002 is for. The AM-12 row above should be read as
|
||||
"session-level cost measured; per-task attribution pending CB-WP-0002".
|
||||
|
||||
## 6. Metrics not reported
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue