tegwick
4be6e020ea
Some checks failed
ci / check (push) Failing after 3s
Adopts both remediations from CB-EV-0001 §4 (maintainer decision). Option A — serde_yaml is now optional behind cb-game-runtime's `scenarios` feature. The scenario module, the ScenarioGame impl and the string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both configurations compile and lint clean under -D warnings. A trap worth recording: `default-features = false` on a *member* dependency is silently ignored when the workspace dependency does not specify it. The first attempt gated nothing while looking correct — the build succeeded and cargo tree still showed all six YAML crates. Fixed by setting it on the workspace dependency. This is the positive-control failure mode in miniature: success was not evidence the change applied. Retarget — AM-4 now measures third-party source under audit, split by build configuration, replacing a crate count that was unreachable without undoing K5/K7 and that does not compare across ecosystems. Re-measured via the new `make dep-weight`, whose own positive control refuses to report when any crate's source cannot be located: shipped runtime 23 crates 246,250 lines target <=250,000 met dev toolchain 29 crates 317,021 lines target <=350,000 met own source 3,408 lines Scenario tooling costs 70,771 lines a shipped game never compiles — the split the single number was hiding. Targets are set at current measurement plus headroom, so they bind on future growth rather than retroactively passing what had failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
193 lines
7 KiB
Markdown
193 lines
7 KiB
Markdown
---
|
||
id: CB-WP-0002
|
||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||
status: proposed
|
||
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
||
---
|
||
|
||
# Purpose
|
||
|
||
CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price
|
||
sheet, formula, staleness rule — and then could not report a number,
|
||
because nothing was ever instrumented. It was recorded as
|
||
"uncomputable".
|
||
|
||
That was wrong, and the retrospective for this workplan should say so
|
||
plainly: **the data existed the whole time.** Every Claude Code session
|
||
transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
|
||
per-message `usage`, including the cache breakdown. Reading it for
|
||
CB-WP-0001's session gives:
|
||
|
||
| Component | Tokens | Cost (Fable 5) |
|
||
|---|---|---|
|
||
| Output | 585,528 | $29.28 |
|
||
| Cache read | 131,863,164 | **$131.86** |
|
||
| Cache write (1h) | 4,365,668 | $87.31 |
|
||
| Input | 1,090 | $0.01 |
|
||
| **Total** | | **$248.46** |
|
||
|
||
The headline finding from that single reading is the reason this
|
||
workplan exists: **53% of the cost is cache reads**, not output. Cost in
|
||
an agentic loop is driven by context size × turn count, not by how much
|
||
the model writes. No D2 decision made on "tokens per task" would have
|
||
surfaced that.
|
||
|
||
This workplan makes cost a first-class measured dimension so that
|
||
future AM-12 equivalents support conclusions instead of decorating an
|
||
evidence file. It follows InnerLoop v1.0, including the rules that pass
|
||
earned: every metric names its instrument, and every harness carries a
|
||
positive control.
|
||
|
||
## Phase A — Instrument
|
||
|
||
## Task: Survey the available cost instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T01
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
|
||
```
|
||
|
||
Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop
|
||
survey template. Candidates at minimum: session transcript JSONL
|
||
(`usage` per assistant message), the Custodian State Hub token-event
|
||
API (`record_token_event`, `update_task_status` token tiers,
|
||
`get_token_summary`), the Claude Code status bar, and the Anthropic
|
||
usage/billing API. Per candidate: what it reports, granularity,
|
||
accuracy, whether it survives compaction, and whether it can attribute
|
||
cost to a unit of work.
|
||
|
||
Note explicitly which are **exact** and which are **estimates** — the
|
||
prior failure was tolerating an estimate-shaped metric. Name the
|
||
benchmark-to-beat per dimension; expect the transcript to lead on
|
||
accuracy and the hub to lead on durability.
|
||
|
||
## Task: Decide the instrument and the attribution model (ADR)
|
||
|
||
```task
|
||
id: CB-WP-0002-T02
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
||
```
|
||
|
||
Adversarial review of T01 first (InnerLoop §Step 2), committed as
|
||
`history/YYMMDD-cost-accounting-{challenge,response}.md`.
|
||
|
||
Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is
|
||
**attribution**: a transcript is a flat message stream, and a "task" is
|
||
a workplan concept. Options to weigh, not assume:
|
||
|
||
- git commit timestamps as task boundaries (the loop commits per
|
||
iteration, so boundaries already exist and are durable)
|
||
- explicit session markers emitted at task start/end
|
||
- hub task status transitions as the time index
|
||
|
||
State the expected advantage per dimension and the known failure modes
|
||
of the chosen model — in particular, what happens to attribution across
|
||
`/compact`, across resumed sessions, and for work spanning a boundary.
|
||
Gate: no collector code before this ADR is committed.
|
||
|
||
## Task: Specify the cost metrics with named instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T03
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
|
||
```
|
||
|
||
Write `specs/CostAccounting.md`: the cost model (input, output, cache
|
||
read, cache write at 5m and 1h, per the price sheet), the attribution
|
||
contract, and the acceptance metrics — each naming the **command that
|
||
produces its number**, per InnerLoop v1.0 §Step 4.
|
||
|
||
Must include a metric for the finding that motivated this workplan:
|
||
**cost composition** (what fraction is cache read vs write vs output),
|
||
not only a total. A single total would have hidden the 53%.
|
||
|
||
Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and
|
||
replace AM-12's definition with one that is computable.
|
||
|
||
## Phase B — Build and prove
|
||
|
||
## Task: Implement the cost collector
|
||
|
||
```task
|
||
id: CB-WP-0002-T04
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
|
||
```
|
||
|
||
Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads
|
||
transcripts, applies the price sheet at
|
||
`benchmarks/baselines/model-prices.toml`, attributes cost per the T02
|
||
model, and emits both a per-task table and a composition breakdown in
|
||
the evidence-row format from T03.
|
||
|
||
**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool
|
||
asserts that attributed tokens sum to the transcript total, and refuses
|
||
to emit numbers when they do not reconcile. An unattributed remainder is
|
||
reported as its own line, never silently dropped — the failure this
|
||
whole workplan exists to prevent was a number that looked fine.
|
||
|
||
Handle multi-model sessions: a session that switches models must price
|
||
each message at its own model's rate.
|
||
|
||
## Task: Validate against CB-WP-0001 and answer a real question
|
||
|
||
```task
|
||
id: CB-WP-0002-T05
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
|
||
```
|
||
|
||
Run the collector over the CB-WP-0001 session and commit
|
||
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
|
||
totals in this workplan's Purpose section is the acceptance test: the
|
||
tool must reproduce $248.46 (Fable 5) from the same transcript, or
|
||
explain the difference.
|
||
|
||
Then use it to answer at least one question that could not be answered
|
||
before, and record the answer. Candidates: which of T01–T09 cost most
|
||
and whether that matched its value; what a `/compact` costs; whether the
|
||
adversarial review paid for itself; how much the six T08 code iterations
|
||
cost relative to the research phase.
|
||
|
||
A tool that produces numbers nobody draws a conclusion from has not
|
||
cleared the bar that CB-WP-0001's AM-12 failed to clear.
|
||
|
||
## Task: Wire cost into the loop
|
||
|
||
```task
|
||
id: CB-WP-0002-T06
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
|
||
```
|
||
|
||
Make cost collection automatic rather than remembered: add the
|
||
collector to the evidence checklist in `specs/InnerLoop.md`, emit hub
|
||
token events so the Token Cost dashboard reflects real numbers rather
|
||
than the 1000/500 heuristic fallback, and add a `make cost` target to
|
||
the one command surface.
|
||
|
||
## Task: Retrospective
|
||
|
||
```task
|
||
id: CB-WP-0002-T07
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
||
```
|
||
|
||
Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what
|
||
this pass teaches about metric design. The specific question to answer:
|
||
CB-WP-0001 produced a fully specified metric that could not be computed,
|
||
and v1.0's "every metric names its instrument" rule was written to stop
|
||
that. Did it? If a metric can still be written without a working
|
||
instrument, the rule needs teeth — for example, requiring that the
|
||
instrument be demonstrated on real data before the metric is accepted.
|