tegwick
8e11fc412e
Some checks failed
ci / check (push) Failing after 4s
AM-4: measured what each remediation option actually buys, rather than leaving one recommendation unquantified. serde_yaml optional removes 6 crates, not 5 — ryu belongs to that group, since serde_json now uses zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23, inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only reimplementing a primitive gets under 20, so the target is unreachable without undoing K5/K7. Also records that crate count compares badly across ecosystems, and offers the alternative the count is a proxy for: 307,317 lines of third-party source under audit against 3,398 of our own. AM-12: corrected from "uncomputable" to measured. The refusal to estimate was right; the claim that no instrument existed was wrong. Session transcripts carry exact per-message usage including the cache breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is cache reads — cost is driven by context size times turn count, not by output volume. What is still missing is per-task attribution, since nothing marks task boundaries in a transcript. CB-WP-0002 makes cost measurable and attributable: survey the instruments, decide the attribution model by ADR, spec metrics that include cost composition rather than a bare total, build a collector whose positive control refuses to emit unreconciled numbers, and prove it by answering a question that could not be answered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
185 lines
6.6 KiB
Markdown
185 lines
6.6 KiB
Markdown
---
|
||
id: CB-WP-0002
|
||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||
status: proposed
|
||
---
|
||
|
||
# Purpose
|
||
|
||
CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price
|
||
sheet, formula, staleness rule — and then could not report a number,
|
||
because nothing was ever instrumented. It was recorded as
|
||
"uncomputable".
|
||
|
||
That was wrong, and the retrospective for this workplan should say so
|
||
plainly: **the data existed the whole time.** Every Claude Code session
|
||
transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
|
||
per-message `usage`, including the cache breakdown. Reading it for
|
||
CB-WP-0001's session gives:
|
||
|
||
| Component | Tokens | Cost (Fable 5) |
|
||
|---|---|---|
|
||
| Output | 585,528 | $29.28 |
|
||
| Cache read | 131,863,164 | **$131.86** |
|
||
| Cache write (1h) | 4,365,668 | $87.31 |
|
||
| Input | 1,090 | $0.01 |
|
||
| **Total** | | **$248.46** |
|
||
|
||
The headline finding from that single reading is the reason this
|
||
workplan exists: **53% of the cost is cache reads**, not output. Cost in
|
||
an agentic loop is driven by context size × turn count, not by how much
|
||
the model writes. No D2 decision made on "tokens per task" would have
|
||
surfaced that.
|
||
|
||
This workplan makes cost a first-class measured dimension so that
|
||
future AM-12 equivalents support conclusions instead of decorating an
|
||
evidence file. It follows InnerLoop v1.0, including the rules that pass
|
||
earned: every metric names its instrument, and every harness carries a
|
||
positive control.
|
||
|
||
## Phase A — Instrument
|
||
|
||
## Task: Survey the available cost instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T01
|
||
status: todo
|
||
priority: high
|
||
```
|
||
|
||
Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop
|
||
survey template. Candidates at minimum: session transcript JSONL
|
||
(`usage` per assistant message), the Custodian State Hub token-event
|
||
API (`record_token_event`, `update_task_status` token tiers,
|
||
`get_token_summary`), the Claude Code status bar, and the Anthropic
|
||
usage/billing API. Per candidate: what it reports, granularity,
|
||
accuracy, whether it survives compaction, and whether it can attribute
|
||
cost to a unit of work.
|
||
|
||
Note explicitly which are **exact** and which are **estimates** — the
|
||
prior failure was tolerating an estimate-shaped metric. Name the
|
||
benchmark-to-beat per dimension; expect the transcript to lead on
|
||
accuracy and the hub to lead on durability.
|
||
|
||
## Task: Decide the instrument and the attribution model (ADR)
|
||
|
||
```task
|
||
id: CB-WP-0002-T02
|
||
status: todo
|
||
priority: high
|
||
```
|
||
|
||
Adversarial review of T01 first (InnerLoop §Step 2), committed as
|
||
`history/YYMMDD-cost-accounting-{challenge,response}.md`.
|
||
|
||
Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is
|
||
**attribution**: a transcript is a flat message stream, and a "task" is
|
||
a workplan concept. Options to weigh, not assume:
|
||
|
||
- git commit timestamps as task boundaries (the loop commits per
|
||
iteration, so boundaries already exist and are durable)
|
||
- explicit session markers emitted at task start/end
|
||
- hub task status transitions as the time index
|
||
|
||
State the expected advantage per dimension and the known failure modes
|
||
of the chosen model — in particular, what happens to attribution across
|
||
`/compact`, across resumed sessions, and for work spanning a boundary.
|
||
Gate: no collector code before this ADR is committed.
|
||
|
||
## Task: Specify the cost metrics with named instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T03
|
||
status: todo
|
||
priority: high
|
||
```
|
||
|
||
Write `specs/CostAccounting.md`: the cost model (input, output, cache
|
||
read, cache write at 5m and 1h, per the price sheet), the attribution
|
||
contract, and the acceptance metrics — each naming the **command that
|
||
produces its number**, per InnerLoop v1.0 §Step 4.
|
||
|
||
Must include a metric for the finding that motivated this workplan:
|
||
**cost composition** (what fraction is cache read vs write vs output),
|
||
not only a total. A single total would have hidden the 53%.
|
||
|
||
Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and
|
||
replace AM-12's definition with one that is computable.
|
||
|
||
## Phase B — Build and prove
|
||
|
||
## Task: Implement the cost collector
|
||
|
||
```task
|
||
id: CB-WP-0002-T04
|
||
status: todo
|
||
priority: medium
|
||
```
|
||
|
||
Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads
|
||
transcripts, applies the price sheet at
|
||
`benchmarks/baselines/model-prices.toml`, attributes cost per the T02
|
||
model, and emits both a per-task table and a composition breakdown in
|
||
the evidence-row format from T03.
|
||
|
||
**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool
|
||
asserts that attributed tokens sum to the transcript total, and refuses
|
||
to emit numbers when they do not reconcile. An unattributed remainder is
|
||
reported as its own line, never silently dropped — the failure this
|
||
whole workplan exists to prevent was a number that looked fine.
|
||
|
||
Handle multi-model sessions: a session that switches models must price
|
||
each message at its own model's rate.
|
||
|
||
## Task: Validate against CB-WP-0001 and answer a real question
|
||
|
||
```task
|
||
id: CB-WP-0002-T05
|
||
status: todo
|
||
priority: medium
|
||
```
|
||
|
||
Run the collector over the CB-WP-0001 session and commit
|
||
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
|
||
totals in this workplan's Purpose section is the acceptance test: the
|
||
tool must reproduce $248.46 (Fable 5) from the same transcript, or
|
||
explain the difference.
|
||
|
||
Then use it to answer at least one question that could not be answered
|
||
before, and record the answer. Candidates: which of T01–T09 cost most
|
||
and whether that matched its value; what a `/compact` costs; whether the
|
||
adversarial review paid for itself; how much the six T08 code iterations
|
||
cost relative to the research phase.
|
||
|
||
A tool that produces numbers nobody draws a conclusion from has not
|
||
cleared the bar that CB-WP-0001's AM-12 failed to clear.
|
||
|
||
## Task: Wire cost into the loop
|
||
|
||
```task
|
||
id: CB-WP-0002-T06
|
||
status: todo
|
||
priority: low
|
||
```
|
||
|
||
Make cost collection automatic rather than remembered: add the
|
||
collector to the evidence checklist in `specs/InnerLoop.md`, emit hub
|
||
token events so the Token Cost dashboard reflects real numbers rather
|
||
than the 1000/500 heuristic fallback, and add a `make cost` target to
|
||
the one command surface.
|
||
|
||
## Task: Retrospective
|
||
|
||
```task
|
||
id: CB-WP-0002-T07
|
||
status: todo
|
||
priority: low
|
||
```
|
||
|
||
Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what
|
||
this pass teaches about metric design. The specific question to answer:
|
||
CB-WP-0001 produced a fully specified metric that could not be computed,
|
||
and v1.0's "every metric names its instrument" rule was written to stop
|
||
that. Did it? If a metric can still be written without a working
|
||
instrument, the rule needs teeth — for example, requiring that the
|
||
instrument be demonstrated on real data before the metric is accepted.
|