tegwick
53c460c992
Duplicated-fact drift is the fourth error class and the only one with no executable rule. No positive control catches it (both copies are internally consistent) and re-derivation does not either (the copy reproduces whatever it was copied from). It is caught only by reading a copy against its source, which nothing in the loop required. facts.toml holds 15 facts and is GENERATED by `make facts-gen` from cb-cost, dep-weight and rule-coverage. The trap this task named — a hand-maintained registry that becomes another drifting copy — is closed by facts-check re-running the instruments and failing when the committed registry disagrees with them. A stale registry cannot certify stale artifacts. An artifact quoting a fact tags it: **$93.15** <!-- fact --> with the key. 17 occurrences across 5 artifacts are now checked. Falsified before being believed: changing CostAccounting.md line 158 from $93.15 to $92.87 — the exact historical drift — produced exit 1 naming the file, the line and the expected value. Tested against the class it exists to catch, on a real artifact, not only in its self-test. It then caught a live tag inside its own documentation example in InnerLoop.md within the hour. Third time a gate has failed on its own pass's work. What it does not close is stated rather than implied: 22 untagged literal copies remain and are reported, not failed. Tagging is opt-in, a number can legitimately recur, and a gate that fires on coincidence gets routed around. Naming the uncovered surface beats claiming the class is closed. InnerLoop single-source-of-fact moves from prose to executable — v1.3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
153 lines
7.2 KiB
Markdown
153 lines
7.2 KiB
Markdown
# CB-EV-0002: cost accounting
|
||
|
||
capability: meta.loop.cost-accounting
|
||
spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0
|
||
decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md)
|
||
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md)
|
||
instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`)
|
||
measured: 2026-07-31
|
||
|
||
Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445`
|
||
(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not
|
||
transcribed by hand (CA-15).
|
||
|
||
---
|
||
|
||
## 1. Acceptance table
|
||
|
||
| ID | Metric | Target | Measured | Verdict |
|
||
|---|---|---|---|---|
|
||
| AC-1 | pinned total, as two components | $93.15 = $92.03 + $1.11 | **$92.03 main + $1.11 subagent = $93.15** | **met** | <!-- fact:pinned_total --><!-- fact:pinned_main --><!-- fact:pinned_subagent -->
|
||
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
|
||
| AC-3 | unattributed share reported | present, 33% | **32.4%, own line** | **met** |
|
||
| AC-4 | composition reported | 5 components | **5 of 5** | **met** |
|
||
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
|
||
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
|
||
| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** |
|
||
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
|
||
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
|
||
|
||
No `unmeasured` rows. AC-1's target was corrected four times before and after this
|
||
run; §4 records why, because the sequence is more useful than the final
|
||
number.
|
||
|
||
## 2. Composition — the finding the metric exists for
|
||
|
||
```text
|
||
input 690 tok $ 0.00 0.0%
|
||
output 323,643 tok $ 11.13 11.9%
|
||
cache_read 80,611,798 tok $ 59.67 64.1%
|
||
write_5m 37,467 tok $ 0.47 0.5%
|
||
write_1h 1,672,854 tok $ 21.87 23.5%
|
||
TOTAL $ 93.15
|
||
```
|
||
|
||
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
|
||
to text written is 249:1. A single total would have shown $93.15 and
|
||
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
|
||
have done, and why that metric is now demoted.
|
||
|
||
## 3. Per-task attribution
|
||
|
||
```text
|
||
UNATTRIBUTED $ 30.14 32.4%
|
||
T08 $ 21.02 22.6%
|
||
T07 $ 12.01 12.9%
|
||
T03 $ 9.91 10.6%
|
||
T04 $ 8.00 8.6%
|
||
T05 $ 7.19 7.7%
|
||
T06 $ 3.58 3.8%
|
||
T09 $ 1.29 1.4%
|
||
```
|
||
|
||
**Stated limit (CA-10):** 32.4% of cost sits in commits whose subject
|
||
carries no `T##` tag, so this table is a view over 67.5% of spend. That is
|
||
a property of commit hygiene, not of the collector.
|
||
|
||
T08 (the GROUND aggregate, six code iterations) at $21.02 is the most
|
||
expensive task and was also the one that produced the most rework — the
|
||
tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong
|
||
benchmark all originated there. Expensive *and* error-dense: the correlation
|
||
is worth watching, not yet a conclusion from n=1.
|
||
|
||
## 4. What it does not support
|
||
|
||
- **The per-task figures are not comparable across passes.** They mix
|
||
models (opus/fable/sonnet) at different price points and different cache
|
||
states. The dollar figure is comparable; a token count is not.
|
||
- **AC-3's 32.4% is a fixture pin, not a quality target.** Improving commit
|
||
tagging will move it, and that is the desired direction.
|
||
- **This is one session.** Every ratio here (cache share, $/turn, the
|
||
compaction effect in §5) is n=1 and should be treated as a hypothesis
|
||
until a second pass reproduces it.
|
||
- ~~**The price sheet cannot express a time-boxed rate.**~~ **Fixed
|
||
2026-07-31 by CB-WP-0003 T11** (CA-16/CA-17). Applying the sonnet
|
||
promotional rate moved AC-1 from $93.32 to **$93.15** — the $0.17 this
|
||
section predicted, now collected rather than merely noted.
|
||
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
|
||
billing API could not independently confirm the total. The transcript
|
||
counters are the same ones billing uses, but that is an argument, not a
|
||
reconciliation.
|
||
|
||
## 5. The question that could not be answered before
|
||
|
||
**"What does `/compact` actually cost, and is a long session quadratic?"**
|
||
|
||
CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context
|
||
grows with turns, making long sessions quadratic. The data **qualifies that
|
||
claim**: it is true only between compactions.
|
||
|
||
The session compacted twice, and the transcript records both directly:
|
||
|
||
| compaction | pre-tokens | post-tokens | reduction |
|
||
|---|---|---|---|
|
||
| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** |
|
||
| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** |
|
||
|
||
Cost per turn across the boundary:
|
||
|
||
| segment | turns | mean context | total | $/turn |
|
||
|---|---|---|---|---|
|
||
| start → C1 | 136 | 304,178 | $62.19 | **$0.457** |
|
||
| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** |
|
||
|
||
**The 202 turns after the first compaction cost less than half of the 136
|
||
turns before it — a 3.1× drop in cost per turn.** Context growth *is* the
|
||
cost driver, and compaction is the control on it. At the pre-compact rate,
|
||
those 202 turns would have cost ~$92 instead of $30.
|
||
|
||
Two consequences for CB-WP-0003:
|
||
|
||
1. **T04's "one task per session" recommendation is not the only remedy,
|
||
and may not be the cheapest one.** Compaction achieved a 27× context
|
||
reduction inside a running session at the cost of one summarization
|
||
call. A fresh session pays a cold-start re-read of the committed
|
||
artifacts; compaction pays a summary. Which is cheaper is now a
|
||
measurable question rather than a matter of taste, and T04 should
|
||
measure it before prescribing.
|
||
2. **The quadratic claim should be restated as bounded-quadratic:** cost
|
||
grows with context between compactions and resets at each one. The
|
||
failure mode is not "a long session" but "a long *uncompacted* session".
|
||
|
||
## 6. Retrospective note
|
||
|
||
The positive control paid for itself on its first execution, which is the
|
||
strongest evidence this project has produced for the InnerLoop v1.0 rule
|
||
that added it. CA-02 asserted that `usage` is identical across the lines of
|
||
one `requestId` — verified twice on the main transcript, by the survey
|
||
(206/206 groups) and independently by the adversarial reviewer. It is false
|
||
in the `subagents/` tree, where `output_tokens` is a running count
|
||
(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to
|
||
print a number. Under the prior first-wins rule it would have printed a
|
||
plausible one.
|
||
|
||
The generalization that failed is worth naming: **a property verified on the
|
||
largest sample was assumed to hold on the smallest one.** The main
|
||
transcript is 338 of 346 responses, so 206/206 felt conclusive; the
|
||
violation lives entirely in the 8 responses nobody checked separately.
|
||
|
||
Cost of the adversarial review this pass: **$1.11**, against a $93.15 pass.
|
||
It found three approval-blocking defects, one of which (the subagent
|
||
exclusion) would have made this evidence file certify a broken collector.
|
||
Second consecutive pass where a ~1% spend on review changed the outcome.
|
||
That is now two data points for CB-WP-0003 T03.
|