T02 review round: survey approved after 15 of 16 findings conceded
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.
The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:
- the dedup positive-control table failed its own checksum (group
sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
groups are 206, not 208) — the worst place for a slip, since that
table is the sole evidence for the load-bearing dedup assumption
- the multi-model table was labelled "deduped" while holding per-line
counts, three lines below the paragraph condemning per-line counts
- the $92.21 acceptance target excluded the subagent tree, which ran
INSIDE the pin window. A collector correctly implementing the
survey's own blind-spot finding would have failed the benchmark.
Target is now $92.87, stated as its two components.
Folded in, each re-derived before conceding:
- attribution must key on sessionId, not wall-clock: two sessions
overlap 4h13m on this repo carrying ~$12 (one of them was this
workplan's own background job)
- only 14 of 33 commits name a task; the other 19 hold 33% of cost
- cache writes bill per TTL — pricing the aggregate at 1h inflates
the subagent by 43%
- the "~87% cache" figure matched nothing; 64.6% of cost is cache
read, 88.4% is all cache
- fable is 35% of calls and 60% of dollars; opus is the inverse
Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
bf57b6f3a4
commit
ac3ac2a8ce
4 changed files with 415 additions and 69 deletions
134
history/260731-cost-accounting-response.md
Normal file
134
history/260731-cost-accounting-response.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# 2026-07-31 — response to the challenge on CB-RES-0002
|
||||
|
||||
Answers to `260731-cost-accounting-challenge.md`. Every contested number was
|
||||
**re-derived here before being conceded** — the rule this survey argued for
|
||||
applies to the challenge too, not only to the survey.
|
||||
|
||||
Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.**
|
||||
Survey moves to `status: approved` after the folds below.
|
||||
|
||||
## Blocking — conceded
|
||||
|
||||
**#1 dedup table fails its own checksum.** Confirmed: the group-size
|
||||
distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657
|
||||
lines, and multi-line groups are **206**. The survey said 76 and 208. The
|
||||
reviewer is right that this is the worst possible place for an arithmetic
|
||||
slip: the table *is* the positive control for the load-bearing dedup
|
||||
assumption. Corrected in the survey.
|
||||
|
||||
Worth naming plainly: a document whose thesis is "re-derive, don't quote"
|
||||
shipped a hand-typed table that did not sum. The finding survives; the
|
||||
discipline did not extend to the evidence *for* the finding.
|
||||
|
||||
**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line
|
||||
counts under a header reading "API responses (deduped)". Deduped over the
|
||||
same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The
|
||||
reviewer's 206 for opus was the per-line figure quoted from the survey's own
|
||||
text; the correct deduped value is 213. Corrected, and the table now shows
|
||||
both columns so the 1.9× inflation is visible rather than asserted.
|
||||
|
||||
**#3 the acceptance target excludes the subagent.** Confirmed and the most
|
||||
consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`,
|
||||
which is **inside** the pin window. Re-derived:
|
||||
|
||||
```text
|
||||
main transcript $92.2075 (339 responses)
|
||||
subagent $ 0.6580 (7 responses)
|
||||
TRUE TOTAL $92.8655
|
||||
```
|
||||
|
||||
The survey stated the C1 blind-spot correctly and then set a benchmark that
|
||||
only a collector *exhibiting* that blind-spot could hit. A correct collector
|
||||
would have failed the acceptance test. The target is now **$92.87**, stated
|
||||
as the sum of both components so a collector reading one tree can be
|
||||
diagnosed rather than merely failed.
|
||||
|
||||
## Material — conceded
|
||||
|
||||
**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's
|
||||
cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The
|
||||
collector's arithmetic was already correct — it prices the `ephemeral_5m` and
|
||||
`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing
|
||||
is the *rule*: the survey named both fields without stating that the
|
||||
top-level `cache_creation_input_tokens` aggregate must never be priced at a
|
||||
single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43%
|
||||
error. Added as an explicit contract for T03 and a risk-list entry, since
|
||||
fan-out passes are where 5m writes dominate.
|
||||
|
||||
**#5 concurrent sessions are already in the data.** Conceded, with an
|
||||
observation the reviewer could not have had: session `f1eb1147` is *this
|
||||
review's own session* — the background job running the workplan. The survey
|
||||
claimed the interleaving risk was "not exercised here"; it was being
|
||||
exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be
|
||||
separated by a wall-clock join. This kills naive wall-clock attribution
|
||||
outright and is now a stated ADR constraint: **attribution must key on
|
||||
`sessionId`, not only time.** The survey's C1 section also now names the
|
||||
second transcript.
|
||||
|
||||
**#6 a third of the money has no task tag.** Re-derived independently with
|
||||
`(prev commit, this commit]` intervals over the 33 pinned commits:
|
||||
**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either
|
||||
end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs
|
||||
matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same
|
||||
conclusion, and the survey now carries the re-derived 33%.) C5's claim that
|
||||
the loop "commits per task iteration with the task in the subject" holds for
|
||||
14 of 33 commits. This is now sized in the survey rather than left as a
|
||||
qualitative hazard, and the unattributed-remainder line in T04 has a known
|
||||
target to beat.
|
||||
|
||||
**#7 the ~87% figure matches nothing.** Conceded — it was a stale
|
||||
hand-carried number from before the pin. The defensible figures are **64.6%**
|
||||
(cache read as a share of cost) and **88.4%** (all cache as a share of cost).
|
||||
C2's argument is unaffected: the hub schema can express neither. Corrected to
|
||||
88.4% with the basis named.
|
||||
|
||||
## Minor — conceded
|
||||
|
||||
- **#8** `<synthetic>` has a valid `requestId` and an all-zero `usage`, not a
|
||||
missing one. Dollar impact is exactly $0.00, but the two guard styles
|
||||
(`if not usage` vs `if model not in prices`) branch differently, so T04's
|
||||
contract now names which it uses.
|
||||
- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned
|
||||
assistant lines carry `usage`. Conclusion stood, count was wrong. Removed.
|
||||
- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1).
|
||||
Guidance changed from "off by the offset" to "parse `%cI` and convert".
|
||||
- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8**
|
||||
minutes, not "3–15". The max is the number that matters.
|
||||
- **#12** interval convention now stated explicitly: *ending-at-commit*,
|
||||
`(prev, this]`, which is the only one consistent with commit-at-end-of-task.
|
||||
- **#13** the sonnet intro rate is a TOML comment, so a collector silently
|
||||
ignores it ($0.17 at the pin). Raised as a price-sheet schema defect —
|
||||
time-boxed rates need a representation. Same class of defect the survey
|
||||
levels at C2, as the reviewer notes.
|
||||
- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable
|
||||
value. Folded into D2 with a number attached.
|
||||
- **#16** accepted as the better framing, and promoted out of "minor":
|
||||
**fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of
|
||||
calls.** In a document arguing that totals mislead, a count-majority
|
||||
presented where a cost-majority was meant is the same error one level down.
|
||||
|
||||
## Partially answered
|
||||
|
||||
**#15 (reviewer found no fault) — noted, with one correction accepted.**
|
||||
The reviewer is right that C3/C4 elimination is categorical rather than
|
||||
performance-based and therefore legitimate. Accepted. The one thing folded
|
||||
in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and
|
||||
is now marked as such rather than reading like a measurement.
|
||||
|
||||
## What this round cost, and whether it paid
|
||||
|
||||
Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet,
|
||||
~$0.60 — the same order as CB-WP-0001's review at $0.66.
|
||||
|
||||
It found three approval-blocking defects, one of which (#3) would have made
|
||||
T05's acceptance test reward a *broken* collector and fail a correct one.
|
||||
That is the second consecutive pass where a ~$0.60 review caught something a
|
||||
$92 pass had missed. The economics are not close, and this belongs in the
|
||||
CB-WP-0003 T03 argument for pointing review at measurement rather than prose.
|
||||
|
||||
The uncomfortable pattern across both rounds: **the survey's arithmetic over
|
||||
raw data was flawless — every one of the reviewer's ~20 spot-checks
|
||||
reproduced to the cent — and its hand-written prose tables were not.** The
|
||||
errors were all in numbers typed by hand into markdown after the computation,
|
||||
never in the computation. That is a mechanical, fixable class: tables that a
|
||||
tool can emit should be emitted by the tool. Raised for T03 and T06.
|
||||
Loading…
Add table
Add a link
Reference in a new issue