T03: point review at the harness, and state what review cannot catch

The review step targeted the survey. Every serious error in this project
has been in measurement or build configuration, so the loop was
adversarially reviewing the artifact cheapest to fix and leaving
unreviewed the one where errors occur.

Step 2 now routes by risk: when the claim rests on numbers, the reviewer
gets the harness and the evidence file too, and must reproduce the
number independently rather than read about it.

The addition that matters more, because it was learned the hard way: a
reviewer re-derives the author's claims and therefore inherits the
author's SAMPLING. CB-WP-0002's dedup invariant was checked twice --
survey 206/206 groups, then the reviewer independently -- and both used
the main transcript. It is false in the 8-response subagent tree neither
looked at. Two independent verifications, one shared blind spot.

  Rule: the reviewer re-derives on a different sample than the author
  used, and where only one sample exists, says so rather than reporting
  a clean verify.

Also recorded: what review demonstrably DOES do. $0.66 and $1.11 across
two passes, ~1% of each, both finding approval-blocking defects. Cost is
not a reason to skip it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:26:01 +02:00
parent e21b9f4250
commit 4fd6322e17
2 changed files with 35 additions and 6 deletions

View file

@ -106,11 +106,40 @@ evidence verdict for those rows caps at `parity`, never `better`.
### Step 2 — Approve (adversarial review)
For tier-L passes, approval is earned through an **adversarial review**: a
separate session (or agent), given only the survey document, attempts to
break it — an omitted candidate, a stale or unverifiable benchmark, an
unmeasured claim presented as measured. Exactly **one round**: challenge,
then response. The survey is approvable only when every challenge is either
answered with evidence or conceded and folded into the survey.
separate session (or agent) attempts to break the work. Exactly **one
round**: challenge, then response. The work is approvable only when every
challenge is either answered with evidence or conceded and folded in.
**The review target follows the risk.** Reviewing the survey was the
original rule, and it is the wrong target for a capability whose claim
rests on numbers — every serious error in this project has been in
measurement or build configuration, not in prose.
| the claim rests on | the reviewer is given | and must attempt |
|---|---|---|
| a survey of candidates | the survey | an omitted candidate; a stale or unverifiable benchmark; an unmeasured claim presented as measured |
| **numbers** | the survey **and the harness and the evidence file** | **reproduce the number independently**; state what the harness would report if the work silently stopped |
**What review cannot do — state this, do not discover it.** A reviewer
re-derives the author's claims and therefore inherits the author's
sampling. So:
> **The reviewer re-derives on a different sample than the author used.**
> Where only one sample exists, the review says so rather than reporting a
> clean verify.
*(v1.1, from CB-WP-0002: the dedup invariant was verified on the main
transcript by the survey — 206/206 groups — and independently re-verified
by the reviewer, who used the same transcript. It is false in the
8-response `subagents/` tree neither examined. Two independent checks,
one blind spot, because both sampled the same way. Only an assertion
running over all the data at execution time caught it.)*
**What review demonstrably does do.** Measured across two passes:
**$0.66** and **$1.11**, roughly 1% of each pass, each finding
approval-blocking defects — in CB-WP-0002 a target that would have made
the evidence file certify a broken collector. The step pays for itself by
a wide margin and the cost is not a reason to skip it.
**Documentation requirement:** the research process, the challenge, and the
resulting improvements to the research are each documented in timestamped