T03: point review at the harness, and state what review cannot catch
The review step targeted the survey. Every serious error in this project has been in measurement or build configuration, so the loop was adversarially reviewing the artifact cheapest to fix and leaving unreviewed the one where errors occur. Step 2 now routes by risk: when the claim rests on numbers, the reviewer gets the harness and the evidence file too, and must reproduce the number independently rather than read about it. The addition that matters more, because it was learned the hard way: a reviewer re-derives the author's claims and therefore inherits the author's SAMPLING. CB-WP-0002's dedup invariant was checked twice -- survey 206/206 groups, then the reviewer independently -- and both used the main transcript. It is false in the 8-response subagent tree neither looked at. Two independent verifications, one shared blind spot. Rule: the reviewer re-derives on a different sample than the author used, and where only one sample exists, says so rather than reporting a clean verify. Also recorded: what review demonstrably DOES do. $0.66 and $1.11 across two passes, ~1% of each, both finding approval-blocking defects. Cost is not a reason to skip it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e21b9f4250
commit
4fd6322e17
2 changed files with 35 additions and 6 deletions
|
|
@ -106,11 +106,40 @@ evidence verdict for those rows caps at `parity`, never `better`.
|
|||
### Step 2 — Approve (adversarial review)
|
||||
|
||||
For tier-L passes, approval is earned through an **adversarial review**: a
|
||||
separate session (or agent), given only the survey document, attempts to
|
||||
break it — an omitted candidate, a stale or unverifiable benchmark, an
|
||||
unmeasured claim presented as measured. Exactly **one round**: challenge,
|
||||
then response. The survey is approvable only when every challenge is either
|
||||
answered with evidence or conceded and folded into the survey.
|
||||
separate session (or agent) attempts to break the work. Exactly **one
|
||||
round**: challenge, then response. The work is approvable only when every
|
||||
challenge is either answered with evidence or conceded and folded in.
|
||||
|
||||
**The review target follows the risk.** Reviewing the survey was the
|
||||
original rule, and it is the wrong target for a capability whose claim
|
||||
rests on numbers — every serious error in this project has been in
|
||||
measurement or build configuration, not in prose.
|
||||
|
||||
| the claim rests on | the reviewer is given | and must attempt |
|
||||
|---|---|---|
|
||||
| a survey of candidates | the survey | an omitted candidate; a stale or unverifiable benchmark; an unmeasured claim presented as measured |
|
||||
| **numbers** | the survey **and the harness and the evidence file** | **reproduce the number independently**; state what the harness would report if the work silently stopped |
|
||||
|
||||
**What review cannot do — state this, do not discover it.** A reviewer
|
||||
re-derives the author's claims and therefore inherits the author's
|
||||
sampling. So:
|
||||
|
||||
> **The reviewer re-derives on a different sample than the author used.**
|
||||
> Where only one sample exists, the review says so rather than reporting a
|
||||
> clean verify.
|
||||
|
||||
*(v1.1, from CB-WP-0002: the dedup invariant was verified on the main
|
||||
transcript by the survey — 206/206 groups — and independently re-verified
|
||||
by the reviewer, who used the same transcript. It is false in the
|
||||
8-response `subagents/` tree neither examined. Two independent checks,
|
||||
one blind spot, because both sampled the same way. Only an assertion
|
||||
running over all the data at execution time caught it.)*
|
||||
|
||||
**What review demonstrably does do.** Measured across two passes:
|
||||
**$0.66** and **$1.11**, roughly 1% of each pass, each finding
|
||||
approval-blocking defects — in CB-WP-0002 a target that would have made
|
||||
the evidence file certify a broken collector. The step pays for itself by
|
||||
a wide margin and the cost is not a reason to skip it.
|
||||
|
||||
**Documentation requirement:** the research process, the challenge, and the
|
||||
resulting improvements to the research are each documented in timestamped
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue