Tier-L survey plus the full review trail. v1 was REJECTED by adversarial
review with four blocking findings; this is v2, and the finding below is
the reviewer's as much as mine.
The root cause is neither v1's story ("coverage scope is too narrow" —
true but shallow) nor the reviewer's cheaper one ("the link check points
at one file" — true but catches 3 of 7). It is:
Every coverage instrument in this project counts NAMES. None counts
ASSERTIONS. M-D1-COV counts covers: tags; M-D1-LNK counts rule-ID
strings. Both answer "is this rule mentioned?" and neither answers
"does anything fail if this rule is violated?"
Four of seven defects found this pass are named in the source and inert,
so no name-based check finds them. This subsumes the oldest error class:
all five prior harness-does-nothing instances have the same shape.
Two are mutation-proven, because the reviewer set that bar for findings
they had only read:
AM-7 — folding a 100k-event log from fresh(999) instead of fresh(42),
an unrelated genesis state, leaves the test green. The hash reaches
only a println!; the sole assertion is on elapsed time. And the log is
built across games seeded 42,43,44... so it could not be asserted as
written. evidence/CB-EV-0001 reports "AM-7 replay met, 2,290x" —
unearned on the hash-identical clause. HDN instance #6.
K9 — making Snapshot::take discard its EventSeq and store 0, which
destroys the half of K9 that says "+ the EventId it includes", leaves
the test green. On the acceptance criterion GameKernel §2.4 is zero of
three, not two of three.
Also found: K10, K14, K18 named nowhere in source; AM-10 vacuous (a K6
determinism lint reported under a D4 leak row, over a cb-*-api crate that
does not exist); AM-11's conformance suite does not exist; AM-1b measured
and omitted from the evidence scoreboard.
All twelve challenge findings conceded, none defended on its facts. v1's
prediction was rejected as unfalsifiable AND already determined; the
replacement names its refuting outcome and is beatable in both
directions.
Chaos d4=1, no override. Declaration 1 of the 12-declaration window.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The question was whether converting agent work to deterministic tooling
recovers capacity or the work reappears elsewhere. Neither, cleanly:
tooling recovers capacity where it removes the possibility of doing the
work by hand, and recovers nothing where it merely offers a better
option.
Sorting the five candidates by that test reproduces the results exactly.
env-setup and task-done left no manual path — env-test fails the build if
the friction returns, and task-done refuses to emit an estimate, so the
old way means knowingly writing fiction. Both met. facts-check and
`make status` left grep and heredocs one keystroke away. Both missed.
This is the principle ADR-0003 already used to reject explicit session
markers — nothing that depends on discipline gets to be the primary
index. It governs tooling economics too, and CB-RES-0003 did not apply
it.
Stated confidence sorted the extremes and said nothing in the middle,
where every judgement call lives: both high met, both medium missed, and
the single low also met, so it was miscalibrated at that end too. Future
reviews state the mechanism instead — after this fix, is the manual path
impossible, absurd, or merely inferior? — and only the first two may
carry a cost prediction.
The confound is symmetric and does not explain the split: building
repo.py and task-done.py also required inspection and patching, and those
categories still went to zero. It inflates the size of the miss, not its
location.
The most valuable output is not the 6 points of 38 recovered. It is that
the loop made a falsifiable quantitative forecast, was wrong by 4-5x, and
could say exactly which parts were wrong and why.
CB-WP-0004 status -> done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Discharges the open item T07 raised. Values unchanged at 250,000 and
350,000; what was missing was a reviewed decision behind them, since
they had been set by the implementer in the commit that measured them
and that also changed the feature gating being measured.
The ADR supplies the argument T07's test requires -- why the targets
bind on FUTURE work rather than merely passing present work:
AM-4a leaves 3,750 lines of headroom (1.5%), about one small crate,
so any new shipped-runtime dependency breaches it almost immediately.
That is intended: the shipped runtime should be effectively frozen.
AM-4b leaves 32,979 lines (10.4%), deliberately looser -- dev tooling
should absorb one moderate dependency without a spec change, not two.
Both are ceilings on a quantity that only grows by choice. Nothing
drifts across them; only adding a dependency does.
Falsification condition stated: if a later pass raises AM-4a to
accommodate a dependency it wants, that is the failure the ceiling
exists to catch, and the answer is an ADR arguing for the dependency.
First ADR written under the correction/retarget test; sets the shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The retrospective question was whether the mechanism set is complete or
each pass still finds a new class. This pass produced both an eighth
instance AND a fourth class, so the answer is the uncomfortable one.
Ledger: 10 instances, 4 classes, across 3 workplans. Every pass has
produced at least one class the previous pass had not seen.
HDN harness-does-nothing 5 executable assertions
TA trusted arithmetic 3 re-derivation
SSB same-sample blind spot 1 assertions over ALL the data
DFD duplicated-fact drift 2 NEW -- reading a copy against source
DFD is genuinely distinct: no positive control catches it, because both
copies are internally consistent, and re-derivation does not either,
because the copy faithfully reproduces what it was copied from. Found
when an inlined price sheet went stale within an hour of T11 changing
the real one.
So v1.2 stops trying to enumerate classes in advance. Every error in
three passes was corrected in-session for under ~1% of the pass, so the
stated design goal is now cheap CORRECTION: keep raw data so numbers are
re-derivable, keep artifacts small and committed so a wrong number is
one grep from everywhere quoting it, give every number a command.
Plus the one rule the new class earns: single source of fact.
The original hypothesis is revised rather than confirmed. "A rule that
cannot be executed is not a rule" is wrong -- the two most valuable
corrections in the project came from a decorative rule that cannot be
automated (re-derive inherited numbers). An executable rule fires
reliably and catches one class; a decorative one fires unreliably and
can catch any class, including unnamed ones. Keep both.
Gates this pass: loop-lint caught 3 real violations on first run, then
failed on its own author within the hour when a T07 edit pushed
InnerLoop.md to 407 lines against its own 400 limit.
CB-WP-0003 complete: 11 of 11 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blanket "no retargeting in the measuring commit" rule would have been
wrong. CB-WP-0002 moved AC-1 three times in exactly that shape and every
move was correct -- each time a new instrument disproved the old figure.
Four legitimate corrections would have been forbidden to catch one bad
retarget.
The test is mechanical rather than a statement of intent:
correction the target moves and the implementation does not; legal in
the same commit provided the instrument's output is there
retarget the same commit changes both the target and the code the
target measures; requires an ADR stating why the new target
binds on future work
Applied retroactively: AM-4a/AM-4b are UNRATIFIED. They were set after
seeing the measurement, in the commit that produced it, with the
implementation changing too -- a retarget by this test. make dep-weight
is currently enforcing a target no reviewed decision stands behind.
Recorded as an open item; ratifying or changing them is a maintainer
decision, not an implementer's.
Also: specs/InnerLoop.md split into InnerLoop.md (process) and
InnerLoopReference.md (rubric, template, rules, definition of done).
Not a stylistic choice -- `make loop-lint` failed on the commit that
pushed the file to 407 lines against its own ~400 limit. The gate added
this morning to make that rule executable caught its own author within
the hour, which is the cheapest possible demonstration that it works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
41 rules classified executable / checkable / decorative, each tagged with
the failure class it catches. Counts: 11 executable, 22 checkable, 4
decorative (one of them dead policy).
Audit: history/260731-inner-loop-rule-audit.md
New tools/loop-lint.py makes 7 rules executable (tier declared, chaos
roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file
loadability, reporting tools expose --self-test). It found three real
violations on its first run, none previously visible:
- specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit
the loop has stated since v0.2 and never measured. Split at its own
section boundaries into Blueprint (1-8) + Runtime (9-15).
- tools/dep-weight.py and tools/rule-coverage.py had positive-control
logic and no --self-test, so nothing verified the control worked.
Adding rule-coverage's self-test exposed a latent instance of the exact
class this workplan is about: if the spec regex stopped matching, rules
was empty, missing was empty, and the tool exited 0 reporting "0/0" --
a silent pass, in the tool that reports our headline AM-1 number. Both
tools now assert they found something before reporting.
Two demotions applied in the spec rather than left implicit: "structured
over prose" is marked guidance (nothing can check it), and the 8k/10k
token budget is struck through and marked DEAD POLICY pointing at T05.
The audit's uncomfortable finding: rule 13 (re-derive inherited numbers)
has no mechanical form, is deliberately left decorative, and caught the
LARGEST error in CB-WP-0002. That is a counter-example to this
workplan's own hypothesis. "A rule that cannot be executed is not a
rule" is wrong as stated; the defensible version is that such a rule
cannot be relied on to fire, so it must not be the only defence for a
class that matters.
Class coverage: harness-does-nothing has five executable rules;
trusted-arithmetic has ZERO and produced the largest single error.
make loop-lint and make self-tests wired into `make all` and CI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.
v1.1 adds:
- the instrument must exist and the target must come out of it;
targets are provisional until the tool emits them
- a number inherited from earlier work is re-derived before use as a
target, or cited as unverified
- every reporting tool exposes --self-test, run before the number
- cost is in the definition of done; M-D2-CST may not be uncomputable
The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.
CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.
Workplan status: done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.
The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:
- the dedup positive-control table failed its own checksum (group
sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
groups are 206, not 208) — the worst place for a slip, since that
table is the sole evidence for the load-bearing dedup assumption
- the multi-model table was labelled "deduped" while holding per-line
counts, three lines below the paragraph condemning per-line counts
- the $92.21 acceptance target excluded the subagent tree, which ran
INSIDE the pin window. A collector correctly implementing the
survey's own blind-spot finding would have failed the benchmark.
Target is now $92.87, stated as its two components.
Folded in, each re-derived before conceding:
- attribution must key on sessionId, not wall-clock: two sessions
overlap 4h13m on this repo carrying ~$12 (one of them was this
workplan's own background job)
- only 14 of 33 commits name a task; the other 19 hold 33% of cost
- cache writes bill per TTL — pricing the aggregate at 1h inflates
the subagent by 43%
- the "~87% cache" figure matched nothing; 64.6% of cost is cache
read, 88.4% is all cache
- fable is 35% of calls and 60% of dollars; opus is the inverse
Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.
Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.
Three errors found in this workplan's own Purpose section, all
overstating cost:
- one API response is written as up to six JSONL lines, each
repeating the same usage object (657 lines, 346 responses)
- a multi-model session was priced entirely at Fable 5 rates,
violating a per-model rule MetricsAndScenarios §1a already had
- the transcript is a live file, so an unpinned total is not
repeatable (346 -> 356 responses between two runs minutes apart)
CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.
Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.
Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop has now survived CB-WP-0001 end to end and is revised from what
that pass actually taught, recorded in
history/260731-inner-loop-retrospective.md.
Held up: the adversarial review (one blocking and three significant
findings on its single use, resolved with new measurements rather than
argument), the parity-cap rule, the provisional U-item mechanism, and
the ADR gate.
The gap: both serious errors in the pass were measurement errors, and
the loop caught neither. Review reads prose; these were claims about
numbers. In both cases a harness ran successfully while doing no work —
a JS harness timing rejected no-ops, and a Rust benchmark computing
throughput for rounds that never completed.
v1.0 adds:
- Measurement validity: a harness must assert it performed the work it
reports. A number from a run that cannot prove it did the work is void.
- Metric feasibility: every metric names its instrument and is checked
reachable against the contracts in its own spec.
- No silently-ignored input; decisions get commands, not defaults;
scaffolds are exercised or marked; coverage gates that count tags say
so.
- Evidence must state what a comparison does not support.
- The chaos roll is recorded even when it changes nothing, so a
mechanism that never fires is visible rather than assumed.
CB-WP-0001 is complete: 9/9 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
