premise never left the repo
Separate agent, second tier-L review in this project. Six of seven
challenges conceded. The survey's headline finding is WITHDRAWN, not
softened.
C4 kills it, and the reviewer ranked it fourth. A FirstLegal policy --
take legal[0], no heuristic at all -- scores 0% at five and six seats
where GreedyPolicy scores 100%, and 77.5% at two seats where greedy scores
66%. Two unsophisticated agents span the entire range at the same seat
count. "The game is too easy at 5-6 seats" is therefore a statement about
GreedyPolicy, not about GROUND. The rescue the reviewer offered -- greedy
hits the 12-point ceiling in 200/200 deals, so the 6p row is a rules claim
-- dies on the same data: FirstLegal reaches that ceiling never.
C1: the per-node cost was wrong by 30-50x. The timer started before the
seed loop, so "us/node" included two setups, an entire greedy game and a
full validate+fold replay, divided by player-decision count. The tell was
in my own published output and I did not look at it: the figure FELL
(161/139/112) as branching ROSE (4.7/7.4/9.1), which no per-enumeration
cost can do. Re-measured with the clock around legal_commands alone:
3.0/3.5/4.1 us, now rising with branching. The reviewer measured 15.6-20.4
by a different isolation; we disagree by ~5x and neither has established
which is right, so T04 must benchmark it with criterion rather than adopt
either number.
C6: "exhaustive search is out at any seat count" is false -- ~3 seconds
over the last two rounds at 3p. With C1's correction the budget is
~10^5-10^6 nodes and bounded endgame search fits, so ADR-0013 cannot open
with "exhaustive is impossible, therefore determinized sampling" --
especially as sampling carries strategy fusion that exhaustive search does
not.
C3: the finding failed the admissibility rule this project wrote nine
hours earlier. 6/9/12 are sums where GROUND-WP-0004 T02 requires
per-priority rows, and the harness has no assertions, no --self-test and
no make target, so nothing can turn it red -- a `default` artifact wearing
a `counterexample` label, by CB-WP-0022 T05's own distinction.
C2: the ratio story explains nothing; 3p and 4p share deal, threshold and
ratio and differ by 12.5 points of win rate. C5: "explains the
maintainer's report" is contradicted by lib.rs:2487, which records his
losses as 3-player games on the pre-ruling deal, arithmetically unwinnable
at 6 against 7.
T06 exists to report to GROUND-WP-0005, which is BLOCKED waiting on a
difficulty baseline. Had this proceeded they would have been invited to
move thresholds on the strength of one bot's behaviour. That is the fifth
wrong premise this project would have sent them, and the second stopped by
an adversarial review rather than by a control. Both tier-L reviews here
have now caught a false headline that every gate passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First adversarial review in this repo run by a genuinely separate agent.
CB-RES-0006's reviewer opened by conceding it could not be, and called its
own findings "a lower bound on what a genuinely separate reviewer would
find." This is the measurement: the separate reviewer ran git log against
the survey's central example and found 2da19a4 had falsified it four days
earlier, while the author -- who wrote that commit -- quoted the dead
number twice.
Seven challenges: four conceded, two conceded in part, one answered.
C1 changes the design. "GR-E01 is admissible because 4/6/9 against 5/7/9
is a computation anyone can rerun" was a computation already rerun: the
edition import measured 6/9/12, the conclusion inverted, and the scenario
was renamed -unreachable- to -reachable-. That finding was one of the TWO
that passed the reproduction rule. So three wrong premises have now
reached ground-game and the third satisfied an existence test -- existence
is not the property that was missing. The rule gains shape (ground-game's
row-level deal table, promoted from a T04 addendum) and a clause the
survey never contemplated: a reproduction must be able to fail. Ours went
green and stayed admissible.
C1 also caught a defect in flight. T06's payload, status todo, still named
4/6/9 and was queued to send it to ground-game as "no dataset reconciles
them." Withdrawn before sending -- the fourth wrong premise, and the only
one stopped.
C2 withdraws the baseline's precision: design-baseline.py is a
hand-maintained dict counting itself, has_reproduction never checks the
file exists (its YES-control is green against a deleted path), and
Makefile:127 runs only --self-test so the reporting path has no CI. The
direction survives; 33% is not a measured rate and T05 must not build on
it. C3: "six provisional defaults" is five, GR-E01 double-counted. C4:
GroundRules §Underdetermined was never evaluated as a candidate and
already delivers four of five benchmarks -- T03's burden flips to arguing
extension over replacement.
Survived: the rule's affordability, and reuse of the provisional
machinery.
loop-lint: no findings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four of six challenges conceded; the survey's two main arguments are
replaced rather than clarified.
C1 The survey concluded the sub-100k region was empty without ever
measuring it. It is not empty: tiny-skia fits at 83,956, and the
real windowed floor is fltk at 140,079 — 1.5x corrected headroom,
not 128x. "Two orders of magnitude" is withdrawn.
C1b Following that concession: wgpu+winit is 1,741,979 marginal lines
against a 250,000 target. AM-4a is incompatible with INTENT stage 2
by 7x, and has been since both were written. Raised for the
maintainer; deliberately not decided by this pass.
C2 "Marginal cost zero" was scored on an axis chosen to produce zero,
and the same relocation already flattered sdl2 and fltk in the
survey's own table. One acquisition rule now covers all three, and
it raises two candidates' cost rather than lowering the winner's.
C3 The survey contradicted itself on the second-use rule. cb-render-api
and cb-render-null are withdrawn; cb-render-html ships against the
existing Project trait, and the port waits for stage 2's second use.
C6 The candidate measurements did carry a positive control. The C1
batch did not — it copied dep-weight.py's measurement function
without its guards — and was re-measured under them before being
cited. Figures reproduced unchanged. The HTML row is relabelled
0 by construction, not by measurement.
C4/C5 become six controls binding on ADR-0007.
The recommendation stands. The reasoning that reached it does not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GR-O01 states 2-6 players; every scenario in the corpus was 3-player.
Now all five counts play to GameEnded under both policies and reproduce
at the same seed, with scenarios at both boundaries and the CLI
transcript run at 2p, 3p and 6p.
Nothing broke — the rules are seat-count-generic. What the boundaries
exposed is arithmetic: with the standard preset's placeholder Problem
values (value = priority), the best total any game can reach is 3 at 2p,
6 at 3-4p, 10 at 5-6p, against GR-E01 thresholds of 5, 7 and 9. Group
success is unreachable below five seats regardless of play, and no
scenario noticed because none had played to scoring with everything
claimed.
GR-S01 calls the fixture a stand-in for scenario Problem data, so this
is evidence the stand-in is not neutral, not that GR-E01 is wrong. It is
pinned by a passing scenario, an arithmetic test, and a provisional
marker owned by ground-game so it ages in `make coverage`. The test
states its own delete-by: it is expected to fail when Problem values
become real data, and that failure is the signal to delete it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A human seat is a Policy like any bot, so the CLI adds no second driver:
HumanPolicy renders the projection, lists the legal commands and reads an
index or `pass`. `make play` runs it; `--all-bots` watches one.
K13's Project trait gains its first implementor after six passes with
none. Hidden: other seats' face-down selections until Reveal, hands and
deck (counts only), a face-down Problem's suit and value, and the seed —
not secret content, but a seat holding it can compute the deck.
A played session becomes an artifact: --record writes it as a scenario
the runner executes, --replay writes a .cbreplay bundle. record.rs is the
inverse of parse_command and its warrant is a round-trip test over every
command shape.
The acceptance test for the projection passed vacuously twice. First it
asserted the text contained "face-down", which every render does because
of Problems. Counted, it then reported zero inspected entries: seats are
asked in order, so a human at P1 is prompted before anyone has selected.
Seated at P3 it inspects ten entries and dies when the projection is
mutated to reveal everything. Counting what the harness examined caught
both, which is the second time that remedy has worked where a stronger
predicate would not have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Policy trait, a seeded random policy and a greedy one with a stated
heuristic, a legal-command generator that filters candidates through
validate, and a driver that runs a 3-player all-bot game to GameEnded.
Same-seed runs are hash-identical (K8), and a different policy seed
produces a different game — without that second assertion the first is
satisfied by a bot that ignores its RNG.
Every failure is loud, because the one a bot driver must not have is the
silent one: no legal move, passing where an action is required, an
out-of-range index (not clamped), a rejected command, and a stall guard.
What the second consumer found, none of it fixed here:
- GR-A13 admits SOLVE against an already-claimed Problem and resolution
then does nothing — the action is silently wasted. The policy avoids
it; the rule is left for a ruling.
- The 3-player standard fixture cannot reach GR-E01's threshold of 7:
three Problems valued 1,2,3 cap the total at 6. No scenario noticed
because none plays to scoring.
- K13's Project trait still has zero implementors. T02 is its first.
Mutation-checked by hand. The first mutation was a no-op and survived;
removing the Resolve call outright turned three tests red for the stated
reason. Third instance of the weak-mutation class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The question was whether M-D1-MUT is a real instrument or a name-counter
with extra steps, given that writing a weak mutation is as easy as writing
a strong one.
It is real, but only because it was hardened three times in one pass. Five
controls now stand between a mutation and a red verdict — the mutation
must apply, the baseline must be green, the tree must be restored and
verified, the failure must match a stated reason, and that stated reason
must be absent from passing output — and every one of them exists because
its failure actually occurred. The last is the sharpest: the FA guard
needed a guard, because my first AM-2 expect was "AM-2", which the passing
report contains.
Generalizable: an instrument that measures whether other instruments work
needs more controls than the instruments it measures. M-D1-MUT carries
five; dep-weight and rule-coverage carry one each. That asymmetry is the
cost of a meta-instrument, and a project adding one should budget for it.
A worse failure mode than CB-WP-0005 predicted: a mutation can become weak
without anyone touching it. AM-6's went SURVIVED when T04 moved its gate
from debug to release — nothing about the row, the mutation or the code
changed, only the headroom. Mutation strength is coupled to measurement
conditions, so a mutation is not a write-once artifact. CB-WP-0005 T08's
stronger remedy — mutations written by someone other than the author — was
NOT tested and should not be assumed unnecessary: EXPECT-VACUOUS covers
the cheap failure, not the expensive one AM-6 demonstrated.
The "removes the manual path" test is settled as a predictor of cost, not
of worth. mutation-check fails it outright and produced six defects
nothing else would have found.
Prediction error collapsed: 4-5x, then 2.5x, now small — because this pass
predicted per task, as a mechanism, with the alternative named. Both
branches are outcomes someone must defend, so the prediction cannot be
dodged. AM-3 and AM-4c took the second branch and are better resolved for
it than if a number had been forced.
No InnerLoop change. v1.4's mutation requirement is one pass old and
changing it before a second use would be the invention-in-isolation INTENT
warns about — the same argument used to amend K14 four hours earlier.
Named next candidate: specs/SessionShape.md. SS-01..SS-05 have been stated
since CB-WP-0003 and none has ever been enforced. This pass ran at 2.5x
the context ceiling its own spec sets and nothing said a word.
CB-WP-0006 status -> done, 9/9.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two rules, two different answers, which is the point of a task phrased
"implement, or amend and say why".
K18 is implemented. "Criterion benches driving the same scenario format at
scale" was false — the bench hardcoded its commands and never touched
ScenarioFile, while MetricsAndScenarios §3 pointed at a benchmarks/
directory containing only baselines/. benchmarks/synthetic-3p.yaml now
holds the workload and both the bench and bench_shape read it: the
workload is data, not code.
A second defect surfaced while fixing the first. After the bench switched
to the file, bench_shape still hardcoded the same sequence, so the
workload existed twice — deleting end_round from the YAML broke bench-test
while bench_shape kept passing. Duplicated-fact drift in executable form.
Both now read the same include_str! and deleting a command breaks both.
Explicitly not claimed: this does not unblock AM-3. AM-3's baseline is a
declarative game object — moves, turn order, rules. synthetic-3p.yaml is a
command list; the rules live in games/ground. Marking it as AM-3's subject
would compare a script to a game definition, which is the category error
AM-3 is blocked on. The file says so in its own header, where the next
person will be tempted.
K14 is amended. CommitWindow had zero non-test users and GROUND enforces
the same contract inline. Wiring GROUND through it was rejected: it would
change the serialized shape of `selections`, which four scenario files
assert by dot-path and every state hash depends on, for the sole benefit
of making a sentence literally true.
The deciding argument is INTENT's, not convenience: abstractions are
extracted from working games rather than invented in isolation, and no
concept becomes canonical until it survives a second concrete use.
CommitWindow was invented before any game needed it and has survived none.
Imposing it on GROUND would manufacture the first use rather than discover
it. So K14 states what is actually guaranteed, CommitWindow is marked
provisional in the source, and it carries a delete-by date of 2026-12-31.
Kernel spec->code link 16/18 -> 18/18, stated with the caveat the gate
prints every run: that is about names, not assertions.
Two self-tests broke and both broke correctly. rule-coverage's gate test
hardcoded "unlinked rules exist today" and failed when the last one was
linked; it now computes that and asserts the gate fails iff rules are
unlinked. facts' text check rejected k_unlinked once it became
legitimately empty; empty now renders as "(none)" and the check
distinguishes absent from empty.
M-D1-MUT: 8 of 14, unchanged — K14 and K18 are kernel rules, not
acceptance rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
INTENT design decision 8 of 10, unimplemented for six passes. cb-sim had
no flag parsing at all, so --replay had nowhere to go.
The bundle is manifest + commands.log + initial.snapshot + expected.yaml,
dev-only behind the scenarios feature and charged to AM-4b. The command
stream goes through the K11 framing built in T05, so a truncated bundle is
detected rather than replayed short — the two tasks compose rather than
duplicating.
The reviewer's D2 correction was real: this was not "a directory of four
files". Pass carried only the end state, RunOutcome::Failed was a
formatted String, and scenario.rs created an EventLog, appended to it and
never read it. All three had to change.
The first round trip failed to reproduce, and the cause is worth keeping:
state_hash_hex over a serde_json::Value is a different canonical form than
over the typed aggregate — Value's map is key-sorted, a struct serializes
in declaration order. The bundle was written with one basis and verified
with the other. A round trip written to recompute its own comparison value
would have PASSED this bug; it failed because the recorded hash came from
the producing process, which is control 2's entire purpose.
make replay-test implements ADR-0005 §6's four controls, 14/14: a
committed deliberately-failing fixture outside the corpus with covers: []
so it neither fails `make sim` nor inflates AM-1; a tampered recorded hash
must fail; a log short by one byte and a corrupted length prefix must be
rejected; and a mutated manifest seed must fail — which bites only because
replay re-derives the initial state from seed+setup and checks it against
the recorded snapshot, since restoring from the snapshot alone would leave
the seed inert. Plus a control on the controls: the bundle must still
replay after every mutation is reverted.
AM-7's hash-identical clause is re-earned. The probe records a hash per
per-game segment and replays each from its own genesis; folding from the
wrong seed now fails. That is the clause ADR-0005 §4 withdrew as
mutation-proven inert. The scaling >= 0.9x clause is still unenforced, so
AM-7 stays PARTIAL — reported, not rounded up.
Kernel coverage 15/18 -> 16/18. facts-check immediately caught the spec's
copy of that number going stale, on a number that moved the same hour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K11 is implemented: crates/cb-events/src/store.rs, magic + version header,
4-byte little-endian length prefix, append-only. Reimplemented not
assimilated per ADR-0005 §2 — no new dependency, and AM-4a/AM-4b are
unchanged at 246,250 / 317,021 because nothing entered the graph.
The operative clause is "detected", so corruption is tested rather than
assumed: a tail short by one byte, a half-written length prefix, a length
prefix corrupted to claim more than the file holds, foreign magic, and a
future format version are each rejected with a distinct error. A reader
that accepts a truncated tail is worse than no format, because it silently
returns a short history that looks complete.
AM-11 is earned. LogStore has two impls — MemLogStore and FileLogStore —
driven through ONE conformance(). The trait carries raw/set_raw precisely
so the corruption controls live in the shared suite: a format contract
that only one impl enforces is not a contract. The same shape is
retro-fitted to KernelRng, which is what AM-11 actually names: ChaChaRng
and NullRng now pass one suite asserting bounds, draw(1) == 0, determinism
across fresh instances, and shuffle preserving the multiset. They were
previously exercised by two separate tests, which is why "met, narrow" was
never earned and ADR-0005 §4 downgraded it.
K9 gets the assertion it did not have: snapshot at seq N + events N+1..M
must equal the from-genesis fold, hash-compared, on GroundState,
single-seed on purpose — AM-7's probe folds a multi-seed log, which is not
a replay of anything, and that defect is not repeated. Two positive
controls: the log must exceed 50 events, and the mid-log snapshot must
differ from the end state or "apply the remainder" is vacuous.
Proof it works: the exact mutation that SURVIVED in CB-WP-0005 — making
Snapshot::take discard its EventSeq — now fails on the K9 assertion.
AM-11's mutation breaks NullRng::draw to return its bound and the shared
suite fails. That is what M-D4-SWAP claims — either impl substitutable —
and exactly what two separate per-impl tests could never demonstrate.
M-D1-MUT: 7 -> 8 of 14. CB-EV-0001's scoreboard is refreshed: AM-2, AM-5
and AM-9 added, AM-6 moved to enforced, and the headline total corrected
from 4 to 8 — it had gone stale inside the same workplan that produced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
T03 reported AM-5 at 87.0 s / 61.3 s and called it a 45% breach of the
60 s target. Re-measured with the fixed instrument on a quiet machine:
load before measuring: 0.14 per CPU over 8 CPUs — quiet
dev toolchain (default features) 37.3 s [ok target <= 60 s]
best of 3: 37.3, 42.9, 46.2 (spread 1.24x)
shipped runtime (--no-default-features) 41.2 s [ok target <= 60 s]
best of 3: 41.2, 50.8, 54.2 (spread 1.32x)
AM-5 is MET with 1.6x headroom. The 87.0 s was measured while the machine
was busy with mutation-check and cargo builds — a timing measurement under
contention measures the contention.
That is the same error class as AM-6's, committed two tasks later in the
same session by the same author, in the row immediately after the one
where it was diagnosed. Knowing the failure mode did not prevent it; only
building the guard did. That is the InnerLoop v1.2 design-goal argument
holding up under a third instance: optimize for cheap correction, because
prevention keeps not converging.
The instrument now refuses to measure above 0.5 load per CPU, takes the
best of 3, and warns when the spread exceeds 1.25x. Best, not worst: a
build-time ceiling asks whether the machine can do it in 60 s, the mirror
of AM-6's best-of-N for a throughput floor. The spread warning fired on
the shipped-runtime samples — consecutive clean builds degrade 37.3 ->
46.2 — so a quiet machine is not a uniform one either.
The escalation to a maintainer decision is withdrawn: there is no breach.
The build profiling done while the breach was believed real is recorded in
the log rather than acted on — 174 s of CPU work at only 3.2x parallelism
on 8 cores, a ~22 s serial proc-macro chain, lto=thin worth ~6 s, and
pinning ppv-lite86 to drop zerocopy making it worse (23 -> 25 crates).
With 1.6x headroom there is nothing to buy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
loop-lint failed at 427 lines. Third workplan in this project to cross the
~400-line loadability limit, which makes it structural rather than a
formatting nuisance: delivery notes accumulate in a file whose job is to
state intent, and every remaining task would push it further over.
history/260801-cb-wp-0006-log.md now holds the per-task narrative; the
workplan keeps a one-line pointer per task. 427 -> 270 lines, with room
for T05-T09. T08 draws on both.
Same fix shape as CB-WP-0005, where the limit exposed a circular reference
rather than mere length. A gate that keeps finding real structure is worth
more than its stated rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
T07 — CB-EV-0004. 70 responses, $16.03.
Test 1 MET: 1 spec -> 2, 58 rules -> 76, 1 source file -> 10, and the new
denominator came in at 83% naming K10/K14/K18. The stated failure mode — a
widened denominator reporting the same percentage — did not occur.
Test 2 UNMET: M-D1-MUT 4 of 14 against >=10, and wrong about what as well
as how much. The diagnosis was three absent kernel rules; the measurement
found eight rows with no instrument at all.
Test 3: quality held, and widening surfaced far more than the seven known
defects — six further uninstrumented rows, HDN #7 (rule-coverage's
self-test green while the tool was broken), a stale $248.46 invisible to
facts-check because it was untagged, and a fifth error class.
The clean test CB-WP-0004 was owed is now run, on a pass that used the
tools without building them. The two categories whose tools removed the
manual path are at 0 turns two passes on; the two that merely offered a
better option are now the entire mechanical cost of a pass. Absolute
mechanical cost fell $51.76 -> $6.13. The mechanism holds.
Reported because nothing else would: SH-3 batching is 0.0% this pass — 67
tool calls across 67 responses — against a 20% target, and mean context
315,170 against 200,000. SessionShape has stated these since CB-WP-0003
and none has ever been enforced.
T08 — the retrospective answers its question: yes, a weak mutation is the
new grep, and it is worse, because it fails in the opposite direction.
mutation-check's first run produced two SURVIVED verdicts and both were
the author's own no-op mutations.
The fifth error class: false accusation. HDN, TA, SSB and DFD all
under-report — a real problem passes. FA over-reports: it publishes the
claim that working code is broken, sends the next pass to fix something
that is not broken, and is more credible than the truth because it arrives
with a measurement attached. Thirteen instances, five classes, six passes,
and the newest class is one that hardening created.
One correction to CB-WP-0004 T06: "a gate only pays if it removes the
manual path" is a predictor of whether a gate saves money, not a criterion
for whether it is worth having. mutation-check fails that test and
produced the most valuable findings of the pass.
InnerLoop v1.4: where a claim rests on numbers, the adversarial reviewer
must read the assertion behind each quoted number and mutate it.
Re-running the command that prints a number is not verification of that
number. Second verification step to inherit the author's blindness; both
fixes replace re-derivation with adversarial execution.
CB-WP-0005 status -> done, 5 of 8 tasks, 3 cancelled into CB-WP-0006.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tier-L survey plus the full review trail. v1 was REJECTED by adversarial
review with four blocking findings; this is v2, and the finding below is
the reviewer's as much as mine.
The root cause is neither v1's story ("coverage scope is too narrow" —
true but shallow) nor the reviewer's cheaper one ("the link check points
at one file" — true but catches 3 of 7). It is:
Every coverage instrument in this project counts NAMES. None counts
ASSERTIONS. M-D1-COV counts covers: tags; M-D1-LNK counts rule-ID
strings. Both answer "is this rule mentioned?" and neither answers
"does anything fail if this rule is violated?"
Four of seven defects found this pass are named in the source and inert,
so no name-based check finds them. This subsumes the oldest error class:
all five prior harness-does-nothing instances have the same shape.
Two are mutation-proven, because the reviewer set that bar for findings
they had only read:
AM-7 — folding a 100k-event log from fresh(999) instead of fresh(42),
an unrelated genesis state, leaves the test green. The hash reaches
only a println!; the sole assertion is on elapsed time. And the log is
built across games seeded 42,43,44... so it could not be asserted as
written. evidence/CB-EV-0001 reports "AM-7 replay met, 2,290x" —
unearned on the hash-identical clause. HDN instance #6.
K9 — making Snapshot::take discard its EventSeq and store 0, which
destroys the half of K9 that says "+ the EventId it includes", leaves
the test green. On the acceptance criterion GameKernel §2.4 is zero of
three, not two of three.
Also found: K10, K14, K18 named nowhere in source; AM-10 vacuous (a K6
determinism lint reported under a D4 leak row, over a cb-*-api crate that
does not exist); AM-11's conformance suite does not exist; AM-1b measured
and omitted from the evidence scoreboard.
All twelve challenge findings conceded, none defended on its facts. v1's
prediction was rejected as unfalsifiable AND already determined; the
replacement names its refuting outcome and is beatable in both
directions.
Chaos d4=1, no override. Declaration 1 of the 12-declaration window.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The question was whether converting agent work to deterministic tooling
recovers capacity or the work reappears elsewhere. Neither, cleanly:
tooling recovers capacity where it removes the possibility of doing the
work by hand, and recovers nothing where it merely offers a better
option.
Sorting the five candidates by that test reproduces the results exactly.
env-setup and task-done left no manual path — env-test fails the build if
the friction returns, and task-done refuses to emit an estimate, so the
old way means knowingly writing fiction. Both met. facts-check and
`make status` left grep and heredocs one keystroke away. Both missed.
This is the principle ADR-0003 already used to reject explicit session
markers — nothing that depends on discipline gets to be the primary
index. It governs tooling economics too, and CB-RES-0003 did not apply
it.
Stated confidence sorted the extremes and said nothing in the middle,
where every judgement call lives: both high met, both medium missed, and
the single low also met, so it was miscalibrated at that end too. Future
reviews state the mechanism instead — after this fix, is the manual path
impossible, absurd, or merely inferior? — and only the first two may
carry a cost prediction.
The confound is symmetric and does not explain the split: building
repo.py and task-done.py also required inspection and patching, and those
categories still went to zero. It inflates the size of the miss, not its
location.
The most valuable output is not the 6 points of 38 recovered. It is that
the loop made a falsifiable quantitative forecast, was wrong by 4-5x, and
could say exactly which parts were wrong and why.
CB-WP-0004 status -> done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Discharges the open item T07 raised. Values unchanged at 250,000 and
350,000; what was missing was a reviewed decision behind them, since
they had been set by the implementer in the commit that measured them
and that also changed the feature gating being measured.
The ADR supplies the argument T07's test requires -- why the targets
bind on FUTURE work rather than merely passing present work:
AM-4a leaves 3,750 lines of headroom (1.5%), about one small crate,
so any new shipped-runtime dependency breaches it almost immediately.
That is intended: the shipped runtime should be effectively frozen.
AM-4b leaves 32,979 lines (10.4%), deliberately looser -- dev tooling
should absorb one moderate dependency without a spec change, not two.
Both are ceilings on a quantity that only grows by choice. Nothing
drifts across them; only adding a dependency does.
Falsification condition stated: if a later pass raises AM-4a to
accommodate a dependency it wants, that is the failure the ceiling
exists to catch, and the answer is an ADR arguing for the dependency.
First ADR written under the correction/retarget test; sets the shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The retrospective question was whether the mechanism set is complete or
each pass still finds a new class. This pass produced both an eighth
instance AND a fourth class, so the answer is the uncomfortable one.
Ledger: 10 instances, 4 classes, across 3 workplans. Every pass has
produced at least one class the previous pass had not seen.
HDN harness-does-nothing 5 executable assertions
TA trusted arithmetic 3 re-derivation
SSB same-sample blind spot 1 assertions over ALL the data
DFD duplicated-fact drift 2 NEW -- reading a copy against source
DFD is genuinely distinct: no positive control catches it, because both
copies are internally consistent, and re-derivation does not either,
because the copy faithfully reproduces what it was copied from. Found
when an inlined price sheet went stale within an hour of T11 changing
the real one.
So v1.2 stops trying to enumerate classes in advance. Every error in
three passes was corrected in-session for under ~1% of the pass, so the
stated design goal is now cheap CORRECTION: keep raw data so numbers are
re-derivable, keep artifacts small and committed so a wrong number is
one grep from everywhere quoting it, give every number a command.
Plus the one rule the new class earns: single source of fact.
The original hypothesis is revised rather than confirmed. "A rule that
cannot be executed is not a rule" is wrong -- the two most valuable
corrections in the project came from a decorative rule that cannot be
automated (re-derive inherited numbers). An executable rule fires
reliably and catches one class; a decorative one fires unreliably and
can catch any class, including unnamed ones. Keep both.
Gates this pass: loop-lint caught 3 real violations on first run, then
failed on its own author within the hour when a T07 edit pushed
InnerLoop.md to 407 lines against its own 400 limit.
CB-WP-0003 complete: 11 of 11 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blanket "no retargeting in the measuring commit" rule would have been
wrong. CB-WP-0002 moved AC-1 three times in exactly that shape and every
move was correct -- each time a new instrument disproved the old figure.
Four legitimate corrections would have been forbidden to catch one bad
retarget.
The test is mechanical rather than a statement of intent:
correction the target moves and the implementation does not; legal in
the same commit provided the instrument's output is there
retarget the same commit changes both the target and the code the
target measures; requires an ADR stating why the new target
binds on future work
Applied retroactively: AM-4a/AM-4b are UNRATIFIED. They were set after
seeing the measurement, in the commit that produced it, with the
implementation changing too -- a retarget by this test. make dep-weight
is currently enforcing a target no reviewed decision stands behind.
Recorded as an open item; ratifying or changing them is a maintainer
decision, not an implementer's.
Also: specs/InnerLoop.md split into InnerLoop.md (process) and
InnerLoopReference.md (rubric, template, rules, definition of done).
Not a stylistic choice -- `make loop-lint` failed on the commit that
pushed the file to 407 lines against its own ~400 limit. The gate added
this morning to make that rule executable caught its own author within
the hour, which is the cheapest possible demonstration that it works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
41 rules classified executable / checkable / decorative, each tagged with
the failure class it catches. Counts: 11 executable, 22 checkable, 4
decorative (one of them dead policy).
Audit: history/260731-inner-loop-rule-audit.md
New tools/loop-lint.py makes 7 rules executable (tier declared, chaos
roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file
loadability, reporting tools expose --self-test). It found three real
violations on its first run, none previously visible:
- specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit
the loop has stated since v0.2 and never measured. Split at its own
section boundaries into Blueprint (1-8) + Runtime (9-15).
- tools/dep-weight.py and tools/rule-coverage.py had positive-control
logic and no --self-test, so nothing verified the control worked.
Adding rule-coverage's self-test exposed a latent instance of the exact
class this workplan is about: if the spec regex stopped matching, rules
was empty, missing was empty, and the tool exited 0 reporting "0/0" --
a silent pass, in the tool that reports our headline AM-1 number. Both
tools now assert they found something before reporting.
Two demotions applied in the spec rather than left implicit: "structured
over prose" is marked guidance (nothing can check it), and the 8k/10k
token budget is struck through and marked DEAD POLICY pointing at T05.
The audit's uncomfortable finding: rule 13 (re-derive inherited numbers)
has no mechanical form, is deliberately left decorative, and caught the
LARGEST error in CB-WP-0002. That is a counter-example to this
workplan's own hypothesis. "A rule that cannot be executed is not a
rule" is wrong as stated; the defensible version is that such a rule
cannot be relied on to fire, so it must not be the only defence for a
class that matters.
Class coverage: harness-does-nothing has five executable rules;
trusted-arithmetic has ZERO and produced the largest single error.
make loop-lint and make self-tests wired into `make all` and CI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.
v1.1 adds:
- the instrument must exist and the target must come out of it;
targets are provisional until the tool emits them
- a number inherited from earlier work is re-derived before use as a
target, or cited as unverified
- every reporting tool exposes --self-test, run before the number
- cost is in the definition of done; M-D2-CST may not be uncomputable
The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.
CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.
Workplan status: done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.
The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:
- the dedup positive-control table failed its own checksum (group
sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
groups are 206, not 208) — the worst place for a slip, since that
table is the sole evidence for the load-bearing dedup assumption
- the multi-model table was labelled "deduped" while holding per-line
counts, three lines below the paragraph condemning per-line counts
- the $92.21 acceptance target excluded the subagent tree, which ran
INSIDE the pin window. A collector correctly implementing the
survey's own blind-spot finding would have failed the benchmark.
Target is now $92.87, stated as its two components.
Folded in, each re-derived before conceding:
- attribution must key on sessionId, not wall-clock: two sessions
overlap 4h13m on this repo carrying ~$12 (one of them was this
workplan's own background job)
- only 14 of 33 commits name a task; the other 19 hold 33% of cost
- cache writes bill per TTL — pricing the aggregate at 1h inflates
the subagent by 43%
- the "~87% cache" figure matched nothing; 64.6% of cost is cache
read, 88.4% is all cache
- fable is 35% of calls and 60% of dollars; opus is the inverse
Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.
Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.
Three errors found in this workplan's own Purpose section, all
overstating cost:
- one API response is written as up to six JSONL lines, each
repeating the same usage object (657 lines, 346 responses)
- a multi-model session was priced entirely at Fable 5 rates,
violating a per-model rule MetricsAndScenarios §1a already had
- the transcript is a live file, so an unpinned total is not
repeatable (346 -> 356 responses between two runs minutes apart)
CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.
Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.
Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop has now survived CB-WP-0001 end to end and is revised from what
that pass actually taught, recorded in
history/260731-inner-loop-retrospective.md.
Held up: the adversarial review (one blocking and three significant
findings on its single use, resolved with new measurements rather than
argument), the parity-cap rule, the provisional U-item mechanism, and
the ADR gate.
The gap: both serious errors in the pass were measurement errors, and
the loop caught neither. Review reads prose; these were claims about
numbers. In both cases a harness ran successfully while doing no work —
a JS harness timing rejected no-ops, and a Rust benchmark computing
throughput for rounds that never completed.
v1.0 adds:
- Measurement validity: a harness must assert it performed the work it
reports. A number from a run that cannot prove it did the work is void.
- Metric feasibility: every metric names its instrument and is checked
reachable against the contracts in its own spec.
- No silently-ignored input; decisions get commands, not defaults;
scaffolds are exercised or marked; coverage gates that count tags say
so.
- Evidence must state what a comparison does not support.
- The chaos roll is recorded even when it changes nothing, so a
mechanism that never fires is visible rather than assumed.
CB-WP-0001 is complete: 9/9 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
