fix-consistency appended state_hub_workstream_id rather than replacing
the empty one, leaving a duplicate YAML key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review of where token-priced turns did work a deterministic tool could
do. Method: classify every turn in both transcripts by the tool calls it
made. The classifier is committed in tools/cb-cost.py and emitted by
`make cost-mix`, so the baseline is reproducible and the same command
can later falsify the predictions.
mech environment setup 84 turns $15.33
mech ad-hoc text patching 75 turns $13.86
git 37 turns $13.85
mech hub task status 25 turns $ 7.46
mech orientation / inspect 49 turns $ 6.87
hub other 32 turns $ 6.22
mech ad-hoc transcript 39 turns $ 4.56
mech workplan status edit 21 turns $ 4.06
MECHANICAL (dedup) 290 turns $51.26 = 38% of pass
Largest category is `cd` and `export PATH` -- pure friction, and
dep-weight.py already patched it at the leaf, which is evidence it was
noticed and fixed in the wrong place. Second is heredocs string-patching
markdown, which is also the mechanism behind duplicated-fact drift, the
error class InnerLoop v1.2 names and cannot gate.
Explicitly NOT automated: git (37 turns, $13.85) is mostly commit
message authorship -- the highest-output turns in the corpus and the
project's reasoning record. Automating it would save money and destroy
what makes corrections cheap.
CB-WP-0004 implements five candidates and predicts $33-41 recovery
(25-30%), below the 38% measured share on purpose: some inspection and
patching is genuinely exploratory.
The control loop is the deliverable, not a formality. T05 tests three
things and must report all: did mechanical turns disappear, did they
RELOCATE into prose, and did quality hold. If mechanical turns fall and
prose rises by as much, the saving is zero and that is the result to
publish.
Also a self-indictment worth recording: every hub update_task_status in
this project carried hand-typed token estimates, in a repo whose central
finding is that estimated token counts are worthless. T02 fixes it by
reading measured values from cb-cost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Discharges the open item T07 raised. Values unchanged at 250,000 and
350,000; what was missing was a reviewed decision behind them, since
they had been set by the implementer in the commit that measured them
and that also changed the feature gating being measured.
The ADR supplies the argument T07's test requires -- why the targets
bind on FUTURE work rather than merely passing present work:
AM-4a leaves 3,750 lines of headroom (1.5%), about one small crate,
so any new shipped-runtime dependency breaches it almost immediately.
That is intended: the shipped runtime should be effectively frozen.
AM-4b leaves 32,979 lines (10.4%), deliberately looser -- dev tooling
should absorb one moderate dependency without a spec change, not two.
Both are ceilings on a quantity that only grows by choice. Nothing
drifts across them; only adding a dependency does.
Falsification condition stated: if a later pass raises AM-4a to
accommodate a dependency it wants, that is the failure the ceiling
exists to catch, and the answer is an ADR arguing for the dependency.
First ADR written under the correction/retarget test; sets the shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The retrospective question was whether the mechanism set is complete or
each pass still finds a new class. This pass produced both an eighth
instance AND a fourth class, so the answer is the uncomfortable one.
Ledger: 10 instances, 4 classes, across 3 workplans. Every pass has
produced at least one class the previous pass had not seen.
HDN harness-does-nothing 5 executable assertions
TA trusted arithmetic 3 re-derivation
SSB same-sample blind spot 1 assertions over ALL the data
DFD duplicated-fact drift 2 NEW -- reading a copy against source
DFD is genuinely distinct: no positive control catches it, because both
copies are internally consistent, and re-derivation does not either,
because the copy faithfully reproduces what it was copied from. Found
when an inlined price sheet went stale within an hour of T11 changing
the real one.
So v1.2 stops trying to enumerate classes in advance. Every error in
three passes was corrected in-session for under ~1% of the pass, so the
stated design goal is now cheap CORRECTION: keep raw data so numbers are
re-derivable, keep artifacts small and committed so a wrong number is
one grep from everywhere quoting it, give every number a command.
Plus the one rule the new class earns: single source of fact.
The original hypothesis is revised rather than confirmed. "A rule that
cannot be executed is not a rule" is wrong -- the two most valuable
corrections in the project came from a decorative rule that cannot be
automated (re-derive inherited numbers). An executable rule fires
reliably and catches one class; a decorative one fires unreliably and
can catch any class, including unnamed ones. Keep both.
Gates this pass: loop-lint caught 3 real violations on first run, then
failed on its own author within the hour when a T07 edit pushed
InnerLoop.md to 407 lines against its own 400 limit.
CB-WP-0003 complete: 11 of 11 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five scenarios encoded U-item defaults with no owner and no review date,
so they could shape the kernel indefinitely while looking handled.
Each now carries provisional_owner and provisional_raised, and the
runtime's ScenarioFile learned both fields (deny_unknown_fields meant
adding them to YAML alone would have failed every scenario -- the parser
had to agree).
make coverage reports every provisional item with its owner and age in
days, warns on any with no owner, and warns past 30 days. It WARNS
rather than breaking the build, on purpose: the ruling is ground-game's
to make and the kernel cannot make it for them. What the loop can
enforce is that evidence files list them, which is now stated in
GroundRules.
rule-coverage --self-test gained an assertion that every provisional
item has both fields, so the next one added without them fails a
command rather than passing quietly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The copy in MetricsAndScenarios 1a went stale within hours of T11: it
still showed the sonnet intro rate as a comment after the real sheet
moved dated rates into data. A spec that duplicates a data file will
always drift from it; the copy is now a pointer that says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
make coverage now reports a second number: how many claimed rules are
also NAMED in the aggregate source. 58/58 tag coverage was weaker
evidence than it read as, and this says how much weaker.
Measured: 49 of 58. Nine rules are claimed by a scenario and appear
nowhere in games/ground/src/lib.rs --
GR-D07 GR-F02 GR-L03 GR-O03 GR-P01 GR-P02 GR-P03 GR-P04 GR-T01.
The gate REPORTS rather than fails, on purpose. Closing the gap by
adding those IDs to comments would satisfy the check without
establishing that any of the nine is implemented -- the overclaim
InnerLoop implementation rule 2 exists to prevent, and one CB-WP-0001
already committed once. Each needs its implementation confirmed before
it is tagged; promoting M-D1-LNK to a failing gate is correct after
that, not before.
Also added: a phantom check that fails when a rule id appears in code
that the spec does not define (currently zero).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review step targeted the survey. Every serious error in this project
has been in measurement or build configuration, so the loop was
adversarially reviewing the artifact cheapest to fix and leaving
unreviewed the one where errors occur.
Step 2 now routes by risk: when the claim rests on numbers, the reviewer
gets the harness and the evidence file too, and must reproduce the
number independently rather than read about it.
The addition that matters more, because it was learned the hard way: a
reviewer re-derives the author's claims and therefore inherits the
author's SAMPLING. CB-WP-0002's dedup invariant was checked twice --
survey 206/206 groups, then the reviewer independently -- and both used
the main transcript. It is false in the 8-response subagent tree neither
looked at. Two independent verifications, one shared blind spot.
Rule: the reviewer re-derives on a different sample than the author
used, and where only one sample exists, says so rather than reporting
a clean verify.
Also recorded: what review demonstrably DOES do. $0.66 and $1.11 across
two passes, ~1% of each, both finding approval-blocking defects. Cost is
not a reason to skip it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blanket "no retargeting in the measuring commit" rule would have been
wrong. CB-WP-0002 moved AC-1 three times in exactly that shape and every
move was correct -- each time a new instrument disproved the old figure.
Four legitimate corrections would have been forbidden to catch one bad
retarget.
The test is mechanical rather than a statement of intent:
correction the target moves and the implementation does not; legal in
the same commit provided the instrument's output is there
retarget the same commit changes both the target and the code the
target measures; requires an ADR stating why the new target
binds on future work
Applied retroactively: AM-4a/AM-4b are UNRATIFIED. They were set after
seeing the measurement, in the commit that produced it, with the
implementation changing too -- a retarget by this test. make dep-weight
is currently enforcing a target no reviewed decision stands behind.
Recorded as an open item; ratifying or changing them is a maintainer
decision, not an implementer's.
Also: specs/InnerLoop.md split into InnerLoop.md (process) and
InnerLoopReference.md (rubric, template, rules, definition of done).
Not a stylistic choice -- `make loop-lint` failed on the commit that
pushed the file to 407 lines against its own ~400 limit. The gate added
this morning to make that rule executable caught its own author within
the hour, which is the cheapest possible demonstration that it works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mechanism never fired. Two rolls across two workplans (9, then 2)
against ~0.2 expected firings at d10. At ~2 tier decisions per workplan
it would take roughly twenty workplans to see four overrides -- a rate
that prevents its own evaluation, which T06 named as the one option to
reject.
Raised to d4 for the next 12 tier declarations, then kept, reverted to
d10, or deleted on the evidence. Expected ~3 firings.
Cost stated up front rather than discovered: a chaos-L override on
S-shaped work buys a full survey + review + ADR, and the comparable
measured figure is CB-WP-0001 T03 at $9.91. The window should cost
$20-30. Cheaper than carrying an unevaluated ritual indefinitely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The price sheet had two defects of one shape -- a schema that could not
hold the fact it needed, the same criticism the cost survey levelled at
the State Hub.
CA-16 time-boxed rates are DATA. Sonnet's intro price lived in a
`# intro ...` comment and was invisible to the collector that
reads the file. Now promo_input/promo_output/promo_until,
applied per response at its own timestamp.
CA-17 the 90-day staleness rule was prose in MetricsAndScenarios 1a
that every M-D2-CST verdict silently inherited. Now `recorded`
+ `max_age_days` in the sheet, and a stale sheet ABORTS.
Both are exercised by make cost-test: the promo rate must apply before
2026-08-31 and lapse after, and a 102-day-old sheet must trip.
Applying CA-16 moved AC-1 from $93.32 to $93.15 -- the $0.17 CB-EV-0002
predicted, now collected rather than noted. That is a legitimate
retarget under T07's distinction: the instrument disproved the target,
and its output is in this commit. The number has now been stated five
times ($248.46, $92.21, $92.87, $93.32, $93.15), each correction from a
different mechanism.
Evidence tables regenerated from the tool rather than hand-patched,
per CA-15 -- which is the rule that exists because hand-typed tables
were the only thing the adversarial review found wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 8k/10k per-task token budget was never referenced or enforced and
T08 blew past it silently. Replaced with a budget that can actually
fire.
The design constraint is the interesting part: per-task cost needs the
commit that CLOSES the task, so a per-task budget is unavoidably
retrospective -- it can only report a breach after the money is spent,
which is the dead-policy failure again. What IS observable mid-task is
spend since the last commit, because the transcript is append-live. So
the budget binds on the open remainder.
CB-01 budget = USD since the last commit, via `make cost-budget`
CB-02 soft $10.00 (state progress, decide), hard $22.00 (stop)
Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50
$1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum
-- it would have fired exactly once on the calibration pass. Hard is ~2x
the observed max, a value never reached in 32 intervals, so reaching it
means the session is doing something the data has no example of.
Both thresholds are set ABOVE every observed value, so they bind on
future work rather than ratifying present work -- the distinction T07
is about.
Stated limit: it is a command, not a daemon. An agent that never runs it
gets no signal, which is the dead-policy failure one level up. Mitigated
only by being free to run and on the one command surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The task's original premise was wrong and is recorded rather than
deleted. It was written to prescribe one task per session; measurement
says the variable is context, not turn count.
Measured:
- compaction cut context 27x (542,991 -> 19,974) and cost/turn 3.1x;
the 202 turns after C1 cost less than half the 136 before it
- a turn costs $0.010 at 20k context and $0.270 at 540k
- break-even for a compaction is 2-11 turns, so: compact whenever
context exceeds ~300k and work remains
- a fresh session is NOT free -- cold start floors at ~51k and must
then re-read the artifacts a compact summary already holds (~66k).
Prefer compaction to continue work; prefer a fresh session when the
task changes, because then prior context is pure overhead.
cb-cost now emits SH-1..SH-3 so the targets come from the instrument
rather than from analysis, per InnerLoop v1.1. All three are UNMET
(mean context 232,982 vs 200,000; p90 492,042 vs 300,000; batching 7.8%
vs 20%) and are reported unmet rather than retargeted -- retargeting in
the commit that first measures is precisely what T07 exists to prevent.
Eighth error instance found while writing this: CB-WP-0001's claim that
"0 of 330 tool calls were batched" is wrong. 330 was the count of
single-call responses, not the total; 31 responses batched, covering 76
calls. It was carried into this workplan unverified. Trusted-arithmetic
class -- the one the T01 audit flagged as having no executable defence,
confirming that finding within hours of making it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both items were completed as part of the T01 audit: the two older
reporting tools gained --self-test entry points, and loop-lint's
self-test check makes the contract mechanical for any new tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
41 rules classified executable / checkable / decorative, each tagged with
the failure class it catches. Counts: 11 executable, 22 checkable, 4
decorative (one of them dead policy).
Audit: history/260731-inner-loop-rule-audit.md
New tools/loop-lint.py makes 7 rules executable (tier declared, chaos
roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file
loadability, reporting tools expose --self-test). It found three real
violations on its first run, none previously visible:
- specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit
the loop has stated since v0.2 and never measured. Split at its own
section boundaries into Blueprint (1-8) + Runtime (9-15).
- tools/dep-weight.py and tools/rule-coverage.py had positive-control
logic and no --self-test, so nothing verified the control worked.
Adding rule-coverage's self-test exposed a latent instance of the exact
class this workplan is about: if the spec regex stopped matching, rules
was empty, missing was empty, and the tool exited 0 reporting "0/0" --
a silent pass, in the tool that reports our headline AM-1 number. Both
tools now assert they found something before reporting.
Two demotions applied in the spec rather than left implicit: "structured
over prose" is marked guidance (nothing can check it), and the 8k/10k
token budget is struck through and marked DEAD POLICY pointing at T05.
The audit's uncomfortable finding: rule 13 (re-derive inherited numbers)
has no mechanical form, is deliberately left decorative, and caught the
LARGEST error in CB-WP-0002. That is a counter-example to this
workplan's own hypothesis. "A rule that cannot be executed is not a
rule" is wrong as stated; the defensible version is that such a rule
cannot be relied on to fire, so it must not be the only defence for a
class that matters.
Class coverage: harness-does-nothing has five executable rules;
trusted-arithmetic has ZERO and produced the largest single error.
make loop-lint and make self-tests wired into `make all` and CI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The framing hypothesis -- a rule that cannot be executed is not a rule
-- survives but was too coarse. CB-WP-0002 corrected one number four
times and each correction came from a DIFFERENT mechanism, none finding
more than one. Two limits are now measured rather than assumed:
- a positive control cannot catch trusted arithmetic (both errors in
$248.46 were sums over data that really existed)
- adversarial review cannot catch a same-sample blind spot: survey and
reviewer both verified dedup on the main transcript, and it is false
in the subagent tree neither examined
Task-level revisions:
T01 also classify each rule by WHICH failure class it catches; seven
instances across three classes are now on record
T02 scope reduced -- the --self-test contract, cb-cost's five
assertions, make cost-test in CI are delivered; remaining work is
rule-coverage/dep-weight plus a check that new tools carry one
T03 now backed by two measurements ($0.66, $1.11 at ~1% of a pass),
and must also state what review CANNOT do
T04 premise overturned -- sessions are bounded-quadratic, not
quadratic. /compact cut context 27x and cost/turn 3.1x. Measure
compaction against a fresh session before prescribing either
T05 unblocked, with calibration data ($1.29-$21.02/task) and the
constraint that a budget binds only on the 67.5% attribution
reaches, only after the closing commit
T07 sharpened -- CB-WP-0002 retargeted three times, correctly, so a
blanket prohibition is wrong; separate instrument-disproved-target
from implementation-missed-target
T10 its prediction is already falsified; the successor question is
whether class-by-class hardening converges at all
T11 NEW -- the price sheet cannot express a dated rate and becomes
wrong on 2026-08-31
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.
v1.1 adds:
- the instrument must exist and the target must come out of it;
targets are provisional until the tool emits them
- a number inherited from earlier work is re-derived before use as a
target, or cited as unverified
- every reporting tool exposes --self-test, run before the number
- cost is in the definition of done; M-D2-CST may not be uncomputable
The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.
CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.
Workplan status: done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- InnerLoop definition-of-done gains a cost row: M-D2-CST may no longer
be recorded uncomputable, and composition must be reported, not only a
total.
- InnerLoop Step 5 gains the --self-test contract: every tool that
reports a number exposes one, and it runs before the number does.
Rationale attached, because the case that motivated it is the one
review cannot catch — survey and reviewer both verified the same large
sample and both missed the small one.
- make cost / cost-test / cost-pin on the one command surface; cost-test
in `make all` and in CI.
- Hub now holds the measured figure for CB-WP-0001: 80.6M in / 323.6k
out against the 401,100 it previously estimated, low by ~200x. The
event states plainly that the hub schema cannot represent the 88% of
cost that is cache, and names `make cost-pin` as the authority.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AC-1 $93.32 ($92.21 main + $1.11 subagent), residual $0.000000. No
unmeasured rows. Composition: 88.0% cache, 11.9% output, 249:1 context
re-read to text written.
The question that could not be answered before: what /compact costs.
The transcript records it directly — 542,991 -> 19,974 tokens, a 27x
reduction. Cost per turn fell from $0.457 (136 turns before) to $0.149
(202 turns after). The 202 post-compact turns cost less than half the
136 before them.
This qualifies CB-WP-0003 T04: sessions are bounded-quadratic, not
quadratic. Cost grows with context between compactions and resets at
each one, so the failure mode is a long UNCOMPACTED session. Whether a
fresh session beats a compaction is now measurable, and T04 should
measure it before prescribing one task per session.
Stale $0.66/$92.87 targets updated across survey and workplan.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.
The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:
CA-02 assumed usage is identical across the lines of one requestId.
True in the main transcript (206/206 groups, verified twice — by the
survey and by the adversarial reviewer). FALSE in the subagents/
tree, where output_tokens is a running count: one response reads
5, 5, 195 across its three lines. First-wins scored it at 5.
So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.
The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.
Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.
make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
T05 still named $248.46 as its acceptance test and the Purpose total
still omitted the subagent tree. Both corrected in place, with the
error sequence recorded rather than overwritten.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).
Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.
The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.
MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Instrument: session transcript JSONL, deduped by requestId, priced per
model and per cache TTL. Hub rejected as a source (no cache fields, ~100%
error on the same work), kept as sink. Status bar and billing API
rejected on availability.
Attribution: (prev_commit, this_commit] ending-at-commit, scoped by
sessionId. Explicit session markers rejected because they depend on an
agent remembering a step — the exact failure this workplan exists to fix.
Hub status transitions rejected as an after-the-fact time index.
Stated honestly: we buy accuracy with specification complexity, and D1
gets worse. That trade is only right because the alternative on offer was
not a simpler correct number but one wrong by two orders of magnitude.
Seven failure modes recorded up front, each becoming a T04 test — most
importantly that dedup fails in the under-reporting direction, so the
invariant is asserted at runtime rather than trusted.
Gate satisfied: T04 may now write collector code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.
The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:
- the dedup positive-control table failed its own checksum (group
sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
groups are 206, not 208) — the worst place for a slip, since that
table is the sole evidence for the load-bearing dedup assumption
- the multi-model table was labelled "deduped" while holding per-line
counts, three lines below the paragraph condemning per-line counts
- the $92.21 acceptance target excluded the subagent tree, which ran
INSIDE the pin window. A collector correctly implementing the
survey's own blind-spot finding would have failed the benchmark.
Target is now $92.87, stated as its two components.
Folded in, each re-derived before conceding:
- attribution must key on sessionId, not wall-clock: two sessions
overlap 4h13m on this repo carrying ~$12 (one of them was this
workplan's own background job)
- only 14 of 33 commits name a task; the other 19 hold 33% of cost
- cache writes bill per TTL — pricing the aggregate at 1h inflates
the subagent by 43%
- the "~87% cache" figure matched nothing; 64.6% of cost is cache
read, 88.4% is all cache
- fable is 35% of calls and 60% of dollars; opus is the inverse
Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.
Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.
Three errors found in this workplan's own Purpose section, all
overstating cost:
- one API response is written as up to six JSONL lines, each
repeating the same usage object (657 lines, 346 responses)
- a multi-model session was priced entirely at Fable 5 rates,
violating a per-model rule MetricsAndScenarios §1a already had
- the transcript is a live file, so an unpinned total is not
repeatable (346 -> 356 responses between two runs minutes apart)
CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.
Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.
Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written back by fix-consistency: workstream and task UUIDs for the ten
CB-WP-0003 tasks, plus the regenerated WORK-RECORDS.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers what the immediate CI fixes did not. Framing finding: writing a
rule into InnerLoop did not prevent the next instance of the failure it
was written for; making it a CI step caught one on its first run. Four
instances of the harness-does-nothing class, zero prevented by prose.
Working hypothesis: a loop rule that cannot be executed is not a rule.
Each item is made executable or deleted, default delete.
Phase A makes rules executable: audit every InnerLoop rule as
executable/checkable/decorative, extend the positive-control gate beyond
benchmarks, and point adversarial review at measurement rather than only
research — all four real errors were in measurement and build config,
which the review never reads.
Phase B is session economics, from measurement: 592 turns, mean context
245k tokens, $0.245/turn in cache reads, zero batched tool calls, cache
reads 58% of cost against output at 12%. Cost is turns x context and
context grows with turns, so a long session is quadratic. Also replaces
the dead 8k/10k token budget, which was never enforced and which T08
exceeded by orders of magnitude with no signal.
Phase C removes or fixes the rest: resolve the chaos roll (never fired,
and at 1-in-10 will stay unevaluated), require retargeting to be an ADR
rather than something the implementer does in the commit that measures
the metric, give provisional items an owner and expiry, and strengthen
the coverage gate beyond tag-counting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gates existed; CI ran half of them and tolerated the failure case.
- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
prefix is a failure, and a run in which nothing executed is a failure.
Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
would have skipped every scenario while the pipeline stayed green.
Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
crate-count metric.
The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.
That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.
Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.
The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adopts both remediations from CB-EV-0001 §4 (maintainer decision).
Option A — serde_yaml is now optional behind cb-game-runtime's
`scenarios` feature. The scenario module, the ScenarioGame impl and the
string parsers behind it are cfg-gated; cb-sim opts in explicitly. Both
configurations compile and lint clean under -D warnings.
A trap worth recording: `default-features = false` on a *member*
dependency is silently ignored when the workspace dependency does not
specify it. The first attempt gated nothing while looking correct — the
build succeeded and cargo tree still showed all six YAML crates. Fixed
by setting it on the workspace dependency. This is the positive-control
failure mode in miniature: success was not evidence the change applied.
Retarget — AM-4 now measures third-party source under audit, split by
build configuration, replacing a crate count that was unreachable
without undoing K5/K7 and that does not compare across ecosystems.
Re-measured via the new `make dep-weight`, whose own positive control
refuses to report when any crate's source cannot be located:
shipped runtime 23 crates 246,250 lines target <=250,000 met
dev toolchain 29 crates 317,021 lines target <=350,000 met
own source 3,408 lines
Scenario tooling costs 70,771 lines a shipped game never compiles —
the split the single number was hiding.
Targets are set at current measurement plus headroom, so they bind on
future growth rather than retroactively passing what had failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AM-4: measured what each remediation option actually buys, rather than
leaving one recommendation unquantified. serde_yaml optional removes 6
crates, not 5 — ryu belongs to that group, since serde_json now uses
zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23,
inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only
reimplementing a primitive gets under 20, so the target is unreachable
without undoing K5/K7.
Also records that crate count compares badly across ecosystems, and
offers the alternative the count is a proxy for: 307,317 lines of
third-party source under audit against 3,398 of our own.
AM-12: corrected from "uncomputable" to measured. The refusal to
estimate was right; the claim that no instrument existed was wrong.
Session transcripts carry exact per-message usage including the cache
breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is
cache reads — cost is driven by context size times turn count, not by
output volume. What is still missing is per-task attribution, since
nothing marks task boundaries in a transcript.
CB-WP-0002 makes cost measurable and attributable: survey the
instruments, decide the attribution model by ADR, spec metrics that
include cost composition rather than a bare total, build a collector
whose positive control refuses to emit unreconciled numbers, and prove
it by answering a question that could not be answered before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop has now survived CB-WP-0001 end to end and is revised from what
that pass actually taught, recorded in
history/260731-inner-loop-retrospective.md.
Held up: the adversarial review (one blocking and three significant
findings on its single use, resolved with new measurements rather than
argument), the parity-cap rule, the provisional U-item mechanism, and
the ADR gate.
The gap: both serious errors in the pass were measurement errors, and
the loop caught neither. Review reads prose; these were claims about
numbers. In both cases a harness ran successfully while doing no work —
a JS harness timing rejected no-ops, and a Rust benchmark computing
throughput for rounds that never completed.
v1.0 adds:
- Measurement validity: a harness must assert it performed the work it
reports. A number from a run that cannot prove it did the work is void.
- Metric feasibility: every metric names its instrument and is checked
reachable against the contracts in its own spec.
- No silently-ignored input; decisions get commands, not defaults;
scaffolds are exercised or marked; coverage gates that count tags say
so.
- Evidence must state what a comparison does not support.
- The chaos roll is recorded even when it changes nothing, so a
mechanism that never fires is visible rather than assumed.
CB-WP-0001 is complete: 9/9 tasks done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evidence/CB-EV-0001-game-kernel.md records the acceptance run against
the CB-RES-0001 baseline.
Met: AM-1 rule coverage 58/58; AM-6 throughput 1.65M events/s against a
100k target; AM-7 scaling 1.08x at 20x workload and a 100k-event replay
in 4.13ms against a 5s budget; AM-8 zero divergence over 10 full runs
with fmt and clippy clean; AM-10 zero foreign collection types.
Not met and reported as such: AM-4 at 33 transitive crates against a
<=20 target. Attribution is in the evidence file. The recommended fix
is making serde_yaml optional (-5, a test-only concern), after which
the remainder is sha2 and rand_chacha, which K5 and K7 require. We are
not hand-rolling crypto primitives to win a dependency count.
AM-12 is recorded as uncomputable: per-task token counts were never
instrumented, and inventing a USD figure would defeat the metric.
A measurement error was found and corrected before publication. The
first benchmark reported 9.3M events/s on a flat curve. The workload
had a player selecting SUPPORT while parked at Stress 4, so GR-R03
rejected it, rounds never completed, and throughput was computed for
rounds that never happened. The bench now asserts the per-round event
count and panics rather than measuring a stalled loop. The corrected
figure is 5.6x lower.
The evidence file states plainly what the boardgame.io comparison does
and does not support: the ~450x command-rate ratio is cross-runtime and
cross-feature-set, so it is a direction, not a verdict, per the
InnerLoop parity-cap rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- GR-R09: after Round 5's End the game ends and scoring runs instead of
the round advancing.
- GR-E01: claimed Problems sum their printed values against the
player-count threshold (2p 5, 3-4p 7, 5-6p 9 in dataset 0.1).
- GR-E02 SHARED GROUND: shared score, Mastery reduced per Blame token
and per Denied Problem.
- GR-E03 COMMON PROBLEM: personal score is claimed value less Blame,
tiebroken by lower Stress then more Bonds.
- GR-E04 BONDED COALITIONS: connected components over Bonds only, so
Rivalries do not connect and an unbonded player is a coalition of
one; tiebroken by lower combined Stress then fewer Blame.
Ties yield every tied candidate rather than an arbitrary pick, which is
what "shared victory" in GR-E03/E04 asks for.
AM-1 rule coverage is now 58/58 (100%), 21 scenarios, 17 tests.
Caveat recorded rather than papered over: GR-E02's "successes" is not
defined in dataset 0.1. It is implemented as the count of claimed
Problems and both scoring scenarios are marked provisional, so a
ground-game ruling flips a scenario rather than the kernel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolution step 3 (GR-R06), the last unimplemented step:
- GR-D02 binding: one stage per consecutive round, DENY then ATTACK
then REVERSE, regardless of the player's later Stress.
- GR-D03 DENY: turn one face-up, unsolved, unprotected Problem face
down and Deny it. Under the U3 default no legal target is a no-op and
the sequence still advances.
- GR-D04 ATTACK: one extra Attack under the normal relation rules, then
place the Focus token beside the target even if it was cancelled.
- GR-D05 REVERSE: flip Focus to Blame, +1 Stress to the holder, one
Protection to the owner, unless the holder's GROUND-ND rejects it.
Under the U5 default the owner takes -2 either way and the sequence
ends.
- GR-D06 early end: a Support through a Bond that predates this round's
Support step cancels the stage and ends the sequence, and the placed
Focus token is removed. GROUND-GR ends it after the stage resolves.
- GR-D07: the marker returns to OFF, so a later End can re-trigger.
The Attack rules are now one routine shared by the chosen ATTACK Action
and the DARVO extra Attack, so GR-A06..A09 cannot drift between them.
Stage targets are named by their own command during Reveal, validated
against what the stage admits: DENY needs an eligible Problem, ATTACK
another player, and only a player with a live sequence may choose.
18 scenarios pass; AM-1 coverage 47/58 (81%), up from 41/58.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the three rules deferred in iter 2 and 3 because each needed a
decision from a player rather than a default I would have invented:
- GR-A11 GROUND-OU: restore a Denied Problem, protect a face-up one
from Deny, or cancel an Attack aimed at the chooser.
- GR-A12 GROUND-ND: remove a Blame token, break a relation involving
the chooser, or reject a Reverse.
- GR-L02 / GR-A05: the Support target answers after Reveal — accept or
decline a Bond where no relation exists, flip or break an existing
Rivalry. A Bond can no longer form without consent.
Each choice is validated against the state it claims to act on: a
restore needs a Denied Problem, a cancel needs an Attack actually aimed
at the chooser, a Blame removal needs that token present, a break needs
that relation to exist. GR takes no sub-choice, and OU and ND may only
use choices from their own list.
GR-A09 ordering follows the U8 default: an OU cancellation is chosen at
step 1 and applies first, so a Protection token is spent only when it
is what actually cancels.
14 scenarios pass; AM-1 coverage 41/58 (70%), up from 34/58.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
make coverage compares the GR-rule IDs defined in specs/GroundRules.md
against the covers: lists in scenarios/ground/*.yaml. It exits 1 if a
scenario claims a rule the spec does not define, so coverage cannot be
inflated by an invented ID, and exits 2 while rules remain uncovered.
Current reading: 34/58 (58%). AM-1 requires 100%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fills in resolution steps 1, 4 and 6 (GR-R06):
- GR-R05 mode choice: a player who revealed GROUND picks GR/OU/ND after
Reveal. Resolution refuses to start while any revealed GROUND lacks a
mode, and only that player may choose it.
- GR-A10 GROUND-GR: self -2 Stress and Freedom readied.
- GR-A01 INVESTIGATE: reveal the chosen hidden non-Denied Problem, then
draw one Solution; the draw still happens when nothing is revealable.
- GR-A02 SOLVE: spend a Solution of the Problem's suit and claim it; a
later resolver the same round spends nothing, per Lead order.
- GR-A13 tightened: INVESTIGATE must target a hidden Problem, SOLVE a
face-up non-Denied one. Previously any existing Problem was accepted.
Deck exhaustion (U4) reshuffles the discard, seeded from the game seed
and round so validate stays a pure function of state. The resulting
order travels inside DeckReshuffled, so replay never re-derives it.
Still pending, each because it needs its own decision command rather
than a default: GROUND-OU and GROUND-ND three-way choices (GR-A11/A12)
and the DARVO stage machine (GR-D02..D07). No scenario claims coverage
of them.
Filler picks in existing scenarios moved from GROUND to INVESTIGATE:
GROUND now has a real Stress effect, which was polluting the Support
and Attack assertions.
11 scenarios pass, 34 rules covered; 17 tests, fmt/clippy green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round machinery, system-driven (GR-R04/R06/R08):
- GR-R06 fixed step order with GR-R07 Lead-first ordering inside a step.
Steps 1 (GROUND), 3 (DARVO stages), 4 (INVESTIGATE) and 6 (SOLVE) are
not implemented yet; their Actions resolve as no-ops and no scenario
claims coverage of them.
- SUPPORT GR-A03/A04/A05 and ATTACK GR-A06..A09, with relation
formation, flip and break per GR-L01/L03/L04 and Protection
cancellation.
- GR-R08 End: DARVO trigger at Stress 5 in Lead order, Lead rotation,
Round advance, per-round flags cleared.
- Stress clamps 0-5 on every application, the U2 default, so a mid-round
spike that is reduced before End does not trigger DARVO.
Two consent-dependent rules are deliberately left out because they need
a decision command rather than a default: Bond formation (GR-L02) and
the target's flip-or-break choice on Support-through-Rivalry (GR-A05).
Both are noted in code and covered by a provisional scenario.
Fixes a defect in the T07 scaffold: relations were keyed by a tuple,
which JSON cannot use as an object key, so state_hash would have
panicked on any state holding a relation. Relation keys are now a Pair
newtype serialized as "a-b", with a regression test.
setup.patch may now create a final key so scenarios can seed open-ended
maps; a typo anywhere earlier in the path is still an error.
8 scenarios pass, 28 rules covered; 17 tests, fmt/clippy green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the RunOutcome::Unimplemented stub with a real runner:
- ScenarioGame trait: games own setup presets and the command
vocabulary, the runner owns execution, assertions, and determinism.
- K8 double-run: every scenario runs twice on the same seed and fails
on state-hash divergence.
- K4/K11: applied events go through Envelope into EventLog, so seq
monotonicity is enforced on the real path, not just in unit tests.
- setup.patch was parsed and silently dropped; the runner now applies
it generically and errors on a path that does not exist, so a typo
in a scenario can never pass as a no-op.
- Assertions: dot-path state lookup over objects and arrays, ordered
event subsequence matching by field subset, exact rejects-set match.
GROUND rules realized: GR-S01..S04 setup (seeded shuffle, deal, Lead,
Surface Problem face up), GR-R02 Select commit, GR-R03 stress gate and
Freedom spend, GR-A13 targeting legality.
cb-sim dispatches by the scenario's game prefix and reports rule
coverage. 3 scenarios pass, 7 rules covered; fmt/clippy/tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
