ADR-0008, tier M (survey and ADR merged).
D1 — SH-3 retired as a gate, kept as a diagnostic. Investigating it
found a third defect, deeper than the two this pass was declared on.
Re-deriving batching from the raw transcripts, independently of cb-cost:
CB-WP-0011 pass 54 with tools 0 batched 0.0%
gap -> next decl 16 with tools 6 batched 37.5%
CB-WP-0012 pass 86 with tools 0 batched 0.0%
gap -> next decl 10 with tools 1 batched 10.0%
CB-WP-0013 so far 10 with tools 0 batched 0.0%
Zero batched turns in 150 in-pass responses; 37.5% in one gap, above the
20% floor. Batching needs two calls whose inputs are known at once —
orientation work. Implementation consumes each step's result before the
next. SH-3's window is since the last commit, which during a pass is
always implementation. The metric could not read above ~0% in the window
it was gated on. A floor the window structurally excludes is not a
target.
This pass's own declaration was also wrong: it claimed batching "has got
worse" (7.8-8.6% vs 1.1-6.3%). Differently-placed windows, not different
behaviour. Withdrawn — the same class of error, in the pass written to
correct it.
Not retargeting to match the measurement: the floor was not moved to 6%,
the gate was removed on an argument about what the quantity is worth.
The number is still reported; only the verdict is gone.
D2/D3 — AM-4a counts --edges normal,no-proc-macro: 157,202, not 246,250.
The target moves down with it, 250,000 -> 161,000, so the correction
hands back essentially nothing (headroom 3,750 -> 3,798). Three controls:
the exclusion drops exactly the five expected crates, only removes and
never adds, and is not a no-op.
The DFD gate then caught the follow-on it exists for — three historical
documents carrying live fact tags for a number that had changed. Not
rewritten; untagged, with a supersession banner.
AM-4b is deliberately not corrected: its proc-macro share is unmeasured.
gate-review now reads 0 due, 0 silent, 0 drifted — GATE-REVIEW earns its
first caught entry by forcing SH-3's re-justification, and the registry
has no silent gates left.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The metric six evidence files reported as 0.0% was never measured.
--shape-budget windows since the last commit, and it is read while
writing the evidence file — right after a commit, when the window holds
one or two responses. SH-1/SH-2 are location statistics and survive n=2.
SH-3 is a rate: at n=2 its only possible values are 0%, 50%, 100%.
cb-cost.py gains sh3_line(), which below a minimum sample prints
"insufficient sample" and no verdict. The floor is derived: if the true
rate were exactly the 20% target, P(zero batched in n) = 0.8^n, and at
n=14 that is 4.4% — so "0 batched in 14" rules out a target-meeting rate
at ~95%. Below that the tool has nothing to say and now says so.
The window was kept rather than split. SH-3 could have been given a
per-pass window, but the budget's purpose is the open remainder since
the last commit, and giving one of three metrics a different window
makes "the window" ambiguous in a tool three specs cite.
Four controls, three mutations, each red for its stated reason —
including the one the evidence files actually hit, where a refusal is
printed as a measured zero.
SessionShape.md §4 carries the correction with the real per-pass figures
(1.1%-6.3%), beside the eighth trusted-arithmetic instance. This is the
ninth, and the second in this same metric. It also shows what the frozen
0.0% hid: against the pinned 7.8%-8.6%, batching has got worse, and six
passes reported a breach that was moving underneath them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR-0006 D1. cb-cost gains pass_costs, which buckets responses into
workplan windows in a single read of the transcripts — calling collect()
once per boundary would re-read every transcript per window, and status
is supposed to stay cheap enough that nobody replaces it with ls.
make status now reports the share over a trailing three passes with the
per-pass breakdown, keeps the lifetime figure labelled NOT the metric,
and prints the D2 exemption so the next reader does not re-derive the
standoff where the budget blocked its own repair. One pass would be a
coin flip: a meta pass reads 100%, a product pass 0%. Three is the
smallest window where the ratio means something and still moves.
First reading: 36% over the last three passes against 49% lifetime,
with CB-WP-0008 at 0%.
The first run reported $0.00 for every window. Transcript stamps are
UTC Z and git prints a local offset, so the string comparison put every
response before the first boundary. Both the fix and a self-test for it
are in; reverting the conversion turns four checks red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scope cut first, on the maintainer's decision after a spend review: the
project is 38% product / 62% loop-meta, cost per response is 2.9x worse
than its best window, and INTENT stage 0 still lacks a CLI player and
bots. CB-WP-0005 and CB-WP-0006 cost ~$74 — 31% of all spend — for zero
measured efficiency gain. T02 and T04 are cancelled unstarted.
T01: SH-1/SH-2/SH-3 now report over the window since the last commit, and
the cumulative figure is retained but labelled "history, NOT the metric".
The prediction held decisively — window 655,744 mean context against
cumulative 255,307, a 2.6x gap against a 20% refutation threshold. A
cumulative mean over 1,094 responses cannot detect a worsening trend
because the history outvotes the present.
T03: `make shape-budget`, modelled on CB-01/CB-02. Soft thresholds are the
existing SessionShape targets; hard is 1.5x, set before the next
measurement per §Step 4. Deliberately not in `make all` — failing the
build on context would block committing, and committing is what closes the
attribution window and is the natural point to compact, so a gate that
blocks the remedy is a trap. It fires HARD on its first run: 656,574
against a 300,000 ceiling.
InnerLoop v1.5 establishes the soft 25% meta budget. Workplans declare
kind: product|meta|mixed and `make status` reports the share; mixed splits
50/50 and says so. Soft on purpose — a task already started may be
finished, because stopping mid-task to satisfy a ratio wastes the work.
What it forbids is opening new meta work above the line. A pass that
exceeds it must say so in its evidence and name the product work
displaced.
First reading: 68% OVER, of $74.22 attributed. Product reads $0.00 because
the only product workplan, CB-WP-0001, predates qualified task ids and its
bare T## labels collide across passes — stated in the output rather than
papered over.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cb-cost gains --since, so the baseline (--pin 578dcbe, 662 responses,
$135.60) and this pass (--since 578dcbe, 83 responses, $7.82) are two
disjoint windows over the same transcripts. Every verdict is on share of
pass, since the windows differ 17x in size.
Test 1 — did mechanical turns disappear? Partly. Mechanical share fell
38.2% -> 32.4%. Six points against a predicted 25-30; reported unmet, and
no target was moved in this commit.
environment setup 11.5% -> 0.6% (85 turns -> 1) met, decisively
hub + workplan 8.5% -> 0.0% (46 turns -> 0) met
text patching 10.4% -> 9.6% not met
orientation 5.1% -> 21.2% not met, worse
The confound is stated before any defence of the numbers: this is the
pass that built the tools, and the two categories that missed are exactly
the two whose tools were under construction. The clean test is the next
pass, and it is carried forward rather than waived.
Test 2 — did the work relocate? Not into prose. Output tokens per
response fell 896 -> 681. Output's rising share of cost is a shrinking
denominator, not more writing. Cost per response halved and the evidence
refuses to claim it: that is compaction (mean context 232,982 ->
117,822), and attributing it to tooling would repeat CB-WP-0002's
original error in a new direction.
Test 3 — did quality hold? Yes, recorded as explicit judgment. make all
green with two gates that did not exist before, and four findings
surfaced this pass, three caught by controls written this pass — one of
them a 5.2x attribution error that would have reached the hub as a
measured number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the three hand-done steps of a task close (46 turns, $11.52 per
CB-RES-0003): the heredoc flipping status in the workplan file, the
hand-written hub call, and the hand-typed token counts.
The third is the reason this task exists. Every update_task_status this
repo produced carried estimated tokens_in/tokens_out — in a project whose
central finding is that estimated token counts are worthless. task-done
reads the measured figure from the transcripts, or refuses; there is no
path through it that emits an estimate.
cb-cost gains by_task_detail: cost, response count, model histogram and
token components per task. task-done imports cb-cost rather than parsing
its printed table, so the hub figure is not a copy that can drift from
its source.
The positive control found a real defect before the tool ran once.
Attribution keyed on a bare T\d\d from the commit subject, so CB-WP-0002
T01, CB-WP-0003 T01 and CB-WP-0004 T01 shared a bucket: the self-test
reported $12.10 for "T01" where the qualified figure is $2.33. That 5.2x
overstatement would have been pushed to the hub as a *measured* number —
the same fiction in a new form. task_label() now keys qualified subjects
on the full id and leaves unqualified ones bare rather than
retro-assigning them to a workplan. The pinned $93.15 benchmark is
unchanged, so historical attribution was not disturbed.
Fourth instance of trusted arithmetic: a number believed because a
program produced it rather than a hand.
Refusals, all exercised by --self-test: unknown id, typo'd id,
already-done task, missing state_hub_task_id, no measured spend, and a
status flip that produced no change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CB-RES-0003 measured 84 turns and $15.33 — the largest mechanical
category — spent prefixing commands with `cd` and
`export PATH="$HOME/.cargo/bin:$PATH"`. Both causes are now fixed once
instead of per-leaf.
tools/repo.py resolves the repo root from __file__ and cargo from PATH
then the standard rustup locations. Every tool imports ROOT from it, so
the repo path is stated once rather than redefined in four files —
single source of fact, the rule DFD earned in InnerLoop v1.2.
rule-coverage and dep-weight now call enter_root(), which is why their
relative paths did not need rewriting one by one.
The Makefile derives REPO from MAKEFILE_LIST and resolves CARGO the same
way, so `make -C <repo> <target>` works from any directory with no
prefix.
make env-test is the positive control, and is in `make all`: every tool
runs from / with PATH=/usr/bin:/bin. Without it this fix could regress
silently and invalidate T05's measurement — the whole point of the
control loop.
dep-weight's "cargo not on PATH" error is kept rather than deleted. It
should now be unreachable, and --self-test asserts cargo_bin() resolves
unaided; a control that never fires is cheaper than a regression.
loop-lint failed on repo.py on its first run — a reporting tool with a
positive control but no --self-test entry point. Second time the gate
has caught work from its own pass within the hour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review of where token-priced turns did work a deterministic tool could
do. Method: classify every turn in both transcripts by the tool calls it
made. The classifier is committed in tools/cb-cost.py and emitted by
`make cost-mix`, so the baseline is reproducible and the same command
can later falsify the predictions.
mech environment setup 84 turns $15.33
mech ad-hoc text patching 75 turns $13.86
git 37 turns $13.85
mech hub task status 25 turns $ 7.46
mech orientation / inspect 49 turns $ 6.87
hub other 32 turns $ 6.22
mech ad-hoc transcript 39 turns $ 4.56
mech workplan status edit 21 turns $ 4.06
MECHANICAL (dedup) 290 turns $51.26 = 38% of pass
Largest category is `cd` and `export PATH` -- pure friction, and
dep-weight.py already patched it at the leaf, which is evidence it was
noticed and fixed in the wrong place. Second is heredocs string-patching
markdown, which is also the mechanism behind duplicated-fact drift, the
error class InnerLoop v1.2 names and cannot gate.
Explicitly NOT automated: git (37 turns, $13.85) is mostly commit
message authorship -- the highest-output turns in the corpus and the
project's reasoning record. Automating it would save money and destroy
what makes corrections cheap.
CB-WP-0004 implements five candidates and predicts $33-41 recovery
(25-30%), below the 38% measured share on purpose: some inspection and
patching is genuinely exploratory.
The control loop is the deliverable, not a formality. T05 tests three
things and must report all: did mechanical turns disappear, did they
RELOCATE into prose, and did quality hold. If mechanical turns fall and
prose rises by as much, the saving is zero and that is the result to
publish.
Also a self-indictment worth recording: every hub update_task_status in
this project carried hand-typed token estimates, in a repo whose central
finding is that estimated token counts are worthless. T02 fixes it by
reading measured values from cb-cost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The price sheet had two defects of one shape -- a schema that could not
hold the fact it needed, the same criticism the cost survey levelled at
the State Hub.
CA-16 time-boxed rates are DATA. Sonnet's intro price lived in a
`# intro ...` comment and was invisible to the collector that
reads the file. Now promo_input/promo_output/promo_until,
applied per response at its own timestamp.
CA-17 the 90-day staleness rule was prose in MetricsAndScenarios 1a
that every M-D2-CST verdict silently inherited. Now `recorded`
+ `max_age_days` in the sheet, and a stale sheet ABORTS.
Both are exercised by make cost-test: the promo rate must apply before
2026-08-31 and lapse after, and a 102-day-old sheet must trip.
Applying CA-16 moved AC-1 from $93.32 to $93.15 -- the $0.17 CB-EV-0002
predicted, now collected rather than noted. That is a legitimate
retarget under T07's distinction: the instrument disproved the target,
and its output is in this commit. The number has now been stated five
times ($248.46, $92.21, $92.87, $93.32, $93.15), each correction from a
different mechanism.
Evidence tables regenerated from the tool rather than hand-patched,
per CA-15 -- which is the rule that exists because hand-typed tables
were the only thing the adversarial review found wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 8k/10k per-task token budget was never referenced or enforced and
T08 blew past it silently. Replaced with a budget that can actually
fire.
The design constraint is the interesting part: per-task cost needs the
commit that CLOSES the task, so a per-task budget is unavoidably
retrospective -- it can only report a breach after the money is spent,
which is the dead-policy failure again. What IS observable mid-task is
spend since the last commit, because the transcript is append-live. So
the budget binds on the open remainder.
CB-01 budget = USD since the last commit, via `make cost-budget`
CB-02 soft $10.00 (state progress, decide), hard $22.00 (stop)
Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50
$1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum
-- it would have fired exactly once on the calibration pass. Hard is ~2x
the observed max, a value never reached in 32 intervals, so reaching it
means the session is doing something the data has no example of.
Both thresholds are set ABOVE every observed value, so they bind on
future work rather than ratifying present work -- the distinction T07
is about.
Stated limit: it is a command, not a daemon. An agent that never runs it
gets no signal, which is the dead-policy failure one level up. Mitigated
only by being free to run and on the one command surface.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The task's original premise was wrong and is recorded rather than
deleted. It was written to prescribe one task per session; measurement
says the variable is context, not turn count.
Measured:
- compaction cut context 27x (542,991 -> 19,974) and cost/turn 3.1x;
the 202 turns after C1 cost less than half the 136 before it
- a turn costs $0.010 at 20k context and $0.270 at 540k
- break-even for a compaction is 2-11 turns, so: compact whenever
context exceeds ~300k and work remains
- a fresh session is NOT free -- cold start floors at ~51k and must
then re-read the artifacts a compact summary already holds (~66k).
Prefer compaction to continue work; prefer a fresh session when the
task changes, because then prior context is pure overhead.
cb-cost now emits SH-1..SH-3 so the targets come from the instrument
rather than from analysis, per InnerLoop v1.1. All three are UNMET
(mean context 232,982 vs 200,000; p90 492,042 vs 300,000; batching 7.8%
vs 20%) and are reported unmet rather than retargeted -- retargeting in
the commit that first measures is precisely what T07 exists to prevent.
Eighth error instance found while writing this: CB-WP-0001's claim that
"0 of 330 tool calls were batched" is wrong. 330 was the count of
single-call responses, not the total; 31 responses batched, covering 76
calls. It was carried into this workplan unverified. Trusted-arithmetic
class -- the one the T01 audit flagged as having no executable defence,
confirming that finding within hours of making it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.
The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:
CA-02 assumed usage is identical across the lines of one requestId.
True in the main transcript (206/206 groups, verified twice — by the
survey and by the adversarial reviewer). FALSE in the subagents/
tree, where output_tokens is a running count: one response reads
5, 5, 195 across its three lines. First-wins scored it at 5.
So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.
The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.
Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.
make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
