coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T07: separate a correction from a retarget with a mechanical test

Browse source 
A blanket "no retargeting in the measuring commit" rule would have been
wrong. CB-WP-0002 moved AC-1 three times in exactly that shape and every
move was correct -- each time a new instrument disproved the old figure.
Four legitimate corrections would have been forbidden to catch one bad
retarget.

The test is mechanical rather than a statement of intent:

  correction  the target moves and the implementation does not; legal in
              the same commit provided the instrument's output is there
  retarget    the same commit changes both the target and the code the
              target measures; requires an ADR stating why the new target
              binds on future work

Applied retroactively: AM-4a/AM-4b are UNRATIFIED. They were set after
seeing the measurement, in the commit that produced it, with the
implementation changing too -- a retarget by this test. make dep-weight
is currently enforcing a target no reviewed decision stands behind.
Recorded as an open item; ratifying or changing them is a maintainer
decision, not an implementer's.

Also: specs/InnerLoop.md split into InnerLoop.md (process) and
InnerLoopReference.md (rubric, template, rules, definition of done).
Not a stylistic choice -- `make loop-lint` failed on the commit that
pushed the file to 407 lines against its own ~400 limit. The gate added
this morning to make that rule executable caught its own author within
the hour, which is the cheapest possible demonstration that it works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:25:22 +02:00
parent ced573ba75
commit e21b9f4250
4 changed files with 189 additions and 130 deletions
Download patch file Download diff file Expand all files Collapse all files

9
history/260731-inner-loop-rule-audit.md
View file

@ -134,6 +134,15 @@ evidence files.
executed cannot be relied on to fire, so it must not be the only defence
for a class that matters.*
## Open items raised by this audit
- **AM-4a / AM-4b are unratified retargets.** Measured at 246,250 and set
at 250,000 in the same commit, by the implementer, with the
implementation changing in that commit too — a retarget under the test
added to InnerLoop §Step 4 by T07, not a correction. They must be
ratified by ADR or changed. Until then `make dep-weight` is enforcing a
target no reviewed decision stands behind.
## Class coverage — where the gaps are
| class | executable rules covering it | assessment |

167
specs/InnerLoop.md
View file

@ -172,6 +172,34 @@ double-counted transcript lines and priced a three-model session at one
model's rate. Neither error was of the harness-does-nothing class; both
sums ran over real data, and a positive control would have passed them.)*
**Retargeting: the instrument may move a target, the implementation may
not.** A metric's target changes in only two ways, and they are not
treated alike:
| | trigger | requirement |
|---|---|---|
| **corrected** | the instrument disproved the target — the target was computed by hand, or by an earlier tool with a defect | legitimate in the same commit, **provided the instrument's output is in that commit** |
| **retargeted** | the implementation missed the target and the target moves to accommodate it | requires an **ADR**: old target, the measurement, and why the new target binds on *future* work rather than merely passing present work |
The distinction is not the implementer's self-report of intent. It is
mechanical: **a correction is one where the target moves and the
implementation does not.** If the same commit changes both the target and
the code the target measures, it is a retarget and needs the ADR.
*(v1.1, from CB-WP-0002/0003: AM-4's targets were measured at 246,250 and
set at 250,000 in one commit by the implementer after seeing the number —
the structure this rule exists to stop. But CB-WP-0002 then moved AC-1
three times, correctly, each time because a new instrument disproved the
old figure ($92.21 → $92.87 → $93.32 → $93.15). A blanket prohibition
would have forbidden four legitimate corrections to catch one bad
retarget.)*
**Applied retroactively:** AM-4a and AM-4b are **unratified** until an ADR
is written or they are changed. They were set by the implementer after
seeing the measurement, in the commit that produced it, and the
implementation changed in that same commit — a retarget by the test above.
Tracked as an open item in `history/260731-inner-loop-rule-audit.md`.
**A metric is checked against the contracts in its own spec.** If a
contract makes a target unreachable, one of the two is wrong and the
conflict is resolved when it is noticed, not at the acceptance run.
@ -244,136 +272,17 @@ state the claim you will defend, and the claim you are not making.
---
## The four-dimension rubric
Every survey, ADR, and acceptance table is organized by these dimensions:
| Dimension | Question | Example measurable proxies |
|---|---|---|
| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session |
| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task |
| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead |
| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility |
Scoring is always **relative to the step-1 baseline**, never absolute:
`better / parity / worse / unmeasured` per proxy, with the number attached.
`unmeasured` is legal in a survey, illegal in an evidence file.
---
## Survey template (research/CB-RES-NNNN-<slug>.md)
## Reference material
```markdown
# CB-RES-NNNN: <capability>
capability: <canonical.capability.id>
status: draft | approved
The four-dimension rubric, the survey template, the implementation rules
each pass earned, the agentic-efficiency requirements, and the
definition of done live in
**[InnerLoopReference.md](InnerLoopReference.md)**. They are normative;
they are separated only so both files load whole.
## Candidates
Per candidate: origin, license, maturity, adoption; data model; mutation
mechanism; determinism/replay story; relevant performance (measured if
runnable locally, cited with source otherwise).
## Baselines (benchmark-to-beat)
| Dimension | Baseline holder | Metric | Value | Provenance |
(one row minimum per dimension; provenance = measured / cited / estimated)
## Verdict
Which candidate leads per dimension; what none of them do well
(the surpass opportunity); risks in the baselines themselves.
```
---
## Implementation rules the first pass earned
These are cheap, and each exists because its absence cost something in
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
1. **No silently-ignored input.** A field that is parsed and then unused
is a defect, not a stub. Inputs are honoured or rejected with an
error — never dropped. *(A scenario `setup.patch` was parsed and
discarded; every scenario using it would have tested the wrong
initial state while passing.)*
2. **Decisions get commands, not defaults.** A rule that requires a
participant's choice is implemented as a command carrying that
choice. Until it is, **nothing claims coverage of it** — no tag, no
scenario, no acceptance row. Inventing a default to make a rule
"done" is the failure this prevents.
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
not evidence. Any scaffold path no test reaches is marked as
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
that would panic on any state holding a relation.)*
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
against `covers:` lists proves no rule is unclaimed and no claimed
rule is invented. It does **not** prove a scenario exercises what it
names. Wherever such a number is reported, that limit is reported
with it.
---
## Agentic-efficiency requirements
The loop exists to be driven by agents. Therefore:
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
split before exceeding, link with relative paths.
2. **Structured over prose** *(guidance, not a requirement — nothing can
check it)* — tables and fenced blocks for anything a later step must
parse (baselines, acceptance metrics, evidence rows).
3. **One command surface** — all checks runnable through repo-root
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
in one place), each supporting deterministic, greppable output.
4. **Self-contained tasks** — a workplan task names its input artifacts and
output artifacts; a fresh session must be able to execute it from the
task text plus linked files alone.
5. **Evidence or it didn't happen** — claims of "better" live in committed
evidence files with numbers, never only in commit messages or chat.
6. ~~**Token discipline** — per the global budget policy, a loop iteration
that exceeds its budget without measurable progress is stopped and
decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task
token budget was never referenced or enforced, and CB-WP-0001 T08
exceeded it by orders of magnitude with no signal. It implies a control
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
this is documentation of a gap, not a rule.
6a. **Live cost budget** *(replaces the above)* — spend since the last
commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`.
Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90
$9.36, max $10.80), so both thresholds bind on future work rather than
ratifying past work. Contract: [CostAccounting.md](CostAccounting.md)
§7 (CB-01, CB-02). It fires on the open remainder rather than per task
because per-task cost needs the commit that closes the task, and a
budget that can only report after the money is spent is the dead policy
this replaces.
**Enforcement status.** Rules above that a command can check are enforced
by `make loop-lint`; the full classification of every InnerLoop rule as
executable / checkable / decorative, with the failure class each catches,
is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance*
or *dead* say so where they appear, so a reader can tell a requirement
from a preference without consulting the audit.
---
## Definition of done — one loop pass
A capability has completed the loop when all of the following are committed:
- [ ] research/CB-RES-NNNN with approved status and full baseline table
- [ ] decisions/ADR-NNNN with per-dimension expected advantage
- [ ] specs/<Capability>.md with acceptance-metrics table
- [ ] passing scenarios covering every numbered spec rule
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
- [ ] every reported number produced by a harness with a positive
control; any metric that could not be instrumented is recorded as
uncomputable rather than estimated
- [ ] every unmet metric reported as unmet, with attribution and the
options for resolving it — a missed target is an output of the
loop, not a reason to move the target quietly
- [ ] **cost recorded**: `make cost` run for the pass, its composition
(not only its total) in the evidence file, and the per-task figures
pushed to the hub. M-D2-CST is no longer allowed to be
`uncomputable` — the instrument exists
([CostAccounting.md](CostAccounting.md))
- [ ] retrospective note (may be one paragraph appended to the evidence
file): what the loop itself should change
*(Split 2026-07-31: this file reached 407 lines against its own ~400-line
loadability rule, and `make loop-lint` — added the same day to make that
rule executable — failed on the commit that pushed it over. The rule
caught its own author within an hour of being written.)*

141
specs/InnerLoopReference.md Normal file
View file

@ -0,0 +1,141 @@
# The Inner Loop — Reference
Normative companion to **[InnerLoop.md](InnerLoop.md)**, which holds the
five steps, the tiers, and the chaos roll. Split from it on 2026-07-31 so
both files stay under the loop's own whole-file loadability limit.
Everything here binds exactly as the process document does.
## The four-dimension rubric
Every survey, ADR, and acceptance table is organized by these dimensions:
| Dimension | Question | Example measurable proxies |
|---|---|---|
| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session |
| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task |
| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead |
| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility |
Scoring is always **relative to the step-1 baseline**, never absolute:
`better / parity / worse / unmeasured` per proxy, with the number attached.
`unmeasured` is legal in a survey, illegal in an evidence file.
---
## Survey template (research/CB-RES-NNNN-<slug>.md)
```markdown
# CB-RES-NNNN: <capability>
capability: <canonical.capability.id>
status: draft | approved
## Candidates
Per candidate: origin, license, maturity, adoption; data model; mutation
mechanism; determinism/replay story; relevant performance (measured if
runnable locally, cited with source otherwise).
## Baselines (benchmark-to-beat)
| Dimension | Baseline holder | Metric | Value | Provenance |
(one row minimum per dimension; provenance = measured / cited / estimated)
## Verdict
Which candidate leads per dimension; what none of them do well
(the surpass opportunity); risks in the baselines themselves.
```
---
## Implementation rules the first pass earned
These are cheap, and each exists because its absence cost something in
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
1. **No silently-ignored input.** A field that is parsed and then unused
is a defect, not a stub. Inputs are honoured or rejected with an
error — never dropped. *(A scenario `setup.patch` was parsed and
discarded; every scenario using it would have tested the wrong
initial state while passing.)*
2. **Decisions get commands, not defaults.** A rule that requires a
participant's choice is implemented as a command carrying that
choice. Until it is, **nothing claims coverage of it** — no tag, no
scenario, no acceptance row. Inventing a default to make a rule
"done" is the failure this prevents.
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
not evidence. Any scaffold path no test reaches is marked as
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
that would panic on any state holding a relation.)*
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
against `covers:` lists proves no rule is unclaimed and no claimed
rule is invented. It does **not** prove a scenario exercises what it
names. Wherever such a number is reported, that limit is reported
with it.
---
## Agentic-efficiency requirements
The loop exists to be driven by agents. Therefore:
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
split before exceeding, link with relative paths.
2. **Structured over prose** *(guidance, not a requirement — nothing can
check it)* — tables and fenced blocks for anything a later step must
parse (baselines, acceptance metrics, evidence rows).
3. **One command surface** — all checks runnable through repo-root
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
in one place), each supporting deterministic, greppable output.
4. **Self-contained tasks** — a workplan task names its input artifacts and
output artifacts; a fresh session must be able to execute it from the
task text plus linked files alone.
5. **Evidence or it didn't happen** — claims of "better" live in committed
evidence files with numbers, never only in commit messages or chat.
6. ~~**Token discipline** — per the global budget policy, a loop iteration
that exceeds its budget without measurable progress is stopped and
decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task
token budget was never referenced or enforced, and CB-WP-0001 T08
exceeded it by orders of magnitude with no signal. It implies a control
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
this is documentation of a gap, not a rule.
6a. **Live cost budget** *(replaces the above)* — spend since the last
commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`.
Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90
$9.36, max $10.80), so both thresholds bind on future work rather than
ratifying past work. Contract: [CostAccounting.md](CostAccounting.md)
§7 (CB-01, CB-02). It fires on the open remainder rather than per task
because per-task cost needs the commit that closes the task, and a
budget that can only report after the money is spent is the dead policy
this replaces.
**Enforcement status.** Rules above that a command can check are enforced
by `make loop-lint`; the full classification of every InnerLoop rule as
executable / checkable / decorative, with the failure class each catches,
is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance*
or *dead* say so where they appear, so a reader can tell a requirement
from a preference without consulting the audit.
---
## Definition of done — one loop pass
A capability has completed the loop when all of the following are committed:
- [ ] research/CB-RES-NNNN with approved status and full baseline table
- [ ] decisions/ADR-NNNN with per-dimension expected advantage
- [ ] specs/<Capability>.md with acceptance-metrics table
- [ ] passing scenarios covering every numbered spec rule
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
- [ ] every reported number produced by a harness with a positive
control; any metric that could not be instrumented is recorded as
uncomputable rather than estimated
- [ ] every unmet metric reported as unmet, with attribution and the
options for resolving it — a missed target is an output of the
loop, not a reason to move the target quietly
- [ ] **cost recorded**: `make cost` run for the pass, its composition
(not only its total) in the evidence file, and the per-task figures
pushed to the hub. M-D2-CST is no longer allowed to be
`uncomputable` — the instrument exists
([CostAccounting.md](CostAccounting.md))
- [ ] retrospective note (may be one paragraph appended to the evidence
file): what the loop itself should change

2
workplans/CB-WP-0003-loop-hardening.md
View file

@ -253,7 +253,7 @@ to reject.
```task
id: CB-WP-0003-T07
status: todo
status: done
priority: medium
state_hub_task_id: "2f702821-7e64-4981-9885-82fa07f638aa"
```