diff --git a/history/260731-inner-loop-rule-audit.md b/history/260731-inner-loop-rule-audit.md index 96203dd..13999eb 100644 --- a/history/260731-inner-loop-rule-audit.md +++ b/history/260731-inner-loop-rule-audit.md @@ -134,6 +134,15 @@ evidence files. executed cannot be relied on to fire, so it must not be the only defence for a class that matters.* +## Open items raised by this audit + +- **AM-4a / AM-4b are unratified retargets.** Measured at 246,250 and set + at 250,000 in the same commit, by the implementer, with the + implementation changing in that commit too — a retarget under the test + added to InnerLoop §Step 4 by T07, not a correction. They must be + ratified by ADR or changed. Until then `make dep-weight` is enforcing a + target no reviewed decision stands behind. + ## Class coverage — where the gaps are | class | executable rules covering it | assessment | diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md index 38c14a8..de5d34f 100644 --- a/specs/InnerLoop.md +++ b/specs/InnerLoop.md @@ -172,6 +172,34 @@ double-counted transcript lines and priced a three-model session at one model's rate. Neither error was of the harness-does-nothing class; both sums ran over real data, and a positive control would have passed them.)* +**Retargeting: the instrument may move a target, the implementation may +not.** A metric's target changes in only two ways, and they are not +treated alike: + +| | trigger | requirement | +|---|---|---| +| **corrected** | the instrument disproved the target — the target was computed by hand, or by an earlier tool with a defect | legitimate in the same commit, **provided the instrument's output is in that commit** | +| **retargeted** | the implementation missed the target and the target moves to accommodate it | requires an **ADR**: old target, the measurement, and why the new target binds on *future* work rather than merely passing present work | + +The distinction is not the implementer's self-report of intent. It is +mechanical: **a correction is one where the target moves and the +implementation does not.** If the same commit changes both the target and +the code the target measures, it is a retarget and needs the ADR. + +*(v1.1, from CB-WP-0002/0003: AM-4's targets were measured at 246,250 and +set at 250,000 in one commit by the implementer after seeing the number — +the structure this rule exists to stop. But CB-WP-0002 then moved AC-1 +three times, correctly, each time because a new instrument disproved the +old figure ($92.21 → $92.87 → $93.32 → $93.15). A blanket prohibition +would have forbidden four legitimate corrections to catch one bad +retarget.)* + +**Applied retroactively:** AM-4a and AM-4b are **unratified** until an ADR +is written or they are changed. They were set by the implementer after +seeing the measurement, in the commit that produced it, and the +implementation changed in that same commit — a retarget by the test above. +Tracked as an open item in `history/260731-inner-loop-rule-audit.md`. + **A metric is checked against the contracts in its own spec.** If a contract makes a target unreachable, one of the two is wrong and the conflict is resolved when it is noticed, not at the acceptance run. @@ -244,136 +272,17 @@ state the claim you will defend, and the claim you are not making. --- -## The four-dimension rubric - -Every survey, ADR, and acceptance table is organized by these dimensions: - -| Dimension | Question | Example measurable proxies | -|---|---|---| -| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session | -| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task | -| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead | -| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility | - -Scoring is always **relative to the step-1 baseline**, never absolute: -`better / parity / worse / unmeasured` per proxy, with the number attached. -`unmeasured` is legal in a survey, illegal in an evidence file. - --- -## Survey template (research/CB-RES-NNNN-.md) +## Reference material -```markdown -# CB-RES-NNNN: -capability: -status: draft | approved +The four-dimension rubric, the survey template, the implementation rules +each pass earned, the agentic-efficiency requirements, and the +definition of done live in +**[InnerLoopReference.md](InnerLoopReference.md)**. They are normative; +they are separated only so both files load whole. -## Candidates -Per candidate: origin, license, maturity, adoption; data model; mutation -mechanism; determinism/replay story; relevant performance (measured if -runnable locally, cited with source otherwise). - -## Baselines (benchmark-to-beat) -| Dimension | Baseline holder | Metric | Value | Provenance | -(one row minimum per dimension; provenance = measured / cited / estimated) - -## Verdict -Which candidate leads per dimension; what none of them do well -(the surpass opportunity); risks in the baselines themselves. -``` - ---- - -## Implementation rules the first pass earned - -These are cheap, and each exists because its absence cost something in -CB-WP-0001. See `history/260731-inner-loop-retrospective.md`. - -1. **No silently-ignored input.** A field that is parsed and then unused - is a defect, not a stub. Inputs are honoured or rejected with an - error — never dropped. *(A scenario `setup.patch` was parsed and - discarded; every scenario using it would have tested the wrong - initial state while passing.)* -2. **Decisions get commands, not defaults.** A rule that requires a - participant's choice is implemented as a command carrying that - choice. Until it is, **nothing claims coverage of it** — no tag, no - scenario, no acceptance row. Inventing a default to make a rule - "done" is the failure this prevents. -3. **Scaffolds are exercised or marked.** A scaffold's green gates are - not evidence. Any scaffold path no test reaches is marked as - unexercised. *(A compiling, fully-green scaffold shipped a state-hash - that would panic on any state holding a relation.)* -4. **Coverage gates that count tags say so.** A gate comparing rule IDs - against `covers:` lists proves no rule is unclaimed and no claimed - rule is invented. It does **not** prove a scenario exercises what it - names. Wherever such a number is reported, that limit is reported - with it. - ---- - -## Agentic-efficiency requirements - -The loop exists to be driven by agents. Therefore: - -1. **Whole-file loadability** — every loop artifact stays under ~400 lines; - split before exceeding, link with relative paths. -2. **Structured over prose** *(guidance, not a requirement — nothing can - check it)* — tables and fenced blocks for anything a later step must - parse (baselines, acceptance metrics, evidence rows). -3. **One command surface** — all checks runnable through repo-root - commands (eventually `cb *`; until then, `make`/`cargo` aliases declared - in one place), each supporting deterministic, greppable output. -4. **Self-contained tasks** — a workplan task names its input artifacts and - output artifacts; a fresh session must be able to execute it from the - task text plus linked files alone. -5. **Evidence or it didn't happen** — claims of "better" live in committed - evidence files with numbers, never only in commit messages or chat. -6. ~~**Token discipline** — per the global budget policy, a loop iteration - that exceeds its budget without measurable progress is stopped and - decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task - token budget was never referenced or enforced, and CB-WP-0001 T08 - exceeded it by orders of magnitude with no signal. It implies a control - that does not exist. Replacement in USD is CB-WP-0003 T05; until then - this is documentation of a gap, not a rule. - -6a. **Live cost budget** *(replaces the above)* — spend since the last - commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`. - Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90 - $9.36, max $10.80), so both thresholds bind on future work rather than - ratifying past work. Contract: [CostAccounting.md](CostAccounting.md) - §7 (CB-01, CB-02). It fires on the open remainder rather than per task - because per-task cost needs the commit that closes the task, and a - budget that can only report after the money is spent is the dead policy - this replaces. - -**Enforcement status.** Rules above that a command can check are enforced -by `make loop-lint`; the full classification of every InnerLoop rule as -executable / checkable / decorative, with the failure class each catches, -is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance* -or *dead* say so where they appear, so a reader can tell a requirement -from a preference without consulting the audit. - ---- - -## Definition of done — one loop pass - -A capability has completed the loop when all of the following are committed: - -- [ ] research/CB-RES-NNNN with approved status and full baseline table -- [ ] decisions/ADR-NNNN with per-dimension expected advantage -- [ ] specs/.md with acceptance-metrics table -- [ ] passing scenarios covering every numbered spec rule -- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured` -- [ ] every reported number produced by a harness with a positive - control; any metric that could not be instrumented is recorded as - uncomputable rather than estimated -- [ ] every unmet metric reported as unmet, with attribution and the - options for resolving it — a missed target is an output of the - loop, not a reason to move the target quietly -- [ ] **cost recorded**: `make cost` run for the pass, its composition - (not only its total) in the evidence file, and the per-task figures - pushed to the hub. M-D2-CST is no longer allowed to be - `uncomputable` — the instrument exists - ([CostAccounting.md](CostAccounting.md)) -- [ ] retrospective note (may be one paragraph appended to the evidence - file): what the loop itself should change +*(Split 2026-07-31: this file reached 407 lines against its own ~400-line +loadability rule, and `make loop-lint` — added the same day to make that +rule executable — failed on the commit that pushed it over. The rule +caught its own author within an hour of being written.)* diff --git a/specs/InnerLoopReference.md b/specs/InnerLoopReference.md new file mode 100644 index 0000000..06af1fd --- /dev/null +++ b/specs/InnerLoopReference.md @@ -0,0 +1,141 @@ +# The Inner Loop — Reference + +Normative companion to **[InnerLoop.md](InnerLoop.md)**, which holds the +five steps, the tiers, and the chaos roll. Split from it on 2026-07-31 so +both files stay under the loop's own whole-file loadability limit. + +Everything here binds exactly as the process document does. + +## The four-dimension rubric + +Every survey, ADR, and acceptance table is organized by these dimensions: + +| Dimension | Question | Example measurable proxies | +|---|---|---| +| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session | +| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task | +| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead | +| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility | + +Scoring is always **relative to the step-1 baseline**, never absolute: +`better / parity / worse / unmeasured` per proxy, with the number attached. +`unmeasured` is legal in a survey, illegal in an evidence file. + +--- + +## Survey template (research/CB-RES-NNNN-.md) + +```markdown +# CB-RES-NNNN: +capability: +status: draft | approved + +## Candidates +Per candidate: origin, license, maturity, adoption; data model; mutation +mechanism; determinism/replay story; relevant performance (measured if +runnable locally, cited with source otherwise). + +## Baselines (benchmark-to-beat) +| Dimension | Baseline holder | Metric | Value | Provenance | +(one row minimum per dimension; provenance = measured / cited / estimated) + +## Verdict +Which candidate leads per dimension; what none of them do well +(the surpass opportunity); risks in the baselines themselves. +``` + +--- + +## Implementation rules the first pass earned + +These are cheap, and each exists because its absence cost something in +CB-WP-0001. See `history/260731-inner-loop-retrospective.md`. + +1. **No silently-ignored input.** A field that is parsed and then unused + is a defect, not a stub. Inputs are honoured or rejected with an + error — never dropped. *(A scenario `setup.patch` was parsed and + discarded; every scenario using it would have tested the wrong + initial state while passing.)* +2. **Decisions get commands, not defaults.** A rule that requires a + participant's choice is implemented as a command carrying that + choice. Until it is, **nothing claims coverage of it** — no tag, no + scenario, no acceptance row. Inventing a default to make a rule + "done" is the failure this prevents. +3. **Scaffolds are exercised or marked.** A scaffold's green gates are + not evidence. Any scaffold path no test reaches is marked as + unexercised. *(A compiling, fully-green scaffold shipped a state-hash + that would panic on any state holding a relation.)* +4. **Coverage gates that count tags say so.** A gate comparing rule IDs + against `covers:` lists proves no rule is unclaimed and no claimed + rule is invented. It does **not** prove a scenario exercises what it + names. Wherever such a number is reported, that limit is reported + with it. + +--- + +## Agentic-efficiency requirements + +The loop exists to be driven by agents. Therefore: + +1. **Whole-file loadability** — every loop artifact stays under ~400 lines; + split before exceeding, link with relative paths. +2. **Structured over prose** *(guidance, not a requirement — nothing can + check it)* — tables and fenced blocks for anything a later step must + parse (baselines, acceptance metrics, evidence rows). +3. **One command surface** — all checks runnable through repo-root + commands (eventually `cb *`; until then, `make`/`cargo` aliases declared + in one place), each supporting deterministic, greppable output. +4. **Self-contained tasks** — a workplan task names its input artifacts and + output artifacts; a fresh session must be able to execute it from the + task text plus linked files alone. +5. **Evidence or it didn't happen** — claims of "better" live in committed + evidence files with numbers, never only in commit messages or chat. +6. ~~**Token discipline** — per the global budget policy, a loop iteration + that exceeds its budget without measurable progress is stopped and + decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task + token budget was never referenced or enforced, and CB-WP-0001 T08 + exceeded it by orders of magnitude with no signal. It implies a control + that does not exist. Replacement in USD is CB-WP-0003 T05; until then + this is documentation of a gap, not a rule. + +6a. **Live cost budget** *(replaces the above)* — spend since the last + commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`. + Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90 + $9.36, max $10.80), so both thresholds bind on future work rather than + ratifying past work. Contract: [CostAccounting.md](CostAccounting.md) + §7 (CB-01, CB-02). It fires on the open remainder rather than per task + because per-task cost needs the commit that closes the task, and a + budget that can only report after the money is spent is the dead policy + this replaces. + +**Enforcement status.** Rules above that a command can check are enforced +by `make loop-lint`; the full classification of every InnerLoop rule as +executable / checkable / decorative, with the failure class each catches, +is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance* +or *dead* say so where they appear, so a reader can tell a requirement +from a preference without consulting the audit. + +--- + +## Definition of done — one loop pass + +A capability has completed the loop when all of the following are committed: + +- [ ] research/CB-RES-NNNN with approved status and full baseline table +- [ ] decisions/ADR-NNNN with per-dimension expected advantage +- [ ] specs/.md with acceptance-metrics table +- [ ] passing scenarios covering every numbered spec rule +- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured` +- [ ] every reported number produced by a harness with a positive + control; any metric that could not be instrumented is recorded as + uncomputable rather than estimated +- [ ] every unmet metric reported as unmet, with attribution and the + options for resolving it — a missed target is an output of the + loop, not a reason to move the target quietly +- [ ] **cost recorded**: `make cost` run for the pass, its composition + (not only its total) in the evidence file, and the per-task figures + pushed to the hub. M-D2-CST is no longer allowed to be + `uncomputable` — the instrument exists + ([CostAccounting.md](CostAccounting.md)) +- [ ] retrospective note (may be one paragraph appended to the evidence + file): what the loop itself should change diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md index c0b41ce..a4ac8c8 100644 --- a/workplans/CB-WP-0003-loop-hardening.md +++ b/workplans/CB-WP-0003-loop-hardening.md @@ -253,7 +253,7 @@ to reject. ```task id: CB-WP-0003-T07 -status: todo +status: done priority: medium state_hub_task_id: "2f702821-7e64-4981-9885-82fa07f638aa" ```