tegwick
e21b9f4250
A blanket "no retargeting in the measuring commit" rule would have been
wrong. CB-WP-0002 moved AC-1 three times in exactly that shape and every
move was correct -- each time a new instrument disproved the old figure.
Four legitimate corrections would have been forbidden to catch one bad
retarget.
The test is mechanical rather than a statement of intent:
correction the target moves and the implementation does not; legal in
the same commit provided the instrument's output is there
retarget the same commit changes both the target and the code the
target measures; requires an ADR stating why the new target
binds on future work
Applied retroactively: AM-4a/AM-4b are UNRATIFIED. They were set after
seeing the measurement, in the commit that produced it, with the
implementation changing too -- a retarget by this test. make dep-weight
is currently enforcing a target no reviewed decision stands behind.
Recorded as an open item; ratifying or changing them is a maintainer
decision, not an implementer's.
Also: specs/InnerLoop.md split into InnerLoop.md (process) and
InnerLoopReference.md (rubric, template, rules, definition of done).
Not a stylistic choice -- `make loop-lint` failed on the commit that
pushed the file to 407 lines against its own ~400 limit. The gate added
this morning to make that rule executable caught its own author within
the hour, which is the cheapest possible demonstration that it works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
141 lines
7.1 KiB
Markdown
141 lines
7.1 KiB
Markdown
# The Inner Loop — Reference
|
|
|
|
Normative companion to **[InnerLoop.md](InnerLoop.md)**, which holds the
|
|
five steps, the tiers, and the chaos roll. Split from it on 2026-07-31 so
|
|
both files stay under the loop's own whole-file loadability limit.
|
|
|
|
Everything here binds exactly as the process document does.
|
|
|
|
## The four-dimension rubric
|
|
|
|
Every survey, ADR, and acceptance table is organized by these dimensions:
|
|
|
|
| Dimension | Question | Example measurable proxies |
|
|
|---|---|---|
|
|
| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session |
|
|
| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task |
|
|
| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead |
|
|
| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility |
|
|
|
|
Scoring is always **relative to the step-1 baseline**, never absolute:
|
|
`better / parity / worse / unmeasured` per proxy, with the number attached.
|
|
`unmeasured` is legal in a survey, illegal in an evidence file.
|
|
|
|
---
|
|
|
|
## Survey template (research/CB-RES-NNNN-<slug>.md)
|
|
|
|
```markdown
|
|
# CB-RES-NNNN: <capability>
|
|
capability: <canonical.capability.id>
|
|
status: draft | approved
|
|
|
|
## Candidates
|
|
Per candidate: origin, license, maturity, adoption; data model; mutation
|
|
mechanism; determinism/replay story; relevant performance (measured if
|
|
runnable locally, cited with source otherwise).
|
|
|
|
## Baselines (benchmark-to-beat)
|
|
| Dimension | Baseline holder | Metric | Value | Provenance |
|
|
(one row minimum per dimension; provenance = measured / cited / estimated)
|
|
|
|
## Verdict
|
|
Which candidate leads per dimension; what none of them do well
|
|
(the surpass opportunity); risks in the baselines themselves.
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation rules the first pass earned
|
|
|
|
These are cheap, and each exists because its absence cost something in
|
|
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
|
|
|
|
1. **No silently-ignored input.** A field that is parsed and then unused
|
|
is a defect, not a stub. Inputs are honoured or rejected with an
|
|
error — never dropped. *(A scenario `setup.patch` was parsed and
|
|
discarded; every scenario using it would have tested the wrong
|
|
initial state while passing.)*
|
|
2. **Decisions get commands, not defaults.** A rule that requires a
|
|
participant's choice is implemented as a command carrying that
|
|
choice. Until it is, **nothing claims coverage of it** — no tag, no
|
|
scenario, no acceptance row. Inventing a default to make a rule
|
|
"done" is the failure this prevents.
|
|
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
|
|
not evidence. Any scaffold path no test reaches is marked as
|
|
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
|
|
that would panic on any state holding a relation.)*
|
|
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
|
|
against `covers:` lists proves no rule is unclaimed and no claimed
|
|
rule is invented. It does **not** prove a scenario exercises what it
|
|
names. Wherever such a number is reported, that limit is reported
|
|
with it.
|
|
|
|
---
|
|
|
|
## Agentic-efficiency requirements
|
|
|
|
The loop exists to be driven by agents. Therefore:
|
|
|
|
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
|
|
split before exceeding, link with relative paths.
|
|
2. **Structured over prose** *(guidance, not a requirement — nothing can
|
|
check it)* — tables and fenced blocks for anything a later step must
|
|
parse (baselines, acceptance metrics, evidence rows).
|
|
3. **One command surface** — all checks runnable through repo-root
|
|
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
|
|
in one place), each supporting deterministic, greppable output.
|
|
4. **Self-contained tasks** — a workplan task names its input artifacts and
|
|
output artifacts; a fresh session must be able to execute it from the
|
|
task text plus linked files alone.
|
|
5. **Evidence or it didn't happen** — claims of "better" live in committed
|
|
evidence files with numbers, never only in commit messages or chat.
|
|
6. ~~**Token discipline** — per the global budget policy, a loop iteration
|
|
that exceeds its budget without measurable progress is stopped and
|
|
decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task
|
|
token budget was never referenced or enforced, and CB-WP-0001 T08
|
|
exceeded it by orders of magnitude with no signal. It implies a control
|
|
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
|
|
this is documentation of a gap, not a rule.
|
|
|
|
6a. **Live cost budget** *(replaces the above)* — spend since the last
|
|
commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`.
|
|
Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90
|
|
$9.36, max $10.80), so both thresholds bind on future work rather than
|
|
ratifying past work. Contract: [CostAccounting.md](CostAccounting.md)
|
|
§7 (CB-01, CB-02). It fires on the open remainder rather than per task
|
|
because per-task cost needs the commit that closes the task, and a
|
|
budget that can only report after the money is spent is the dead policy
|
|
this replaces.
|
|
|
|
**Enforcement status.** Rules above that a command can check are enforced
|
|
by `make loop-lint`; the full classification of every InnerLoop rule as
|
|
executable / checkable / decorative, with the failure class each catches,
|
|
is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance*
|
|
or *dead* say so where they appear, so a reader can tell a requirement
|
|
from a preference without consulting the audit.
|
|
|
|
---
|
|
|
|
## Definition of done — one loop pass
|
|
|
|
A capability has completed the loop when all of the following are committed:
|
|
|
|
- [ ] research/CB-RES-NNNN with approved status and full baseline table
|
|
- [ ] decisions/ADR-NNNN with per-dimension expected advantage
|
|
- [ ] specs/<Capability>.md with acceptance-metrics table
|
|
- [ ] passing scenarios covering every numbered spec rule
|
|
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
|
|
- [ ] every reported number produced by a harness with a positive
|
|
control; any metric that could not be instrumented is recorded as
|
|
uncomputable rather than estimated
|
|
- [ ] every unmet metric reported as unmet, with attribution and the
|
|
options for resolving it — a missed target is an output of the
|
|
loop, not a reason to move the target quietly
|
|
- [ ] **cost recorded**: `make cost` run for the pass, its composition
|
|
(not only its total) in the evidence file, and the per-task figures
|
|
pushed to the hub. M-D2-CST is no longer allowed to be
|
|
`uncomputable` — the instrument exists
|
|
([CostAccounting.md](CostAccounting.md))
|
|
- [ ] retrospective note (may be one paragraph appended to the evidence
|
|
file): what the loop itself should change
|