tegwick
fed422a3a3
41 rules classified executable / checkable / decorative, each tagged with
the failure class it catches. Counts: 11 executable, 22 checkable, 4
decorative (one of them dead policy).
Audit: history/260731-inner-loop-rule-audit.md
New tools/loop-lint.py makes 7 rules executable (tier declared, chaos
roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file
loadability, reporting tools expose --self-test). It found three real
violations on its first run, none previously visible:
- specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit
the loop has stated since v0.2 and never measured. Split at its own
section boundaries into Blueprint (1-8) + Runtime (9-15).
- tools/dep-weight.py and tools/rule-coverage.py had positive-control
logic and no --self-test, so nothing verified the control worked.
Adding rule-coverage's self-test exposed a latent instance of the exact
class this workplan is about: if the spec regex stopped matching, rules
was empty, missing was empty, and the tool exited 0 reporting "0/0" --
a silent pass, in the tool that reports our headline AM-1 number. Both
tools now assert they found something before reporting.
Two demotions applied in the spec rather than left implicit: "structured
over prose" is marked guidance (nothing can check it), and the 8k/10k
token budget is struck through and marked DEAD POLICY pointing at T05.
The audit's uncomfortable finding: rule 13 (re-derive inherited numbers)
has no mechanical form, is deliberately left decorative, and caught the
LARGEST error in CB-WP-0002. That is a counter-example to this
workplan's own hypothesis. "A rule that cannot be executed is not a
rule" is wrong as stated; the defensible version is that such a rule
cannot be relied on to fire, so it must not be the only defence for a
class that matters.
Class coverage: harness-does-nothing has five executable rules;
trusted-arithmetic has ZERO and produced the largest single error.
make loop-lint and make self-tests wired into `make all` and CI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
350 lines
17 KiB
Markdown
350 lines
17 KiB
Markdown
# The Inner Loop — Assimilate and Surpass
|
||
|
||
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
|
||
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||
target; inherited numbers are re-derived before use; every reporting tool
|
||
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||
`history/260731-cost-accounting-retrospective.md`.
|
||
|
||
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||
validity (the positive control), metric feasibility and instrument naming,
|
||
four implementation rules the pass earned, and the requirement that
|
||
evidence state what it does not support. Rationale and the failures behind
|
||
each: `history/260731-inner-loop-retrospective.md`.
|
||
|
||
Normative process for building every Clay-Borg capability. Referenced by
|
||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||
every artifact it produces must be small enough to load whole, structured
|
||
enough to act on without interpretation, and falsifiable enough that an
|
||
agent can judge its own work without a human in the iteration.
|
||
|
||
---
|
||
|
||
## The five steps
|
||
|
||
```text
|
||
1 RESEARCH → research/CB-RES-NNNN-<slug>.md (survey, baselines)
|
||
2 APPROVE → decision recorded in the ADR (gate: survey complete?)
|
||
3 DECIDE → decisions/ADR-NNNN-<slug>.md (assimilate/reimplement/hybrid)
|
||
4 SPECIFY → specs/<Capability>.md (contracts + acceptance metrics)
|
||
5 CODE LOOP → code + scenarios + benchmarks (iterate until metrics beat baseline)
|
||
```
|
||
|
||
**Hard gate: no implementation code for a capability exists before its ADR
|
||
(step 3) is committed.**
|
||
|
||
## Loop tiers and the chaos roll
|
||
|
||
Every work packet declares a tier before work starts. The tier sets how
|
||
heavy steps 1–3 are; steps 4–5 (spec with metrics, code loop with evidence)
|
||
are never skipped for code-producing work.
|
||
|
||
| Tier | Weight of steps 1–3 | Structural trigger (forces at least this tier) |
|
||
|---|---|---|
|
||
| **L** | Full: separate survey, adversarial review, ADR | Creates a new capability port, or is named a high-leverage pass by the maintainer |
|
||
| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, or adds/updates an external dependency |
|
||
| **S** | One provenance paragraph in the commit message | Everything else (utilities, fixes, refactors inside a boundary) |
|
||
|
||
**The chaos roll.** After deriving the structural tier, roll d10
|
||
(`shuf -i 1-10 -n 1`). On a **10**, the tier is instead picked uniformly at
|
||
random (`shuf -e S M L -n 1`), overriding the structural derivation — up or
|
||
down. Both rolls are recorded in the tier declaration
|
||
(`tier: M (structural L, chaos 10→M)`). **Record the roll every time,
|
||
including when it changes nothing** (`tier: L (structural L, chaos 4)`),
|
||
so a mechanism that never fires is visible rather than assumed. Purpose:
|
||
an occasional random
|
||
reweighting keeps the classification honest — arguing everything into S
|
||
stops paying off when audits can compare argued tiers against the random
|
||
sample — and occasionally forces a deep look at something "obviously
|
||
trivial", which is where local optima hide.
|
||
|
||
Chaos limits: a rolled-down tier relaxes *process* weight only. Invariants
|
||
(zero foreign types in canonical interfaces, determinism, passing
|
||
conformance suites) bind at every tier, and a rolled-down pass touching a
|
||
canonical interface still requires the interface change to be flagged in
|
||
the commit for retrospective review.
|
||
|
||
### Step 1 — Research
|
||
|
||
Identify the best implementation in existence for this capability. Produce
|
||
`research/CB-RES-NNNN-<slug>.md` following the survey template (below).
|
||
The survey is done when it can name, per dimension, a concrete
|
||
**benchmark-to-beat**: a number, a property, or a reproducible comparison —
|
||
not an impression.
|
||
|
||
**Runnable-baseline option.** For passes judged high-leverage (declared by
|
||
the maintainer or proposed in the survey and confirmed in the ADR), cited
|
||
numbers are not enough: the survey must ship a reproducible **baseline
|
||
harness** that runs the leading candidate on our machine against our
|
||
workload — the same scenario files where feasible. The harness ships with a
|
||
*fidelity note* stating what was and wasn't faithfully reproduced, so a
|
||
hastily wired competitor setup cannot silently inflate our advantage.
|
||
Where the option is not invoked (or the candidate isn't practically
|
||
runnable), comparisons against cited-only numbers are **directional**: the
|
||
evidence verdict for those rows caps at `parity`, never `better`.
|
||
|
||
### Step 2 — Approve (adversarial review)
|
||
|
||
For tier-L passes, approval is earned through an **adversarial review**: a
|
||
separate session (or agent), given only the survey document, attempts to
|
||
break it — an omitted candidate, a stale or unverifiable benchmark, an
|
||
unmeasured claim presented as measured. Exactly **one round**: challenge,
|
||
then response. The survey is approvable only when every challenge is either
|
||
answered with evidence or conceded and folded into the survey.
|
||
|
||
**Documentation requirement:** the research process, the challenge, and the
|
||
resulting improvements to the research are each documented in timestamped
|
||
markdown files under `history/`:
|
||
|
||
```text
|
||
history/YYMMDD-<slug>-research.md # how the survey was conducted: sources,
|
||
# queries, what was measured vs cited, dead ends
|
||
history/YYMMDD-<slug>-challenge.md # the adversarial attack, verbatim
|
||
history/YYMMDD-<slug>-response.md # answers/concessions and what changed in the survey
|
||
```
|
||
|
||
The polished survey artifact remains `research/CB-RES-NNNN-<slug>.md`; the
|
||
history files preserve the unpolished trail so a later reader can judge how
|
||
hard the survey was actually tested. For tier-M passes the review is
|
||
optional but, when performed, follows the same format. If not approvable
|
||
after the round, the loop returns to step 1 with the named gaps.
|
||
|
||
### Step 3 — Decide
|
||
|
||
`decisions/ADR-NNNN-<slug>.md`: assimilate behind a port, reimplement, or
|
||
hybrid — with the **expected advantage stated per dimension** (see rubric).
|
||
An honest "worse here, better there, and why that trade is right" beats a
|
||
claimed sweep of all four dimensions.
|
||
|
||
### Step 4 — Specify
|
||
|
||
`specs/<Capability>.md`: the contracts, invariants, and — mandatory — the
|
||
**acceptance metrics table**, each row tied to a baseline from step 1.
|
||
A spec without measurable acceptance criteria is not done. Metrics follow
|
||
the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md),
|
||
including the rule that metric selection itself passes through a mini
|
||
research step (metric provenance).
|
||
|
||
**Every metric names its instrument, and is checked reachable.** A row
|
||
in the acceptance table carries the command that produces its number.
|
||
A metric with no named instrument is a wish, not a metric.
|
||
|
||
**The instrument must exist, and the target must come out of it.**
|
||
Naming a command is not the same as running one. A target computed by
|
||
hand and merely *labelled* with a command is the same defect the rule
|
||
was written to stop, one level down. Where the instrument is built later
|
||
in the pass, the target is marked `provisional:` until the instrument
|
||
emits it, and the spec is amended to whatever the instrument returns.
|
||
|
||
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
|
||
`cb-cost --pin fc76445` before that tool existed, and set the target to
|
||
a hand-computed $92.87. When the tool was built it returned $93.32 —
|
||
the hand computation carried a dedup bug the tool's own positive control
|
||
caught. The metric satisfied v1.0's rule completely and was still
|
||
wrong.)*
|
||
|
||
**A number inherited from earlier work is re-derived before it is used
|
||
as a target, or it is cited as unverified.** Quoting is not measuring.
|
||
|
||
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
|
||
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
|
||
double-counted transcript lines and priced a three-model session at one
|
||
model's rate. Neither error was of the harness-does-nothing class; both
|
||
sums ran over real data, and a positive control would have passed them.)*
|
||
|
||
**A metric is checked against the contracts in its own spec.** If a
|
||
contract makes a target unreachable, one of the two is wrong and the
|
||
conflict is resolved when it is noticed, not at the acceptance run.
|
||
Re-check the table whenever a contract is added.
|
||
|
||
*(v1.0, from CB-WP-0001: AM-4's ≤20-crate target was made unreachable by
|
||
the K5 and K7 contracts written after it, and AM-12's cost metric was
|
||
fully specified and never instrumented, so it could not be computed.)*
|
||
|
||
### Step 5 — Code loop
|
||
|
||
Implement iteratively. Each iteration:
|
||
|
||
```text
|
||
change → cb-check (fmt, clippy, tests) → scenarios → benchmarks
|
||
→ compare against acceptance table → evidence row appended
|
||
```
|
||
|
||
Done when every acceptance metric meets or beats its baseline and the
|
||
comparison numbers are committed as an evidence file
|
||
(`evidence/CB-EV-NNNN-<slug>.md`). A failed scenario must yield a replay
|
||
artifact an agent can re-execute locally.
|
||
|
||
#### Measurement validity — the positive control
|
||
|
||
**Every benchmark and harness must assert that it performed the work it
|
||
reports.** Completing without error is not evidence of having done
|
||
anything: a loop whose commands are all rejected runs fast and reports a
|
||
throughput for work that never happened.
|
||
|
||
Concretely, a measurement harness must, on every run:
|
||
|
||
- assert the unit of work produced its expected effect (events applied,
|
||
rows written, moves accepted) — not merely that the call returned;
|
||
- fail loudly rather than report a number when that assertion fails;
|
||
- state the divisor used to convert raw timings into the metric's unit,
|
||
pinned by a test so a workload change cannot silently rescale it.
|
||
|
||
**A number from a run that cannot prove it did the work is void** and
|
||
must not reach an evidence file.
|
||
|
||
**Every tool that reports a number exposes `--self-test`**, and that
|
||
self-test runs before the number is produced (`make cost` depends on
|
||
`make cost-test`). The assertion must name a failure it detects, not
|
||
merely exercise the happy path.
|
||
|
||
*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first
|
||
run against real data and aborted, catching a rule that was verified on
|
||
206/206 groups of the main transcript and false in the 8-response
|
||
subagent tree. The generalization that failed — a property confirmed on
|
||
the largest sample assumed to hold on the smallest — is not one review
|
||
catches, because both the survey and the adversarial reviewer checked
|
||
the same large sample.)*
|
||
|
||
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
|
||
exactly this shape. A JS harness reported 8.4s for 100k moves while
|
||
every move was being rejected, and a Rust benchmark reported 9.3M
|
||
events/s — a 93× beat — while most rounds never completed because a
|
||
stress gate rejected one player's action. The corrected figure was 5.6×
|
||
lower. Adversarial review caught neither; both were claims about
|
||
numbers, and review reads prose.)*
|
||
|
||
#### Evidence states what it does not support
|
||
|
||
An evidence file that compares across runtimes, languages, or feature
|
||
sets **names the disanalogies explicitly**, in the same section as the
|
||
number. The reader must not have to infer that a ratio is not
|
||
like-for-like. This is the parity-cap rule applied to the write-up:
|
||
state the claim you will defend, and the claim you are not making.
|
||
|
||
---
|
||
|
||
## The four-dimension rubric
|
||
|
||
Every survey, ADR, and acceptance table is organized by these dimensions:
|
||
|
||
| Dimension | Question | Example measurable proxies |
|
||
|---|---|---|
|
||
| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session |
|
||
| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task |
|
||
| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead |
|
||
| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility |
|
||
|
||
Scoring is always **relative to the step-1 baseline**, never absolute:
|
||
`better / parity / worse / unmeasured` per proxy, with the number attached.
|
||
`unmeasured` is legal in a survey, illegal in an evidence file.
|
||
|
||
---
|
||
|
||
## Survey template (research/CB-RES-NNNN-<slug>.md)
|
||
|
||
```markdown
|
||
# CB-RES-NNNN: <capability>
|
||
capability: <canonical.capability.id>
|
||
status: draft | approved
|
||
|
||
## Candidates
|
||
Per candidate: origin, license, maturity, adoption; data model; mutation
|
||
mechanism; determinism/replay story; relevant performance (measured if
|
||
runnable locally, cited with source otherwise).
|
||
|
||
## Baselines (benchmark-to-beat)
|
||
| Dimension | Baseline holder | Metric | Value | Provenance |
|
||
(one row minimum per dimension; provenance = measured / cited / estimated)
|
||
|
||
## Verdict
|
||
Which candidate leads per dimension; what none of them do well
|
||
(the surpass opportunity); risks in the baselines themselves.
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation rules the first pass earned
|
||
|
||
These are cheap, and each exists because its absence cost something in
|
||
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
|
||
|
||
1. **No silently-ignored input.** A field that is parsed and then unused
|
||
is a defect, not a stub. Inputs are honoured or rejected with an
|
||
error — never dropped. *(A scenario `setup.patch` was parsed and
|
||
discarded; every scenario using it would have tested the wrong
|
||
initial state while passing.)*
|
||
2. **Decisions get commands, not defaults.** A rule that requires a
|
||
participant's choice is implemented as a command carrying that
|
||
choice. Until it is, **nothing claims coverage of it** — no tag, no
|
||
scenario, no acceptance row. Inventing a default to make a rule
|
||
"done" is the failure this prevents.
|
||
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
|
||
not evidence. Any scaffold path no test reaches is marked as
|
||
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
|
||
that would panic on any state holding a relation.)*
|
||
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
|
||
against `covers:` lists proves no rule is unclaimed and no claimed
|
||
rule is invented. It does **not** prove a scenario exercises what it
|
||
names. Wherever such a number is reported, that limit is reported
|
||
with it.
|
||
|
||
---
|
||
|
||
## Agentic-efficiency requirements
|
||
|
||
The loop exists to be driven by agents. Therefore:
|
||
|
||
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
|
||
split before exceeding, link with relative paths.
|
||
2. **Structured over prose** *(guidance, not a requirement — nothing can
|
||
check it)* — tables and fenced blocks for anything a later step must
|
||
parse (baselines, acceptance metrics, evidence rows).
|
||
3. **One command surface** — all checks runnable through repo-root
|
||
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
|
||
in one place), each supporting deterministic, greppable output.
|
||
4. **Self-contained tasks** — a workplan task names its input artifacts and
|
||
output artifacts; a fresh session must be able to execute it from the
|
||
task text plus linked files alone.
|
||
5. **Evidence or it didn't happen** — claims of "better" live in committed
|
||
evidence files with numbers, never only in commit messages or chat.
|
||
6. ~~**Token discipline** — per the global budget policy, a loop iteration
|
||
that exceeds its budget without measurable progress is stopped and
|
||
decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task
|
||
token budget was never referenced or enforced, and CB-WP-0001 T08
|
||
exceeded it by orders of magnitude with no signal. It implies a control
|
||
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
|
||
this is documentation of a gap, not a rule.
|
||
|
||
**Enforcement status.** Rules above that a command can check are enforced
|
||
by `make loop-lint`; the full classification of every InnerLoop rule as
|
||
executable / checkable / decorative, with the failure class each catches,
|
||
is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance*
|
||
or *dead* say so where they appear, so a reader can tell a requirement
|
||
from a preference without consulting the audit.
|
||
|
||
---
|
||
|
||
## Definition of done — one loop pass
|
||
|
||
A capability has completed the loop when all of the following are committed:
|
||
|
||
- [ ] research/CB-RES-NNNN with approved status and full baseline table
|
||
- [ ] decisions/ADR-NNNN with per-dimension expected advantage
|
||
- [ ] specs/<Capability>.md with acceptance-metrics table
|
||
- [ ] passing scenarios covering every numbered spec rule
|
||
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
|
||
- [ ] every reported number produced by a harness with a positive
|
||
control; any metric that could not be instrumented is recorded as
|
||
uncomputable rather than estimated
|
||
- [ ] every unmet metric reported as unmet, with attribution and the
|
||
options for resolving it — a missed target is an output of the
|
||
loop, not a reason to move the target quietly
|
||
- [ ] **cost recorded**: `make cost` run for the pass, its composition
|
||
(not only its total) in the evidence file, and the per-task figures
|
||
pushed to the hub. M-D2-CST is no longer allowed to be
|
||
`uncomputable` — the instrument exists
|
||
([CostAccounting.md](CostAccounting.md))
|
||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||
file): what the loop itself should change
|