T01: audit every InnerLoop rule, and make the checkable ones executable
41 rules classified executable / checkable / decorative, each tagged with
the failure class it catches. Counts: 11 executable, 22 checkable, 4
decorative (one of them dead policy).
Audit: history/260731-inner-loop-rule-audit.md
New tools/loop-lint.py makes 7 rules executable (tier declared, chaos
roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file
loadability, reporting tools expose --self-test). It found three real
violations on its first run, none previously visible:
- specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit
the loop has stated since v0.2 and never measured. Split at its own
section boundaries into Blueprint (1-8) + Runtime (9-15).
- tools/dep-weight.py and tools/rule-coverage.py had positive-control
logic and no --self-test, so nothing verified the control worked.
Adding rule-coverage's self-test exposed a latent instance of the exact
class this workplan is about: if the spec regex stopped matching, rules
was empty, missing was empty, and the tool exited 0 reporting "0/0" --
a silent pass, in the tool that reports our headline AM-1 number. Both
tools now assert they found something before reporting.
Two demotions applied in the spec rather than left implicit: "structured
over prose" is marked guidance (nothing can check it), and the 8k/10k
token budget is struck through and marked DEAD POLICY pointing at T05.
The audit's uncomfortable finding: rule 13 (re-derive inherited numbers)
has no mechanical form, is deliberately left decorative, and caught the
LARGEST error in CB-WP-0002. That is a counter-example to this
workplan's own hypothesis. "A rule that cannot be executed is not a
rule" is wrong as stated; the defensible version is that such a rule
cannot be relied on to fire, so it must not be the only defence for a
class that matters.
Class coverage: harness-does-nothing has five executable rules;
trusted-arithmetic has ZERO and produced the largest single error.
make loop-lint and make self-tests wired into `make all` and CI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ee1ee485b2
commit
fed422a3a3
10 changed files with 827 additions and 242 deletions
|
|
@ -151,11 +151,12 @@ as a target, or it is cited as unverified.** Quoting is not measuring.
|
|||
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
|
||||
double-counted transcript lines and priced a three-model session at one
|
||||
model's rate. Neither error was of the harness-does-nothing class; both
|
||||
sums ran over real data, and a positive control would have passed them.)* A metric must
|
||||
also be checked against the contracts in the *same spec*: if a contract
|
||||
makes a target unreachable, one of the two is wrong and the conflict is
|
||||
resolved when it is noticed, not at the acceptance run. Re-check the
|
||||
table whenever a contract is added.
|
||||
sums ran over real data, and a positive control would have passed them.)*
|
||||
|
||||
**A metric is checked against the contracts in its own spec.** If a
|
||||
contract makes a target unreachable, one of the two is wrong and the
|
||||
conflict is resolved when it is noticed, not at the acceptance run.
|
||||
Re-check the table whenever a contract is added.
|
||||
|
||||
*(v1.0, from CB-WP-0001: AM-4's ≤20-crate target was made unreachable by
|
||||
the K5 and K7 contracts written after it, and AM-12's cost metric was
|
||||
|
|
@ -297,8 +298,9 @@ The loop exists to be driven by agents. Therefore:
|
|||
|
||||
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
|
||||
split before exceeding, link with relative paths.
|
||||
2. **Structured over prose** — tables and fenced blocks for anything a
|
||||
later step must parse (baselines, acceptance metrics, evidence rows).
|
||||
2. **Structured over prose** *(guidance, not a requirement — nothing can
|
||||
check it)* — tables and fenced blocks for anything a later step must
|
||||
parse (baselines, acceptance metrics, evidence rows).
|
||||
3. **One command surface** — all checks runnable through repo-root
|
||||
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
|
||||
in one place), each supporting deterministic, greppable output.
|
||||
|
|
@ -307,9 +309,20 @@ The loop exists to be driven by agents. Therefore:
|
|||
task text plus linked files alone.
|
||||
5. **Evidence or it didn't happen** — claims of "better" live in committed
|
||||
evidence files with numbers, never only in commit messages or chat.
|
||||
6. **Token discipline** — per the global budget policy, a loop iteration
|
||||
6. ~~**Token discipline** — per the global budget policy, a loop iteration
|
||||
that exceeds its budget without measurable progress is stopped and
|
||||
decomposed, not pushed through.
|
||||
decomposed, not pushed through.~~ **DEAD POLICY.** The 8k/10k per-task
|
||||
token budget was never referenced or enforced, and CB-WP-0001 T08
|
||||
exceeded it by orders of magnitude with no signal. It implies a control
|
||||
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
|
||||
this is documentation of a gap, not a rule.
|
||||
|
||||
**Enforcement status.** Rules above that a command can check are enforced
|
||||
by `make loop-lint`; the full classification of every InnerLoop rule as
|
||||
executable / checkable / decorative, with the failure class each catches,
|
||||
is in `history/260731-inner-loop-rule-audit.md`. Rules marked *guidance*
|
||||
or *dead* say so where they appear, so a reader can tell a requirement
|
||||
from a preference without consulting the audit.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -335,4 +348,3 @@ A capability has completed the loop when all of the following are committed:
|
|||
([CostAccounting.md](CostAccounting.md))
|
||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||
file): what the loop itself should change
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue