tegwick
561facce43
Some checks failed
ci / check (push) Has been cancelled
T05. tools/design.py, make design, and the register in GroundRules.md --
14 rows, no new file, because ADR-0012 D2 made §Underdetermined the
register rather than building one beside it.
Backfill was the test and it caught two things the ADR did not have.
First, a `role` column. The first report alarmed on U2 and was wrong to:
U2's scenario is green BECAUSE the provisional default it documents is
implemented, which says nothing about whether ground-game agrees. GR-E01's
was a counterexample that went green. Same colour, opposite meaning -- a
register that cannot tell them apart either alarms constantly or never.
Only a green counterexample alarms. Folded back into GameDesign §1.3.
Second, it contradicted the survey. CB-RES-0007 said "six of the ten
already have provisional scenarios." Measured -- grep -lE "\bU<n>\b" over
scenarios/ground -- exactly ONE U-item names itself. Five provisional
scenarios exist and four probably encode U-item defaults, but the mapping
is not written down, so it is not checkable. Same defect class as the
wrong premises, found inside the survey that proposed the fix. Now a
reported debt: open, lacking a reproduction: 9, target 0.
design.py carries the control design-baseline.py never had, asserted
directly: a row citing a nonexistent file must not count as reproduced,
using the exact path 2da19a4 deleted -- which the old tool called green.
design-baseline.py is marked superseded rather than deleted; it is the
evidence for how a wrong number got into a survey.
T06. The report is a FILE in ground-game under GROUND-WP-0002, committed
there, with a hub message that only points at it. It asks for no ruling:
it carries GR-E01's withdrawal, our reproduction debt, and two notes that
are explicitly not findings.
And it had to acknowledge something nobody anticipated. GROUND-WP-0002 is
finished -- all ten U-items were RULED 2026-08-03, every one confirmed as
the default we simulate, plus five of six provisional scenarios. The
survey said "0 of 10 ruled" two days later and this register was built
saying `reported`. That is the unread-inbox failure running in the
opposite direction: they answered and we did not collect it. The
instrument's first run surfaced it. They are `ruled`, not `applied` --
lifting the now-settled provisional flags is owed and is not done, and
make design shows them open until it is.
T07. evidence/CB-EV-0021. Two of six catches in this pass came from
execution rather than process (the role distinction from building it, the
ten uncollected rulings from running it), which is InnerLoop §Design
goal's prediction holding.
make self-tests, facts-check, loop-lint: clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
217 lines
9.7 KiB
Markdown
217 lines
9.7 KiB
Markdown
# CB-EV-0021 — the design instrument
|
||
|
||
CB-WP-0022 T07. Tier L, declaration 5 of chaos window 2 (structural L,
|
||
chaos d8=6 → no override). Closed 2026-08-05.
|
||
|
||
**Delivered:** INTENT property 4 (*Instrument*),
|
||
[ADR-0012](../decisions/ADR-0012-the-design-instrument.md) (nine
|
||
decisions), [specs/GameDesign.md](../specs/GameDesign.md) v1.0, the
|
||
register in [GroundRules.md](../specs/GroundRules.md) (14 rows),
|
||
`tools/design.py` + `make design`, and a report file in `ground-game`.
|
||
|
||
---
|
||
|
||
## 1. The headline: a separate reviewer found the survey's showcase was false
|
||
|
||
This is the first adversarial review in this repo **run by a genuinely
|
||
separate agent**. CB-RES-0006's reviewer opened by conceding it could not
|
||
be, and wrote:
|
||
|
||
> *"Treat the challenges below as a lower bound on what a genuinely
|
||
> separate reviewer would find."*
|
||
|
||
That was a testable claim. **This is the test.** The separate reviewer ran
|
||
`git log` against the survey's central example and found our own commit
|
||
had falsified it four days earlier — while the author, who wrote that
|
||
commit, quoted the dead number twice in the same workplan.
|
||
|
||
`CB-WP-0022:61` justified the whole proposal with *"GR-E01 is admissible
|
||
because 4/6/9 against 5/7/9 is a computation anyone can rerun."*
|
||
`2da19a4` had rerun it: **6/9/12**, scenario renamed `-unreachable-` →
|
||
`-reachable-`, conclusion inverted. And GR-E01 was **one of the two
|
||
findings that passed the reproduction rule**.
|
||
|
||
**Verdict on same-session review: it is not a weak form of the control, it
|
||
is a different control.** Four of the seven challenges rest on facts a
|
||
same-session reviewer had already seen and not questioned. The cost was
|
||
~70k subagent tokens; it caught a defect in flight (§2) and rewrote the
|
||
pass's central rule.
|
||
|
||
## 2. What it caught in flight
|
||
|
||
T06's payload, status `todo`, still read *"the deal count puts 4/6/9
|
||
points in play … no dataset reconciles them"* and was queued for
|
||
transmission to `ground-game`.
|
||
|
||
It would have been the **fourth** wrong premise sent downstream, after
|
||
*"12 in the file"* (a sum with no deal table) and *"SOLVE offered on a
|
||
face-down Problem"* (the wrong condition named). **It is the only one
|
||
stopped before sending.**
|
||
|
||
| wrong premise | reached ground-game? | passed an existence test? |
|
||
|---|---|---|
|
||
| *"12 in the file"* | yes | — |
|
||
| *"SOLVE on a face-down Problem"* | yes | — |
|
||
| *"4/6/9, no dataset reconciles them"* | **stopped 2026-08-05** | **yes** |
|
||
|
||
## 3. The rule the pass came to build was one clause short
|
||
|
||
The declaration's load-bearing rule was an **existence** test. Three of
|
||
three wrong premises would have passed it, and one did.
|
||
|
||
What shipped instead (GameDesign §1): a reproduction must **exist**, have
|
||
the **shape ground-game ruled** (row-level deal table, never a sum —
|
||
promoted from a T04 addendum to part of the rule), and **be able to
|
||
fail**.
|
||
|
||
**Clause 3 is the one this pass bought.** GR-E01's scenario went green
|
||
when the edition landed and the finding stayed admissible and stayed
|
||
queued, because nothing treated a passing artifact as a signal. *A green
|
||
reproduction is an alarm, not a reassurance.*
|
||
|
||
## 4. Building it produced a distinction the ADR does not have
|
||
|
||
`make design`'s first run alarmed on **U2** — and was wrong to.
|
||
|
||
U2's scenario is green **because the provisional default it documents is
|
||
implemented**, which says nothing about whether ground-game agrees.
|
||
GR-E01's was a *counterexample* that went green. **Same colour, opposite
|
||
meaning.**
|
||
|
||
So the register gained a `role` column — `counterexample` | `default` —
|
||
and only a green counterexample alarms. **A register that cannot tell them
|
||
apart either alarms constantly or never**, and this was invisible until
|
||
the metric ran against real rows. Folded back into GameDesign §1.3.
|
||
|
||
## 5. Did backfilling change the design?
|
||
|
||
**T07 asked to be suspicious if all six findings fit the first taxonomy.**
|
||
They did — five kinds absorbed everything, no sixth invented. Two reasons
|
||
that is less reassuring than it looks and one reason it is fine:
|
||
|
||
- the taxonomy was **derived from these six**, so fitting them is nearly
|
||
circular. Its real test is the seventh finding;
|
||
- **one of the six was deleted during backfill.** *"Six provisional
|
||
defaults"* was not entered: C3 showed it double-counted GR-E01, and
|
||
provisional scenarios are *reproductions for* underdetermined items, not
|
||
a finding. So the taxonomy absorbed five, not six;
|
||
- what it did do is **force the note tier to earn itself** — GR-A13 and
|
||
GR-E03/E04 have no artifact and became notes, which is the first time
|
||
the distinction cost anything.
|
||
|
||
## 6. The numbers, and why none of the survey's survive
|
||
|
||
**CB-RES-0007's measurements are withdrawn.** C2 dismantled the
|
||
instrument: `design-baseline.py` is a hand-maintained dict where
|
||
`n = len(FINDINGS)`; `has_reproduction` is `startswith("scenarios/")` and
|
||
never stats the file, so its own YES-control was **green against a path
|
||
`2da19a4` deleted**; the 4-day figure is two date literals subtracted
|
||
(mutated to 2020-01-01 → 2406 days, all checks still green); and
|
||
`Makefile:127` ran only `--self-test`, so the reporting path had no CI.
|
||
|
||
It is **superseded, not deleted** — it is the evidence for how a wrong
|
||
number got into a survey, and the header now says so.
|
||
|
||
Measured by `make design`, 2026-08-05, over rows that name real files:
|
||
|
||
```
|
||
findings 12 (+2 notes)
|
||
with a resolving reproduction 3/12 = 25% target 100%
|
||
open, lacking a reproduction 9 target 0
|
||
reproductions green while open 0 target 0
|
||
notes past 30 days 0 target 0
|
||
closed (log) 2 [F11, F13]
|
||
```
|
||
|
||
**Two survey claims died against measurement:**
|
||
|
||
| survey said | measured |
|
||
|---|---|
|
||
| *"six of the ten already have provisional scenarios"* | **one.** `grep -lE "\bU<n>\b" scenarios/ground/*.yaml` → only U2 |
|
||
| *"2/6 = 33% reproduced"* | withdrawn; GR-E01 was double-counted and the detector was broken |
|
||
|
||
The first is the same defect class as the wrong premises — **an
|
||
uncheckable link asserted as a fact** — found inside the survey that
|
||
proposed the fix.
|
||
|
||
## 7. The register's first run found ten answers nobody had collected
|
||
|
||
The most useful thing `make design` did was not a metric.
|
||
|
||
**GROUND-WP-0002 T05 ruled all ten U-items on 2026-08-03 — every one
|
||
confirmed.** T03 confirmed five of the six provisional scenarios. The
|
||
workplan is `finished`.
|
||
|
||
CB-RES-0007 reported *"0 of 10 ruled"* on 2026-08-03, and the register
|
||
built this session initially recorded them as `reported`. **Both were two
|
||
days stale on the day they were written.**
|
||
|
||
**This is the unread-inbox failure running in the opposite direction.**
|
||
The pass was designed around *we send findings and nobody reads them*. The
|
||
symmetric failure — *they answer and we do not collect it* — was not
|
||
anticipated by the declaration, the survey, or the review, and the first
|
||
run of the instrument surfaced it immediately.
|
||
|
||
The items are `ruled`, not `applied`: the rulings confirmed our defaults,
|
||
so the rules did not move, but the scenarios still carry `provisional:
|
||
true` for settled choices. **Lifting those flags is owed and is not done**
|
||
— `make design` shows them open until it is, which is the register
|
||
working.
|
||
|
||
## 8. The engine-evolution question, as the review left it
|
||
|
||
Declared judgment: no second register, because engine evolution already
|
||
lives in `evidence/`, `decisions/`, `gates.toml` and workplans.
|
||
|
||
**The conclusion survives; the argument did not.** C5 found the "third
|
||
thing" the maintainer meant is visible in `specs/InnerLoopReference.md`
|
||
(383 lines, two sections literally headed *"moved from InnerLoop v1.6"*)
|
||
and 769 lines of `history/` retrospectives — **neither of which the
|
||
inventory named**. So the content exists and found homes somewhat by
|
||
accident.
|
||
|
||
ADR-0012 D7 records it as **argued and narrowed, not settled**, with the
|
||
revisit condition: if InnerLoopReference keeps absorbing material that is
|
||
neither a decision nor a finding, build the register.
|
||
|
||
## 9. What tier L cost against what it caught
|
||
|
||
Second full-weight L pass in this window, third in the project.
|
||
|
||
| caught | by |
|
||
|---|---|
|
||
| the central example was false | **only** the separate reviewer |
|
||
| a wrong premise in flight to ground-game | the same |
|
||
| the baseline instrument counts itself | the same |
|
||
| the incumbent register was never evaluated | the same |
|
||
| `role`: green default ≠ green counterexample | **building it**, not reviewing it |
|
||
| ten uncollected rulings | **running it**, not building it |
|
||
|
||
**Two of six were caught by execution, not by process** — the pattern
|
||
InnerLoop §Design goal predicts (cheap correction beats exhaustive
|
||
prevention), and an argument against front-loading more review rather than
|
||
less.
|
||
|
||
**The one number that matters for the loop's own calibration:** the review
|
||
step is what distinguished this pass from a competent one. Without it the
|
||
pass ships a rule justified by a false example, sends a fourth bad premise
|
||
downstream, and builds `make design` on a dict that counts itself. The
|
||
correct reading of CB-RES-0006's fidelity note is **not** *"our reviews
|
||
were somewhat weaker"* — it is *"our reviews were not performing this
|
||
function at all."*
|
||
|
||
## 10. Cost
|
||
|
||
CB-WP-0021's cost, by re-running the instrument: see `make cost` /
|
||
`tools/cb-cost.py`. **Not quoted here as a literal** — §Single source of
|
||
fact, and this pass has just spent itself proving what typed numbers do.
|
||
|
||
## Open after this pass
|
||
|
||
- **U1–U10 → `applied`**: lift `provisional: true` where a ruling confirmed
|
||
the default, and record each ruling. Nine items of bookkeeping.
|
||
- **The U-item ↔ scenario mapping is unwritten** for 9 of 10. Cheap, and
|
||
it is what makes the register checkable.
|
||
- **F12 and F14 are notes** and expire 2026-08-31. GROUND-WP-0003's
|
||
playtest closes F14 and produces its artifact via `cb-play --record`.
|
||
- **The taxonomy's real test is the seventh finding**, not these six.
|