CB-WP-0021 T06: fix AM-7's measurement, not its floor
Some checks failed
ci / check (push) Has been cancelled
Some checks failed
ci / check (push) Has been cancelled
The row folded a 5,000-event log against a 100,000-event log and compared throughputs, which confounds 'does cost per event grow with history' (the property it claims) with 'does streaming a 20x longer Vec cost more per element' (a memory-hierarchy fact true of any program). It measured the second and reported it as the first: importing the edition enlarged the aggregate and the ratio fell to 0.845 with the state bounded. Corrected to time the SAME 5,000 events on a state at depth 0 and on a state at depth 100,000. Equal windows, equal event mix, so the only difference left is history depth. corrected: clean 1.004, mutated 0.589 (red) old: clean 0.845 (red on healthy code), mutated 0.751 It also runs in 8.5s instead of timing out: the first version re-walked the 100k prefix every repetition, 200M untimed folds per sample, which under the mutation never finished. A control that cannot be run is not a control. It now advances to depth once per sample and clones. Two of my own measurements here were wrong and both were caught by measuring again. A 2-minute timeout killed the shell line before its restoring cp ran, so three readings were taken on MUTATED code -- I diagnosed an event-mix confound that did not exist and 'fixed' it. The fix is kept on its merits; the justification was fiction. And the probe that proved state was bounded had checked four of eleven collections. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
2da19a49b7
commit
503966bce9
5 changed files with 155 additions and 82 deletions
|
|
@ -2,7 +2,7 @@
|
|||
id: CB-WP-0021
|
||||
kind: product
|
||||
title: "Import the edition: the game plays its own data"
|
||||
status: ready
|
||||
status: active
|
||||
state_hub_workstream_id: "782b1c37-f3a7-469b-87a3-fa73ebe758d2"
|
||||
---
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ alternatives to take, and that is the ADR.
|
|||
|
||||
```task
|
||||
id: CB-WP-0021-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "68e4fe63-eec6-4fb8-a84f-32c7edee19af"
|
||||
```
|
||||
|
|
@ -155,7 +155,7 @@ replay determinism, is not a data-loading change.
|
|||
|
||||
```task
|
||||
id: CB-WP-0021-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "c77c0b39-0841-40bf-8078-135486d7ed55"
|
||||
```
|
||||
|
|
@ -182,7 +182,7 @@ numbers nobody ruled on.
|
|||
|
||||
```task
|
||||
id: CB-WP-0021-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "28c3ff2c-16ae-47b5-9474-10e754936c60"
|
||||
```
|
||||
|
|
@ -207,7 +207,7 @@ Hidden → face down) and `hidden_priority`.
|
|||
|
||||
```task
|
||||
id: CB-WP-0021-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "ff3bd923-9066-49ce-aadd-a3552e4964ff"
|
||||
```
|
||||
|
|
@ -228,6 +228,51 @@ ruled. Message `ground-game` with the outcome.
|
|||
Do **not** quietly delete a failing-in-fact scenario. CB-EV-0005: *a score
|
||||
improved by deleting the question is not an improvement.*
|
||||
|
||||
## Task: fix AM-7's measurement, not its floor
|
||||
|
||||
```task
|
||||
id: CB-WP-0021-T06
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
T05 turned AM-7 red: median **0.845** against a 0.9 floor, all nine
|
||||
samples below. The maintainer chose to **fix the measurement** rather than
|
||||
ADR the floor or optimise the fold.
|
||||
|
||||
**The row was measuring the wrong thing.** It folded a 5,000-event log and
|
||||
a 100,000-event log and compared throughputs, which confounds:
|
||||
|
||||
1. does cost per event grow with how many events have been folded? — the
|
||||
property AM-7 claims; and
|
||||
2. does streaming a 20× longer `Vec` cost more per element? — a
|
||||
memory-hierarchy fact true of any program.
|
||||
|
||||
It measured (2) and reported it as (1). Importing the edition enlarged the
|
||||
aggregate and the ratio fell, **with the state bounded**.
|
||||
|
||||
**Corrected:** time a 5,000-event window on a state at depth 0, and *the
|
||||
same events* on a state at depth 100,000. Equal windows, equal event mix;
|
||||
the only difference left is history depth.
|
||||
|
||||
| | clean | history-proportional mutation |
|
||||
|---|---|---|
|
||||
| corrected | **1.004** | **0.589 — red** |
|
||||
| old | 0.845 (red on healthy code) | 0.751 |
|
||||
|
||||
Renamed `am7_cost_per_event_does_not_grow_with_history`, because the old
|
||||
name described the confounded measurement.
|
||||
|
||||
**Two of my own measurements in this task were wrong, and both were caught
|
||||
by measuring again.** A 2-minute timeout killed the shell line before its
|
||||
restoring `cp` ran, so the next three readings were taken on **mutated
|
||||
code** — I diagnosed a "second confound" from event-mix that did not
|
||||
exist, and "fixed" it by folding identical events on both legs. That
|
||||
change is kept, on its own merits: identical events remove a real
|
||||
potential confound. But the justification I gave for it was fiction, and
|
||||
the probe that proved state was bounded had only checked four of eleven
|
||||
collections.
|
||||
|
||||
## Task: evidence
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue