60 lines
9.4 KiB
Markdown
60 lines
9.4 KiB
Markdown
# 2026-07-31 — adversarial challenge: CB-RES-0001 (T04)
|
||
|
||
**Verdict: approvable with corrections** — the measured numbers are real and the determinism/dependency claims verify against the harness, but the survey's central architectural claim about boardgame.io's superlinear degradation is contradicted by a documented tuning option the harness never tested, the memory methodology is invalid, and at least two squarely in-scope candidates (OpenSpiel, Ludii) are omitted.
|
||
|
||
Reviewed cold: only `research/CB-RES-0001-game-kernel.md` plus the harness under `research/CB-RES-0001-harness/boardgame-io/` (`bench.js`, `results-260731.json`, `package.json`, and the installed `boardgame.io@0.50.2` source in `node_modules`).
|
||
|
||
---
|
||
|
||
## Blocking
|
||
|
||
### B1. "Architectural, not a tuning artifact" is asserted, untested, and likely false as stated
|
||
|
||
Survey text: *"Per-move cost grows with history length (log accumulation + state pipeline): throughput halves as move count doubles — superlinear total cost. This is architectural (unbounded redux log per client), not a tuning artifact."* This attribution underwrites the D3-scaling baseline row (0.45×), the "boardgame.io is slow *and* degrades" verdict, and plausibly the "did not finish in 300s" @100k row.
|
||
|
||
I inspected the installed library. boardgame.io v0.50.2 has a documented game option `disableUndo` (`node_modules/boardgame.io/dist/cjs/initialize-648ccd94.js:49`, `server.js:1912`). Without it, every move executes `_undo: [...state._undo, undoEntry]` (`reducer-6f7cf6b0.js:853`) — an O(n) array spread **per move**, i.e. O(n²) total, retaining a G/ctx/plugins snapshot per move. That alone is sufficient to produce exactly the observed halving-per-doubling and the RSS growth, and it is switched off by a one-line, documented config flag. The harness (`bench.js`) never sets `disableUndo`, never tests the bare `Master`, and the survey nowhere acknowledges the flag. The client-side `log` accumulation the survey blames may or may not remain superlinear with undo disabled — nobody measured it.
|
||
|
||
The survey's Risks section hedges the *measurement surface* ("headless client pipeline... a bare server-side master could differ") but not the *attribution*: "architectural, not a tuning artifact" is precisely a claim that no tuning knob exists, and one demonstrably does. **Required correction:** rerun the harness with `disableUndo: true` (and ideally the server `Master` path). If scaling flattens, the D3-scaling row, the @100k DNF row, and the "degrades" verdict language must be rewritten; if it stays superlinear, the claim becomes defensible and should cite the flag explicitly.
|
||
|
||
## Significant
|
||
|
||
### S1. Memory baseline (RSS 224 MB @5k) is methodologically invalid
|
||
|
||
`bench.js` measures `process.memoryUsage()` (line 101) **after** the warmup run (5,000 moves, line 92), the main run, *and* three additional determinism runs of 4,000 moves each (lines 96–98) — five client instances, each with its own unbounded log and undo stack, with no GC or isolation between them. The "D3 memory | RSS @5k moves | 224 MB | measured" baseline row attributes to a 5k-move workload an RSS that actually reflects ~22k moves across five clients plus Node baseline. The per-run `rss_mb` column in `results-260731.json` inherits the same flaw. The row should either be re-measured with an isolated single run (report heap delta, not raw RSS) or downgraded from "measured" to "indicative".
|
||
|
||
### S2. Omitted candidates: OpenSpiel and Ludii are squarely in scope; BGA Studio and Colyseus deserve at least a line
|
||
|
||
The survey's stated scope is "deterministic authoritative state, command → validation → events, simultaneous commit/reveal, hidden information, replay". Two mature, well-known systems sit dead-center in that scope and are absent:
|
||
|
||
- **OpenSpiel** (DeepMind, C++/Python, active): a games kernel explicitly built for **simultaneous-move and imperfect-information games**, deterministic, with serializable state and replay via action histories. It is arguably the *best* existing reference for the commit/reveal + hidden-information combination the verdict claims "none of them do", and as an in-process C++ kernel it is a real D3 comparator that could replace the unsourced event-sourcing row (S3).
|
||
- **Ludii** (and the GGP/GDL lineage): general game systems whose entire point is **ease of rule specification** — the D1 dimension. A survey whose D1 baseline is "~45 LOC of JS" while ignoring the ludeme/GDL literature has a hole exactly where it sets a benchmark-to-beat.
|
||
|
||
Secondary: **Board Game Arena Studio** (the dominant commercial turn-based rules framework: server-authoritative state machine, hidden info, full replay — a stronger commercial reference than TTS, which the survey itself calls the anti-model) and **Colyseus** (active JS authoritative-state server — the natural "active JS alternative" counterweight to inactive boardgame.io). The verdict's surpass claim ("what none of them do") cannot stand on a candidate set that excludes the systems most likely to falsify it, OpenSpiel in particular.
|
||
|
||
### S3. The D3 ceiling row — and the headline ADR benchmark derived from it — rests on an unsourced number
|
||
|
||
"~10⁵–10⁶ events/s per core is the commonly reported range" carries no citation: no benchmark link, no repo, no paper. The survey is honest that it is "cited/estimated — directional" and caps verdicts at parity — good — but then the **headline benchmark-to-beat (≥100,000 events/s, flat scaling)** is set from exactly this unsourced range. A benchmark-to-beat whose floor comes from a number nobody can check is soft. Correction: either cite a concrete reproducible source (e.g. a named cqrs-es or fold-benchmark), or measure a minimal Rust comparator, or explicitly label the 100k floor as a stipulated engineering target rather than a competitor-derived baseline.
|
||
|
||
### S4. Timed-loop contamination: `updatePlayerID` inside the measured region
|
||
|
||
`bench.js` calls `client.updatePlayerID(...)` once per move inside the `hrtime` window (lines 73, 76). That is client-object bookkeeping, not move application, and its cost is unquantified. At 1 call per move it plausibly inflates per-move cost by a constant factor; it does not explain the superlinear shape, but the absolute moves/s baseline (1,930 @5k) is not purely "applied moves". Either hoist per-player clients out of the loop or note the contamination in the provenance column.
|
||
|
||
## Minor
|
||
|
||
- **M1 — vacuous baseline row:** D1 "rule-to-scenario traceability: 0% — none of the candidates link rules to tests" is trivially beatable by any nonzero effort; as a "benchmark-to-beat" it is gameable to the point of being a free pass. Keep it as an observation, not a baseline row.
|
||
- **M2 — D1 LOC number is generous and gameable:** the game definition in `bench.js` (lines 15–50) is ~36 lines, not ~45; more importantly LOC-of-a-synthetic-game is an easily gamed metric (formatting, helper extraction). Acceptable as a rough bar; should not be the only quantitative D1 row (see S2 — Ludii/GDL is the real D1 literature).
|
||
- **M3 — hidden information claimed as a feature, never exercised:** the survey credits boardgame.io's `playerView`/`STRIP_SECRET`, but the harness never invokes a playerView, and the synthetic "hands" never deplete (`commitAction` reads `hands[playerID][0]` and never removes cards — line 30), so the hidden-info and hand mechanics are dead weight in the workload. The commit/reveal shape is real; the hidden-info cost is unmeasured. Provenance should say so.
|
||
- **M4 — unverifiable maturity citations:** "12.4k stars", "Snyk: Inactive" are plausible but not independently checkable from the artifact; low stakes since the ~4-years-since-release fact is checkable from npm.
|
||
|
||
## What was attacked and survived
|
||
|
||
- **Determinism claim (measured ✅):** verified in `bench.js` lines 96–98 — same-seed double run compared by SHA-256 of `G`, different-seed run diverges; `results-260731.json` records `true`/`true`. The seeded RNG *is* exercised (`resolveAll` calls `random.D6()`). Sound.
|
||
- **Dependency/weight numbers:** independently re-verified — `node_modules` contains exactly 120 packages, 37 MB (`du -sh`), matching the D2 row precisely. Trustworthy.
|
||
- **Warmup:** present (line 92, 5,000-move JIT warmup before the timed run). Sound.
|
||
- **Throughput arithmetic:** applied-moves accounting (rounds × 4), elapsed_ms, and moves/s are internally consistent between `bench.js` and `results-260731.json`; the 2.6 s / 1,930 moves/s @5k row reproduces from the JSON.
|
||
- **Measured-vs-cited labeling:** with the exceptions in B1/S1/S4, the survey is unusually disciplined about tagging rows measured/cited/observed and its Risks section preemptively flags the client-pipeline and cited-not-measured caveats. No claim tagged "measured" is fabricated; the failures are attribution (B1) and methodology (S1), not honesty.
|
||
- **Verdict internal consistency:** the "×50 headline" arithmetic (100k vs 1,930) checks out, and the verdict correctly refuses to claim a D3 win over the event-sourcing class it hasn't measured. The one overclaim is the B1 "architectural" sentence.
|
||
|
||
## Disposition
|
||
|
||
Approvable after: (1) rerunning the harness with `disableUndo: true` and revising or vindicating the superlinear-is-architectural claim; (2) fixing or relabeling the memory row; (3) adding OpenSpiel and Ludii (at minimum as cited candidates, with the verdict's "what none of them do" paragraph re-checked against OpenSpiel specifically); (4) sourcing or relabeling the event-sourcing throughput range. None of these require new candidate-grade engineering; all are within a day's work.
|