diff --git a/research/ModelMatching.md b/research/ModelMatching.md index a3caba8..9443302 100644 --- a/research/ModelMatching.md +++ b/research/ModelMatching.md @@ -1,9 +1,10 @@ # Model matching and regime flipping -**Status:** v0.2 (2026-10-13), ASSEMBLY-WP-0012 and WP-0017. Matching on -observed data works. The IMM filter (§ 4.1) detects accelerations reliably, -but detects slowdowns only after 2.5–3.75 years. The acceptance test is not -yet passed, so no measured flip may be declared yet. +**Status:** v0.3 (2026-10-13), ASSEMBLY-WP-0012 and WP-0017. The IMM filter +passes its synthetic acceptance test (§ 4.2) under two caveats: a perfect +model, and an investment indicator that the real data provide only roughly. +Measured flips may be declared only after an AI-specific investment series is +in place (WP-0017-T06). Code: `simulations/gadm02/matching.py`. Data: `observed_series.json`. Results: `results/matching.md` (observed), `results/matching_synthetic.md` @@ -153,3 +154,31 @@ Observed series for the real filter: US adoption (BTOS), US investment (L5, quarterly; operator rough), US discovery channel (StatCounter, monthly), EU and DE adoption (annual). Sentiment points after checkpoint 0 do not exist yet. + +### 4.2 Flips judged on anchors: acceptance passed (WP-0017-T04) + +The slow slowdown detection in § 4.1 came from the flip rule, not from the +evidence. Investment halves immediately when B3 switches to B2 (0.090 → +0.044), but the new anchor's weight split between B2/loop-off and B2/loop-on, +which are indistinguishable while sentiment is flat. Neither reached 0.75. +The rule now judges **anchor flips on anchor weights** (summed over the +loop state) and loop flips on loop weights. False flips count both. + +| Indicators | Anchor correct at start | False flips | Anchor switches within 1.5 y | Delays | +|---|---|---|---|---| +| adoption + investment + Pd + sentiment | 6 of 6 | 0 of 6 | **6 of 6** | 0.5 y each (two quarters, the minimum) | +| adoption + Pd + sentiment (no investment) | — | **5 of 6** | 5 of 6 | 0.5–3.0 y | + +**Acceptance passed** with all four indicators (`results/matching_imm.md`). + +Caveats: + +1. **Perfect model.** Truth and candidates share the model structure. Real + data contain misspecification, so real performance is worse. +2. **The investment indicator carries the result.** Without it, false flips + rise to 5 of 6. The real investment series (L5) covers all + information-processing equipment, not AI only. Before measured flips may be + declared, an AI-specific investment series is needed (WP-0017-T06). +3. **The loop state stays undetectable while sentiment is flat.** Loop flips + are therefore not declared from the filter at all until a sentiment shock + gives evidence. diff --git a/simulations/gadm02/matching.py b/simulations/gadm02/matching.py index fdce655..36eee95 100644 --- a/simulations/gadm02/matching.py +++ b/simulations/gadm02/matching.py @@ -121,8 +121,22 @@ def best(res: dict) -> tuple: return max(res, key=lambda k: res[k]["w"]) -def flips(evaluations: list[tuple[float, dict]]) -> list[tuple[float, tuple, tuple]]: - """Flip events: best changes and the new best holds weight >= FLIP_WEIGHT twice in a row.""" +def marginal(res: dict, part: int) -> dict: + """Weights summed over the other part of the candidate (0 = anchor, 1 = loop state).""" + out: dict = {} + for c, v in res.items(): + out[(c[part],)] = out.get((c[part],), 0.0) + v["w"] + return {k: {"w": w} for k, w in out.items()} + + +def flips(evaluations: list[tuple[float, dict]], by: int | None = None) -> list[tuple[float, tuple, tuple]]: + """Flip events: best changes and the new best holds weight >= FLIP_WEIGHT twice in a row. + + by = 0 judges the anchor (weights summed over the loop state), by = 1 the loop state; None the full + candidate. Judging the anchor matters when the loop is not identifiable: its weight would otherwise + split between two indistinguishable candidates and never reach FLIP_WEIGHT.""" + if by is not None: + evaluations = [(t, marginal(res, by)) for t, res in evaluations] events, current, pending = [], None, None for t, res in evaluations: b, wb = best(res), res[best(res)]["w"] @@ -406,7 +420,8 @@ def main_imm() -> None: "switching at t 4; observed from t 2 to t 8. Indicators and noise: adoption biweekly (log sd 0.03),", "investment quarterly (log sd 0.05), discovery channel Pd monthly (sd 0.01), sentiment twice a", f"year (sd 0.02). IMM filter, switch probability {IMM_SWITCH} per quarter. Flip: new best with weight", - f"≥ {FLIP_WEIGHT} twice in a row.", "", + f"≥ {FLIP_WEIGHT} twice in a row, judged on anchor weights (summed over the loop state) and, for the", + "loop case, on loop weights (summed over anchors). False flips count both.", "", "**Acceptance (anchor regimes):** anchor correct at the first evaluation in ≥ 5 of 6 no-switch runs,", "no false flips in 6 no-switch runs, and ≥ 4 of 6 anchor switches detected within 1.5 years. The", "loop switch is reported separately: with sentiment anchored, switching the loop on is invisible", @@ -421,10 +436,11 @@ def main_imm() -> None: truth = synthetic_rows("US", first_c, second, ts or T_PATH) for seed in (1, 2, 3): ev = imm_run(truth, 2.0, 6.0, seed, kinds) - fl = flips(ev) + fl = flips(ev, by=0) if kind != "loop" else flips(ev, by=1) b0 = best(ev[0][1]) if kind == "none": correct += b0[0] == first_c[0] + fl = flips(ev, by=0) + flips(ev, by=1) false += bool(fl) verdict = ("anchor correct" if b0[0] == first_c[0] else "anchor wrong") + ("; FALSE FLIP" if fl else "") else: @@ -432,12 +448,12 @@ def main_imm() -> None: hit = [e for e in fl if ts <= e[0] <= ts + 1.5 and e[2][0] == second[0]] det += bool(hit) else: - hit = [e for e in fl if ts <= e[0] <= ts + 1.5 and e[2][1] == second[1]] + hit = [e for e in fl if ts <= e[0] <= ts + 1.5 and e[2][0] == second[1]] loop_det += bool(hit) verdict = f"detected, delay {hit[0][0] - ts:.2f} y" if hit else "missed" label = fmt(first_c) + (f" → {fmt(second)} at t {ts}" if second else "") lines.append(f"| {label} | {seed} | {fmt(b0)} | " - + ("; ".join(f"{t:.2f}: {fmt(a)} → {fmt(b)}" for t, a, b in fl) or "none") + + ("; ".join(f"{t:.2f}: {a[0]} → {b[0]}" for t, a, b in fl) or "none") + f" | {verdict} |") ok = correct >= 5 and false == 0 and det >= 4 lines += ["", f"Anchor correct at start {correct}/6, false flips {false}/6, anchor switches detected {det}/6, " diff --git a/simulations/gadm02/results/matching_imm.md b/simulations/gadm02/results/matching_imm.md index 73214ba..cce1267 100644 --- a/simulations/gadm02/results/matching_imm.md +++ b/simulations/gadm02/results/matching_imm.md @@ -5,7 +5,8 @@ US calibrated state, sentiment anchored (BR-009 fix). Truth from a known candida switching at t 4; observed from t 2 to t 8. Indicators and noise: adoption biweekly (log sd 0.03), investment quarterly (log sd 0.05), discovery channel Pd monthly (sd 0.01), sentiment twice a year (sd 0.02). IMM filter, switch probability 0.1 per quarter. Flip: new best with weight -≥ 0.75 twice in a row. +≥ 0.75 twice in a row, judged on anchor weights (summed over the loop state) and, for the +loop case, on loop weights (summed over anchors). False flips count both. **Acceptance (anchor regimes):** anchor correct at the first evaluation in ≥ 5 of 6 no-switch runs, no false flips in 6 no-switch runs, and ≥ 4 of 6 anchor switches detected within 1.5 years. The @@ -20,14 +21,14 @@ until perceived benefit moves. | B2/loop on | 1 | B2/loop off | none | anchor correct | | B2/loop on | 2 | B2/loop off | none | anchor correct | | B2/loop on | 3 | B2/loop off | none | anchor correct | -| B2/loop off → B4/loop off at t 4.0 | 1 | B2/loop off | 5.00: B2/loop on → B4/loop off | detected, delay 1.00 y | -| B2/loop off → B4/loop off at t 4.0 | 2 | B2/loop off | 5.50: B2/loop off → B4/loop off | detected, delay 1.50 y | -| B2/loop off → B4/loop off at t 4.0 | 3 | B2/loop off | 5.25: B2/loop off → B4/loop off | detected, delay 1.25 y | -| B3/loop off → B2/loop off at t 4.0 | 1 | B3/loop off | 7.75: B3/loop off → B2/loop on | missed | -| B3/loop off → B2/loop off at t 4.0 | 2 | B3/loop off | 7.00: B3/loop off → B2/loop off | missed | -| B3/loop off → B2/loop off at t 4.0 | 3 | B3/loop off | 6.50: B3/loop off → B2/loop off | missed | +| B2/loop off → B4/loop off at t 4.0 | 1 | B2/loop off | 4.50: B2 → B4 | detected, delay 0.50 y | +| B2/loop off → B4/loop off at t 4.0 | 2 | B2/loop off | 4.50: B2 → B4 | detected, delay 0.50 y | +| B2/loop off → B4/loop off at t 4.0 | 3 | B2/loop off | 4.50: B2 → B4 | detected, delay 0.50 y | +| B3/loop off → B2/loop off at t 4.0 | 1 | B3/loop off | 4.50: B3 → B2 | detected, delay 0.50 y | +| B3/loop off → B2/loop off at t 4.0 | 2 | B3/loop off | 4.50: B3 → B2 | detected, delay 0.50 y | +| B3/loop off → B2/loop off at t 4.0 | 3 | B3/loop off | 4.50: B3 → B2 | detected, delay 0.50 y | | B3/loop off → B3/loop on at t 4.0 | 1 | B3/loop off | none | missed | | B3/loop off → B3/loop on at t 4.0 | 2 | B3/loop off | none | missed | | B3/loop off → B3/loop on at t 4.0 | 3 | B3/loop off | none | missed | -Anchor correct at start 6/6, false flips 0/6, anchor switches detected 3/6, loop switches detected 0/3. Acceptance: **not passed**. +Anchor correct at start 6/6, false flips 0/6, anchor switches detected 6/6, loop switches detected 0/3. Acceptance: **passed**. diff --git a/simulations/gadm02/test_matching.py b/simulations/gadm02/test_matching.py index 47d868a..5d56d93 100644 --- a/simulations/gadm02/test_matching.py +++ b/simulations/gadm02/test_matching.py @@ -24,6 +24,13 @@ class TestMatching(unittest.TestCase): self.assertEqual(m.flips([ev(0, 0.9), ev(1, 0.1), ev(2, 0.1)]), [(2, a, b)]) self.assertEqual(m.flips([ev(0, 0.5), ev(1, 0.1), ev(2, 0.1)]), []) # nothing established yet + def test_anchor_flip_ignores_unidentifiable_loop_split(self): + def ev(t, w): + return (t, {("B3", "off"): {"w": w[0]}, ("B2", "off"): {"w": w[1]}, ("B2", "on"): {"w": w[2]}}) + evals = [ev(0, (0.9, 0.05, 0.05)), ev(1, (0.1, 0.45, 0.45)), ev(2, (0.1, 0.45, 0.45))] + self.assertEqual(m.flips(evals), []) # full candidates: split, no flip + self.assertEqual(m.flips(evals, by=0), [(2, ("B3",), ("B2",))]) + if __name__ == "__main__": unittest.main() diff --git a/workplans/ASSEMBLY-WP-0017-reliable-flip-detection.md b/workplans/ASSEMBLY-WP-0017-reliable-flip-detection.md index d31e1b2..42f4aca 100644 --- a/workplans/ASSEMBLY-WP-0017-reliable-flip-detection.md +++ b/workplans/ASSEMBLY-WP-0017-reliable-flip-detection.md @@ -81,12 +81,12 @@ state. ```task id: ASSEMBLY-WP-0017-T04 -status: progress +status: done priority: high state_hub_task_id: "5e4d8270-8499-5635-98d4-55f3186258d2" ``` -**2026-10-13, not passed:** anchors correct 6/6, false flips 0/6, B2 → B4 detected 3/3 within 1.0–1.5 y, B3 → B2 detected 3/3 but only after 2.5–3.75 y (acceptance needs ≥ 4/6 within 1.5 y). Next: a slowdown-sensitive indicator (for example the growth rate of investment, or adoption *intentions*, BTOS Q24) or a longer acceptance window for slowdowns. The criterion must not be loosened without a reason recorded here. +**Done 2026-10-13, passed:** flips judged on anchor weights (the loop split was the cause of the slow detection). Anchors 6/6, false flips 0/6, switches 6/6 after 0.5 y. Caveats: perfect model; without the investment indicator false flips rise to 5/6 (`research/ModelMatching.md` § 4.2). Earlier note: anchors correct 6/6, false flips 0/6, B2 → B4 detected 3/3 within 1.0–1.5 y, B3 → B2 detected 3/3 but only after 2.5–3.75 y (acceptance needs ≥ 4/6 within 1.5 y). Next: a slowdown-sensitive indicator (for example the growth rate of investment, or adoption *intentions*, BTOS Q24) or a longer acceptance window for slowdowns. The criterion must not be loosened without a reason recorded here. Acceptance: at least 5 of 6 regimes correct at the first evaluation, no false flips in 6 no-switch runs, and at least 4 of 6 switches detected within 1.5 @@ -101,7 +101,23 @@ priority: medium state_hub_task_id: "93477fd0-6d90-51dd-a41a-75a22715bf29" ``` -Waits for T04. Then run matching at every full checkpoint (optionally at -light ones), report the current regime and any flip candidates in the +Waits for T06 (T04 passed, but the result rests on an AI-specific investment +series). Then run matching at every full checkpoint (optionally at light ones), report the current regime and any flip candidates in the checkpoint record, and coordinate the runner change with WP-0014 (governed execution profile). + +## AI-specific investment series + +```task +id: ASSEMBLY-WP-0017-T06 +status: todo +priority: high +``` + +The filter's reliability rests on investment (without it, false flips are 5 of +6). L5 covers all information-processing equipment. Find and wire a series +closer to organism investment, for example hyperscaler capital expenditure +(quarterly company filings, summed), BEA data-center structures investment, +or a Census AI-spending question if one exists. Add it to `sources.json` +(if machine-readable) and `observed_series.json`. Then re-run +`matching.py --imm` with this series' realistic noise.