tegwick cf682281d7 Complete local generalisation tasks and document remaining experiment blockers

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4

2026-09-28 12:06:24 +02:00

2.1 KiB

Raw Blame History

title

status

hypotheses

task

created

E-001

Mechanical recovery and defect discrimination over the labelled mutation set

EXECUTED

H-001

H-002

H-004

TD-WP-0002-T08

2026-08-22

E-001 — Mechanical recovery and defect discrimination

Design

Run the same verification asset against every labelled lab mutation (T05), in two arms:

arm A — semantic action realized by the agentic driver;
arm B — a recorded selector-based sequence captured against the baseline, built with the most robust selector strategy reasonably available.

Arm B is the control for H-001 and must actually be run.

Outcomes recorded per run

surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts (S3) · classification emitted · claim/invariant diff (expected empty) · tokens · wall time · retries.

Decision rules

False Adaptation Rate > 0 → stop. H-004 falsified; redesign, do not tune.
Arm B recovery ≥ arm A recovery → H-001 falsified.
Recovery achieved with a non-empty claim diff → H-002 falsified and a framework defect, since D-02 forbids the write path.

Status

EXECUTED 2026-08-22 (T08). Arm A run over 23 mutations; arm B over the 12 mechanical ones. Results in H-001 and H-004. FAR 0/7.

Expanded measurement — 2026-09-28 (TD-WP-0003-T03)

38 mutations, including ten that drop test ids and seven new paired structural controls. Discovery and recorded selectors both pass 17/17 preserved-id cases; on dropped ids discovery passes 9/10 and recorded selectors 0/10. M22 remains unrealizable. The full journey accepts 26/27 mechanical cases; FAR remains 0/7 on the unchanged defect set. Claim/provenance indices remain unchanged.

Reproduce: PYTHONPATH=src:. python3 tools/measure_e001.py /tmp/e001.json. Raw result. These selected and correlated synthetic HTML cases do not establish population reliability, visual/browser coverage, model capability or economics. See review for the mechanism list and limits. H-001 remains supported only in the narrowed identifier-loss setting.

2.1 KiB Raw Blame History