2026-08-22 23:11:21 +02:00
|
|
|
---
|
|
|
|
|
id: E-001
|
|
|
|
|
title: Mechanical recovery and defect discrimination over the labelled mutation set
|
2026-09-28 12:06:24 +02:00
|
|
|
status: EXECUTED
|
2026-08-22 23:11:21 +02:00
|
|
|
hypotheses: [H-001, H-002, H-004]
|
|
|
|
|
task: TD-WP-0002-T08
|
|
|
|
|
created: "2026-08-22"
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# E-001 — Mechanical recovery and defect discrimination
|
|
|
|
|
|
|
|
|
|
## Design
|
|
|
|
|
|
|
|
|
|
Run the same verification asset against every labelled lab mutation (T05),
|
|
|
|
|
in two arms:
|
|
|
|
|
|
|
|
|
|
- **arm A** — semantic action realized by the agentic driver;
|
|
|
|
|
- **arm B** — a recorded selector-based sequence captured against the baseline,
|
|
|
|
|
built with the most robust selector strategy reasonably available.
|
|
|
|
|
|
|
|
|
|
Arm B is the control for H-001 and must actually be run.
|
|
|
|
|
|
|
|
|
|
## Outcomes recorded per run
|
|
|
|
|
|
|
|
|
|
surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts
|
|
|
|
|
(S3) · classification emitted · claim/invariant diff (expected empty) ·
|
|
|
|
|
tokens · wall time · retries.
|
|
|
|
|
|
|
|
|
|
## Decision rules
|
|
|
|
|
|
|
|
|
|
- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune.
|
|
|
|
|
- Arm B recovery ≥ arm A recovery → H-001 falsified.
|
|
|
|
|
- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a
|
|
|
|
|
framework defect, since D-02 forbids the write path.
|
|
|
|
|
|
|
|
|
|
## Status
|
|
|
|
|
|
T08: the classifier, measured and attacked
False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003
attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety
result is not bought by escalating everything.
- classification.py: total function over three signals, rule order chosen so
every rule that could excuse a regression sits after the rule that reports
one. SAFE_TO_ACCEPT is a two-element closed set, asserted.
- CompositeDriver plus scenarios/full_journey.py: one asset crossing both
surfaces, so UI mutations are visible as surface differences while the
claims they do not touch stay green.
- E-003: surface substitution (new M23), concurrent mechanical+defect,
evidence starvation, provenance laundering. All held.
F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an
outcome the table could produce. It cannot - M12 and M19 are behaviourally
identical, as the lab has asserted since T05. PRODUCT_DEFECT and
SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and
the distinction becomes a human adjudication. INTENT_CHANGED survives but is
detected by the claim fingerprint moving, not inferred from behaviour.
Two classifier defects found and fixed rather than reported: claims downstream
of a failed realization now yield INCONCLUSIVE rather than FAIL (a false
accusation is the mirror image of a false adaptation), and the browser driver
records a page signature so surface change is detectable when the interaction
path is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-23 00:02:58 +02:00
|
|
|
`EXECUTED` 2026-08-22 (T08). Arm A run over 23 mutations; arm B over
|
|
|
|
|
the 12 mechanical ones. Results in H-001 and H-004. FAR 0/7.
|
2026-09-28 12:06:24 +02:00
|
|
|
|
|
|
|
|
## Expanded measurement — 2026-09-28 (TD-WP-0003-T03)
|
|
|
|
|
|
|
|
|
|
38 mutations, including ten that drop test ids and seven new paired structural
|
|
|
|
|
controls. Discovery and recorded selectors both pass 17/17 preserved-id cases;
|
|
|
|
|
on dropped ids discovery passes 9/10 and recorded selectors 0/10. M22 remains
|
|
|
|
|
unrealizable. The full journey accepts 26/27 mechanical cases; FAR remains 0/7
|
|
|
|
|
on the unchanged defect set. Claim/provenance indices remain unchanged.
|
|
|
|
|
|
|
|
|
|
Reproduce: `PYTHONPATH=src:. python3 tools/measure_e001.py /tmp/e001.json`.
|
|
|
|
|
[Raw result](../evidence/2026-09-28-e001.json).
|
|
|
|
|
These selected and correlated synthetic HTML cases do not establish population
|
|
|
|
|
reliability, visual/browser coverage, model capability or economics. See
|
|
|
|
|
[review](../../docs/TestDriverGeneralisationReview.md) for the mechanism list and
|
|
|
|
|
limits. H-001 remains supported only in the narrowed identifier-loss setting.
|