T08: the classifier, measured and attacked

False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003
attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety
result is not bought by escalating everything.

- classification.py: total function over three signals, rule order chosen so
  every rule that could excuse a regression sits after the rule that reports
  one. SAFE_TO_ACCEPT is a two-element closed set, asserted.
- CompositeDriver plus scenarios/full_journey.py: one asset crossing both
  surfaces, so UI mutations are visible as surface differences while the
  claims they do not touch stay green.
- E-003: surface substitution (new M23), concurrent mechanical+defect,
  evidence starvation, provenance laundering. All held.

F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an
outcome the table could produce. It cannot - M12 and M19 are behaviourally
identical, as the lab has asserted since T05. PRODUCT_DEFECT and
SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and
the distinction becomes a human adjudication. INTENT_CHANGED survives but is
detected by the claim fingerprint moving, not inferred from behaviour.

Two classifier defects found and fixed rather than reported: claims downstream
of a failed realization now yield INCONCLUSIVE rather than FAIL (a false
accusation is the mirror image of a false adaptation), and the browser driver
records a page signature so surface change is detectable when the interaction
path is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39

src/testdriver/browser.py

@@ -99,6 +99,7 @@ class BrowserDriver:
             )
 
         document = Document.parse(body)
+        signature = page_signature(document)
         args = {"resource_id": self._resource_id, **dict(action.args)}
 
         try:
@@ -110,7 +111,7 @@ class BrowserDriver:
                     "action": action.name,
                     "stage": "discovery",
                     "runtime": getattr(self._runtime, "name", "?"),
-                    "page_bytes": len(body),
+                    "page_signature": signature,
                 },
                 raised=f"RealizationFailed: {failure}",
             )
@@ -121,6 +122,7 @@ class BrowserDriver:
             "runtime": plan.metrics.runtime,
             "rationale": plan.rationale,
             "element_path": plan.element_path,
+            "page_signature": signature,
             "target": plan.action,
             "fields": sorted(plan.fields),
             "metrics": plan.metrics.as_dict(),
@@ -136,6 +138,32 @@ class BrowserDriver:
         return Realization(self.surface.id, mechanics)
 
 
+def page_signature(document: Document) -> str:
+    """A coarse, stable shape of the surface as the driver saw it.
+
+    Element counts by tag plus the names of every control. Enough that a
+    reworded label, an extra confirmation step or a changed element type shows up
+    as a surface difference — without being so fine that incidental whitespace
+    makes every run look like an adaptation.
+    """
+    from collections import Counter
+
+    counts = Counter(element.tag for element in document.elements())
+    controls = sorted(
+        element.attrs.get("name", "")
+        for element in document.elements("input", "select", "textarea")
+    )
+    labels = sorted(
+        element.full_text.strip().lower()
+        for element in document.elements("button", "a", "label")
+        if element.full_text.strip()
+    )
+    return json.dumps(
+        {"tags": dict(sorted(counts.items())), "controls": controls, "labels": labels},
+        sort_keys=True,
+    )
+
+
 def record_baseline_selectors(html: str) -> dict[str, str]:
     """Capture the control arm's selectors from a baseline page.
 

src/testdriver/classification.py

@@ -0,0 +1,245 @@
+"""Why did this run differ from the last accepted one?
+
+The centrepiece. Everything else in the framework exists so that this question
+can be answered from evidence rather than guessed.
+
+The classifier is never asked *"is this failure acceptable?"* — it has no
+authority to accept anything, and claims are run inputs it cannot reach (D-02).
+It is asked only *"what kind of difference is this?"*, and the safe answers
+outnumber the convenient ones.
+
+## What the T02 design got wrong
+
+`docs/TestDriverClassificationDesign.md` lists `SEMANTIC_CHANGE` as an outcome
+the decision table can produce. Building it showed that it cannot — see F-0006.
+A deliberate product decision and a defect are behaviourally identical (M12 and
+M19 in the lab prove it), so no amount of evidence separates them. What the
+classifier can honestly say is *"behaviour changed against intent that did not"*,
+and hand that to a human.
+
+Intent change **is** detectable, but only when a human has actually changed the
+intent: the claim fingerprint moves. That is a fact about the recorded use case,
+not an inference about behaviour.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Mapping
+
+
+class Classification(str, Enum):
+    UNCHANGED = "UNCHANGED"
+    """Same surface, same verdicts. Nothing to report."""
+
+    MECHANICAL_ADAPTATION = "MECHANICAL_ADAPTATION"
+    """The surface moved; every protected claim still holds. Safe to accept."""
+
+    BEHAVIOUR_CHANGED = "BEHAVIOUR_CHANGED"
+    """A claim that held no longer holds. Never adapted to; always escalated.
+
+    Whether this is a product defect or a deliberate change of intent is a
+    question about intent, not about evidence, and is resolved by a human.
+    """
+
+    INTENT_CHANGED = "INTENT_CHANGED"
+    """The recorded claim set itself changed. A human has already acted."""
+
+    REALIZATION_FAILED = "REALIZATION_FAILED"
+    """The action could not be performed at all. Not a verdict about the system."""
+
+    AMBIGUOUS = "AMBIGUOUS"
+    """The evidence does not support a conclusion. Escalate; never default."""
+
+
+#: Classifications that permit the run to be accepted without a human.
+SAFE_TO_ACCEPT = frozenset({Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION})
+
+
+@dataclass(frozen=True, slots=True)
+class Signals:
+    """The three strata, reduced to what classification actually needs."""
+
+    surface_differs: bool
+    realization_failed: bool
+    postcondition_met: bool | None
+    verdicts_regressed: bool
+    verdicts_inconclusive: bool
+    claims_differ: bool
+    evidence_incomplete: bool
+
+    def as_dict(self) -> dict[str, Any]:
+        return {
+            "surface_differs": self.surface_differs,
+            "realization_failed": self.realization_failed,
+            "postcondition_met": self.postcondition_met,
+            "verdicts_regressed": self.verdicts_regressed,
+            "verdicts_inconclusive": self.verdicts_inconclusive,
+            "claims_differ": self.claims_differ,
+            "evidence_incomplete": self.evidence_incomplete,
+        }
+
+
+@dataclass(frozen=True, slots=True)
+class Outcome:
+    classification: Classification
+    reason: str
+    signals: Signals
+    regressions: tuple[str, ...] = ()
+
+    @property
+    def safe_to_accept(self) -> bool:
+        return self.classification in SAFE_TO_ACCEPT
+
+    def as_dict(self) -> dict[str, Any]:
+        return {
+            "classification": self.classification.value,
+            "reason": self.reason,
+            "signals": self.signals.as_dict(),
+            "regressions": list(self.regressions),
+        }
+
+
+# --- signal extraction ----------------------------------------------------
+
+
+def _verdict_map(pack: Mapping[str, Any]) -> dict[tuple[str, str], str]:
+    return {(v["assertion_id"], v["step_id"]): v["verdict"] for v in pack["verdicts"]}
+
+
+def _surface_fingerprint(pack: Mapping[str, Any]) -> str:
+    """How the run was performed, reduced to a comparable shape.
+
+    Deliberately excludes timings and free text: those change run to run and
+    would make every run look like a mechanical adaptation.
+    """
+    parts = []
+    for obs in pack["observations"]:
+        if obs["kind"] != "realization":
+            continue
+        mechanics = obs["data"].get("mechanics", {})
+        parts.append({
+            "step": obs["step_id"],
+            "surface": obs["data"].get("surface"),
+            "target": mechanics.get("target") or mechanics.get("operation"),
+            "fields": sorted(mechanics.get("fields") or mechanics.get("arguments") or []),
+            "element_path": mechanics.get("element_path"),
+            "page": mechanics.get("page_signature"),
+        })
+    return json.dumps(parts, sort_keys=True)
+
+
+def _claim_fingerprint(pack: Mapping[str, Any]) -> str:
+    return json.dumps(sorted(pack.get("provenance_index", {}).items()))
+
+
+def extract_signals(baseline: Mapping[str, Any], candidate: Mapping[str, Any]) -> Signals:
+    before, after = _verdict_map(baseline), _verdict_map(candidate)
+
+    regressed = any(
+        after.get(key) == "FAIL" and verdict != "FAIL" for key, verdict in before.items()
+    ) or any(v == "FAIL" for k, v in after.items() if k not in before)
+
+    realization_failed = any(
+        obs["kind"] == "realization" and obs["data"].get("raised")
+        for obs in candidate["observations"]
+    )
+    checks = [
+        obs["data"] for obs in candidate["observations"]
+        if obs["kind"] == "realization_check"
+    ]
+    met: bool | None
+    if not checks:
+        met = None
+    elif any(c.get("postcondition_met") is False for c in checks):
+        met = False
+    elif any(c.get("postcondition_met") is None for c in checks):
+        met = None
+    else:
+        met = True
+
+    return Signals(
+        surface_differs=_surface_fingerprint(baseline) != _surface_fingerprint(candidate),
+        realization_failed=realization_failed,
+        postcondition_met=met,
+        verdicts_regressed=regressed,
+        verdicts_inconclusive=any(v == "INCONCLUSIVE" for v in after.values()),
+        claims_differ=_claim_fingerprint(baseline) != _claim_fingerprint(candidate),
+        evidence_incomplete=not after or not candidate.get("sut_version"),
+    )
+
+
+def regressions(baseline: Mapping[str, Any], candidate: Mapping[str, Any]) -> tuple[str, ...]:
+    before, after = _verdict_map(baseline), _verdict_map(candidate)
+    return tuple(sorted(
+        f"{assertion}@{step}"
+        for (assertion, step), verdict in after.items()
+        if verdict == "FAIL" and before.get((assertion, step)) != "FAIL"
+    ))
+
+
+# --- the decision table ---------------------------------------------------
+
+
+def classify(baseline: Mapping[str, Any], candidate: Mapping[str, Any]) -> Outcome:
+    """Total over the signal space. Order matters, and the order is the safety argument.
+
+    Every rule that could excuse a regression is placed *after* the rule that
+    reports one. A surface change occurring alongside a broken claim is a
+    regression that happens to coincide with a surface change — never a surface
+    change that happens to break a claim.
+    """
+    signals = extract_signals(baseline, candidate)
+    failures = regressions(baseline, candidate)
+
+    # 1. Evidence first. A conclusion drawn from incomplete evidence is worse
+    #    than no conclusion, in either direction.
+    if signals.evidence_incomplete:
+        return Outcome(Classification.AMBIGUOUS,
+                       "the run did not retain enough evidence to classify", signals)
+    if signals.verdicts_inconclusive:
+        return Outcome(Classification.AMBIGUOUS,
+                       "at least one assertion could not be judged from the evidence",
+                       signals, failures)
+
+    # 2. Intent moving is a fact about the recorded use case, not an inference.
+    if signals.claims_differ:
+        return Outcome(Classification.INTENT_CHANGED,
+                       "the recorded claim set differs from the baseline; "
+                       "a human changed what is being asserted", signals, failures)
+
+    # 3. A regression is reported before anything is allowed to explain it away.
+    #    This is decision-table row 3: coincidence is not exoneration.
+    if signals.verdicts_regressed:
+        note = (" — a surface change occurred in the same run and does not excuse it"
+                if signals.surface_differs else "")
+        return Outcome(Classification.BEHAVIOUR_CHANGED,
+                       f"claims that held at baseline no longer hold{note}",
+                       signals, failures)
+
+    # 4. An action the system accepted but did not perform is a behaviour
+    #    change, observed independently of any claim. It is caught here even
+    #    when no claim happens to cover it.
+    if signals.postcondition_met is False:
+        return Outcome(Classification.BEHAVIOUR_CHANGED,
+                       "the action was accepted but its declared effect did not occur",
+                       signals, failures)
+
+    # 5. Could not act at all. Says nothing about whether the system is correct.
+    if signals.realization_failed:
+        return Outcome(Classification.REALIZATION_FAILED,
+                       "the action could not be realized on this surface", signals)
+    if signals.postcondition_met is None:
+        return Outcome(Classification.AMBIGUOUS,
+                       "the action's postcondition could not be evaluated", signals)
+
+    # 6. Only now, with every claim intact and the action verified, may a
+    #    surface difference be called an adaptation.
+    if signals.surface_differs:
+        return Outcome(Classification.MECHANICAL_ADAPTATION,
+                       "the surface changed; every protected claim still holds", signals)
+
+    return Outcome(Classification.UNCHANGED, "no observable difference from baseline",
+                   signals)

src/testdriver/drivers.py

@@ -74,3 +74,33 @@ class DirectDriver:
             return Realization(self.surface.id, mechanics, raised=f"{type(exc).__name__}: {exc}")
         mechanics["result"] = result
         return Realization(self.surface.id, mechanics)
+
+
+class CompositeDriver:
+    """Routes each semantic action to the driver for the surface it permits.
+
+    A real journey crosses surfaces: a share happens in the UI, an audit check
+    against the API. Binding one asset to one driver would force every step
+    through the same door and hide exactly the mechanical changes that only
+    appear on one of them.
+
+    Routing is by the action's *permitted* surfaces (D-05), never by which driver
+    happens to be able to do it. A driver that could perform an action it was not
+    permitted to perform is surface substitution, not convenience.
+    """
+
+    def __init__(self, drivers: dict[str, Any], default: str) -> None:
+        self._drivers = drivers
+        self._default = default
+        self.surface = drivers[default].surface
+
+    def realize(self, actor: Actor, action: SemanticAction) -> Realization:
+        permitted = action.permitted_surfaces or {self._default}
+        for surface_id in sorted(permitted):
+            driver = self._drivers.get(surface_id)
+            if driver is not None:
+                return driver.realize(actor, action)
+        raise UnsupportedAction(
+            f"no driver for any permitted surface of {action.name!r}: "
+            f"{sorted(permitted)}"
+        )

src/testdriver/runner.py

@@ -112,6 +112,7 @@ class Runner:
         )
 
         judgments: list[Judgment] = []
+        scenario_sound = True
         claims_by_step: dict[str, list] = {}
         for claim in scenario.use_case.claims:
             claims_by_step.setdefault(claim.after_step, []).append(claim)
@@ -171,13 +172,35 @@ class Runner:
                 step.id,
             )
 
+            # Only an action that did not *happen* makes the scenario unsound.
+            # An action that was accepted but did not take effect did happen —
+            # and that is a statement about the system, judged below, not a
+            # reason to stop judging.
+            if refused and not step.expect_refusal:
+                scenario_sound = False
+
             # --- invariants after every step -----------------------------
+            # Invariants are judged regardless: they are supposed to hold at all
+            # times, however far the scenario got.
             for invariant in scenario.use_case.invariants:
                 judgments.append(self._oracle.judge(invariant, snapshot, step.id))
 
             # --- claims attached to this step ----------------------------
+            # Claims describe the state a *completed* scenario should reach. If a
+            # step did not happen, a claim about the state after it is not
+            # failing — it is unevaluable. Reporting FAIL there would accuse the
+            # system of a defect on the strength of the test's own inability to
+            # act, which is the mirror image of a false adaptation and just as
+            # dishonest.
             for claim in claims_by_step.get(step.id, ()):
-                judgments.append(self._oracle.judge(claim, snapshot, step.id))
+                if scenario_sound:
+                    judgments.append(self._oracle.judge(claim, snapshot, step.id))
+                else:
+                    judgments.append(Judgment(
+                        claim.id, claim.text, Verdict.INCONCLUSIVE, step.id,
+                        {"reason": "an earlier step in this scenario did not complete, "
+                                   "so the state this claim describes was never reached"},
+                    ))
 
         pack.verdicts = [j.as_dict() for j in judgments]
         pack.finished_at = datetime.now(timezone.utc).isoformat()