coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T08: the classifier, measured and attacked

Browse source 
False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003
attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety
result is not bought by escalating everything.

- classification.py: total function over three signals, rule order chosen so
  every rule that could excuse a regression sits after the rule that reports
  one. SAFE_TO_ACCEPT is a two-element closed set, asserted.
- CompositeDriver plus scenarios/full_journey.py: one asset crossing both
  surfaces, so UI mutations are visible as surface differences while the
  claims they do not touch stay green.
- E-003: surface substitution (new M23), concurrent mechanical+defect,
  evidence starvation, provenance laundering. All held.

F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an
outcome the table could produce. It cannot - M12 and M19 are behaviourally
identical, as the lab has asserted since T05. PRODUCT_DEFECT and
SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and
the distinction becomes a human adjudication. INTENT_CHANGED survives but is
detected by the claim fingerprint moving, not inferred from behaviour.

Two classifier defects found and fixed rather than reported: claims downstream
of a failed realization now yield INCONCLUSIVE rather than FAIL (a false
accusation is the mirror image of a false adaptation), and the browser driver
records a page signature so surface change is detectable when the interaction
path is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-23 00:02:58 +02:00
parent 4b78ce4597
commit 84848e9a0e
26 changed files with 824 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/TestDriverClassificationDesign.md
View file

@ -66,11 +66,21 @@ to the last accepted run of the same verification asset.
| yes | yes | unchanged | `MECHANICAL_ADAPTATION` |
| no | yes | changed to FAIL | `PRODUCT_DEFECT` |
| yes | yes | changed to FAIL | `PRODUCT_DEFECT` |
| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human |
| any | yes | unchanged, but the asset's declared claim set differs from the use case | ~~`SEMANTIC_CHANGE` → human~~ → **`INTENT_CHANGED`**, see below |
| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` |
| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` |
| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate |
> **Revised at T08 — see `research/findings/F-0006-classifier-cannot-infer-intent.md`.**
> `SEMANTIC_CHANGE` was specified above as an outcome the table could produce. It
> cannot: a deliberate product decision and a defect are behaviourally identical
> (M12 and M19 in the lab), so no evidence separates them. `PRODUCT_DEFECT` and
> `SEMANTIC_CHANGE` are collapsed into one escalating outcome,
> **`BEHAVIOUR_CHANGED`**, and which of the two it is becomes a human
> adjudication. `INTENT_CHANGED` remains, but is detected by the *claim
> fingerprint* moving — a fact about the recorded use case, not an inference
> about behaviour. The row above described that, filed under the wrong heading.
Two rows carry the whole safety argument:
- **Row 3** — a surface change occurring *alongside* a verdict change is classified