tegwick 84848e9a0e T08: the classifier, measured and attacked

False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003
attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety
result is not bought by escalating everything.

- classification.py: total function over three signals, rule order chosen so
  every rule that could excuse a regression sits after the rule that reports
  one. SAFE_TO_ACCEPT is a two-element closed set, asserted.
- CompositeDriver plus scenarios/full_journey.py: one asset crossing both
  surfaces, so UI mutations are visible as surface differences while the
  claims they do not touch stay green.
- E-003: surface substitution (new M23), concurrent mechanical+defect,
  evidence starvation, provenance laundering. All held.

F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an
outcome the table could produce. It cannot - M12 and M19 are behaviourally
identical, as the lab has asserted since T05. PRODUCT_DEFECT and
SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and
the distinction becomes a human adjudication. INTENT_CHANGED survives but is
detected by the claim fingerprint moving, not inferred from behaviour.

Two classifier defects found and fixed rather than reported: claims downstream
of a failed realization now yield INCONCLUSIVE rather than FAIL (a false
accusation is the mirror image of a false adaptation), and the browser driver
records a page signature so surface change is detectable when the interaction
path is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39

2026-08-23 00:02:58 +02:00

3.1 KiB

Raw Blame History

title

status

created

experiments

concepts

H-004

Independent Judgment

EXPERIMENTING

2026-08-22

E-001

E-003

C-oracle-independence

C-intent-provenance

H-004 — Independent Judgment

Claim

Separating actor execution from deterministic oracles reduces false-positive adaptation to defective behaviour.

Falsification condition

The framework classifies any seeded DEFECT-labelled mutation as MECHANICAL_ADAPTATION — i.e. False Adaptation Rate > 0 over the labelled set.

This is the project's existential safety metric. A non-zero result is a stop-and-redesign signal, not a tuning exercise.

Measurement

False Adaptation Rate over DEFECT-labelled mutations. Target: exactly 0.
Classification precision/recall over the full labelled set.
Rate of AMBIGUOUS escalation — tracked as a cost, not as a failure. A system that escalates everything is safe and useless.

Relationship to the design

docs/TestDriverClassificationDesign.md argues that FAR = 0 follows architecturally from D-02 (claims are run inputs with no adaptation write path) rather than from classifier accuracy. This hypothesis therefore tests the architecture, not a model.

The most plausible route to falsification is not a misclassification but surface-substitution: an actor routing around a broken authorization control via another surface and scoring as a successful recovery. D-05 exists to close that route; E-003 exists to attack it deliberately.

Result (TD-WP-0002-T08)

False Adaptation Rate = 0/7. No DEFECT-labelled mutation was classified as safe to accept, including the three deliberate attacks in E-003.

Ground truth	Accepted without a human
MECHANICAL (12)	11
SEMANTIC (4)	2 — both genuinely inert for this scenario
DEFECT (7)	0

E-003 attacks, all held:

surface substitution (M23, UI control removed, API left open) → AMBIGUOUS. The driver did not route around; discovery failed loudly.
concurrent mechanical + defect (M01+M15, M02+M17, M21+M20) → BEHAVIOUR_CHANGED, with the reason explicitly noting that the coincident surface change does not excuse it.
evidence starvation → AMBIGUOUS, never a pass.
provenance laundering → rejected at authoring, and caught in the record.

Note what this does and does not establish. FAR = 0 follows largely from architecture — claims are run inputs with no adaptation write path (D-02), and SAFE_TO_ACCEPT is a two-element closed set. The experiment confirms the architecture behaves as designed over 23 mutations; it does not establish that the architecture is correct for mutations nobody thought of. That distinction should survive into any external claim.

The other side of the trade is asserted too: a classifier that escalated everything would score a perfect FAR and be useless. test_mechanical_changes_are_mostly_absorbed pins 11 of 12.

Status log

2026-08-22 PROPOSED. No evidence.
2026-08-22 EXPERIMENTING. FAR 0/7 over the labelled set plus E-003 attacks. Not promoted to SUPPORTED: 23 hand-written mutations is a small, self-chosen sample.

3.1 KiB Raw Blame History