tegwick
84848e9a0e
False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003 attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety result is not bought by escalating everything. - classification.py: total function over three signals, rule order chosen so every rule that could excuse a regression sits after the rule that reports one. SAFE_TO_ACCEPT is a two-element closed set, asserted. - CompositeDriver plus scenarios/full_journey.py: one asset crossing both surfaces, so UI mutations are visible as surface differences while the claims they do not touch stay green. - E-003: surface substitution (new M23), concurrent mechanical+defect, evidence starvation, provenance laundering. All held. F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an outcome the table could produce. It cannot - M12 and M19 are behaviourally identical, as the lab has asserted since T05. PRODUCT_DEFECT and SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and the distinction becomes a human adjudication. INTENT_CHANGED survives but is detected by the claim fingerprint moving, not inferred from behaviour. Two classifier defects found and fixed rather than reported: claims downstream of a failed realization now yield INCONCLUSIVE rather than FAIL (a false accusation is the mirror image of a false adaptation), and the browser driver records a page signature so surface change is detectable when the interaction path is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
93 lines
3.8 KiB
Markdown
93 lines
3.8 KiB
Markdown
---
|
|
id: F-0006
|
|
type: framework-finding
|
|
class: CONCEPT_DRIFT
|
|
status: resolved
|
|
discovered: "2026-08-22"
|
|
resolved: "2026-08-22"
|
|
discovered_by: TD-WP-0002-T08
|
|
workplan: TD-WP-0002
|
|
task: TD-WP-0002-T08
|
|
hypotheses: [H-004]
|
|
---
|
|
|
|
# F-0006 — The classifier cannot infer a semantic change, only escalate
|
|
|
|
## What the design said
|
|
|
|
`docs/TestDriverClassificationDesign.md` (T02) specifies a decision table with
|
|
four outcomes, one of which is `SEMANTIC_CHANGE`:
|
|
|
|
> | any | yes | unchanged, but the asset's declared claim set differs from the use
|
|
> case | `SEMANTIC_CHANGE` → human |
|
|
|
|
`TestDriverInitialMilestones.md` M6 states the same four-way path:
|
|
`IMPLEMENTATION CHANGE` / `INTENT CHANGE` / `PRODUCT DEFECT` / `AMBIGUOUS`.
|
|
|
|
Both read as though a classifier could look at a run and determine that intended
|
|
behaviour had changed.
|
|
|
|
## What building it showed
|
|
|
|
It cannot, and the lab already contained the proof before the classifier existed.
|
|
|
|
**M12** (revocation deferred by product decision, `SEMANTIC`) and **M19**
|
|
(revocation propagates after a delay, `DEFECT`) produce *identical* evidence:
|
|
same failing assertion, same step, same snapshot, same audit trail.
|
|
`test_deferred_revoke_and_revoke_race_are_behaviourally_identical` has asserted
|
|
this since T05.
|
|
|
|
No quantity of evidence distinguishes them, because the difference is not in the
|
|
system. It is in whether someone decided the new behaviour was wanted. A
|
|
classifier that emitted `SEMANTIC_CHANGE` from evidence would be guessing, and
|
|
guessing in the one direction the project cannot afford: `SEMANTIC_CHANGE`
|
|
sounds benign, and "the team must have meant it" is exactly the reasoning that
|
|
normalizes a defect.
|
|
|
|
## Resolution
|
|
|
|
Path 2 — the concept is deliberately revised.
|
|
|
|
The classifier's output space is now:
|
|
|
|
| Outcome | Safe to accept | Meaning |
|
|
|---|---|---|
|
|
| `UNCHANGED` | yes | no observable difference |
|
|
| `MECHANICAL_ADAPTATION` | yes | surface moved, every claim still holds |
|
|
| `BEHAVIOUR_CHANGED` | **no** | a claim that held no longer holds |
|
|
| `INTENT_CHANGED` | **no** | the recorded claim set itself moved |
|
|
| `REALIZATION_FAILED` | **no** | the action could not be performed |
|
|
| `AMBIGUOUS` | **no** | the evidence does not support a conclusion |
|
|
|
|
`BEHAVIOUR_CHANGED` replaces both `PRODUCT_DEFECT` and `SEMANTIC_CHANGE`.
|
|
Which of the two it is remains a real and important question — it is simply a
|
|
question for a human, recorded as an adjudication, not inferred from a run.
|
|
|
|
**`INTENT_CHANGED` is detectable**, but note what makes it so: the *claim
|
|
fingerprint* changed, meaning a human already edited what is being asserted.
|
|
That is a fact about the recorded use case, not an inference about behaviour.
|
|
This is what the T02 table's fourth row was really describing; it was filed under
|
|
the wrong heading.
|
|
|
|
## Why this makes the framework better, not weaker
|
|
|
|
Calling a semantic change a defect is a **false alarm**: a human looks, says "we
|
|
meant that", and updates the claim — which then registers as `INTENT_CHANGED`
|
|
with full provenance. Cost: one review.
|
|
|
|
Calling a defect a semantic change is a **false adaptation**: the thesis dies.
|
|
|
|
Collapsing the two into one escalating outcome makes the framework err only in
|
|
the direction it can afford. The measured consequence is visible in the T08
|
|
matrix: 2 of 4 `SEMANTIC` mutations are escalated as `BEHAVIOUR_CHANGED`, which
|
|
looks like imprecision and is in fact the design working.
|
|
|
|
## Consequences
|
|
|
|
- `docs/TestDriverClassificationDesign.md` decision table updated in place, with
|
|
the original row preserved and marked.
|
|
- `TestDriverInitialMilestones.md` M6's four-way path is superseded by this
|
|
finding; the milestone's *intent* (do not normalize defects) is unchanged and
|
|
is met.
|
|
- H-004's falsification condition is unaffected: False Adaptation Rate over
|
|
`DEFECT`-labelled mutations. Measured **0/7** at T08.
|