T08: the classifier, measured and attacked
False Adaptation Rate = 0/7 across the labelled catalogue and the three E-003 attacks. 11 of 12 mechanical mutations absorbed without a human, so the safety result is not bought by escalating everything. - classification.py: total function over three signals, rule order chosen so every rule that could excuse a regression sits after the rule that reports one. SAFE_TO_ACCEPT is a two-element closed set, asserted. - CompositeDriver plus scenarios/full_journey.py: one asset crossing both surfaces, so UI mutations are visible as surface differences while the claims they do not touch stay green. - E-003: surface substitution (new M23), concurrent mechanical+defect, evidence starvation, provenance laundering. All held. F-0006 (CONCEPT_DRIFT, resolved): the T02 design listed SEMANTIC_CHANGE as an outcome the table could produce. It cannot - M12 and M19 are behaviourally identical, as the lab has asserted since T05. PRODUCT_DEFECT and SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and the distinction becomes a human adjudication. INTENT_CHANGED survives but is detected by the claim fingerprint moving, not inferred from behaviour. Two classifier defects found and fixed rather than reported: claims downstream of a failed realization now yield INCONCLUSIVE rather than FAIL (a false accusation is the mirror image of a false adaptation), and the browser driver records a page signature so surface change is detectable when the interaction path is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
4b78ce4597
commit
84848e9a0e
26 changed files with 824 additions and 26 deletions
|
|
@ -112,6 +112,7 @@ class Runner:
|
|||
)
|
||||
|
||||
judgments: list[Judgment] = []
|
||||
scenario_sound = True
|
||||
claims_by_step: dict[str, list] = {}
|
||||
for claim in scenario.use_case.claims:
|
||||
claims_by_step.setdefault(claim.after_step, []).append(claim)
|
||||
|
|
@ -171,13 +172,35 @@ class Runner:
|
|||
step.id,
|
||||
)
|
||||
|
||||
# Only an action that did not *happen* makes the scenario unsound.
|
||||
# An action that was accepted but did not take effect did happen —
|
||||
# and that is a statement about the system, judged below, not a
|
||||
# reason to stop judging.
|
||||
if refused and not step.expect_refusal:
|
||||
scenario_sound = False
|
||||
|
||||
# --- invariants after every step -----------------------------
|
||||
# Invariants are judged regardless: they are supposed to hold at all
|
||||
# times, however far the scenario got.
|
||||
for invariant in scenario.use_case.invariants:
|
||||
judgments.append(self._oracle.judge(invariant, snapshot, step.id))
|
||||
|
||||
# --- claims attached to this step ----------------------------
|
||||
# Claims describe the state a *completed* scenario should reach. If a
|
||||
# step did not happen, a claim about the state after it is not
|
||||
# failing — it is unevaluable. Reporting FAIL there would accuse the
|
||||
# system of a defect on the strength of the test's own inability to
|
||||
# act, which is the mirror image of a false adaptation and just as
|
||||
# dishonest.
|
||||
for claim in claims_by_step.get(step.id, ()):
|
||||
judgments.append(self._oracle.judge(claim, snapshot, step.id))
|
||||
if scenario_sound:
|
||||
judgments.append(self._oracle.judge(claim, snapshot, step.id))
|
||||
else:
|
||||
judgments.append(Judgment(
|
||||
claim.id, claim.text, Verdict.INCONCLUSIVE, step.id,
|
||||
{"reason": "an earlier step in this scenario did not complete, "
|
||||
"so the state this claim describes was never reached"},
|
||||
))
|
||||
|
||||
pack.verdicts = [j.as_dict() for j in judgments]
|
||||
pack.finished_at = datetime.now(timezone.utc).isoformat()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue