test-driver/docs/TestDriverClassificationDesign.md

# Adaptation Classification and Intent Provenance

**Status:** v0.1 — design decision
**Workplan:** `TD-WP-0002-T02`
**Supersedes nothing.** Extends `TestDriverConceptModel.md` §§ 2.3, 7.8, 9.4.

This note answers the two questions that gate every implementation task in
`TD-WP-0002`. Both were unanswered in the concept corpus, and both are cheaper to
answer on paper than to discover at canonical M6.

---

## Part A — The classifier

### The question

What **deterministic evidence** separates *"the sharing button moved"* from
*"Bob can still read R after revocation"*?

The concept corpus assumes such a discriminator exists but never names it. If the
honest answer were "a model decides", Concept Model § 2.3 would be violated and
the framework's central safety claim would collapse.

### The reframe

The question as posed is harder than the question the framework actually has to
answer, because it presumes a single classifier looking at an undifferentiated
failure and deciding what kind it is.

Instead, **evidence is stratified**, and each stratum has a different authority:

| Stratum | Content | Gathered by | May a model touch it? |
|---|---|---|---|
| **S1 — Surface** | selectors, DOM, URLs, route shapes, coordinates, screenshots | the actor, incidentally, while realizing an action | **yes** — this is mechanics |
| **S2 — Realization** | did the semantic action's declared postcondition hold; which surface was actually used | an Observer, out-of-band from the actor | no |
| **S3 — Judgment** | claims and invariants evaluated against domain-state observations | Observers + Oracles, independent of the actor | **never** |

S1 is how. S2 is whether it happened. S3 is whether that was correct.

### The structural rule

> **A change in S1 can never license a change in S3.**

This is enforced architecturally rather than by classifier accuracy:

- **Claims and invariants are inputs to a run, not outputs of it.** The adaptation
  machinery has no write path to them. There is no code path by which a retry, a
  re-realization, or a learned trajectory can alter what is asserted.
- **Adaptation is confined to the realization loop.** It may change *how* a
  semantic action is performed. It cannot change *what follows from* performing it.
- Therefore the classifier is never asked "is this failure acceptable?" It is only
  asked "why did this run fail to reach the point of judgment?"

This makes `False Adaptation Rate = 0` an **architectural property, not a tuning
target**. The system does not merely try hard to avoid normalizing a defect — it
cannot express the operation. That matters because a rate driven near zero by
tuning regresses silently; a rate that is zero by construction does not.

### The decision table

Classification is a total function over three observed signals. Baseline refers
to the last accepted run of the same verification asset.

| S1 surface differs | S2 postcondition met | S3 verdicts vs baseline | Classification |
|---|---|---|---|
| yes | yes | unchanged | `MECHANICAL_ADAPTATION` |
| no | yes | changed to FAIL | `PRODUCT_DEFECT` |
| yes | yes | changed to FAIL | `PRODUCT_DEFECT` |
| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human |
| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` |
| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` |
| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate |

Two rows carry the whole safety argument:

- **Row 3** — a surface change occurring *alongside* a verdict change is classified
  as a defect, never as an adaptation. Coincidence is not exoneration. This is the
  row that a naive self-healing tool gets wrong, and it is where the seeded M05
  authorization defect must land even when it ships together with M01/M02.
- **Row 7** — missing or contradictory evidence yields `AMBIGUOUS`, never a
  default-to-pass and never a default-to-adapt. `INCONCLUSIVE` already exists as a
  first-class verdict; this row is what gives it teeth.

### Realization legitimacy

S2 records not only *whether* the postcondition held but *which surface produced
it*. A semantic action declares the surfaces it may legitimately use; an actor
that achieves `grant_access` by calling the HTTP API when the scenario constrains
it to the browser has not realized the action — it has performed an unrequested
**surface-substitution**, which is itself one of the catalogued security
mutations.

Without this check, an agent can "recover" from a UI authorization control by
routing around it and the run would score as a successful mechanical adaptation.
That is the most plausible concrete route to a false adaptation, so it is checked
explicitly rather than assumed away.

### Where model judgment is and is not used

Concept Model § 2.3 is satisfied, and the boundary is now precise:

- **Used:** locating a moved control, proposing a path through a changed DOM,
  summarizing a trajectory. All S1. All independently verified afterwards by S2.
- **Not used:** evaluating a claim, evaluating an invariant, producing a verdict,
  deciding whether an adaptation was legitimate, deciding whether a run passed.

A model can be wrong about where the button is; the run then fails at S2 and is
retried or reported. A model is never in a position to be wrong about whether Bob
could still read R.

### What this costs

The design requires an out-of-band observer capable of evaluating every semantic
action's postcondition and every claim without going through the actor's surface.
In the lab this is a direct database or admin-API reader. For a real system under
test it is the main integration burden the framework imposes, and it should be
stated plainly to any future adopter: **test-driver requires an independent
observation channel.** Without one, S2 and S3 collapse into S1 and the safety
argument is void.

This is a real limitation and is recorded as such rather than discovered later.

---

## Part B — Intent provenance

### The question

If an agent writes the implementation, an agent writes the use case, and an agent
realizes the test, then "oracles are independent from actors" is *procedurally*
true and *epistemically* empty. Independence of components does not give
independence of belief when all components derive from the same source.

### The rule

> A claim may only be authored by a source **causally independent of the
> implementation it constrains.**

`UseCase` and `Claim` therefore carry a required `provenance` field:

| Value | Meaning | Admissible as a claim? |
|---|---|---|
| `human` | authored by a person | yes |
| `spec` | derived from a specification artifact that demonstrably predates the implementation (ref + commit SHA recorded) | yes |
| `agent-from-spec` | an agent transcribed a human or spec artifact; a human accepted the result | yes, with the accepting identity recorded |
| `agent-from-implementation` | an agent derived it by observing what the system does | **no** |

`agent-from-implementation` is not forbidden as an activity — it is genuinely
useful for T0 exploration. It is forbidden as a **claim**. Such output enters the
system as an *exploratory hypothesis*, which can drive runs and accumulate
evidence but cannot produce a `FAIL`, cannot enter an invariant, and cannot be
crystallized. Promotion to a claim requires an explicit human acceptance event,
recorded with identity and timestamp.

Without this rule, the framework's guarantee reduces to "the implementation
agrees with itself", which is exactly the failure the project exists to prevent.

### Consequences

- Provenance is carried into every Evidence Pack, so any verdict can later be
  audited for independence. A `FAIL` whose claim has inadmissible provenance is
  itself an `EVIDENCE_FAILURE`.
- Crystallization preserves provenance. A deterministic test generated from an
  agentic ancestor inherits the ancestor's claim provenance and cannot launder it.
- A fourth self-test joins the M3 set: `td://self/intent-independence` — asserting
  that no admissible claim in the corpus has implementation-derived provenance.

### Honest limitation

This rule constrains *provenance*, not *quality*. A human can write a claim that
merely restates what they saw the system do. The rule makes the independence
question answerable and auditable; it does not make it automatic. That residual
is accepted for the spike and revisited at T10.

---

## Decisions recorded

- **D-01** Evidence is stratified S1/S2/S3; adaptation may write only to S1.
- **D-02** Claims and invariants are run inputs; no adaptation path may modify them.
- **D-03** Classification is a total function over (S1 differs, S2 met, S3 vs baseline),
  defaulting to `AMBIGUOUS` on missing evidence.
- **D-04** A surface change concurrent with a verdict change classifies as
  `PRODUCT_DEFECT`, never `MECHANICAL_ADAPTATION`.
- **D-05** Semantic actions declare permitted surfaces; realization via another
  surface is a finding, not a recovery.
- **D-06** Claims require causally independent provenance;
  `agent-from-implementation` output is an exploratory hypothesis until a human
  promotes it.
- **D-07** test-driver requires an independent observation channel into the system
  under test. Stated as an adoption precondition.