diff --git a/docs/TestDriverClassificationDesign.md b/docs/TestDriverClassificationDesign.md new file mode 100644 index 0000000..d65d619 --- /dev/null +++ b/docs/TestDriverClassificationDesign.md @@ -0,0 +1,191 @@ +# Adaptation Classification and Intent Provenance + +**Status:** v0.1 — design decision +**Workplan:** `TD-WP-0002-T02` +**Supersedes nothing.** Extends `TestDriverConceptModel.md` §§ 2.3, 7.8, 9.4. + +This note answers the two questions that gate every implementation task in +`TD-WP-0002`. Both were unanswered in the concept corpus, and both are cheaper to +answer on paper than to discover at canonical M6. + +--- + +## Part A — The classifier + +### The question + +What **deterministic evidence** separates *"the sharing button moved"* from +*"Bob can still read R after revocation"*? + +The concept corpus assumes such a discriminator exists but never names it. If the +honest answer were "a model decides", Concept Model § 2.3 would be violated and +the framework's central safety claim would collapse. + +### The reframe + +The question as posed is harder than the question the framework actually has to +answer, because it presumes a single classifier looking at an undifferentiated +failure and deciding what kind it is. + +Instead, **evidence is stratified**, and each stratum has a different authority: + +| Stratum | Content | Gathered by | May a model touch it? | +|---|---|---|---| +| **S1 — Surface** | selectors, DOM, URLs, route shapes, coordinates, screenshots | the actor, incidentally, while realizing an action | **yes** — this is mechanics | +| **S2 — Realization** | did the semantic action's declared postcondition hold; which surface was actually used | an Observer, out-of-band from the actor | no | +| **S3 — Judgment** | claims and invariants evaluated against domain-state observations | Observers + Oracles, independent of the actor | **never** | + +S1 is how. S2 is whether it happened. S3 is whether that was correct. + +### The structural rule + +> **A change in S1 can never license a change in S3.** + +This is enforced architecturally rather than by classifier accuracy: + +- **Claims and invariants are inputs to a run, not outputs of it.** The adaptation + machinery has no write path to them. There is no code path by which a retry, a + re-realization, or a learned trajectory can alter what is asserted. +- **Adaptation is confined to the realization loop.** It may change *how* a + semantic action is performed. It cannot change *what follows from* performing it. +- Therefore the classifier is never asked "is this failure acceptable?" It is only + asked "why did this run fail to reach the point of judgment?" + +This makes `False Adaptation Rate = 0` an **architectural property, not a tuning +target**. The system does not merely try hard to avoid normalizing a defect — it +cannot express the operation. That matters because a rate driven near zero by +tuning regresses silently; a rate that is zero by construction does not. + +### The decision table + +Classification is a total function over three observed signals. Baseline refers +to the last accepted run of the same verification asset. + +| S1 surface differs | S2 postcondition met | S3 verdicts vs baseline | Classification | +|---|---|---|---| +| yes | yes | unchanged | `MECHANICAL_ADAPTATION` | +| no | yes | changed to FAIL | `PRODUCT_DEFECT` | +| yes | yes | changed to FAIL | `PRODUCT_DEFECT` | +| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human | +| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` | +| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` | +| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate | + +Two rows carry the whole safety argument: + +- **Row 3** — a surface change occurring *alongside* a verdict change is classified + as a defect, never as an adaptation. Coincidence is not exoneration. This is the + row that a naive self-healing tool gets wrong, and it is where the seeded M05 + authorization defect must land even when it ships together with M01/M02. +- **Row 7** — missing or contradictory evidence yields `AMBIGUOUS`, never a + default-to-pass and never a default-to-adapt. `INCONCLUSIVE` already exists as a + first-class verdict; this row is what gives it teeth. + +### Realization legitimacy + +S2 records not only *whether* the postcondition held but *which surface produced +it*. A semantic action declares the surfaces it may legitimately use; an actor +that achieves `grant_access` by calling the HTTP API when the scenario constrains +it to the browser has not realized the action — it has performed an unrequested +**surface-substitution**, which is itself one of the catalogued security +mutations. + +Without this check, an agent can "recover" from a UI authorization control by +routing around it and the run would score as a successful mechanical adaptation. +That is the most plausible concrete route to a false adaptation, so it is checked +explicitly rather than assumed away. + +### Where model judgment is and is not used + +Concept Model § 2.3 is satisfied, and the boundary is now precise: + +- **Used:** locating a moved control, proposing a path through a changed DOM, + summarizing a trajectory. All S1. All independently verified afterwards by S2. +- **Not used:** evaluating a claim, evaluating an invariant, producing a verdict, + deciding whether an adaptation was legitimate, deciding whether a run passed. + +A model can be wrong about where the button is; the run then fails at S2 and is +retried or reported. A model is never in a position to be wrong about whether Bob +could still read R. + +### What this costs + +The design requires an out-of-band observer capable of evaluating every semantic +action's postcondition and every claim without going through the actor's surface. +In the lab this is a direct database or admin-API reader. For a real system under +test it is the main integration burden the framework imposes, and it should be +stated plainly to any future adopter: **test-driver requires an independent +observation channel.** Without one, S2 and S3 collapse into S1 and the safety +argument is void. + +This is a real limitation and is recorded as such rather than discovered later. + +--- + +## Part B — Intent provenance + +### The question + +If an agent writes the implementation, an agent writes the use case, and an agent +realizes the test, then "oracles are independent from actors" is *procedurally* +true and *epistemically* empty. Independence of components does not give +independence of belief when all components derive from the same source. + +### The rule + +> A claim may only be authored by a source **causally independent of the +> implementation it constrains.** + +`UseCase` and `Claim` therefore carry a required `provenance` field: + +| Value | Meaning | Admissible as a claim? | +|---|---|---| +| `human` | authored by a person | yes | +| `spec` | derived from a specification artifact that demonstrably predates the implementation (ref + commit SHA recorded) | yes | +| `agent-from-spec` | an agent transcribed a human or spec artifact; a human accepted the result | yes, with the accepting identity recorded | +| `agent-from-implementation` | an agent derived it by observing what the system does | **no** | + +`agent-from-implementation` is not forbidden as an activity — it is genuinely +useful for T0 exploration. It is forbidden as a **claim**. Such output enters the +system as an *exploratory hypothesis*, which can drive runs and accumulate +evidence but cannot produce a `FAIL`, cannot enter an invariant, and cannot be +crystallized. Promotion to a claim requires an explicit human acceptance event, +recorded with identity and timestamp. + +Without this rule, the framework's guarantee reduces to "the implementation +agrees with itself", which is exactly the failure the project exists to prevent. + +### Consequences + +- Provenance is carried into every Evidence Pack, so any verdict can later be + audited for independence. A `FAIL` whose claim has inadmissible provenance is + itself an `EVIDENCE_FAILURE`. +- Crystallization preserves provenance. A deterministic test generated from an + agentic ancestor inherits the ancestor's claim provenance and cannot launder it. +- A fourth self-test joins the M3 set: `td://self/intent-independence` — asserting + that no admissible claim in the corpus has implementation-derived provenance. + +### Honest limitation + +This rule constrains *provenance*, not *quality*. A human can write a claim that +merely restates what they saw the system do. The rule makes the independence +question answerable and auditable; it does not make it automatic. That residual +is accepted for the spike and revisited at T10. + +--- + +## Decisions recorded + +- **D-01** Evidence is stratified S1/S2/S3; adaptation may write only to S1. +- **D-02** Claims and invariants are run inputs; no adaptation path may modify them. +- **D-03** Classification is a total function over (S1 differs, S2 met, S3 vs baseline), + defaulting to `AMBIGUOUS` on missing evidence. +- **D-04** A surface change concurrent with a verdict change classifies as + `PRODUCT_DEFECT`, never `MECHANICAL_ADAPTATION`. +- **D-05** Semantic actions declare permitted surfaces; realization via another + surface is a finding, not a recovery. +- **D-06** Claims require causally independent provenance; + `agent-from-implementation` output is an exploratory hypothesis until a human + promotes it. +- **D-07** test-driver requires an independent observation channel into the system + under test. Stated as an adoption precondition. diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index c7c4363..6af86f9 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -69,6 +69,9 @@ satisfying to build, and impossible to validate at this stage: One process, one database, one browser engine, one application under test. - Novelty belongs in the verification model, never in the infrastructure. - No LLM judgment where a deterministic oracle is available (Concept Model §2.3). +- Evidence stratification S1/S2/S3 and claim provenance per + `docs/TestDriverClassificationDesign.md` (decision + `fef5213f-ce9b-44c2-b327-a0b0ba4b6270`). Adaptation writes only to S1. --- @@ -101,7 +104,7 @@ taxonomy has no class for document-to-document drift. ```task id: TD-WP-0002-T02 -status: todo +status: done priority: high state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63" ``` @@ -122,6 +125,23 @@ survives in each case. Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision. +**Done 2026-08-22.** Design note written; decision +`fef5213f-ce9b-44c2-b327-a0b0ba4b6270` recorded in the hub. + +Outcome in one line: **adaptation may write only to surface evidence (S1); claims +and invariants are run inputs with no adaptation write path.** False Adaptation +Rate = 0 therefore becomes an architectural property rather than a tuning target — +the system cannot express "accept a defect as an adaptation". Classification is a +total function over three signals defaulting to `AMBIGUOUS`; a surface change +concurrent with a verdict change classifies as `PRODUCT_DEFECT`, never as +adaptation. Claims require causally independent provenance. + +Two consequences that change later tasks: +- **D-07** — an independent observation channel into the system under test is now + an explicit adoption precondition, not an implementation detail. T05 must expose + one in the lab; without it S2/S3 collapse into S1. +- A fourth self-test, `td://self/intent-independence`, joins the T06 set. + ## Minimal research control plane ```task