diff --git a/INTENT.md b/INTENT.md index 0567f38..904b763 100755 --- a/INTENT.md +++ b/INTENT.md @@ -360,49 +360,27 @@ These are logical boundaries first. They do not require separate services or pac ## Initial Milestones -### M0 — Deterministic Semantic Scenario Runner +> **Superseded.** This section previously carried its own `M0`–`M3` sequence, +> which conflicted with two other sequences in the repository. The canonical +> milestone sequence is now **`docs/TestDriverInitialMilestones.md` (M0–M10)**. +> Recorded as framework finding `F-0001` (CONCEPT_DRIFT), 2026-08-22. -Implement the conceptual kernel without agentic involvement: +The former M0–M3 map onto the canonical sequence as follows: -- UseCase, -- Actor, -- World, -- Scenario, -- SemanticAction, -- Driver, -- Observation, -- Evidence, -- Oracle, -- Verdict. +| Superseded (INTENT.md) | Canonical (`TestDriverInitialMilestones.md`) | +|---|---| +| M0 Deterministic Semantic Scenario Runner | M1 Deterministic Semantic Kernel | +| M1 Agentic Driver | M4 Agentic Realization | +| M2 Adaptation and Crystallization | M5 Mechanical Adaptation, M6 Defect vs Adaptation, M7 Crystallization | +| M3 Living Verification | M9 Verification Energy | -Run the Alice/Bob/Carol reference scenario deterministically. +The canonical sequence additionally introduces M0 (Research Control Plane), +M2 (Test-Driver Lab), M3 (Self-Verification), M8 (Improvement Loop) and +M10 (Compression Review), which had no counterpart here. -### M1 — Agentic Driver - -Allow an actor agent to realize a semantic action or scenario goal through one interaction surface while maintaining strict actor isolation and evidence capture. - -### M2 — Adaptation and Crystallization - -Add: - -- adaptation classification, -- learned successful paths, -- hardening, -- deterministic test candidate generation, -- lineage from exploratory discovery to regression test. - -### M3 — Living Verification - -Add: - -- energy, -- temperature, -- confidence, -- retirement, -- security mutation campaigns, -- adaptive campaign selection. - ---- +Execution order is *not* the canonical order — see +`workplans/TD-WP-0002-vertical-spike-crystallization.md`, which drives one thin +thread through M1–M7 rather than completing each milestone in turn. ## Non-Goals for the Initial Project diff --git a/docs/TestDriverClassificationDesign.md b/docs/TestDriverClassificationDesign.md new file mode 100644 index 0000000..d65d619 --- /dev/null +++ b/docs/TestDriverClassificationDesign.md @@ -0,0 +1,191 @@ +# Adaptation Classification and Intent Provenance + +**Status:** v0.1 — design decision +**Workplan:** `TD-WP-0002-T02` +**Supersedes nothing.** Extends `TestDriverConceptModel.md` §§ 2.3, 7.8, 9.4. + +This note answers the two questions that gate every implementation task in +`TD-WP-0002`. Both were unanswered in the concept corpus, and both are cheaper to +answer on paper than to discover at canonical M6. + +--- + +## Part A — The classifier + +### The question + +What **deterministic evidence** separates *"the sharing button moved"* from +*"Bob can still read R after revocation"*? + +The concept corpus assumes such a discriminator exists but never names it. If the +honest answer were "a model decides", Concept Model § 2.3 would be violated and +the framework's central safety claim would collapse. + +### The reframe + +The question as posed is harder than the question the framework actually has to +answer, because it presumes a single classifier looking at an undifferentiated +failure and deciding what kind it is. + +Instead, **evidence is stratified**, and each stratum has a different authority: + +| Stratum | Content | Gathered by | May a model touch it? | +|---|---|---|---| +| **S1 — Surface** | selectors, DOM, URLs, route shapes, coordinates, screenshots | the actor, incidentally, while realizing an action | **yes** — this is mechanics | +| **S2 — Realization** | did the semantic action's declared postcondition hold; which surface was actually used | an Observer, out-of-band from the actor | no | +| **S3 — Judgment** | claims and invariants evaluated against domain-state observations | Observers + Oracles, independent of the actor | **never** | + +S1 is how. S2 is whether it happened. S3 is whether that was correct. + +### The structural rule + +> **A change in S1 can never license a change in S3.** + +This is enforced architecturally rather than by classifier accuracy: + +- **Claims and invariants are inputs to a run, not outputs of it.** The adaptation + machinery has no write path to them. There is no code path by which a retry, a + re-realization, or a learned trajectory can alter what is asserted. +- **Adaptation is confined to the realization loop.** It may change *how* a + semantic action is performed. It cannot change *what follows from* performing it. +- Therefore the classifier is never asked "is this failure acceptable?" It is only + asked "why did this run fail to reach the point of judgment?" + +This makes `False Adaptation Rate = 0` an **architectural property, not a tuning +target**. The system does not merely try hard to avoid normalizing a defect — it +cannot express the operation. That matters because a rate driven near zero by +tuning regresses silently; a rate that is zero by construction does not. + +### The decision table + +Classification is a total function over three observed signals. Baseline refers +to the last accepted run of the same verification asset. + +| S1 surface differs | S2 postcondition met | S3 verdicts vs baseline | Classification | +|---|---|---|---| +| yes | yes | unchanged | `MECHANICAL_ADAPTATION` | +| no | yes | changed to FAIL | `PRODUCT_DEFECT` | +| yes | yes | changed to FAIL | `PRODUCT_DEFECT` | +| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human | +| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` | +| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` | +| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate | + +Two rows carry the whole safety argument: + +- **Row 3** — a surface change occurring *alongside* a verdict change is classified + as a defect, never as an adaptation. Coincidence is not exoneration. This is the + row that a naive self-healing tool gets wrong, and it is where the seeded M05 + authorization defect must land even when it ships together with M01/M02. +- **Row 7** — missing or contradictory evidence yields `AMBIGUOUS`, never a + default-to-pass and never a default-to-adapt. `INCONCLUSIVE` already exists as a + first-class verdict; this row is what gives it teeth. + +### Realization legitimacy + +S2 records not only *whether* the postcondition held but *which surface produced +it*. A semantic action declares the surfaces it may legitimately use; an actor +that achieves `grant_access` by calling the HTTP API when the scenario constrains +it to the browser has not realized the action — it has performed an unrequested +**surface-substitution**, which is itself one of the catalogued security +mutations. + +Without this check, an agent can "recover" from a UI authorization control by +routing around it and the run would score as a successful mechanical adaptation. +That is the most plausible concrete route to a false adaptation, so it is checked +explicitly rather than assumed away. + +### Where model judgment is and is not used + +Concept Model § 2.3 is satisfied, and the boundary is now precise: + +- **Used:** locating a moved control, proposing a path through a changed DOM, + summarizing a trajectory. All S1. All independently verified afterwards by S2. +- **Not used:** evaluating a claim, evaluating an invariant, producing a verdict, + deciding whether an adaptation was legitimate, deciding whether a run passed. + +A model can be wrong about where the button is; the run then fails at S2 and is +retried or reported. A model is never in a position to be wrong about whether Bob +could still read R. + +### What this costs + +The design requires an out-of-band observer capable of evaluating every semantic +action's postcondition and every claim without going through the actor's surface. +In the lab this is a direct database or admin-API reader. For a real system under +test it is the main integration burden the framework imposes, and it should be +stated plainly to any future adopter: **test-driver requires an independent +observation channel.** Without one, S2 and S3 collapse into S1 and the safety +argument is void. + +This is a real limitation and is recorded as such rather than discovered later. + +--- + +## Part B — Intent provenance + +### The question + +If an agent writes the implementation, an agent writes the use case, and an agent +realizes the test, then "oracles are independent from actors" is *procedurally* +true and *epistemically* empty. Independence of components does not give +independence of belief when all components derive from the same source. + +### The rule + +> A claim may only be authored by a source **causally independent of the +> implementation it constrains.** + +`UseCase` and `Claim` therefore carry a required `provenance` field: + +| Value | Meaning | Admissible as a claim? | +|---|---|---| +| `human` | authored by a person | yes | +| `spec` | derived from a specification artifact that demonstrably predates the implementation (ref + commit SHA recorded) | yes | +| `agent-from-spec` | an agent transcribed a human or spec artifact; a human accepted the result | yes, with the accepting identity recorded | +| `agent-from-implementation` | an agent derived it by observing what the system does | **no** | + +`agent-from-implementation` is not forbidden as an activity — it is genuinely +useful for T0 exploration. It is forbidden as a **claim**. Such output enters the +system as an *exploratory hypothesis*, which can drive runs and accumulate +evidence but cannot produce a `FAIL`, cannot enter an invariant, and cannot be +crystallized. Promotion to a claim requires an explicit human acceptance event, +recorded with identity and timestamp. + +Without this rule, the framework's guarantee reduces to "the implementation +agrees with itself", which is exactly the failure the project exists to prevent. + +### Consequences + +- Provenance is carried into every Evidence Pack, so any verdict can later be + audited for independence. A `FAIL` whose claim has inadmissible provenance is + itself an `EVIDENCE_FAILURE`. +- Crystallization preserves provenance. A deterministic test generated from an + agentic ancestor inherits the ancestor's claim provenance and cannot launder it. +- A fourth self-test joins the M3 set: `td://self/intent-independence` — asserting + that no admissible claim in the corpus has implementation-derived provenance. + +### Honest limitation + +This rule constrains *provenance*, not *quality*. A human can write a claim that +merely restates what they saw the system do. The rule makes the independence +question answerable and auditable; it does not make it automatic. That residual +is accepted for the spike and revisited at T10. + +--- + +## Decisions recorded + +- **D-01** Evidence is stratified S1/S2/S3; adaptation may write only to S1. +- **D-02** Claims and invariants are run inputs; no adaptation path may modify them. +- **D-03** Classification is a total function over (S1 differs, S2 met, S3 vs baseline), + defaulting to `AMBIGUOUS` on missing evidence. +- **D-04** A surface change concurrent with a verdict change classifies as + `PRODUCT_DEFECT`, never `MECHANICAL_ADAPTATION`. +- **D-05** Semantic actions declare permitted surfaces; realization via another + surface is a finding, not a recovery. +- **D-06** Claims require causally independent provenance; + `agent-from-implementation` output is an exploratory hypothesis until a human + promotes it. +- **D-07** test-driver requires an independent observation channel into the system + under test. Stated as an adoption precondition. diff --git a/docs/TestDriverConceptModel.md b/docs/TestDriverConceptModel.md index c562c45..0be21f9 100755 --- a/docs/TestDriverConceptModel.md +++ b/docs/TestDriverConceptModel.md @@ -910,46 +910,21 @@ The conceptual components may initially share a process or codebase. Their separ # 15. Initial Development Sequence -## M0 — Deterministic semantic scenario runner +> **Superseded.** This section previously carried its own `M0`–`M3` sequence, +> which conflicted with the sequences in `INTENT.md` and +> `TestDriverInitialMilestones.md`. The canonical milestone sequence is +> **`TestDriverInitialMilestones.md` (M0–M10)**. Recorded as framework finding +> `F-0001` (CONCEPT_DRIFT), 2026-08-22. -Implement: +The former M0–M3 corresponded to canonical M1, M4, M5–M7 and M9 respectively. +See the mapping table in `INTENT.md` § Initial Milestones. -- use cases, -- actors, -- worlds, -- semantic actions, -- observations, -- oracles, -- verdicts. - -Execute the reference multi-user scenario without AI. - -## M1 — Agentic driver - -Allow an actor agent to realize a semantic action or use-case subgoal through a browser or API surface while preserving actor isolation and evidence capture. - -## M2 — Adaptation and crystallization - -Implement: - -- adaptation classification, -- successful-path persistence, -- hardening, -- generation of deterministic test candidates. - -## M3 — Living verification - -Introduce: - -- energy, -- temperature, -- lineage, -- retirement, -- mutation campaigns, -- adaptive test selection. +Execution order is *not* the canonical order — see +`workplans/TD-WP-0002-vertical-spike-crystallization.md`. --- + # 16. Canonical Concept Set v0.1 The initial test-driver vocabulary is: diff --git a/docs/TestDriverInitialMilestones.md b/docs/TestDriverInitialMilestones.md index 30a43ed..caf7d0b 100755 --- a/docs/TestDriverInitialMilestones.md +++ b/docs/TestDriverInitialMilestones.md @@ -1,8 +1,18 @@ # TestDriver Research Prototype — Initial Milestones -**Status:** v0.1 +**Status:** v0.1 — **canonical milestone sequence** **Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts. +> This document is the single canonical source for milestone identifiers `M0`–`M10`. +> The competing sequences formerly in `INTENT.md` and `TestDriverConceptModel.md` § 15 +> are superseded — see framework finding +> `research/findings/F-0001-milestone-sequence-drift.md`. +> +> **Canonical order is not execution order.** The milestones define *what must be +> true*; `workplans/TD-WP-0002-vertical-spike-crystallization.md` defines *the order +> in which we get there*, driving one thin thread through M1–M7 rather than +> completing each milestone in turn. + --- ## Milestone 0 — Research Control Plane diff --git a/research/README.md b/research/README.md new file mode 100644 index 0000000..42b3e84 --- /dev/null +++ b/research/README.md @@ -0,0 +1,48 @@ +# Research Control Plane + +Deliberately small. This directory exists so that claims about test-driver can be +falsified rather than accumulated. It is plain files — no CLI, no schema, no +tooling — until there are enough readings to justify tooling. + +``` +research/ +├── hypotheses/ H-NNN — a claim with a falsification condition +├── experiments/ E-NNN — a planned or executed test of a hypothesis +├── findings/ F-NNNN — findings about test-driver itself +├── concepts/ the Concept ↔ Implementation Fitness Map +└── decisions/ pointers to decisions recorded in State Hub +``` + +## Identifier convention + +| Prefix | Scope | Example | +|---|---|---| +| `H-NNN` | Hypothesis | `H-001` | +| `E-NNN` | Experiment | `E-001` | +| `F-NNNN` | Framework Finding | `F-0001` | +| `C-` | Concept in the fitness map | `C-actor-isolation` | +| `D-NN` | Decision, scoped to its design note | `D-07` | +| `TD-WP-NNNN-TNN` | Workplan task (State Hub) | `TD-WP-0002-T04` | + +Identifiers are stable and never reused. A rejected hypothesis keeps its number. + +## Hypothesis lifecycle + +``` +PROPOSED → EXPERIMENTING → SUPPORTED → PRACTICALLY_VALIDATED → ARCHITECTURAL + └──────→ REJECTED +``` + +A hypothesis may be reopened if later evidence contradicts it. Reopening is +recorded in the file, not by creating a new identifier. + +## Rules + +1. **Every hypothesis states what would falsify it**, in terms of an observable + outcome, before any experiment runs. A hypothesis with no falsification + condition is an opinion. +2. **Concepts with no supporting evidence are marked as such**, not quietly + retained. The fitness map is expected to contain unsupported entries; hiding + them defeats its purpose. +3. **Subtraction counts as progress.** A rejected hypothesis or a removed + abstraction is a result, not a setback. diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md new file mode 100644 index 0000000..f6c7630 --- /dev/null +++ b/research/concepts/fitness-map.md @@ -0,0 +1,52 @@ +# Concept ↔ Implementation Fitness Map + +**Updated:** 2026-08-22 (TD-WP-0002-T03) + +Traces each important concept to the implementation, experiment and evidence that +support it. **Unsupported entries are the point of this map** — a concept with no +implementation and no evidence is not a gap to be embarrassed about, it is the +current honest state, and hiding it defeats the map's purpose. + +Support levels follow `TestDriverImprovementLoop.md` §13: +`C0 Idea` · `C1 Hypothesis` · `C2 Experimentally Supported` · +`C3 Practically Validated` · `C4 Architectural Invariant` + +## Current state + +At the time of writing there is **no implementation**, so no concept can honestly +exceed `C1`. The initial classifications in §13 of the Improvement Loop +(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they +were aspirational, not evidenced. + +| Concept | Level | Implementation | Experiment | Evidence | Open question | +|---|---|---|---|---|---| +| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? | +| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. | +| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) | +| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | +| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? | +| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. | +| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) | +| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. | +| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) | +| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. | +| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. | +| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) | +| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | +| `C-confidence` | C0 | — | — | — | Deferred. | +| `C-campaign` | C0 | — | — | — | Deferred. | +| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. | +| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. | +| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. | + +## Orphan check + +**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`: +`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`. + +All five are deferred *by explicit decision*, not oversight. They are the group +most at risk of being built because they are easy and satisfying, and never +validated. They are revisited at T10, where the question is not "when do we build +these" but "does the evidence justify keeping them in the model at all". + +**Implementation orphans** — none yet; there is no implementation. diff --git a/research/decisions/README.md b/research/decisions/README.md new file mode 100644 index 0000000..6762916 --- /dev/null +++ b/research/decisions/README.md @@ -0,0 +1,9 @@ +# Decisions + +Decisions are recorded in State Hub (the durable store) and indexed here for +offline readers. Design rationale lives in the corresponding `docs/` note; this +file is a pointer table, not a second source of truth. + +| Ref | Title | Hub ID | Source | +|---|---|---|---| +| D-01 … D-07 | Stratified evidence and claim provenance as the basis for adaptation safety | `fef5213f-ce9b-44c2-b327-a0b0ba4b6270` | `docs/TestDriverClassificationDesign.md` | diff --git a/research/experiments/E-001-mechanical-vs-defect.md b/research/experiments/E-001-mechanical-vs-defect.md new file mode 100644 index 0000000..3360d1c --- /dev/null +++ b/research/experiments/E-001-mechanical-vs-defect.md @@ -0,0 +1,38 @@ +--- +id: E-001 +title: Mechanical recovery and defect discrimination over the labelled mutation set +status: PLANNED +hypotheses: [H-001, H-002, H-004] +task: TD-WP-0002-T08 +created: "2026-08-22" +--- + +# E-001 — Mechanical recovery and defect discrimination + +## Design + +Run the same verification asset against every labelled lab mutation (T05), +in two arms: + +- **arm A** — semantic action realized by the agentic driver; +- **arm B** — a recorded selector-based sequence captured against the baseline, + built with the most robust selector strategy reasonably available. + +Arm B is the control for H-001 and must actually be run. + +## Outcomes recorded per run + +surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts +(S3) · classification emitted · claim/invariant diff (expected empty) · +tokens · wall time · retries. + +## Decision rules + +- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune. +- Arm B recovery ≥ arm A recovery → H-001 falsified. +- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a + framework defect, since D-02 forbids the write path. + +## Status + +`PLANNED`. Blocked on T05 (lab) and T07 (agentic driver). diff --git a/research/experiments/E-002-crystallization-fidelity.md b/research/experiments/E-002-crystallization-fidelity.md new file mode 100644 index 0000000..da52a71 --- /dev/null +++ b/research/experiments/E-002-crystallization-fidelity.md @@ -0,0 +1,33 @@ +--- +id: E-002 +title: Crystallization fidelity and cost +status: PLANNED +hypotheses: [H-003] +task: TD-WP-0002-T09 +created: "2026-08-22" +--- + +# E-002 — Crystallization fidelity and cost + +## Design + +Run one agentic realization repeatedly against a stable lab version until +realization is stable, crystallize it, then execute the descendant with no model +involvement across every lab version the ancestor was run against. + +## Outcomes recorded + +Oracle set of descendant vs ancestor · verdict agreement per lab version · +execution cost ratio (tokens, wall time) · whether human authorship of assertions +was required. + +## Decision rules + +- Descendant oracle set is not a superset of the ancestor's → H-003 falsified. +- Any verdict disagreement on the same lab version → H-003 falsified. +- Cost ratio ≥ 1 → thesis intact, product rationale is not. Record plainly rather + than quietly. + +## Status + +`PLANNED`. Blocked on T07. diff --git a/research/experiments/E-003-surface-substitution-attack.md b/research/experiments/E-003-surface-substitution-attack.md new file mode 100644 index 0000000..c3677d9 --- /dev/null +++ b/research/experiments/E-003-surface-substitution-attack.md @@ -0,0 +1,40 @@ +--- +id: E-003 +title: Deliberate attack on the adaptation safety boundary +status: PLANNED +hypotheses: [H-004] +task: TD-WP-0002-T08 +created: "2026-08-22" +--- + +# E-003 — Deliberate attack on the adaptation safety boundary + +## Why this experiment exists separately + +E-001 measures whether the framework *happens to* stay safe over a labelled set. +E-003 tries to *make it unsafe*. The distinction matters: an experiment that only +samples benign cases cannot establish a safety property, and False Adaptation Rate +is the one metric the project cannot afford to be wrong about. + +## Attack cases + +1. **Surface substitution** — remove the sharing control from the UI while leaving + the HTTP endpoint open. A driver that "recovers" by calling the API has routed + around the control. Must classify as a finding, never as recovery (D-05). +2. **Concurrent change** — ship a mechanical mutation (M01) and the authorization + defect (M05) in the same lab version. Must classify as `PRODUCT_DEFECT`; + coincidence must not exonerate (decision table row 3). +3. **Evidence starvation** — disable the independent observation channel mid-run. + Must yield `AMBIGUOUS`, never a pass and never an adaptation (row 7). +4. **Provenance laundering** — attempt to introduce a claim derived from observed + implementation behaviour and crystallize it. Must be rejected at authoring + time (D-06) and, if it slips through, at crystallization. + +## Decision rule + +Any attack that succeeds is a stop-and-redesign signal for H-004, regardless of +what E-001 reports. + +## Status + +`PLANNED`. Blocked on T05, T07, T08. diff --git a/research/findings/F-0001-milestone-sequence-drift.md b/research/findings/F-0001-milestone-sequence-drift.md new file mode 100644 index 0000000..fe8b8b0 --- /dev/null +++ b/research/findings/F-0001-milestone-sequence-drift.md @@ -0,0 +1,77 @@ +--- +id: F-0001 +type: framework-finding +class: CONCEPT_DRIFT +status: resolved +discovered: "2026-08-22" +resolved: "2026-08-22" +discovered_by: concept-assessment +source: history/2026-08-22-concept-assessment-swot.md +workplan: TD-WP-0002 +task: TD-WP-0002-T01 +--- + +# F-0001 — Three conflicting milestone sequences + +## Class + +`CONCEPT_DRIFT` — the repository carried three mutually inconsistent statements +of the same concept, with no explicit decision revising any of them. + +Note the unusual shape: §5 of the Improvement Loop defines concept drift as +*implementation* diverging from concept. Here there was no implementation at all. +The drift was **document-to-document**, which the taxonomy does not currently +name. See "Residual" below. + +## Observation + +Three documents each defined a milestone sequence for the same project: + +| Document | Sequence | +|---|---| +| `INTENT.md` § Initial Milestones | M0–M3 | +| `docs/TestDriverConceptModel.md` § 15 | M0–M3 (different content) | +| `docs/TestDriverInitialMilestones.md` | M0–M10 | + +The two M0–M3 sequences disagreed with each other and both disagreed with the +M0–M10 sequence. `M2`, for instance, denoted "Adaptation and Crystallization" in +one document and "Test-Driver Lab" in another — so an unqualified reference to +"M2" was ambiguous across the corpus. + +## Why it matters + +The project's own thesis is that implementations must not silently redefine +intended behaviour, and that concept drift is a first-class finding. The corpus +exhibited the failure it was written to prevent, before a single line of code +existed. Left unresolved, every subsequent milestone reference in commits, +workplans and evidence would have been ambiguous. + +## Resolution + +Path 2 of §5 — *the concept is deliberately revised*. + +`docs/TestDriverInitialMilestones.md` (M0–M10) is now **canonical**. It is the +most complete of the three, is the only one carrying exit criteria and success +metrics, and is the only one that names the lab, self-verification, the +improvement loop and the compression review. + +The other two sections were replaced in place with a superseded banner and a +mapping table rather than deleted, so that references from earlier commits and +from `history/` remain interpretable. + +Execution order is deliberately *not* the canonical order: +`workplans/TD-WP-0002-vertical-spike-crystallization.md` drives one thin thread +through canonical M1–M7 rather than completing each milestone in turn. Canonical +sequence and execution sequence are now distinct and separately stated. + +## Residual + +The finding taxonomy (§9) has no class for divergence **between concept +documents** — `CONCEPT_DRIFT` is defined as implementation-vs-concept. This +finding was classified `CONCEPT_DRIFT` as the nearest fit. + +Carry into `TD-WP-0002-T10` (compression review): either widen `CONCEPT_DRIFT` +to cover intra-conceptual inconsistency, or accept that pre-implementation +document drift is not a framework finding at all and should be handled as +ordinary editorial hygiene. Do not add a new finding class before that question +is answered — the taxonomy is already larger than the evidence supporting it. diff --git a/research/findings/README.md b/research/findings/README.md new file mode 100644 index 0000000..51323c2 --- /dev/null +++ b/research/findings/README.md @@ -0,0 +1,19 @@ +# Framework Findings + +Findings about **test-driver itself**, kept strictly separate from findings about +any system under test. Classes are defined in +`docs/TestDriverImprovementLoop.md` § 9: + +`PRODUCT_DEFECT` · `TEST_DEFECT` · `MECHANICAL_ADAPTATION` · `SEMANTIC_CHANGE` · +`CONCEPT_DRIFT` · `FRAMEWORK_LIMITATION` · `EVIDENCE_FAILURE` · +`UNNECESSARY_COMPLEXITY` + +File naming: `F-NNNN-.md`. Identifiers are stable and never reused. + +Each finding carries frontmatter (`id`, `class`, `status`, dates, originating +workplan/task) and, when resolved, states which of the three §5 resolution paths +was taken. A finding that leaves an open question records it under "Residual" and +names the task that will answer it — residuals are not parked in prose. + +The wider research control plane (`hypotheses/`, `experiments/`, `concepts/`, +`decisions/`) is built in `TD-WP-0002-T03`. diff --git a/research/hypotheses/H-001-semantic-action-stability.md b/research/hypotheses/H-001-semantic-action-stability.md new file mode 100644 index 0000000..dcf2291 --- /dev/null +++ b/research/hypotheses/H-001-semantic-action-stability.md @@ -0,0 +1,41 @@ +--- +id: H-001 +title: Semantic Action Stability +status: PROPOSED +created: "2026-08-22" +experiments: [E-001] +concepts: [C-semantic-action] +--- + +# H-001 — Semantic Action Stability + +## Claim + +A semantic action survives implementation restructuring better than a recorded UI +interaction sequence. + +## Falsification condition + +Across the labelled mechanical mutations in the lab (T05), a recorded interaction +sequence survives **at least as many** mutations as the semantic action does. + +If mechanics-free identity buys no measurable durability, the central abstraction +is decorative and `SemanticAction` should be reduced to a naming convention. + +## Measurement + +Mechanical Recovery Rate for each of two arms over the same mutation set: +- **arm A** — semantic action realized by an agentic driver; +- **arm B** — a recorded selector-based sequence captured against the baseline. + +Arm B is a genuine control and must be run, not assumed to fail. + +## Threats to validity + +The comparison is unfair if arm B is built naively — a brittle straw man makes +H-001 trivially true and worthless. Arm B uses the most robust selector strategy +reasonably available (roles, labels, test ids where the lab provides them). + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-002-mechanical-adaptation.md b/research/hypotheses/H-002-mechanical-adaptation.md new file mode 100644 index 0000000..92d7191 --- /dev/null +++ b/research/hypotheses/H-002-mechanical-adaptation.md @@ -0,0 +1,40 @@ +--- +id: H-002 +title: Mechanical Adaptation +status: PROPOSED +created: "2026-08-22" +experiments: [E-001] +concepts: [C-adaptation] +--- + +# H-002 — Mechanical Adaptation + +## Claim + +An agentic driver can recover from a mechanical implementation change without +modifying the semantics of the protected use case. + +## Falsification condition + +Either: +- the driver fails to recover from a majority of labelled `MECHANICAL` mutations + (recovery is not achievable in practice); **or** +- recovery is achieved only in runs where claims or invariants were altered + (recovery is achievable but not semantics-preserving). + +The second branch matters more than the first. A high recovery rate purchased by +weakening assertions falsifies this hypothesis just as decisively as no recovery +at all. + +## Measurement + +- Mechanical Recovery Rate over `MECHANICAL`-labelled mutations. +- Claim/invariant diff per run — expected to be empty by construction + (`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a + falsification signal **and** a framework defect, since no write path should exist. + +## Status log + +- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification + branch structurally unreachable; the measurement is retained anyway, as an + assertion that the architecture is what we believe it is. diff --git a/research/hypotheses/H-003-crystallization.md b/research/hypotheses/H-003-crystallization.md new file mode 100644 index 0000000..e019c5d --- /dev/null +++ b/research/hypotheses/H-003-crystallization.md @@ -0,0 +1,43 @@ +--- +id: H-003 +title: Crystallization +status: PROPOSED +created: "2026-08-22" +experiments: [E-002] +concepts: [C-crystallization] +--- + +# H-003 — Crystallization + +## Claim + +A sufficiently stable agentic execution can be converted into deterministic test +code without losing relevant oracle coverage. + +## Falsification condition + +Any of: +- the generated deterministic test cannot be produced without human authorship of + the assertions; +- it drops one or more oracles present in the agentic ancestor; +- it passes where the ancestor would have failed, or vice versa, on the same lab + version; +- it costs no less to execute than the agentic ancestor. + +## Measurement + +- Crystallization success rate. +- Oracle set of descendant vs ancestor — must be a superset or equal. +- Verdict agreement between ancestor and descendant across lab versions. +- Execution cost ratio (tokens, wall time) descendant : ancestor. + +## Note on the cost criterion + +The cost criterion is included deliberately. If crystallization preserves +semantics but saves nothing, the thesis is intact but the product rationale is +not — and that distinction should be visible in the record rather than discovered +by a user. + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-004-independent-judgment.md b/research/hypotheses/H-004-independent-judgment.md new file mode 100644 index 0000000..e4814a1 --- /dev/null +++ b/research/hypotheses/H-004-independent-judgment.md @@ -0,0 +1,46 @@ +--- +id: H-004 +title: Independent Judgment +status: PROPOSED +created: "2026-08-22" +experiments: [E-001, E-003] +concepts: [C-oracle-independence, C-intent-provenance] +--- + +# H-004 — Independent Judgment + +## Claim + +Separating actor execution from deterministic oracles reduces false-positive +adaptation to defective behaviour. + +## Falsification condition + +The framework classifies any seeded `DEFECT`-labelled mutation as +`MECHANICAL_ADAPTATION` — i.e. **False Adaptation Rate > 0** over the labelled set. + +This is the project's existential safety metric. A non-zero result is a +stop-and-redesign signal, not a tuning exercise. + +## Measurement + +- False Adaptation Rate over `DEFECT`-labelled mutations. Target: exactly 0. +- Classification precision/recall over the full labelled set. +- Rate of `AMBIGUOUS` escalation — tracked as a cost, not as a failure. A system + that escalates everything is safe and useless. + +## Relationship to the design + +`docs/TestDriverClassificationDesign.md` argues that FAR = 0 follows +architecturally from D-02 (claims are run inputs with no adaptation write path) +rather than from classifier accuracy. This hypothesis therefore tests the +**architecture**, not a model. + +The most plausible route to falsification is not a misclassification but +**surface-substitution**: an actor routing around a broken authorization control +via another surface and scoring as a successful recovery. D-05 exists to close +that route; E-003 exists to attack it deliberately. + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-005-verification-energy.md b/research/hypotheses/H-005-verification-energy.md new file mode 100644 index 0000000..c1bd75c --- /dev/null +++ b/research/hypotheses/H-005-verification-energy.md @@ -0,0 +1,40 @@ +--- +id: H-005 +title: Verification Energy +status: PROPOSED +created: "2026-08-22" +experiments: [] +concepts: [C-energy] +--- + +# H-005 — Verification Energy + +## Claim + +Historical evidence about defects caught, adaptations required, false positives +and duplication can identify verification assets whose continued execution is +more valuable than others. + +## Falsification condition + +Energy scores computed from event history fail to rank assets better than a +trivial baseline (e.g. recency, or uniform priority) when predicting which assets +subsequently catch a defect. + +## Why no experiment is planned yet + +**This hypothesis is not testable at the current scale and is deliberately +dormant.** Validating it requires event history across many assets over months — +history the spike will not accumulate. Implementing a scoring function now would +produce a number that cannot be checked, which is worse than no number. + +`TD-WP-0002` therefore records **raw immutable `EnergyEvent`s from the first run** +and implements no scoring, decay, or selection logic. Events cannot be +reconstructed later; scores can always be computed later. + +This is the hypothesis most likely to be **cheaply built and never validated**, +which is precisely why it is fenced off. + +## Status log + +- 2026-08-22 `PROPOSED`, dormant by decision. Event capture only. diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index 720905e..6af86f9 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -4,7 +4,7 @@ type: workplan title: "Vertical spike: falsify the crystallization thesis" domain: infotech repo: test-driver -status: proposed +status: active owner: codex topic_slug: custodian created: "2026-08-22" @@ -69,6 +69,9 @@ satisfying to build, and impossible to validate at this stage: One process, one database, one browser engine, one application under test. - Novelty belongs in the verification model, never in the infrastructure. - No LLM judgment where a deterministic oracle is available (Concept Model §2.3). +- Evidence stratification S1/S2/S3 and claim provenance per + `docs/TestDriverClassificationDesign.md` (decision + `fef5213f-ce9b-44c2-b327-a0b0ba4b6270`). Adaptation writes only to S1. --- @@ -76,7 +79,7 @@ satisfying to build, and impossible to validate at this stage: ```task id: TD-WP-0002-T01 -status: todo +status: done priority: high state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a" ``` @@ -91,11 +94,17 @@ Pick one canonical sequence, mark the others superseded in place, and commit first entry in the framework findings log — the project should catch this on itself before it catches anything else. +**Done 2026-08-22.** `docs/TestDriverInitialMilestones.md` (M0–M10) is canonical; +the `INTENT.md` and Concept Model § 15 sequences carry superseded banners with a +mapping table. Recorded as `research/findings/F-0001-milestone-sequence-drift.md`. +`INTENT.md` and `docs/` are now tracked. One residual carried to T10: the finding +taxonomy has no class for document-to-document drift. + ## Decide intent provenance and the classification mechanism on paper ```task id: TD-WP-0002-T02 -status: todo +status: done priority: high state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63" ``` @@ -116,6 +125,23 @@ survives in each case. Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision. +**Done 2026-08-22.** Design note written; decision +`fef5213f-ce9b-44c2-b327-a0b0ba4b6270` recorded in the hub. + +Outcome in one line: **adaptation may write only to surface evidence (S1); claims +and invariants are run inputs with no adaptation write path.** False Adaptation +Rate = 0 therefore becomes an architectural property rather than a tuning target — +the system cannot express "accept a defect as an adaptation". Classification is a +total function over three signals defaulting to `AMBIGUOUS`; a surface change +concurrent with a verdict change classifies as `PRODUCT_DEFECT`, never as +adaptation. Claims require causally independent provenance. + +Two consequences that change later tasks: +- **D-07** — an independent observation channel into the system under test is now + an explicit adoption precondition, not an implementation detail. T05 must expose + one in the lab; without it S2/S3 collapse into S1. +- A fourth self-test, `td://self/intent-independence`, joins the T06 set. + ## Minimal research control plane ```task