test-driver/docs/TestDriverInitialMilestones.md

TestDriver Research Prototype — Initial Milestones

Status: v0.1
Purpose: Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.

---

Milestone 0 — Research Control Plane

Goal

Make the conceptual development of test-driver explicit, traceable and falsifiable before substantial framework code accumulates.

Implement

• repository structure for concepts, hypotheses, experiments, findings and evidence;
• hypothesis register;
• concept maturity register;
• Concept ↔ Implementation Fitness Map;
• framework finding taxonomy;
• stable identifiers for concepts, hypotheses, experiments and findings;
• lightweight CLI or file conventions for recording decisions.

Suggested structure:

research/
├── hypotheses/
├── experiments/
├── findings/
├── concepts/
└── decisions/

Initial hypotheses

At minimum register:

• H-001 Semantic Action Stability
• H-002 Mechanical Adaptation
• H-003 Crystallization
• H-004 Independent Judgment
• H-005 Verification Energy

Exit Criteria

• every major v0.1 concept has a stable identifier;
• every implemented subsystem can be linked to at least one concept or hypothesis;
• at least one hypothesis is expressed with a falsification condition and planned experiment;
• framework findings can be recorded independently from product findings.

Evidence Produced

The first Concept ↔ Implementation Fitness Map and hypothesis register.

---

Milestone 1 — Deterministic Semantic Kernel

Goal

Prove the core model without agentic complexity.

Implement

Minimal executable representations of:

• UseCase;
• Actor;
• World;
• Scenario;
• SemanticAction;
• Observation;
• Oracle;
• Verdict;
• VerificationAsset;
• Run.

Implement one deterministic driver, preferably HTTP or direct application adapter.

Reference Use Case

Alice owns resource R.
Alice grants Bob READ access.
Bob can read R.
Carol cannot read R.
Alice revokes Bob.
Bob can no longer read R.

Exit Criteria

• the complete use case runs deterministically;
• Alice, Bob and Carol have isolated identities and sessions;
• all important outcomes are judged by independent deterministic oracles;
• a run produces structured evidence;
• the same scenario can be replayed from known initial state.

Evidence Produced

The first reproducible Evidence Pack.

---

Milestone 2 — Test-Driver Lab

Goal

Create a controlled evolutionary environment in which test-driver claims can be deliberately challenged.

Implement

A deliberately small application supporting:

• users;
• tenants/workspaces;
• authentication;
• resources;
• sharing;
• read/write permissions;
• revoke;
• audit history;
• HTTP API;
• minimal browser UI.

Add explicit mutation switches or tagged lab versions.

Initial Mutations

• M01 move/rename sharing control;
• M02 restructure the DOM;
• M03 change compatible API representation;
• M04 introduce an additional legitimate workflow step;
• M05 introduce authorization defect;
• M06 introduce propagation delay.

Exit Criteria

• the reference use case works against the baseline lab;
• each mutation can be enabled reproducibly;
• mutations can be classified as mechanical, semantic or defective;
• baseline and mutated versions retain explicit version identifiers.

Evidence Produced

A repeatable benchmark environment for framework development.

---

Milestone 3 — Self-Verification v0

Goal

Make test-driver test its own foundational guarantees.

Implement

Initial td://self/... verification assets:

td://self/actor-isolation
td://self/oracle-independence
td://self/evidence-reproducibility

The self-tests should operate against observable behavior rather than internal implementation details where practical.

Exit Criteria

• intentionally breaking actor isolation makes the corresponding self-test fail;
• allowing an actor to determine its own verdict makes oracle-independence fail;
• corrupting or omitting required run evidence makes evidence-reproducibility fail;
• each failure creates a Framework Finding.

Evidence Produced

Proof that conceptual regressions can be detected as framework regressions.

---

Milestone 4 — Agentic Realization

Goal

Introduce agentic flexibility only at the realization layer while retaining deterministic truth.

Implement

• browser driver;
• one agentic Actor Runtime;
• strict per-actor context isolation;
• semantic goal → UI realization loop;
• full action/evidence recording;
• bounded navigation and tool permissions.

Use the semantic action:

grant_access(Bob, R, READ)

The agent may discover how to accomplish it through the UI.

The oracle must remain deterministic.

Exit Criteria

• an agent can realize the reference semantic action from intent;
• the actor cannot access another actor's private context;
• deterministic oracles independently establish success/failure;
• agent/model/configuration identity is recorded in evidence;
• failures can be replayed sufficiently to diagnose them.

Evidence Produced

First trustworthy agentic run.

---

Milestone 5 — Mechanical Adaptation

Goal

Demonstrate the core fluid-development thesis.

Experiment

Run the same Verification Asset against:

1. lab baseline;
2. M01 moved/renamed control;
3. M02 changed DOM structure.

The use-case semantics remain unchanged.

Implement

• adaptation detection;
• adaptation classification;
• preservation of semantic action identity;
• adaptation evidence;
• adaptation history on the Verification Asset.

Exit Criteria

• agentic execution recovers from at least two mechanical mutations;
• original deterministic oracles remain unchanged;
• semantic intent is not modified;
• adaptation is classified as mechanical;
• the framework reports an adaptation rather than a product defect.

Success Metric

Mechanical Recovery Rate

Critical Safety Metric

False Semantic Adaptation Rate = 0 for the experiment set.

Evidence Produced

Support or rejection for H-001 and H-002.

---

Milestone 6 — Defect vs. Adaptation Discrimination

Goal

Prove that adaptive testing does not simply learn to accept broken software.

Experiment

Use:

• M01/M02 as legitimate mechanical changes;
• M05 as an authorization defect;
• M04 or M10-style mutation as a deliberate semantic requirement change.

Implement

Classification path:

IMPLEMENTATION CHANGE
INTENT CHANGE
PRODUCT DEFECT
AMBIGUOUS

Add escalation for semantic changes and ambiguity.

Exit Criteria

• mechanical changes adapt without altering claims/invariants;
• authorization defect creates a Product Finding;
• deliberate requirement change creates a Semantic Change finding;
• ambiguous evidence produces INCONCLUSIVE rather than silent adaptation;
• no seeded defect is normalized as adaptation.

Success Metric

Classification precision/recall over controlled mutations.

Evidence Produced

The first meaningful measurement of adaptation safety.

---

Milestone 7 — Crystallization v0

Goal

Show that agentic flexibility can harden into deterministic regression.

Implement

• semantic action trajectory capture;
• stable-realization detection;
• deterministic candidate generation;
• candidate comparison against existing oracle set;
• provenance/lineage from agentic ancestor to deterministic descendant;
• manual acceptance step initially.

Experiment

Run the same agentic realization repeatedly against a stable lab version, crystallize it, then execute without any model involvement.

Exit Criteria

• one agentic Verification Asset produces a deterministic test candidate;
• deterministic execution preserves the relevant claims and oracles;
• the generated/hardened test runs with zero agentic involvement;
• lineage remains visible;
• execution cost is measurably lower than agentic execution.

Success Metrics

• crystallization success rate;
• semantic coverage retained;
• execution cost reduction.

Evidence Produced

Support or rejection for H-003.

---

Milestone 8 — Framework Finding → Improvement Loop

Goal

Close the first actual self-improvement cycle.

Implement

Workflow:

Framework Finding
      ↓
Classification
      ↓
Improvement Hypothesis
      ↓
Controlled Experiment
      ↓
Evidence
      ↓
Accept / Reject
      ↓
Self-Verification / Regression

Use a real framework weakness discovered during Milestones 1–7 rather than inventing one if possible.

Exit Criteria

• a framework finding produces an explicit improvement hypothesis;
• the hypothesis is experimentally evaluated;
• the accepted change links back to concept and evidence;
• the discovered framework failure leaves behind a permanent self-verification or deterministic regression;
• the Concept ↔ Implementation Fitness Map is updated.

Evidence Produced

The first completed Concept–Implementation Fitness Loop.

This is the milestone at which the self-improvement system genuinely exists.

---

Milestone 9 — Verification Energy v0

Goal

Begin measuring test value without prematurely optimizing the scoring model.

Implement

Record immutable Energy Events such as:

DEFECT_DETECTED
REGRESSION_CAUGHT
MECHANICAL_ADAPTATION
SEMANTIC_ADAPTATION
TEST_DEFECT
FALSE_POSITIVE
DUPLICATE
CRYSTALLIZED
USECASE_DEPRECATED

Initially calculate only a simple transparent score.

Exit Criteria

• Energy is derived from event history rather than stored as unexplained state;
• every score change is explainable;
• Energy can influence campaign priority;
• criticality can override retirement;
• no automatic deletion is implemented yet.

Evidence Produced

A dataset suitable for later testing whether Energy actually predicts verification value.

---

Milestone 10 — First Compression Review

Goal

Prevent the research prototype from turning into premature platform architecture.

Review

Ask:

• Which concepts have no supporting evidence?
• Which implementation abstractions have no conceptual justification?
• Which metadata has not informed a decision?
• Which agentic behavior can now be deterministic?
• Which capabilities can be merged or removed?
• What have the experiments falsified?

Exit Criteria

• at least one simplification is seriously evaluated;
• rejected concepts are marked as such rather than silently retained;
• architecture reflects experimental learning;
• updated Concept Model and Improvement Loop remain smaller or more precise where evidence permits.

Evidence Produced

The first proof that self-improvement includes subtraction, not only accumulation.

---

Recommended Execution Order

M0  Research Control Plane
 |
M1  Deterministic Semantic Kernel
 |
M2  Test-Driver Lab
 |
M3  Self-Verification v0
 |
M4  Agentic Realization
 |
M5  Mechanical Adaptation
 |
M6  Defect vs Adaptation
 |
M7  Crystallization
 |
M8  Closed Improvement Loop
 |
M9  Verification Energy
 |
M10 Compression Review

The first major research gate is M8.

Before M8, test-driver has promising mechanisms.

At M8, it has demonstrated a complete evidence-driven self-improvement cycle.

---

Prototype Success Gate

The initial research prototype should be considered successful enough to justify broader framework investment when it can demonstrate all of the following in one coherent system:

1. a multi-user use case expressed independently of implementation details;
2. deterministic independent oracles;
3. agentic realization of at least one semantic action;
4. recovery from legitimate mechanical implementation change;
5. rejection of a seeded semantic/security defect as a mere adaptation;
6. reproducible evidence;
7. crystallization into deterministic execution;
8. a framework failure converted into a permanent self-verification;
9. explicit concept-to-implementation traceability;
10. measured human effort and execution cost.

That demonstration is more valuable than broad feature coverage.