test-driver/docs/TestDriverResearchPrototype.md

# Stage 1 Test Driver Validation

The biggest risk is not technical feasibility. It is that **test-driver becomes conceptually elegant but too broad to prove itself quickly**.

I would improve the odds of success by treating the next phase as an experiment in whether three specific ideas actually work:

1. **A use case can survive implementation change better than a conventional test script.**
2. **Agentic execution can bridge unstable implementation without corrupting the intended semantics.**
3. **Successful agentic tests can crystallize into cheaper deterministic tests.**

If those three work, the rest—energy, campaigns, security mutation, multi-user orchestration—has a strong foundation.

### Narrow the first battlefield

Choose exactly one real application and perhaps three use cases. They should deliberately include the difficult characteristics test-driver is meant to solve: authentication, several users, state transitions, authorization, and an evolving UI/API.

For example:

```text
UC-01  Alice creates a workspace
UC-02  Alice invites Bob and Bob joins
UC-03  Alice revokes Bob and Bob loses access
```

UC-03 already gives you functional, interaction, temporal and security semantics.

Avoid building a generic testing platform first. Make test-driver extraordinarily good at this one sequence.

### Make the semantic layer the core intellectual property

The critical interface is not the LLM integration or browser automation. It is:

```text
UseCase
   ↓
Semantic Goal
   ↓
Semantic Action
   ↓
Concrete realization
```

For example:

```text
grant_access(Bob, resource, READ)
```

may currently mean six browser interactions.

Later it may mean an API operation.

Test-driver should care about the semantic action. Drivers care about realization.

If this abstraction is good, the framework survives technology changes. If it is poor, agentic execution becomes sophisticated screen scraping.

### Force the framework to distinguish discovery from truth

One of the strongest architectural principles should be:

> **Agents discover paths. Oracles establish truth.**

For the first implementation, make every important oracle deterministic.

For example, let the agent discover how Alice invites Bob through the UI, but verify independently through an API or database-facing test interface that:

```text
membership(Bob, Workspace) == MEMBER
membership(Carol, Workspace) == NONE
```

Do not let the actor agent conclude, “It looks like Bob joined.”

This separation will prevent many future problems.

### Build reproducibility before intelligence

For every run, capture at least:

```text
use-case version
scenario version
application version/commit
actor identities and roles
initial world
semantic actions
actual actions
random seed
timestamps
observations
oracle results
screenshots/traces where useful
agent/model/config version
```

An agentically discovered failure that cannot be reproduced is much less valuable.

The first impressive demonstration should therefore not be “the agent found a bug.”

It should be:

> “The agent found a bug, test-driver reduced it to this scenario, and the failure can now be replayed deterministically.”

### Treat crystallization as an explicit deliverable

Don't postpone crystallization until later.

Make the first milestone contain this lifecycle:

```text
new use case
    ↓
agentic execution
    ↓
stable semantic trajectory
    ↓
candidate deterministic implementation
    ↓
deterministic regression
```

You need to learn early whether this transition can actually be automated or assisted effectively.

A useful success metric might be:

> **How many agentic verification assets can be downgraded to deterministic execution without losing semantic coverage?**

That is much more meaningful than counting generated tests.

### Introduce energy only after you have event history

I like Test Energy a lot, but I would avoid optimizing its formula early.

Start by recording events:

```text
found-defect
false-positive
mechanical-adaptation
semantic-adaptation
duplicate-detected
usecase-changed
crystallized
regression-caught
```

Then initially compute a crude score.

After a few hundred runs you can inspect whether the proposed energy changes actually correspond to human intuition about test value.

In other words:

> **Store the evidence first; invent the fitness function second.**

Otherwise you'll encode assumptions before you have data.

### Separate three kinds of change

This will probably become one of the framework's most important capabilities.

Whenever a test stops matching the system, classify the change as:

```text
IMPLEMENTATION CHANGE
same behavior, different realization

INTENT CHANGE
the product is deliberately supposed to behave differently

DEFECT
implementation no longer satisfies unchanged intent
```

This classification is what makes adaptive tests safe.

A lot of test maintenance today implicitly assumes the first case. Test-driver must never make that assumption.

### Give humans a small number of meaningful decision points

Agentic systems fail when humans either have to approve everything or are expected to trust everything.

Aim for perhaps three escalation types:

```text
SEMANTIC CHANGE
"The expected workflow appears to have changed."

INVARIANT CONFLICT
"The observed behavior conflicts with a protected invariant."

AMBIGUOUS VERDICT
"Available evidence cannot establish correctness."
```

Everything mechanical beneath that should eventually be autonomous.

That creates a good human/agent boundary.

### Make security an early proof point

Security testing is especially good for proving the framework because ordinary functional use cases naturally generate adversarial variants.

From:

```text
Alice grants Bob access.
```

derive:

```text
Carol tries access.
Bob attempts WRITE.
Bob accesses before grant.
Bob accesses after revoke.
Bob substitutes another resource ID.
Bob attempts cross-tenant access.
```

If test-driver can systematically produce these from the ordinary use case, you already have something substantially more interesting than “an LLM controlling Playwright.”

### Use a deliberately boring implementation stack

The novelty should be in the verification model, not infrastructure.

For the first implementation, I'd prefer something like:

```text
Python
pytest
Playwright
Pydantic/dataclasses
YAML
SQLite
OpenTelemetry where useful
```

Maybe later a graph store, distributed workers, richer agent infrastructure, etc.

But initially:

> **One process, one database, one browser engine, one application under test.**

Complex infrastructure will hide conceptual flaws.

### Dogfood aggressively

Test-driver itself should become one of the systems tested by test-driver.

As soon as there is a usable CLI or web surface, introduce its own use cases:

```text
define use case
execute campaign
inspect evidence
accept finding
crystallize test
retire verification
```

That gives you continuous pressure to make the framework usable rather than merely architecturally sophisticated.

### Define success quantitatively

I'd track a small scorecard from the first month:

| Measure                                      | Why it matters   |
| -------------------------------------------- | ---------------- |
| Time to express a new use case               | usability        |
| Agentic scenario success rate                | executability    |
| Mechanical adaptations handled automatically | fluidity benefit |
| False semantic adaptations                   | safety           |
| Genuine defects discovered                   | effectiveness    |
| Defects reproducibly replayed                | trustworthiness  |
| Agentic → deterministic crystallizations     | core thesis      |
| Median execution cost per maturity level     | economics        |
| Human interventions per 100 runs             | autonomy         |
| Security variants derived per use case       | leverage         |

The key metric could eventually be something like:

> **Verified behavior per unit of human maintenance effort.**

That captures what test-driver is fundamentally trying to improve.

### Keep the architecture open, but the semantics opinionated

Drivers, agents and storage should be replaceable.

The following should not be casually replaceable:

```text
UseCase
Actor
Scenario
Claim
Invariant
SemanticAction
Observation
Oracle
VerificationAsset
Finding
Adaptation
Crystallization
EnergyEvent
Lineage
```

Those are the conceptual backbone.

If they change constantly, you don't yet understand the domain.

### And establish a ruthless non-goal list

For v0.1 I would explicitly not try to become:

```text
a generic unit-test framework
a load-testing replacement
a fuzzing engine
a vulnerability scanner
a test-management SaaS
a CI/CD system
an observability platform
a universal browser agent
```

Test-driver may eventually integrate all of those.

Its unique purpose should remain:

> **Maintain trustworthy verification of use cases while software changes rapidly, using agentic flexibility where necessary and deterministic verification wherever possible.**

If I were running the project, the next concrete milestone would be a **Test Driver Research Prototype**, not yet a product: one target application, three linked multi-user use cases, deterministic oracles, one agentic browser driver, complete evidence capture, one mutation mechanism, and one successful crystallization into a deterministic regression test.

If that demonstrably works, we will have validated the hardest and most original part of the idea.

xxx