# test-driver

## Intent

`test-driver` is a use-case-driven verification framework for integration, end-to-end, multi-user interaction, authorization, security and resilience testing in software systems that evolve through fast and increasingly agentic development cycles.

The project exists to make verification itself adaptive without allowing tests to simply conform to whatever an implementation happens to do.

Its central idea is that tests should **mature together with the behavior they protect**:

> Emerging software is verified with fluid, exploratory and agentic scenarios. As expected behavior and implementation stabilize, those scenarios are progressively specified, hardened and crystallized into deterministic regression tests that require no agentic involvement.

`test-driver` therefore treats a test not primarily as code, but as a **verification asset** with identity, intent, evidence, lineage, maturity, temperature and energy over time.

---

## Why

Modern software development increasingly operates under conditions where:

- product and implementation cycles are very short,
- interfaces and workflows change rapidly,
- multiple services and interaction surfaces participate in one user outcome,
- authorization and tenant boundaries must be verified continuously,
- multi-user behavior produces temporal and concurrency problems that linear tests poorly represent,
- agentic software development can change code faster than conventional test suites can be manually maintained,
- exploratory testing remains valuable but is expensive and difficult to reproduce,
- mature behavior should ultimately be protected by cheap, deterministic regression tests.

Conventional test automation often forces a premature choice between brittle scripted tests and expensive exploratory testing.

`test-driver` aims to provide a continuum between them.

---

## Core Model

The framework starts from **UseCases** describing intended behavior rather than implementation mechanics.

A use case is projected into one or more **VerificationAssets** and exercised through different lenses such as:

- integration,
- journey / end-to-end,
- multi-user interaction,
- security,
- resilience,
- later, scale and performance.

A concrete **Scenario** combines:

```text
UseCase + Actors + World + Schedule + Surfaces + Variant
```

Actors execute through **Drivers** such as browsers, APIs, CLIs or messaging systems.

**Observers** collect evidence independently from actors.

**Oracles** evaluate claims and invariants from that evidence and produce explicit verdicts.

Agentic actors may decide how to accomplish goals or explore alternatives, but they should not normally be the sole authority deciding whether the system behaved correctly.

---

## Verification Evolution

Verification assets progress through a maturity continuum:

```text
T0 Exploratory
T1 Agentic
T2 Adaptive
T3 Specified
T4 Hardened
T5 Deterministic
```

This progression is called **Crystallization**.

A typical lifecycle is:

```text
Explore
  -> Discover
  -> Reproduce
  -> Minimize
  -> Specify
  -> Harden
  -> Crystallize
  -> Deterministic Regression
```

Crystallization is reversible: a major redesign may temporarily require a mature verification asset to become adaptive again.

---

## Temperature

Capabilities and implementations have a **Temperature** representing their degree of change:

```text
HOT   actively being invented
WARM  frequently changing
COOL  stabilizing
COLD  mature / contractual
```

Temperature influences the preferred verification mode:

```text
HOT   -> exploratory / agentic
WARM  -> adaptive
COOL  -> hardened
COLD  -> deterministic
```

The framework should make it natural for tests to crystallize as software cools.

> **Under review since T10 — see `research/findings/F-0008-temperature-may-be-redundant.md`.**
> Crystallization was built and demonstrated without Temperature ever being
> consulted. What actually triggered freezing was *measured stability of
> realization* — the same path found across several runs — which is observed,
> not declared. Temperature is a label a human would have to maintain and keep
> honest. If measured stability is sufficient, Temperature is a concept the
> framework does not need.

---

## Energy

Every verification asset may carry **Energy** representing the current value of retaining and executing it.

Energy increases when a test proves useful, for example by:

- detecting a confirmed defect,
- detecting a security violation,
- preventing recurrence of a previous defect,
- protecting an important and actively changing capability.

Energy decreases when a test creates maintenance cost without sufficient value, for example when it:

- repeatedly requires adaptation to legitimate implementation changes,
- produces false positives,
- becomes flaky,
- duplicates stronger verification,
- protects behavior that is no longer relevant.

Energy is not correctness.

A failing test must not automatically be adapted to the current implementation. A discrepancy may represent an implementation defect, intended requirement change, test defect or ambiguous condition requiring investigation.

Low-energy tests may move from active execution to low-frequency campaigns, archival state and finally retirement.

Critical contractual, regulatory or security invariants may define retirement floors or prohibitions.

---

## Security by Use-Case Mutation

Security testing should not be a disconnected universe of hand-maintained tests.

Ordinary use cases should be transformable into adversarial scenarios through reusable mutations such as:

```text
actor-substitution
resource-substitution
tenant-substitution
sequence-reordering
step-skipping
replay
repetition
concurrency
surface-substitution
invalid-state
privilege-mutation
dependency-failure
```

For example, from:

> Alice shares resource R with Bob.

`test-driver` should be able to derive questions such as:

- Can Carol access R?
- Can Bob write when only read permission was granted?
- Can Bob substitute another resource identifier?
- Can access survive revocation?
- Can a forbidden operation be performed through another surface?
- What happens if grant and revoke race each other?

---

## Multi-User Isolation

Actors must be real independent execution entities from the framework's perspective.

Each actor owns its own:

- identity,
- credentials,
- session,
- permissions,
- private memory,
- known resources,
- interaction surfaces.

The orchestrator may know the whole world, but actors must not implicitly share information merely because the same agent technology is used to execute them.

---

## Semantic Actions

`test-driver` should prefer **SemanticActions** over low-level recorded mechanics.

Examples:

```text
grant_access(Bob, READ)
revoke_access(Bob)
approve_invoice(I)
open_resource(R)
```

An agent may initially discover how a semantic action maps onto a changing user interface.

As the implementation stabilizes, the corresponding driver may acquire a deterministic implementation of the same semantic action.

Semantic actions therefore provide the bridge between agentic exploration and deterministic crystallization.

---

## Evidence and Oracles

Every meaningful run should produce durable evidence sufficient for later verification and diagnosis.

Evidence may include:

- scenario and run identifiers,
- actor and role information,
- system and component versions,
- action timeline,
- screenshots,
- requests and responses,
- domain-state observations,
- audit records,
- logs,
- metrics,
- traces,
- oracle evaluations.

Oracles evaluate explicit claims and invariants and should prefer deterministic evidence where possible.

Initial verdicts are:

```text
PASS
FAIL
INCONCLUSIVE
```

`SUSPICIOUS` was listed here originally and **removed at T10**: no oracle could
produce it, and a verdict nothing can emit is a promise the framework does not
keep. It returns, with an identifier, if a mechanism ever needs it.

---

## Lineage

Every verification asset should retain enough provenance to answer:

> Why does this test exist?

A test may originate from:

- a use case,
- a requirement,
- a previous test,
- a defect,
- a security incident,
- an exploratory finding.

Findings should be able to generate new hardened regression assets.

---

## Campaigns

A **Campaign** selects which verification assets and scenario variants to execute.

Expected early campaign types include:

```text
smoke
regression
release qualification
authorization
tenant isolation
concurrency
resilience
exploratory
```

Future campaign selection may consider:

- test energy,
- use-case criticality,
- changed-system proximity,
- risk,
- time since last execution,
- execution cost,
- previous findings.

This creates an adaptive **Test Metabolism** rather than treating every historical test as equally relevant forever.

---

## First Reference Scenario

The initial end-to-end reference scenario is deliberately multi-user:

> Alice owns resource R. Alice grants Bob read access. Bob can access R. Carol cannot access R. Alice revokes Bob's access. Bob can no longer access R.

The scenario is intended to exercise:

- actor isolation,
- identity and authorization,
- shared state,
- positive and negative verification,
- temporal behavior,
- security mutation,
- evidence collection,
- deterministic oracles,
- agentic realization,
- later crystallization.

---

## Initial Architecture Boundaries

The first implementation should preserve these conceptual boundaries:

```text
UseCase Parser
      |
      v
Scenario Planner
      |
      v
Actor Runtime
      |
      v
Drivers
      |
      v
System Under Test

Observers -> Evidence -> Oracle Engine -> Verdict

Verification Metadata:
  maturity
  temperature
  energy
  confidence
  lineage
```

These are logical boundaries first. They do not require separate services or packages in the initial implementation.

---

## Initial Milestones

> **Superseded.** This section previously carried its own `M0`–`M3` sequence,
> which conflicted with two other sequences in the repository. The canonical
> milestone sequence is now **`docs/TestDriverInitialMilestones.md` (M0–M10)**.
> Recorded as framework finding `F-0001` (CONCEPT_DRIFT), 2026-08-22.

The former M0–M3 map onto the canonical sequence as follows:

| Superseded (INTENT.md) | Canonical (`TestDriverInitialMilestones.md`) |
|---|---|
| M0 Deterministic Semantic Scenario Runner | M1 Deterministic Semantic Kernel |
| M1 Agentic Driver | M4 Agentic Realization |
| M2 Adaptation and Crystallization | M5 Mechanical Adaptation, M6 Defect vs Adaptation, M7 Crystallization |
| M3 Living Verification | M9 Verification Energy |

The canonical sequence additionally introduces M0 (Research Control Plane),
M2 (Test-Driver Lab), M3 (Self-Verification), M8 (Improvement Loop) and
M10 (Compression Review), which had no counterpart here.

Execution order is *not* the canonical order — see
`workplans/TD-WP-0002-vertical-spike-crystallization.md`, which drives one thin
thread through M1–M7 rather than completing each milestone in turn.

## Non-Goals for the Initial Project

The first implementation does **not** aim to:

- replace every existing unit-test framework,
- invent a new browser automation engine,
- make all tests agentic,
- use LLM judgment where deterministic oracles are available,
- automatically rewrite semantic requirements to match implementation,
- exhaustively enumerate every possible scenario permutation,
- build a distributed test cloud before the conceptual model is proven,
- solve large-scale performance testing in the first milestone.

`test-driver` should integrate with mature lower-level testing and automation tools rather than reimplement them unnecessarily.

---

## Design Heuristics

1. **Describe intent before mechanics.**
2. **Keep actor knowledge isolated.**
3. **Prefer semantic actions over UI coordinates or selectors.**
4. **Keep oracles independent from actors.**
5. **Treat security as mutation of normal behavior.**
6. **Allow agents to explore, but harden what becomes known.**
7. **Crystallize stable behavior into deterministic code.**
8. **Retain evidence and lineage for explainability.**
9. **Track the changing value of tests rather than assuming immortality.**
10. **Do not let implementation silently redefine intended behavior.**

---

## Success Criterion

The project succeeds when a new or changing use case can begin with a comparatively fluid behavioral description, be exercised safely with agentic assistance, accumulate evidence and useful variants, and then naturally evolve into reliable deterministic verification as the software stabilizes.

In short:

> **test-driver keeps verification fluid while software is fluid, and turns learned behavior into deterministic confidence when the software cools.**