Register with State Hub, persist concept assessment, seed first workplan

- history/2026-08-22-concept-assessment-swot.md: SWOT assessment of the
  concept corpus with recommendations for the first workplan
- statehub register: infotech domain, TD-WP prefix, generated AGENTS.md,
  .custodian-brief.md and TD-WP-0001 bootstrap workplan
- .repo-classification.yaml: category research, domain infotech
- SCOPE.md rewritten with real repo boundaries
- TD-WP-0002: vertical spike reordering M0-M10 into one end-to-end thread
  that can falsify the crystallization thesis early
- commit previously untracked INTENT.md and docs/

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-22 22:40:39 +02:00
parent b471bed707
commit 7249c6a403
13 changed files with 3900 additions and 0 deletions

1044
docs/TestDriverConceptModel.md Executable file

File diff suppressed because it is too large Load diff

609
docs/TestDriverImprovementLoop.md Executable file
View file

@ -0,0 +1,609 @@
# TestDriver Improvement Loop
**Status:** Concept v0.1
**Purpose:** Establish a self-improvement loop that keeps the implementation of `test-driver` aligned with its conceptual model while generating evidence about which concepts actually work.
---
## 1. Intent
`test-driver` is itself an evolving software system. Its implementation must therefore be subject to the same principles it applies to systems under test:
- intended behavior must remain distinguishable from implementation details;
- change must generate evidence rather than silently redefine correctness;
- exploratory mechanisms should harden into deterministic mechanisms as understanding grows;
- failures should improve future verification;
- obsolete complexity should be allowed to disappear.
The improvement loop exists to continuously reconcile:
1. **Concept** — what test-driver claims should be true;
2. **Implementation** — what the framework actually does;
3. **Evidence** — what experiments and executions demonstrate.
The loop should make conceptual drift visible and turn important framework failures into durable self-verification assets.
---
## 2. Core Principle
> Every meaningful implementation increment should generate evidence about whether test-driver is becoming a better realization of its own concept.
The framework therefore has two simultaneous feedback loops:
```text
Concept / Intent / Hypotheses
|
v
Select Experiment
|
v
Implement
|
v
Exercise on SUT
|
v
Evidence
|
+----------+----------+
| |
v v
Product Finding Framework Finding
| |
v v
Improve target Improve test-driver
|
v
Reconcile with Concept
|
+-----> next cycle
```
A run may therefore produce findings about the system under test and findings about the verification framework itself.
---
## 3. Development as Experimental Work
Implementation work should increasingly be framed as hypotheses rather than feature requests.
Examples:
### H-001 — Semantic Action Stability
> A semantic action survives implementation restructuring better than a recorded UI interaction sequence.
### H-002 — Mechanical Adaptation
> An agentic driver can recover from a mechanical implementation change without modifying the semantics of the protected use case.
### H-003 — Crystallization
> A sufficiently stable agentic execution can be converted into deterministic test code without losing relevant oracle coverage.
### H-004 — Independent Judgment
> Separating actor execution from deterministic oracles reduces false-positive adaptation to defective behavior.
### H-005 — Verification Energy
> Historical evidence about defects caught, adaptations required, false positives and duplication can identify verification assets whose continued execution is more valuable than others.
Each hypothesis should have:
- an identifier;
- a claim;
- a falsification condition;
- one or more experiments;
- evidence;
- a status;
- resulting implementation or concept changes.
Suggested lifecycle:
```text
PROPOSED
|
v
EXPERIMENTING
|
+------> REJECTED
|
v
SUPPORTED
|
v
PRACTICALLY_VALIDATED
|
v
ARCHITECTURAL
```
A hypothesis may also be reopened if later evidence contradicts it.
---
## 4. ConceptImplementation Fitness Map
Every important concept should become traceable to the implementation and evidence that support it.
Conceptual relationship:
```text
Concept
|
+-- implementation
+-- experiment
+-- evidence
+-- self-verification
+-- unresolved questions
```
Example:
```yaml
concept: actor-isolation
claim: >
One actor must not obtain private state, credentials, observations,
or memory belonging to another actor except through modeled
communication channels.
implementation:
- testdriver/runtime/actor_context.py
experiments:
- H-006
self_verifications:
- td://self/actor-isolation
status: supported
```
The map should expose two forms of drift.
### Conceptual Orphaning
A concept is claimed but has no implementation or verification evidence.
### Implementation Orphaning
A subsystem or abstraction exists without a concept, requirement, or experiment that explains why it is needed.
Both should be visible during review.
---
## 5. Concept Drift
A **Concept Drift Finding** occurs when implementation behavior diverges from an established concept without an explicit decision to revise that concept.
Example:
```text
Concept:
Agents discover paths; independent oracles judge outcomes.
Implementation:
Browser agent declares the scenario successful.
Finding:
CONCEPT_DRIFT
```
A concept-drift finding must resolve in one of three ways:
1. implementation changes to match the concept;
2. the concept is deliberately revised;
3. an experiment demonstrates that the distinction is no longer useful.
Implementation reality must not silently redefine the conceptual model.
---
## 6. Self-Verification
`test-driver` should become a system under test for `test-driver`.
Self-verification use cases use the namespace:
```text
td://self/...
```
Initial candidates:
```text
td://self/actor-isolation
td://self/oracle-independence
td://self/evidence-reproducibility
td://self/mechanical-adaptation
td://self/semantic-change-detection
td://self/crystallization
td://self/test-retirement
```
These scenarios should verify framework-level promises rather than implementation internals whenever possible.
Example:
### `td://self/mechanical-adaptation`
1. execute a stable use case against lab version A;
2. change the UI mechanically without changing semantics;
3. rerun the use case;
4. allow agentic navigation to recover;
5. verify that the same semantic action and oracles remain valid;
6. record the adaptation and evidence.
Expected result:
```text
mechanical implementation change
|
v
adaptation detected
|
v
alternative realization discovered
|
v
semantic action preserved
|
v
original oracle still passes
```
---
## 7. Test-Driver Lab
A purpose-built mutable application should provide controlled evolutionary pressure for the framework.
Suggested repository or module name:
```text
test-driver-lab
```
The lab should be intentionally small but support:
- multiple users;
- organizations or tenants;
- authentication;
- resources;
- sharing;
- permissions;
- simple workflows;
- audit events;
- API interaction;
- browser interaction.
The lab should also support deliberate implementation mutations.
Examples:
```text
M01 move or rename a UI control
M02 replace the DOM structure
M03 change a compatible API representation
M04 add a legitimate workflow step
M05 introduce an authorization defect
M06 introduce eventual-consistency delay
M07 introduce intermittent dependency failure
M08 remove or deprecate a capability
M09 create a concurrency race
M10 change the intended business requirement
```
The purpose is not to build a representative product. It is to create a controlled environment in which claims about test-driver can be falsified.
---
## 8. Dual Mutation
Mutation should operate in two directions.
### Use-Case Mutation
Mutate actor, resource, order, timing, state, or permissions to test the robustness of the system under test.
```text
UseCase Mutation
|
v
tests robustness of application
```
### Implementation Mutation
Mutate the target implementation while holding the use-case semantics constant to test the robustness of test-driver.
```text
Implementation Mutation
|
v
tests robustness of test-driver
```
This duality allows the framework to test both the application and its own verification strategy.
---
## 9. Framework Findings
The framework should maintain finding classes distinct from ordinary product defects.
Initial set:
### PRODUCT_DEFECT
The system under test violates unchanged intent.
### TEST_DEFECT
The verification asset or oracle is incorrect.
### MECHANICAL_ADAPTATION
Implementation mechanics changed while protected semantics remain equivalent.
### SEMANTIC_CHANGE
The intended product behavior has changed.
### CONCEPT_DRIFT
The implementation of test-driver no longer matches an established framework concept.
### FRAMEWORK_LIMITATION
A valid scenario cannot be expressed, executed, observed, or judged adequately.
### EVIDENCE_FAILURE
A finding cannot be reproduced or supported from the retained evidence.
### UNNECESSARY_COMPLEXITY
An abstraction or subsystem adds material maintenance cost without sufficient conceptual or experimental justification.
These findings feed the improvement loop.
---
## 10. Improvement Cycle
The canonical loop is:
```text
OBSERVE
|
v
CLASSIFY
|
v
EXPLAIN
|
v
PROPOSE
|
v
EXPERIMENT
|
v
MEASURE
|
v
ACCEPT / REJECT
|
v
CRYSTALLIZE
```
### Observe
Collect evidence from test-driver runs, self-tests, implementation work and lab experiments.
### Classify
Determine whether the observation represents a product defect, test defect, adaptation, concept drift, framework limitation or other finding class.
### Explain
Produce the smallest useful causal explanation supported by evidence.
### Propose
Generate one or more candidate improvements.
### Experiment
Change one relevant variable where practical and attempt to falsify the proposed improvement.
### Measure
Evaluate the result against explicit success criteria.
### Accept / Reject
Retain improvements that produce sufficient evidence. Reject or revise those that do not.
### Crystallize
Convert learned behavior into a more deterministic, cheaper and more maintainable form whenever possible.
---
## 11. Agentic Roles
Self-improvement should not rely on one omnipotent self-modifying agent.
Distinct roles create productive tension.
### Builder
Implements the current hypothesis or improvement proposal.
### Experimenter
Designs experiments intended to falsify claims.
### Critic
Looks for false success, hidden assumptions and semantic drift.
### Auditor
Checks concept-to-implementation traceability.
### Maintainer
Looks for unnecessary abstractions, duplication and maintenance burden.
These are conceptual roles. Initially they may simply correspond to separate prompts or workflow phases.
---
## 12. Fitness Scorecard
The framework should be measured against its thesis rather than implementation volume.
Initial dimensions:
| Dimension | Example Measure |
|---|---|
| Adaptability | Mechanical changes recovered automatically |
| Semantic integrity | False semantic adaptations |
| Detection | Seeded defects correctly discovered |
| Reproducibility | Findings replayable from retained evidence |
| Crystallization | Agentic assets converted to deterministic execution |
| Efficiency | Cost per verified use case |
| Autonomy | Human interventions per 100 runs |
| Robustness | Success across controlled implementation mutations |
| Traceability | Concepts connected to implementation and evidence |
| Simplicity | Complexity required per supported capability |
A particularly important safety metric is:
> **False Adaptation Rate:** the frequency with which test-driver treats an actual product defect as a legitimate adaptation.
This should be aggressively minimized.
---
## 13. Concept Maturity
Concepts should mature based on evidence rather than attractive terminology.
Suggested levels:
```text
C0 Idea
C1 Hypothesis
C2 Experimentally Supported
C3 Practically Validated
C4 Architectural Invariant
```
Examples at the beginning of the research prototype may be approximately:
```text
Actor Isolation C2
Independent Oracles C2
Semantic Actions C1
Crystallization C1
Verification Energy C0-C1
```
These classifications are provisional and should change with evidence.
---
## 14. Compression
Self-improvement must include deletion.
Learning does not necessarily imply adding features or abstractions.
At regular intervals, perform a compression review:
- Which concepts can be merged?
- Which abstractions lack evidence?
- Which agentic mechanisms can crystallize into deterministic code?
- Which metadata has never informed a decision?
- Which subsystem can be removed?
- Which verification assets have become redundant?
The desired outcome is not maximal capability count.
It is:
> **the smallest framework that reliably realizes the validated test-driver concepts.**
---
## 15. Improvement Evidence
Every accepted framework improvement should retain:
```text
Improvement ID
Triggering finding(s)
Affected concept(s)
Hypothesis
Experiment
Before state
After state
Evidence
Measured outcome
Decision
Resulting self-verification
Resulting deterministic regression, if applicable
```
This creates a lineage from conceptual claim through evidence to implementation.
---
## 16. Initial Control Loop
The first working version does not require autonomous self-modification.
A minimal loop is sufficient:
```text
Framework run
|
v
Framework finding
|
v
Human/agent classification
|
v
Improvement hypothesis
|
v
Controlled lab experiment
|
v
Evidence
|
v
Accept / reject
|
v
Self-verification added
```
Only after this loop reliably produces good improvements should more of the process become agentic.
---
## 17. Success Condition
The improvement loop is successful when test-driver can repeatedly demonstrate that:
1. conceptual claims are traceable to implementation and evidence;
2. implementation changes that violate those claims are detected;
3. controlled experiments can distinguish defects, adaptations and semantic changes;
4. important framework failures become durable self-verifications;
5. agentic mechanisms harden into deterministic mechanisms where possible;
6. the framework becomes simpler or more effective as evidence accumulates;
7. framework evolution does not silently redefine correctness.
The long-term objective is a self-hosting verification system whose own evolution is governed by the evidence-driven principles it applies to other software.

View file

@ -0,0 +1,512 @@
# TestDriver Research Prototype — Initial Milestones
**Status:** v0.1
**Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.
---
## Milestone 0 — Research Control Plane
### Goal
Make the conceptual development of test-driver explicit, traceable and falsifiable before substantial framework code accumulates.
### Implement
- repository structure for concepts, hypotheses, experiments, findings and evidence;
- hypothesis register;
- concept maturity register;
- Concept ↔ Implementation Fitness Map;
- framework finding taxonomy;
- stable identifiers for concepts, hypotheses, experiments and findings;
- lightweight CLI or file conventions for recording decisions.
Suggested structure:
```text
research/
├── hypotheses/
├── experiments/
├── findings/
├── concepts/
└── decisions/
```
### Initial hypotheses
At minimum register:
- H-001 Semantic Action Stability
- H-002 Mechanical Adaptation
- H-003 Crystallization
- H-004 Independent Judgment
- H-005 Verification Energy
### Exit Criteria
- every major v0.1 concept has a stable identifier;
- every implemented subsystem can be linked to at least one concept or hypothesis;
- at least one hypothesis is expressed with a falsification condition and planned experiment;
- framework findings can be recorded independently from product findings.
### Evidence Produced
The first Concept ↔ Implementation Fitness Map and hypothesis register.
---
## Milestone 1 — Deterministic Semantic Kernel
### Goal
Prove the core model without agentic complexity.
### Implement
Minimal executable representations of:
- UseCase;
- Actor;
- World;
- Scenario;
- SemanticAction;
- Observation;
- Oracle;
- Verdict;
- VerificationAsset;
- Run.
Implement one deterministic driver, preferably HTTP or direct application adapter.
### Reference Use Case
```text
Alice owns resource R.
Alice grants Bob READ access.
Bob can read R.
Carol cannot read R.
Alice revokes Bob.
Bob can no longer read R.
```
### Exit Criteria
- the complete use case runs deterministically;
- Alice, Bob and Carol have isolated identities and sessions;
- all important outcomes are judged by independent deterministic oracles;
- a run produces structured evidence;
- the same scenario can be replayed from known initial state.
### Evidence Produced
The first reproducible Evidence Pack.
---
## Milestone 2 — Test-Driver Lab
### Goal
Create a controlled evolutionary environment in which test-driver claims can be deliberately challenged.
### Implement
A deliberately small application supporting:
- users;
- tenants/workspaces;
- authentication;
- resources;
- sharing;
- read/write permissions;
- revoke;
- audit history;
- HTTP API;
- minimal browser UI.
Add explicit mutation switches or tagged lab versions.
### Initial Mutations
- M01 move/rename sharing control;
- M02 restructure the DOM;
- M03 change compatible API representation;
- M04 introduce an additional legitimate workflow step;
- M05 introduce authorization defect;
- M06 introduce propagation delay.
### Exit Criteria
- the reference use case works against the baseline lab;
- each mutation can be enabled reproducibly;
- mutations can be classified as mechanical, semantic or defective;
- baseline and mutated versions retain explicit version identifiers.
### Evidence Produced
A repeatable benchmark environment for framework development.
---
## Milestone 3 — Self-Verification v0
### Goal
Make test-driver test its own foundational guarantees.
### Implement
Initial `td://self/...` verification assets:
```text
td://self/actor-isolation
td://self/oracle-independence
td://self/evidence-reproducibility
```
The self-tests should operate against observable behavior rather than internal implementation details where practical.
### Exit Criteria
- intentionally breaking actor isolation makes the corresponding self-test fail;
- allowing an actor to determine its own verdict makes oracle-independence fail;
- corrupting or omitting required run evidence makes evidence-reproducibility fail;
- each failure creates a Framework Finding.
### Evidence Produced
Proof that conceptual regressions can be detected as framework regressions.
---
## Milestone 4 — Agentic Realization
### Goal
Introduce agentic flexibility only at the realization layer while retaining deterministic truth.
### Implement
- browser driver;
- one agentic Actor Runtime;
- strict per-actor context isolation;
- semantic goal → UI realization loop;
- full action/evidence recording;
- bounded navigation and tool permissions.
Use the semantic action:
```text
grant_access(Bob, R, READ)
```
The agent may discover how to accomplish it through the UI.
The oracle must remain deterministic.
### Exit Criteria
- an agent can realize the reference semantic action from intent;
- the actor cannot access another actor's private context;
- deterministic oracles independently establish success/failure;
- agent/model/configuration identity is recorded in evidence;
- failures can be replayed sufficiently to diagnose them.
### Evidence Produced
First trustworthy agentic run.
---
## Milestone 5 — Mechanical Adaptation
### Goal
Demonstrate the core fluid-development thesis.
### Experiment
Run the same Verification Asset against:
1. lab baseline;
2. M01 moved/renamed control;
3. M02 changed DOM structure.
The use-case semantics remain unchanged.
### Implement
- adaptation detection;
- adaptation classification;
- preservation of semantic action identity;
- adaptation evidence;
- adaptation history on the Verification Asset.
### Exit Criteria
- agentic execution recovers from at least two mechanical mutations;
- original deterministic oracles remain unchanged;
- semantic intent is not modified;
- adaptation is classified as mechanical;
- the framework reports an adaptation rather than a product defect.
### Success Metric
**Mechanical Recovery Rate**
### Critical Safety Metric
**False Semantic Adaptation Rate = 0** for the experiment set.
### Evidence Produced
Support or rejection for H-001 and H-002.
---
## Milestone 6 — Defect vs. Adaptation Discrimination
### Goal
Prove that adaptive testing does not simply learn to accept broken software.
### Experiment
Use:
- M01/M02 as legitimate mechanical changes;
- M05 as an authorization defect;
- M04 or M10-style mutation as a deliberate semantic requirement change.
### Implement
Classification path:
```text
IMPLEMENTATION CHANGE
INTENT CHANGE
PRODUCT DEFECT
AMBIGUOUS
```
Add escalation for semantic changes and ambiguity.
### Exit Criteria
- mechanical changes adapt without altering claims/invariants;
- authorization defect creates a Product Finding;
- deliberate requirement change creates a Semantic Change finding;
- ambiguous evidence produces `INCONCLUSIVE` rather than silent adaptation;
- no seeded defect is normalized as adaptation.
### Success Metric
Classification precision/recall over controlled mutations.
### Evidence Produced
The first meaningful measurement of adaptation safety.
---
## Milestone 7 — Crystallization v0
### Goal
Show that agentic flexibility can harden into deterministic regression.
### Implement
- semantic action trajectory capture;
- stable-realization detection;
- deterministic candidate generation;
- candidate comparison against existing oracle set;
- provenance/lineage from agentic ancestor to deterministic descendant;
- manual acceptance step initially.
### Experiment
Run the same agentic realization repeatedly against a stable lab version, crystallize it, then execute without any model involvement.
### Exit Criteria
- one agentic Verification Asset produces a deterministic test candidate;
- deterministic execution preserves the relevant claims and oracles;
- the generated/hardened test runs with zero agentic involvement;
- lineage remains visible;
- execution cost is measurably lower than agentic execution.
### Success Metrics
- crystallization success rate;
- semantic coverage retained;
- execution cost reduction.
### Evidence Produced
Support or rejection for H-003.
---
## Milestone 8 — Framework Finding → Improvement Loop
### Goal
Close the first actual self-improvement cycle.
### Implement
Workflow:
```text
Framework Finding
Classification
Improvement Hypothesis
Controlled Experiment
Evidence
Accept / Reject
Self-Verification / Regression
```
Use a real framework weakness discovered during Milestones 17 rather than inventing one if possible.
### Exit Criteria
- a framework finding produces an explicit improvement hypothesis;
- the hypothesis is experimentally evaluated;
- the accepted change links back to concept and evidence;
- the discovered framework failure leaves behind a permanent self-verification or deterministic regression;
- the Concept ↔ Implementation Fitness Map is updated.
### Evidence Produced
The first completed **ConceptImplementation Fitness Loop**.
This is the milestone at which the self-improvement system genuinely exists.
---
## Milestone 9 — Verification Energy v0
### Goal
Begin measuring test value without prematurely optimizing the scoring model.
### Implement
Record immutable Energy Events such as:
```text
DEFECT_DETECTED
REGRESSION_CAUGHT
MECHANICAL_ADAPTATION
SEMANTIC_ADAPTATION
TEST_DEFECT
FALSE_POSITIVE
DUPLICATE
CRYSTALLIZED
USECASE_DEPRECATED
```
Initially calculate only a simple transparent score.
### Exit Criteria
- Energy is derived from event history rather than stored as unexplained state;
- every score change is explainable;
- Energy can influence campaign priority;
- criticality can override retirement;
- no automatic deletion is implemented yet.
### Evidence Produced
A dataset suitable for later testing whether Energy actually predicts verification value.
---
## Milestone 10 — First Compression Review
### Goal
Prevent the research prototype from turning into premature platform architecture.
### Review
Ask:
- Which concepts have no supporting evidence?
- Which implementation abstractions have no conceptual justification?
- Which metadata has not informed a decision?
- Which agentic behavior can now be deterministic?
- Which capabilities can be merged or removed?
- What have the experiments falsified?
### Exit Criteria
- at least one simplification is seriously evaluated;
- rejected concepts are marked as such rather than silently retained;
- architecture reflects experimental learning;
- updated Concept Model and Improvement Loop remain smaller or more precise where evidence permits.
### Evidence Produced
The first proof that self-improvement includes subtraction, not only accumulation.
---
# Recommended Execution Order
```text
M0 Research Control Plane
|
M1 Deterministic Semantic Kernel
|
M2 Test-Driver Lab
|
M3 Self-Verification v0
|
M4 Agentic Realization
|
M5 Mechanical Adaptation
|
M6 Defect vs Adaptation
|
M7 Crystallization
|
M8 Closed Improvement Loop
|
M9 Verification Energy
|
M10 Compression Review
```
The first major research gate is **M8**.
Before M8, test-driver has promising mechanisms.
At M8, it has demonstrated a complete evidence-driven self-improvement cycle.
---
# Prototype Success Gate
The initial research prototype should be considered successful enough to justify broader framework investment when it can demonstrate all of the following in one coherent system:
1. a multi-user use case expressed independently of implementation details;
2. deterministic independent oracles;
3. agentic realization of at least one semantic action;
4. recovery from legitimate mechanical implementation change;
5. rejection of a seeded semantic/security defect as a mere adaptation;
6. reproducible evidence;
7. crystallization into deterministic execution;
8. a framework failure converted into a permanent self-verification;
9. explicit concept-to-implementation traceability;
10. measured human effort and execution cost.
That demonstration is more valuable than broad feature coverage.

View file

@ -0,0 +1,338 @@
# Stage 1 Test Driver Validation
The biggest risk is not technical feasibility. It is that **test-driver becomes conceptually elegant but too broad to prove itself quickly**.
I would improve the odds of success by treating the next phase as an experiment in whether three specific ideas actually work:
1. **A use case can survive implementation change better than a conventional test script.**
2. **Agentic execution can bridge unstable implementation without corrupting the intended semantics.**
3. **Successful agentic tests can crystallize into cheaper deterministic tests.**
If those three work, the rest—energy, campaigns, security mutation, multi-user orchestration—has a strong foundation.
### Narrow the first battlefield
Choose exactly one real application and perhaps three use cases. They should deliberately include the difficult characteristics test-driver is meant to solve: authentication, several users, state transitions, authorization, and an evolving UI/API.
For example:
```text
UC-01 Alice creates a workspace
UC-02 Alice invites Bob and Bob joins
UC-03 Alice revokes Bob and Bob loses access
```
UC-03 already gives you functional, interaction, temporal and security semantics.
Avoid building a generic testing platform first. Make test-driver extraordinarily good at this one sequence.
### Make the semantic layer the core intellectual property
The critical interface is not the LLM integration or browser automation. It is:
```text
UseCase
Semantic Goal
Semantic Action
Concrete realization
```
For example:
```text
grant_access(Bob, resource, READ)
```
may currently mean six browser interactions.
Later it may mean an API operation.
Test-driver should care about the semantic action. Drivers care about realization.
If this abstraction is good, the framework survives technology changes. If it is poor, agentic execution becomes sophisticated screen scraping.
### Force the framework to distinguish discovery from truth
One of the strongest architectural principles should be:
> **Agents discover paths. Oracles establish truth.**
For the first implementation, make every important oracle deterministic.
For example, let the agent discover how Alice invites Bob through the UI, but verify independently through an API or database-facing test interface that:
```text
membership(Bob, Workspace) == MEMBER
membership(Carol, Workspace) == NONE
```
Do not let the actor agent conclude, “It looks like Bob joined.”
This separation will prevent many future problems.
### Build reproducibility before intelligence
For every run, capture at least:
```text
use-case version
scenario version
application version/commit
actor identities and roles
initial world
semantic actions
actual actions
random seed
timestamps
observations
oracle results
screenshots/traces where useful
agent/model/config version
```
An agentically discovered failure that cannot be reproduced is much less valuable.
The first impressive demonstration should therefore not be “the agent found a bug.”
It should be:
> “The agent found a bug, test-driver reduced it to this scenario, and the failure can now be replayed deterministically.”
### Treat crystallization as an explicit deliverable
Don't postpone crystallization until later.
Make the first milestone contain this lifecycle:
```text
new use case
agentic execution
stable semantic trajectory
candidate deterministic implementation
deterministic regression
```
You need to learn early whether this transition can actually be automated or assisted effectively.
A useful success metric might be:
> **How many agentic verification assets can be downgraded to deterministic execution without losing semantic coverage?**
That is much more meaningful than counting generated tests.
### Introduce energy only after you have event history
I like Test Energy a lot, but I would avoid optimizing its formula early.
Start by recording events:
```text
found-defect
false-positive
mechanical-adaptation
semantic-adaptation
duplicate-detected
usecase-changed
crystallized
regression-caught
```
Then initially compute a crude score.
After a few hundred runs you can inspect whether the proposed energy changes actually correspond to human intuition about test value.
In other words:
> **Store the evidence first; invent the fitness function second.**
Otherwise you'll encode assumptions before you have data.
### Separate three kinds of change
This will probably become one of the framework's most important capabilities.
Whenever a test stops matching the system, classify the change as:
```text
IMPLEMENTATION CHANGE
same behavior, different realization
INTENT CHANGE
the product is deliberately supposed to behave differently
DEFECT
implementation no longer satisfies unchanged intent
```
This classification is what makes adaptive tests safe.
A lot of test maintenance today implicitly assumes the first case. Test-driver must never make that assumption.
### Give humans a small number of meaningful decision points
Agentic systems fail when humans either have to approve everything or are expected to trust everything.
Aim for perhaps three escalation types:
```text
SEMANTIC CHANGE
"The expected workflow appears to have changed."
INVARIANT CONFLICT
"The observed behavior conflicts with a protected invariant."
AMBIGUOUS VERDICT
"Available evidence cannot establish correctness."
```
Everything mechanical beneath that should eventually be autonomous.
That creates a good human/agent boundary.
### Make security an early proof point
Security testing is especially good for proving the framework because ordinary functional use cases naturally generate adversarial variants.
From:
```text
Alice grants Bob access.
```
derive:
```text
Carol tries access.
Bob attempts WRITE.
Bob accesses before grant.
Bob accesses after revoke.
Bob substitutes another resource ID.
Bob attempts cross-tenant access.
```
If test-driver can systematically produce these from the ordinary use case, you already have something substantially more interesting than “an LLM controlling Playwright.”
### Use a deliberately boring implementation stack
The novelty should be in the verification model, not infrastructure.
For the first implementation, I'd prefer something like:
```text
Python
pytest
Playwright
Pydantic/dataclasses
YAML
SQLite
OpenTelemetry where useful
```
Maybe later a graph store, distributed workers, richer agent infrastructure, etc.
But initially:
> **One process, one database, one browser engine, one application under test.**
Complex infrastructure will hide conceptual flaws.
### Dogfood aggressively
Test-driver itself should become one of the systems tested by test-driver.
As soon as there is a usable CLI or web surface, introduce its own use cases:
```text
define use case
execute campaign
inspect evidence
accept finding
crystallize test
retire verification
```
That gives you continuous pressure to make the framework usable rather than merely architecturally sophisticated.
### Define success quantitatively
I'd track a small scorecard from the first month:
| Measure | Why it matters |
| -------------------------------------------- | ---------------- |
| Time to express a new use case | usability |
| Agentic scenario success rate | executability |
| Mechanical adaptations handled automatically | fluidity benefit |
| False semantic adaptations | safety |
| Genuine defects discovered | effectiveness |
| Defects reproducibly replayed | trustworthiness |
| Agentic → deterministic crystallizations | core thesis |
| Median execution cost per maturity level | economics |
| Human interventions per 100 runs | autonomy |
| Security variants derived per use case | leverage |
The key metric could eventually be something like:
> **Verified behavior per unit of human maintenance effort.**
That captures what test-driver is fundamentally trying to improve.
### Keep the architecture open, but the semantics opinionated
Drivers, agents and storage should be replaceable.
The following should not be casually replaceable:
```text
UseCase
Actor
Scenario
Claim
Invariant
SemanticAction
Observation
Oracle
VerificationAsset
Finding
Adaptation
Crystallization
EnergyEvent
Lineage
```
Those are the conceptual backbone.
If they change constantly, you don't yet understand the domain.
### And establish a ruthless non-goal list
For v0.1 I would explicitly not try to become:
```text
a generic unit-test framework
a load-testing replacement
a fuzzing engine
a vulnerability scanner
a test-management SaaS
a CI/CD system
an observability platform
a universal browser agent
```
Test-driver may eventually integrate all of those.
Its unique purpose should remain:
> **Maintain trustworthy verification of use cases while software changes rapidly, using agentic flexibility where necessary and deterministic verification wherever possible.**
If I were running the project, the next concrete milestone would be a **Test Driver Research Prototype**, not yet a product: one target application, three linked multi-user use cases, deterministic oracles, one agentic browser driver, complete evidence capture, one mutation mechanism, and one successful crystallization into a deterministic regression test.
If that demonstrably works, we will have validated the hardest and most original part of the idea.
xxx