Register with State Hub, persist concept assessment, seed first workplan
- history/2026-08-22-concept-assessment-swot.md: SWOT assessment of the concept corpus with recommendations for the first workplan - statehub register: infotech domain, TD-WP prefix, generated AGENTS.md, .custodian-brief.md and TD-WP-0001 bootstrap workplan - .repo-classification.yaml: category research, domain infotech - SCOPE.md rewritten with real repo boundaries - TD-WP-0002: vertical spike reordering M0-M10 into one end-to-end thread that can falsify the crystallization thesis early - commit previously untracked INTENT.md and docs/ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
b471bed707
commit
7249c6a403
13 changed files with 3900 additions and 0 deletions
1044
docs/TestDriverConceptModel.md
Executable file
1044
docs/TestDriverConceptModel.md
Executable file
File diff suppressed because it is too large
Load diff
609
docs/TestDriverImprovementLoop.md
Executable file
609
docs/TestDriverImprovementLoop.md
Executable file
|
|
@ -0,0 +1,609 @@
|
|||
# TestDriver Improvement Loop
|
||||
|
||||
**Status:** Concept v0.1
|
||||
**Purpose:** Establish a self-improvement loop that keeps the implementation of `test-driver` aligned with its conceptual model while generating evidence about which concepts actually work.
|
||||
|
||||
---
|
||||
|
||||
## 1. Intent
|
||||
|
||||
`test-driver` is itself an evolving software system. Its implementation must therefore be subject to the same principles it applies to systems under test:
|
||||
|
||||
- intended behavior must remain distinguishable from implementation details;
|
||||
- change must generate evidence rather than silently redefine correctness;
|
||||
- exploratory mechanisms should harden into deterministic mechanisms as understanding grows;
|
||||
- failures should improve future verification;
|
||||
- obsolete complexity should be allowed to disappear.
|
||||
|
||||
The improvement loop exists to continuously reconcile:
|
||||
|
||||
1. **Concept** — what test-driver claims should be true;
|
||||
2. **Implementation** — what the framework actually does;
|
||||
3. **Evidence** — what experiments and executions demonstrate.
|
||||
|
||||
The loop should make conceptual drift visible and turn important framework failures into durable self-verification assets.
|
||||
|
||||
---
|
||||
|
||||
## 2. Core Principle
|
||||
|
||||
> Every meaningful implementation increment should generate evidence about whether test-driver is becoming a better realization of its own concept.
|
||||
|
||||
The framework therefore has two simultaneous feedback loops:
|
||||
|
||||
```text
|
||||
Concept / Intent / Hypotheses
|
||||
|
|
||||
v
|
||||
Select Experiment
|
||||
|
|
||||
v
|
||||
Implement
|
||||
|
|
||||
v
|
||||
Exercise on SUT
|
||||
|
|
||||
v
|
||||
Evidence
|
||||
|
|
||||
+----------+----------+
|
||||
| |
|
||||
v v
|
||||
Product Finding Framework Finding
|
||||
| |
|
||||
v v
|
||||
Improve target Improve test-driver
|
||||
|
|
||||
v
|
||||
Reconcile with Concept
|
||||
|
|
||||
+-----> next cycle
|
||||
```
|
||||
|
||||
A run may therefore produce findings about the system under test and findings about the verification framework itself.
|
||||
|
||||
---
|
||||
|
||||
## 3. Development as Experimental Work
|
||||
|
||||
Implementation work should increasingly be framed as hypotheses rather than feature requests.
|
||||
|
||||
Examples:
|
||||
|
||||
### H-001 — Semantic Action Stability
|
||||
|
||||
> A semantic action survives implementation restructuring better than a recorded UI interaction sequence.
|
||||
|
||||
### H-002 — Mechanical Adaptation
|
||||
|
||||
> An agentic driver can recover from a mechanical implementation change without modifying the semantics of the protected use case.
|
||||
|
||||
### H-003 — Crystallization
|
||||
|
||||
> A sufficiently stable agentic execution can be converted into deterministic test code without losing relevant oracle coverage.
|
||||
|
||||
### H-004 — Independent Judgment
|
||||
|
||||
> Separating actor execution from deterministic oracles reduces false-positive adaptation to defective behavior.
|
||||
|
||||
### H-005 — Verification Energy
|
||||
|
||||
> Historical evidence about defects caught, adaptations required, false positives and duplication can identify verification assets whose continued execution is more valuable than others.
|
||||
|
||||
Each hypothesis should have:
|
||||
|
||||
- an identifier;
|
||||
- a claim;
|
||||
- a falsification condition;
|
||||
- one or more experiments;
|
||||
- evidence;
|
||||
- a status;
|
||||
- resulting implementation or concept changes.
|
||||
|
||||
Suggested lifecycle:
|
||||
|
||||
```text
|
||||
PROPOSED
|
||||
|
|
||||
v
|
||||
EXPERIMENTING
|
||||
|
|
||||
+------> REJECTED
|
||||
|
|
||||
v
|
||||
SUPPORTED
|
||||
|
|
||||
v
|
||||
PRACTICALLY_VALIDATED
|
||||
|
|
||||
v
|
||||
ARCHITECTURAL
|
||||
```
|
||||
|
||||
A hypothesis may also be reopened if later evidence contradicts it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Concept–Implementation Fitness Map
|
||||
|
||||
Every important concept should become traceable to the implementation and evidence that support it.
|
||||
|
||||
Conceptual relationship:
|
||||
|
||||
```text
|
||||
Concept
|
||||
|
|
||||
+-- implementation
|
||||
+-- experiment
|
||||
+-- evidence
|
||||
+-- self-verification
|
||||
+-- unresolved questions
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
concept: actor-isolation
|
||||
claim: >
|
||||
One actor must not obtain private state, credentials, observations,
|
||||
or memory belonging to another actor except through modeled
|
||||
communication channels.
|
||||
|
||||
implementation:
|
||||
- testdriver/runtime/actor_context.py
|
||||
|
||||
experiments:
|
||||
- H-006
|
||||
|
||||
self_verifications:
|
||||
- td://self/actor-isolation
|
||||
|
||||
status: supported
|
||||
```
|
||||
|
||||
The map should expose two forms of drift.
|
||||
|
||||
### Conceptual Orphaning
|
||||
|
||||
A concept is claimed but has no implementation or verification evidence.
|
||||
|
||||
### Implementation Orphaning
|
||||
|
||||
A subsystem or abstraction exists without a concept, requirement, or experiment that explains why it is needed.
|
||||
|
||||
Both should be visible during review.
|
||||
|
||||
---
|
||||
|
||||
## 5. Concept Drift
|
||||
|
||||
A **Concept Drift Finding** occurs when implementation behavior diverges from an established concept without an explicit decision to revise that concept.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Concept:
|
||||
Agents discover paths; independent oracles judge outcomes.
|
||||
|
||||
Implementation:
|
||||
Browser agent declares the scenario successful.
|
||||
|
||||
Finding:
|
||||
CONCEPT_DRIFT
|
||||
```
|
||||
|
||||
A concept-drift finding must resolve in one of three ways:
|
||||
|
||||
1. implementation changes to match the concept;
|
||||
2. the concept is deliberately revised;
|
||||
3. an experiment demonstrates that the distinction is no longer useful.
|
||||
|
||||
Implementation reality must not silently redefine the conceptual model.
|
||||
|
||||
---
|
||||
|
||||
## 6. Self-Verification
|
||||
|
||||
`test-driver` should become a system under test for `test-driver`.
|
||||
|
||||
Self-verification use cases use the namespace:
|
||||
|
||||
```text
|
||||
td://self/...
|
||||
```
|
||||
|
||||
Initial candidates:
|
||||
|
||||
```text
|
||||
td://self/actor-isolation
|
||||
td://self/oracle-independence
|
||||
td://self/evidence-reproducibility
|
||||
td://self/mechanical-adaptation
|
||||
td://self/semantic-change-detection
|
||||
td://self/crystallization
|
||||
td://self/test-retirement
|
||||
```
|
||||
|
||||
These scenarios should verify framework-level promises rather than implementation internals whenever possible.
|
||||
|
||||
Example:
|
||||
|
||||
### `td://self/mechanical-adaptation`
|
||||
|
||||
1. execute a stable use case against lab version A;
|
||||
2. change the UI mechanically without changing semantics;
|
||||
3. rerun the use case;
|
||||
4. allow agentic navigation to recover;
|
||||
5. verify that the same semantic action and oracles remain valid;
|
||||
6. record the adaptation and evidence.
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
mechanical implementation change
|
||||
|
|
||||
v
|
||||
adaptation detected
|
||||
|
|
||||
v
|
||||
alternative realization discovered
|
||||
|
|
||||
v
|
||||
semantic action preserved
|
||||
|
|
||||
v
|
||||
original oracle still passes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Test-Driver Lab
|
||||
|
||||
A purpose-built mutable application should provide controlled evolutionary pressure for the framework.
|
||||
|
||||
Suggested repository or module name:
|
||||
|
||||
```text
|
||||
test-driver-lab
|
||||
```
|
||||
|
||||
The lab should be intentionally small but support:
|
||||
|
||||
- multiple users;
|
||||
- organizations or tenants;
|
||||
- authentication;
|
||||
- resources;
|
||||
- sharing;
|
||||
- permissions;
|
||||
- simple workflows;
|
||||
- audit events;
|
||||
- API interaction;
|
||||
- browser interaction.
|
||||
|
||||
The lab should also support deliberate implementation mutations.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
M01 move or rename a UI control
|
||||
M02 replace the DOM structure
|
||||
M03 change a compatible API representation
|
||||
M04 add a legitimate workflow step
|
||||
M05 introduce an authorization defect
|
||||
M06 introduce eventual-consistency delay
|
||||
M07 introduce intermittent dependency failure
|
||||
M08 remove or deprecate a capability
|
||||
M09 create a concurrency race
|
||||
M10 change the intended business requirement
|
||||
```
|
||||
|
||||
The purpose is not to build a representative product. It is to create a controlled environment in which claims about test-driver can be falsified.
|
||||
|
||||
---
|
||||
|
||||
## 8. Dual Mutation
|
||||
|
||||
Mutation should operate in two directions.
|
||||
|
||||
### Use-Case Mutation
|
||||
|
||||
Mutate actor, resource, order, timing, state, or permissions to test the robustness of the system under test.
|
||||
|
||||
```text
|
||||
UseCase Mutation
|
||||
|
|
||||
v
|
||||
tests robustness of application
|
||||
```
|
||||
|
||||
### Implementation Mutation
|
||||
|
||||
Mutate the target implementation while holding the use-case semantics constant to test the robustness of test-driver.
|
||||
|
||||
```text
|
||||
Implementation Mutation
|
||||
|
|
||||
v
|
||||
tests robustness of test-driver
|
||||
```
|
||||
|
||||
This duality allows the framework to test both the application and its own verification strategy.
|
||||
|
||||
---
|
||||
|
||||
## 9. Framework Findings
|
||||
|
||||
The framework should maintain finding classes distinct from ordinary product defects.
|
||||
|
||||
Initial set:
|
||||
|
||||
### PRODUCT_DEFECT
|
||||
The system under test violates unchanged intent.
|
||||
|
||||
### TEST_DEFECT
|
||||
The verification asset or oracle is incorrect.
|
||||
|
||||
### MECHANICAL_ADAPTATION
|
||||
Implementation mechanics changed while protected semantics remain equivalent.
|
||||
|
||||
### SEMANTIC_CHANGE
|
||||
The intended product behavior has changed.
|
||||
|
||||
### CONCEPT_DRIFT
|
||||
The implementation of test-driver no longer matches an established framework concept.
|
||||
|
||||
### FRAMEWORK_LIMITATION
|
||||
A valid scenario cannot be expressed, executed, observed, or judged adequately.
|
||||
|
||||
### EVIDENCE_FAILURE
|
||||
A finding cannot be reproduced or supported from the retained evidence.
|
||||
|
||||
### UNNECESSARY_COMPLEXITY
|
||||
An abstraction or subsystem adds material maintenance cost without sufficient conceptual or experimental justification.
|
||||
|
||||
These findings feed the improvement loop.
|
||||
|
||||
---
|
||||
|
||||
## 10. Improvement Cycle
|
||||
|
||||
The canonical loop is:
|
||||
|
||||
```text
|
||||
OBSERVE
|
||||
|
|
||||
v
|
||||
CLASSIFY
|
||||
|
|
||||
v
|
||||
EXPLAIN
|
||||
|
|
||||
v
|
||||
PROPOSE
|
||||
|
|
||||
v
|
||||
EXPERIMENT
|
||||
|
|
||||
v
|
||||
MEASURE
|
||||
|
|
||||
v
|
||||
ACCEPT / REJECT
|
||||
|
|
||||
v
|
||||
CRYSTALLIZE
|
||||
```
|
||||
|
||||
### Observe
|
||||
|
||||
Collect evidence from test-driver runs, self-tests, implementation work and lab experiments.
|
||||
|
||||
### Classify
|
||||
|
||||
Determine whether the observation represents a product defect, test defect, adaptation, concept drift, framework limitation or other finding class.
|
||||
|
||||
### Explain
|
||||
|
||||
Produce the smallest useful causal explanation supported by evidence.
|
||||
|
||||
### Propose
|
||||
|
||||
Generate one or more candidate improvements.
|
||||
|
||||
### Experiment
|
||||
|
||||
Change one relevant variable where practical and attempt to falsify the proposed improvement.
|
||||
|
||||
### Measure
|
||||
|
||||
Evaluate the result against explicit success criteria.
|
||||
|
||||
### Accept / Reject
|
||||
|
||||
Retain improvements that produce sufficient evidence. Reject or revise those that do not.
|
||||
|
||||
### Crystallize
|
||||
|
||||
Convert learned behavior into a more deterministic, cheaper and more maintainable form whenever possible.
|
||||
|
||||
---
|
||||
|
||||
## 11. Agentic Roles
|
||||
|
||||
Self-improvement should not rely on one omnipotent self-modifying agent.
|
||||
|
||||
Distinct roles create productive tension.
|
||||
|
||||
### Builder
|
||||
|
||||
Implements the current hypothesis or improvement proposal.
|
||||
|
||||
### Experimenter
|
||||
|
||||
Designs experiments intended to falsify claims.
|
||||
|
||||
### Critic
|
||||
|
||||
Looks for false success, hidden assumptions and semantic drift.
|
||||
|
||||
### Auditor
|
||||
|
||||
Checks concept-to-implementation traceability.
|
||||
|
||||
### Maintainer
|
||||
|
||||
Looks for unnecessary abstractions, duplication and maintenance burden.
|
||||
|
||||
These are conceptual roles. Initially they may simply correspond to separate prompts or workflow phases.
|
||||
|
||||
---
|
||||
|
||||
## 12. Fitness Scorecard
|
||||
|
||||
The framework should be measured against its thesis rather than implementation volume.
|
||||
|
||||
Initial dimensions:
|
||||
|
||||
| Dimension | Example Measure |
|
||||
|---|---|
|
||||
| Adaptability | Mechanical changes recovered automatically |
|
||||
| Semantic integrity | False semantic adaptations |
|
||||
| Detection | Seeded defects correctly discovered |
|
||||
| Reproducibility | Findings replayable from retained evidence |
|
||||
| Crystallization | Agentic assets converted to deterministic execution |
|
||||
| Efficiency | Cost per verified use case |
|
||||
| Autonomy | Human interventions per 100 runs |
|
||||
| Robustness | Success across controlled implementation mutations |
|
||||
| Traceability | Concepts connected to implementation and evidence |
|
||||
| Simplicity | Complexity required per supported capability |
|
||||
|
||||
A particularly important safety metric is:
|
||||
|
||||
> **False Adaptation Rate:** the frequency with which test-driver treats an actual product defect as a legitimate adaptation.
|
||||
|
||||
This should be aggressively minimized.
|
||||
|
||||
---
|
||||
|
||||
## 13. Concept Maturity
|
||||
|
||||
Concepts should mature based on evidence rather than attractive terminology.
|
||||
|
||||
Suggested levels:
|
||||
|
||||
```text
|
||||
C0 Idea
|
||||
C1 Hypothesis
|
||||
C2 Experimentally Supported
|
||||
C3 Practically Validated
|
||||
C4 Architectural Invariant
|
||||
```
|
||||
|
||||
Examples at the beginning of the research prototype may be approximately:
|
||||
|
||||
```text
|
||||
Actor Isolation C2
|
||||
Independent Oracles C2
|
||||
Semantic Actions C1
|
||||
Crystallization C1
|
||||
Verification Energy C0-C1
|
||||
```
|
||||
|
||||
These classifications are provisional and should change with evidence.
|
||||
|
||||
---
|
||||
|
||||
## 14. Compression
|
||||
|
||||
Self-improvement must include deletion.
|
||||
|
||||
Learning does not necessarily imply adding features or abstractions.
|
||||
|
||||
At regular intervals, perform a compression review:
|
||||
|
||||
- Which concepts can be merged?
|
||||
- Which abstractions lack evidence?
|
||||
- Which agentic mechanisms can crystallize into deterministic code?
|
||||
- Which metadata has never informed a decision?
|
||||
- Which subsystem can be removed?
|
||||
- Which verification assets have become redundant?
|
||||
|
||||
The desired outcome is not maximal capability count.
|
||||
|
||||
It is:
|
||||
|
||||
> **the smallest framework that reliably realizes the validated test-driver concepts.**
|
||||
|
||||
---
|
||||
|
||||
## 15. Improvement Evidence
|
||||
|
||||
Every accepted framework improvement should retain:
|
||||
|
||||
```text
|
||||
Improvement ID
|
||||
Triggering finding(s)
|
||||
Affected concept(s)
|
||||
Hypothesis
|
||||
Experiment
|
||||
Before state
|
||||
After state
|
||||
Evidence
|
||||
Measured outcome
|
||||
Decision
|
||||
Resulting self-verification
|
||||
Resulting deterministic regression, if applicable
|
||||
```
|
||||
|
||||
This creates a lineage from conceptual claim through evidence to implementation.
|
||||
|
||||
---
|
||||
|
||||
## 16. Initial Control Loop
|
||||
|
||||
The first working version does not require autonomous self-modification.
|
||||
|
||||
A minimal loop is sufficient:
|
||||
|
||||
```text
|
||||
Framework run
|
||||
|
|
||||
v
|
||||
Framework finding
|
||||
|
|
||||
v
|
||||
Human/agent classification
|
||||
|
|
||||
v
|
||||
Improvement hypothesis
|
||||
|
|
||||
v
|
||||
Controlled lab experiment
|
||||
|
|
||||
v
|
||||
Evidence
|
||||
|
|
||||
v
|
||||
Accept / reject
|
||||
|
|
||||
v
|
||||
Self-verification added
|
||||
```
|
||||
|
||||
Only after this loop reliably produces good improvements should more of the process become agentic.
|
||||
|
||||
---
|
||||
|
||||
## 17. Success Condition
|
||||
|
||||
The improvement loop is successful when test-driver can repeatedly demonstrate that:
|
||||
|
||||
1. conceptual claims are traceable to implementation and evidence;
|
||||
2. implementation changes that violate those claims are detected;
|
||||
3. controlled experiments can distinguish defects, adaptations and semantic changes;
|
||||
4. important framework failures become durable self-verifications;
|
||||
5. agentic mechanisms harden into deterministic mechanisms where possible;
|
||||
6. the framework becomes simpler or more effective as evidence accumulates;
|
||||
7. framework evolution does not silently redefine correctness.
|
||||
|
||||
The long-term objective is a self-hosting verification system whose own evolution is governed by the evidence-driven principles it applies to other software.
|
||||
512
docs/TestDriverInitialMilestones.md
Executable file
512
docs/TestDriverInitialMilestones.md
Executable file
|
|
@ -0,0 +1,512 @@
|
|||
# TestDriver Research Prototype — Initial Milestones
|
||||
|
||||
**Status:** v0.1
|
||||
**Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 0 — Research Control Plane
|
||||
|
||||
### Goal
|
||||
|
||||
Make the conceptual development of test-driver explicit, traceable and falsifiable before substantial framework code accumulates.
|
||||
|
||||
### Implement
|
||||
|
||||
- repository structure for concepts, hypotheses, experiments, findings and evidence;
|
||||
- hypothesis register;
|
||||
- concept maturity register;
|
||||
- Concept ↔ Implementation Fitness Map;
|
||||
- framework finding taxonomy;
|
||||
- stable identifiers for concepts, hypotheses, experiments and findings;
|
||||
- lightweight CLI or file conventions for recording decisions.
|
||||
|
||||
Suggested structure:
|
||||
|
||||
```text
|
||||
research/
|
||||
├── hypotheses/
|
||||
├── experiments/
|
||||
├── findings/
|
||||
├── concepts/
|
||||
└── decisions/
|
||||
```
|
||||
|
||||
### Initial hypotheses
|
||||
|
||||
At minimum register:
|
||||
|
||||
- H-001 Semantic Action Stability
|
||||
- H-002 Mechanical Adaptation
|
||||
- H-003 Crystallization
|
||||
- H-004 Independent Judgment
|
||||
- H-005 Verification Energy
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- every major v0.1 concept has a stable identifier;
|
||||
- every implemented subsystem can be linked to at least one concept or hypothesis;
|
||||
- at least one hypothesis is expressed with a falsification condition and planned experiment;
|
||||
- framework findings can be recorded independently from product findings.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first Concept ↔ Implementation Fitness Map and hypothesis register.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 1 — Deterministic Semantic Kernel
|
||||
|
||||
### Goal
|
||||
|
||||
Prove the core model without agentic complexity.
|
||||
|
||||
### Implement
|
||||
|
||||
Minimal executable representations of:
|
||||
|
||||
- UseCase;
|
||||
- Actor;
|
||||
- World;
|
||||
- Scenario;
|
||||
- SemanticAction;
|
||||
- Observation;
|
||||
- Oracle;
|
||||
- Verdict;
|
||||
- VerificationAsset;
|
||||
- Run.
|
||||
|
||||
Implement one deterministic driver, preferably HTTP or direct application adapter.
|
||||
|
||||
### Reference Use Case
|
||||
|
||||
```text
|
||||
Alice owns resource R.
|
||||
Alice grants Bob READ access.
|
||||
Bob can read R.
|
||||
Carol cannot read R.
|
||||
Alice revokes Bob.
|
||||
Bob can no longer read R.
|
||||
```
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- the complete use case runs deterministically;
|
||||
- Alice, Bob and Carol have isolated identities and sessions;
|
||||
- all important outcomes are judged by independent deterministic oracles;
|
||||
- a run produces structured evidence;
|
||||
- the same scenario can be replayed from known initial state.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first reproducible Evidence Pack.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 2 — Test-Driver Lab
|
||||
|
||||
### Goal
|
||||
|
||||
Create a controlled evolutionary environment in which test-driver claims can be deliberately challenged.
|
||||
|
||||
### Implement
|
||||
|
||||
A deliberately small application supporting:
|
||||
|
||||
- users;
|
||||
- tenants/workspaces;
|
||||
- authentication;
|
||||
- resources;
|
||||
- sharing;
|
||||
- read/write permissions;
|
||||
- revoke;
|
||||
- audit history;
|
||||
- HTTP API;
|
||||
- minimal browser UI.
|
||||
|
||||
Add explicit mutation switches or tagged lab versions.
|
||||
|
||||
### Initial Mutations
|
||||
|
||||
- M01 move/rename sharing control;
|
||||
- M02 restructure the DOM;
|
||||
- M03 change compatible API representation;
|
||||
- M04 introduce an additional legitimate workflow step;
|
||||
- M05 introduce authorization defect;
|
||||
- M06 introduce propagation delay.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- the reference use case works against the baseline lab;
|
||||
- each mutation can be enabled reproducibly;
|
||||
- mutations can be classified as mechanical, semantic or defective;
|
||||
- baseline and mutated versions retain explicit version identifiers.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
A repeatable benchmark environment for framework development.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 3 — Self-Verification v0
|
||||
|
||||
### Goal
|
||||
|
||||
Make test-driver test its own foundational guarantees.
|
||||
|
||||
### Implement
|
||||
|
||||
Initial `td://self/...` verification assets:
|
||||
|
||||
```text
|
||||
td://self/actor-isolation
|
||||
td://self/oracle-independence
|
||||
td://self/evidence-reproducibility
|
||||
```
|
||||
|
||||
The self-tests should operate against observable behavior rather than internal implementation details where practical.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- intentionally breaking actor isolation makes the corresponding self-test fail;
|
||||
- allowing an actor to determine its own verdict makes oracle-independence fail;
|
||||
- corrupting or omitting required run evidence makes evidence-reproducibility fail;
|
||||
- each failure creates a Framework Finding.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Proof that conceptual regressions can be detected as framework regressions.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 4 — Agentic Realization
|
||||
|
||||
### Goal
|
||||
|
||||
Introduce agentic flexibility only at the realization layer while retaining deterministic truth.
|
||||
|
||||
### Implement
|
||||
|
||||
- browser driver;
|
||||
- one agentic Actor Runtime;
|
||||
- strict per-actor context isolation;
|
||||
- semantic goal → UI realization loop;
|
||||
- full action/evidence recording;
|
||||
- bounded navigation and tool permissions.
|
||||
|
||||
Use the semantic action:
|
||||
|
||||
```text
|
||||
grant_access(Bob, R, READ)
|
||||
```
|
||||
|
||||
The agent may discover how to accomplish it through the UI.
|
||||
|
||||
The oracle must remain deterministic.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- an agent can realize the reference semantic action from intent;
|
||||
- the actor cannot access another actor's private context;
|
||||
- deterministic oracles independently establish success/failure;
|
||||
- agent/model/configuration identity is recorded in evidence;
|
||||
- failures can be replayed sufficiently to diagnose them.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
First trustworthy agentic run.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 5 — Mechanical Adaptation
|
||||
|
||||
### Goal
|
||||
|
||||
Demonstrate the core fluid-development thesis.
|
||||
|
||||
### Experiment
|
||||
|
||||
Run the same Verification Asset against:
|
||||
|
||||
1. lab baseline;
|
||||
2. M01 moved/renamed control;
|
||||
3. M02 changed DOM structure.
|
||||
|
||||
The use-case semantics remain unchanged.
|
||||
|
||||
### Implement
|
||||
|
||||
- adaptation detection;
|
||||
- adaptation classification;
|
||||
- preservation of semantic action identity;
|
||||
- adaptation evidence;
|
||||
- adaptation history on the Verification Asset.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- agentic execution recovers from at least two mechanical mutations;
|
||||
- original deterministic oracles remain unchanged;
|
||||
- semantic intent is not modified;
|
||||
- adaptation is classified as mechanical;
|
||||
- the framework reports an adaptation rather than a product defect.
|
||||
|
||||
### Success Metric
|
||||
|
||||
**Mechanical Recovery Rate**
|
||||
|
||||
### Critical Safety Metric
|
||||
|
||||
**False Semantic Adaptation Rate = 0** for the experiment set.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Support or rejection for H-001 and H-002.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 6 — Defect vs. Adaptation Discrimination
|
||||
|
||||
### Goal
|
||||
|
||||
Prove that adaptive testing does not simply learn to accept broken software.
|
||||
|
||||
### Experiment
|
||||
|
||||
Use:
|
||||
|
||||
- M01/M02 as legitimate mechanical changes;
|
||||
- M05 as an authorization defect;
|
||||
- M04 or M10-style mutation as a deliberate semantic requirement change.
|
||||
|
||||
### Implement
|
||||
|
||||
Classification path:
|
||||
|
||||
```text
|
||||
IMPLEMENTATION CHANGE
|
||||
INTENT CHANGE
|
||||
PRODUCT DEFECT
|
||||
AMBIGUOUS
|
||||
```
|
||||
|
||||
Add escalation for semantic changes and ambiguity.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- mechanical changes adapt without altering claims/invariants;
|
||||
- authorization defect creates a Product Finding;
|
||||
- deliberate requirement change creates a Semantic Change finding;
|
||||
- ambiguous evidence produces `INCONCLUSIVE` rather than silent adaptation;
|
||||
- no seeded defect is normalized as adaptation.
|
||||
|
||||
### Success Metric
|
||||
|
||||
Classification precision/recall over controlled mutations.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first meaningful measurement of adaptation safety.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 7 — Crystallization v0
|
||||
|
||||
### Goal
|
||||
|
||||
Show that agentic flexibility can harden into deterministic regression.
|
||||
|
||||
### Implement
|
||||
|
||||
- semantic action trajectory capture;
|
||||
- stable-realization detection;
|
||||
- deterministic candidate generation;
|
||||
- candidate comparison against existing oracle set;
|
||||
- provenance/lineage from agentic ancestor to deterministic descendant;
|
||||
- manual acceptance step initially.
|
||||
|
||||
### Experiment
|
||||
|
||||
Run the same agentic realization repeatedly against a stable lab version, crystallize it, then execute without any model involvement.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- one agentic Verification Asset produces a deterministic test candidate;
|
||||
- deterministic execution preserves the relevant claims and oracles;
|
||||
- the generated/hardened test runs with zero agentic involvement;
|
||||
- lineage remains visible;
|
||||
- execution cost is measurably lower than agentic execution.
|
||||
|
||||
### Success Metrics
|
||||
|
||||
- crystallization success rate;
|
||||
- semantic coverage retained;
|
||||
- execution cost reduction.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Support or rejection for H-003.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 8 — Framework Finding → Improvement Loop
|
||||
|
||||
### Goal
|
||||
|
||||
Close the first actual self-improvement cycle.
|
||||
|
||||
### Implement
|
||||
|
||||
Workflow:
|
||||
|
||||
```text
|
||||
Framework Finding
|
||||
↓
|
||||
Classification
|
||||
↓
|
||||
Improvement Hypothesis
|
||||
↓
|
||||
Controlled Experiment
|
||||
↓
|
||||
Evidence
|
||||
↓
|
||||
Accept / Reject
|
||||
↓
|
||||
Self-Verification / Regression
|
||||
```
|
||||
|
||||
Use a real framework weakness discovered during Milestones 1–7 rather than inventing one if possible.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- a framework finding produces an explicit improvement hypothesis;
|
||||
- the hypothesis is experimentally evaluated;
|
||||
- the accepted change links back to concept and evidence;
|
||||
- the discovered framework failure leaves behind a permanent self-verification or deterministic regression;
|
||||
- the Concept ↔ Implementation Fitness Map is updated.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first completed **Concept–Implementation Fitness Loop**.
|
||||
|
||||
This is the milestone at which the self-improvement system genuinely exists.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 9 — Verification Energy v0
|
||||
|
||||
### Goal
|
||||
|
||||
Begin measuring test value without prematurely optimizing the scoring model.
|
||||
|
||||
### Implement
|
||||
|
||||
Record immutable Energy Events such as:
|
||||
|
||||
```text
|
||||
DEFECT_DETECTED
|
||||
REGRESSION_CAUGHT
|
||||
MECHANICAL_ADAPTATION
|
||||
SEMANTIC_ADAPTATION
|
||||
TEST_DEFECT
|
||||
FALSE_POSITIVE
|
||||
DUPLICATE
|
||||
CRYSTALLIZED
|
||||
USECASE_DEPRECATED
|
||||
```
|
||||
|
||||
Initially calculate only a simple transparent score.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Energy is derived from event history rather than stored as unexplained state;
|
||||
- every score change is explainable;
|
||||
- Energy can influence campaign priority;
|
||||
- criticality can override retirement;
|
||||
- no automatic deletion is implemented yet.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
A dataset suitable for later testing whether Energy actually predicts verification value.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 10 — First Compression Review
|
||||
|
||||
### Goal
|
||||
|
||||
Prevent the research prototype from turning into premature platform architecture.
|
||||
|
||||
### Review
|
||||
|
||||
Ask:
|
||||
|
||||
- Which concepts have no supporting evidence?
|
||||
- Which implementation abstractions have no conceptual justification?
|
||||
- Which metadata has not informed a decision?
|
||||
- Which agentic behavior can now be deterministic?
|
||||
- Which capabilities can be merged or removed?
|
||||
- What have the experiments falsified?
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- at least one simplification is seriously evaluated;
|
||||
- rejected concepts are marked as such rather than silently retained;
|
||||
- architecture reflects experimental learning;
|
||||
- updated Concept Model and Improvement Loop remain smaller or more precise where evidence permits.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first proof that self-improvement includes subtraction, not only accumulation.
|
||||
|
||||
---
|
||||
|
||||
# Recommended Execution Order
|
||||
|
||||
```text
|
||||
M0 Research Control Plane
|
||||
|
|
||||
M1 Deterministic Semantic Kernel
|
||||
|
|
||||
M2 Test-Driver Lab
|
||||
|
|
||||
M3 Self-Verification v0
|
||||
|
|
||||
M4 Agentic Realization
|
||||
|
|
||||
M5 Mechanical Adaptation
|
||||
|
|
||||
M6 Defect vs Adaptation
|
||||
|
|
||||
M7 Crystallization
|
||||
|
|
||||
M8 Closed Improvement Loop
|
||||
|
|
||||
M9 Verification Energy
|
||||
|
|
||||
M10 Compression Review
|
||||
```
|
||||
|
||||
The first major research gate is **M8**.
|
||||
|
||||
Before M8, test-driver has promising mechanisms.
|
||||
|
||||
At M8, it has demonstrated a complete evidence-driven self-improvement cycle.
|
||||
|
||||
---
|
||||
|
||||
# Prototype Success Gate
|
||||
|
||||
The initial research prototype should be considered successful enough to justify broader framework investment when it can demonstrate all of the following in one coherent system:
|
||||
|
||||
1. a multi-user use case expressed independently of implementation details;
|
||||
2. deterministic independent oracles;
|
||||
3. agentic realization of at least one semantic action;
|
||||
4. recovery from legitimate mechanical implementation change;
|
||||
5. rejection of a seeded semantic/security defect as a mere adaptation;
|
||||
6. reproducible evidence;
|
||||
7. crystallization into deterministic execution;
|
||||
8. a framework failure converted into a permanent self-verification;
|
||||
9. explicit concept-to-implementation traceability;
|
||||
10. measured human effort and execution cost.
|
||||
|
||||
That demonstration is more valuable than broad feature coverage.
|
||||
338
docs/TestDriverResearchPrototype.md
Normal file
338
docs/TestDriverResearchPrototype.md
Normal file
|
|
@ -0,0 +1,338 @@
|
|||
# Stage 1 Test Driver Validation
|
||||
|
||||
The biggest risk is not technical feasibility. It is that **test-driver becomes conceptually elegant but too broad to prove itself quickly**.
|
||||
|
||||
I would improve the odds of success by treating the next phase as an experiment in whether three specific ideas actually work:
|
||||
|
||||
1. **A use case can survive implementation change better than a conventional test script.**
|
||||
2. **Agentic execution can bridge unstable implementation without corrupting the intended semantics.**
|
||||
3. **Successful agentic tests can crystallize into cheaper deterministic tests.**
|
||||
|
||||
If those three work, the rest—energy, campaigns, security mutation, multi-user orchestration—has a strong foundation.
|
||||
|
||||
### Narrow the first battlefield
|
||||
|
||||
Choose exactly one real application and perhaps three use cases. They should deliberately include the difficult characteristics test-driver is meant to solve: authentication, several users, state transitions, authorization, and an evolving UI/API.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
UC-01 Alice creates a workspace
|
||||
UC-02 Alice invites Bob and Bob joins
|
||||
UC-03 Alice revokes Bob and Bob loses access
|
||||
```
|
||||
|
||||
UC-03 already gives you functional, interaction, temporal and security semantics.
|
||||
|
||||
Avoid building a generic testing platform first. Make test-driver extraordinarily good at this one sequence.
|
||||
|
||||
### Make the semantic layer the core intellectual property
|
||||
|
||||
The critical interface is not the LLM integration or browser automation. It is:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
↓
|
||||
Semantic Goal
|
||||
↓
|
||||
Semantic Action
|
||||
↓
|
||||
Concrete realization
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
grant_access(Bob, resource, READ)
|
||||
```
|
||||
|
||||
may currently mean six browser interactions.
|
||||
|
||||
Later it may mean an API operation.
|
||||
|
||||
Test-driver should care about the semantic action. Drivers care about realization.
|
||||
|
||||
If this abstraction is good, the framework survives technology changes. If it is poor, agentic execution becomes sophisticated screen scraping.
|
||||
|
||||
### Force the framework to distinguish discovery from truth
|
||||
|
||||
One of the strongest architectural principles should be:
|
||||
|
||||
> **Agents discover paths. Oracles establish truth.**
|
||||
|
||||
For the first implementation, make every important oracle deterministic.
|
||||
|
||||
For example, let the agent discover how Alice invites Bob through the UI, but verify independently through an API or database-facing test interface that:
|
||||
|
||||
```text
|
||||
membership(Bob, Workspace) == MEMBER
|
||||
membership(Carol, Workspace) == NONE
|
||||
```
|
||||
|
||||
Do not let the actor agent conclude, “It looks like Bob joined.”
|
||||
|
||||
This separation will prevent many future problems.
|
||||
|
||||
### Build reproducibility before intelligence
|
||||
|
||||
For every run, capture at least:
|
||||
|
||||
```text
|
||||
use-case version
|
||||
scenario version
|
||||
application version/commit
|
||||
actor identities and roles
|
||||
initial world
|
||||
semantic actions
|
||||
actual actions
|
||||
random seed
|
||||
timestamps
|
||||
observations
|
||||
oracle results
|
||||
screenshots/traces where useful
|
||||
agent/model/config version
|
||||
```
|
||||
|
||||
An agentically discovered failure that cannot be reproduced is much less valuable.
|
||||
|
||||
The first impressive demonstration should therefore not be “the agent found a bug.”
|
||||
|
||||
It should be:
|
||||
|
||||
> “The agent found a bug, test-driver reduced it to this scenario, and the failure can now be replayed deterministically.”
|
||||
|
||||
### Treat crystallization as an explicit deliverable
|
||||
|
||||
Don't postpone crystallization until later.
|
||||
|
||||
Make the first milestone contain this lifecycle:
|
||||
|
||||
```text
|
||||
new use case
|
||||
↓
|
||||
agentic execution
|
||||
↓
|
||||
stable semantic trajectory
|
||||
↓
|
||||
candidate deterministic implementation
|
||||
↓
|
||||
deterministic regression
|
||||
```
|
||||
|
||||
You need to learn early whether this transition can actually be automated or assisted effectively.
|
||||
|
||||
A useful success metric might be:
|
||||
|
||||
> **How many agentic verification assets can be downgraded to deterministic execution without losing semantic coverage?**
|
||||
|
||||
That is much more meaningful than counting generated tests.
|
||||
|
||||
### Introduce energy only after you have event history
|
||||
|
||||
I like Test Energy a lot, but I would avoid optimizing its formula early.
|
||||
|
||||
Start by recording events:
|
||||
|
||||
```text
|
||||
found-defect
|
||||
false-positive
|
||||
mechanical-adaptation
|
||||
semantic-adaptation
|
||||
duplicate-detected
|
||||
usecase-changed
|
||||
crystallized
|
||||
regression-caught
|
||||
```
|
||||
|
||||
Then initially compute a crude score.
|
||||
|
||||
After a few hundred runs you can inspect whether the proposed energy changes actually correspond to human intuition about test value.
|
||||
|
||||
In other words:
|
||||
|
||||
> **Store the evidence first; invent the fitness function second.**
|
||||
|
||||
Otherwise you'll encode assumptions before you have data.
|
||||
|
||||
### Separate three kinds of change
|
||||
|
||||
This will probably become one of the framework's most important capabilities.
|
||||
|
||||
Whenever a test stops matching the system, classify the change as:
|
||||
|
||||
```text
|
||||
IMPLEMENTATION CHANGE
|
||||
same behavior, different realization
|
||||
|
||||
INTENT CHANGE
|
||||
the product is deliberately supposed to behave differently
|
||||
|
||||
DEFECT
|
||||
implementation no longer satisfies unchanged intent
|
||||
```
|
||||
|
||||
This classification is what makes adaptive tests safe.
|
||||
|
||||
A lot of test maintenance today implicitly assumes the first case. Test-driver must never make that assumption.
|
||||
|
||||
### Give humans a small number of meaningful decision points
|
||||
|
||||
Agentic systems fail when humans either have to approve everything or are expected to trust everything.
|
||||
|
||||
Aim for perhaps three escalation types:
|
||||
|
||||
```text
|
||||
SEMANTIC CHANGE
|
||||
"The expected workflow appears to have changed."
|
||||
|
||||
INVARIANT CONFLICT
|
||||
"The observed behavior conflicts with a protected invariant."
|
||||
|
||||
AMBIGUOUS VERDICT
|
||||
"Available evidence cannot establish correctness."
|
||||
```
|
||||
|
||||
Everything mechanical beneath that should eventually be autonomous.
|
||||
|
||||
That creates a good human/agent boundary.
|
||||
|
||||
### Make security an early proof point
|
||||
|
||||
Security testing is especially good for proving the framework because ordinary functional use cases naturally generate adversarial variants.
|
||||
|
||||
From:
|
||||
|
||||
```text
|
||||
Alice grants Bob access.
|
||||
```
|
||||
|
||||
derive:
|
||||
|
||||
```text
|
||||
Carol tries access.
|
||||
Bob attempts WRITE.
|
||||
Bob accesses before grant.
|
||||
Bob accesses after revoke.
|
||||
Bob substitutes another resource ID.
|
||||
Bob attempts cross-tenant access.
|
||||
```
|
||||
|
||||
If test-driver can systematically produce these from the ordinary use case, you already have something substantially more interesting than “an LLM controlling Playwright.”
|
||||
|
||||
### Use a deliberately boring implementation stack
|
||||
|
||||
The novelty should be in the verification model, not infrastructure.
|
||||
|
||||
For the first implementation, I'd prefer something like:
|
||||
|
||||
```text
|
||||
Python
|
||||
pytest
|
||||
Playwright
|
||||
Pydantic/dataclasses
|
||||
YAML
|
||||
SQLite
|
||||
OpenTelemetry where useful
|
||||
```
|
||||
|
||||
Maybe later a graph store, distributed workers, richer agent infrastructure, etc.
|
||||
|
||||
But initially:
|
||||
|
||||
> **One process, one database, one browser engine, one application under test.**
|
||||
|
||||
Complex infrastructure will hide conceptual flaws.
|
||||
|
||||
### Dogfood aggressively
|
||||
|
||||
Test-driver itself should become one of the systems tested by test-driver.
|
||||
|
||||
As soon as there is a usable CLI or web surface, introduce its own use cases:
|
||||
|
||||
```text
|
||||
define use case
|
||||
execute campaign
|
||||
inspect evidence
|
||||
accept finding
|
||||
crystallize test
|
||||
retire verification
|
||||
```
|
||||
|
||||
That gives you continuous pressure to make the framework usable rather than merely architecturally sophisticated.
|
||||
|
||||
### Define success quantitatively
|
||||
|
||||
I'd track a small scorecard from the first month:
|
||||
|
||||
| Measure | Why it matters |
|
||||
| -------------------------------------------- | ---------------- |
|
||||
| Time to express a new use case | usability |
|
||||
| Agentic scenario success rate | executability |
|
||||
| Mechanical adaptations handled automatically | fluidity benefit |
|
||||
| False semantic adaptations | safety |
|
||||
| Genuine defects discovered | effectiveness |
|
||||
| Defects reproducibly replayed | trustworthiness |
|
||||
| Agentic → deterministic crystallizations | core thesis |
|
||||
| Median execution cost per maturity level | economics |
|
||||
| Human interventions per 100 runs | autonomy |
|
||||
| Security variants derived per use case | leverage |
|
||||
|
||||
The key metric could eventually be something like:
|
||||
|
||||
> **Verified behavior per unit of human maintenance effort.**
|
||||
|
||||
That captures what test-driver is fundamentally trying to improve.
|
||||
|
||||
### Keep the architecture open, but the semantics opinionated
|
||||
|
||||
Drivers, agents and storage should be replaceable.
|
||||
|
||||
The following should not be casually replaceable:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
Actor
|
||||
Scenario
|
||||
Claim
|
||||
Invariant
|
||||
SemanticAction
|
||||
Observation
|
||||
Oracle
|
||||
VerificationAsset
|
||||
Finding
|
||||
Adaptation
|
||||
Crystallization
|
||||
EnergyEvent
|
||||
Lineage
|
||||
```
|
||||
|
||||
Those are the conceptual backbone.
|
||||
|
||||
If they change constantly, you don't yet understand the domain.
|
||||
|
||||
### And establish a ruthless non-goal list
|
||||
|
||||
For v0.1 I would explicitly not try to become:
|
||||
|
||||
```text
|
||||
a generic unit-test framework
|
||||
a load-testing replacement
|
||||
a fuzzing engine
|
||||
a vulnerability scanner
|
||||
a test-management SaaS
|
||||
a CI/CD system
|
||||
an observability platform
|
||||
a universal browser agent
|
||||
```
|
||||
|
||||
Test-driver may eventually integrate all of those.
|
||||
|
||||
Its unique purpose should remain:
|
||||
|
||||
> **Maintain trustworthy verification of use cases while software changes rapidly, using agentic flexibility where necessary and deterministic verification wherever possible.**
|
||||
|
||||
If I were running the project, the next concrete milestone would be a **Test Driver Research Prototype**, not yet a product: one target application, three linked multi-user use cases, deterministic oracles, one agentic browser driver, complete evidence capture, one mutation mechanism, and one successful crystallization into a deterministic regression test.
|
||||
|
||||
If that demonstrably works, we will have validated the hardest and most original part of the idea.
|
||||
|
||||
xxx
|
||||
Loading…
Add table
Add a link
Reference in a new issue