# TestDriver Improvement Loop **Status:** Concept v0.1 **Purpose:** Establish a self-improvement loop that keeps the implementation of `test-driver` aligned with its conceptual model while generating evidence about which concepts actually work. --- ## 1. Intent `test-driver` is itself an evolving software system. Its implementation must therefore be subject to the same principles it applies to systems under test: - intended behavior must remain distinguishable from implementation details; - change must generate evidence rather than silently redefine correctness; - exploratory mechanisms should harden into deterministic mechanisms as understanding grows; - failures should improve future verification; - obsolete complexity should be allowed to disappear. The improvement loop exists to continuously reconcile: 1. **Concept** — what test-driver claims should be true; 2. **Implementation** — what the framework actually does; 3. **Evidence** — what experiments and executions demonstrate. The loop should make conceptual drift visible and turn important framework failures into durable self-verification assets. --- ## 2. Core Principle > Every meaningful implementation increment should generate evidence about whether test-driver is becoming a better realization of its own concept. The framework therefore has two simultaneous feedback loops: ```text Concept / Intent / Hypotheses | v Select Experiment | v Implement | v Exercise on SUT | v Evidence | +----------+----------+ | | v v Product Finding Framework Finding | | v v Improve target Improve test-driver | v Reconcile with Concept | +-----> next cycle ``` A run may therefore produce findings about the system under test and findings about the verification framework itself. --- ## 3. Development as Experimental Work Implementation work should increasingly be framed as hypotheses rather than feature requests. Examples: ### H-001 — Semantic Action Stability > A semantic action survives implementation restructuring better than a recorded UI interaction sequence. ### H-002 — Mechanical Adaptation > An agentic driver can recover from a mechanical implementation change without modifying the semantics of the protected use case. ### H-003 — Crystallization > A sufficiently stable agentic execution can be converted into deterministic test code without losing relevant oracle coverage. ### H-004 — Independent Judgment > Separating actor execution from deterministic oracles reduces false-positive adaptation to defective behavior. ### H-005 — Verification Energy > Historical evidence about defects caught, adaptations required, false positives and duplication can identify verification assets whose continued execution is more valuable than others. Each hypothesis should have: - an identifier; - a claim; - a falsification condition; - one or more experiments; - evidence; - a status; - resulting implementation or concept changes. Suggested lifecycle: ```text PROPOSED | v EXPERIMENTING | +------> REJECTED | v SUPPORTED | v PRACTICALLY_VALIDATED | v ARCHITECTURAL ``` A hypothesis may also be reopened if later evidence contradicts it. --- ## 4. Concept–Implementation Fitness Map Every important concept should become traceable to the implementation and evidence that support it. Conceptual relationship: ```text Concept | +-- implementation +-- experiment +-- evidence +-- self-verification +-- unresolved questions ``` Example: ```yaml concept: actor-isolation claim: > One actor must not obtain private state, credentials, observations, or memory belonging to another actor except through modeled communication channels. implementation: - testdriver/runtime/actor_context.py experiments: - H-006 self_verifications: - td://self/actor-isolation status: supported ``` The map should expose two forms of drift. ### Conceptual Orphaning A concept is claimed but has no implementation or verification evidence. ### Implementation Orphaning A subsystem or abstraction exists without a concept, requirement, or experiment that explains why it is needed. Both should be visible during review. --- ## 5. Concept Drift A **Concept Drift Finding** occurs when implementation behavior diverges from an established concept without an explicit decision to revise that concept. Example: ```text Concept: Agents discover paths; independent oracles judge outcomes. Implementation: Browser agent declares the scenario successful. Finding: CONCEPT_DRIFT ``` A concept-drift finding must resolve in one of three ways: 1. implementation changes to match the concept; 2. the concept is deliberately revised; 3. an experiment demonstrates that the distinction is no longer useful. Implementation reality must not silently redefine the conceptual model. --- ## 6. Self-Verification `test-driver` should become a system under test for `test-driver`. Self-verification use cases use the namespace: ```text td://self/... ``` Initial candidates: ```text td://self/actor-isolation td://self/oracle-independence td://self/evidence-reproducibility td://self/mechanical-adaptation td://self/semantic-change-detection td://self/crystallization td://self/test-retirement ``` These scenarios should verify framework-level promises rather than implementation internals whenever possible. Example: ### `td://self/mechanical-adaptation` 1. execute a stable use case against lab version A; 2. change the UI mechanically without changing semantics; 3. rerun the use case; 4. allow agentic navigation to recover; 5. verify that the same semantic action and oracles remain valid; 6. record the adaptation and evidence. Expected result: ```text mechanical implementation change | v adaptation detected | v alternative realization discovered | v semantic action preserved | v original oracle still passes ``` --- ## 7. Test-Driver Lab A purpose-built mutable application should provide controlled evolutionary pressure for the framework. Suggested repository or module name: ```text test-driver-lab ``` The lab should be intentionally small but support: - multiple users; - organizations or tenants; - authentication; - resources; - sharing; - permissions; - simple workflows; - audit events; - API interaction; - browser interaction. The lab should also support deliberate implementation mutations. Examples: ```text M01 move or rename a UI control M02 replace the DOM structure M03 change a compatible API representation M04 add a legitimate workflow step M05 introduce an authorization defect M06 introduce eventual-consistency delay M07 introduce intermittent dependency failure M08 remove or deprecate a capability M09 create a concurrency race M10 change the intended business requirement ``` The purpose is not to build a representative product. It is to create a controlled environment in which claims about test-driver can be falsified. --- ## 8. Dual Mutation Mutation should operate in two directions. ### Use-Case Mutation Mutate actor, resource, order, timing, state, or permissions to test the robustness of the system under test. ```text UseCase Mutation | v tests robustness of application ``` ### Implementation Mutation Mutate the target implementation while holding the use-case semantics constant to test the robustness of test-driver. ```text Implementation Mutation | v tests robustness of test-driver ``` This duality allows the framework to test both the application and its own verification strategy. --- ## 9. Framework Findings The framework should maintain finding classes distinct from ordinary product defects. Initial set: ### PRODUCT_DEFECT The system under test violates unchanged intent. ### TEST_DEFECT The verification asset or oracle is incorrect. ### MECHANICAL_ADAPTATION Implementation mechanics changed while protected semantics remain equivalent. ### SEMANTIC_CHANGE The intended product behavior has changed. ### CONCEPT_DRIFT The implementation of test-driver no longer matches an established framework concept. ### FRAMEWORK_LIMITATION A valid scenario cannot be expressed, executed, observed, or judged adequately. ### EVIDENCE_FAILURE A finding cannot be reproduced or supported from the retained evidence. ### UNNECESSARY_COMPLEXITY An abstraction or subsystem adds material maintenance cost without sufficient conceptual or experimental justification. These findings feed the improvement loop. --- ## 10. Improvement Cycle The canonical loop is: ```text OBSERVE | v CLASSIFY | v EXPLAIN | v PROPOSE | v EXPERIMENT | v MEASURE | v ACCEPT / REJECT | v CRYSTALLIZE ``` ### Observe Collect evidence from test-driver runs, self-tests, implementation work and lab experiments. ### Classify Determine whether the observation represents a product defect, test defect, adaptation, concept drift, framework limitation or other finding class. ### Explain Produce the smallest useful causal explanation supported by evidence. ### Propose Generate one or more candidate improvements. ### Experiment Change one relevant variable where practical and attempt to falsify the proposed improvement. ### Measure Evaluate the result against explicit success criteria. ### Accept / Reject Retain improvements that produce sufficient evidence. Reject or revise those that do not. ### Crystallize Convert learned behavior into a more deterministic, cheaper and more maintainable form whenever possible. --- ## 11. Agentic Roles Self-improvement should not rely on one omnipotent self-modifying agent. Distinct roles create productive tension. ### Builder Implements the current hypothesis or improvement proposal. ### Experimenter Designs experiments intended to falsify claims. ### Critic Looks for false success, hidden assumptions and semantic drift. ### Auditor Checks concept-to-implementation traceability. ### Maintainer Looks for unnecessary abstractions, duplication and maintenance burden. These are conceptual roles. Initially they may simply correspond to separate prompts or workflow phases. --- ## 12. Fitness Scorecard The framework should be measured against its thesis rather than implementation volume. Initial dimensions: | Dimension | Example Measure | |---|---| | Adaptability | Mechanical changes recovered automatically | | Semantic integrity | False semantic adaptations | | Detection | Seeded defects correctly discovered | | Reproducibility | Findings replayable from retained evidence | | Crystallization | Agentic assets converted to deterministic execution | | Efficiency | Cost per verified use case | | Autonomy | Human interventions per 100 runs | | Robustness | Success across controlled implementation mutations | | Traceability | Concepts connected to implementation and evidence | | Simplicity | Complexity required per supported capability | A particularly important safety metric is: > **False Adaptation Rate:** the frequency with which test-driver treats an actual product defect as a legitimate adaptation. This should be aggressively minimized. --- ## 13. Concept Maturity Concepts should mature based on evidence rather than attractive terminology. Suggested levels: ```text C0 Idea C1 Hypothesis C2 Experimentally Supported C3 Practically Validated C4 Architectural Invariant ``` Examples at the beginning of the research prototype may be approximately: ```text Actor Isolation C2 Independent Oracles C2 Semantic Actions C1 Crystallization C1 Verification Energy C0-C1 ``` These classifications are provisional and should change with evidence. --- ## 14. Compression Self-improvement must include deletion. Learning does not necessarily imply adding features or abstractions. At regular intervals, perform a compression review: - Which concepts can be merged? - Which abstractions lack evidence? - Which agentic mechanisms can crystallize into deterministic code? - Which metadata has never informed a decision? - Which subsystem can be removed? - Which verification assets have become redundant? The desired outcome is not maximal capability count. It is: > **the smallest framework that reliably realizes the validated test-driver concepts.** --- ## 15. Improvement Evidence Every accepted framework improvement should retain: ```text Improvement ID Triggering finding(s) Affected concept(s) Hypothesis Experiment Before state After state Evidence Measured outcome Decision Resulting self-verification Resulting deterministic regression, if applicable ``` This creates a lineage from conceptual claim through evidence to implementation. --- ## 16. Initial Control Loop The first working version does not require autonomous self-modification. A minimal loop is sufficient: ```text Framework run | v Framework finding | v Human/agent classification | v Improvement hypothesis | v Controlled lab experiment | v Evidence | v Accept / reject | v Self-verification added ``` Only after this loop reliably produces good improvements should more of the process become agentic. --- ## 17. Success Condition The improvement loop is successful when test-driver can repeatedly demonstrate that: 1. conceptual claims are traceable to implementation and evidence; 2. implementation changes that violate those claims are detected; 3. controlled experiments can distinguish defects, adaptations and semantic changes; 4. important framework failures become durable self-verifications; 5. agentic mechanisms harden into deterministic mechanisms where possible; 6. the framework becomes simpler or more effective as evidence accumulates; 7. framework evolution does not silently redefine correctness. The long-term objective is a self-hosting verification system whose own evolution is governed by the evidence-driven principles it applies to other software.