test-driver/history/2026-08-22-concept-assessment-swot.md

test-driver — Concept Assessment (SWOT)

Date: 2026-08-22 Author: Claude (Opus 5) with Bernd Worsch Scope: Assessment of the pre-implementation concept corpus — INTENT.md, docs/TestDriverConceptModel.md, docs/TestDriverImprovementLoop.md, docs/TestDriverInitialMilestones.md, docs/TestDriverResearchPrototype.md. Repo state at assessment: 2,950 lines of concept documentation, 1 commit, no executable code, docs/ and INTENT.md untracked.

---

Orientation

The repository is concept-complete and code-empty. Four documents describe a use-case-driven verification framework whose thesis is that verification should mature alongside the behaviour it protects: fluid and agentic while software is hot, deterministic once it cools.

The ratio — roughly 3,000 lines of theory against zero lines executable — is the single most important fact shaping this assessment.

---

Strengths

S1 — The core thesis is original and load-bearing. The crystallization continuum (T0 Exploratory → T5 Deterministic) coupled to implementation Temperature is not a repackaging of "self-healing tests". It makes a falsifiable claim: verification mode should be a function of how fast the system under test is changing.

S2 — The right primitive is identified. SemanticAction as the bridge between agentic discovery and deterministic code generation is the correct pivot point, and the documents know it. grant_access(Bob, R, READ) surviving a DOM restructure is the product.

S3 — Safety is designed in, not bolted on. Oracle independence from actors, "implementation is not the truth", INCONCLUSIVE as a first-class verdict, and False Adaptation Rate as the headline safety metric. This is what separates the concept from self-healing test vendors, which structurally cannot distinguish a moved button from a broken authorization check.

S4 — Falsifiable research posture. H-001…H-005 carry falsification conditions; the lab is a measuring instrument; milestone exit criteria are stated as observable outcomes. Most framework projects have a roadmap — this has an experiment design.

S5 — Security-as-mutation is high leverage and cheap. Deriving twelve adversarial variants from one shared-resource use case is demonstrable early, valuable independently of the crystallization thesis, and requires no maturity beyond T1.

S6 — Anti-bloat discipline is pre-committed. Explicit non-goal list, M10 Compression Review, "self-improvement includes subtraction", and a deliberately boring stack (Python / pytest / Playwright / SQLite / YAML). The documents already contain their own best critique.

---

Weaknesses

W1 — Concept surface vastly exceeds validated ground. Roughly 40 canonical concepts, of which Energy, Confidence, Metabolism, Campaign selection, Retirement floors and Lineage graphs are untestable until M7+. Each is an invitation to build now and validate never.

W2 — Three documents already disagree. INTENT.md numbers milestones M0–M3; TestDriverInitialMilestones.md numbers M0–M10; TestDriverConceptModel.md §15 has a third M0–M3. Concept drift has appeared before any code exists — precisely the failure mode the Improvement Loop document was written to prevent.

W3 — The three hardest problems are the least specified. (a) How a SemanticAction retains identity across a changed surface — the binding and matching mechanism is nowhere described. (b) How adaptation is classified MECHANICAL vs SEMANTIC vs DEFECT without an LLM making the call, which principle 2.3 forbids. (c) How a captured trajectory becomes deterministic code with oracle coverage preserved. These three are the project; the remainder is scaffolding.

W4 — Energy is the weakest concept and the easiest to build. A dangerous combination. Proving that Energy "predicts verification value" requires years of history the prototype will never accumulate. M9 exists mainly to collect a dataset for a study that will not happen at this stage.

W5 — The evaluation set is far too small for its claims. Six seeded mutations (M01–M06) cannot support "classification precision/recall". Dozens of labelled mutations across several dimensions are needed before any rate is more than anecdote.

W6 — Self-verification is circular. td://self/oracle-independence uses the framework to test whether the framework's oracles are independent. Without an out-of-band assertion layer (plain pytest) as ground truth, M3 proves nothing.

W7 — No cost or nondeterminism model. Agentic runs are stochastic and expensive. Nothing budgets for model flake, retries, or per-run cost — yet "execution cost reduction" is a stated M7 success metric.

---

Opportunities

O1 — The timing window is open now. Agentic development is outrunning test maintenance in exactly the way the Why section describes. The pain is acute and current.

O2 — The lab is a publishable asset in its own right. A benchmark of seeded mechanical / semantic / defect mutations with ground-truth labels is something the field lacks entirely. It could earn credibility and contributors faster than the framework does, at a fraction of the cost.

O3 — "False Adaptation Rate" is a naming land-grab. If self-healing test tools come to be held against a metric this project defined, the framing is won regardless of adoption.

O4 — Integration rather than replacement lowers adoption cost to near zero. Crystallization that emits ordinary pytest / Playwright files drops into CI systems that already exist. Users can adopt the output without adopting the framework — and then adopt the framework to keep producing it.

O5 — Agent-native distribution. Exposed over MCP, this becomes the verification layer coding agents call on themselves — a far larger surface than "a test framework a human runs".

O6 — A real second system under test is already available. The Custodian ecosystem (multi-domain, cross-repo, permissioned) is a better dogfooding target than a toy lab and exercises the multi-user and tenant-boundary claims honestly.

---

Threats

T1 — One public false adaptation kills the thesis. If the framework ever normalises a genuine authorization defect as a legitimate mechanical change, the concept is dead and cannot be rescued by a better version. This asymmetry should shape every design decision.

T2 — The independence problem is philosophically serious. If an agent writes the code, an agent writes the use case, and an agent realises the test, then "oracles independent from actors" is procedurally true but epistemically thin. Intent artefacts need human or spec-derived provenance, or the guarantee is a shell game. This is not addressed anywhere in the documents.

T3 — Competitors with distribution. Testim / mabl / Functionize on the self-healing axis; agentic QA startups on the exploration axis; Playwright plus a competent agent covering the naive 80%. The differentiator — defect-vs-adaptation discrimination — must be demonstrated, not described, and the window is roughly 12–18 months.

T4 — The economics may not close. If agentic realisation costs more per run than simply asking an agent to rewrite the broken test, crystallization becomes an aesthetic preference rather than a value proposition. This must be measured early, not at M7.

T5 — Platform drift before evidence. Eleven milestones, a research control plane, five agentic roles and a ten- dimension fitness scorecard — for a project with no runnable code and one maintainer.

---

Recommendations for the first workplan

R1 — Change the sequence: build a narrow vertical spike, not layered milestones. The documented order (M0 registers → M1 kernel → M2 lab → M3 self-verification → M4 agentic …) completes four layers before the thesis is touched once. Invert it: drive one thread end to end — one use case (Alice/Bob/Carol) → deterministic kernel → lab with three mutations (M01 moved control, M02 changed DOM, M05 authorization defect) → agentic realisation of exactly one semantic action → adaptation classification → one crystallization. Thin at every layer, complete end to end. That addresses H-001/H-002/H-003 in one workplan instead of seven.

R2 — Make the classifier the centrepiece, not a downstream milestone. M6 (defect vs adaptation) is where the project either has a reason to exist or does not. Design the classification mechanism first — specifically, what deterministic evidence separates "the button moved" from "Bob can still read after revoke". If the honest answer is "an LLM decides", that violates principle 2.3 and must be resolved on paper before code.

R3 — Cap M0 at roughly half a day. Five hypothesis files, one fitness map, an ID convention. No CLI, no register tooling, no schema. The research control plane is overhead until there are readings to record.

R4 — Defer Energy, Temperature, Confidence, Campaigns and Retirement entirely. Record raw immutable EnergyEvents from day one — they cost nothing and cannot be reconstructed later — but implement no scoring, decay or selection logic. Each of these is cheap to build, satisfying to build, and impossible to validate at this stage.

R5 — Build the lab larger than feels necessary, and label ground truth. It is the measuring instrument for every claim the project makes; a weak lab caps the credibility of all downstream results. Target 15–20 labelled mutations rather than 6, each a reproducible toggle with a recorded expected classification. This is also O2 — the standalone asset.

R6 — Establish out-of-band ground truth before self-verification. Write the actor-isolation and oracle-independence checks as plain pytest against observable behaviour, outside the framework. Otherwise M3 is a system certifying itself.

R7 — Instrument cost and nondeterminism from the first agentic run. Tokens, wall time, retry count, run-to-run variance per maturity level. T4 is an existential economic question; the data is free to collect from run one and impossible to backfill.

R8 — Reconcile the three milestone numberings and commit docs/. Pick one canonical sequence and mark the others superseded. Untracked, mutually inconsistent design documents are exactly the CONCEPT_DRIFT finding the Improvement Loop defines — the project should catch it on itself before it catches anything else.

R9 — Fix a single quantitative gate for the first workplan. Proposed: agentic realisation recovers from M01 and M02 with zero claim or invariant changes, flags M05 as a Product Finding, and produces one deterministic test that runs with no model involvement. Achieving that is worth more than any other ten items on the roadmap; failing it is worth knowing early and cheaply.

R10 — Decide where use-case intent comes from. Record, in the first workplan, whether use cases are human-authored, spec-derived, or agent-generated, and what independence guarantee survives in each case (T2). This determines whether the central promise is real or procedural — a paragraph of thinking now versus a redesign later.

---

Summary judgement

The concept is strong, unusually well-critiqued by its own documents, and correctly identifies its own hardest problem. It is also roughly 3,000 lines of theory ahead of its evidence. The first workplan's job is to close that gap with the thinnest possible slice capable of falsifying the thesis — not to build the architecture the documents describe.