test-driver/workplans/TD-WP-0002-vertical-spike-crystallization.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	state_hub_workstream_id
TD-WP-0002	workplan	Vertical spike: falsify the crystallization thesis	infotech	test-driver	active	codex	custodian	2026-08-22	2026-08-22	68b252b8-03e7-55c7-8dec-ac13f6ca0aa1

Vertical spike: falsify the crystallization thesis

Why this workplan exists

The concept corpus (INTENT.md, docs/) describes eleven milestones that build four complete layers — research control plane, deterministic kernel, lab, self-verification — before the central thesis is exercised even once. The repo currently holds ~3,000 lines of theory and zero lines of executable code.

This workplan inverts that order. It drives one thin thread end to end through every layer of the model, so that the thesis can be supported or falsified cheaply and early:

one use case → deterministic kernel → labelled mutation lab
  → agentic realisation of one semantic action
  → adaptation classification → one crystallization

Reasoning and the full assessment behind this sequencing: history/2026-08-22-concept-assessment-swot.md.

Success gate

The spike succeeds when, in one coherent run:

1. agentic realisation recovers from lab mutations M01 (moved/renamed control) and M02 (restructured DOM) with zero changes to claims or invariants;
2. lab mutation M05 (authorization defect) is reported as a Product Finding, not adapted to;
3. one verification asset crystallizes into a deterministic test that runs with no model involvement and preserves the relevant oracles;
4. False Adaptation Rate = 0 across the labelled mutation set.

Failing this gate early and cheaply is a valid and valuable outcome. Passing it is worth more than any other ten items on the M0–M10 roadmap.

Explicitly deferred

Not in this workplan, by decision rather than omission — each is cheap to build, satisfying to build, and impossible to validate at this stage:

• Energy scoring, decay, and campaign selection (raw immutable EnergyEvents are recorded from the first run; no scoring logic is implemented);
• Temperature and Confidence as computed values;
• Test Metabolism, Campaigns, Retirement floors;
• Lineage graph storage beyond a parent pointer;
• the five agentic roles (Builder / Experimenter / Critic / Auditor / Maintainer);
• any surface beyond one HTTP API and one browser UI.

Constraints

• Stack: Python, pytest, Playwright, Pydantic/dataclasses, YAML, SQLite. One process, one database, one browser engine, one application under test.
• Novelty belongs in the verification model, never in the infrastructure.
• No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
• Evidence stratification S1/S2/S3 and claim provenance per docs/TestDriverClassificationDesign.md (decision fef5213f-ce9b-44c2-b327-a0b0ba4b6270). Adaptation writes only to S1.

---

Reconcile milestone numbering and commit the concept corpus

id: TD-WP-0002-T01
status: done
priority: high
state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a"

Three documents carry three different milestone sequences: INTENT.md (M0–M3), docs/TestDriverInitialMilestones.md (M0–M10), and docs/TestDriverConceptModel.md §15 (a third M0–M3). This is a CONCEPT_DRIFT finding by the project's own taxonomy, present before any code exists.

Pick one canonical sequence, mark the others superseded in place, and commit INTENT.md and docs/ to git (currently untracked). Record the drift as the first entry in the framework findings log — the project should catch this on itself before it catches anything else.

Done 2026-08-22. docs/TestDriverInitialMilestones.md (M0–M10) is canonical; the INTENT.md and Concept Model § 15 sequences carry superseded banners with a mapping table. Recorded as research/findings/F-0001-milestone-sequence-drift.md. INTENT.md and docs/ are now tracked. One residual carried to T10: the finding taxonomy has no class for document-to-document drift.

Decide intent provenance and the classification mechanism on paper

id: TD-WP-0002-T02
status: done
priority: high
state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63"

Two design questions gate everything downstream. Answer both in a written design note plus a State Hub decision record, before writing kernel code.

(a) The classifier. What deterministic evidence separates "the button moved" from "Bob can still read after revoke"? Specify the signal, not the intent. If the honest answer is "a model decides", that violates Concept Model §2.3 and must be resolved now rather than discovered at M6.

(b) Intent provenance. If an agent writes the implementation, an agent writes the use case, and an agent realises the test, then "oracles independent from actors" is procedurally true but epistemically thin. Record whether use cases are human-authored, spec-derived, or agent-generated, and what independence guarantee survives in each case.

Deliverable: docs/TestDriverClassificationDesign.md + one recorded decision.

Done 2026-08-22. Design note written; decision fef5213f-ce9b-44c2-b327-a0b0ba4b6270 recorded in the hub.

Outcome in one line: adaptation may write only to surface evidence (S1); claims and invariants are run inputs with no adaptation write path. False Adaptation Rate = 0 therefore becomes an architectural property rather than a tuning target — the system cannot express "accept a defect as an adaptation". Classification is a total function over three signals defaulting to AMBIGUOUS; a surface change concurrent with a verdict change classifies as PRODUCT_DEFECT, never as adaptation. Claims require causally independent provenance.

Two consequences that change later tasks:

• D-07 — an independent observation channel into the system under test is now an explicit adoption precondition, not an implementation detail. T05 must expose one in the lab; without it S2/S3 collapse into S1.
• A fourth self-test, td://self/intent-independence, joins the T06 set.

Minimal research control plane

id: TD-WP-0002-T03
status: done
priority: medium
state_hub_task_id: "31b88522-decd-5f4c-b0e2-05b6981f33b8"

Timebox: half a day. Five hypothesis files (H-001…H-005) each with an explicit falsification condition, one Concept ↔ Implementation Fitness Map, and a stable identifier convention. Plain files under research/.

No CLI, no register tooling, no schema. The control plane is overhead until there are readings to record.

Done 2026-08-22. research/ holds H-001…H-005 (each with an explicit falsification condition), E-001…E-003, the fitness map, the findings log and the ID convention.

Three things worth carrying forward:

• The fitness map corrects the Improvement Loop § 13 starting classifications downward. Nothing can honestly exceed C1 with no implementation; C2 was aspirational.
• H-005 (Energy) is dormant by decision — untestable at this scale, so events are captured and no scoring is written.
• H-001 requires a real control arm (a robust recorded selector sequence). A straw-man control would make the semantic-action thesis trivially true and worthless.

Deterministic semantic kernel

id: TD-WP-0002-T04
status: done
priority: high
state_hub_task_id: "ffbcc9e8-c1bd-5c50-b62d-495ad9e135a4"

Minimal executable representations of UseCase, Actor, World, Scenario, SemanticAction, Observation, Evidence, Oracle, Verdict, VerificationAsset, Run. One deterministic driver (HTTP or direct adapter).

Run the reference use case end to end:

Alice owns resource R. Alice grants Bob READ. Bob can read R.
Carol cannot read R. Alice revokes Bob. Bob can no longer read R.

Exit: Alice, Bob and Carol hold genuinely isolated identities and sessions; all outcomes are judged by independent deterministic oracles; a run emits a structured Evidence Pack; the scenario replays from known initial state.

Emit raw EnergyEvent records from this point onward. Implement no scoring.

Done 2026-08-22. src/testdriver/ (11 modules), lab/minimal.py, scenarios/alice_bob_carol.py, 16 passing tests. The reference scenario runs end to end and replays identically from the same seed; evidence comes out stratified 3/3/3 across S1/S2/S3.

Three things that came out of building it rather than designing it:

• The observation channel needs two probes, not one. Reading stored state alone verifies test-driver's reimplementation of the rules rather than the system's enforcement of them; probing enforcement alone cannot notice that record and enforcement disagree. The lab exposes both, and their disagreement is now an invariant (i-enforcement-matches-record). That invariant is what catches an authorization defect which leaves the audit trail looking correct.
• A seeded RevokeIsCosmetic lab already fails the run — both the claim and the independent invariant fire, and the claim set is provably untouched. Early evidence for H-004, though not yet the experiment.
• Scenarios are Python, not YAML. Claims are predicates over observations; a YAML dialect able to express them would be a programming language with worse tooling. Revisit once we know which predicates actually recur.

Test-driver lab with labelled ground truth

id: TD-WP-0002-T05
status: done
priority: high
state_hub_task_id: "595a89c2-1462-57fc-8b08-a5a6b875fd48"

A deliberately small application: users, tenants, auth, resources, sharing, read/write permissions, revoke, audit history, HTTP API, minimal browser UI.

Build 15–20 labelled mutations, not the six sketched in the milestones doc. Six cannot support any statement about precision or recall. Each mutation is a reproducible toggle carrying a recorded expected classification (MECHANICAL / SEMANTIC / DEFECT), with baseline and mutated versions carrying explicit version identifiers.

The lab is the measuring instrument for every claim the framework makes — a weak lab caps the credibility of all downstream results. It is also potentially the project's first publishable artefact in its own right.

Done 2026-08-22. lab/app.py (users, tenants, auth, resources, sharing, read/write, revoke, audit), lab/http_api.py (JSON API + browser UI, stdlib only), lab/mutations.py (20 labelled, composable, version-stamped mutations), lab/GROUND-TRUTH.md. 48 tests pass. Detection: MECHANICAL 0/10 flagged, DEFECT 6/6, SEMANTIC 2/4.

Three results worth carrying:

• F-0002 — two seeded defects were initially invisible. M16 (READ grant confers WRITE) and M18 (revocation unaudited) both passed cleanly: nothing flaky, nothing INCONCLUSIVE, the framework simply had nothing to say. A use case protects exactly what it asserts and not one thing more. Resolved by adding two claims already stated as intent in INTENT.md. Had the six-mutation catalogue from the milestones doc been used, this would never have surfaced — the argument for the larger catalogue is now evidenced rather than asserted.
• The test-id axis. Stable data-td selectors survive most UI mutations, which would make H-001 trivially false. Rather than rig the catalogue, mutations now vary on preserves_test_ids, and H-001 must be analysed split by that axis. A semantic action earns its keep exactly where stable identifiers are absent or not carried forward — that is the honest shape of the claim.
• M12 vs M19 are behaviourally identical. A deliberate deferred-revocation decision and a revocation race produce the same failure, same step, same evidence. Only intent separates them. This is the discrimination problem in its sharpest form and is now a test, not a paragraph.

Out-of-band ground truth for self-verification

id: TD-WP-0002-T06
status: todo
priority: medium
state_hub_task_id: "4fb31485-8650-556c-bee4-b4e067a89d04"

Write the actor-isolation and oracle-independence checks as plain pytest against observable behaviour, outside the framework. Using test-driver to verify that test-driver's oracles are independent is a system certifying itself.

Exit: deliberately breaking actor isolation fails the out-of-band check; allowing an actor to determine its own verdict fails oracle-independence; each failure produces a Framework Finding.

Agentic realisation of one semantic action, fully instrumented

id: TD-WP-0002-T07
status: todo
priority: high
state_hub_task_id: "b58cf7ce-dc18-5218-8eac-179291626467"

Browser driver plus one agentic actor runtime, realising exactly one semantic action — grant_access(Bob, R, READ) — from intent, against the lab UI. Strict per-actor context isolation; bounded navigation and tool permissions; full action and evidence recording; agent/model/configuration identity captured in evidence. Oracles stay deterministic.

Instrument cost and nondeterminism from the very first run: tokens, wall time, retry count, and run-to-run variance. If agentic realisation costs more per run than simply asking an agent to rewrite the broken test, the crystallization argument is an aesthetic preference rather than a value proposition. This data is free to collect from run one and impossible to backfill.

Adaptation detection and the defect-vs-adaptation classifier

id: TD-WP-0002-T08
status: todo
priority: high
state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0"

The centrepiece. Implement the mechanism designed in T02 and run it across the full labelled mutation set from T05.

Classification path: IMPLEMENTATION CHANGE / INTENT CHANGE / PRODUCT DEFECT / AMBIGUOUS, with escalation for semantic changes and ambiguity. Preserve semantic action identity across adaptation; record adaptation history on the verification asset; never modify claims or invariants to accommodate an observed behaviour.

Measure: Mechanical Recovery Rate, classification precision/recall, and False Adaptation Rate. The last is the project's existential safety metric — one publicly normalised authorization defect kills the thesis permanently. Target is zero across the set, and a non-zero result is a stop-and-redesign signal, not a tuning exercise.

Crystallize one asset into deterministic regression

id: TD-WP-0002-T09
status: todo
priority: high
state_hub_task_id: "855a1f41-b839-57fc-87b8-198ce2a9f6b1"

Semantic action trajectory capture, stable-realisation detection, deterministic candidate generation, candidate comparison against the existing oracle set, and lineage from the agentic ancestor to the deterministic descendant. Manual acceptance step for now.

Emit ordinary pytest/Playwright code. Output that drops into a CI system which already exists lets a user adopt the result without adopting the framework.

Exit: the generated test runs with zero agentic involvement, preserves the relevant claims and oracles, retains visible lineage, and measurably costs less to execute than the agentic ancestor.

Gate review and first compression pass

id: TD-WP-0002-T10
status: todo
priority: medium
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"

Evaluate the four success-gate criteria against collected evidence and write the result up in history/ regardless of outcome.

Then run the compression questions from Milestone 10 while the spike is still small: which concepts have no supporting evidence, which abstractions have no conceptual justification, which metadata never informed a decision, what did the experiments falsify. Mark rejected concepts as rejected rather than silently retaining them, and update the Concept ↔ Implementation Fitness Map.

Convert at least one framework finding discovered during T04–T09 into a permanent self-verification or deterministic regression — that closes the first genuine Concept–Implementation Fitness Loop.