test-driver/workplans/TD-WP-0002-vertical-spike-crystallization.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	state_hub_workstream_id
TD-WP-0002	workplan	Vertical spike: falsify the crystallization thesis	infotech	test-driver	finished	codex	custodian	2026-08-22	2026-08-23	68b252b8-03e7-55c7-8dec-ac13f6ca0aa1

Vertical spike: falsify the crystallization thesis

Why this workplan exists

The concept corpus (INTENT.md, docs/) describes eleven milestones that build four complete layers — research control plane, deterministic kernel, lab, self-verification — before the central thesis is exercised even once. The repo currently holds ~3,000 lines of theory and zero lines of executable code.

This workplan inverts that order. It drives one thin thread end to end through every layer of the model, so that the thesis can be supported or falsified cheaply and early:

one use case → deterministic kernel → labelled mutation lab
  → agentic realisation of one semantic action
  → adaptation classification → one crystallization

Reasoning and the full assessment behind this sequencing: history/2026-08-22-concept-assessment-swot.md.

Success gate

The spike succeeds when, in one coherent run:

1. agentic realisation recovers from lab mutations M01 (moved/renamed control) and M02 (restructured DOM) with zero changes to claims or invariants;
2. lab mutation M05 (authorization defect) is reported as a Product Finding, not adapted to;
3. one verification asset crystallizes into a deterministic test that runs with no model involvement and preserves the relevant oracles;
4. False Adaptation Rate = 0 across the labelled mutation set.

Failing this gate early and cheaply is a valid and valuable outcome. Passing it is worth more than any other ten items on the M0–M10 roadmap.

Explicitly deferred

Not in this workplan, by decision rather than omission — each is cheap to build, satisfying to build, and impossible to validate at this stage:

• Energy scoring, decay, and campaign selection (raw immutable EnergyEvents are recorded from the first run; no scoring logic is implemented);
• Temperature and Confidence as computed values;
• Test Metabolism, Campaigns, Retirement floors;
• Lineage graph storage beyond a parent pointer;
• the five agentic roles (Builder / Experimenter / Critic / Auditor / Maintainer);
• any surface beyond one HTTP API and one browser UI.

Constraints

• Stack: Python, pytest, Playwright, Pydantic/dataclasses, YAML, SQLite. One process, one database, one browser engine, one application under test.
• Novelty belongs in the verification model, never in the infrastructure.
• No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
• Evidence stratification S1/S2/S3 and claim provenance per docs/TestDriverClassificationDesign.md (decision fef5213f-ce9b-44c2-b327-a0b0ba4b6270). Adaptation writes only to S1.

---

Reconcile milestone numbering and commit the concept corpus

id: TD-WP-0002-T01
status: done
priority: high
state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a"

Three documents carry three different milestone sequences: INTENT.md (M0–M3), docs/TestDriverInitialMilestones.md (M0–M10), and docs/TestDriverConceptModel.md §15 (a third M0–M3). This is a CONCEPT_DRIFT finding by the project's own taxonomy, present before any code exists.

Pick one canonical sequence, mark the others superseded in place, and commit INTENT.md and docs/ to git (currently untracked). Record the drift as the first entry in the framework findings log — the project should catch this on itself before it catches anything else.

Done 2026-08-22. docs/TestDriverInitialMilestones.md (M0–M10) is canonical; the INTENT.md and Concept Model § 15 sequences carry superseded banners with a mapping table. Recorded as research/findings/F-0001-milestone-sequence-drift.md. INTENT.md and docs/ are now tracked. One residual carried to T10: the finding taxonomy has no class for document-to-document drift.

Decide intent provenance and the classification mechanism on paper

id: TD-WP-0002-T02
status: done
priority: high
state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63"

Two design questions gate everything downstream. Answer both in a written design note plus a State Hub decision record, before writing kernel code.

(a) The classifier. What deterministic evidence separates "the button moved" from "Bob can still read after revoke"? Specify the signal, not the intent. If the honest answer is "a model decides", that violates Concept Model §2.3 and must be resolved now rather than discovered at M6.

(b) Intent provenance. If an agent writes the implementation, an agent writes the use case, and an agent realises the test, then "oracles independent from actors" is procedurally true but epistemically thin. Record whether use cases are human-authored, spec-derived, or agent-generated, and what independence guarantee survives in each case.

Deliverable: docs/TestDriverClassificationDesign.md + one recorded decision.

Done 2026-08-22. Design note written; decision fef5213f-ce9b-44c2-b327-a0b0ba4b6270 recorded in the hub.

Outcome in one line: adaptation may write only to surface evidence (S1); claims and invariants are run inputs with no adaptation write path. False Adaptation Rate = 0 therefore becomes an architectural property rather than a tuning target — the system cannot express "accept a defect as an adaptation". Classification is a total function over three signals defaulting to AMBIGUOUS; a surface change concurrent with a verdict change classifies as PRODUCT_DEFECT, never as adaptation. Claims require causally independent provenance.

Two consequences that change later tasks:

• D-07 — an independent observation channel into the system under test is now an explicit adoption precondition, not an implementation detail. T05 must expose one in the lab; without it S2/S3 collapse into S1.
• A fourth self-test, td://self/intent-independence, joins the T06 set.

Minimal research control plane

id: TD-WP-0002-T03
status: done
priority: medium
state_hub_task_id: "31b88522-decd-5f4c-b0e2-05b6981f33b8"

Timebox: half a day. Five hypothesis files (H-001…H-005) each with an explicit falsification condition, one Concept ↔ Implementation Fitness Map, and a stable identifier convention. Plain files under research/.

No CLI, no register tooling, no schema. The control plane is overhead until there are readings to record.

Done 2026-08-22. research/ holds H-001…H-005 (each with an explicit falsification condition), E-001…E-003, the fitness map, the findings log and the ID convention.

Three things worth carrying forward:

• The fitness map corrects the Improvement Loop § 13 starting classifications downward. Nothing can honestly exceed C1 with no implementation; C2 was aspirational.
• H-005 (Energy) is dormant by decision — untestable at this scale, so events are captured and no scoring is written.
• H-001 requires a real control arm (a robust recorded selector sequence). A straw-man control would make the semantic-action thesis trivially true and worthless.

Deterministic semantic kernel

id: TD-WP-0002-T04
status: done
priority: high
state_hub_task_id: "ffbcc9e8-c1bd-5c50-b62d-495ad9e135a4"

Minimal executable representations of UseCase, Actor, World, Scenario, SemanticAction, Observation, Evidence, Oracle, Verdict, VerificationAsset, Run. One deterministic driver (HTTP or direct adapter).

Run the reference use case end to end:

Alice owns resource R. Alice grants Bob READ. Bob can read R.
Carol cannot read R. Alice revokes Bob. Bob can no longer read R.

Exit: Alice, Bob and Carol hold genuinely isolated identities and sessions; all outcomes are judged by independent deterministic oracles; a run emits a structured Evidence Pack; the scenario replays from known initial state.

Emit raw EnergyEvent records from this point onward. Implement no scoring.

Done 2026-08-22. src/testdriver/ (11 modules), lab/minimal.py, scenarios/alice_bob_carol.py, 16 passing tests. The reference scenario runs end to end and replays identically from the same seed; evidence comes out stratified 3/3/3 across S1/S2/S3.

Three things that came out of building it rather than designing it:

• The observation channel needs two probes, not one. Reading stored state alone verifies test-driver's reimplementation of the rules rather than the system's enforcement of them; probing enforcement alone cannot notice that record and enforcement disagree. The lab exposes both, and their disagreement is now an invariant (i-enforcement-matches-record). That invariant is what catches an authorization defect which leaves the audit trail looking correct.
• A seeded RevokeIsCosmetic lab already fails the run — both the claim and the independent invariant fire, and the claim set is provably untouched. Early evidence for H-004, though not yet the experiment.
• Scenarios are Python, not YAML. Claims are predicates over observations; a YAML dialect able to express them would be a programming language with worse tooling. Revisit once we know which predicates actually recur.

Test-driver lab with labelled ground truth

id: TD-WP-0002-T05
status: done
priority: high
state_hub_task_id: "595a89c2-1462-57fc-8b08-a5a6b875fd48"

A deliberately small application: users, tenants, auth, resources, sharing, read/write permissions, revoke, audit history, HTTP API, minimal browser UI.

Build 15–20 labelled mutations, not the six sketched in the milestones doc. Six cannot support any statement about precision or recall. Each mutation is a reproducible toggle carrying a recorded expected classification (MECHANICAL / SEMANTIC / DEFECT), with baseline and mutated versions carrying explicit version identifiers.

The lab is the measuring instrument for every claim the framework makes — a weak lab caps the credibility of all downstream results. It is also potentially the project's first publishable artefact in its own right.

Done 2026-08-22. lab/app.py (users, tenants, auth, resources, sharing, read/write, revoke, audit), lab/http_api.py (JSON API + browser UI, stdlib only), lab/mutations.py (20 labelled, composable, version-stamped mutations), lab/GROUND-TRUTH.md. 48 tests pass. Detection: MECHANICAL 0/10 flagged, DEFECT 6/6, SEMANTIC 2/4.

Three results worth carrying:

• F-0002 — two seeded defects were initially invisible. M16 (READ grant confers WRITE) and M18 (revocation unaudited) both passed cleanly: nothing flaky, nothing INCONCLUSIVE, the framework simply had nothing to say. A use case protects exactly what it asserts and not one thing more. Resolved by adding two claims already stated as intent in INTENT.md. Had the six-mutation catalogue from the milestones doc been used, this would never have surfaced — the argument for the larger catalogue is now evidenced rather than asserted.
• The test-id axis. Stable data-td selectors survive most UI mutations, which would make H-001 trivially false. Rather than rig the catalogue, mutations now vary on preserves_test_ids, and H-001 must be analysed split by that axis. A semantic action earns its keep exactly where stable identifiers are absent or not carried forward — that is the honest shape of the claim.
• M12 vs M19 are behaviourally identical. A deliberate deferred-revocation decision and a revocation race produce the same failure, same step, same evidence. Only intent separates them. This is the discrimination problem in its sharpest form and is now a test, not a paragraph.

Out-of-band ground truth for self-verification

id: TD-WP-0002-T06
status: done
priority: medium
state_hub_task_id: "4fb31485-8650-556c-bee4-b4e067a89d04"

Write the actor-isolation and oracle-independence checks as plain pytest against observable behaviour, outside the framework. Using test-driver to verify that test-driver's oracles are independent is a system certifying itself.

Exit: deliberately breaking actor isolation fails the out-of-band check; allowing an actor to determine its own verdict fails oracle-independence; each failure produces a Framework Finding.

Done 2026-08-22. tests/selfverification/ — checks as plain functions over a serialized Evidence Pack, plus 12 tests that they hold and 12 that they can fail. All four td://self/... identifiers are covered. 72 tests pass overall.

• The strongest check turned out to be verdict reproducibility from S3 alone: re-evaluating each assertion against the stored snapshots, with no actor, driver or live system in reach, must reproduce exactly what the run reported. It is asserted on failing runs too — a check that only works when everything is green verifies luck, not independence.
• F-0003 (open) — actor isolation leaves no trace in ordinary evidence. The self-test catches a shared memory store only because the harness plants per-actor canaries first; without them, a run where all three actors share one memory dict produces an Evidence Pack identical to a correct one. Isolation is currently a property of a scenario written to expose it, not of runs in general. Carried to T10, reassess after T07.
• The same finding notes the mirror-image problem: oracle independence is enforced by construction, so no real run can produce the violating artefact and the check must be exercised against a hand-built pack. A guarantee enforced by construction cannot be verified by observing real runs. Four green self-tests are not four equivalent proofs, and the suite now says so.

Agentic realisation of one semantic action, fully instrumented

id: TD-WP-0002-T07
status: done
priority: high
state_hub_task_id: "b58cf7ce-dc18-5218-8eac-179291626467"

Browser driver plus one agentic actor runtime, realising exactly one semantic action — grant_access(Bob, R, READ) — from intent, against the lab UI. Strict per-actor context isolation; bounded navigation and tool permissions; full action and evidence recording; agent/model/configuration identity captured in evidence. Oracles stay deterministic.

Instrument cost and nondeterminism from the very first run: tokens, wall time, retry count, and run-to-run variance. If agentic realisation costs more per run than simply asking an agent to rewrite the broken test, the crystallization argument is an aesthetic preference rather than a value proposition. This data is free to collect from run one and impossible to backfill.

Done 2026-08-22. html.py (stdlib DOM), agentic.py (two runtimes), browser.py (per-actor sessions over real HTTP), scenarios/browser_grant.py. 107 tests pass. Two decisions taken with the operator: stdlib HTML driver instead of Playwright (F-0004) and a deterministic discovery runtime instead of a live model, both behind interfaces that let the alternatives drop in later.

The headline result is a narrowing, not a confirmation:

	Discovery (agentic)	Recorded selectors (control)
test ids preserved (9 mutations)	9/9	9/9
test ids dropped (3 mutations)	2/3	0/3

• F-0005 — semantic actions earn their keep more narrowly than claimed. Where an application keeps stable identifiers, the conventional approach matches the agentic one exactly and is cheaper, faster and deterministic. The semantic action wins only where identifiers are absent or not carried forward. Filed as CONCEPT_DRIFT: the concept model overstates this and should be revised to match the evidence. Three mutations on the deciding side is directionally clear and statistically nothing — recorded rather than rounded up.
• M22 marks where a scripted runtime stops. Renaming form fields defeats the heuristic, but the page still carries a "Person" label a model could read. This is the first concrete evidence that a live model would add capability rather than only cost — worth more than the general argument that it might.
• Two mutations (M21, M22) were added mid-task because the deciding side of the test-id axis was N=1 after the first run. Extending the instrument when the evidence shows it is too thin is the intended behaviour.
• Recovery happened with no claim or invariant diff in any run, and the one failure failed loudly — RealizationFailed in evidence, not a silent pass.

Adaptation detection and the defect-vs-adaptation classifier

id: TD-WP-0002-T08
status: done
priority: high
state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0"

The centrepiece. Implement the mechanism designed in T02 and run it across the full labelled mutation set from T05.

Classification path: IMPLEMENTATION CHANGE / INTENT CHANGE / PRODUCT DEFECT / AMBIGUOUS, with escalation for semantic changes and ambiguity. Preserve semantic action identity across adaptation; record adaptation history on the verification asset; never modify claims or invariants to accommodate an observed behaviour.

Measure: Mechanical Recovery Rate, classification precision/recall, and False Adaptation Rate. The last is the project's existential safety metric — one publicly normalised authorization defect kills the thesis permanently. Target is zero across the set, and a non-zero result is a stop-and-redesign signal, not a tuning exercise.

Done 2026-08-22. classification.py, CompositeDriver, scenarios/full_journey.py (one asset crossing both surfaces). 141 tests pass.

False Adaptation Rate = 0/7, including the three E-003 attacks. The other side of the trade is asserted too — 11 of 12 mechanical mutations are absorbed without a human, so the safety result is not bought by escalating everything.

Ground truth	Accepted without a human
MECHANICAL (12)	11
SEMANTIC (4)	2 — both genuinely inert here
DEFECT (7)	0

• F-0006 (resolved) — the T02 decision table listed SEMANTIC_CHANGE as an outcome it could produce. It cannot: M12 and M19 are behaviourally identical, as the lab has asserted since T05. PRODUCT_DEFECT and SEMANTIC_CHANGE collapse into one escalating outcome, BEHAVIOUR_CHANGED, and the distinction becomes a recorded human adjudication. INTENT_CHANGED survives but is detected by the claim fingerprint moving — a fact about the recorded use case, not an inference about behaviour. Erring toward "false alarm" costs one review; erring toward "false adaptation" ends the project.
• Two classifier defects found and fixed rather than reported. Claims downstream of a failed realization now yield INCONCLUSIVE rather than FAIL — accusing the system of a defect on the strength of the test's own inability to act is the mirror image of a false adaptation. And the browser driver now records a page signature, so a surface change is detectable even when the chosen interaction path is identical.
• What this does not establish. FAR = 0 follows largely from architecture (D-02, plus SAFE_TO_ACCEPT being a closed two-element set). The experiment confirms the architecture behaves as designed over 23 hand-written mutations; it says nothing about mutations nobody thought of. That distinction should survive into any external claim.

Crystallize one asset into deterministic regression

id: TD-WP-0002-T09
status: done
priority: high
state_hub_task_id: "855a1f41-b839-57fc-87b8-198ce2a9f6b1"

Semantic action trajectory capture, stable-realisation detection, deterministic candidate generation, candidate comparison against the existing oracle set, and lineage from the agentic ancestor to the deterministic descendant. Manual acceptance step for now.

Emit ordinary pytest/Playwright code. Output that drops into a CI system which already exists lets a user adopt the result without adopting the framework.

Exit: the generated test runs with zero agentic involvement, preserves the relevant claims and oracles, retains visible lineage, and measurably costs less to execute than the agentic ancestor.

Done 2026-08-23. crystallization.py, crystallized/test_grant_access.py (generated), 163 tests pass. All four exit criteria met.

• Freezing requires the same path across several runs, not one success. One run proves the agent can find a way; it does not show the surface has stopped moving, and freezing on a single observation is how a crystallized test becomes flaky the first time a page renders differently.
• Reversibility demonstrated in both directions. M24 (grant endpoint renamed) breaks the frozen path; the descendant fails loudly rather than falling back to searching — falling back would quietly turn a T5 asset into a T1 one and hide the signal crystallization exists to produce — while the agentic ancestor recovers from the same mutation. That is what distinguishes crystallization from ordinary codegen: the agentic form is not discarded.
• F-0007 (open) — the economic case is unmeasurable. The descendant is 54% cheaper, and that number should not be quoted in support of the thesis. The T07 runtime consumes zero tokens, so the entire saving is one page fetch, one parse and a two-candidate scoring pass. The saving crystallization actually claims — model tokens, latency, retry variance — is two or three orders of magnitude larger and entirely absent from this measurement.
• The generated test is not fully standalone: realization is plain urllib, but assertions are imported rather than restated, because a generated test that paraphrases its claims creates a second unverified statement of intent. Right call, and it qualifies the "drops into existing CI" story — what drops in is the realization; the claims still need the use-case module.

Gate review and first compression pass

id: TD-WP-0002-T10
status: done
priority: medium
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"

Evaluate the four success-gate criteria against collected evidence and write the result up in history/ regardless of outcome.

Then run the compression questions from Milestone 10 while the spike is still small: which concepts have no supporting evidence, which abstractions have no conceptual justification, which metadata never informed a decision, what did the experiments falsify. Mark rejected concepts as rejected rather than silently retaining them, and update the Concept ↔ Implementation Fitness Map.

Convert at least one framework finding discovered during T04–T09 into a permanent self-verification or deterministic regression — that closes the first genuine Concept–Implementation Fitness Loop.

Done 2026-08-23. Full write-up: history/2026-08-23-td-wp-0002-gate-review.md. 178 tests pass.

All four gate criteria met. M01/M02 absorbed with the claim set unchanged; the authorization defects escalated with regressions named; one asset crystallized and running with no model; False Adaptation Rate 0/7, with 12 of 13 mechanical mutations absorbed so the safety result is not bought by escalating everything.

Fitness loop closed via F-0003. Actor isolation left no trace in ordinary evidence — it was a property of scenarios written to expose it. Every actor is now seeded with an automatic private marker and the runner examines all actors on every scenario, with two permanent regressions behind it, one of which catches a leak with the harness planting nothing.

Compression — six abstractions removed, each declared and never used: Verdict.SUSPICIOUS (a verdict no oracle could emit), Step.expect_refusal, ActorIsolationError, World.seed, EvidencePack.latest(), Trajectory.method.

F-0008 — Temperature may be redundant. Crystallization was built end to end without it ever being consulted; what triggered freezing was measured stability of realization, which is observed rather than declared. A declared temperature is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it are recorded — but gated, along with energy.py: if the next workplan closes without a decision consulting either, they go.

Two classifications had never run. INTENT_CHANGED and REALIZATION_FAILED were declared at T08 and produced by no test. Both now have purpose-built cases, and test_no_classification_is_unreachable fails if a seventh is added without one.

What this leaves. The thesis survived contact with its own evidence in narrowed form, and the demonstration is architectural rather than statistical. The highest-value next step is a bounded live-model experiment — F-0005 and F-0007 converge on it independently, and one experiment settles both whether a model adds capability and whether crystallization has a product rationale. That should come before any further framework feature.