test-driver/workplans/TD-WP-0002-vertical-spike-crystallization.md

---
id: TD-WP-0002
type: workplan
title: "Vertical spike: falsify the crystallization thesis"
domain: infotech
repo: test-driver
status: finished
owner: codex
topic_slug: custodian
created: "2026-08-22"
updated: "2026-08-23"
state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1"
---

# Vertical spike: falsify the crystallization thesis

## Why this workplan exists

The concept corpus (`INTENT.md`, `docs/`) describes eleven milestones that build
four complete layers — research control plane, deterministic kernel, lab,
self-verification — before the central thesis is exercised even once. The repo
currently holds ~3,000 lines of theory and zero lines of executable code.

This workplan inverts that order. It drives **one thin thread end to end**
through every layer of the model, so that the thesis can be supported or
falsified cheaply and early:

```
one use case → deterministic kernel → labelled mutation lab
  → agentic realisation of one semantic action
  → adaptation classification → one crystallization
```

Reasoning and the full assessment behind this sequencing:
`history/2026-08-22-concept-assessment-swot.md`.

## Success gate

The spike succeeds when, in one coherent run:

1. agentic realisation recovers from lab mutations **M01** (moved/renamed
   control) and **M02** (restructured DOM) with **zero** changes to claims or
   invariants;
2. lab mutation **M05** (authorization defect) is reported as a **Product
   Finding**, not adapted to;
3. one verification asset crystallizes into a deterministic test that runs with
   **no model involvement** and preserves the relevant oracles;
4. **False Adaptation Rate = 0** across the labelled mutation set.

Failing this gate early and cheaply is a valid and valuable outcome. Passing it
is worth more than any other ten items on the M0–M10 roadmap.

## Explicitly deferred

Not in this workplan, by decision rather than omission — each is cheap to build,
satisfying to build, and impossible to validate at this stage:

- Energy scoring, decay, and campaign selection (raw immutable `EnergyEvent`s are
  recorded from the first run; **no** scoring logic is implemented);
- Temperature and Confidence as computed values;
- Test Metabolism, Campaigns, Retirement floors;
- Lineage graph storage beyond a parent pointer;
- the five agentic roles (Builder / Experimenter / Critic / Auditor / Maintainer);
- any surface beyond one HTTP API and one browser UI.

## Constraints

- Stack: Python, pytest, Playwright, Pydantic/dataclasses, YAML, SQLite.
  One process, one database, one browser engine, one application under test.
- Novelty belongs in the verification model, never in the infrastructure.
- No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
- Evidence stratification S1/S2/S3 and claim provenance per
  `docs/TestDriverClassificationDesign.md` (decision
  `fef5213f-ce9b-44c2-b327-a0b0ba4b6270`). Adaptation writes only to S1.

---

## Reconcile milestone numbering and commit the concept corpus

```task
id: TD-WP-0002-T01
status: done
priority: high
state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a"
```

Three documents carry three different milestone sequences: `INTENT.md` (M0–M3),
`docs/TestDriverInitialMilestones.md` (M0–M10), and
`docs/TestDriverConceptModel.md` §15 (a third M0–M3). This is a CONCEPT_DRIFT
finding by the project's own taxonomy, present before any code exists.

Pick one canonical sequence, mark the others superseded in place, and commit
`INTENT.md` and `docs/` to git (currently untracked). Record the drift as the
first entry in the framework findings log — the project should catch this on
itself before it catches anything else.

**Done 2026-08-22.** `docs/TestDriverInitialMilestones.md` (M0–M10) is canonical;
the `INTENT.md` and Concept Model § 15 sequences carry superseded banners with a
mapping table. Recorded as `research/findings/F-0001-milestone-sequence-drift.md`.
`INTENT.md` and `docs/` are now tracked. One residual carried to T10: the finding
taxonomy has no class for document-to-document drift.

## Decide intent provenance and the classification mechanism on paper

```task
id: TD-WP-0002-T02
status: done
priority: high
state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63"
```

Two design questions gate everything downstream. Answer both in a written design
note plus a State Hub decision record, **before** writing kernel code.

**(a) The classifier.** What *deterministic evidence* separates "the button
moved" from "Bob can still read after revoke"? Specify the signal, not the
intent. If the honest answer is "a model decides", that violates Concept Model
§2.3 and must be resolved now rather than discovered at M6.

**(b) Intent provenance.** If an agent writes the implementation, an agent writes
the use case, and an agent realises the test, then "oracles independent from
actors" is procedurally true but epistemically thin. Record whether use cases are
human-authored, spec-derived, or agent-generated, and what independence guarantee
survives in each case.

Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision.

**Done 2026-08-22.** Design note written; decision
`fef5213f-ce9b-44c2-b327-a0b0ba4b6270` recorded in the hub.

Outcome in one line: **adaptation may write only to surface evidence (S1); claims
and invariants are run inputs with no adaptation write path.** False Adaptation
Rate = 0 therefore becomes an architectural property rather than a tuning target —
the system cannot express "accept a defect as an adaptation". Classification is a
total function over three signals defaulting to `AMBIGUOUS`; a surface change
concurrent with a verdict change classifies as `PRODUCT_DEFECT`, never as
adaptation. Claims require causally independent provenance.

Two consequences that change later tasks:
- **D-07** — an independent observation channel into the system under test is now
  an explicit adoption precondition, not an implementation detail. T05 must expose
  one in the lab; without it S2/S3 collapse into S1.
- A fourth self-test, `td://self/intent-independence`, joins the T06 set.

## Minimal research control plane

```task
id: TD-WP-0002-T03
status: done
priority: medium
state_hub_task_id: "31b88522-decd-5f4c-b0e2-05b6981f33b8"
```

Timebox: half a day. Five hypothesis files (H-001…H-005) each with an explicit
falsification condition, one Concept ↔ Implementation Fitness Map, and a stable
identifier convention. Plain files under `research/`.

No CLI, no register tooling, no schema. The control plane is overhead until there
are readings to record.

**Done 2026-08-22.** `research/` holds H-001…H-005 (each with an explicit
falsification condition), E-001…E-003, the fitness map, the findings log and the
ID convention.

Three things worth carrying forward:
- The fitness map corrects the Improvement Loop § 13 starting classifications
  **downward**. Nothing can honestly exceed `C1` with no implementation; `C2` was
  aspirational.
- **H-005 (Energy) is dormant by decision** — untestable at this scale, so events
  are captured and no scoring is written.
- **H-001 requires a real control arm** (a robust recorded selector sequence). A
  straw-man control would make the semantic-action thesis trivially true and
  worthless.

## Deterministic semantic kernel

```task
id: TD-WP-0002-T04
status: done
priority: high
state_hub_task_id: "ffbcc9e8-c1bd-5c50-b62d-495ad9e135a4"
```

Minimal executable representations of UseCase, Actor, World, Scenario,
SemanticAction, Observation, Evidence, Oracle, Verdict, VerificationAsset, Run.
One deterministic driver (HTTP or direct adapter).

Run the reference use case end to end:

```
Alice owns resource R. Alice grants Bob READ. Bob can read R.
Carol cannot read R. Alice revokes Bob. Bob can no longer read R.
```

Exit: Alice, Bob and Carol hold genuinely isolated identities and sessions; all
outcomes are judged by independent deterministic oracles; a run emits a
structured Evidence Pack; the scenario replays from known initial state.

Emit raw `EnergyEvent` records from this point onward. Implement no scoring.

**Done 2026-08-22.** `src/testdriver/` (11 modules), `lab/minimal.py`,
`scenarios/alice_bob_carol.py`, 16 passing tests. The reference scenario runs
end to end and replays identically from the same seed; evidence comes out
stratified 3/3/3 across S1/S2/S3.

Three things that came out of building it rather than designing it:

- **The observation channel needs two probes, not one.** Reading stored state
  alone verifies test-driver's reimplementation of the rules rather than the
  system's enforcement of them; probing enforcement alone cannot notice that
  record and enforcement disagree. The lab exposes both, and their disagreement
  is now an invariant (`i-enforcement-matches-record`). That invariant is what
  catches an authorization defect which leaves the audit trail looking correct.
- **A seeded `RevokeIsCosmetic` lab already fails the run** — both the claim and
  the independent invariant fire, and the claim set is provably untouched. Early
  evidence for H-004, though not yet the experiment.
- **Scenarios are Python, not YAML.** Claims are predicates over observations; a
  YAML dialect able to express them would be a programming language with worse
  tooling. Revisit once we know which predicates actually recur.

## Test-driver lab with labelled ground truth

```task
id: TD-WP-0002-T05
status: done
priority: high
state_hub_task_id: "595a89c2-1462-57fc-8b08-a5a6b875fd48"
```

A deliberately small application: users, tenants, auth, resources, sharing,
read/write permissions, revoke, audit history, HTTP API, minimal browser UI.

Build **15–20 labelled mutations**, not the six sketched in the milestones doc.
Six cannot support any statement about precision or recall. Each mutation is a
reproducible toggle carrying a recorded expected classification
(`MECHANICAL` / `SEMANTIC` / `DEFECT`), with baseline and mutated versions
carrying explicit version identifiers.

The lab is the measuring instrument for every claim the framework makes — a weak
lab caps the credibility of all downstream results. It is also potentially the
project's first publishable artefact in its own right.

**Done 2026-08-22.** `lab/app.py` (users, tenants, auth, resources, sharing,
read/write, revoke, audit), `lab/http_api.py` (JSON API + browser UI, stdlib
only), `lab/mutations.py` (20 labelled, composable, version-stamped mutations),
`lab/GROUND-TRUTH.md`. 48 tests pass. Detection: MECHANICAL 0/10 flagged,
DEFECT 6/6, SEMANTIC 2/4.

Three results worth carrying:

- **F-0002 — two seeded defects were initially invisible.** M16 (READ grant
  confers WRITE) and M18 (revocation unaudited) both passed cleanly: nothing
  flaky, nothing `INCONCLUSIVE`, the framework simply had nothing to say. A use
  case protects exactly what it asserts and not one thing more. Resolved by
  adding two claims already stated as intent in `INTENT.md`. Had the six-mutation
  catalogue from the milestones doc been used, this would never have surfaced —
  the argument for the larger catalogue is now evidenced rather than asserted.
- **The test-id axis.** Stable `data-td` selectors survive most UI mutations,
  which would make H-001 trivially *false*. Rather than rig the catalogue,
  mutations now vary on `preserves_test_ids`, and H-001 must be analysed split by
  that axis. A semantic action earns its keep exactly where stable identifiers
  are absent or not carried forward — that is the honest shape of the claim.
- **M12 vs M19 are behaviourally identical.** A deliberate deferred-revocation
  decision and a revocation race produce the same failure, same step, same
  evidence. Only intent separates them. This is the discrimination problem in its
  sharpest form and is now a test, not a paragraph.

## Out-of-band ground truth for self-verification

```task
id: TD-WP-0002-T06
status: done
priority: medium
state_hub_task_id: "4fb31485-8650-556c-bee4-b4e067a89d04"
```

Write the actor-isolation and oracle-independence checks as **plain pytest**
against observable behaviour, outside the framework. Using test-driver to verify
that test-driver's oracles are independent is a system certifying itself.

Exit: deliberately breaking actor isolation fails the out-of-band check;
allowing an actor to determine its own verdict fails oracle-independence;
each failure produces a Framework Finding.

**Done 2026-08-22.** `tests/selfverification/` — checks as plain functions over a
serialized Evidence Pack, plus 12 tests that they hold and 12 that they *can
fail*. All four `td://self/...` identifiers are covered. 72 tests pass overall.

- The strongest check turned out to be **verdict reproducibility from S3 alone**:
  re-evaluating each assertion against the stored snapshots, with no actor,
  driver or live system in reach, must reproduce exactly what the run reported.
  It is asserted on failing runs too — a check that only works when everything is
  green verifies luck, not independence.
- **F-0003 (open)** — actor isolation leaves no trace in ordinary evidence. The
  self-test catches a shared memory store only because the harness plants
  per-actor canaries first; without them, a run where all three actors share one
  memory dict produces an Evidence Pack identical to a correct one. Isolation is
  currently a property of a scenario written to expose it, not of runs in
  general. Carried to T10, reassess after T07.
- The same finding notes the mirror-image problem: oracle independence is
  enforced by construction, so no real run can produce the violating artefact and
  the check must be exercised against a hand-built pack. **A guarantee enforced
  by construction cannot be verified by observing real runs.** Four green
  self-tests are not four equivalent proofs, and the suite now says so.

## Agentic realisation of one semantic action, fully instrumented

```task
id: TD-WP-0002-T07
status: done
priority: high
state_hub_task_id: "b58cf7ce-dc18-5218-8eac-179291626467"
```

Browser driver plus one agentic actor runtime, realising exactly one semantic
action — `grant_access(Bob, R, READ)` — from intent, against the lab UI. Strict
per-actor context isolation; bounded navigation and tool permissions; full
action and evidence recording; agent/model/configuration identity captured in
evidence. Oracles stay deterministic.

**Instrument cost and nondeterminism from the very first run:** tokens, wall
time, retry count, and run-to-run variance. If agentic realisation costs more per
run than simply asking an agent to rewrite the broken test, the crystallization
argument is an aesthetic preference rather than a value proposition. This data is
free to collect from run one and impossible to backfill.

**Done 2026-08-22.** `html.py` (stdlib DOM), `agentic.py` (two runtimes),
`browser.py` (per-actor sessions over real HTTP), `scenarios/browser_grant.py`.
107 tests pass. Two decisions taken with the operator: **stdlib HTML driver
instead of Playwright** (F-0004) and **a deterministic discovery runtime instead
of a live model**, both behind interfaces that let the alternatives drop in later.

The headline result is a narrowing, not a confirmation:

| | Discovery (agentic) | Recorded selectors (control) |
|---|---|---|
| test ids preserved (9 mutations) | 9/9 | **9/9** |
| test ids dropped (3 mutations) | 2/3 | 0/3 |

- **F-0005 — semantic actions earn their keep more narrowly than claimed.** Where
  an application keeps stable identifiers, the conventional approach matches the
  agentic one exactly and is cheaper, faster and deterministic. The semantic
  action wins only where identifiers are absent or not carried forward. Filed as
  `CONCEPT_DRIFT`: the concept model overstates this and should be revised to
  match the evidence. Three mutations on the deciding side is directionally clear
  and statistically nothing — recorded rather than rounded up.
- **M22 marks where a scripted runtime stops.** Renaming form fields defeats the
  heuristic, but the page still carries a "Person" label a model could read. This
  is the first concrete evidence that a live model would add *capability* rather
  than only cost — worth more than the general argument that it might.
- Two mutations (M21, M22) were added mid-task because the deciding side of the
  test-id axis was N=1 after the first run. Extending the instrument when the
  evidence shows it is too thin is the intended behaviour.
- Recovery happened with **no claim or invariant diff in any run**, and the one
  failure failed *loudly* — `RealizationFailed` in evidence, not a silent pass.

## Adaptation detection and the defect-vs-adaptation classifier

```task
id: TD-WP-0002-T08
status: done
priority: high
state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0"
```

The centrepiece. Implement the mechanism designed in T02 and run it across the
full labelled mutation set from T05.

Classification path: `IMPLEMENTATION CHANGE` / `INTENT CHANGE` /
`PRODUCT DEFECT` / `AMBIGUOUS`, with escalation for semantic changes and
ambiguity. Preserve semantic action identity across adaptation; record adaptation
history on the verification asset; never modify claims or invariants to
accommodate an observed behaviour.

Measure: Mechanical Recovery Rate, classification precision/recall, and
**False Adaptation Rate**. The last is the project's existential safety metric —
one publicly normalised authorization defect kills the thesis permanently. Target
is zero across the set, and a non-zero result is a stop-and-redesign signal, not
a tuning exercise.

**Done 2026-08-22.** `classification.py`, `CompositeDriver`,
`scenarios/full_journey.py` (one asset crossing both surfaces). 141 tests pass.

**False Adaptation Rate = 0/7**, including the three E-003 attacks. The other
side of the trade is asserted too — 11 of 12 mechanical mutations are absorbed
without a human, so the safety result is not bought by escalating everything.

| Ground truth | Accepted without a human |
|---|---|
| MECHANICAL (12) | 11 |
| SEMANTIC (4) | 2 — both genuinely inert here |
| DEFECT (7) | **0** |

- **F-0006 (resolved)** — the T02 decision table listed `SEMANTIC_CHANGE` as an
  outcome it could produce. It cannot: M12 and M19 are behaviourally identical,
  as the lab has asserted since T05. `PRODUCT_DEFECT` and `SEMANTIC_CHANGE`
  collapse into one escalating outcome, `BEHAVIOUR_CHANGED`, and the distinction
  becomes a recorded human adjudication. `INTENT_CHANGED` survives but is
  detected by the *claim fingerprint* moving — a fact about the recorded use
  case, not an inference about behaviour. Erring toward "false alarm" costs one
  review; erring toward "false adaptation" ends the project.
- **Two classifier defects found and fixed rather than reported.** Claims
  downstream of a failed realization now yield `INCONCLUSIVE` rather than `FAIL`
  — accusing the system of a defect on the strength of the test's own inability
  to act is the mirror image of a false adaptation. And the browser driver now
  records a page signature, so a surface change is detectable even when the
  chosen interaction path is identical.
- **What this does not establish.** FAR = 0 follows largely from architecture
  (D-02, plus `SAFE_TO_ACCEPT` being a closed two-element set). The experiment
  confirms the architecture behaves as designed over 23 hand-written mutations;
  it says nothing about mutations nobody thought of. That distinction should
  survive into any external claim.

## Crystallize one asset into deterministic regression

```task
id: TD-WP-0002-T09
status: done
priority: high
state_hub_task_id: "855a1f41-b839-57fc-87b8-198ce2a9f6b1"
```

Semantic action trajectory capture, stable-realisation detection, deterministic
candidate generation, candidate comparison against the existing oracle set, and
lineage from the agentic ancestor to the deterministic descendant. Manual
acceptance step for now.

Emit ordinary pytest/Playwright code. Output that drops into a CI system which
already exists lets a user adopt the result without adopting the framework.

Exit: the generated test runs with zero agentic involvement, preserves the
relevant claims and oracles, retains visible lineage, and measurably costs less
to execute than the agentic ancestor.

**Done 2026-08-23.** `crystallization.py`, `crystallized/test_grant_access.py`
(generated), 163 tests pass. All four exit criteria met.

- Freezing requires the **same** path across several runs, not one success. One
  run proves the agent can find a way; it does not show the surface has stopped
  moving, and freezing on a single observation is how a crystallized test becomes
  flaky the first time a page renders differently.
- **Reversibility demonstrated in both directions.** M24 (grant endpoint renamed)
  breaks the frozen path; the descendant fails *loudly* rather than falling back
  to searching — falling back would quietly turn a T5 asset into a T1 one and
  hide the signal crystallization exists to produce — while the agentic ancestor
  recovers from the same mutation. That is what distinguishes crystallization
  from ordinary codegen: the agentic form is not discarded.
- **F-0007 (open) — the economic case is unmeasurable.** The descendant is 54%
  cheaper, and that number should not be quoted in support of the thesis. The
  T07 runtime consumes zero tokens, so the entire saving is one page fetch, one
  parse and a two-candidate scoring pass. The saving crystallization actually
  claims — model tokens, latency, retry variance — is two or three orders of
  magnitude larger and entirely absent from this measurement.
- The generated test is **not fully standalone**: realization is plain `urllib`,
  but assertions are *imported* rather than restated, because a generated test
  that paraphrases its claims creates a second unverified statement of intent.
  Right call, and it qualifies the "drops into existing CI" story — what drops in
  is the realization; the claims still need the use-case module.

## Gate review and first compression pass

```task
id: TD-WP-0002-T10
status: done
priority: medium
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"
```

Evaluate the four success-gate criteria against collected evidence and write the
result up in `history/` regardless of outcome.

Then run the compression questions from Milestone 10 while the spike is still
small: which concepts have no supporting evidence, which abstractions have no
conceptual justification, which metadata never informed a decision, what did the
experiments falsify. Mark rejected concepts as rejected rather than silently
retaining them, and update the Concept ↔ Implementation Fitness Map.

Convert at least one framework finding discovered during T04–T09 into a permanent
self-verification or deterministic regression — that closes the first genuine
Concept–Implementation Fitness Loop.

**Done 2026-08-23.** Full write-up:
`history/2026-08-23-td-wp-0002-gate-review.md`. 178 tests pass.

**All four gate criteria met.** M01/M02 absorbed with the claim set unchanged;
the authorization defects escalated with regressions named; one asset
crystallized and running with no model; **False Adaptation Rate 0/7**, with 12 of
13 mechanical mutations absorbed so the safety result is not bought by escalating
everything.

**Fitness loop closed via F-0003.** Actor isolation left no trace in ordinary
evidence — it was a property of scenarios written to expose it. Every actor is
now seeded with an automatic private marker and the runner examines all actors on
every scenario, with two permanent regressions behind it, one of which catches a
leak with the harness planting nothing.

**Compression — six abstractions removed**, each declared and never used:
`Verdict.SUSPICIOUS` (a verdict no oracle could emit), `Step.expect_refusal`,
`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`,
`Trajectory.method`.

**F-0008 — Temperature may be redundant.** Crystallization was built end to end
without it ever being consulted; what triggered freezing was *measured stability
of realization*, which is observed rather than declared. A declared temperature
is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it
are recorded — but gated, along with `energy.py`: if the next workplan closes
without a decision consulting either, they go.

**Two classifications had never run.** `INTENT_CHANGED` and `REALIZATION_FAILED`
were declared at T08 and produced by no test. Both now have purpose-built cases,
and `test_no_classification_is_unreachable` fails if a seventh is added without
one.

**What this leaves.** The thesis survived contact with its own evidence in
narrowed form, and the demonstration is architectural rather than statistical.
The highest-value next step is a **bounded live-model experiment** — F-0005 and
F-0007 converge on it independently, and one experiment settles both whether a
model adds capability and whether crystallization has a product rationale. That
should come before any further framework feature.