test-driver/research/findings/F-0007-crystallization-cost-unmeasurable.md

---
id: F-0007
type: framework-finding
class: FRAMEWORK_LIMITATION
status: open
discovered: "2026-08-23"
discovered_by: TD-WP-0002-T09
workplan: TD-WP-0002
task: TD-WP-0002-T09
hypotheses: [H-003]
carried_to: TD-WP-0002-T10
---

# F-0007 — Crystallization's economic case cannot be measured yet

## The criterion

H-003 includes a cost clause, deliberately:

> - it costs no less to execute than the agentic ancestor.

with the note: *if crystallization preserves semantics but saves nothing, the
thesis is intact but the product rationale is not.*

## What was measured

| | median per run |
|---|---|
| agentic ancestor | 6.87 ms |
| crystallized descendant | 3.17 ms |
| **reduction** | **53.9 %** |

The criterion is met — the descendant is measurably cheaper. But the number is
close to meaningless as evidence for the thesis.

## Why it is close to meaningless

The T07 runtime is a **deterministic heuristic**, chosen with the operator to
avoid API cost and nondeterminism. It consumes zero tokens. So the entire
measured saving is one page fetch, one HTML parse and a two-candidate scoring
pass — a few milliseconds of local work.

The saving crystallization actually claims is of a different kind and two or
three orders of magnitude larger: **model tokens, model latency, and the
variance that forces retries.** None of those exist in this measurement, because
none of those exist in this runtime.

So the honest statement is:

> Crystallization is measurably cheaper than the ancestor it was frozen from.
> The measured 54 % is a **floor** produced by removing local discovery work, and
> it says nothing about the saving that motivates the concept.

Quoting "54 % cheaper" as support for the crystallization thesis would be
misleading, and this finding exists so that nobody does.

## What would make it measurable

A live-model runtime behind the same `ActorRuntime` interface. The
`RealizationMetrics` fields (`tokens_in`, `tokens_out`, `model`, `retries`) were
populated from the first run precisely so this comparison becomes a subtraction
rather than a re-run of everything — see T07.

**F-0005 already gives an independent reason to want one:** M22 defeats the
heuristic runtime while remaining solvable by reading a visible label. So a live
model would settle two open questions at once — whether it adds *capability*
(F-0005) and whether crystallization has an economic case (this finding).

That makes a bounded live-model experiment the highest-value next investment,
above any further framework feature.

## A second, smaller limitation

The generated test is **not fully standalone**. Its realization is plain
`urllib` with no framework dependency, but its assertions are *imported* from the
originating scenario module rather than restated.

That was the right call — a generated test that paraphrases its claims creates a
second, unverified statement of intent, and drift between them would be silent.
But it qualifies the adoption story in the workplan ("output that drops into a CI
system which already exists"): what drops in is the realization, while the claims
still require the use-case module on the path.

Fully standalone generation would need claims expressible in a serializable form
rather than as Python predicates. That is a real design question — it is the same
question as "should scenarios be YAML", deferred at T04 — and both should be
answered together at T10, with evidence about which predicates actually recur.

## Interface settlement — 2026-09-28

TD-WP-0003-T05 deliberately retains Python predicates and imported original
claims. Standalone serialization is not a promised deliverable. See
`docs/TestDriverGeneralisationReview.md` for the published contract. The economic
finding stays open under TD-WP-0003-T01; no live-model measurement was performed.