T09: crystallization
A stable agentic realization becomes deterministic code. All four exit criteria met; 163 tests pass. - crystallization.py: trajectory capture, stability assessment requiring the same path across several runs, CrystallizedDriver, pytest codegen - crystallized/test_grant_access.py: generated, runs with no model, carries its lineage in the docstring - descendant preserves the ancestor's oracle set, agrees with it across five lab versions, and still catches a seeded defect - reversibility shown both ways via new M24 (grant endpoint renamed): the frozen descendant fails loudly rather than searching, and the agentic ancestor recovers from the same mutation F-0007 (open): the 54% cost reduction must not be quoted in support of the thesis. The T07 runtime is token-free, so the measured saving is one page fetch, one parse and a two-candidate scoring pass. The saving the concept actually claims - tokens, latency, retry variance - is unmeasured. Together with F-0005 this makes a bounded live-model experiment the highest-value next investment. Assertions in the generated test are imported rather than restated, so it is not fully standalone. Deliberate: paraphrased claims would be a second unverified statement of intent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
b9ad259468
commit
eee7722714
28 changed files with 794 additions and 10 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Concept ↔ Implementation Fitness Map
|
||||
|
||||
**Updated:** 2026-08-22 (TD-WP-0002-T08)
|
||||
**Updated:** 2026-08-23 (TD-WP-0002-T09)
|
||||
|
||||
Traces each important concept to the implementation, experiment and evidence that
|
||||
support it. **Unsupported entries are the point of this map** — a concept with no
|
||||
|
|
@ -36,9 +36,9 @@ were aspirational, not evidenced.
|
|||
| `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
|
||||
| `C-adaptation` | **C2** | `classification.py`, `agentic.py` | E-001 | T08 matrix | 11/12 mechanical absorbed without a human. |
|
||||
| `C-classification` | **C2** | `classification.py` | E-001, E-003 | T08 matrix, FAR 0/7 | **F-0006** — cannot infer SEMANTIC vs DEFECT; collapses to one escalating outcome. |
|
||||
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
|
||||
| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. |
|
||||
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. |
|
||||
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
|
||||
| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. |
|
||||
| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) |
|
||||
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
||||
| `C-confidence` | C0 | — | — | — | Deferred. |
|
||||
|
|
|
|||
|
|
@ -30,4 +30,5 @@ was required.
|
|||
|
||||
## Status
|
||||
|
||||
`PLANNED`. Blocked on T07.
|
||||
`EXECUTED` 2026-08-23 (T09). Fidelity criteria met across five lab versions;
|
||||
the cost criterion is met but uninformative — F-0007.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,87 @@
|
|||
---
|
||||
id: F-0007
|
||||
type: framework-finding
|
||||
class: FRAMEWORK_LIMITATION
|
||||
status: open
|
||||
discovered: "2026-08-23"
|
||||
discovered_by: TD-WP-0002-T09
|
||||
workplan: TD-WP-0002
|
||||
task: TD-WP-0002-T09
|
||||
hypotheses: [H-003]
|
||||
carried_to: TD-WP-0002-T10
|
||||
---
|
||||
|
||||
# F-0007 — Crystallization's economic case cannot be measured yet
|
||||
|
||||
## The criterion
|
||||
|
||||
H-003 includes a cost clause, deliberately:
|
||||
|
||||
> - it costs no less to execute than the agentic ancestor.
|
||||
|
||||
with the note: *if crystallization preserves semantics but saves nothing, the
|
||||
thesis is intact but the product rationale is not.*
|
||||
|
||||
## What was measured
|
||||
|
||||
| | median per run |
|
||||
|---|---|
|
||||
| agentic ancestor | 6.87 ms |
|
||||
| crystallized descendant | 3.17 ms |
|
||||
| **reduction** | **53.9 %** |
|
||||
|
||||
The criterion is met — the descendant is measurably cheaper. But the number is
|
||||
close to meaningless as evidence for the thesis.
|
||||
|
||||
## Why it is close to meaningless
|
||||
|
||||
The T07 runtime is a **deterministic heuristic**, chosen with the operator to
|
||||
avoid API cost and nondeterminism. It consumes zero tokens. So the entire
|
||||
measured saving is one page fetch, one HTML parse and a two-candidate scoring
|
||||
pass — a few milliseconds of local work.
|
||||
|
||||
The saving crystallization actually claims is of a different kind and two or
|
||||
three orders of magnitude larger: **model tokens, model latency, and the
|
||||
variance that forces retries.** None of those exist in this measurement, because
|
||||
none of those exist in this runtime.
|
||||
|
||||
So the honest statement is:
|
||||
|
||||
> Crystallization is measurably cheaper than the ancestor it was frozen from.
|
||||
> The measured 54 % is a **floor** produced by removing local discovery work, and
|
||||
> it says nothing about the saving that motivates the concept.
|
||||
|
||||
Quoting "54 % cheaper" as support for the crystallization thesis would be
|
||||
misleading, and this finding exists so that nobody does.
|
||||
|
||||
## What would make it measurable
|
||||
|
||||
A live-model runtime behind the same `ActorRuntime` interface. The
|
||||
`RealizationMetrics` fields (`tokens_in`, `tokens_out`, `model`, `retries`) were
|
||||
populated from the first run precisely so this comparison becomes a subtraction
|
||||
rather than a re-run of everything — see T07.
|
||||
|
||||
**F-0005 already gives an independent reason to want one:** M22 defeats the
|
||||
heuristic runtime while remaining solvable by reading a visible label. So a live
|
||||
model would settle two open questions at once — whether it adds *capability*
|
||||
(F-0005) and whether crystallization has an economic case (this finding).
|
||||
|
||||
That makes a bounded live-model experiment the highest-value next investment,
|
||||
above any further framework feature.
|
||||
|
||||
## A second, smaller limitation
|
||||
|
||||
The generated test is **not fully standalone**. Its realization is plain
|
||||
`urllib` with no framework dependency, but its assertions are *imported* from the
|
||||
originating scenario module rather than restated.
|
||||
|
||||
That was the right call — a generated test that paraphrases its claims creates a
|
||||
second, unverified statement of intent, and drift between them would be silent.
|
||||
But it qualifies the adoption story in the workplan ("output that drops into a CI
|
||||
system which already exists"): what drops in is the realization, while the claims
|
||||
still require the use-case module on the path.
|
||||
|
||||
Fully standalone generation would need claims expressible in a serializable form
|
||||
rather than as Python predicates. That is a real design question — it is the same
|
||||
question as "should scenarios be YAML", deferred at T04 — and both should be
|
||||
answered together at T10, with evidence about which predicates actually recur.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: H-003
|
||||
title: Crystallization
|
||||
status: PROPOSED
|
||||
status: EXPERIMENTING
|
||||
created: "2026-08-22"
|
||||
experiments: [E-002]
|
||||
concepts: [C-crystallization]
|
||||
|
|
@ -38,6 +38,28 @@ semantics but saves nothing, the thesis is intact but the product rationale is
|
|||
not — and that distinction should be visible in the record rather than discovered
|
||||
by a user.
|
||||
|
||||
## Result (TD-WP-0002-T09)
|
||||
|
||||
| Criterion | Outcome |
|
||||
|---|---|
|
||||
| produced without human authorship of assertions | **met** — assertions are imported, not restated |
|
||||
| drops no oracle from the ancestor | **met** — descendant's set is a superset |
|
||||
| agrees with the ancestor across lab versions | **met** — baseline, M15, M17, M16, M13 |
|
||||
| costs less to execute | **met, but see F-0007** |
|
||||
|
||||
Also demonstrated: the descendant still catches a seeded defect (M17), it fails
|
||||
*loudly* rather than searching when its frozen path breaks (M24), and the agentic
|
||||
ancestor recovers from that same M24 where the descendant cannot — the
|
||||
reversibility case from `INTENT.md`, shown in both directions.
|
||||
|
||||
**Not falsified. Not yet supported on the criterion that matters most.** The
|
||||
measured 54 % cost reduction comes entirely from removing local discovery work,
|
||||
because the T07 runtime consumes no tokens. The saving crystallization actually
|
||||
claims — model tokens, latency, retry variance — is unmeasured and unmeasurable
|
||||
with this runtime. See `research/findings/F-0007-...`.
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`. No evidence.
|
||||
- 2026-08-23 `EXPERIMENTING`. Fidelity criteria met; the economic criterion is
|
||||
met only in a sense too weak to support the thesis.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue