test-driver/workplans/TD-WP-0003-generalise-and-settle.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated
TD-WP-0003	workplan	Generalise the model and settle the open questions	infotech	test-driver	proposed	codex	custodian	2026-08-23	2026-08-23

Generalise the model and settle the open questions

Why this workplan exists

TD-WP-0002 passed its gate: False Adaptation Rate 0/7, one asset crystallized and running with no model, the first Concept–Implementation Fitness Loop closed. What it does not yet have is generality or settled economics.

Everything demonstrated so far rests on one use case, one application, and one token-free runtime. Three of the project's own findings say plainly that this is too narrow a base to build real work on:

• F-0005 — H-001 is supported 2/3 against 0/3 on the deciding axis. Directionally clear, statistically nothing.
• F-0007 — crystallization's economic case cannot be measured with a runtime that consumes no tokens.
• F-0008 — Temperature was never consulted while building the thing it was supposed to govern.

This workplan widens the base and closes those questions before the framework meets a real system. Its purpose is structure and established experience, not new capability.

Success gate

1. The conceptual model expresses three substantively different multi-user use cases without new framework concepts being invented for each.
2. A live-model runtime has produced a measured answer on both open questions: does a model add capability beyond the heuristic (F-0005), and does crystallization save anything that matters (F-0007).
3. Every concept gated at T10 is either used in a decision or removed.
4. False Adaptation Rate remains 0 across a materially larger catalogue.
5. Explicit, written criteria exist for taking the framework to a real system — and an honest verdict on whether they are met.

Explicitly not in scope

• Applying test-driver to audit-core or any production system. That is the next workplan, gated on criterion 5 above. Deliberate: real work should land on a base that has been generalised, not on one that happens to fit its first example.
• Energy scoring, campaign selection, retirement — still deferred, and now additionally gated for removal (T04 below).
• Any new lens (resilience, scale) beyond what the three use cases require.

Coordination note

Another session is working in this repo on audit-core use cases (usecases/, tests/test_audit_core_e2_use_case.py, AUDIT-WP-0008). That work consumes the kernel as a library. Two consequences for this workplan:

• Do not use broad git add -A. Stage explicitly. A T10 commit here swept that session's in-progress files and a set of __pycache__ artefacts.
• Interface changes to testdriver are no longer private. T05 in particular touches how claims are expressed, so it needs a published interface change rather than a quiet edit.

---

Bounded live-model runtime experiment

id: TD-WP-0003-T01
status: todo
priority: high

Implement a live-model ActorRuntime behind the existing interface and run a bounded experiment — a fixed run count and a stated cost ceiling agreed with the operator before the first call.

Two questions, one experiment. They are the two highest-value unknowns in the project and neither can be answered by reasoning:

• Capability (F-0005). M22 defeats the heuristic runtime — form fields renamed, test ids dropped — while remaining solvable by reading the visible "Person" label. Does a model in fact solve it? If yes, the agentic arm's advantage is wider than the T07 measurement showed. If no, the semantic-action thesis is narrower still.
• Economics (F-0007). RealizationMetrics already carries tokens_in, tokens_out, model and retries, populated from the first run precisely so this is a subtraction rather than a re-run. Measure cost per agentic realization against cost per crystallized execution, and compare both against the honest alternative: asking an agent to rewrite the broken test.

Record run-to-run variance, not only means. A stochastic runtime that succeeds on average and fails unpredictably is a different product from one that succeeds reliably, and the difference will not show in an average.

Needs an operator decision before starting: cost ceiling, model, and whether the live runtime ever runs in the default test suite (recommendation: no — keep the suite deterministic and free, run the live arm on demand).

Two further use cases

id: TD-WP-0003-T02
status: todo
priority: high

Alice/Bob/Carol is one shape: a resource, an owner, a grant, a revoke. Every concept in the model was designed against it, so of course it fits.

Add two substantively different multi-user use cases — candidates: an approval workflow with sequencing and delegation, and a tenant-scoped lifecycle with cross-boundary invariants. Choose for difference, not coverage.

The question is not whether they pass. It is what the framework had to grow to express them, and every such growth is evidence:

• a concept invented per use case is a modelling failure;
• a concept that finally gets used (Schedule, Lens, Variant are all declared and unbuilt) is a concept validated;
• a concept still untouched after three use cases is a candidate for removal.

Record the answer in the fitness map either way.

Strengthen the instrument on the deciding axis

id: TD-WP-0003-T03
status: todo
priority: high

H-001's whole result turns on three mutations that drop stable identifiers, of which discovery handles two. Three is not a sample.

Build the dropped-identifier side out to at least ten mutations spanning genuinely different mechanisms — rewritten markup, renamed fields, restructured forms, relocated controls, changed element semantics. Keep the preserved-identifier side proportionate so the comparison stays fair.

Then re-run E-001 and restate H-001 with a number that means something. Be prepared for the honest outcome that the narrowed claim narrows further.

Settle every gated concept

id: TD-WP-0003-T04
status: todo
priority: medium

T10 gated two things on this workplan. Settle both, and do not extend the gate:

• Temperature (F-0008). Crystallization was built without it; measured stability of realization did its job, and is observed rather than declared. Either a decision in T01–T03 consults it, or it leaves the concept model.
• energy.py. Events recorded since T04 have informed nothing. Either the history is used, or the module goes and H-005 is marked dormant-indefinite.

Also review Confidence, Campaign, Metabolism and Retirement — all declared, unimplemented, never consulted. After three use cases there is enough evidence to say whether they are deferred or dead.

Removing a concept is a result. Extending a gate is not.

Decide how claims are expressed

id: TD-WP-0003-T05
status: todo
priority: medium

Deferred twice, and now blocking two things at once (F-0007, and the T04 note in TD-WP-0002): claims are Python predicates, so a generated regression test cannot be fully standalone and scenarios cannot be declarative.

Answerable only after T02, which is the point of the ordering — the question "which predicates actually recur" needs more than one use case to answer.

Three candidate answers, and the middle one should not win by default:

1. keep predicates as code and accept that generated tests import their claims;
2. a small declarative claim vocabulary covering what recurs, with an escape hatch to code;
3. full serialization, at the cost of a language nobody asked for.

Whatever is chosen, publish it as an interface change — the audit-core session consumes Claim directly.

A browser-engine driver, if it is warranted

id: TD-WP-0003-T06
status: wait
priority: low

Blocked on the operator establishing Playwright and clearing its licence.

F-0004 records what the stdlib driver cannot see: JavaScript, screenshots, and — most consequentially — visual relayout, which is a large fraction of real mechanical change. H-001 currently measures structural durability only, and that caveat is attached to every statement of the result.

Scope is one experiment, not a rewrite: implement Driver with a browser engine, run the same semantic actions against the same mutations, and answer whether the H-001 result changes. If it does not, F-0004 closes and the stdlib driver stands as the default.

Do not start this before T01–T03. It widens the surface; those three settle what the surface is for.

Measure the cost of expressing a use case

id: TD-WP-0003-T07
status: todo
priority: medium

TestDriverResearchPrototype.md puts time to express a new use case first in its scorecard, and it has never been measured. T02 creates the only honest opportunity — the first time someone writes a use case that was not designed alongside the framework.

Record, for each of the two new use cases: elapsed time, lines written, concepts consulted, points where the model resisted, and anything that had to be worked around rather than expressed.

This feeds the metric the project ultimately cares about — verified behaviour per unit of human maintenance effort — and it is the number an adopter will ask for first. It cannot be reconstructed later.

Readiness review for real-system application

id: TD-WP-0003-T08
status: todo
priority: high

Write explicit criteria for taking test-driver to a real system, then answer them honestly against the evidence. Not a formality: the audit-core work is already in this repo and will proceed on whatever this concludes.

At minimum the criteria must cover:

• the independent observation channel D-07 requires, and what it costs to provide on a system that was not built with one — the single largest adoption barrier and still untested outside the lab;
• whether claim provenance survives contact with a real backlog, where "the spec" is often a ticket written after the code;
• what happens when a real system's behaviour is legitimately ambiguous rather than seeded-ambiguous;
• what a false adaptation would cost in that specific engagement, which is the only setting where FAR = 0 stops being an abstraction.

Then state a verdict, including the verdict "not yet, and here is what is missing". A readiness review that cannot conclude not ready is not a review.