Add TD-WP-0003: generalise the model and settle the open questions

Consolidation before application. Everything TD-WP-0002 demonstrated rests on one use case, one application and one token-free runtime; three of the project's own findings say that is too narrow a base for real work. - T01 bounded live-model experiment: settles F-0005 (capability) and F-0007 (economics) in one run. Needs an operator decision on cost ceiling first. - T02 two further use cases: the model was designed against Alice/Bob/Carol, so of course it fits. What it has to grow to express something else is the evidence. - T03 build the dropped-identifier mutation side to ten; H-001 currently turns on three. - T04 settle every gated concept - Temperature and energy.py are used in a decision or removed. Extending a gate is not a result. - T05 decide how claims are expressed, answerable only after T02. - T06 browser-engine driver if warranted (blocked on Playwright). - T07 measure time to express a use case - first in the scorecard, never measured, and reconstructable only during T02. - T08 readiness review with explicit criteria for real-system application. Applying test-driver to audit-core is explicitly out of scope and gated on T08. Coordination note added: another session works in this repo, so staging must be explicit and interface changes published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-23 00:53:07 +02:00 · 2026-08-23 00:53:07 +02:00 · faef069b17
commit faef069b17
parent cee87e763f
1 changed files with 267 additions and 0 deletions
--- a/workplans/TD-WP-0003-generalise-and-settle.md
+++ b/workplans/TD-WP-0003-generalise-and-settle.md
@ -0,0 +1,267 @@
+---
+id: TD-WP-0003
+type: workplan
+title: "Generalise the model and settle the open questions"
+domain: infotech
+repo: test-driver
+status: proposed
+owner: codex
+topic_slug: custodian
+created: "2026-08-23"
+updated: "2026-08-23"
+---
+
+# Generalise the model and settle the open questions
+
+## Why this workplan exists
+
+`TD-WP-0002` passed its gate: False Adaptation Rate 0/7, one asset crystallized
+and running with no model, the first Concept–Implementation Fitness Loop closed.
+What it does **not** yet have is generality or settled economics.
+
+Everything demonstrated so far rests on **one use case**, **one application**, and
+**one token-free runtime**. Three of the project's own findings say plainly that
+this is too narrow a base to build real work on:
+
+- **F-0005** — H-001 is supported 2/3 against 0/3 on the deciding axis. Directionally
+  clear, statistically nothing.
+- **F-0007** — crystallization's economic case cannot be measured with a runtime
+  that consumes no tokens.
+- **F-0008** — `Temperature` was never consulted while building the thing it was
+  supposed to govern.
+
+This workplan widens the base and closes those questions **before** the framework
+meets a real system. Its purpose is structure and established experience, not new
+capability.
+
+## Success gate
+
+1. The conceptual model expresses **three** substantively different multi-user use
+   cases without new framework concepts being invented for each.
+2. A live-model runtime has produced a measured answer on **both** open questions:
+   does a model add capability beyond the heuristic (F-0005), and does
+   crystallization save anything that matters (F-0007).
+3. Every concept gated at T10 is either **used in a decision or removed**.
+4. False Adaptation Rate remains **0** across a materially larger catalogue.
+5. Explicit, written criteria exist for taking the framework to a real system —
+   and an honest verdict on whether they are met.
+
+## Explicitly not in scope
+
+- **Applying test-driver to audit-core or any production system.** That is the
+  next workplan, gated on criterion 5 above. Deliberate: real work should land on
+  a base that has been generalised, not on one that happens to fit its first
+  example.
+- Energy scoring, campaign selection, retirement — still deferred, and now
+  additionally *gated for removal* (T04 below).
+- Any new lens (resilience, scale) beyond what the three use cases require.
+
+## Coordination note
+
+Another session is working in this repo on audit-core use cases
+(`usecases/`, `tests/test_audit_core_e2_use_case.py`, `AUDIT-WP-0008`). That work
+consumes the kernel as a library. Two consequences for this workplan:
+
+- **Do not use broad `git add -A`.** Stage explicitly. A T10 commit here swept
+  that session's in-progress files and a set of `__pycache__` artefacts.
+- Interface changes to `testdriver` are no longer private. T05 in particular
+  touches how claims are expressed, so it needs a published interface change
+  rather than a quiet edit.
+
+---
+
+## Bounded live-model runtime experiment
+
+```task
+id: TD-WP-0003-T01
+status: todo
+priority: high
+```
+
+Implement a live-model `ActorRuntime` behind the existing interface and run a
+**bounded** experiment — a fixed run count and a stated cost ceiling agreed with
+the operator before the first call.
+
+Two questions, one experiment. They are the two highest-value unknowns in the
+project and neither can be answered by reasoning:
+
+- **Capability (F-0005).** M22 defeats the heuristic runtime — form fields renamed,
+  test ids dropped — while remaining solvable by reading the visible `"Person"`
+  label. Does a model in fact solve it? If yes, the agentic arm's advantage is
+  wider than the T07 measurement showed. If no, the semantic-action thesis is
+  narrower still.
+- **Economics (F-0007).** `RealizationMetrics` already carries `tokens_in`,
+  `tokens_out`, `model` and `retries`, populated from the first run precisely so
+  this is a subtraction rather than a re-run. Measure cost per agentic
+  realization against cost per crystallized execution, and compare both against
+  the honest alternative: *asking an agent to rewrite the broken test*.
+
+Record run-to-run variance, not only means. A stochastic runtime that succeeds on
+average and fails unpredictably is a different product from one that succeeds
+reliably, and the difference will not show in an average.
+
+**Needs an operator decision before starting:** cost ceiling, model, and whether
+the live runtime ever runs in the default test suite (recommendation: no — keep
+the suite deterministic and free, run the live arm on demand).
+
+## Two further use cases
+
+```task
+id: TD-WP-0003-T02
+status: todo
+priority: high
+```
+
+Alice/Bob/Carol is one shape: a resource, an owner, a grant, a revoke. Every
+concept in the model was designed against it, so of course it fits.
+
+Add two substantively different multi-user use cases — candidates: an
+**approval workflow** with sequencing and delegation, and a **tenant-scoped
+lifecycle** with cross-boundary invariants. Choose for difference, not coverage.
+
+The question is not whether they pass. It is **what the framework had to grow to
+express them**, and every such growth is evidence:
+
+- a concept invented per use case is a modelling failure;
+- a concept that finally gets used (`Schedule`, `Lens`, `Variant` are all declared
+  and unbuilt) is a concept validated;
+- a concept still untouched after three use cases is a candidate for removal.
+
+Record the answer in the fitness map either way.
+
+## Strengthen the instrument on the deciding axis
+
+```task
+id: TD-WP-0003-T03
+status: todo
+priority: high
+```
+
+H-001's whole result turns on three mutations that drop stable identifiers, of
+which discovery handles two. Three is not a sample.
+
+Build the dropped-identifier side out to at least ten mutations spanning
+genuinely different mechanisms — rewritten markup, renamed fields, restructured
+forms, relocated controls, changed element semantics. Keep the preserved-identifier
+side proportionate so the comparison stays fair.
+
+Then re-run E-001 and restate H-001 with a number that means something. Be
+prepared for the honest outcome that the narrowed claim narrows further.
+
+## Settle every gated concept
+
+```task
+id: TD-WP-0003-T04
+status: todo
+priority: medium
+```
+
+T10 gated two things on this workplan. Settle both, and do not extend the gate:
+
+- **`Temperature` (F-0008).** Crystallization was built without it; measured
+  stability of realization did its job, and is observed rather than declared.
+  Either a decision in T01–T03 consults it, or it leaves the concept model.
+- **`energy.py`.** Events recorded since T04 have informed nothing. Either the
+  history is used, or the module goes and H-005 is marked dormant-indefinite.
+
+Also review `Confidence`, `Campaign`, `Metabolism` and `Retirement` — all
+declared, unimplemented, never consulted. After three use cases there is enough
+evidence to say whether they are deferred or dead.
+
+Removing a concept is a result. Extending a gate is not.
+
+## Decide how claims are expressed
+
+```task
+id: TD-WP-0003-T05
+status: todo
+priority: medium
+```
+
+Deferred twice, and now blocking two things at once (F-0007, and the T04 note in
+`TD-WP-0002`): claims are Python predicates, so a generated regression test cannot
+be fully standalone and scenarios cannot be declarative.
+
+Answerable only **after T02**, which is the point of the ordering — the question
+"which predicates actually recur" needs more than one use case to answer.
+
+Three candidate answers, and the middle one should not win by default:
+
+1. keep predicates as code and accept that generated tests import their claims;
+2. a small declarative claim vocabulary covering what recurs, with an escape
+   hatch to code;
+3. full serialization, at the cost of a language nobody asked for.
+
+Whatever is chosen, publish it as an interface change — the audit-core session
+consumes `Claim` directly.
+
+## A browser-engine driver, if it is warranted
+
+```task
+id: TD-WP-0003-T06
+status: wait
+priority: low
+```
+
+Blocked on the operator establishing Playwright and clearing its licence.
+
+**F-0004** records what the stdlib driver cannot see: JavaScript, screenshots, and
+— most consequentially — **visual relayout**, which is a large fraction of real
+mechanical change. H-001 currently measures *structural* durability only, and that
+caveat is attached to every statement of the result.
+
+Scope is one experiment, not a rewrite: implement `Driver` with a browser engine,
+run the same semantic actions against the same mutations, and answer whether the
+H-001 result changes. If it does not, F-0004 closes and the stdlib driver stands
+as the default.
+
+Do not start this before T01–T03. It widens the surface; those three settle what
+the surface is for.
+
+## Measure the cost of expressing a use case
+
+```task
+id: TD-WP-0003-T07
+status: todo
+priority: medium
+```
+
+`TestDriverResearchPrototype.md` puts *time to express a new use case* first in its
+scorecard, and it has never been measured. T02 creates the only honest opportunity
+— the first time someone writes a use case that was not designed alongside the
+framework.
+
+Record, for each of the two new use cases: elapsed time, lines written, concepts
+consulted, points where the model resisted, and anything that had to be worked
+around rather than expressed.
+
+This feeds the metric the project ultimately cares about — *verified behaviour per
+unit of human maintenance effort* — and it is the number an adopter will ask for
+first. It cannot be reconstructed later.
+
+## Readiness review for real-system application
+
+```task
+id: TD-WP-0003-T08
+status: todo
+priority: high
+```
+
+Write explicit criteria for taking test-driver to a real system, then answer them
+honestly against the evidence. Not a formality: the audit-core work is already in
+this repo and will proceed on whatever this concludes.
+
+At minimum the criteria must cover:
+
+- the independent observation channel **D-07** requires, and what it costs to
+  provide on a system that was not built with one — the single largest adoption
+  barrier and still untested outside the lab;
+- whether claim provenance survives contact with a real backlog, where "the spec"
+  is often a ticket written after the code;
+- what happens when a real system's behaviour is legitimately ambiguous rather
+  than seeded-ambiguous;
+- what a false adaptation would cost **in that specific engagement**, which is the
+  only setting where FAR = 0 stops being an abstraction.
+
+Then state a verdict, including the verdict "not yet, and here is what is missing".
+A readiness review that cannot conclude *not ready* is not a review.