From faef069b17dd6e67dc9c334e3642568d9127d765 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 23 Aug 2026 00:53:07 +0200 Subject: [PATCH] Add TD-WP-0003: generalise the model and settle the open questions Consolidation before application. Everything TD-WP-0002 demonstrated rests on one use case, one application and one token-free runtime; three of the project's own findings say that is too narrow a base for real work. - T01 bounded live-model experiment: settles F-0005 (capability) and F-0007 (economics) in one run. Needs an operator decision on cost ceiling first. - T02 two further use cases: the model was designed against Alice/Bob/Carol, so of course it fits. What it has to grow to express something else is the evidence. - T03 build the dropped-identifier mutation side to ten; H-001 currently turns on three. - T04 settle every gated concept - Temperature and energy.py are used in a decision or removed. Extending a gate is not a result. - T05 decide how claims are expressed, answerable only after T02. - T06 browser-engine driver if warranted (blocked on Playwright). - T07 measure time to express a use case - first in the scorecard, never measured, and reconstructable only during T02. - T08 readiness review with explicit criteria for real-system application. Applying test-driver to audit-core is explicitly out of scope and gated on T08. Coordination note added: another session works in this repo, so staging must be explicit and interface changes published. Co-Authored-By: Claude Opus 5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39 --- workplans/TD-WP-0003-generalise-and-settle.md | 267 ++++++++++++++++++ 1 file changed, 267 insertions(+) create mode 100644 workplans/TD-WP-0003-generalise-and-settle.md diff --git a/workplans/TD-WP-0003-generalise-and-settle.md b/workplans/TD-WP-0003-generalise-and-settle.md new file mode 100644 index 0000000..b17607f --- /dev/null +++ b/workplans/TD-WP-0003-generalise-and-settle.md @@ -0,0 +1,267 @@ +--- +id: TD-WP-0003 +type: workplan +title: "Generalise the model and settle the open questions" +domain: infotech +repo: test-driver +status: proposed +owner: codex +topic_slug: custodian +created: "2026-08-23" +updated: "2026-08-23" +--- + +# Generalise the model and settle the open questions + +## Why this workplan exists + +`TD-WP-0002` passed its gate: False Adaptation Rate 0/7, one asset crystallized +and running with no model, the first Concept–Implementation Fitness Loop closed. +What it does **not** yet have is generality or settled economics. + +Everything demonstrated so far rests on **one use case**, **one application**, and +**one token-free runtime**. Three of the project's own findings say plainly that +this is too narrow a base to build real work on: + +- **F-0005** — H-001 is supported 2/3 against 0/3 on the deciding axis. Directionally + clear, statistically nothing. +- **F-0007** — crystallization's economic case cannot be measured with a runtime + that consumes no tokens. +- **F-0008** — `Temperature` was never consulted while building the thing it was + supposed to govern. + +This workplan widens the base and closes those questions **before** the framework +meets a real system. Its purpose is structure and established experience, not new +capability. + +## Success gate + +1. The conceptual model expresses **three** substantively different multi-user use + cases without new framework concepts being invented for each. +2. A live-model runtime has produced a measured answer on **both** open questions: + does a model add capability beyond the heuristic (F-0005), and does + crystallization save anything that matters (F-0007). +3. Every concept gated at T10 is either **used in a decision or removed**. +4. False Adaptation Rate remains **0** across a materially larger catalogue. +5. Explicit, written criteria exist for taking the framework to a real system — + and an honest verdict on whether they are met. + +## Explicitly not in scope + +- **Applying test-driver to audit-core or any production system.** That is the + next workplan, gated on criterion 5 above. Deliberate: real work should land on + a base that has been generalised, not on one that happens to fit its first + example. +- Energy scoring, campaign selection, retirement — still deferred, and now + additionally *gated for removal* (T04 below). +- Any new lens (resilience, scale) beyond what the three use cases require. + +## Coordination note + +Another session is working in this repo on audit-core use cases +(`usecases/`, `tests/test_audit_core_e2_use_case.py`, `AUDIT-WP-0008`). That work +consumes the kernel as a library. Two consequences for this workplan: + +- **Do not use broad `git add -A`.** Stage explicitly. A T10 commit here swept + that session's in-progress files and a set of `__pycache__` artefacts. +- Interface changes to `testdriver` are no longer private. T05 in particular + touches how claims are expressed, so it needs a published interface change + rather than a quiet edit. + +--- + +## Bounded live-model runtime experiment + +```task +id: TD-WP-0003-T01 +status: todo +priority: high +``` + +Implement a live-model `ActorRuntime` behind the existing interface and run a +**bounded** experiment — a fixed run count and a stated cost ceiling agreed with +the operator before the first call. + +Two questions, one experiment. They are the two highest-value unknowns in the +project and neither can be answered by reasoning: + +- **Capability (F-0005).** M22 defeats the heuristic runtime — form fields renamed, + test ids dropped — while remaining solvable by reading the visible `"Person"` + label. Does a model in fact solve it? If yes, the agentic arm's advantage is + wider than the T07 measurement showed. If no, the semantic-action thesis is + narrower still. +- **Economics (F-0007).** `RealizationMetrics` already carries `tokens_in`, + `tokens_out`, `model` and `retries`, populated from the first run precisely so + this is a subtraction rather than a re-run. Measure cost per agentic + realization against cost per crystallized execution, and compare both against + the honest alternative: *asking an agent to rewrite the broken test*. + +Record run-to-run variance, not only means. A stochastic runtime that succeeds on +average and fails unpredictably is a different product from one that succeeds +reliably, and the difference will not show in an average. + +**Needs an operator decision before starting:** cost ceiling, model, and whether +the live runtime ever runs in the default test suite (recommendation: no — keep +the suite deterministic and free, run the live arm on demand). + +## Two further use cases + +```task +id: TD-WP-0003-T02 +status: todo +priority: high +``` + +Alice/Bob/Carol is one shape: a resource, an owner, a grant, a revoke. Every +concept in the model was designed against it, so of course it fits. + +Add two substantively different multi-user use cases — candidates: an +**approval workflow** with sequencing and delegation, and a **tenant-scoped +lifecycle** with cross-boundary invariants. Choose for difference, not coverage. + +The question is not whether they pass. It is **what the framework had to grow to +express them**, and every such growth is evidence: + +- a concept invented per use case is a modelling failure; +- a concept that finally gets used (`Schedule`, `Lens`, `Variant` are all declared + and unbuilt) is a concept validated; +- a concept still untouched after three use cases is a candidate for removal. + +Record the answer in the fitness map either way. + +## Strengthen the instrument on the deciding axis + +```task +id: TD-WP-0003-T03 +status: todo +priority: high +``` + +H-001's whole result turns on three mutations that drop stable identifiers, of +which discovery handles two. Three is not a sample. + +Build the dropped-identifier side out to at least ten mutations spanning +genuinely different mechanisms — rewritten markup, renamed fields, restructured +forms, relocated controls, changed element semantics. Keep the preserved-identifier +side proportionate so the comparison stays fair. + +Then re-run E-001 and restate H-001 with a number that means something. Be +prepared for the honest outcome that the narrowed claim narrows further. + +## Settle every gated concept + +```task +id: TD-WP-0003-T04 +status: todo +priority: medium +``` + +T10 gated two things on this workplan. Settle both, and do not extend the gate: + +- **`Temperature` (F-0008).** Crystallization was built without it; measured + stability of realization did its job, and is observed rather than declared. + Either a decision in T01–T03 consults it, or it leaves the concept model. +- **`energy.py`.** Events recorded since T04 have informed nothing. Either the + history is used, or the module goes and H-005 is marked dormant-indefinite. + +Also review `Confidence`, `Campaign`, `Metabolism` and `Retirement` — all +declared, unimplemented, never consulted. After three use cases there is enough +evidence to say whether they are deferred or dead. + +Removing a concept is a result. Extending a gate is not. + +## Decide how claims are expressed + +```task +id: TD-WP-0003-T05 +status: todo +priority: medium +``` + +Deferred twice, and now blocking two things at once (F-0007, and the T04 note in +`TD-WP-0002`): claims are Python predicates, so a generated regression test cannot +be fully standalone and scenarios cannot be declarative. + +Answerable only **after T02**, which is the point of the ordering — the question +"which predicates actually recur" needs more than one use case to answer. + +Three candidate answers, and the middle one should not win by default: + +1. keep predicates as code and accept that generated tests import their claims; +2. a small declarative claim vocabulary covering what recurs, with an escape + hatch to code; +3. full serialization, at the cost of a language nobody asked for. + +Whatever is chosen, publish it as an interface change — the audit-core session +consumes `Claim` directly. + +## A browser-engine driver, if it is warranted + +```task +id: TD-WP-0003-T06 +status: wait +priority: low +``` + +Blocked on the operator establishing Playwright and clearing its licence. + +**F-0004** records what the stdlib driver cannot see: JavaScript, screenshots, and +— most consequentially — **visual relayout**, which is a large fraction of real +mechanical change. H-001 currently measures *structural* durability only, and that +caveat is attached to every statement of the result. + +Scope is one experiment, not a rewrite: implement `Driver` with a browser engine, +run the same semantic actions against the same mutations, and answer whether the +H-001 result changes. If it does not, F-0004 closes and the stdlib driver stands +as the default. + +Do not start this before T01–T03. It widens the surface; those three settle what +the surface is for. + +## Measure the cost of expressing a use case + +```task +id: TD-WP-0003-T07 +status: todo +priority: medium +``` + +`TestDriverResearchPrototype.md` puts *time to express a new use case* first in its +scorecard, and it has never been measured. T02 creates the only honest opportunity +— the first time someone writes a use case that was not designed alongside the +framework. + +Record, for each of the two new use cases: elapsed time, lines written, concepts +consulted, points where the model resisted, and anything that had to be worked +around rather than expressed. + +This feeds the metric the project ultimately cares about — *verified behaviour per +unit of human maintenance effort* — and it is the number an adopter will ask for +first. It cannot be reconstructed later. + +## Readiness review for real-system application + +```task +id: TD-WP-0003-T08 +status: todo +priority: high +``` + +Write explicit criteria for taking test-driver to a real system, then answer them +honestly against the evidence. Not a formality: the audit-core work is already in +this repo and will proceed on whatever this concludes. + +At minimum the criteria must cover: + +- the independent observation channel **D-07** requires, and what it costs to + provide on a system that was not built with one — the single largest adoption + barrier and still untested outside the lab; +- whether claim provenance survives contact with a real backlog, where "the spec" + is often a ticket written after the code; +- what happens when a real system's behaviour is legitimately ambiguous rather + than seeded-ambiguous; +- what a false adaptation would cost **in that specific engagement**, which is the + only setting where FAR = 0 stops being an abstraction. + +Then state a verdict, including the verdict "not yet, and here is what is missing". +A readiness review that cannot conclude *not ready* is not a review.