Compare commits

...

3 commits

Author SHA1 Message Date
bec7fdc465 T03: research control plane
H-001..H-005 with explicit falsification conditions, E-001..E-003, the
Concept-Implementation Fitness Map, findings log and ID convention. Plain
files, no tooling.

- fitness map corrects Improvement Loop section 13 levels downward: nothing
  exceeds C1 without implementation
- H-005 (Energy) dormant by decision - events captured, no scoring written
- H-001 gets a genuine control arm so the semantic-action thesis is not
  trivially true
- E-003 added: deliberately attacks the safety boundary rather than only
  sampling benign cases

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-22 23:11:21 +02:00
15b55c807a T02: adaptation classification and intent provenance design
Answers the two questions gating the rest of TD-WP-0002.

Classifier: stratify evidence into S1 surface / S2 realization / S3 judgment.
Adaptation writes only to S1; claims and invariants are run inputs with no
adaptation write path, so False Adaptation Rate = 0 is architectural rather
than a tuning target. Classification is a total function over three signals,
defaulting to AMBIGUOUS on missing evidence; a surface change concurrent with
a verdict change is PRODUCT_DEFECT. Semantic actions declare permitted
surfaces so surface-substitution cannot pass as recovery.

Provenance: claims require a source causally independent of the
implementation. agent-from-implementation output is an exploratory hypothesis
until a human promotes it.

New adoption precondition D-07: test-driver requires an independent
observation channel into the system under test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-22 23:09:32 +02:00
c856cb9e64 T01: establish canonical milestone sequence, record F-0001
Three documents defined conflicting milestone sequences; an unqualified
reference to e.g. 'M2' was ambiguous across the corpus.

- docs/TestDriverInitialMilestones.md (M0-M10) is now canonical
- INTENT.md and ConceptModel section 15 carry superseded banners plus a
  mapping table rather than being deleted
- canonical order is explicitly distinguished from execution order
- research/findings/ established; F-0001 filed as CONCEPT_DRIFT, resolved
  via path 2 (concept deliberately revised), with one residual for T10

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-22 23:07:31 +02:00
18 changed files with 784 additions and 78 deletions

View file

@ -360,49 +360,27 @@ These are logical boundaries first. They do not require separate services or pac
## Initial Milestones
### M0 — Deterministic Semantic Scenario Runner
> **Superseded.** This section previously carried its own `M0``M3` sequence,
> which conflicted with two other sequences in the repository. The canonical
> milestone sequence is now **`docs/TestDriverInitialMilestones.md` (M0M10)**.
> Recorded as framework finding `F-0001` (CONCEPT_DRIFT), 2026-08-22.
Implement the conceptual kernel without agentic involvement:
The former M0M3 map onto the canonical sequence as follows:
- UseCase,
- Actor,
- World,
- Scenario,
- SemanticAction,
- Driver,
- Observation,
- Evidence,
- Oracle,
- Verdict.
| Superseded (INTENT.md) | Canonical (`TestDriverInitialMilestones.md`) |
|---|---|
| M0 Deterministic Semantic Scenario Runner | M1 Deterministic Semantic Kernel |
| M1 Agentic Driver | M4 Agentic Realization |
| M2 Adaptation and Crystallization | M5 Mechanical Adaptation, M6 Defect vs Adaptation, M7 Crystallization |
| M3 Living Verification | M9 Verification Energy |
Run the Alice/Bob/Carol reference scenario deterministically.
The canonical sequence additionally introduces M0 (Research Control Plane),
M2 (Test-Driver Lab), M3 (Self-Verification), M8 (Improvement Loop) and
M10 (Compression Review), which had no counterpart here.
### M1 — Agentic Driver
Allow an actor agent to realize a semantic action or scenario goal through one interaction surface while maintaining strict actor isolation and evidence capture.
### M2 — Adaptation and Crystallization
Add:
- adaptation classification,
- learned successful paths,
- hardening,
- deterministic test candidate generation,
- lineage from exploratory discovery to regression test.
### M3 — Living Verification
Add:
- energy,
- temperature,
- confidence,
- retirement,
- security mutation campaigns,
- adaptive campaign selection.
---
Execution order is *not* the canonical order — see
`workplans/TD-WP-0002-vertical-spike-crystallization.md`, which drives one thin
thread through M1M7 rather than completing each milestone in turn.
## Non-Goals for the Initial Project

View file

@ -0,0 +1,191 @@
# Adaptation Classification and Intent Provenance
**Status:** v0.1 — design decision
**Workplan:** `TD-WP-0002-T02`
**Supersedes nothing.** Extends `TestDriverConceptModel.md` §§ 2.3, 7.8, 9.4.
This note answers the two questions that gate every implementation task in
`TD-WP-0002`. Both were unanswered in the concept corpus, and both are cheaper to
answer on paper than to discover at canonical M6.
---
## Part A — The classifier
### The question
What **deterministic evidence** separates *"the sharing button moved"* from
*"Bob can still read R after revocation"*?
The concept corpus assumes such a discriminator exists but never names it. If the
honest answer were "a model decides", Concept Model § 2.3 would be violated and
the framework's central safety claim would collapse.
### The reframe
The question as posed is harder than the question the framework actually has to
answer, because it presumes a single classifier looking at an undifferentiated
failure and deciding what kind it is.
Instead, **evidence is stratified**, and each stratum has a different authority:
| Stratum | Content | Gathered by | May a model touch it? |
|---|---|---|---|
| **S1 — Surface** | selectors, DOM, URLs, route shapes, coordinates, screenshots | the actor, incidentally, while realizing an action | **yes** — this is mechanics |
| **S2 — Realization** | did the semantic action's declared postcondition hold; which surface was actually used | an Observer, out-of-band from the actor | no |
| **S3 — Judgment** | claims and invariants evaluated against domain-state observations | Observers + Oracles, independent of the actor | **never** |
S1 is how. S2 is whether it happened. S3 is whether that was correct.
### The structural rule
> **A change in S1 can never license a change in S3.**
This is enforced architecturally rather than by classifier accuracy:
- **Claims and invariants are inputs to a run, not outputs of it.** The adaptation
machinery has no write path to them. There is no code path by which a retry, a
re-realization, or a learned trajectory can alter what is asserted.
- **Adaptation is confined to the realization loop.** It may change *how* a
semantic action is performed. It cannot change *what follows from* performing it.
- Therefore the classifier is never asked "is this failure acceptable?" It is only
asked "why did this run fail to reach the point of judgment?"
This makes `False Adaptation Rate = 0` an **architectural property, not a tuning
target**. The system does not merely try hard to avoid normalizing a defect — it
cannot express the operation. That matters because a rate driven near zero by
tuning regresses silently; a rate that is zero by construction does not.
### The decision table
Classification is a total function over three observed signals. Baseline refers
to the last accepted run of the same verification asset.
| S1 surface differs | S2 postcondition met | S3 verdicts vs baseline | Classification |
|---|---|---|---|
| yes | yes | unchanged | `MECHANICAL_ADAPTATION` |
| no | yes | changed to FAIL | `PRODUCT_DEFECT` |
| yes | yes | changed to FAIL | `PRODUCT_DEFECT` |
| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human |
| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` |
| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` |
| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate |
Two rows carry the whole safety argument:
- **Row 3** — a surface change occurring *alongside* a verdict change is classified
as a defect, never as an adaptation. Coincidence is not exoneration. This is the
row that a naive self-healing tool gets wrong, and it is where the seeded M05
authorization defect must land even when it ships together with M01/M02.
- **Row 7** — missing or contradictory evidence yields `AMBIGUOUS`, never a
default-to-pass and never a default-to-adapt. `INCONCLUSIVE` already exists as a
first-class verdict; this row is what gives it teeth.
### Realization legitimacy
S2 records not only *whether* the postcondition held but *which surface produced
it*. A semantic action declares the surfaces it may legitimately use; an actor
that achieves `grant_access` by calling the HTTP API when the scenario constrains
it to the browser has not realized the action — it has performed an unrequested
**surface-substitution**, which is itself one of the catalogued security
mutations.
Without this check, an agent can "recover" from a UI authorization control by
routing around it and the run would score as a successful mechanical adaptation.
That is the most plausible concrete route to a false adaptation, so it is checked
explicitly rather than assumed away.
### Where model judgment is and is not used
Concept Model § 2.3 is satisfied, and the boundary is now precise:
- **Used:** locating a moved control, proposing a path through a changed DOM,
summarizing a trajectory. All S1. All independently verified afterwards by S2.
- **Not used:** evaluating a claim, evaluating an invariant, producing a verdict,
deciding whether an adaptation was legitimate, deciding whether a run passed.
A model can be wrong about where the button is; the run then fails at S2 and is
retried or reported. A model is never in a position to be wrong about whether Bob
could still read R.
### What this costs
The design requires an out-of-band observer capable of evaluating every semantic
action's postcondition and every claim without going through the actor's surface.
In the lab this is a direct database or admin-API reader. For a real system under
test it is the main integration burden the framework imposes, and it should be
stated plainly to any future adopter: **test-driver requires an independent
observation channel.** Without one, S2 and S3 collapse into S1 and the safety
argument is void.
This is a real limitation and is recorded as such rather than discovered later.
---
## Part B — Intent provenance
### The question
If an agent writes the implementation, an agent writes the use case, and an agent
realizes the test, then "oracles are independent from actors" is *procedurally*
true and *epistemically* empty. Independence of components does not give
independence of belief when all components derive from the same source.
### The rule
> A claim may only be authored by a source **causally independent of the
> implementation it constrains.**
`UseCase` and `Claim` therefore carry a required `provenance` field:
| Value | Meaning | Admissible as a claim? |
|---|---|---|
| `human` | authored by a person | yes |
| `spec` | derived from a specification artifact that demonstrably predates the implementation (ref + commit SHA recorded) | yes |
| `agent-from-spec` | an agent transcribed a human or spec artifact; a human accepted the result | yes, with the accepting identity recorded |
| `agent-from-implementation` | an agent derived it by observing what the system does | **no** |
`agent-from-implementation` is not forbidden as an activity — it is genuinely
useful for T0 exploration. It is forbidden as a **claim**. Such output enters the
system as an *exploratory hypothesis*, which can drive runs and accumulate
evidence but cannot produce a `FAIL`, cannot enter an invariant, and cannot be
crystallized. Promotion to a claim requires an explicit human acceptance event,
recorded with identity and timestamp.
Without this rule, the framework's guarantee reduces to "the implementation
agrees with itself", which is exactly the failure the project exists to prevent.
### Consequences
- Provenance is carried into every Evidence Pack, so any verdict can later be
audited for independence. A `FAIL` whose claim has inadmissible provenance is
itself an `EVIDENCE_FAILURE`.
- Crystallization preserves provenance. A deterministic test generated from an
agentic ancestor inherits the ancestor's claim provenance and cannot launder it.
- A fourth self-test joins the M3 set: `td://self/intent-independence` — asserting
that no admissible claim in the corpus has implementation-derived provenance.
### Honest limitation
This rule constrains *provenance*, not *quality*. A human can write a claim that
merely restates what they saw the system do. The rule makes the independence
question answerable and auditable; it does not make it automatic. That residual
is accepted for the spike and revisited at T10.
---
## Decisions recorded
- **D-01** Evidence is stratified S1/S2/S3; adaptation may write only to S1.
- **D-02** Claims and invariants are run inputs; no adaptation path may modify them.
- **D-03** Classification is a total function over (S1 differs, S2 met, S3 vs baseline),
defaulting to `AMBIGUOUS` on missing evidence.
- **D-04** A surface change concurrent with a verdict change classifies as
`PRODUCT_DEFECT`, never `MECHANICAL_ADAPTATION`.
- **D-05** Semantic actions declare permitted surfaces; realization via another
surface is a finding, not a recovery.
- **D-06** Claims require causally independent provenance;
`agent-from-implementation` output is an exploratory hypothesis until a human
promotes it.
- **D-07** test-driver requires an independent observation channel into the system
under test. Stated as an adoption precondition.

View file

@ -910,46 +910,21 @@ The conceptual components may initially share a process or codebase. Their separ
# 15. Initial Development Sequence
## M0 — Deterministic semantic scenario runner
> **Superseded.** This section previously carried its own `M0``M3` sequence,
> which conflicted with the sequences in `INTENT.md` and
> `TestDriverInitialMilestones.md`. The canonical milestone sequence is
> **`TestDriverInitialMilestones.md` (M0M10)**. Recorded as framework finding
> `F-0001` (CONCEPT_DRIFT), 2026-08-22.
Implement:
The former M0M3 corresponded to canonical M1, M4, M5M7 and M9 respectively.
See the mapping table in `INTENT.md` § Initial Milestones.
- use cases,
- actors,
- worlds,
- semantic actions,
- observations,
- oracles,
- verdicts.
Execute the reference multi-user scenario without AI.
## M1 — Agentic driver
Allow an actor agent to realize a semantic action or use-case subgoal through a browser or API surface while preserving actor isolation and evidence capture.
## M2 — Adaptation and crystallization
Implement:
- adaptation classification,
- successful-path persistence,
- hardening,
- generation of deterministic test candidates.
## M3 — Living verification
Introduce:
- energy,
- temperature,
- lineage,
- retirement,
- mutation campaigns,
- adaptive test selection.
Execution order is *not* the canonical order — see
`workplans/TD-WP-0002-vertical-spike-crystallization.md`.
---
# 16. Canonical Concept Set v0.1
The initial test-driver vocabulary is:

View file

@ -1,8 +1,18 @@
# TestDriver Research Prototype — Initial Milestones
**Status:** v0.1
**Status:** v0.1**canonical milestone sequence**
**Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.
> This document is the single canonical source for milestone identifiers `M0``M10`.
> The competing sequences formerly in `INTENT.md` and `TestDriverConceptModel.md` § 15
> are superseded — see framework finding
> `research/findings/F-0001-milestone-sequence-drift.md`.
>
> **Canonical order is not execution order.** The milestones define *what must be
> true*; `workplans/TD-WP-0002-vertical-spike-crystallization.md` defines *the order
> in which we get there*, driving one thin thread through M1M7 rather than
> completing each milestone in turn.
---
## Milestone 0 — Research Control Plane

48
research/README.md Normal file
View file

@ -0,0 +1,48 @@
# Research Control Plane
Deliberately small. This directory exists so that claims about test-driver can be
falsified rather than accumulated. It is plain files — no CLI, no schema, no
tooling — until there are enough readings to justify tooling.
```
research/
├── hypotheses/ H-NNN — a claim with a falsification condition
├── experiments/ E-NNN — a planned or executed test of a hypothesis
├── findings/ F-NNNN — findings about test-driver itself
├── concepts/ the Concept ↔ Implementation Fitness Map
└── decisions/ pointers to decisions recorded in State Hub
```
## Identifier convention
| Prefix | Scope | Example |
|---|---|---|
| `H-NNN` | Hypothesis | `H-001` |
| `E-NNN` | Experiment | `E-001` |
| `F-NNNN` | Framework Finding | `F-0001` |
| `C-<slug>` | Concept in the fitness map | `C-actor-isolation` |
| `D-NN` | Decision, scoped to its design note | `D-07` |
| `TD-WP-NNNN-TNN` | Workplan task (State Hub) | `TD-WP-0002-T04` |
Identifiers are stable and never reused. A rejected hypothesis keeps its number.
## Hypothesis lifecycle
```
PROPOSED → EXPERIMENTING → SUPPORTED → PRACTICALLY_VALIDATED → ARCHITECTURAL
└──────→ REJECTED
```
A hypothesis may be reopened if later evidence contradicts it. Reopening is
recorded in the file, not by creating a new identifier.
## Rules
1. **Every hypothesis states what would falsify it**, in terms of an observable
outcome, before any experiment runs. A hypothesis with no falsification
condition is an opinion.
2. **Concepts with no supporting evidence are marked as such**, not quietly
retained. The fitness map is expected to contain unsupported entries; hiding
them defeats its purpose.
3. **Subtraction counts as progress.** A rejected hypothesis or a removed
abstraction is a result, not a setback.

View file

@ -0,0 +1,52 @@
# Concept ↔ Implementation Fitness Map
**Updated:** 2026-08-22 (TD-WP-0002-T03)
Traces each important concept to the implementation, experiment and evidence that
support it. **Unsupported entries are the point of this map** — a concept with no
implementation and no evidence is not a gap to be embarrassed about, it is the
current honest state, and hiding it defeats the map's purpose.
Support levels follow `TestDriverImprovementLoop.md` §13:
`C0 Idea` · `C1 Hypothesis` · `C2 Experimentally Supported` ·
`C3 Practically Validated` · `C4 Architectural Invariant`
## Current state
At the time of writing there is **no implementation**, so no concept can honestly
exceed `C1`. The initial classifications in §13 of the Improvement Loop
(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they
were aspirational, not evidenced.
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|---|---|---|---|---|---|
| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? |
| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. |
| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) |
| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? |
| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) |
| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. |
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. |
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) |
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
| `C-confidence` | C0 | — | — | — | Deferred. |
| `C-campaign` | C0 | — | — | — | Deferred. |
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. |
## Orphan check
**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`:
`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`.
All five are deferred *by explicit decision*, not oversight. They are the group
most at risk of being built because they are easy and satisfying, and never
validated. They are revisited at T10, where the question is not "when do we build
these" but "does the evidence justify keeping them in the model at all".
**Implementation orphans** — none yet; there is no implementation.

View file

@ -0,0 +1,9 @@
# Decisions
Decisions are recorded in State Hub (the durable store) and indexed here for
offline readers. Design rationale lives in the corresponding `docs/` note; this
file is a pointer table, not a second source of truth.
| Ref | Title | Hub ID | Source |
|---|---|---|---|
| D-01 … D-07 | Stratified evidence and claim provenance as the basis for adaptation safety | `fef5213f-ce9b-44c2-b327-a0b0ba4b6270` | `docs/TestDriverClassificationDesign.md` |

View file

@ -0,0 +1,38 @@
---
id: E-001
title: Mechanical recovery and defect discrimination over the labelled mutation set
status: PLANNED
hypotheses: [H-001, H-002, H-004]
task: TD-WP-0002-T08
created: "2026-08-22"
---
# E-001 — Mechanical recovery and defect discrimination
## Design
Run the same verification asset against every labelled lab mutation (T05),
in two arms:
- **arm A** — semantic action realized by the agentic driver;
- **arm B** — a recorded selector-based sequence captured against the baseline,
built with the most robust selector strategy reasonably available.
Arm B is the control for H-001 and must actually be run.
## Outcomes recorded per run
surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts
(S3) · classification emitted · claim/invariant diff (expected empty) ·
tokens · wall time · retries.
## Decision rules
- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune.
- Arm B recovery ≥ arm A recovery → H-001 falsified.
- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a
framework defect, since D-02 forbids the write path.
## Status
`PLANNED`. Blocked on T05 (lab) and T07 (agentic driver).

View file

@ -0,0 +1,33 @@
---
id: E-002
title: Crystallization fidelity and cost
status: PLANNED
hypotheses: [H-003]
task: TD-WP-0002-T09
created: "2026-08-22"
---
# E-002 — Crystallization fidelity and cost
## Design
Run one agentic realization repeatedly against a stable lab version until
realization is stable, crystallize it, then execute the descendant with no model
involvement across every lab version the ancestor was run against.
## Outcomes recorded
Oracle set of descendant vs ancestor · verdict agreement per lab version ·
execution cost ratio (tokens, wall time) · whether human authorship of assertions
was required.
## Decision rules
- Descendant oracle set is not a superset of the ancestor's → H-003 falsified.
- Any verdict disagreement on the same lab version → H-003 falsified.
- Cost ratio ≥ 1 → thesis intact, product rationale is not. Record plainly rather
than quietly.
## Status
`PLANNED`. Blocked on T07.

View file

@ -0,0 +1,40 @@
---
id: E-003
title: Deliberate attack on the adaptation safety boundary
status: PLANNED
hypotheses: [H-004]
task: TD-WP-0002-T08
created: "2026-08-22"
---
# E-003 — Deliberate attack on the adaptation safety boundary
## Why this experiment exists separately
E-001 measures whether the framework *happens to* stay safe over a labelled set.
E-003 tries to *make it unsafe*. The distinction matters: an experiment that only
samples benign cases cannot establish a safety property, and False Adaptation Rate
is the one metric the project cannot afford to be wrong about.
## Attack cases
1. **Surface substitution** — remove the sharing control from the UI while leaving
the HTTP endpoint open. A driver that "recovers" by calling the API has routed
around the control. Must classify as a finding, never as recovery (D-05).
2. **Concurrent change** — ship a mechanical mutation (M01) and the authorization
defect (M05) in the same lab version. Must classify as `PRODUCT_DEFECT`;
coincidence must not exonerate (decision table row 3).
3. **Evidence starvation** — disable the independent observation channel mid-run.
Must yield `AMBIGUOUS`, never a pass and never an adaptation (row 7).
4. **Provenance laundering** — attempt to introduce a claim derived from observed
implementation behaviour and crystallize it. Must be rejected at authoring
time (D-06) and, if it slips through, at crystallization.
## Decision rule
Any attack that succeeds is a stop-and-redesign signal for H-004, regardless of
what E-001 reports.
## Status
`PLANNED`. Blocked on T05, T07, T08.

View file

@ -0,0 +1,77 @@
---
id: F-0001
type: framework-finding
class: CONCEPT_DRIFT
status: resolved
discovered: "2026-08-22"
resolved: "2026-08-22"
discovered_by: concept-assessment
source: history/2026-08-22-concept-assessment-swot.md
workplan: TD-WP-0002
task: TD-WP-0002-T01
---
# F-0001 — Three conflicting milestone sequences
## Class
`CONCEPT_DRIFT` — the repository carried three mutually inconsistent statements
of the same concept, with no explicit decision revising any of them.
Note the unusual shape: §5 of the Improvement Loop defines concept drift as
*implementation* diverging from concept. Here there was no implementation at all.
The drift was **document-to-document**, which the taxonomy does not currently
name. See "Residual" below.
## Observation
Three documents each defined a milestone sequence for the same project:
| Document | Sequence |
|---|---|
| `INTENT.md` § Initial Milestones | M0M3 |
| `docs/TestDriverConceptModel.md` § 15 | M0M3 (different content) |
| `docs/TestDriverInitialMilestones.md` | M0M10 |
The two M0M3 sequences disagreed with each other and both disagreed with the
M0M10 sequence. `M2`, for instance, denoted "Adaptation and Crystallization" in
one document and "Test-Driver Lab" in another — so an unqualified reference to
"M2" was ambiguous across the corpus.
## Why it matters
The project's own thesis is that implementations must not silently redefine
intended behaviour, and that concept drift is a first-class finding. The corpus
exhibited the failure it was written to prevent, before a single line of code
existed. Left unresolved, every subsequent milestone reference in commits,
workplans and evidence would have been ambiguous.
## Resolution
Path 2 of §5 — *the concept is deliberately revised*.
`docs/TestDriverInitialMilestones.md` (M0M10) is now **canonical**. It is the
most complete of the three, is the only one carrying exit criteria and success
metrics, and is the only one that names the lab, self-verification, the
improvement loop and the compression review.
The other two sections were replaced in place with a superseded banner and a
mapping table rather than deleted, so that references from earlier commits and
from `history/` remain interpretable.
Execution order is deliberately *not* the canonical order:
`workplans/TD-WP-0002-vertical-spike-crystallization.md` drives one thin thread
through canonical M1M7 rather than completing each milestone in turn. Canonical
sequence and execution sequence are now distinct and separately stated.
## Residual
The finding taxonomy (§9) has no class for divergence **between concept
documents** — `CONCEPT_DRIFT` is defined as implementation-vs-concept. This
finding was classified `CONCEPT_DRIFT` as the nearest fit.
Carry into `TD-WP-0002-T10` (compression review): either widen `CONCEPT_DRIFT`
to cover intra-conceptual inconsistency, or accept that pre-implementation
document drift is not a framework finding at all and should be handled as
ordinary editorial hygiene. Do not add a new finding class before that question
is answered — the taxonomy is already larger than the evidence supporting it.

View file

@ -0,0 +1,19 @@
# Framework Findings
Findings about **test-driver itself**, kept strictly separate from findings about
any system under test. Classes are defined in
`docs/TestDriverImprovementLoop.md` § 9:
`PRODUCT_DEFECT` · `TEST_DEFECT` · `MECHANICAL_ADAPTATION` · `SEMANTIC_CHANGE` ·
`CONCEPT_DRIFT` · `FRAMEWORK_LIMITATION` · `EVIDENCE_FAILURE` ·
`UNNECESSARY_COMPLEXITY`
File naming: `F-NNNN-<slug>.md`. Identifiers are stable and never reused.
Each finding carries frontmatter (`id`, `class`, `status`, dates, originating
workplan/task) and, when resolved, states which of the three §5 resolution paths
was taken. A finding that leaves an open question records it under "Residual" and
names the task that will answer it — residuals are not parked in prose.
The wider research control plane (`hypotheses/`, `experiments/`, `concepts/`,
`decisions/`) is built in `TD-WP-0002-T03`.

View file

@ -0,0 +1,41 @@
---
id: H-001
title: Semantic Action Stability
status: PROPOSED
created: "2026-08-22"
experiments: [E-001]
concepts: [C-semantic-action]
---
# H-001 — Semantic Action Stability
## Claim
A semantic action survives implementation restructuring better than a recorded UI
interaction sequence.
## Falsification condition
Across the labelled mechanical mutations in the lab (T05), a recorded interaction
sequence survives **at least as many** mutations as the semantic action does.
If mechanics-free identity buys no measurable durability, the central abstraction
is decorative and `SemanticAction` should be reduced to a naming convention.
## Measurement
Mechanical Recovery Rate for each of two arms over the same mutation set:
- **arm A** — semantic action realized by an agentic driver;
- **arm B** — a recorded selector-based sequence captured against the baseline.
Arm B is a genuine control and must be run, not assumed to fail.
## Threats to validity
The comparison is unfair if arm B is built naively — a brittle straw man makes
H-001 trivially true and worthless. Arm B uses the most robust selector strategy
reasonably available (roles, labels, test ids where the lab provides them).
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,40 @@
---
id: H-002
title: Mechanical Adaptation
status: PROPOSED
created: "2026-08-22"
experiments: [E-001]
concepts: [C-adaptation]
---
# H-002 — Mechanical Adaptation
## Claim
An agentic driver can recover from a mechanical implementation change without
modifying the semantics of the protected use case.
## Falsification condition
Either:
- the driver fails to recover from a majority of labelled `MECHANICAL` mutations
(recovery is not achievable in practice); **or**
- recovery is achieved only in runs where claims or invariants were altered
(recovery is achievable but not semantics-preserving).
The second branch matters more than the first. A high recovery rate purchased by
weakening assertions falsifies this hypothesis just as decisively as no recovery
at all.
## Measurement
- Mechanical Recovery Rate over `MECHANICAL`-labelled mutations.
- Claim/invariant diff per run — expected to be empty by construction
(`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a
falsification signal **and** a framework defect, since no write path should exist.
## Status log
- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification
branch structurally unreachable; the measurement is retained anyway, as an
assertion that the architecture is what we believe it is.

View file

@ -0,0 +1,43 @@
---
id: H-003
title: Crystallization
status: PROPOSED
created: "2026-08-22"
experiments: [E-002]
concepts: [C-crystallization]
---
# H-003 — Crystallization
## Claim
A sufficiently stable agentic execution can be converted into deterministic test
code without losing relevant oracle coverage.
## Falsification condition
Any of:
- the generated deterministic test cannot be produced without human authorship of
the assertions;
- it drops one or more oracles present in the agentic ancestor;
- it passes where the ancestor would have failed, or vice versa, on the same lab
version;
- it costs no less to execute than the agentic ancestor.
## Measurement
- Crystallization success rate.
- Oracle set of descendant vs ancestor — must be a superset or equal.
- Verdict agreement between ancestor and descendant across lab versions.
- Execution cost ratio (tokens, wall time) descendant : ancestor.
## Note on the cost criterion
The cost criterion is included deliberately. If crystallization preserves
semantics but saves nothing, the thesis is intact but the product rationale is
not — and that distinction should be visible in the record rather than discovered
by a user.
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,46 @@
---
id: H-004
title: Independent Judgment
status: PROPOSED
created: "2026-08-22"
experiments: [E-001, E-003]
concepts: [C-oracle-independence, C-intent-provenance]
---
# H-004 — Independent Judgment
## Claim
Separating actor execution from deterministic oracles reduces false-positive
adaptation to defective behaviour.
## Falsification condition
The framework classifies any seeded `DEFECT`-labelled mutation as
`MECHANICAL_ADAPTATION` — i.e. **False Adaptation Rate > 0** over the labelled set.
This is the project's existential safety metric. A non-zero result is a
stop-and-redesign signal, not a tuning exercise.
## Measurement
- False Adaptation Rate over `DEFECT`-labelled mutations. Target: exactly 0.
- Classification precision/recall over the full labelled set.
- Rate of `AMBIGUOUS` escalation — tracked as a cost, not as a failure. A system
that escalates everything is safe and useless.
## Relationship to the design
`docs/TestDriverClassificationDesign.md` argues that FAR = 0 follows
architecturally from D-02 (claims are run inputs with no adaptation write path)
rather than from classifier accuracy. This hypothesis therefore tests the
**architecture**, not a model.
The most plausible route to falsification is not a misclassification but
**surface-substitution**: an actor routing around a broken authorization control
via another surface and scoring as a successful recovery. D-05 exists to close
that route; E-003 exists to attack it deliberately.
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,40 @@
---
id: H-005
title: Verification Energy
status: PROPOSED
created: "2026-08-22"
experiments: []
concepts: [C-energy]
---
# H-005 — Verification Energy
## Claim
Historical evidence about defects caught, adaptations required, false positives
and duplication can identify verification assets whose continued execution is
more valuable than others.
## Falsification condition
Energy scores computed from event history fail to rank assets better than a
trivial baseline (e.g. recency, or uniform priority) when predicting which assets
subsequently catch a defect.
## Why no experiment is planned yet
**This hypothesis is not testable at the current scale and is deliberately
dormant.** Validating it requires event history across many assets over months —
history the spike will not accumulate. Implementing a scoring function now would
produce a number that cannot be checked, which is worse than no number.
`TD-WP-0002` therefore records **raw immutable `EnergyEvent`s from the first run**
and implements no scoring, decay, or selection logic. Events cannot be
reconstructed later; scores can always be computed later.
This is the hypothesis most likely to be **cheaply built and never validated**,
which is precisely why it is fenced off.
## Status log
- 2026-08-22 `PROPOSED`, dormant by decision. Event capture only.

View file

@ -4,7 +4,7 @@ type: workplan
title: "Vertical spike: falsify the crystallization thesis"
domain: infotech
repo: test-driver
status: proposed
status: active
owner: codex
topic_slug: custodian
created: "2026-08-22"
@ -69,6 +69,9 @@ satisfying to build, and impossible to validate at this stage:
One process, one database, one browser engine, one application under test.
- Novelty belongs in the verification model, never in the infrastructure.
- No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
- Evidence stratification S1/S2/S3 and claim provenance per
`docs/TestDriverClassificationDesign.md` (decision
`fef5213f-ce9b-44c2-b327-a0b0ba4b6270`). Adaptation writes only to S1.
---
@ -76,7 +79,7 @@ satisfying to build, and impossible to validate at this stage:
```task
id: TD-WP-0002-T01
status: todo
status: done
priority: high
state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a"
```
@ -91,11 +94,17 @@ Pick one canonical sequence, mark the others superseded in place, and commit
first entry in the framework findings log — the project should catch this on
itself before it catches anything else.
**Done 2026-08-22.** `docs/TestDriverInitialMilestones.md` (M0M10) is canonical;
the `INTENT.md` and Concept Model § 15 sequences carry superseded banners with a
mapping table. Recorded as `research/findings/F-0001-milestone-sequence-drift.md`.
`INTENT.md` and `docs/` are now tracked. One residual carried to T10: the finding
taxonomy has no class for document-to-document drift.
## Decide intent provenance and the classification mechanism on paper
```task
id: TD-WP-0002-T02
status: todo
status: done
priority: high
state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63"
```
@ -116,6 +125,23 @@ survives in each case.
Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision.
**Done 2026-08-22.** Design note written; decision
`fef5213f-ce9b-44c2-b327-a0b0ba4b6270` recorded in the hub.
Outcome in one line: **adaptation may write only to surface evidence (S1); claims
and invariants are run inputs with no adaptation write path.** False Adaptation
Rate = 0 therefore becomes an architectural property rather than a tuning target —
the system cannot express "accept a defect as an adaptation". Classification is a
total function over three signals defaulting to `AMBIGUOUS`; a surface change
concurrent with a verdict change classifies as `PRODUCT_DEFECT`, never as
adaptation. Claims require causally independent provenance.
Two consequences that change later tasks:
- **D-07** — an independent observation channel into the system under test is now
an explicit adoption precondition, not an implementation detail. T05 must expose
one in the lab; without it S2/S3 collapse into S1.
- A fourth self-test, `td://self/intent-independence`, joins the T06 set.
## Minimal research control plane
```task