Compare commits
3 commits
b21f23cabb
...
bec7fdc465
| Author | SHA1 | Date | |
|---|---|---|---|
| bec7fdc465 | |||
| 15b55c807a | |||
| c856cb9e64 |
18 changed files with 784 additions and 78 deletions
56
INTENT.md
56
INTENT.md
|
|
@ -360,49 +360,27 @@ These are logical boundaries first. They do not require separate services or pac
|
|||
|
||||
## Initial Milestones
|
||||
|
||||
### M0 — Deterministic Semantic Scenario Runner
|
||||
> **Superseded.** This section previously carried its own `M0`–`M3` sequence,
|
||||
> which conflicted with two other sequences in the repository. The canonical
|
||||
> milestone sequence is now **`docs/TestDriverInitialMilestones.md` (M0–M10)**.
|
||||
> Recorded as framework finding `F-0001` (CONCEPT_DRIFT), 2026-08-22.
|
||||
|
||||
Implement the conceptual kernel without agentic involvement:
|
||||
The former M0–M3 map onto the canonical sequence as follows:
|
||||
|
||||
- UseCase,
|
||||
- Actor,
|
||||
- World,
|
||||
- Scenario,
|
||||
- SemanticAction,
|
||||
- Driver,
|
||||
- Observation,
|
||||
- Evidence,
|
||||
- Oracle,
|
||||
- Verdict.
|
||||
| Superseded (INTENT.md) | Canonical (`TestDriverInitialMilestones.md`) |
|
||||
|---|---|
|
||||
| M0 Deterministic Semantic Scenario Runner | M1 Deterministic Semantic Kernel |
|
||||
| M1 Agentic Driver | M4 Agentic Realization |
|
||||
| M2 Adaptation and Crystallization | M5 Mechanical Adaptation, M6 Defect vs Adaptation, M7 Crystallization |
|
||||
| M3 Living Verification | M9 Verification Energy |
|
||||
|
||||
Run the Alice/Bob/Carol reference scenario deterministically.
|
||||
The canonical sequence additionally introduces M0 (Research Control Plane),
|
||||
M2 (Test-Driver Lab), M3 (Self-Verification), M8 (Improvement Loop) and
|
||||
M10 (Compression Review), which had no counterpart here.
|
||||
|
||||
### M1 — Agentic Driver
|
||||
|
||||
Allow an actor agent to realize a semantic action or scenario goal through one interaction surface while maintaining strict actor isolation and evidence capture.
|
||||
|
||||
### M2 — Adaptation and Crystallization
|
||||
|
||||
Add:
|
||||
|
||||
- adaptation classification,
|
||||
- learned successful paths,
|
||||
- hardening,
|
||||
- deterministic test candidate generation,
|
||||
- lineage from exploratory discovery to regression test.
|
||||
|
||||
### M3 — Living Verification
|
||||
|
||||
Add:
|
||||
|
||||
- energy,
|
||||
- temperature,
|
||||
- confidence,
|
||||
- retirement,
|
||||
- security mutation campaigns,
|
||||
- adaptive campaign selection.
|
||||
|
||||
---
|
||||
Execution order is *not* the canonical order — see
|
||||
`workplans/TD-WP-0002-vertical-spike-crystallization.md`, which drives one thin
|
||||
thread through M1–M7 rather than completing each milestone in turn.
|
||||
|
||||
## Non-Goals for the Initial Project
|
||||
|
||||
|
|
|
|||
191
docs/TestDriverClassificationDesign.md
Normal file
191
docs/TestDriverClassificationDesign.md
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
# Adaptation Classification and Intent Provenance
|
||||
|
||||
**Status:** v0.1 — design decision
|
||||
**Workplan:** `TD-WP-0002-T02`
|
||||
**Supersedes nothing.** Extends `TestDriverConceptModel.md` §§ 2.3, 7.8, 9.4.
|
||||
|
||||
This note answers the two questions that gate every implementation task in
|
||||
`TD-WP-0002`. Both were unanswered in the concept corpus, and both are cheaper to
|
||||
answer on paper than to discover at canonical M6.
|
||||
|
||||
---
|
||||
|
||||
## Part A — The classifier
|
||||
|
||||
### The question
|
||||
|
||||
What **deterministic evidence** separates *"the sharing button moved"* from
|
||||
*"Bob can still read R after revocation"*?
|
||||
|
||||
The concept corpus assumes such a discriminator exists but never names it. If the
|
||||
honest answer were "a model decides", Concept Model § 2.3 would be violated and
|
||||
the framework's central safety claim would collapse.
|
||||
|
||||
### The reframe
|
||||
|
||||
The question as posed is harder than the question the framework actually has to
|
||||
answer, because it presumes a single classifier looking at an undifferentiated
|
||||
failure and deciding what kind it is.
|
||||
|
||||
Instead, **evidence is stratified**, and each stratum has a different authority:
|
||||
|
||||
| Stratum | Content | Gathered by | May a model touch it? |
|
||||
|---|---|---|---|
|
||||
| **S1 — Surface** | selectors, DOM, URLs, route shapes, coordinates, screenshots | the actor, incidentally, while realizing an action | **yes** — this is mechanics |
|
||||
| **S2 — Realization** | did the semantic action's declared postcondition hold; which surface was actually used | an Observer, out-of-band from the actor | no |
|
||||
| **S3 — Judgment** | claims and invariants evaluated against domain-state observations | Observers + Oracles, independent of the actor | **never** |
|
||||
|
||||
S1 is how. S2 is whether it happened. S3 is whether that was correct.
|
||||
|
||||
### The structural rule
|
||||
|
||||
> **A change in S1 can never license a change in S3.**
|
||||
|
||||
This is enforced architecturally rather than by classifier accuracy:
|
||||
|
||||
- **Claims and invariants are inputs to a run, not outputs of it.** The adaptation
|
||||
machinery has no write path to them. There is no code path by which a retry, a
|
||||
re-realization, or a learned trajectory can alter what is asserted.
|
||||
- **Adaptation is confined to the realization loop.** It may change *how* a
|
||||
semantic action is performed. It cannot change *what follows from* performing it.
|
||||
- Therefore the classifier is never asked "is this failure acceptable?" It is only
|
||||
asked "why did this run fail to reach the point of judgment?"
|
||||
|
||||
This makes `False Adaptation Rate = 0` an **architectural property, not a tuning
|
||||
target**. The system does not merely try hard to avoid normalizing a defect — it
|
||||
cannot express the operation. That matters because a rate driven near zero by
|
||||
tuning regresses silently; a rate that is zero by construction does not.
|
||||
|
||||
### The decision table
|
||||
|
||||
Classification is a total function over three observed signals. Baseline refers
|
||||
to the last accepted run of the same verification asset.
|
||||
|
||||
| S1 surface differs | S2 postcondition met | S3 verdicts vs baseline | Classification |
|
||||
|---|---|---|---|
|
||||
| yes | yes | unchanged | `MECHANICAL_ADAPTATION` |
|
||||
| no | yes | changed to FAIL | `PRODUCT_DEFECT` |
|
||||
| yes | yes | changed to FAIL | `PRODUCT_DEFECT` |
|
||||
| any | yes | unchanged, but the asset's declared claim set differs from the use case | `SEMANTIC_CHANGE` → human |
|
||||
| any | no, and no legitimate surface affords it | — | `PRODUCT_DEFECT` |
|
||||
| any | no, but the action is expressible and the actor simply failed | — | `FRAMEWORK_LIMITATION` |
|
||||
| any | any | any oracle `INCONCLUSIVE`, or required evidence missing | `AMBIGUOUS` → escalate |
|
||||
|
||||
Two rows carry the whole safety argument:
|
||||
|
||||
- **Row 3** — a surface change occurring *alongside* a verdict change is classified
|
||||
as a defect, never as an adaptation. Coincidence is not exoneration. This is the
|
||||
row that a naive self-healing tool gets wrong, and it is where the seeded M05
|
||||
authorization defect must land even when it ships together with M01/M02.
|
||||
- **Row 7** — missing or contradictory evidence yields `AMBIGUOUS`, never a
|
||||
default-to-pass and never a default-to-adapt. `INCONCLUSIVE` already exists as a
|
||||
first-class verdict; this row is what gives it teeth.
|
||||
|
||||
### Realization legitimacy
|
||||
|
||||
S2 records not only *whether* the postcondition held but *which surface produced
|
||||
it*. A semantic action declares the surfaces it may legitimately use; an actor
|
||||
that achieves `grant_access` by calling the HTTP API when the scenario constrains
|
||||
it to the browser has not realized the action — it has performed an unrequested
|
||||
**surface-substitution**, which is itself one of the catalogued security
|
||||
mutations.
|
||||
|
||||
Without this check, an agent can "recover" from a UI authorization control by
|
||||
routing around it and the run would score as a successful mechanical adaptation.
|
||||
That is the most plausible concrete route to a false adaptation, so it is checked
|
||||
explicitly rather than assumed away.
|
||||
|
||||
### Where model judgment is and is not used
|
||||
|
||||
Concept Model § 2.3 is satisfied, and the boundary is now precise:
|
||||
|
||||
- **Used:** locating a moved control, proposing a path through a changed DOM,
|
||||
summarizing a trajectory. All S1. All independently verified afterwards by S2.
|
||||
- **Not used:** evaluating a claim, evaluating an invariant, producing a verdict,
|
||||
deciding whether an adaptation was legitimate, deciding whether a run passed.
|
||||
|
||||
A model can be wrong about where the button is; the run then fails at S2 and is
|
||||
retried or reported. A model is never in a position to be wrong about whether Bob
|
||||
could still read R.
|
||||
|
||||
### What this costs
|
||||
|
||||
The design requires an out-of-band observer capable of evaluating every semantic
|
||||
action's postcondition and every claim without going through the actor's surface.
|
||||
In the lab this is a direct database or admin-API reader. For a real system under
|
||||
test it is the main integration burden the framework imposes, and it should be
|
||||
stated plainly to any future adopter: **test-driver requires an independent
|
||||
observation channel.** Without one, S2 and S3 collapse into S1 and the safety
|
||||
argument is void.
|
||||
|
||||
This is a real limitation and is recorded as such rather than discovered later.
|
||||
|
||||
---
|
||||
|
||||
## Part B — Intent provenance
|
||||
|
||||
### The question
|
||||
|
||||
If an agent writes the implementation, an agent writes the use case, and an agent
|
||||
realizes the test, then "oracles are independent from actors" is *procedurally*
|
||||
true and *epistemically* empty. Independence of components does not give
|
||||
independence of belief when all components derive from the same source.
|
||||
|
||||
### The rule
|
||||
|
||||
> A claim may only be authored by a source **causally independent of the
|
||||
> implementation it constrains.**
|
||||
|
||||
`UseCase` and `Claim` therefore carry a required `provenance` field:
|
||||
|
||||
| Value | Meaning | Admissible as a claim? |
|
||||
|---|---|---|
|
||||
| `human` | authored by a person | yes |
|
||||
| `spec` | derived from a specification artifact that demonstrably predates the implementation (ref + commit SHA recorded) | yes |
|
||||
| `agent-from-spec` | an agent transcribed a human or spec artifact; a human accepted the result | yes, with the accepting identity recorded |
|
||||
| `agent-from-implementation` | an agent derived it by observing what the system does | **no** |
|
||||
|
||||
`agent-from-implementation` is not forbidden as an activity — it is genuinely
|
||||
useful for T0 exploration. It is forbidden as a **claim**. Such output enters the
|
||||
system as an *exploratory hypothesis*, which can drive runs and accumulate
|
||||
evidence but cannot produce a `FAIL`, cannot enter an invariant, and cannot be
|
||||
crystallized. Promotion to a claim requires an explicit human acceptance event,
|
||||
recorded with identity and timestamp.
|
||||
|
||||
Without this rule, the framework's guarantee reduces to "the implementation
|
||||
agrees with itself", which is exactly the failure the project exists to prevent.
|
||||
|
||||
### Consequences
|
||||
|
||||
- Provenance is carried into every Evidence Pack, so any verdict can later be
|
||||
audited for independence. A `FAIL` whose claim has inadmissible provenance is
|
||||
itself an `EVIDENCE_FAILURE`.
|
||||
- Crystallization preserves provenance. A deterministic test generated from an
|
||||
agentic ancestor inherits the ancestor's claim provenance and cannot launder it.
|
||||
- A fourth self-test joins the M3 set: `td://self/intent-independence` — asserting
|
||||
that no admissible claim in the corpus has implementation-derived provenance.
|
||||
|
||||
### Honest limitation
|
||||
|
||||
This rule constrains *provenance*, not *quality*. A human can write a claim that
|
||||
merely restates what they saw the system do. The rule makes the independence
|
||||
question answerable and auditable; it does not make it automatic. That residual
|
||||
is accepted for the spike and revisited at T10.
|
||||
|
||||
---
|
||||
|
||||
## Decisions recorded
|
||||
|
||||
- **D-01** Evidence is stratified S1/S2/S3; adaptation may write only to S1.
|
||||
- **D-02** Claims and invariants are run inputs; no adaptation path may modify them.
|
||||
- **D-03** Classification is a total function over (S1 differs, S2 met, S3 vs baseline),
|
||||
defaulting to `AMBIGUOUS` on missing evidence.
|
||||
- **D-04** A surface change concurrent with a verdict change classifies as
|
||||
`PRODUCT_DEFECT`, never `MECHANICAL_ADAPTATION`.
|
||||
- **D-05** Semantic actions declare permitted surfaces; realization via another
|
||||
surface is a finding, not a recovery.
|
||||
- **D-06** Claims require causally independent provenance;
|
||||
`agent-from-implementation` output is an exploratory hypothesis until a human
|
||||
promotes it.
|
||||
- **D-07** test-driver requires an independent observation channel into the system
|
||||
under test. Stated as an adoption precondition.
|
||||
|
|
@ -910,46 +910,21 @@ The conceptual components may initially share a process or codebase. Their separ
|
|||
|
||||
# 15. Initial Development Sequence
|
||||
|
||||
## M0 — Deterministic semantic scenario runner
|
||||
> **Superseded.** This section previously carried its own `M0`–`M3` sequence,
|
||||
> which conflicted with the sequences in `INTENT.md` and
|
||||
> `TestDriverInitialMilestones.md`. The canonical milestone sequence is
|
||||
> **`TestDriverInitialMilestones.md` (M0–M10)**. Recorded as framework finding
|
||||
> `F-0001` (CONCEPT_DRIFT), 2026-08-22.
|
||||
|
||||
Implement:
|
||||
The former M0–M3 corresponded to canonical M1, M4, M5–M7 and M9 respectively.
|
||||
See the mapping table in `INTENT.md` § Initial Milestones.
|
||||
|
||||
- use cases,
|
||||
- actors,
|
||||
- worlds,
|
||||
- semantic actions,
|
||||
- observations,
|
||||
- oracles,
|
||||
- verdicts.
|
||||
|
||||
Execute the reference multi-user scenario without AI.
|
||||
|
||||
## M1 — Agentic driver
|
||||
|
||||
Allow an actor agent to realize a semantic action or use-case subgoal through a browser or API surface while preserving actor isolation and evidence capture.
|
||||
|
||||
## M2 — Adaptation and crystallization
|
||||
|
||||
Implement:
|
||||
|
||||
- adaptation classification,
|
||||
- successful-path persistence,
|
||||
- hardening,
|
||||
- generation of deterministic test candidates.
|
||||
|
||||
## M3 — Living verification
|
||||
|
||||
Introduce:
|
||||
|
||||
- energy,
|
||||
- temperature,
|
||||
- lineage,
|
||||
- retirement,
|
||||
- mutation campaigns,
|
||||
- adaptive test selection.
|
||||
Execution order is *not* the canonical order — see
|
||||
`workplans/TD-WP-0002-vertical-spike-crystallization.md`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
# 16. Canonical Concept Set v0.1
|
||||
|
||||
The initial test-driver vocabulary is:
|
||||
|
|
|
|||
|
|
@ -1,8 +1,18 @@
|
|||
# TestDriver Research Prototype — Initial Milestones
|
||||
|
||||
**Status:** v0.1
|
||||
**Status:** v0.1 — **canonical milestone sequence**
|
||||
**Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.
|
||||
|
||||
> This document is the single canonical source for milestone identifiers `M0`–`M10`.
|
||||
> The competing sequences formerly in `INTENT.md` and `TestDriverConceptModel.md` § 15
|
||||
> are superseded — see framework finding
|
||||
> `research/findings/F-0001-milestone-sequence-drift.md`.
|
||||
>
|
||||
> **Canonical order is not execution order.** The milestones define *what must be
|
||||
> true*; `workplans/TD-WP-0002-vertical-spike-crystallization.md` defines *the order
|
||||
> in which we get there*, driving one thin thread through M1–M7 rather than
|
||||
> completing each milestone in turn.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 0 — Research Control Plane
|
||||
|
|
|
|||
48
research/README.md
Normal file
48
research/README.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# Research Control Plane
|
||||
|
||||
Deliberately small. This directory exists so that claims about test-driver can be
|
||||
falsified rather than accumulated. It is plain files — no CLI, no schema, no
|
||||
tooling — until there are enough readings to justify tooling.
|
||||
|
||||
```
|
||||
research/
|
||||
├── hypotheses/ H-NNN — a claim with a falsification condition
|
||||
├── experiments/ E-NNN — a planned or executed test of a hypothesis
|
||||
├── findings/ F-NNNN — findings about test-driver itself
|
||||
├── concepts/ the Concept ↔ Implementation Fitness Map
|
||||
└── decisions/ pointers to decisions recorded in State Hub
|
||||
```
|
||||
|
||||
## Identifier convention
|
||||
|
||||
| Prefix | Scope | Example |
|
||||
|---|---|---|
|
||||
| `H-NNN` | Hypothesis | `H-001` |
|
||||
| `E-NNN` | Experiment | `E-001` |
|
||||
| `F-NNNN` | Framework Finding | `F-0001` |
|
||||
| `C-<slug>` | Concept in the fitness map | `C-actor-isolation` |
|
||||
| `D-NN` | Decision, scoped to its design note | `D-07` |
|
||||
| `TD-WP-NNNN-TNN` | Workplan task (State Hub) | `TD-WP-0002-T04` |
|
||||
|
||||
Identifiers are stable and never reused. A rejected hypothesis keeps its number.
|
||||
|
||||
## Hypothesis lifecycle
|
||||
|
||||
```
|
||||
PROPOSED → EXPERIMENTING → SUPPORTED → PRACTICALLY_VALIDATED → ARCHITECTURAL
|
||||
└──────→ REJECTED
|
||||
```
|
||||
|
||||
A hypothesis may be reopened if later evidence contradicts it. Reopening is
|
||||
recorded in the file, not by creating a new identifier.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Every hypothesis states what would falsify it**, in terms of an observable
|
||||
outcome, before any experiment runs. A hypothesis with no falsification
|
||||
condition is an opinion.
|
||||
2. **Concepts with no supporting evidence are marked as such**, not quietly
|
||||
retained. The fitness map is expected to contain unsupported entries; hiding
|
||||
them defeats its purpose.
|
||||
3. **Subtraction counts as progress.** A rejected hypothesis or a removed
|
||||
abstraction is a result, not a setback.
|
||||
52
research/concepts/fitness-map.md
Normal file
52
research/concepts/fitness-map.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# Concept ↔ Implementation Fitness Map
|
||||
|
||||
**Updated:** 2026-08-22 (TD-WP-0002-T03)
|
||||
|
||||
Traces each important concept to the implementation, experiment and evidence that
|
||||
support it. **Unsupported entries are the point of this map** — a concept with no
|
||||
implementation and no evidence is not a gap to be embarrassed about, it is the
|
||||
current honest state, and hiding it defeats the map's purpose.
|
||||
|
||||
Support levels follow `TestDriverImprovementLoop.md` §13:
|
||||
`C0 Idea` · `C1 Hypothesis` · `C2 Experimentally Supported` ·
|
||||
`C3 Practically Validated` · `C4 Architectural Invariant`
|
||||
|
||||
## Current state
|
||||
|
||||
At the time of writing there is **no implementation**, so no concept can honestly
|
||||
exceed `C1`. The initial classifications in §13 of the Improvement Loop
|
||||
(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they
|
||||
were aspirational, not evidenced.
|
||||
|
||||
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
||||
|---|---|---|---|---|---|
|
||||
| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? |
|
||||
| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. |
|
||||
| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) |
|
||||
| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
||||
| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? |
|
||||
| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
|
||||
| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) |
|
||||
| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. |
|
||||
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
|
||||
| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. |
|
||||
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
|
||||
| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) |
|
||||
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
||||
| `C-confidence` | C0 | — | — | — | Deferred. |
|
||||
| `C-campaign` | C0 | — | — | — | Deferred. |
|
||||
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||
| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. |
|
||||
|
||||
## Orphan check
|
||||
|
||||
**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`:
|
||||
`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`.
|
||||
|
||||
All five are deferred *by explicit decision*, not oversight. They are the group
|
||||
most at risk of being built because they are easy and satisfying, and never
|
||||
validated. They are revisited at T10, where the question is not "when do we build
|
||||
these" but "does the evidence justify keeping them in the model at all".
|
||||
|
||||
**Implementation orphans** — none yet; there is no implementation.
|
||||
9
research/decisions/README.md
Normal file
9
research/decisions/README.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Decisions
|
||||
|
||||
Decisions are recorded in State Hub (the durable store) and indexed here for
|
||||
offline readers. Design rationale lives in the corresponding `docs/` note; this
|
||||
file is a pointer table, not a second source of truth.
|
||||
|
||||
| Ref | Title | Hub ID | Source |
|
||||
|---|---|---|---|
|
||||
| D-01 … D-07 | Stratified evidence and claim provenance as the basis for adaptation safety | `fef5213f-ce9b-44c2-b327-a0b0ba4b6270` | `docs/TestDriverClassificationDesign.md` |
|
||||
38
research/experiments/E-001-mechanical-vs-defect.md
Normal file
38
research/experiments/E-001-mechanical-vs-defect.md
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
id: E-001
|
||||
title: Mechanical recovery and defect discrimination over the labelled mutation set
|
||||
status: PLANNED
|
||||
hypotheses: [H-001, H-002, H-004]
|
||||
task: TD-WP-0002-T08
|
||||
created: "2026-08-22"
|
||||
---
|
||||
|
||||
# E-001 — Mechanical recovery and defect discrimination
|
||||
|
||||
## Design
|
||||
|
||||
Run the same verification asset against every labelled lab mutation (T05),
|
||||
in two arms:
|
||||
|
||||
- **arm A** — semantic action realized by the agentic driver;
|
||||
- **arm B** — a recorded selector-based sequence captured against the baseline,
|
||||
built with the most robust selector strategy reasonably available.
|
||||
|
||||
Arm B is the control for H-001 and must actually be run.
|
||||
|
||||
## Outcomes recorded per run
|
||||
|
||||
surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts
|
||||
(S3) · classification emitted · claim/invariant diff (expected empty) ·
|
||||
tokens · wall time · retries.
|
||||
|
||||
## Decision rules
|
||||
|
||||
- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune.
|
||||
- Arm B recovery ≥ arm A recovery → H-001 falsified.
|
||||
- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a
|
||||
framework defect, since D-02 forbids the write path.
|
||||
|
||||
## Status
|
||||
|
||||
`PLANNED`. Blocked on T05 (lab) and T07 (agentic driver).
|
||||
33
research/experiments/E-002-crystallization-fidelity.md
Normal file
33
research/experiments/E-002-crystallization-fidelity.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
---
|
||||
id: E-002
|
||||
title: Crystallization fidelity and cost
|
||||
status: PLANNED
|
||||
hypotheses: [H-003]
|
||||
task: TD-WP-0002-T09
|
||||
created: "2026-08-22"
|
||||
---
|
||||
|
||||
# E-002 — Crystallization fidelity and cost
|
||||
|
||||
## Design
|
||||
|
||||
Run one agentic realization repeatedly against a stable lab version until
|
||||
realization is stable, crystallize it, then execute the descendant with no model
|
||||
involvement across every lab version the ancestor was run against.
|
||||
|
||||
## Outcomes recorded
|
||||
|
||||
Oracle set of descendant vs ancestor · verdict agreement per lab version ·
|
||||
execution cost ratio (tokens, wall time) · whether human authorship of assertions
|
||||
was required.
|
||||
|
||||
## Decision rules
|
||||
|
||||
- Descendant oracle set is not a superset of the ancestor's → H-003 falsified.
|
||||
- Any verdict disagreement on the same lab version → H-003 falsified.
|
||||
- Cost ratio ≥ 1 → thesis intact, product rationale is not. Record plainly rather
|
||||
than quietly.
|
||||
|
||||
## Status
|
||||
|
||||
`PLANNED`. Blocked on T07.
|
||||
40
research/experiments/E-003-surface-substitution-attack.md
Normal file
40
research/experiments/E-003-surface-substitution-attack.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
id: E-003
|
||||
title: Deliberate attack on the adaptation safety boundary
|
||||
status: PLANNED
|
||||
hypotheses: [H-004]
|
||||
task: TD-WP-0002-T08
|
||||
created: "2026-08-22"
|
||||
---
|
||||
|
||||
# E-003 — Deliberate attack on the adaptation safety boundary
|
||||
|
||||
## Why this experiment exists separately
|
||||
|
||||
E-001 measures whether the framework *happens to* stay safe over a labelled set.
|
||||
E-003 tries to *make it unsafe*. The distinction matters: an experiment that only
|
||||
samples benign cases cannot establish a safety property, and False Adaptation Rate
|
||||
is the one metric the project cannot afford to be wrong about.
|
||||
|
||||
## Attack cases
|
||||
|
||||
1. **Surface substitution** — remove the sharing control from the UI while leaving
|
||||
the HTTP endpoint open. A driver that "recovers" by calling the API has routed
|
||||
around the control. Must classify as a finding, never as recovery (D-05).
|
||||
2. **Concurrent change** — ship a mechanical mutation (M01) and the authorization
|
||||
defect (M05) in the same lab version. Must classify as `PRODUCT_DEFECT`;
|
||||
coincidence must not exonerate (decision table row 3).
|
||||
3. **Evidence starvation** — disable the independent observation channel mid-run.
|
||||
Must yield `AMBIGUOUS`, never a pass and never an adaptation (row 7).
|
||||
4. **Provenance laundering** — attempt to introduce a claim derived from observed
|
||||
implementation behaviour and crystallize it. Must be rejected at authoring
|
||||
time (D-06) and, if it slips through, at crystallization.
|
||||
|
||||
## Decision rule
|
||||
|
||||
Any attack that succeeds is a stop-and-redesign signal for H-004, regardless of
|
||||
what E-001 reports.
|
||||
|
||||
## Status
|
||||
|
||||
`PLANNED`. Blocked on T05, T07, T08.
|
||||
77
research/findings/F-0001-milestone-sequence-drift.md
Normal file
77
research/findings/F-0001-milestone-sequence-drift.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
id: F-0001
|
||||
type: framework-finding
|
||||
class: CONCEPT_DRIFT
|
||||
status: resolved
|
||||
discovered: "2026-08-22"
|
||||
resolved: "2026-08-22"
|
||||
discovered_by: concept-assessment
|
||||
source: history/2026-08-22-concept-assessment-swot.md
|
||||
workplan: TD-WP-0002
|
||||
task: TD-WP-0002-T01
|
||||
---
|
||||
|
||||
# F-0001 — Three conflicting milestone sequences
|
||||
|
||||
## Class
|
||||
|
||||
`CONCEPT_DRIFT` — the repository carried three mutually inconsistent statements
|
||||
of the same concept, with no explicit decision revising any of them.
|
||||
|
||||
Note the unusual shape: §5 of the Improvement Loop defines concept drift as
|
||||
*implementation* diverging from concept. Here there was no implementation at all.
|
||||
The drift was **document-to-document**, which the taxonomy does not currently
|
||||
name. See "Residual" below.
|
||||
|
||||
## Observation
|
||||
|
||||
Three documents each defined a milestone sequence for the same project:
|
||||
|
||||
| Document | Sequence |
|
||||
|---|---|
|
||||
| `INTENT.md` § Initial Milestones | M0–M3 |
|
||||
| `docs/TestDriverConceptModel.md` § 15 | M0–M3 (different content) |
|
||||
| `docs/TestDriverInitialMilestones.md` | M0–M10 |
|
||||
|
||||
The two M0–M3 sequences disagreed with each other and both disagreed with the
|
||||
M0–M10 sequence. `M2`, for instance, denoted "Adaptation and Crystallization" in
|
||||
one document and "Test-Driver Lab" in another — so an unqualified reference to
|
||||
"M2" was ambiguous across the corpus.
|
||||
|
||||
## Why it matters
|
||||
|
||||
The project's own thesis is that implementations must not silently redefine
|
||||
intended behaviour, and that concept drift is a first-class finding. The corpus
|
||||
exhibited the failure it was written to prevent, before a single line of code
|
||||
existed. Left unresolved, every subsequent milestone reference in commits,
|
||||
workplans and evidence would have been ambiguous.
|
||||
|
||||
## Resolution
|
||||
|
||||
Path 2 of §5 — *the concept is deliberately revised*.
|
||||
|
||||
`docs/TestDriverInitialMilestones.md` (M0–M10) is now **canonical**. It is the
|
||||
most complete of the three, is the only one carrying exit criteria and success
|
||||
metrics, and is the only one that names the lab, self-verification, the
|
||||
improvement loop and the compression review.
|
||||
|
||||
The other two sections were replaced in place with a superseded banner and a
|
||||
mapping table rather than deleted, so that references from earlier commits and
|
||||
from `history/` remain interpretable.
|
||||
|
||||
Execution order is deliberately *not* the canonical order:
|
||||
`workplans/TD-WP-0002-vertical-spike-crystallization.md` drives one thin thread
|
||||
through canonical M1–M7 rather than completing each milestone in turn. Canonical
|
||||
sequence and execution sequence are now distinct and separately stated.
|
||||
|
||||
## Residual
|
||||
|
||||
The finding taxonomy (§9) has no class for divergence **between concept
|
||||
documents** — `CONCEPT_DRIFT` is defined as implementation-vs-concept. This
|
||||
finding was classified `CONCEPT_DRIFT` as the nearest fit.
|
||||
|
||||
Carry into `TD-WP-0002-T10` (compression review): either widen `CONCEPT_DRIFT`
|
||||
to cover intra-conceptual inconsistency, or accept that pre-implementation
|
||||
document drift is not a framework finding at all and should be handled as
|
||||
ordinary editorial hygiene. Do not add a new finding class before that question
|
||||
is answered — the taxonomy is already larger than the evidence supporting it.
|
||||
19
research/findings/README.md
Normal file
19
research/findings/README.md
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# Framework Findings
|
||||
|
||||
Findings about **test-driver itself**, kept strictly separate from findings about
|
||||
any system under test. Classes are defined in
|
||||
`docs/TestDriverImprovementLoop.md` § 9:
|
||||
|
||||
`PRODUCT_DEFECT` · `TEST_DEFECT` · `MECHANICAL_ADAPTATION` · `SEMANTIC_CHANGE` ·
|
||||
`CONCEPT_DRIFT` · `FRAMEWORK_LIMITATION` · `EVIDENCE_FAILURE` ·
|
||||
`UNNECESSARY_COMPLEXITY`
|
||||
|
||||
File naming: `F-NNNN-<slug>.md`. Identifiers are stable and never reused.
|
||||
|
||||
Each finding carries frontmatter (`id`, `class`, `status`, dates, originating
|
||||
workplan/task) and, when resolved, states which of the three §5 resolution paths
|
||||
was taken. A finding that leaves an open question records it under "Residual" and
|
||||
names the task that will answer it — residuals are not parked in prose.
|
||||
|
||||
The wider research control plane (`hypotheses/`, `experiments/`, `concepts/`,
|
||||
`decisions/`) is built in `TD-WP-0002-T03`.
|
||||
41
research/hypotheses/H-001-semantic-action-stability.md
Normal file
41
research/hypotheses/H-001-semantic-action-stability.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
id: H-001
|
||||
title: Semantic Action Stability
|
||||
status: PROPOSED
|
||||
created: "2026-08-22"
|
||||
experiments: [E-001]
|
||||
concepts: [C-semantic-action]
|
||||
---
|
||||
|
||||
# H-001 — Semantic Action Stability
|
||||
|
||||
## Claim
|
||||
|
||||
A semantic action survives implementation restructuring better than a recorded UI
|
||||
interaction sequence.
|
||||
|
||||
## Falsification condition
|
||||
|
||||
Across the labelled mechanical mutations in the lab (T05), a recorded interaction
|
||||
sequence survives **at least as many** mutations as the semantic action does.
|
||||
|
||||
If mechanics-free identity buys no measurable durability, the central abstraction
|
||||
is decorative and `SemanticAction` should be reduced to a naming convention.
|
||||
|
||||
## Measurement
|
||||
|
||||
Mechanical Recovery Rate for each of two arms over the same mutation set:
|
||||
- **arm A** — semantic action realized by an agentic driver;
|
||||
- **arm B** — a recorded selector-based sequence captured against the baseline.
|
||||
|
||||
Arm B is a genuine control and must be run, not assumed to fail.
|
||||
|
||||
## Threats to validity
|
||||
|
||||
The comparison is unfair if arm B is built naively — a brittle straw man makes
|
||||
H-001 trivially true and worthless. Arm B uses the most robust selector strategy
|
||||
reasonably available (roles, labels, test ids where the lab provides them).
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`. No evidence.
|
||||
40
research/hypotheses/H-002-mechanical-adaptation.md
Normal file
40
research/hypotheses/H-002-mechanical-adaptation.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
id: H-002
|
||||
title: Mechanical Adaptation
|
||||
status: PROPOSED
|
||||
created: "2026-08-22"
|
||||
experiments: [E-001]
|
||||
concepts: [C-adaptation]
|
||||
---
|
||||
|
||||
# H-002 — Mechanical Adaptation
|
||||
|
||||
## Claim
|
||||
|
||||
An agentic driver can recover from a mechanical implementation change without
|
||||
modifying the semantics of the protected use case.
|
||||
|
||||
## Falsification condition
|
||||
|
||||
Either:
|
||||
- the driver fails to recover from a majority of labelled `MECHANICAL` mutations
|
||||
(recovery is not achievable in practice); **or**
|
||||
- recovery is achieved only in runs where claims or invariants were altered
|
||||
(recovery is achievable but not semantics-preserving).
|
||||
|
||||
The second branch matters more than the first. A high recovery rate purchased by
|
||||
weakening assertions falsifies this hypothesis just as decisively as no recovery
|
||||
at all.
|
||||
|
||||
## Measurement
|
||||
|
||||
- Mechanical Recovery Rate over `MECHANICAL`-labelled mutations.
|
||||
- Claim/invariant diff per run — expected to be empty by construction
|
||||
(`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a
|
||||
falsification signal **and** a framework defect, since no write path should exist.
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification
|
||||
branch structurally unreachable; the measurement is retained anyway, as an
|
||||
assertion that the architecture is what we believe it is.
|
||||
43
research/hypotheses/H-003-crystallization.md
Normal file
43
research/hypotheses/H-003-crystallization.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
id: H-003
|
||||
title: Crystallization
|
||||
status: PROPOSED
|
||||
created: "2026-08-22"
|
||||
experiments: [E-002]
|
||||
concepts: [C-crystallization]
|
||||
---
|
||||
|
||||
# H-003 — Crystallization
|
||||
|
||||
## Claim
|
||||
|
||||
A sufficiently stable agentic execution can be converted into deterministic test
|
||||
code without losing relevant oracle coverage.
|
||||
|
||||
## Falsification condition
|
||||
|
||||
Any of:
|
||||
- the generated deterministic test cannot be produced without human authorship of
|
||||
the assertions;
|
||||
- it drops one or more oracles present in the agentic ancestor;
|
||||
- it passes where the ancestor would have failed, or vice versa, on the same lab
|
||||
version;
|
||||
- it costs no less to execute than the agentic ancestor.
|
||||
|
||||
## Measurement
|
||||
|
||||
- Crystallization success rate.
|
||||
- Oracle set of descendant vs ancestor — must be a superset or equal.
|
||||
- Verdict agreement between ancestor and descendant across lab versions.
|
||||
- Execution cost ratio (tokens, wall time) descendant : ancestor.
|
||||
|
||||
## Note on the cost criterion
|
||||
|
||||
The cost criterion is included deliberately. If crystallization preserves
|
||||
semantics but saves nothing, the thesis is intact but the product rationale is
|
||||
not — and that distinction should be visible in the record rather than discovered
|
||||
by a user.
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`. No evidence.
|
||||
46
research/hypotheses/H-004-independent-judgment.md
Normal file
46
research/hypotheses/H-004-independent-judgment.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
id: H-004
|
||||
title: Independent Judgment
|
||||
status: PROPOSED
|
||||
created: "2026-08-22"
|
||||
experiments: [E-001, E-003]
|
||||
concepts: [C-oracle-independence, C-intent-provenance]
|
||||
---
|
||||
|
||||
# H-004 — Independent Judgment
|
||||
|
||||
## Claim
|
||||
|
||||
Separating actor execution from deterministic oracles reduces false-positive
|
||||
adaptation to defective behaviour.
|
||||
|
||||
## Falsification condition
|
||||
|
||||
The framework classifies any seeded `DEFECT`-labelled mutation as
|
||||
`MECHANICAL_ADAPTATION` — i.e. **False Adaptation Rate > 0** over the labelled set.
|
||||
|
||||
This is the project's existential safety metric. A non-zero result is a
|
||||
stop-and-redesign signal, not a tuning exercise.
|
||||
|
||||
## Measurement
|
||||
|
||||
- False Adaptation Rate over `DEFECT`-labelled mutations. Target: exactly 0.
|
||||
- Classification precision/recall over the full labelled set.
|
||||
- Rate of `AMBIGUOUS` escalation — tracked as a cost, not as a failure. A system
|
||||
that escalates everything is safe and useless.
|
||||
|
||||
## Relationship to the design
|
||||
|
||||
`docs/TestDriverClassificationDesign.md` argues that FAR = 0 follows
|
||||
architecturally from D-02 (claims are run inputs with no adaptation write path)
|
||||
rather than from classifier accuracy. This hypothesis therefore tests the
|
||||
**architecture**, not a model.
|
||||
|
||||
The most plausible route to falsification is not a misclassification but
|
||||
**surface-substitution**: an actor routing around a broken authorization control
|
||||
via another surface and scoring as a successful recovery. D-05 exists to close
|
||||
that route; E-003 exists to attack it deliberately.
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`. No evidence.
|
||||
40
research/hypotheses/H-005-verification-energy.md
Normal file
40
research/hypotheses/H-005-verification-energy.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
id: H-005
|
||||
title: Verification Energy
|
||||
status: PROPOSED
|
||||
created: "2026-08-22"
|
||||
experiments: []
|
||||
concepts: [C-energy]
|
||||
---
|
||||
|
||||
# H-005 — Verification Energy
|
||||
|
||||
## Claim
|
||||
|
||||
Historical evidence about defects caught, adaptations required, false positives
|
||||
and duplication can identify verification assets whose continued execution is
|
||||
more valuable than others.
|
||||
|
||||
## Falsification condition
|
||||
|
||||
Energy scores computed from event history fail to rank assets better than a
|
||||
trivial baseline (e.g. recency, or uniform priority) when predicting which assets
|
||||
subsequently catch a defect.
|
||||
|
||||
## Why no experiment is planned yet
|
||||
|
||||
**This hypothesis is not testable at the current scale and is deliberately
|
||||
dormant.** Validating it requires event history across many assets over months —
|
||||
history the spike will not accumulate. Implementing a scoring function now would
|
||||
produce a number that cannot be checked, which is worse than no number.
|
||||
|
||||
`TD-WP-0002` therefore records **raw immutable `EnergyEvent`s from the first run**
|
||||
and implements no scoring, decay, or selection logic. Events cannot be
|
||||
reconstructed later; scores can always be computed later.
|
||||
|
||||
This is the hypothesis most likely to be **cheaply built and never validated**,
|
||||
which is precisely why it is fenced off.
|
||||
|
||||
## Status log
|
||||
|
||||
- 2026-08-22 `PROPOSED`, dormant by decision. Event capture only.
|
||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Vertical spike: falsify the crystallization thesis"
|
||||
domain: infotech
|
||||
repo: test-driver
|
||||
status: proposed
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-08-22"
|
||||
|
|
@ -69,6 +69,9 @@ satisfying to build, and impossible to validate at this stage:
|
|||
One process, one database, one browser engine, one application under test.
|
||||
- Novelty belongs in the verification model, never in the infrastructure.
|
||||
- No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
|
||||
- Evidence stratification S1/S2/S3 and claim provenance per
|
||||
`docs/TestDriverClassificationDesign.md` (decision
|
||||
`fef5213f-ce9b-44c2-b327-a0b0ba4b6270`). Adaptation writes only to S1.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -76,7 +79,7 @@ satisfying to build, and impossible to validate at this stage:
|
|||
|
||||
```task
|
||||
id: TD-WP-0002-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "b1755e17-279a-56cf-86eb-359e9739070a"
|
||||
```
|
||||
|
|
@ -91,11 +94,17 @@ Pick one canonical sequence, mark the others superseded in place, and commit
|
|||
first entry in the framework findings log — the project should catch this on
|
||||
itself before it catches anything else.
|
||||
|
||||
**Done 2026-08-22.** `docs/TestDriverInitialMilestones.md` (M0–M10) is canonical;
|
||||
the `INTENT.md` and Concept Model § 15 sequences carry superseded banners with a
|
||||
mapping table. Recorded as `research/findings/F-0001-milestone-sequence-drift.md`.
|
||||
`INTENT.md` and `docs/` are now tracked. One residual carried to T10: the finding
|
||||
taxonomy has no class for document-to-document drift.
|
||||
|
||||
## Decide intent provenance and the classification mechanism on paper
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "e5776cf5-0475-5dc6-90c3-db772ce24d63"
|
||||
```
|
||||
|
|
@ -116,6 +125,23 @@ survives in each case.
|
|||
|
||||
Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision.
|
||||
|
||||
**Done 2026-08-22.** Design note written; decision
|
||||
`fef5213f-ce9b-44c2-b327-a0b0ba4b6270` recorded in the hub.
|
||||
|
||||
Outcome in one line: **adaptation may write only to surface evidence (S1); claims
|
||||
and invariants are run inputs with no adaptation write path.** False Adaptation
|
||||
Rate = 0 therefore becomes an architectural property rather than a tuning target —
|
||||
the system cannot express "accept a defect as an adaptation". Classification is a
|
||||
total function over three signals defaulting to `AMBIGUOUS`; a surface change
|
||||
concurrent with a verdict change classifies as `PRODUCT_DEFECT`, never as
|
||||
adaptation. Claims require causally independent provenance.
|
||||
|
||||
Two consequences that change later tasks:
|
||||
- **D-07** — an independent observation channel into the system under test is now
|
||||
an explicit adoption precondition, not an implementation detail. T05 must expose
|
||||
one in the lab; without it S2/S3 collapse into S1.
|
||||
- A fourth self-test, `td://self/intent-independence`, joins the T06 set.
|
||||
|
||||
## Minimal research control plane
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue