T03: research control plane

H-001..H-005 with explicit falsification conditions, E-001..E-003, the
Concept-Implementation Fitness Map, findings log and ID convention. Plain
files, no tooling.

- fitness map corrects Improvement Loop section 13 levels downward: nothing
  exceeds C1 without implementation
- H-005 (Energy) dormant by decision - events captured, no scoring written
- H-001 gets a genuine control arm so the semantic-action thesis is not
  trivially true
- E-003 added: deliberately attacks the safety boundary rather than only
  sampling benign cases

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-22 23:11:21 +02:00
parent 15b55c807a
commit bec7fdc465
11 changed files with 430 additions and 0 deletions

48
research/README.md Normal file
View file

@ -0,0 +1,48 @@
# Research Control Plane
Deliberately small. This directory exists so that claims about test-driver can be
falsified rather than accumulated. It is plain files — no CLI, no schema, no
tooling — until there are enough readings to justify tooling.
```
research/
├── hypotheses/ H-NNN — a claim with a falsification condition
├── experiments/ E-NNN — a planned or executed test of a hypothesis
├── findings/ F-NNNN — findings about test-driver itself
├── concepts/ the Concept ↔ Implementation Fitness Map
└── decisions/ pointers to decisions recorded in State Hub
```
## Identifier convention
| Prefix | Scope | Example |
|---|---|---|
| `H-NNN` | Hypothesis | `H-001` |
| `E-NNN` | Experiment | `E-001` |
| `F-NNNN` | Framework Finding | `F-0001` |
| `C-<slug>` | Concept in the fitness map | `C-actor-isolation` |
| `D-NN` | Decision, scoped to its design note | `D-07` |
| `TD-WP-NNNN-TNN` | Workplan task (State Hub) | `TD-WP-0002-T04` |
Identifiers are stable and never reused. A rejected hypothesis keeps its number.
## Hypothesis lifecycle
```
PROPOSED → EXPERIMENTING → SUPPORTED → PRACTICALLY_VALIDATED → ARCHITECTURAL
└──────→ REJECTED
```
A hypothesis may be reopened if later evidence contradicts it. Reopening is
recorded in the file, not by creating a new identifier.
## Rules
1. **Every hypothesis states what would falsify it**, in terms of an observable
outcome, before any experiment runs. A hypothesis with no falsification
condition is an opinion.
2. **Concepts with no supporting evidence are marked as such**, not quietly
retained. The fitness map is expected to contain unsupported entries; hiding
them defeats its purpose.
3. **Subtraction counts as progress.** A rejected hypothesis or a removed
abstraction is a result, not a setback.

View file

@ -0,0 +1,52 @@
# Concept ↔ Implementation Fitness Map
**Updated:** 2026-08-22 (TD-WP-0002-T03)
Traces each important concept to the implementation, experiment and evidence that
support it. **Unsupported entries are the point of this map** — a concept with no
implementation and no evidence is not a gap to be embarrassed about, it is the
current honest state, and hiding it defeats the map's purpose.
Support levels follow `TestDriverImprovementLoop.md` §13:
`C0 Idea` · `C1 Hypothesis` · `C2 Experimentally Supported` ·
`C3 Practically Validated` · `C4 Architectural Invariant`
## Current state
At the time of writing there is **no implementation**, so no concept can honestly
exceed `C1`. The initial classifications in §13 of the Improvement Loop
(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they
were aspirational, not evidenced.
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|---|---|---|---|---|---|
| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? |
| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. |
| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) |
| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? |
| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) |
| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. |
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. |
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) |
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
| `C-confidence` | C0 | — | — | — | Deferred. |
| `C-campaign` | C0 | — | — | — | Deferred. |
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. |
## Orphan check
**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`:
`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`.
All five are deferred *by explicit decision*, not oversight. They are the group
most at risk of being built because they are easy and satisfying, and never
validated. They are revisited at T10, where the question is not "when do we build
these" but "does the evidence justify keeping them in the model at all".
**Implementation orphans** — none yet; there is no implementation.

View file

@ -0,0 +1,9 @@
# Decisions
Decisions are recorded in State Hub (the durable store) and indexed here for
offline readers. Design rationale lives in the corresponding `docs/` note; this
file is a pointer table, not a second source of truth.
| Ref | Title | Hub ID | Source |
|---|---|---|---|
| D-01 … D-07 | Stratified evidence and claim provenance as the basis for adaptation safety | `fef5213f-ce9b-44c2-b327-a0b0ba4b6270` | `docs/TestDriverClassificationDesign.md` |

View file

@ -0,0 +1,38 @@
---
id: E-001
title: Mechanical recovery and defect discrimination over the labelled mutation set
status: PLANNED
hypotheses: [H-001, H-002, H-004]
task: TD-WP-0002-T08
created: "2026-08-22"
---
# E-001 — Mechanical recovery and defect discrimination
## Design
Run the same verification asset against every labelled lab mutation (T05),
in two arms:
- **arm A** — semantic action realized by the agentic driver;
- **arm B** — a recorded selector-based sequence captured against the baseline,
built with the most robust selector strategy reasonably available.
Arm B is the control for H-001 and must actually be run.
## Outcomes recorded per run
surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts
(S3) · classification emitted · claim/invariant diff (expected empty) ·
tokens · wall time · retries.
## Decision rules
- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune.
- Arm B recovery ≥ arm A recovery → H-001 falsified.
- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a
framework defect, since D-02 forbids the write path.
## Status
`PLANNED`. Blocked on T05 (lab) and T07 (agentic driver).

View file

@ -0,0 +1,33 @@
---
id: E-002
title: Crystallization fidelity and cost
status: PLANNED
hypotheses: [H-003]
task: TD-WP-0002-T09
created: "2026-08-22"
---
# E-002 — Crystallization fidelity and cost
## Design
Run one agentic realization repeatedly against a stable lab version until
realization is stable, crystallize it, then execute the descendant with no model
involvement across every lab version the ancestor was run against.
## Outcomes recorded
Oracle set of descendant vs ancestor · verdict agreement per lab version ·
execution cost ratio (tokens, wall time) · whether human authorship of assertions
was required.
## Decision rules
- Descendant oracle set is not a superset of the ancestor's → H-003 falsified.
- Any verdict disagreement on the same lab version → H-003 falsified.
- Cost ratio ≥ 1 → thesis intact, product rationale is not. Record plainly rather
than quietly.
## Status
`PLANNED`. Blocked on T07.

View file

@ -0,0 +1,40 @@
---
id: E-003
title: Deliberate attack on the adaptation safety boundary
status: PLANNED
hypotheses: [H-004]
task: TD-WP-0002-T08
created: "2026-08-22"
---
# E-003 — Deliberate attack on the adaptation safety boundary
## Why this experiment exists separately
E-001 measures whether the framework *happens to* stay safe over a labelled set.
E-003 tries to *make it unsafe*. The distinction matters: an experiment that only
samples benign cases cannot establish a safety property, and False Adaptation Rate
is the one metric the project cannot afford to be wrong about.
## Attack cases
1. **Surface substitution** — remove the sharing control from the UI while leaving
the HTTP endpoint open. A driver that "recovers" by calling the API has routed
around the control. Must classify as a finding, never as recovery (D-05).
2. **Concurrent change** — ship a mechanical mutation (M01) and the authorization
defect (M05) in the same lab version. Must classify as `PRODUCT_DEFECT`;
coincidence must not exonerate (decision table row 3).
3. **Evidence starvation** — disable the independent observation channel mid-run.
Must yield `AMBIGUOUS`, never a pass and never an adaptation (row 7).
4. **Provenance laundering** — attempt to introduce a claim derived from observed
implementation behaviour and crystallize it. Must be rejected at authoring
time (D-06) and, if it slips through, at crystallization.
## Decision rule
Any attack that succeeds is a stop-and-redesign signal for H-004, regardless of
what E-001 reports.
## Status
`PLANNED`. Blocked on T05, T07, T08.

View file

@ -0,0 +1,41 @@
---
id: H-001
title: Semantic Action Stability
status: PROPOSED
created: "2026-08-22"
experiments: [E-001]
concepts: [C-semantic-action]
---
# H-001 — Semantic Action Stability
## Claim
A semantic action survives implementation restructuring better than a recorded UI
interaction sequence.
## Falsification condition
Across the labelled mechanical mutations in the lab (T05), a recorded interaction
sequence survives **at least as many** mutations as the semantic action does.
If mechanics-free identity buys no measurable durability, the central abstraction
is decorative and `SemanticAction` should be reduced to a naming convention.
## Measurement
Mechanical Recovery Rate for each of two arms over the same mutation set:
- **arm A** — semantic action realized by an agentic driver;
- **arm B** — a recorded selector-based sequence captured against the baseline.
Arm B is a genuine control and must be run, not assumed to fail.
## Threats to validity
The comparison is unfair if arm B is built naively — a brittle straw man makes
H-001 trivially true and worthless. Arm B uses the most robust selector strategy
reasonably available (roles, labels, test ids where the lab provides them).
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,40 @@
---
id: H-002
title: Mechanical Adaptation
status: PROPOSED
created: "2026-08-22"
experiments: [E-001]
concepts: [C-adaptation]
---
# H-002 — Mechanical Adaptation
## Claim
An agentic driver can recover from a mechanical implementation change without
modifying the semantics of the protected use case.
## Falsification condition
Either:
- the driver fails to recover from a majority of labelled `MECHANICAL` mutations
(recovery is not achievable in practice); **or**
- recovery is achieved only in runs where claims or invariants were altered
(recovery is achievable but not semantics-preserving).
The second branch matters more than the first. A high recovery rate purchased by
weakening assertions falsifies this hypothesis just as decisively as no recovery
at all.
## Measurement
- Mechanical Recovery Rate over `MECHANICAL`-labelled mutations.
- Claim/invariant diff per run — expected to be empty by construction
(`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a
falsification signal **and** a framework defect, since no write path should exist.
## Status log
- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification
branch structurally unreachable; the measurement is retained anyway, as an
assertion that the architecture is what we believe it is.

View file

@ -0,0 +1,43 @@
---
id: H-003
title: Crystallization
status: PROPOSED
created: "2026-08-22"
experiments: [E-002]
concepts: [C-crystallization]
---
# H-003 — Crystallization
## Claim
A sufficiently stable agentic execution can be converted into deterministic test
code without losing relevant oracle coverage.
## Falsification condition
Any of:
- the generated deterministic test cannot be produced without human authorship of
the assertions;
- it drops one or more oracles present in the agentic ancestor;
- it passes where the ancestor would have failed, or vice versa, on the same lab
version;
- it costs no less to execute than the agentic ancestor.
## Measurement
- Crystallization success rate.
- Oracle set of descendant vs ancestor — must be a superset or equal.
- Verdict agreement between ancestor and descendant across lab versions.
- Execution cost ratio (tokens, wall time) descendant : ancestor.
## Note on the cost criterion
The cost criterion is included deliberately. If crystallization preserves
semantics but saves nothing, the thesis is intact but the product rationale is
not — and that distinction should be visible in the record rather than discovered
by a user.
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,46 @@
---
id: H-004
title: Independent Judgment
status: PROPOSED
created: "2026-08-22"
experiments: [E-001, E-003]
concepts: [C-oracle-independence, C-intent-provenance]
---
# H-004 — Independent Judgment
## Claim
Separating actor execution from deterministic oracles reduces false-positive
adaptation to defective behaviour.
## Falsification condition
The framework classifies any seeded `DEFECT`-labelled mutation as
`MECHANICAL_ADAPTATION` — i.e. **False Adaptation Rate > 0** over the labelled set.
This is the project's existential safety metric. A non-zero result is a
stop-and-redesign signal, not a tuning exercise.
## Measurement
- False Adaptation Rate over `DEFECT`-labelled mutations. Target: exactly 0.
- Classification precision/recall over the full labelled set.
- Rate of `AMBIGUOUS` escalation — tracked as a cost, not as a failure. A system
that escalates everything is safe and useless.
## Relationship to the design
`docs/TestDriverClassificationDesign.md` argues that FAR = 0 follows
architecturally from D-02 (claims are run inputs with no adaptation write path)
rather than from classifier accuracy. This hypothesis therefore tests the
**architecture**, not a model.
The most plausible route to falsification is not a misclassification but
**surface-substitution**: an actor routing around a broken authorization control
via another surface and scoring as a successful recovery. D-05 exists to close
that route; E-003 exists to attack it deliberately.
## Status log
- 2026-08-22 `PROPOSED`. No evidence.

View file

@ -0,0 +1,40 @@
---
id: H-005
title: Verification Energy
status: PROPOSED
created: "2026-08-22"
experiments: []
concepts: [C-energy]
---
# H-005 — Verification Energy
## Claim
Historical evidence about defects caught, adaptations required, false positives
and duplication can identify verification assets whose continued execution is
more valuable than others.
## Falsification condition
Energy scores computed from event history fail to rank assets better than a
trivial baseline (e.g. recency, or uniform priority) when predicting which assets
subsequently catch a defect.
## Why no experiment is planned yet
**This hypothesis is not testable at the current scale and is deliberately
dormant.** Validating it requires event history across many assets over months —
history the spike will not accumulate. Implementing a scoring function now would
produce a number that cannot be checked, which is worse than no number.
`TD-WP-0002` therefore records **raw immutable `EnergyEvent`s from the first run**
and implements no scoring, decay, or selection logic. Events cannot be
reconstructed later; scores can always be computed later.
This is the hypothesis most likely to be **cheaply built and never validated**,
which is precisely why it is fenced off.
## Status log
- 2026-08-22 `PROPOSED`, dormant by decision. Event capture only.