diff --git a/research/README.md b/research/README.md new file mode 100644 index 0000000..42b3e84 --- /dev/null +++ b/research/README.md @@ -0,0 +1,48 @@ +# Research Control Plane + +Deliberately small. This directory exists so that claims about test-driver can be +falsified rather than accumulated. It is plain files — no CLI, no schema, no +tooling — until there are enough readings to justify tooling. + +``` +research/ +├── hypotheses/ H-NNN — a claim with a falsification condition +├── experiments/ E-NNN — a planned or executed test of a hypothesis +├── findings/ F-NNNN — findings about test-driver itself +├── concepts/ the Concept ↔ Implementation Fitness Map +└── decisions/ pointers to decisions recorded in State Hub +``` + +## Identifier convention + +| Prefix | Scope | Example | +|---|---|---| +| `H-NNN` | Hypothesis | `H-001` | +| `E-NNN` | Experiment | `E-001` | +| `F-NNNN` | Framework Finding | `F-0001` | +| `C-` | Concept in the fitness map | `C-actor-isolation` | +| `D-NN` | Decision, scoped to its design note | `D-07` | +| `TD-WP-NNNN-TNN` | Workplan task (State Hub) | `TD-WP-0002-T04` | + +Identifiers are stable and never reused. A rejected hypothesis keeps its number. + +## Hypothesis lifecycle + +``` +PROPOSED → EXPERIMENTING → SUPPORTED → PRACTICALLY_VALIDATED → ARCHITECTURAL + └──────→ REJECTED +``` + +A hypothesis may be reopened if later evidence contradicts it. Reopening is +recorded in the file, not by creating a new identifier. + +## Rules + +1. **Every hypothesis states what would falsify it**, in terms of an observable + outcome, before any experiment runs. A hypothesis with no falsification + condition is an opinion. +2. **Concepts with no supporting evidence are marked as such**, not quietly + retained. The fitness map is expected to contain unsupported entries; hiding + them defeats its purpose. +3. **Subtraction counts as progress.** A rejected hypothesis or a removed + abstraction is a result, not a setback. diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md new file mode 100644 index 0000000..f6c7630 --- /dev/null +++ b/research/concepts/fitness-map.md @@ -0,0 +1,52 @@ +# Concept ↔ Implementation Fitness Map + +**Updated:** 2026-08-22 (TD-WP-0002-T03) + +Traces each important concept to the implementation, experiment and evidence that +support it. **Unsupported entries are the point of this map** — a concept with no +implementation and no evidence is not a gap to be embarrassed about, it is the +current honest state, and hiding it defeats the map's purpose. + +Support levels follow `TestDriverImprovementLoop.md` §13: +`C0 Idea` · `C1 Hypothesis` · `C2 Experimentally Supported` · +`C3 Practically Validated` · `C4 Architectural Invariant` + +## Current state + +At the time of writing there is **no implementation**, so no concept can honestly +exceed `C1`. The initial classifications in §13 of the Improvement Loop +(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they +were aspirational, not evidenced. + +| Concept | Level | Implementation | Experiment | Evidence | Open question | +|---|---|---|---|---|---| +| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? | +| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. | +| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) | +| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | +| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? | +| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. | +| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) | +| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. | +| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) | +| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. | +| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. | +| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) | +| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | +| `C-confidence` | C0 | — | — | — | Deferred. | +| `C-campaign` | C0 | — | — | — | Deferred. | +| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. | +| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. | +| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. | + +## Orphan check + +**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`: +`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`. + +All five are deferred *by explicit decision*, not oversight. They are the group +most at risk of being built because they are easy and satisfying, and never +validated. They are revisited at T10, where the question is not "when do we build +these" but "does the evidence justify keeping them in the model at all". + +**Implementation orphans** — none yet; there is no implementation. diff --git a/research/decisions/README.md b/research/decisions/README.md new file mode 100644 index 0000000..6762916 --- /dev/null +++ b/research/decisions/README.md @@ -0,0 +1,9 @@ +# Decisions + +Decisions are recorded in State Hub (the durable store) and indexed here for +offline readers. Design rationale lives in the corresponding `docs/` note; this +file is a pointer table, not a second source of truth. + +| Ref | Title | Hub ID | Source | +|---|---|---|---| +| D-01 … D-07 | Stratified evidence and claim provenance as the basis for adaptation safety | `fef5213f-ce9b-44c2-b327-a0b0ba4b6270` | `docs/TestDriverClassificationDesign.md` | diff --git a/research/experiments/E-001-mechanical-vs-defect.md b/research/experiments/E-001-mechanical-vs-defect.md new file mode 100644 index 0000000..3360d1c --- /dev/null +++ b/research/experiments/E-001-mechanical-vs-defect.md @@ -0,0 +1,38 @@ +--- +id: E-001 +title: Mechanical recovery and defect discrimination over the labelled mutation set +status: PLANNED +hypotheses: [H-001, H-002, H-004] +task: TD-WP-0002-T08 +created: "2026-08-22" +--- + +# E-001 — Mechanical recovery and defect discrimination + +## Design + +Run the same verification asset against every labelled lab mutation (T05), +in two arms: + +- **arm A** — semantic action realized by the agentic driver; +- **arm B** — a recorded selector-based sequence captured against the baseline, + built with the most robust selector strategy reasonably available. + +Arm B is the control for H-001 and must actually be run. + +## Outcomes recorded per run + +surface diff (S1) · postcondition met and surface used (S2) · oracle verdicts +(S3) · classification emitted · claim/invariant diff (expected empty) · +tokens · wall time · retries. + +## Decision rules + +- **False Adaptation Rate > 0** → stop. H-004 falsified; redesign, do not tune. +- Arm B recovery ≥ arm A recovery → H-001 falsified. +- Recovery achieved with a non-empty claim diff → H-002 falsified *and* a + framework defect, since D-02 forbids the write path. + +## Status + +`PLANNED`. Blocked on T05 (lab) and T07 (agentic driver). diff --git a/research/experiments/E-002-crystallization-fidelity.md b/research/experiments/E-002-crystallization-fidelity.md new file mode 100644 index 0000000..da52a71 --- /dev/null +++ b/research/experiments/E-002-crystallization-fidelity.md @@ -0,0 +1,33 @@ +--- +id: E-002 +title: Crystallization fidelity and cost +status: PLANNED +hypotheses: [H-003] +task: TD-WP-0002-T09 +created: "2026-08-22" +--- + +# E-002 — Crystallization fidelity and cost + +## Design + +Run one agentic realization repeatedly against a stable lab version until +realization is stable, crystallize it, then execute the descendant with no model +involvement across every lab version the ancestor was run against. + +## Outcomes recorded + +Oracle set of descendant vs ancestor · verdict agreement per lab version · +execution cost ratio (tokens, wall time) · whether human authorship of assertions +was required. + +## Decision rules + +- Descendant oracle set is not a superset of the ancestor's → H-003 falsified. +- Any verdict disagreement on the same lab version → H-003 falsified. +- Cost ratio ≥ 1 → thesis intact, product rationale is not. Record plainly rather + than quietly. + +## Status + +`PLANNED`. Blocked on T07. diff --git a/research/experiments/E-003-surface-substitution-attack.md b/research/experiments/E-003-surface-substitution-attack.md new file mode 100644 index 0000000..c3677d9 --- /dev/null +++ b/research/experiments/E-003-surface-substitution-attack.md @@ -0,0 +1,40 @@ +--- +id: E-003 +title: Deliberate attack on the adaptation safety boundary +status: PLANNED +hypotheses: [H-004] +task: TD-WP-0002-T08 +created: "2026-08-22" +--- + +# E-003 — Deliberate attack on the adaptation safety boundary + +## Why this experiment exists separately + +E-001 measures whether the framework *happens to* stay safe over a labelled set. +E-003 tries to *make it unsafe*. The distinction matters: an experiment that only +samples benign cases cannot establish a safety property, and False Adaptation Rate +is the one metric the project cannot afford to be wrong about. + +## Attack cases + +1. **Surface substitution** — remove the sharing control from the UI while leaving + the HTTP endpoint open. A driver that "recovers" by calling the API has routed + around the control. Must classify as a finding, never as recovery (D-05). +2. **Concurrent change** — ship a mechanical mutation (M01) and the authorization + defect (M05) in the same lab version. Must classify as `PRODUCT_DEFECT`; + coincidence must not exonerate (decision table row 3). +3. **Evidence starvation** — disable the independent observation channel mid-run. + Must yield `AMBIGUOUS`, never a pass and never an adaptation (row 7). +4. **Provenance laundering** — attempt to introduce a claim derived from observed + implementation behaviour and crystallize it. Must be rejected at authoring + time (D-06) and, if it slips through, at crystallization. + +## Decision rule + +Any attack that succeeds is a stop-and-redesign signal for H-004, regardless of +what E-001 reports. + +## Status + +`PLANNED`. Blocked on T05, T07, T08. diff --git a/research/hypotheses/H-001-semantic-action-stability.md b/research/hypotheses/H-001-semantic-action-stability.md new file mode 100644 index 0000000..dcf2291 --- /dev/null +++ b/research/hypotheses/H-001-semantic-action-stability.md @@ -0,0 +1,41 @@ +--- +id: H-001 +title: Semantic Action Stability +status: PROPOSED +created: "2026-08-22" +experiments: [E-001] +concepts: [C-semantic-action] +--- + +# H-001 — Semantic Action Stability + +## Claim + +A semantic action survives implementation restructuring better than a recorded UI +interaction sequence. + +## Falsification condition + +Across the labelled mechanical mutations in the lab (T05), a recorded interaction +sequence survives **at least as many** mutations as the semantic action does. + +If mechanics-free identity buys no measurable durability, the central abstraction +is decorative and `SemanticAction` should be reduced to a naming convention. + +## Measurement + +Mechanical Recovery Rate for each of two arms over the same mutation set: +- **arm A** — semantic action realized by an agentic driver; +- **arm B** — a recorded selector-based sequence captured against the baseline. + +Arm B is a genuine control and must be run, not assumed to fail. + +## Threats to validity + +The comparison is unfair if arm B is built naively — a brittle straw man makes +H-001 trivially true and worthless. Arm B uses the most robust selector strategy +reasonably available (roles, labels, test ids where the lab provides them). + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-002-mechanical-adaptation.md b/research/hypotheses/H-002-mechanical-adaptation.md new file mode 100644 index 0000000..92d7191 --- /dev/null +++ b/research/hypotheses/H-002-mechanical-adaptation.md @@ -0,0 +1,40 @@ +--- +id: H-002 +title: Mechanical Adaptation +status: PROPOSED +created: "2026-08-22" +experiments: [E-001] +concepts: [C-adaptation] +--- + +# H-002 — Mechanical Adaptation + +## Claim + +An agentic driver can recover from a mechanical implementation change without +modifying the semantics of the protected use case. + +## Falsification condition + +Either: +- the driver fails to recover from a majority of labelled `MECHANICAL` mutations + (recovery is not achievable in practice); **or** +- recovery is achieved only in runs where claims or invariants were altered + (recovery is achievable but not semantics-preserving). + +The second branch matters more than the first. A high recovery rate purchased by +weakening assertions falsifies this hypothesis just as decisively as no recovery +at all. + +## Measurement + +- Mechanical Recovery Rate over `MECHANICAL`-labelled mutations. +- Claim/invariant diff per run — expected to be empty by construction + (`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a + falsification signal **and** a framework defect, since no write path should exist. + +## Status log + +- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification + branch structurally unreachable; the measurement is retained anyway, as an + assertion that the architecture is what we believe it is. diff --git a/research/hypotheses/H-003-crystallization.md b/research/hypotheses/H-003-crystallization.md new file mode 100644 index 0000000..e019c5d --- /dev/null +++ b/research/hypotheses/H-003-crystallization.md @@ -0,0 +1,43 @@ +--- +id: H-003 +title: Crystallization +status: PROPOSED +created: "2026-08-22" +experiments: [E-002] +concepts: [C-crystallization] +--- + +# H-003 — Crystallization + +## Claim + +A sufficiently stable agentic execution can be converted into deterministic test +code without losing relevant oracle coverage. + +## Falsification condition + +Any of: +- the generated deterministic test cannot be produced without human authorship of + the assertions; +- it drops one or more oracles present in the agentic ancestor; +- it passes where the ancestor would have failed, or vice versa, on the same lab + version; +- it costs no less to execute than the agentic ancestor. + +## Measurement + +- Crystallization success rate. +- Oracle set of descendant vs ancestor — must be a superset or equal. +- Verdict agreement between ancestor and descendant across lab versions. +- Execution cost ratio (tokens, wall time) descendant : ancestor. + +## Note on the cost criterion + +The cost criterion is included deliberately. If crystallization preserves +semantics but saves nothing, the thesis is intact but the product rationale is +not — and that distinction should be visible in the record rather than discovered +by a user. + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-004-independent-judgment.md b/research/hypotheses/H-004-independent-judgment.md new file mode 100644 index 0000000..e4814a1 --- /dev/null +++ b/research/hypotheses/H-004-independent-judgment.md @@ -0,0 +1,46 @@ +--- +id: H-004 +title: Independent Judgment +status: PROPOSED +created: "2026-08-22" +experiments: [E-001, E-003] +concepts: [C-oracle-independence, C-intent-provenance] +--- + +# H-004 — Independent Judgment + +## Claim + +Separating actor execution from deterministic oracles reduces false-positive +adaptation to defective behaviour. + +## Falsification condition + +The framework classifies any seeded `DEFECT`-labelled mutation as +`MECHANICAL_ADAPTATION` — i.e. **False Adaptation Rate > 0** over the labelled set. + +This is the project's existential safety metric. A non-zero result is a +stop-and-redesign signal, not a tuning exercise. + +## Measurement + +- False Adaptation Rate over `DEFECT`-labelled mutations. Target: exactly 0. +- Classification precision/recall over the full labelled set. +- Rate of `AMBIGUOUS` escalation — tracked as a cost, not as a failure. A system + that escalates everything is safe and useless. + +## Relationship to the design + +`docs/TestDriverClassificationDesign.md` argues that FAR = 0 follows +architecturally from D-02 (claims are run inputs with no adaptation write path) +rather than from classifier accuracy. This hypothesis therefore tests the +**architecture**, not a model. + +The most plausible route to falsification is not a misclassification but +**surface-substitution**: an actor routing around a broken authorization control +via another surface and scoring as a successful recovery. D-05 exists to close +that route; E-003 exists to attack it deliberately. + +## Status log + +- 2026-08-22 `PROPOSED`. No evidence. diff --git a/research/hypotheses/H-005-verification-energy.md b/research/hypotheses/H-005-verification-energy.md new file mode 100644 index 0000000..c1bd75c --- /dev/null +++ b/research/hypotheses/H-005-verification-energy.md @@ -0,0 +1,40 @@ +--- +id: H-005 +title: Verification Energy +status: PROPOSED +created: "2026-08-22" +experiments: [] +concepts: [C-energy] +--- + +# H-005 — Verification Energy + +## Claim + +Historical evidence about defects caught, adaptations required, false positives +and duplication can identify verification assets whose continued execution is +more valuable than others. + +## Falsification condition + +Energy scores computed from event history fail to rank assets better than a +trivial baseline (e.g. recency, or uniform priority) when predicting which assets +subsequently catch a defect. + +## Why no experiment is planned yet + +**This hypothesis is not testable at the current scale and is deliberately +dormant.** Validating it requires event history across many assets over months — +history the spike will not accumulate. Implementing a scoring function now would +produce a number that cannot be checked, which is worse than no number. + +`TD-WP-0002` therefore records **raw immutable `EnergyEvent`s from the first run** +and implements no scoring, decay, or selection logic. Events cannot be +reconstructed later; scores can always be computed later. + +This is the hypothesis most likely to be **cheaply built and never validated**, +which is precisely why it is fenced off. + +## Status log + +- 2026-08-22 `PROPOSED`, dormant by decision. Event capture only.