coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Complete local generalisation tasks and document remaining experiment blockers

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4
This commit is contained in:
tegwick 2026-09-28 12:06:24 +02:00
parent d54af628df
commit cf682281d7
36 changed files with 5394 additions and 279 deletions
Download patch file Download diff file Expand all files Collapse all files

52
research/concepts/fitness-map.md
View file

@ -1,6 +1,6 @@
# Concept ↔ Implementation Fitness Map
**Updated:** 2026-08-23 (TD-WP-0002-T10)
**Updated:** 2026-09-28 (TD-WP-0003-T02/T03/T04)
Traces each important concept to the implementation, experiment and evidence that
support it. **Unsupported entries are the point of this map** — a concept with no
@ -28,43 +28,39 @@ were aspirational, not evidenced.
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|---|---|---|---|---|---|
| `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? |
| `C-use-case` | C1 | `intent.py`, three reference scenarios | T02 | `tests/test_generalisation.py` | Three synthetic use cases fit; external authoring still unmeasured. |
| `C-actor-isolation` | **C2** | `world.py`, `runner.py` | E-001 | `td://self/actor-isolation`, examined every run | **F-0003 resolved** — automatic canaries; observable on every scenario. |
| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. |
| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 | 2026-09-28 two-arm receipt | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. |
| `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
| `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. |
| `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
| `C-adaptation` | **C2** | `classification.py`, `agentic.py` | E-001 | T08 matrix | 11/12 mechanical absorbed without a human. |
| `C-adaptation` | **C2** | `classification.py`, `agentic.py` | E-001 | T08 matrix | 26/27 mechanical absorbed; ten dropped-id cases now included. |
| `C-classification` | **C2** | `classification.py` | E-001, E-003 | T08 matrix, FAR 0/7 | **F-0006** — cannot infer SEMANTIC vs DEFECT; collapses to one escalating outcome. |
| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. |
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. |
| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. |
| `C-energy` | C0 | `energy.py`, capture only | — | none | Dormant. **Gated**: if the next workplan ends with no decision having used the history, remove. |
| `C-temperature` | **C0 — under review** | — | — | none | **F-0008** — crystallization was built without it; measured stability did the work. Gated for removal. |
| `C-confidence` | C0 | — | — | none | Declared, unimplemented, never consulted. Same position as `C-temperature` without a built alternative. |
| `C-campaign` | C0 | — | — | — | Deferred. |
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
| `C-energy` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-temperature` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-confidence` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-campaign` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-metabolism` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-retirement` | Removed | — | T04 | Generalisation review | No demonstrated decision; not an implementation promise. |
| `C-security-mutation` | C1 | `lab/mutations.py` | E-003 | `lab/GROUND-TRUTH.md` | Catalogue is hand-written; no derivation mechanism from use cases yet. |
## Orphan check
## Generalisation and compression review — 2026-09-28
**Conceptual orphans** — concepts with no planned implementation in `TD-WP-0002`:
`C-temperature`, `C-confidence`, `C-campaign`, `C-metabolism`, `C-retirement`.
No new kernel concept was required for delegation/sequencing or tenant lifecycle.
Ordered `Step`s supply the Schedule; scenario `variant` identifies seeded defects.
No independent scheduler or Variant class was needed. Integration/security lenses
remain descriptive groupings; no Lens runtime object was consulted or validated.
The generic observer interface works, but each new domain needs a custom snapshot
collector: observation-channel adoption cost remains a real concern.
All five are deferred *by explicit decision*, not oversight. They are the group
most at risk of being built because they are easy and satisfying, and never
validated. They are revisited at T10, where the question is not "when do we build
these" but "does the evidence justify keeping them in the model at all".
Temperature, Energy, Confidence, Campaign, Metabolism and Retirement are removed,
not deferred to another gate. `energy.py`, its exports and evidence field are
removed; H-005 is dormant-indefinite and F-0008 is resolved. Ordinary evidence,
judgments, realization metrics and lineage remain. See
[review](../../docs/TestDriverGeneralisationReview.md) for compatibility and limits.
**Implementation orphans** — none. Every module in `src/testdriver/` traces to a
concept above.
**Removed at T10** (compression pass — see the gate review § 3):
`Verdict.SUSPICIOUS`, `Step.expect_refusal`, `ActorIsolationError`,
`World.seed`, `EvidencePack.latest()`, `Trajectory.method`. Each was declared and
never used; `SUSPICIOUS` was additionally a verdict no oracle could emit.
**Gated for removal**: `C-temperature` (F-0008) and `energy.py`. Both survive the
current review on the strength of being cheap, not of being used. If the next
workplan closes without a decision consulting either, they go.
Previous compression (T10): `Verdict.SUSPICIOUS`, `Step.expect_refusal`,
`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`, `Trajectory.method`.