Register with State Hub, persist concept assessment, seed first workplan
- history/2026-08-22-concept-assessment-swot.md: SWOT assessment of the concept corpus with recommendations for the first workplan - statehub register: infotech domain, TD-WP prefix, generated AGENTS.md, .custodian-brief.md and TD-WP-0001 bootstrap workplan - .repo-classification.yaml: category research, domain infotech - SCOPE.md rewritten with real repo boundaries - TD-WP-0002: vertical spike reordering M0-M10 into one end-to-end thread that can falsify the crystallization thesis early - commit previously untracked INTENT.md and docs/ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
b471bed707
commit
7249c6a403
13 changed files with 3900 additions and 0 deletions
27
.custodian-brief.md
Normal file
27
.custodian-brief.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
<!-- custodian-brief: generated by statehub register; fix-consistency may replace this file -->
|
||||
# Custodian Brief - test-driver
|
||||
|
||||
**Project:** test-driver
|
||||
**Domain:** infotech
|
||||
**State Hub:** http://127.0.0.1:8000
|
||||
**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a`
|
||||
|
||||
## Open Workplans
|
||||
|
||||
### Bootstrap State Hub integration
|
||||
|
||||
Workplan file: `workplans/TD-WP-0001-statehub-bootstrap.md`
|
||||
|
||||
Open tasks:
|
||||
- T01 - Review generated integration files
|
||||
- T02 - Verify local developer workflow
|
||||
- T03 - Seed first real workplan
|
||||
|
||||
## Session Start
|
||||
|
||||
1. Read `INTENT.md`, `SCOPE.md`, and `AGENTS.md`.
|
||||
2. Check inbox: `GET /messages/?to_agent=test-driver&unread_only=true`.
|
||||
3. Scan `workplans/`.
|
||||
4. Update task statuses in workplan files as work progresses.
|
||||
|
||||
Last generated: 2026-08-22
|
||||
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# state-hub: track .claude/rules
|
||||
# Claude Code local state (track shared rules; ignore machine-specific files)
|
||||
.claude/*
|
||||
!.claude/rules/
|
||||
!.claude/rules/*.md
|
||||
32
.repo-classification.yaml
Normal file
32
.repo-classification.yaml
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-08-22'
|
||||
classified_by: human
|
||||
category: research
|
||||
domain: infotech
|
||||
secondary_domains:
|
||||
- agents
|
||||
capability_tags:
|
||||
- testing
|
||||
- verification
|
||||
- quality-assurance
|
||||
- authorization
|
||||
- security
|
||||
- evidence
|
||||
- orchestration
|
||||
business_stake:
|
||||
- technology
|
||||
- product
|
||||
- execution
|
||||
- automation
|
||||
- intelligence
|
||||
business_mechanics:
|
||||
- intention
|
||||
- control
|
||||
- adaptation
|
||||
notes: >-
|
||||
Research prototype validating the crystallization thesis (agentic exploration
|
||||
hardening into deterministic regression). Standard §5.2 — category is
|
||||
`research` until the prototype success gate is passed; revisit to `project`
|
||||
at that point.
|
||||
209
AGENTS.md
Normal file
209
AGENTS.md
Normal file
|
|
@ -0,0 +1,209 @@
|
|||
# test-driver — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Use-case-driven verification framework for integration, end-to-end, multi-user, authorization and security testing that matures tests from agentic exploration into deterministic regression.
|
||||
|
||||
**Domain:** infotech
|
||||
**Repo slug:** test-driver
|
||||
**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a`
|
||||
**Workplan prefix:** `TD-WP-`
|
||||
|
||||
---
|
||||
|
||||
## State Hub Integration
|
||||
|
||||
The Custodian State Hub tracks work across all domains. Codex uses HTTP REST and
|
||||
the `statehub` CLI by default. MCP is opt-in because the current Codex MCP bridge
|
||||
adds severe call latency; the full administrative MCP surface remains available
|
||||
to clients that need it.
|
||||
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
| Optional local edge relay | http://127.0.0.1:18080 |
|
||||
|
||||
When an operator has enabled the edge relay, set API_BASE to the relay URL.
|
||||
Queueable writes return an explicit queued receipt if the central hub is
|
||||
unreachable. Treat that as pending local evidence, then ask the operator to run
|
||||
statehub outbox status/replay after connectivity returns.
|
||||
|
||||
Codex workspace-write sandboxes need network access enabled to reach the host's
|
||||
loopback listener. Bootstrap this once with `make -C ~/state-hub configure-codex`
|
||||
and restart Codex. The canonical REST health endpoint is `/state/health`, not
|
||||
`/health`. If a sandboxed loopback probe fails, retry it with escalated execution
|
||||
before declaring State Hub unavailable; a managed Codex permission profile may
|
||||
still enforce isolated networking. Experimental MCP can be enabled explicitly
|
||||
with `make -C ~/state-hub configure-codex WITH_MCP=1`.
|
||||
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workplans for this domain
|
||||
curl -s "http://127.0.0.1:8000/workplans/?topic_id=cee7bedf-2b48-46ef-8601-006474f2ad7a&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=test-driver&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workplan_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Omit `workplan_id` / `task_id` when not applicable.
|
||||
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=test-driver&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. If finishing a workplan: hand off **residuals** as live work records first
|
||||
(intake with `origin: residual` + `origin_ref: <WP-id>`, or a next workplan /
|
||||
decision / engagement). Do not park leftovers only in prose or `SCOPE.md`.
|
||||
Canon: `the-custodian/canon/standards/work-record-types_v0.1.md` § Residuals.
|
||||
3. Log: `POST /progress/` with a summary of what changed (name handoff ids)
|
||||
4. After workplan file changes, run:
|
||||
```bash
|
||||
statehub fix-consistency
|
||||
```
|
||||
Coding agents should run this directly; ask the operator only if the CLI or
|
||||
State Hub API is unavailable. This syncs task status from files into the hub DB.
|
||||
If C-06/C-11 reports that this host is not the identifier registrar, do not
|
||||
retry, export `STATEHUB_REGISTRAR`, or register records by hand. Commit and
|
||||
push the file-backed work first, then run the repo-manager fallback once:
|
||||
```bash
|
||||
uv run --project ~/repo-manager rmgr registrar-reconcile \
|
||||
--path . --confirm-primary --push
|
||||
```
|
||||
If unavailable, send one deduplicated registrar request to `repo-manager`
|
||||
naming the repo and canonical ids; UUID absence does not block local work.
|
||||
|
||||
---
|
||||
|
||||
{CREDENTIAL_ROUTING}
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/TD-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-TD-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: TD-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: infotech
|
||||
repo: test-driver
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # fix-consistency — do not edit (legacy field name; workplan UUID)
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Terminology:** workplan is the fleet term; `workstream` appears only in legacy
|
||||
API/MCP/frontmatter bridges until `STATE-WP-0069` retires them — see
|
||||
`the-custodian/canon/standards/workplan-terminology-fleet_v0.1.md`.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: TD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
**Residuals when finishing:** actionable leftovers become live work records
|
||||
before `status: finished` — usually an intake (`origin: residual`,
|
||||
`origin_ref: TD-WP-NNNN`) or a spawned workplan. Residual is a *role*,
|
||||
not a kind. Fleet list lives on State Hub, not in `SCOPE.md`.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Run `statehub fix-consistency` locally.
|
||||
3. On a non-registrar C-06/C-11 skip, use the repo-manager fallback documented
|
||||
above exactly once; never set registrar authority directly.
|
||||
445
INTENT.md
Executable file
445
INTENT.md
Executable file
|
|
@ -0,0 +1,445 @@
|
|||
# test-driver
|
||||
|
||||
## Intent
|
||||
|
||||
`test-driver` is a use-case-driven verification framework for integration, end-to-end, multi-user interaction, authorization, security and resilience testing in software systems that evolve through fast and increasingly agentic development cycles.
|
||||
|
||||
The project exists to make verification itself adaptive without allowing tests to simply conform to whatever an implementation happens to do.
|
||||
|
||||
Its central idea is that tests should **mature together with the behavior they protect**:
|
||||
|
||||
> Emerging software is verified with fluid, exploratory and agentic scenarios. As expected behavior and implementation stabilize, those scenarios are progressively specified, hardened and crystallized into deterministic regression tests that require no agentic involvement.
|
||||
|
||||
`test-driver` therefore treats a test not primarily as code, but as a **verification asset** with identity, intent, evidence, lineage, maturity, temperature and energy over time.
|
||||
|
||||
---
|
||||
|
||||
## Why
|
||||
|
||||
Modern software development increasingly operates under conditions where:
|
||||
|
||||
- product and implementation cycles are very short,
|
||||
- interfaces and workflows change rapidly,
|
||||
- multiple services and interaction surfaces participate in one user outcome,
|
||||
- authorization and tenant boundaries must be verified continuously,
|
||||
- multi-user behavior produces temporal and concurrency problems that linear tests poorly represent,
|
||||
- agentic software development can change code faster than conventional test suites can be manually maintained,
|
||||
- exploratory testing remains valuable but is expensive and difficult to reproduce,
|
||||
- mature behavior should ultimately be protected by cheap, deterministic regression tests.
|
||||
|
||||
Conventional test automation often forces a premature choice between brittle scripted tests and expensive exploratory testing.
|
||||
|
||||
`test-driver` aims to provide a continuum between them.
|
||||
|
||||
---
|
||||
|
||||
## Core Model
|
||||
|
||||
The framework starts from **UseCases** describing intended behavior rather than implementation mechanics.
|
||||
|
||||
A use case is projected into one or more **VerificationAssets** and exercised through different lenses such as:
|
||||
|
||||
- integration,
|
||||
- journey / end-to-end,
|
||||
- multi-user interaction,
|
||||
- security,
|
||||
- resilience,
|
||||
- later, scale and performance.
|
||||
|
||||
A concrete **Scenario** combines:
|
||||
|
||||
```text
|
||||
UseCase + Actors + World + Schedule + Surfaces + Variant
|
||||
```
|
||||
|
||||
Actors execute through **Drivers** such as browsers, APIs, CLIs or messaging systems.
|
||||
|
||||
**Observers** collect evidence independently from actors.
|
||||
|
||||
**Oracles** evaluate claims and invariants from that evidence and produce explicit verdicts.
|
||||
|
||||
Agentic actors may decide how to accomplish goals or explore alternatives, but they should not normally be the sole authority deciding whether the system behaved correctly.
|
||||
|
||||
---
|
||||
|
||||
## Verification Evolution
|
||||
|
||||
Verification assets progress through a maturity continuum:
|
||||
|
||||
```text
|
||||
T0 Exploratory
|
||||
T1 Agentic
|
||||
T2 Adaptive
|
||||
T3 Specified
|
||||
T4 Hardened
|
||||
T5 Deterministic
|
||||
```
|
||||
|
||||
This progression is called **Crystallization**.
|
||||
|
||||
A typical lifecycle is:
|
||||
|
||||
```text
|
||||
Explore
|
||||
-> Discover
|
||||
-> Reproduce
|
||||
-> Minimize
|
||||
-> Specify
|
||||
-> Harden
|
||||
-> Crystallize
|
||||
-> Deterministic Regression
|
||||
```
|
||||
|
||||
Crystallization is reversible: a major redesign may temporarily require a mature verification asset to become adaptive again.
|
||||
|
||||
---
|
||||
|
||||
## Temperature
|
||||
|
||||
Capabilities and implementations have a **Temperature** representing their degree of change:
|
||||
|
||||
```text
|
||||
HOT actively being invented
|
||||
WARM frequently changing
|
||||
COOL stabilizing
|
||||
COLD mature / contractual
|
||||
```
|
||||
|
||||
Temperature influences the preferred verification mode:
|
||||
|
||||
```text
|
||||
HOT -> exploratory / agentic
|
||||
WARM -> adaptive
|
||||
COOL -> hardened
|
||||
COLD -> deterministic
|
||||
```
|
||||
|
||||
The framework should make it natural for tests to crystallize as software cools.
|
||||
|
||||
---
|
||||
|
||||
## Energy
|
||||
|
||||
Every verification asset may carry **Energy** representing the current value of retaining and executing it.
|
||||
|
||||
Energy increases when a test proves useful, for example by:
|
||||
|
||||
- detecting a confirmed defect,
|
||||
- detecting a security violation,
|
||||
- preventing recurrence of a previous defect,
|
||||
- protecting an important and actively changing capability.
|
||||
|
||||
Energy decreases when a test creates maintenance cost without sufficient value, for example when it:
|
||||
|
||||
- repeatedly requires adaptation to legitimate implementation changes,
|
||||
- produces false positives,
|
||||
- becomes flaky,
|
||||
- duplicates stronger verification,
|
||||
- protects behavior that is no longer relevant.
|
||||
|
||||
Energy is not correctness.
|
||||
|
||||
A failing test must not automatically be adapted to the current implementation. A discrepancy may represent an implementation defect, intended requirement change, test defect or ambiguous condition requiring investigation.
|
||||
|
||||
Low-energy tests may move from active execution to low-frequency campaigns, archival state and finally retirement.
|
||||
|
||||
Critical contractual, regulatory or security invariants may define retirement floors or prohibitions.
|
||||
|
||||
---
|
||||
|
||||
## Security by Use-Case Mutation
|
||||
|
||||
Security testing should not be a disconnected universe of hand-maintained tests.
|
||||
|
||||
Ordinary use cases should be transformable into adversarial scenarios through reusable mutations such as:
|
||||
|
||||
```text
|
||||
actor-substitution
|
||||
resource-substitution
|
||||
tenant-substitution
|
||||
sequence-reordering
|
||||
step-skipping
|
||||
replay
|
||||
repetition
|
||||
concurrency
|
||||
surface-substitution
|
||||
invalid-state
|
||||
privilege-mutation
|
||||
dependency-failure
|
||||
```
|
||||
|
||||
For example, from:
|
||||
|
||||
> Alice shares resource R with Bob.
|
||||
|
||||
`test-driver` should be able to derive questions such as:
|
||||
|
||||
- Can Carol access R?
|
||||
- Can Bob write when only read permission was granted?
|
||||
- Can Bob substitute another resource identifier?
|
||||
- Can access survive revocation?
|
||||
- Can a forbidden operation be performed through another surface?
|
||||
- What happens if grant and revoke race each other?
|
||||
|
||||
---
|
||||
|
||||
## Multi-User Isolation
|
||||
|
||||
Actors must be real independent execution entities from the framework's perspective.
|
||||
|
||||
Each actor owns its own:
|
||||
|
||||
- identity,
|
||||
- credentials,
|
||||
- session,
|
||||
- permissions,
|
||||
- private memory,
|
||||
- known resources,
|
||||
- interaction surfaces.
|
||||
|
||||
The orchestrator may know the whole world, but actors must not implicitly share information merely because the same agent technology is used to execute them.
|
||||
|
||||
---
|
||||
|
||||
## Semantic Actions
|
||||
|
||||
`test-driver` should prefer **SemanticActions** over low-level recorded mechanics.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
grant_access(Bob, READ)
|
||||
revoke_access(Bob)
|
||||
approve_invoice(I)
|
||||
open_resource(R)
|
||||
```
|
||||
|
||||
An agent may initially discover how a semantic action maps onto a changing user interface.
|
||||
|
||||
As the implementation stabilizes, the corresponding driver may acquire a deterministic implementation of the same semantic action.
|
||||
|
||||
Semantic actions therefore provide the bridge between agentic exploration and deterministic crystallization.
|
||||
|
||||
---
|
||||
|
||||
## Evidence and Oracles
|
||||
|
||||
Every meaningful run should produce durable evidence sufficient for later verification and diagnosis.
|
||||
|
||||
Evidence may include:
|
||||
|
||||
- scenario and run identifiers,
|
||||
- actor and role information,
|
||||
- system and component versions,
|
||||
- action timeline,
|
||||
- screenshots,
|
||||
- requests and responses,
|
||||
- domain-state observations,
|
||||
- audit records,
|
||||
- logs,
|
||||
- metrics,
|
||||
- traces,
|
||||
- oracle evaluations.
|
||||
|
||||
Oracles evaluate explicit claims and invariants and should prefer deterministic evidence where possible.
|
||||
|
||||
Initial verdicts are:
|
||||
|
||||
```text
|
||||
PASS
|
||||
FAIL
|
||||
SUSPICIOUS
|
||||
INCONCLUSIVE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lineage
|
||||
|
||||
Every verification asset should retain enough provenance to answer:
|
||||
|
||||
> Why does this test exist?
|
||||
|
||||
A test may originate from:
|
||||
|
||||
- a use case,
|
||||
- a requirement,
|
||||
- a previous test,
|
||||
- a defect,
|
||||
- a security incident,
|
||||
- an exploratory finding.
|
||||
|
||||
Findings should be able to generate new hardened regression assets.
|
||||
|
||||
---
|
||||
|
||||
## Campaigns
|
||||
|
||||
A **Campaign** selects which verification assets and scenario variants to execute.
|
||||
|
||||
Expected early campaign types include:
|
||||
|
||||
```text
|
||||
smoke
|
||||
regression
|
||||
release qualification
|
||||
authorization
|
||||
tenant isolation
|
||||
concurrency
|
||||
resilience
|
||||
exploratory
|
||||
```
|
||||
|
||||
Future campaign selection may consider:
|
||||
|
||||
- test energy,
|
||||
- use-case criticality,
|
||||
- changed-system proximity,
|
||||
- risk,
|
||||
- time since last execution,
|
||||
- execution cost,
|
||||
- previous findings.
|
||||
|
||||
This creates an adaptive **Test Metabolism** rather than treating every historical test as equally relevant forever.
|
||||
|
||||
---
|
||||
|
||||
## First Reference Scenario
|
||||
|
||||
The initial end-to-end reference scenario is deliberately multi-user:
|
||||
|
||||
> Alice owns resource R. Alice grants Bob read access. Bob can access R. Carol cannot access R. Alice revokes Bob's access. Bob can no longer access R.
|
||||
|
||||
The scenario is intended to exercise:
|
||||
|
||||
- actor isolation,
|
||||
- identity and authorization,
|
||||
- shared state,
|
||||
- positive and negative verification,
|
||||
- temporal behavior,
|
||||
- security mutation,
|
||||
- evidence collection,
|
||||
- deterministic oracles,
|
||||
- agentic realization,
|
||||
- later crystallization.
|
||||
|
||||
---
|
||||
|
||||
## Initial Architecture Boundaries
|
||||
|
||||
The first implementation should preserve these conceptual boundaries:
|
||||
|
||||
```text
|
||||
UseCase Parser
|
||||
|
|
||||
v
|
||||
Scenario Planner
|
||||
|
|
||||
v
|
||||
Actor Runtime
|
||||
|
|
||||
v
|
||||
Drivers
|
||||
|
|
||||
v
|
||||
System Under Test
|
||||
|
||||
Observers -> Evidence -> Oracle Engine -> Verdict
|
||||
|
||||
Verification Metadata:
|
||||
maturity
|
||||
temperature
|
||||
energy
|
||||
confidence
|
||||
lineage
|
||||
```
|
||||
|
||||
These are logical boundaries first. They do not require separate services or packages in the initial implementation.
|
||||
|
||||
---
|
||||
|
||||
## Initial Milestones
|
||||
|
||||
### M0 — Deterministic Semantic Scenario Runner
|
||||
|
||||
Implement the conceptual kernel without agentic involvement:
|
||||
|
||||
- UseCase,
|
||||
- Actor,
|
||||
- World,
|
||||
- Scenario,
|
||||
- SemanticAction,
|
||||
- Driver,
|
||||
- Observation,
|
||||
- Evidence,
|
||||
- Oracle,
|
||||
- Verdict.
|
||||
|
||||
Run the Alice/Bob/Carol reference scenario deterministically.
|
||||
|
||||
### M1 — Agentic Driver
|
||||
|
||||
Allow an actor agent to realize a semantic action or scenario goal through one interaction surface while maintaining strict actor isolation and evidence capture.
|
||||
|
||||
### M2 — Adaptation and Crystallization
|
||||
|
||||
Add:
|
||||
|
||||
- adaptation classification,
|
||||
- learned successful paths,
|
||||
- hardening,
|
||||
- deterministic test candidate generation,
|
||||
- lineage from exploratory discovery to regression test.
|
||||
|
||||
### M3 — Living Verification
|
||||
|
||||
Add:
|
||||
|
||||
- energy,
|
||||
- temperature,
|
||||
- confidence,
|
||||
- retirement,
|
||||
- security mutation campaigns,
|
||||
- adaptive campaign selection.
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals for the Initial Project
|
||||
|
||||
The first implementation does **not** aim to:
|
||||
|
||||
- replace every existing unit-test framework,
|
||||
- invent a new browser automation engine,
|
||||
- make all tests agentic,
|
||||
- use LLM judgment where deterministic oracles are available,
|
||||
- automatically rewrite semantic requirements to match implementation,
|
||||
- exhaustively enumerate every possible scenario permutation,
|
||||
- build a distributed test cloud before the conceptual model is proven,
|
||||
- solve large-scale performance testing in the first milestone.
|
||||
|
||||
`test-driver` should integrate with mature lower-level testing and automation tools rather than reimplement them unnecessarily.
|
||||
|
||||
---
|
||||
|
||||
## Design Heuristics
|
||||
|
||||
1. **Describe intent before mechanics.**
|
||||
2. **Keep actor knowledge isolated.**
|
||||
3. **Prefer semantic actions over UI coordinates or selectors.**
|
||||
4. **Keep oracles independent from actors.**
|
||||
5. **Treat security as mutation of normal behavior.**
|
||||
6. **Allow agents to explore, but harden what becomes known.**
|
||||
7. **Crystallize stable behavior into deterministic code.**
|
||||
8. **Retain evidence and lineage for explainability.**
|
||||
9. **Track the changing value of tests rather than assuming immortality.**
|
||||
10. **Do not let implementation silently redefine intended behavior.**
|
||||
|
||||
---
|
||||
|
||||
## Success Criterion
|
||||
|
||||
The project succeeds when a new or changing use case can begin with a comparatively fluid behavioral description, be exercised safely with agentic assistance, accumulate evidence and useful variants, and then naturally evolve into reliable deterministic verification as the software stabilizes.
|
||||
|
||||
In short:
|
||||
|
||||
> **test-driver keeps verification fluid while software is fluid, and turns learned behavior into deterministic confidence when the software cools.**
|
||||
103
SCOPE.md
Normal file
103
SCOPE.md
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
# SCOPE
|
||||
|
||||
> This file helps you quickly understand what this repository is about,
|
||||
> when it is relevant, and when it is not.
|
||||
|
||||
---
|
||||
|
||||
## One-liner
|
||||
|
||||
`test-driver` is a use-case-driven verification framework whose tests mature
|
||||
alongside the software they protect — fluid and agentic while behaviour is hot,
|
||||
deterministic once it cools.
|
||||
|
||||
---
|
||||
|
||||
## Core Idea
|
||||
|
||||
A test is treated not as code but as a **verification asset** with identity,
|
||||
intent, evidence, lineage, maturity, temperature and energy. Use cases are the
|
||||
primary behavioural source; integration, journey, multi-user, security and
|
||||
resilience tests are projections of the same use case rather than separate
|
||||
suites.
|
||||
|
||||
Verification assets progress along a maturity continuum
|
||||
(`T0 Exploratory → T5 Deterministic`) called **Crystallization**. Agents may
|
||||
explore and realise semantic actions against unstable interfaces; oracles remain
|
||||
deterministic and independent from actors, so the framework can distinguish a
|
||||
legitimate mechanical change from a product defect rather than adapting to
|
||||
whatever the implementation happens to do.
|
||||
|
||||
**Current state:** research prototype. Concept corpus is complete
|
||||
(`INTENT.md`, `docs/`); implementation has not started. See
|
||||
`history/2026-08-22-concept-assessment-swot.md` for the standing assessment and
|
||||
the reasoning behind the current workplan sequence.
|
||||
|
||||
---
|
||||
|
||||
## In Scope
|
||||
|
||||
- The conceptual model: UseCase, Actor, Scenario, SemanticAction, Observation,
|
||||
Oracle, Verdict, VerificationAsset, Finding, Adaptation, Crystallization.
|
||||
- A deterministic semantic scenario kernel and its evidence format.
|
||||
- The **test-driver lab** — a small mutable application under test carrying
|
||||
labelled mechanical, semantic and defect mutations as ground truth.
|
||||
- Adaptation detection and the defect-vs-adaptation classifier.
|
||||
- Crystallization of agentic realisations into deterministic regression tests.
|
||||
- Security testing expressed as mutation of ordinary use cases.
|
||||
- Self-verification of the framework's own foundational guarantees.
|
||||
- The research control plane: hypotheses, experiments, findings, fitness map.
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Replacing unit-test frameworks, browser automation engines, or CI systems.
|
||||
- Building a load-testing, fuzzing, vulnerability-scanning or observability
|
||||
platform.
|
||||
- Test-management SaaS, distributed test clouds, or multi-tenant hosting.
|
||||
- Making all tests agentic, or using model judgment where a deterministic oracle
|
||||
is available.
|
||||
- Rewriting semantic requirements to match implementation behaviour.
|
||||
- Scale and performance verification before the conceptual model is proven.
|
||||
|
||||
---
|
||||
|
||||
## Relevant When
|
||||
|
||||
- You need the test-driver conceptual vocabulary or its canonical concept set.
|
||||
- You are working on the crystallization, adaptation-classification, or
|
||||
semantic-action binding mechanisms.
|
||||
- You are extending the lab or its mutation catalogue.
|
||||
- You need the framework's hypotheses, fitness scorecard, or evidence format.
|
||||
|
||||
---
|
||||
|
||||
## Not Relevant When
|
||||
|
||||
- You need ordinary unit or component tests for another repo — use that repo's
|
||||
own test stack.
|
||||
- You are looking for fleet coordination or cross-repo memory — that is State
|
||||
Hub.
|
||||
|
||||
---
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
1. `INTENT.md` — purpose, thesis, design heuristics, non-goals.
|
||||
2. `docs/TestDriverConceptModel.md` — the canonical concept set v0.1.
|
||||
3. `docs/TestDriverImprovementLoop.md` — hypotheses, findings taxonomy,
|
||||
self-improvement cycle.
|
||||
4. `docs/TestDriverInitialMilestones.md` — M0–M10 and the prototype success gate.
|
||||
5. `history/2026-08-22-concept-assessment-swot.md` — assessment and the reasons
|
||||
the first workplan reorders those milestones into a vertical spike.
|
||||
6. `workplans/` — current work. Agent instructions: `AGENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Stack
|
||||
|
||||
Deliberately boring, per `docs/TestDriverResearchPrototype.md`: Python, pytest,
|
||||
Playwright, Pydantic/dataclasses, YAML, SQLite. One process, one database, one
|
||||
browser engine, one application under test. Novelty belongs in the verification
|
||||
model, not the infrastructure.
|
||||
1044
docs/TestDriverConceptModel.md
Executable file
1044
docs/TestDriverConceptModel.md
Executable file
File diff suppressed because it is too large
Load diff
609
docs/TestDriverImprovementLoop.md
Executable file
609
docs/TestDriverImprovementLoop.md
Executable file
|
|
@ -0,0 +1,609 @@
|
|||
# TestDriver Improvement Loop
|
||||
|
||||
**Status:** Concept v0.1
|
||||
**Purpose:** Establish a self-improvement loop that keeps the implementation of `test-driver` aligned with its conceptual model while generating evidence about which concepts actually work.
|
||||
|
||||
---
|
||||
|
||||
## 1. Intent
|
||||
|
||||
`test-driver` is itself an evolving software system. Its implementation must therefore be subject to the same principles it applies to systems under test:
|
||||
|
||||
- intended behavior must remain distinguishable from implementation details;
|
||||
- change must generate evidence rather than silently redefine correctness;
|
||||
- exploratory mechanisms should harden into deterministic mechanisms as understanding grows;
|
||||
- failures should improve future verification;
|
||||
- obsolete complexity should be allowed to disappear.
|
||||
|
||||
The improvement loop exists to continuously reconcile:
|
||||
|
||||
1. **Concept** — what test-driver claims should be true;
|
||||
2. **Implementation** — what the framework actually does;
|
||||
3. **Evidence** — what experiments and executions demonstrate.
|
||||
|
||||
The loop should make conceptual drift visible and turn important framework failures into durable self-verification assets.
|
||||
|
||||
---
|
||||
|
||||
## 2. Core Principle
|
||||
|
||||
> Every meaningful implementation increment should generate evidence about whether test-driver is becoming a better realization of its own concept.
|
||||
|
||||
The framework therefore has two simultaneous feedback loops:
|
||||
|
||||
```text
|
||||
Concept / Intent / Hypotheses
|
||||
|
|
||||
v
|
||||
Select Experiment
|
||||
|
|
||||
v
|
||||
Implement
|
||||
|
|
||||
v
|
||||
Exercise on SUT
|
||||
|
|
||||
v
|
||||
Evidence
|
||||
|
|
||||
+----------+----------+
|
||||
| |
|
||||
v v
|
||||
Product Finding Framework Finding
|
||||
| |
|
||||
v v
|
||||
Improve target Improve test-driver
|
||||
|
|
||||
v
|
||||
Reconcile with Concept
|
||||
|
|
||||
+-----> next cycle
|
||||
```
|
||||
|
||||
A run may therefore produce findings about the system under test and findings about the verification framework itself.
|
||||
|
||||
---
|
||||
|
||||
## 3. Development as Experimental Work
|
||||
|
||||
Implementation work should increasingly be framed as hypotheses rather than feature requests.
|
||||
|
||||
Examples:
|
||||
|
||||
### H-001 — Semantic Action Stability
|
||||
|
||||
> A semantic action survives implementation restructuring better than a recorded UI interaction sequence.
|
||||
|
||||
### H-002 — Mechanical Adaptation
|
||||
|
||||
> An agentic driver can recover from a mechanical implementation change without modifying the semantics of the protected use case.
|
||||
|
||||
### H-003 — Crystallization
|
||||
|
||||
> A sufficiently stable agentic execution can be converted into deterministic test code without losing relevant oracle coverage.
|
||||
|
||||
### H-004 — Independent Judgment
|
||||
|
||||
> Separating actor execution from deterministic oracles reduces false-positive adaptation to defective behavior.
|
||||
|
||||
### H-005 — Verification Energy
|
||||
|
||||
> Historical evidence about defects caught, adaptations required, false positives and duplication can identify verification assets whose continued execution is more valuable than others.
|
||||
|
||||
Each hypothesis should have:
|
||||
|
||||
- an identifier;
|
||||
- a claim;
|
||||
- a falsification condition;
|
||||
- one or more experiments;
|
||||
- evidence;
|
||||
- a status;
|
||||
- resulting implementation or concept changes.
|
||||
|
||||
Suggested lifecycle:
|
||||
|
||||
```text
|
||||
PROPOSED
|
||||
|
|
||||
v
|
||||
EXPERIMENTING
|
||||
|
|
||||
+------> REJECTED
|
||||
|
|
||||
v
|
||||
SUPPORTED
|
||||
|
|
||||
v
|
||||
PRACTICALLY_VALIDATED
|
||||
|
|
||||
v
|
||||
ARCHITECTURAL
|
||||
```
|
||||
|
||||
A hypothesis may also be reopened if later evidence contradicts it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Concept–Implementation Fitness Map
|
||||
|
||||
Every important concept should become traceable to the implementation and evidence that support it.
|
||||
|
||||
Conceptual relationship:
|
||||
|
||||
```text
|
||||
Concept
|
||||
|
|
||||
+-- implementation
|
||||
+-- experiment
|
||||
+-- evidence
|
||||
+-- self-verification
|
||||
+-- unresolved questions
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
concept: actor-isolation
|
||||
claim: >
|
||||
One actor must not obtain private state, credentials, observations,
|
||||
or memory belonging to another actor except through modeled
|
||||
communication channels.
|
||||
|
||||
implementation:
|
||||
- testdriver/runtime/actor_context.py
|
||||
|
||||
experiments:
|
||||
- H-006
|
||||
|
||||
self_verifications:
|
||||
- td://self/actor-isolation
|
||||
|
||||
status: supported
|
||||
```
|
||||
|
||||
The map should expose two forms of drift.
|
||||
|
||||
### Conceptual Orphaning
|
||||
|
||||
A concept is claimed but has no implementation or verification evidence.
|
||||
|
||||
### Implementation Orphaning
|
||||
|
||||
A subsystem or abstraction exists without a concept, requirement, or experiment that explains why it is needed.
|
||||
|
||||
Both should be visible during review.
|
||||
|
||||
---
|
||||
|
||||
## 5. Concept Drift
|
||||
|
||||
A **Concept Drift Finding** occurs when implementation behavior diverges from an established concept without an explicit decision to revise that concept.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Concept:
|
||||
Agents discover paths; independent oracles judge outcomes.
|
||||
|
||||
Implementation:
|
||||
Browser agent declares the scenario successful.
|
||||
|
||||
Finding:
|
||||
CONCEPT_DRIFT
|
||||
```
|
||||
|
||||
A concept-drift finding must resolve in one of three ways:
|
||||
|
||||
1. implementation changes to match the concept;
|
||||
2. the concept is deliberately revised;
|
||||
3. an experiment demonstrates that the distinction is no longer useful.
|
||||
|
||||
Implementation reality must not silently redefine the conceptual model.
|
||||
|
||||
---
|
||||
|
||||
## 6. Self-Verification
|
||||
|
||||
`test-driver` should become a system under test for `test-driver`.
|
||||
|
||||
Self-verification use cases use the namespace:
|
||||
|
||||
```text
|
||||
td://self/...
|
||||
```
|
||||
|
||||
Initial candidates:
|
||||
|
||||
```text
|
||||
td://self/actor-isolation
|
||||
td://self/oracle-independence
|
||||
td://self/evidence-reproducibility
|
||||
td://self/mechanical-adaptation
|
||||
td://self/semantic-change-detection
|
||||
td://self/crystallization
|
||||
td://self/test-retirement
|
||||
```
|
||||
|
||||
These scenarios should verify framework-level promises rather than implementation internals whenever possible.
|
||||
|
||||
Example:
|
||||
|
||||
### `td://self/mechanical-adaptation`
|
||||
|
||||
1. execute a stable use case against lab version A;
|
||||
2. change the UI mechanically without changing semantics;
|
||||
3. rerun the use case;
|
||||
4. allow agentic navigation to recover;
|
||||
5. verify that the same semantic action and oracles remain valid;
|
||||
6. record the adaptation and evidence.
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
mechanical implementation change
|
||||
|
|
||||
v
|
||||
adaptation detected
|
||||
|
|
||||
v
|
||||
alternative realization discovered
|
||||
|
|
||||
v
|
||||
semantic action preserved
|
||||
|
|
||||
v
|
||||
original oracle still passes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Test-Driver Lab
|
||||
|
||||
A purpose-built mutable application should provide controlled evolutionary pressure for the framework.
|
||||
|
||||
Suggested repository or module name:
|
||||
|
||||
```text
|
||||
test-driver-lab
|
||||
```
|
||||
|
||||
The lab should be intentionally small but support:
|
||||
|
||||
- multiple users;
|
||||
- organizations or tenants;
|
||||
- authentication;
|
||||
- resources;
|
||||
- sharing;
|
||||
- permissions;
|
||||
- simple workflows;
|
||||
- audit events;
|
||||
- API interaction;
|
||||
- browser interaction.
|
||||
|
||||
The lab should also support deliberate implementation mutations.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
M01 move or rename a UI control
|
||||
M02 replace the DOM structure
|
||||
M03 change a compatible API representation
|
||||
M04 add a legitimate workflow step
|
||||
M05 introduce an authorization defect
|
||||
M06 introduce eventual-consistency delay
|
||||
M07 introduce intermittent dependency failure
|
||||
M08 remove or deprecate a capability
|
||||
M09 create a concurrency race
|
||||
M10 change the intended business requirement
|
||||
```
|
||||
|
||||
The purpose is not to build a representative product. It is to create a controlled environment in which claims about test-driver can be falsified.
|
||||
|
||||
---
|
||||
|
||||
## 8. Dual Mutation
|
||||
|
||||
Mutation should operate in two directions.
|
||||
|
||||
### Use-Case Mutation
|
||||
|
||||
Mutate actor, resource, order, timing, state, or permissions to test the robustness of the system under test.
|
||||
|
||||
```text
|
||||
UseCase Mutation
|
||||
|
|
||||
v
|
||||
tests robustness of application
|
||||
```
|
||||
|
||||
### Implementation Mutation
|
||||
|
||||
Mutate the target implementation while holding the use-case semantics constant to test the robustness of test-driver.
|
||||
|
||||
```text
|
||||
Implementation Mutation
|
||||
|
|
||||
v
|
||||
tests robustness of test-driver
|
||||
```
|
||||
|
||||
This duality allows the framework to test both the application and its own verification strategy.
|
||||
|
||||
---
|
||||
|
||||
## 9. Framework Findings
|
||||
|
||||
The framework should maintain finding classes distinct from ordinary product defects.
|
||||
|
||||
Initial set:
|
||||
|
||||
### PRODUCT_DEFECT
|
||||
The system under test violates unchanged intent.
|
||||
|
||||
### TEST_DEFECT
|
||||
The verification asset or oracle is incorrect.
|
||||
|
||||
### MECHANICAL_ADAPTATION
|
||||
Implementation mechanics changed while protected semantics remain equivalent.
|
||||
|
||||
### SEMANTIC_CHANGE
|
||||
The intended product behavior has changed.
|
||||
|
||||
### CONCEPT_DRIFT
|
||||
The implementation of test-driver no longer matches an established framework concept.
|
||||
|
||||
### FRAMEWORK_LIMITATION
|
||||
A valid scenario cannot be expressed, executed, observed, or judged adequately.
|
||||
|
||||
### EVIDENCE_FAILURE
|
||||
A finding cannot be reproduced or supported from the retained evidence.
|
||||
|
||||
### UNNECESSARY_COMPLEXITY
|
||||
An abstraction or subsystem adds material maintenance cost without sufficient conceptual or experimental justification.
|
||||
|
||||
These findings feed the improvement loop.
|
||||
|
||||
---
|
||||
|
||||
## 10. Improvement Cycle
|
||||
|
||||
The canonical loop is:
|
||||
|
||||
```text
|
||||
OBSERVE
|
||||
|
|
||||
v
|
||||
CLASSIFY
|
||||
|
|
||||
v
|
||||
EXPLAIN
|
||||
|
|
||||
v
|
||||
PROPOSE
|
||||
|
|
||||
v
|
||||
EXPERIMENT
|
||||
|
|
||||
v
|
||||
MEASURE
|
||||
|
|
||||
v
|
||||
ACCEPT / REJECT
|
||||
|
|
||||
v
|
||||
CRYSTALLIZE
|
||||
```
|
||||
|
||||
### Observe
|
||||
|
||||
Collect evidence from test-driver runs, self-tests, implementation work and lab experiments.
|
||||
|
||||
### Classify
|
||||
|
||||
Determine whether the observation represents a product defect, test defect, adaptation, concept drift, framework limitation or other finding class.
|
||||
|
||||
### Explain
|
||||
|
||||
Produce the smallest useful causal explanation supported by evidence.
|
||||
|
||||
### Propose
|
||||
|
||||
Generate one or more candidate improvements.
|
||||
|
||||
### Experiment
|
||||
|
||||
Change one relevant variable where practical and attempt to falsify the proposed improvement.
|
||||
|
||||
### Measure
|
||||
|
||||
Evaluate the result against explicit success criteria.
|
||||
|
||||
### Accept / Reject
|
||||
|
||||
Retain improvements that produce sufficient evidence. Reject or revise those that do not.
|
||||
|
||||
### Crystallize
|
||||
|
||||
Convert learned behavior into a more deterministic, cheaper and more maintainable form whenever possible.
|
||||
|
||||
---
|
||||
|
||||
## 11. Agentic Roles
|
||||
|
||||
Self-improvement should not rely on one omnipotent self-modifying agent.
|
||||
|
||||
Distinct roles create productive tension.
|
||||
|
||||
### Builder
|
||||
|
||||
Implements the current hypothesis or improvement proposal.
|
||||
|
||||
### Experimenter
|
||||
|
||||
Designs experiments intended to falsify claims.
|
||||
|
||||
### Critic
|
||||
|
||||
Looks for false success, hidden assumptions and semantic drift.
|
||||
|
||||
### Auditor
|
||||
|
||||
Checks concept-to-implementation traceability.
|
||||
|
||||
### Maintainer
|
||||
|
||||
Looks for unnecessary abstractions, duplication and maintenance burden.
|
||||
|
||||
These are conceptual roles. Initially they may simply correspond to separate prompts or workflow phases.
|
||||
|
||||
---
|
||||
|
||||
## 12. Fitness Scorecard
|
||||
|
||||
The framework should be measured against its thesis rather than implementation volume.
|
||||
|
||||
Initial dimensions:
|
||||
|
||||
| Dimension | Example Measure |
|
||||
|---|---|
|
||||
| Adaptability | Mechanical changes recovered automatically |
|
||||
| Semantic integrity | False semantic adaptations |
|
||||
| Detection | Seeded defects correctly discovered |
|
||||
| Reproducibility | Findings replayable from retained evidence |
|
||||
| Crystallization | Agentic assets converted to deterministic execution |
|
||||
| Efficiency | Cost per verified use case |
|
||||
| Autonomy | Human interventions per 100 runs |
|
||||
| Robustness | Success across controlled implementation mutations |
|
||||
| Traceability | Concepts connected to implementation and evidence |
|
||||
| Simplicity | Complexity required per supported capability |
|
||||
|
||||
A particularly important safety metric is:
|
||||
|
||||
> **False Adaptation Rate:** the frequency with which test-driver treats an actual product defect as a legitimate adaptation.
|
||||
|
||||
This should be aggressively minimized.
|
||||
|
||||
---
|
||||
|
||||
## 13. Concept Maturity
|
||||
|
||||
Concepts should mature based on evidence rather than attractive terminology.
|
||||
|
||||
Suggested levels:
|
||||
|
||||
```text
|
||||
C0 Idea
|
||||
C1 Hypothesis
|
||||
C2 Experimentally Supported
|
||||
C3 Practically Validated
|
||||
C4 Architectural Invariant
|
||||
```
|
||||
|
||||
Examples at the beginning of the research prototype may be approximately:
|
||||
|
||||
```text
|
||||
Actor Isolation C2
|
||||
Independent Oracles C2
|
||||
Semantic Actions C1
|
||||
Crystallization C1
|
||||
Verification Energy C0-C1
|
||||
```
|
||||
|
||||
These classifications are provisional and should change with evidence.
|
||||
|
||||
---
|
||||
|
||||
## 14. Compression
|
||||
|
||||
Self-improvement must include deletion.
|
||||
|
||||
Learning does not necessarily imply adding features or abstractions.
|
||||
|
||||
At regular intervals, perform a compression review:
|
||||
|
||||
- Which concepts can be merged?
|
||||
- Which abstractions lack evidence?
|
||||
- Which agentic mechanisms can crystallize into deterministic code?
|
||||
- Which metadata has never informed a decision?
|
||||
- Which subsystem can be removed?
|
||||
- Which verification assets have become redundant?
|
||||
|
||||
The desired outcome is not maximal capability count.
|
||||
|
||||
It is:
|
||||
|
||||
> **the smallest framework that reliably realizes the validated test-driver concepts.**
|
||||
|
||||
---
|
||||
|
||||
## 15. Improvement Evidence
|
||||
|
||||
Every accepted framework improvement should retain:
|
||||
|
||||
```text
|
||||
Improvement ID
|
||||
Triggering finding(s)
|
||||
Affected concept(s)
|
||||
Hypothesis
|
||||
Experiment
|
||||
Before state
|
||||
After state
|
||||
Evidence
|
||||
Measured outcome
|
||||
Decision
|
||||
Resulting self-verification
|
||||
Resulting deterministic regression, if applicable
|
||||
```
|
||||
|
||||
This creates a lineage from conceptual claim through evidence to implementation.
|
||||
|
||||
---
|
||||
|
||||
## 16. Initial Control Loop
|
||||
|
||||
The first working version does not require autonomous self-modification.
|
||||
|
||||
A minimal loop is sufficient:
|
||||
|
||||
```text
|
||||
Framework run
|
||||
|
|
||||
v
|
||||
Framework finding
|
||||
|
|
||||
v
|
||||
Human/agent classification
|
||||
|
|
||||
v
|
||||
Improvement hypothesis
|
||||
|
|
||||
v
|
||||
Controlled lab experiment
|
||||
|
|
||||
v
|
||||
Evidence
|
||||
|
|
||||
v
|
||||
Accept / reject
|
||||
|
|
||||
v
|
||||
Self-verification added
|
||||
```
|
||||
|
||||
Only after this loop reliably produces good improvements should more of the process become agentic.
|
||||
|
||||
---
|
||||
|
||||
## 17. Success Condition
|
||||
|
||||
The improvement loop is successful when test-driver can repeatedly demonstrate that:
|
||||
|
||||
1. conceptual claims are traceable to implementation and evidence;
|
||||
2. implementation changes that violate those claims are detected;
|
||||
3. controlled experiments can distinguish defects, adaptations and semantic changes;
|
||||
4. important framework failures become durable self-verifications;
|
||||
5. agentic mechanisms harden into deterministic mechanisms where possible;
|
||||
6. the framework becomes simpler or more effective as evidence accumulates;
|
||||
7. framework evolution does not silently redefine correctness.
|
||||
|
||||
The long-term objective is a self-hosting verification system whose own evolution is governed by the evidence-driven principles it applies to other software.
|
||||
512
docs/TestDriverInitialMilestones.md
Executable file
512
docs/TestDriverInitialMilestones.md
Executable file
|
|
@ -0,0 +1,512 @@
|
|||
# TestDriver Research Prototype — Initial Milestones
|
||||
|
||||
**Status:** v0.1
|
||||
**Purpose:** Establish the minimum evidence-producing development loop needed to validate the core test-driver concepts.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 0 — Research Control Plane
|
||||
|
||||
### Goal
|
||||
|
||||
Make the conceptual development of test-driver explicit, traceable and falsifiable before substantial framework code accumulates.
|
||||
|
||||
### Implement
|
||||
|
||||
- repository structure for concepts, hypotheses, experiments, findings and evidence;
|
||||
- hypothesis register;
|
||||
- concept maturity register;
|
||||
- Concept ↔ Implementation Fitness Map;
|
||||
- framework finding taxonomy;
|
||||
- stable identifiers for concepts, hypotheses, experiments and findings;
|
||||
- lightweight CLI or file conventions for recording decisions.
|
||||
|
||||
Suggested structure:
|
||||
|
||||
```text
|
||||
research/
|
||||
├── hypotheses/
|
||||
├── experiments/
|
||||
├── findings/
|
||||
├── concepts/
|
||||
└── decisions/
|
||||
```
|
||||
|
||||
### Initial hypotheses
|
||||
|
||||
At minimum register:
|
||||
|
||||
- H-001 Semantic Action Stability
|
||||
- H-002 Mechanical Adaptation
|
||||
- H-003 Crystallization
|
||||
- H-004 Independent Judgment
|
||||
- H-005 Verification Energy
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- every major v0.1 concept has a stable identifier;
|
||||
- every implemented subsystem can be linked to at least one concept or hypothesis;
|
||||
- at least one hypothesis is expressed with a falsification condition and planned experiment;
|
||||
- framework findings can be recorded independently from product findings.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first Concept ↔ Implementation Fitness Map and hypothesis register.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 1 — Deterministic Semantic Kernel
|
||||
|
||||
### Goal
|
||||
|
||||
Prove the core model without agentic complexity.
|
||||
|
||||
### Implement
|
||||
|
||||
Minimal executable representations of:
|
||||
|
||||
- UseCase;
|
||||
- Actor;
|
||||
- World;
|
||||
- Scenario;
|
||||
- SemanticAction;
|
||||
- Observation;
|
||||
- Oracle;
|
||||
- Verdict;
|
||||
- VerificationAsset;
|
||||
- Run.
|
||||
|
||||
Implement one deterministic driver, preferably HTTP or direct application adapter.
|
||||
|
||||
### Reference Use Case
|
||||
|
||||
```text
|
||||
Alice owns resource R.
|
||||
Alice grants Bob READ access.
|
||||
Bob can read R.
|
||||
Carol cannot read R.
|
||||
Alice revokes Bob.
|
||||
Bob can no longer read R.
|
||||
```
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- the complete use case runs deterministically;
|
||||
- Alice, Bob and Carol have isolated identities and sessions;
|
||||
- all important outcomes are judged by independent deterministic oracles;
|
||||
- a run produces structured evidence;
|
||||
- the same scenario can be replayed from known initial state.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first reproducible Evidence Pack.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 2 — Test-Driver Lab
|
||||
|
||||
### Goal
|
||||
|
||||
Create a controlled evolutionary environment in which test-driver claims can be deliberately challenged.
|
||||
|
||||
### Implement
|
||||
|
||||
A deliberately small application supporting:
|
||||
|
||||
- users;
|
||||
- tenants/workspaces;
|
||||
- authentication;
|
||||
- resources;
|
||||
- sharing;
|
||||
- read/write permissions;
|
||||
- revoke;
|
||||
- audit history;
|
||||
- HTTP API;
|
||||
- minimal browser UI.
|
||||
|
||||
Add explicit mutation switches or tagged lab versions.
|
||||
|
||||
### Initial Mutations
|
||||
|
||||
- M01 move/rename sharing control;
|
||||
- M02 restructure the DOM;
|
||||
- M03 change compatible API representation;
|
||||
- M04 introduce an additional legitimate workflow step;
|
||||
- M05 introduce authorization defect;
|
||||
- M06 introduce propagation delay.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- the reference use case works against the baseline lab;
|
||||
- each mutation can be enabled reproducibly;
|
||||
- mutations can be classified as mechanical, semantic or defective;
|
||||
- baseline and mutated versions retain explicit version identifiers.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
A repeatable benchmark environment for framework development.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 3 — Self-Verification v0
|
||||
|
||||
### Goal
|
||||
|
||||
Make test-driver test its own foundational guarantees.
|
||||
|
||||
### Implement
|
||||
|
||||
Initial `td://self/...` verification assets:
|
||||
|
||||
```text
|
||||
td://self/actor-isolation
|
||||
td://self/oracle-independence
|
||||
td://self/evidence-reproducibility
|
||||
```
|
||||
|
||||
The self-tests should operate against observable behavior rather than internal implementation details where practical.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- intentionally breaking actor isolation makes the corresponding self-test fail;
|
||||
- allowing an actor to determine its own verdict makes oracle-independence fail;
|
||||
- corrupting or omitting required run evidence makes evidence-reproducibility fail;
|
||||
- each failure creates a Framework Finding.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Proof that conceptual regressions can be detected as framework regressions.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 4 — Agentic Realization
|
||||
|
||||
### Goal
|
||||
|
||||
Introduce agentic flexibility only at the realization layer while retaining deterministic truth.
|
||||
|
||||
### Implement
|
||||
|
||||
- browser driver;
|
||||
- one agentic Actor Runtime;
|
||||
- strict per-actor context isolation;
|
||||
- semantic goal → UI realization loop;
|
||||
- full action/evidence recording;
|
||||
- bounded navigation and tool permissions.
|
||||
|
||||
Use the semantic action:
|
||||
|
||||
```text
|
||||
grant_access(Bob, R, READ)
|
||||
```
|
||||
|
||||
The agent may discover how to accomplish it through the UI.
|
||||
|
||||
The oracle must remain deterministic.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- an agent can realize the reference semantic action from intent;
|
||||
- the actor cannot access another actor's private context;
|
||||
- deterministic oracles independently establish success/failure;
|
||||
- agent/model/configuration identity is recorded in evidence;
|
||||
- failures can be replayed sufficiently to diagnose them.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
First trustworthy agentic run.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 5 — Mechanical Adaptation
|
||||
|
||||
### Goal
|
||||
|
||||
Demonstrate the core fluid-development thesis.
|
||||
|
||||
### Experiment
|
||||
|
||||
Run the same Verification Asset against:
|
||||
|
||||
1. lab baseline;
|
||||
2. M01 moved/renamed control;
|
||||
3. M02 changed DOM structure.
|
||||
|
||||
The use-case semantics remain unchanged.
|
||||
|
||||
### Implement
|
||||
|
||||
- adaptation detection;
|
||||
- adaptation classification;
|
||||
- preservation of semantic action identity;
|
||||
- adaptation evidence;
|
||||
- adaptation history on the Verification Asset.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- agentic execution recovers from at least two mechanical mutations;
|
||||
- original deterministic oracles remain unchanged;
|
||||
- semantic intent is not modified;
|
||||
- adaptation is classified as mechanical;
|
||||
- the framework reports an adaptation rather than a product defect.
|
||||
|
||||
### Success Metric
|
||||
|
||||
**Mechanical Recovery Rate**
|
||||
|
||||
### Critical Safety Metric
|
||||
|
||||
**False Semantic Adaptation Rate = 0** for the experiment set.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Support or rejection for H-001 and H-002.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 6 — Defect vs. Adaptation Discrimination
|
||||
|
||||
### Goal
|
||||
|
||||
Prove that adaptive testing does not simply learn to accept broken software.
|
||||
|
||||
### Experiment
|
||||
|
||||
Use:
|
||||
|
||||
- M01/M02 as legitimate mechanical changes;
|
||||
- M05 as an authorization defect;
|
||||
- M04 or M10-style mutation as a deliberate semantic requirement change.
|
||||
|
||||
### Implement
|
||||
|
||||
Classification path:
|
||||
|
||||
```text
|
||||
IMPLEMENTATION CHANGE
|
||||
INTENT CHANGE
|
||||
PRODUCT DEFECT
|
||||
AMBIGUOUS
|
||||
```
|
||||
|
||||
Add escalation for semantic changes and ambiguity.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- mechanical changes adapt without altering claims/invariants;
|
||||
- authorization defect creates a Product Finding;
|
||||
- deliberate requirement change creates a Semantic Change finding;
|
||||
- ambiguous evidence produces `INCONCLUSIVE` rather than silent adaptation;
|
||||
- no seeded defect is normalized as adaptation.
|
||||
|
||||
### Success Metric
|
||||
|
||||
Classification precision/recall over controlled mutations.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first meaningful measurement of adaptation safety.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 7 — Crystallization v0
|
||||
|
||||
### Goal
|
||||
|
||||
Show that agentic flexibility can harden into deterministic regression.
|
||||
|
||||
### Implement
|
||||
|
||||
- semantic action trajectory capture;
|
||||
- stable-realization detection;
|
||||
- deterministic candidate generation;
|
||||
- candidate comparison against existing oracle set;
|
||||
- provenance/lineage from agentic ancestor to deterministic descendant;
|
||||
- manual acceptance step initially.
|
||||
|
||||
### Experiment
|
||||
|
||||
Run the same agentic realization repeatedly against a stable lab version, crystallize it, then execute without any model involvement.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- one agentic Verification Asset produces a deterministic test candidate;
|
||||
- deterministic execution preserves the relevant claims and oracles;
|
||||
- the generated/hardened test runs with zero agentic involvement;
|
||||
- lineage remains visible;
|
||||
- execution cost is measurably lower than agentic execution.
|
||||
|
||||
### Success Metrics
|
||||
|
||||
- crystallization success rate;
|
||||
- semantic coverage retained;
|
||||
- execution cost reduction.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
Support or rejection for H-003.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 8 — Framework Finding → Improvement Loop
|
||||
|
||||
### Goal
|
||||
|
||||
Close the first actual self-improvement cycle.
|
||||
|
||||
### Implement
|
||||
|
||||
Workflow:
|
||||
|
||||
```text
|
||||
Framework Finding
|
||||
↓
|
||||
Classification
|
||||
↓
|
||||
Improvement Hypothesis
|
||||
↓
|
||||
Controlled Experiment
|
||||
↓
|
||||
Evidence
|
||||
↓
|
||||
Accept / Reject
|
||||
↓
|
||||
Self-Verification / Regression
|
||||
```
|
||||
|
||||
Use a real framework weakness discovered during Milestones 1–7 rather than inventing one if possible.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- a framework finding produces an explicit improvement hypothesis;
|
||||
- the hypothesis is experimentally evaluated;
|
||||
- the accepted change links back to concept and evidence;
|
||||
- the discovered framework failure leaves behind a permanent self-verification or deterministic regression;
|
||||
- the Concept ↔ Implementation Fitness Map is updated.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first completed **Concept–Implementation Fitness Loop**.
|
||||
|
||||
This is the milestone at which the self-improvement system genuinely exists.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 9 — Verification Energy v0
|
||||
|
||||
### Goal
|
||||
|
||||
Begin measuring test value without prematurely optimizing the scoring model.
|
||||
|
||||
### Implement
|
||||
|
||||
Record immutable Energy Events such as:
|
||||
|
||||
```text
|
||||
DEFECT_DETECTED
|
||||
REGRESSION_CAUGHT
|
||||
MECHANICAL_ADAPTATION
|
||||
SEMANTIC_ADAPTATION
|
||||
TEST_DEFECT
|
||||
FALSE_POSITIVE
|
||||
DUPLICATE
|
||||
CRYSTALLIZED
|
||||
USECASE_DEPRECATED
|
||||
```
|
||||
|
||||
Initially calculate only a simple transparent score.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Energy is derived from event history rather than stored as unexplained state;
|
||||
- every score change is explainable;
|
||||
- Energy can influence campaign priority;
|
||||
- criticality can override retirement;
|
||||
- no automatic deletion is implemented yet.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
A dataset suitable for later testing whether Energy actually predicts verification value.
|
||||
|
||||
---
|
||||
|
||||
## Milestone 10 — First Compression Review
|
||||
|
||||
### Goal
|
||||
|
||||
Prevent the research prototype from turning into premature platform architecture.
|
||||
|
||||
### Review
|
||||
|
||||
Ask:
|
||||
|
||||
- Which concepts have no supporting evidence?
|
||||
- Which implementation abstractions have no conceptual justification?
|
||||
- Which metadata has not informed a decision?
|
||||
- Which agentic behavior can now be deterministic?
|
||||
- Which capabilities can be merged or removed?
|
||||
- What have the experiments falsified?
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- at least one simplification is seriously evaluated;
|
||||
- rejected concepts are marked as such rather than silently retained;
|
||||
- architecture reflects experimental learning;
|
||||
- updated Concept Model and Improvement Loop remain smaller or more precise where evidence permits.
|
||||
|
||||
### Evidence Produced
|
||||
|
||||
The first proof that self-improvement includes subtraction, not only accumulation.
|
||||
|
||||
---
|
||||
|
||||
# Recommended Execution Order
|
||||
|
||||
```text
|
||||
M0 Research Control Plane
|
||||
|
|
||||
M1 Deterministic Semantic Kernel
|
||||
|
|
||||
M2 Test-Driver Lab
|
||||
|
|
||||
M3 Self-Verification v0
|
||||
|
|
||||
M4 Agentic Realization
|
||||
|
|
||||
M5 Mechanical Adaptation
|
||||
|
|
||||
M6 Defect vs Adaptation
|
||||
|
|
||||
M7 Crystallization
|
||||
|
|
||||
M8 Closed Improvement Loop
|
||||
|
|
||||
M9 Verification Energy
|
||||
|
|
||||
M10 Compression Review
|
||||
```
|
||||
|
||||
The first major research gate is **M8**.
|
||||
|
||||
Before M8, test-driver has promising mechanisms.
|
||||
|
||||
At M8, it has demonstrated a complete evidence-driven self-improvement cycle.
|
||||
|
||||
---
|
||||
|
||||
# Prototype Success Gate
|
||||
|
||||
The initial research prototype should be considered successful enough to justify broader framework investment when it can demonstrate all of the following in one coherent system:
|
||||
|
||||
1. a multi-user use case expressed independently of implementation details;
|
||||
2. deterministic independent oracles;
|
||||
3. agentic realization of at least one semantic action;
|
||||
4. recovery from legitimate mechanical implementation change;
|
||||
5. rejection of a seeded semantic/security defect as a mere adaptation;
|
||||
6. reproducible evidence;
|
||||
7. crystallization into deterministic execution;
|
||||
8. a framework failure converted into a permanent self-verification;
|
||||
9. explicit concept-to-implementation traceability;
|
||||
10. measured human effort and execution cost.
|
||||
|
||||
That demonstration is more valuable than broad feature coverage.
|
||||
338
docs/TestDriverResearchPrototype.md
Normal file
338
docs/TestDriverResearchPrototype.md
Normal file
|
|
@ -0,0 +1,338 @@
|
|||
# Stage 1 Test Driver Validation
|
||||
|
||||
The biggest risk is not technical feasibility. It is that **test-driver becomes conceptually elegant but too broad to prove itself quickly**.
|
||||
|
||||
I would improve the odds of success by treating the next phase as an experiment in whether three specific ideas actually work:
|
||||
|
||||
1. **A use case can survive implementation change better than a conventional test script.**
|
||||
2. **Agentic execution can bridge unstable implementation without corrupting the intended semantics.**
|
||||
3. **Successful agentic tests can crystallize into cheaper deterministic tests.**
|
||||
|
||||
If those three work, the rest—energy, campaigns, security mutation, multi-user orchestration—has a strong foundation.
|
||||
|
||||
### Narrow the first battlefield
|
||||
|
||||
Choose exactly one real application and perhaps three use cases. They should deliberately include the difficult characteristics test-driver is meant to solve: authentication, several users, state transitions, authorization, and an evolving UI/API.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
UC-01 Alice creates a workspace
|
||||
UC-02 Alice invites Bob and Bob joins
|
||||
UC-03 Alice revokes Bob and Bob loses access
|
||||
```
|
||||
|
||||
UC-03 already gives you functional, interaction, temporal and security semantics.
|
||||
|
||||
Avoid building a generic testing platform first. Make test-driver extraordinarily good at this one sequence.
|
||||
|
||||
### Make the semantic layer the core intellectual property
|
||||
|
||||
The critical interface is not the LLM integration or browser automation. It is:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
↓
|
||||
Semantic Goal
|
||||
↓
|
||||
Semantic Action
|
||||
↓
|
||||
Concrete realization
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
grant_access(Bob, resource, READ)
|
||||
```
|
||||
|
||||
may currently mean six browser interactions.
|
||||
|
||||
Later it may mean an API operation.
|
||||
|
||||
Test-driver should care about the semantic action. Drivers care about realization.
|
||||
|
||||
If this abstraction is good, the framework survives technology changes. If it is poor, agentic execution becomes sophisticated screen scraping.
|
||||
|
||||
### Force the framework to distinguish discovery from truth
|
||||
|
||||
One of the strongest architectural principles should be:
|
||||
|
||||
> **Agents discover paths. Oracles establish truth.**
|
||||
|
||||
For the first implementation, make every important oracle deterministic.
|
||||
|
||||
For example, let the agent discover how Alice invites Bob through the UI, but verify independently through an API or database-facing test interface that:
|
||||
|
||||
```text
|
||||
membership(Bob, Workspace) == MEMBER
|
||||
membership(Carol, Workspace) == NONE
|
||||
```
|
||||
|
||||
Do not let the actor agent conclude, “It looks like Bob joined.”
|
||||
|
||||
This separation will prevent many future problems.
|
||||
|
||||
### Build reproducibility before intelligence
|
||||
|
||||
For every run, capture at least:
|
||||
|
||||
```text
|
||||
use-case version
|
||||
scenario version
|
||||
application version/commit
|
||||
actor identities and roles
|
||||
initial world
|
||||
semantic actions
|
||||
actual actions
|
||||
random seed
|
||||
timestamps
|
||||
observations
|
||||
oracle results
|
||||
screenshots/traces where useful
|
||||
agent/model/config version
|
||||
```
|
||||
|
||||
An agentically discovered failure that cannot be reproduced is much less valuable.
|
||||
|
||||
The first impressive demonstration should therefore not be “the agent found a bug.”
|
||||
|
||||
It should be:
|
||||
|
||||
> “The agent found a bug, test-driver reduced it to this scenario, and the failure can now be replayed deterministically.”
|
||||
|
||||
### Treat crystallization as an explicit deliverable
|
||||
|
||||
Don't postpone crystallization until later.
|
||||
|
||||
Make the first milestone contain this lifecycle:
|
||||
|
||||
```text
|
||||
new use case
|
||||
↓
|
||||
agentic execution
|
||||
↓
|
||||
stable semantic trajectory
|
||||
↓
|
||||
candidate deterministic implementation
|
||||
↓
|
||||
deterministic regression
|
||||
```
|
||||
|
||||
You need to learn early whether this transition can actually be automated or assisted effectively.
|
||||
|
||||
A useful success metric might be:
|
||||
|
||||
> **How many agentic verification assets can be downgraded to deterministic execution without losing semantic coverage?**
|
||||
|
||||
That is much more meaningful than counting generated tests.
|
||||
|
||||
### Introduce energy only after you have event history
|
||||
|
||||
I like Test Energy a lot, but I would avoid optimizing its formula early.
|
||||
|
||||
Start by recording events:
|
||||
|
||||
```text
|
||||
found-defect
|
||||
false-positive
|
||||
mechanical-adaptation
|
||||
semantic-adaptation
|
||||
duplicate-detected
|
||||
usecase-changed
|
||||
crystallized
|
||||
regression-caught
|
||||
```
|
||||
|
||||
Then initially compute a crude score.
|
||||
|
||||
After a few hundred runs you can inspect whether the proposed energy changes actually correspond to human intuition about test value.
|
||||
|
||||
In other words:
|
||||
|
||||
> **Store the evidence first; invent the fitness function second.**
|
||||
|
||||
Otherwise you'll encode assumptions before you have data.
|
||||
|
||||
### Separate three kinds of change
|
||||
|
||||
This will probably become one of the framework's most important capabilities.
|
||||
|
||||
Whenever a test stops matching the system, classify the change as:
|
||||
|
||||
```text
|
||||
IMPLEMENTATION CHANGE
|
||||
same behavior, different realization
|
||||
|
||||
INTENT CHANGE
|
||||
the product is deliberately supposed to behave differently
|
||||
|
||||
DEFECT
|
||||
implementation no longer satisfies unchanged intent
|
||||
```
|
||||
|
||||
This classification is what makes adaptive tests safe.
|
||||
|
||||
A lot of test maintenance today implicitly assumes the first case. Test-driver must never make that assumption.
|
||||
|
||||
### Give humans a small number of meaningful decision points
|
||||
|
||||
Agentic systems fail when humans either have to approve everything or are expected to trust everything.
|
||||
|
||||
Aim for perhaps three escalation types:
|
||||
|
||||
```text
|
||||
SEMANTIC CHANGE
|
||||
"The expected workflow appears to have changed."
|
||||
|
||||
INVARIANT CONFLICT
|
||||
"The observed behavior conflicts with a protected invariant."
|
||||
|
||||
AMBIGUOUS VERDICT
|
||||
"Available evidence cannot establish correctness."
|
||||
```
|
||||
|
||||
Everything mechanical beneath that should eventually be autonomous.
|
||||
|
||||
That creates a good human/agent boundary.
|
||||
|
||||
### Make security an early proof point
|
||||
|
||||
Security testing is especially good for proving the framework because ordinary functional use cases naturally generate adversarial variants.
|
||||
|
||||
From:
|
||||
|
||||
```text
|
||||
Alice grants Bob access.
|
||||
```
|
||||
|
||||
derive:
|
||||
|
||||
```text
|
||||
Carol tries access.
|
||||
Bob attempts WRITE.
|
||||
Bob accesses before grant.
|
||||
Bob accesses after revoke.
|
||||
Bob substitutes another resource ID.
|
||||
Bob attempts cross-tenant access.
|
||||
```
|
||||
|
||||
If test-driver can systematically produce these from the ordinary use case, you already have something substantially more interesting than “an LLM controlling Playwright.”
|
||||
|
||||
### Use a deliberately boring implementation stack
|
||||
|
||||
The novelty should be in the verification model, not infrastructure.
|
||||
|
||||
For the first implementation, I'd prefer something like:
|
||||
|
||||
```text
|
||||
Python
|
||||
pytest
|
||||
Playwright
|
||||
Pydantic/dataclasses
|
||||
YAML
|
||||
SQLite
|
||||
OpenTelemetry where useful
|
||||
```
|
||||
|
||||
Maybe later a graph store, distributed workers, richer agent infrastructure, etc.
|
||||
|
||||
But initially:
|
||||
|
||||
> **One process, one database, one browser engine, one application under test.**
|
||||
|
||||
Complex infrastructure will hide conceptual flaws.
|
||||
|
||||
### Dogfood aggressively
|
||||
|
||||
Test-driver itself should become one of the systems tested by test-driver.
|
||||
|
||||
As soon as there is a usable CLI or web surface, introduce its own use cases:
|
||||
|
||||
```text
|
||||
define use case
|
||||
execute campaign
|
||||
inspect evidence
|
||||
accept finding
|
||||
crystallize test
|
||||
retire verification
|
||||
```
|
||||
|
||||
That gives you continuous pressure to make the framework usable rather than merely architecturally sophisticated.
|
||||
|
||||
### Define success quantitatively
|
||||
|
||||
I'd track a small scorecard from the first month:
|
||||
|
||||
| Measure | Why it matters |
|
||||
| -------------------------------------------- | ---------------- |
|
||||
| Time to express a new use case | usability |
|
||||
| Agentic scenario success rate | executability |
|
||||
| Mechanical adaptations handled automatically | fluidity benefit |
|
||||
| False semantic adaptations | safety |
|
||||
| Genuine defects discovered | effectiveness |
|
||||
| Defects reproducibly replayed | trustworthiness |
|
||||
| Agentic → deterministic crystallizations | core thesis |
|
||||
| Median execution cost per maturity level | economics |
|
||||
| Human interventions per 100 runs | autonomy |
|
||||
| Security variants derived per use case | leverage |
|
||||
|
||||
The key metric could eventually be something like:
|
||||
|
||||
> **Verified behavior per unit of human maintenance effort.**
|
||||
|
||||
That captures what test-driver is fundamentally trying to improve.
|
||||
|
||||
### Keep the architecture open, but the semantics opinionated
|
||||
|
||||
Drivers, agents and storage should be replaceable.
|
||||
|
||||
The following should not be casually replaceable:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
Actor
|
||||
Scenario
|
||||
Claim
|
||||
Invariant
|
||||
SemanticAction
|
||||
Observation
|
||||
Oracle
|
||||
VerificationAsset
|
||||
Finding
|
||||
Adaptation
|
||||
Crystallization
|
||||
EnergyEvent
|
||||
Lineage
|
||||
```
|
||||
|
||||
Those are the conceptual backbone.
|
||||
|
||||
If they change constantly, you don't yet understand the domain.
|
||||
|
||||
### And establish a ruthless non-goal list
|
||||
|
||||
For v0.1 I would explicitly not try to become:
|
||||
|
||||
```text
|
||||
a generic unit-test framework
|
||||
a load-testing replacement
|
||||
a fuzzing engine
|
||||
a vulnerability scanner
|
||||
a test-management SaaS
|
||||
a CI/CD system
|
||||
an observability platform
|
||||
a universal browser agent
|
||||
```
|
||||
|
||||
Test-driver may eventually integrate all of those.
|
||||
|
||||
Its unique purpose should remain:
|
||||
|
||||
> **Maintain trustworthy verification of use cases while software changes rapidly, using agentic flexibility where necessary and deterministic verification wherever possible.**
|
||||
|
||||
If I were running the project, the next concrete milestone would be a **Test Driver Research Prototype**, not yet a product: one target application, three linked multi-user use cases, deterministic oracles, one agentic browser driver, complete evidence capture, one mutation mechanism, and one successful crystallization into a deterministic regression test.
|
||||
|
||||
If that demonstrably works, we will have validated the hardest and most original part of the idea.
|
||||
|
||||
xxx
|
||||
240
history/2026-08-22-concept-assessment-swot.md
Normal file
240
history/2026-08-22-concept-assessment-swot.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# test-driver — Concept Assessment (SWOT)
|
||||
|
||||
**Date:** 2026-08-22
|
||||
**Author:** Claude (Opus 5) with Bernd Worsch
|
||||
**Scope:** Assessment of the pre-implementation concept corpus — `INTENT.md`,
|
||||
`docs/TestDriverConceptModel.md`, `docs/TestDriverImprovementLoop.md`,
|
||||
`docs/TestDriverInitialMilestones.md`, `docs/TestDriverResearchPrototype.md`.
|
||||
**Repo state at assessment:** 2,950 lines of concept documentation, 1 commit,
|
||||
no executable code, `docs/` and `INTENT.md` untracked.
|
||||
|
||||
---
|
||||
|
||||
## Orientation
|
||||
|
||||
The repository is **concept-complete and code-empty**. Four documents describe a
|
||||
use-case-driven verification framework whose thesis is that verification should
|
||||
mature alongside the behaviour it protects: fluid and agentic while software is
|
||||
hot, deterministic once it cools.
|
||||
|
||||
The ratio — roughly 3,000 lines of theory against zero lines executable — is the
|
||||
single most important fact shaping this assessment.
|
||||
|
||||
---
|
||||
|
||||
## Strengths
|
||||
|
||||
**S1 — The core thesis is original and load-bearing.**
|
||||
The crystallization continuum (T0 Exploratory → T5 Deterministic) coupled to
|
||||
implementation *Temperature* is not a repackaging of "self-healing tests". It
|
||||
makes a falsifiable claim: verification mode should be a function of how fast the
|
||||
system under test is changing.
|
||||
|
||||
**S2 — The right primitive is identified.**
|
||||
`SemanticAction` as the bridge between agentic discovery and deterministic code
|
||||
generation is the correct pivot point, and the documents know it.
|
||||
`grant_access(Bob, R, READ)` surviving a DOM restructure *is* the product.
|
||||
|
||||
**S3 — Safety is designed in, not bolted on.**
|
||||
Oracle independence from actors, "implementation is not the truth",
|
||||
`INCONCLUSIVE` as a first-class verdict, and **False Adaptation Rate** as the
|
||||
headline safety metric. This is what separates the concept from self-healing
|
||||
test vendors, which structurally cannot distinguish a moved button from a broken
|
||||
authorization check.
|
||||
|
||||
**S4 — Falsifiable research posture.**
|
||||
H-001…H-005 carry falsification conditions; the lab is a measuring instrument;
|
||||
milestone exit criteria are stated as observable outcomes. Most framework
|
||||
projects have a roadmap — this has an experiment design.
|
||||
|
||||
**S5 — Security-as-mutation is high leverage and cheap.**
|
||||
Deriving twelve adversarial variants from one shared-resource use case is
|
||||
demonstrable early, valuable independently of the crystallization thesis, and
|
||||
requires no maturity beyond T1.
|
||||
|
||||
**S6 — Anti-bloat discipline is pre-committed.**
|
||||
Explicit non-goal list, M10 Compression Review, "self-improvement includes
|
||||
subtraction", and a deliberately boring stack (Python / pytest / Playwright /
|
||||
SQLite / YAML). The documents already contain their own best critique.
|
||||
|
||||
---
|
||||
|
||||
## Weaknesses
|
||||
|
||||
**W1 — Concept surface vastly exceeds validated ground.**
|
||||
Roughly 40 canonical concepts, of which Energy, Confidence, Metabolism, Campaign
|
||||
selection, Retirement floors and Lineage graphs are untestable until M7+. Each is
|
||||
an invitation to build now and validate never.
|
||||
|
||||
**W2 — Three documents already disagree.**
|
||||
`INTENT.md` numbers milestones M0–M3; `TestDriverInitialMilestones.md` numbers
|
||||
M0–M10; `TestDriverConceptModel.md` §15 has a third M0–M3. Concept drift has
|
||||
appeared before any code exists — precisely the failure mode the Improvement Loop
|
||||
document was written to prevent.
|
||||
|
||||
**W3 — The three hardest problems are the least specified.**
|
||||
(a) How a `SemanticAction` retains *identity* across a changed surface — the
|
||||
binding and matching mechanism is nowhere described.
|
||||
(b) How adaptation is classified MECHANICAL vs SEMANTIC vs DEFECT without an LLM
|
||||
making the call, which principle 2.3 forbids.
|
||||
(c) How a captured trajectory becomes deterministic code with oracle coverage
|
||||
preserved.
|
||||
These three are the project; the remainder is scaffolding.
|
||||
|
||||
**W4 — Energy is the weakest concept and the easiest to build.**
|
||||
A dangerous combination. Proving that Energy "predicts verification value"
|
||||
requires years of history the prototype will never accumulate. M9 exists mainly
|
||||
to collect a dataset for a study that will not happen at this stage.
|
||||
|
||||
**W5 — The evaluation set is far too small for its claims.**
|
||||
Six seeded mutations (M01–M06) cannot support "classification precision/recall".
|
||||
Dozens of labelled mutations across several dimensions are needed before any rate
|
||||
is more than anecdote.
|
||||
|
||||
**W6 — Self-verification is circular.**
|
||||
`td://self/oracle-independence` uses the framework to test whether the
|
||||
framework's oracles are independent. Without an out-of-band assertion layer
|
||||
(plain pytest) as ground truth, M3 proves nothing.
|
||||
|
||||
**W7 — No cost or nondeterminism model.**
|
||||
Agentic runs are stochastic and expensive. Nothing budgets for model flake,
|
||||
retries, or per-run cost — yet "execution cost reduction" is a stated M7 success
|
||||
metric.
|
||||
|
||||
---
|
||||
|
||||
## Opportunities
|
||||
|
||||
**O1 — The timing window is open now.**
|
||||
Agentic development is outrunning test maintenance in exactly the way the *Why*
|
||||
section describes. The pain is acute and current.
|
||||
|
||||
**O2 — The lab is a publishable asset in its own right.**
|
||||
A benchmark of seeded mechanical / semantic / defect mutations with ground-truth
|
||||
labels is something the field lacks entirely. It could earn credibility and
|
||||
contributors faster than the framework does, at a fraction of the cost.
|
||||
|
||||
**O3 — "False Adaptation Rate" is a naming land-grab.**
|
||||
If self-healing test tools come to be held against a metric this project defined,
|
||||
the framing is won regardless of adoption.
|
||||
|
||||
**O4 — Integration rather than replacement lowers adoption cost to near zero.**
|
||||
Crystallization that emits ordinary pytest / Playwright files drops into CI
|
||||
systems that already exist. Users can adopt the output without adopting the
|
||||
framework — and then adopt the framework to keep producing it.
|
||||
|
||||
**O5 — Agent-native distribution.**
|
||||
Exposed over MCP, this becomes the verification layer coding agents call on
|
||||
themselves — a far larger surface than "a test framework a human runs".
|
||||
|
||||
**O6 — A real second system under test is already available.**
|
||||
The Custodian ecosystem (multi-domain, cross-repo, permissioned) is a better
|
||||
dogfooding target than a toy lab and exercises the multi-user and tenant-boundary
|
||||
claims honestly.
|
||||
|
||||
---
|
||||
|
||||
## Threats
|
||||
|
||||
**T1 — One public false adaptation kills the thesis.**
|
||||
If the framework ever normalises a genuine authorization defect as a legitimate
|
||||
mechanical change, the concept is dead and cannot be rescued by a better version.
|
||||
This asymmetry should shape every design decision.
|
||||
|
||||
**T2 — The independence problem is philosophically serious.**
|
||||
If an agent writes the code, an agent writes the use case, and an agent realises
|
||||
the test, then "oracles independent from actors" is procedurally true but
|
||||
epistemically thin. Intent artefacts need human or spec-derived provenance, or
|
||||
the guarantee is a shell game. This is not addressed anywhere in the documents.
|
||||
|
||||
**T3 — Competitors with distribution.**
|
||||
Testim / mabl / Functionize on the self-healing axis; agentic QA startups on the
|
||||
exploration axis; Playwright plus a competent agent covering the naive 80%. The
|
||||
differentiator — defect-vs-adaptation discrimination — must be *demonstrated*,
|
||||
not described, and the window is roughly 12–18 months.
|
||||
|
||||
**T4 — The economics may not close.**
|
||||
If agentic realisation costs more per run than simply asking an agent to rewrite
|
||||
the broken test, crystallization becomes an aesthetic preference rather than a
|
||||
value proposition. This must be measured early, not at M7.
|
||||
|
||||
**T5 — Platform drift before evidence.**
|
||||
Eleven milestones, a research control plane, five agentic roles and a ten-
|
||||
dimension fitness scorecard — for a project with no runnable code and one
|
||||
maintainer.
|
||||
|
||||
---
|
||||
|
||||
## Recommendations for the first workplan
|
||||
|
||||
**R1 — Change the sequence: build a narrow vertical spike, not layered
|
||||
milestones.** The documented order (M0 registers → M1 kernel → M2 lab → M3
|
||||
self-verification → M4 agentic …) completes four layers before the thesis is
|
||||
touched once. Invert it: drive one thread end to end — one use case
|
||||
(Alice/Bob/Carol) → deterministic kernel → lab with three mutations (M01 moved
|
||||
control, M02 changed DOM, M05 authorization defect) → agentic realisation of
|
||||
exactly one semantic action → adaptation classification → one crystallization.
|
||||
Thin at every layer, complete end to end. That addresses H-001/H-002/H-003 in one
|
||||
workplan instead of seven.
|
||||
|
||||
**R2 — Make the classifier the centrepiece, not a downstream milestone.**
|
||||
M6 (defect vs adaptation) is where the project either has a reason to exist or
|
||||
does not. Design the classification mechanism *first* — specifically, what
|
||||
deterministic evidence separates "the button moved" from "Bob can still read
|
||||
after revoke". If the honest answer is "an LLM decides", that violates principle
|
||||
2.3 and must be resolved on paper before code.
|
||||
|
||||
**R3 — Cap M0 at roughly half a day.**
|
||||
Five hypothesis files, one fitness map, an ID convention. No CLI, no register
|
||||
tooling, no schema. The research control plane is overhead until there are
|
||||
readings to record.
|
||||
|
||||
**R4 — Defer Energy, Temperature, Confidence, Campaigns and Retirement
|
||||
entirely.** Record raw immutable `EnergyEvent`s from day one — they cost nothing
|
||||
and cannot be reconstructed later — but implement no scoring, decay or selection
|
||||
logic. Each of these is cheap to build, satisfying to build, and impossible to
|
||||
validate at this stage.
|
||||
|
||||
**R5 — Build the lab larger than feels necessary, and label ground truth.**
|
||||
It is the measuring instrument for every claim the project makes; a weak lab caps
|
||||
the credibility of all downstream results. Target 15–20 labelled mutations rather
|
||||
than 6, each a reproducible toggle with a recorded expected classification. This
|
||||
is also O2 — the standalone asset.
|
||||
|
||||
**R6 — Establish out-of-band ground truth before self-verification.**
|
||||
Write the actor-isolation and oracle-independence checks as plain pytest against
|
||||
observable behaviour, outside the framework. Otherwise M3 is a system certifying
|
||||
itself.
|
||||
|
||||
**R7 — Instrument cost and nondeterminism from the first agentic run.**
|
||||
Tokens, wall time, retry count, run-to-run variance per maturity level. T4 is an
|
||||
existential economic question; the data is free to collect from run one and
|
||||
impossible to backfill.
|
||||
|
||||
**R8 — Reconcile the three milestone numberings and commit `docs/`.**
|
||||
Pick one canonical sequence and mark the others superseded. Untracked, mutually
|
||||
inconsistent design documents are exactly the CONCEPT_DRIFT finding the
|
||||
Improvement Loop defines — the project should catch it on itself before it
|
||||
catches anything else.
|
||||
|
||||
**R9 — Fix a single quantitative gate for the first workplan.**
|
||||
Proposed: *agentic realisation recovers from M01 and M02 with zero claim or
|
||||
invariant changes, flags M05 as a Product Finding, and produces one deterministic
|
||||
test that runs with no model involvement.* Achieving that is worth more than any
|
||||
other ten items on the roadmap; failing it is worth knowing early and cheaply.
|
||||
|
||||
**R10 — Decide where use-case intent comes from.**
|
||||
Record, in the first workplan, whether use cases are human-authored,
|
||||
spec-derived, or agent-generated, and what independence guarantee survives in
|
||||
each case (T2). This determines whether the central promise is real or
|
||||
procedural — a paragraph of thinking now versus a redesign later.
|
||||
|
||||
---
|
||||
|
||||
## Summary judgement
|
||||
|
||||
The concept is strong, unusually well-critiqued by its own documents, and
|
||||
correctly identifies its own hardest problem. It is also roughly 3,000 lines of
|
||||
theory ahead of its evidence. The first workplan's job is to close that gap with
|
||||
the thinnest possible slice capable of falsifying the thesis — not to build the
|
||||
architecture the documents describe.
|
||||
61
workplans/TD-WP-0001-statehub-bootstrap.md
Normal file
61
workplans/TD-WP-0001-statehub-bootstrap.md
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
id: TD-WP-0001
|
||||
type: workplan
|
||||
title: "Bootstrap State Hub integration"
|
||||
domain: infotech
|
||||
repo: test-driver
|
||||
status: ready
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-08-22"
|
||||
updated: "2026-08-22"
|
||||
---
|
||||
|
||||
# Bootstrap State Hub integration
|
||||
|
||||
Use-case-driven verification framework for integration, end-to-end, multi-user, authorization and security testing that matures tests from agentic exploration into deterministic regression.
|
||||
|
||||
## Review Generated Integration Files
|
||||
|
||||
```task
|
||||
id: TD-WP-0001-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`.
|
||||
Replace generated placeholders with repo-specific facts where needed.
|
||||
|
||||
## Verify Local Developer Workflow
|
||||
|
||||
```task
|
||||
id: TD-WP-0001-T02
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Identify the repo's install, test, lint, build, and run commands. Add or refine
|
||||
those commands in the agent instructions so future coding sessions can verify
|
||||
changes confidently.
|
||||
|
||||
## Seed First Real Workplan
|
||||
|
||||
```task
|
||||
id: TD-WP-0001-T03
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Create the first implementation workplan for the repository's most important
|
||||
next change. After workplan file updates, run the sync locally from this repo
|
||||
checkout:
|
||||
|
||||
```bash
|
||||
statehub fix-consistency
|
||||
```
|
||||
|
||||
Blocked until the stack exists: no code, no dependency manifest and no test
|
||||
runner are present yet. Unblocks with TD-WP-0002-T04 (deterministic semantic
|
||||
kernel), which introduces the first Python package and pytest configuration.
|
||||
|
||||
Seeded workplan: `workplans/TD-WP-0002-vertical-spike-crystallization.md`.
|
||||
275
workplans/TD-WP-0002-vertical-spike-crystallization.md
Normal file
275
workplans/TD-WP-0002-vertical-spike-crystallization.md
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
---
|
||||
id: TD-WP-0002
|
||||
type: workplan
|
||||
title: "Vertical spike: falsify the crystallization thesis"
|
||||
domain: infotech
|
||||
repo: test-driver
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-08-22"
|
||||
updated: "2026-08-22"
|
||||
---
|
||||
|
||||
# Vertical spike: falsify the crystallization thesis
|
||||
|
||||
## Why this workplan exists
|
||||
|
||||
The concept corpus (`INTENT.md`, `docs/`) describes eleven milestones that build
|
||||
four complete layers — research control plane, deterministic kernel, lab,
|
||||
self-verification — before the central thesis is exercised even once. The repo
|
||||
currently holds ~3,000 lines of theory and zero lines of executable code.
|
||||
|
||||
This workplan inverts that order. It drives **one thin thread end to end**
|
||||
through every layer of the model, so that the thesis can be supported or
|
||||
falsified cheaply and early:
|
||||
|
||||
```
|
||||
one use case → deterministic kernel → labelled mutation lab
|
||||
→ agentic realisation of one semantic action
|
||||
→ adaptation classification → one crystallization
|
||||
```
|
||||
|
||||
Reasoning and the full assessment behind this sequencing:
|
||||
`history/2026-08-22-concept-assessment-swot.md`.
|
||||
|
||||
## Success gate
|
||||
|
||||
The spike succeeds when, in one coherent run:
|
||||
|
||||
1. agentic realisation recovers from lab mutations **M01** (moved/renamed
|
||||
control) and **M02** (restructured DOM) with **zero** changes to claims or
|
||||
invariants;
|
||||
2. lab mutation **M05** (authorization defect) is reported as a **Product
|
||||
Finding**, not adapted to;
|
||||
3. one verification asset crystallizes into a deterministic test that runs with
|
||||
**no model involvement** and preserves the relevant oracles;
|
||||
4. **False Adaptation Rate = 0** across the labelled mutation set.
|
||||
|
||||
Failing this gate early and cheaply is a valid and valuable outcome. Passing it
|
||||
is worth more than any other ten items on the M0–M10 roadmap.
|
||||
|
||||
## Explicitly deferred
|
||||
|
||||
Not in this workplan, by decision rather than omission — each is cheap to build,
|
||||
satisfying to build, and impossible to validate at this stage:
|
||||
|
||||
- Energy scoring, decay, and campaign selection (raw immutable `EnergyEvent`s are
|
||||
recorded from the first run; **no** scoring logic is implemented);
|
||||
- Temperature and Confidence as computed values;
|
||||
- Test Metabolism, Campaigns, Retirement floors;
|
||||
- Lineage graph storage beyond a parent pointer;
|
||||
- the five agentic roles (Builder / Experimenter / Critic / Auditor / Maintainer);
|
||||
- any surface beyond one HTTP API and one browser UI.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Stack: Python, pytest, Playwright, Pydantic/dataclasses, YAML, SQLite.
|
||||
One process, one database, one browser engine, one application under test.
|
||||
- Novelty belongs in the verification model, never in the infrastructure.
|
||||
- No LLM judgment where a deterministic oracle is available (Concept Model §2.3).
|
||||
|
||||
---
|
||||
|
||||
## Reconcile milestone numbering and commit the concept corpus
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Three documents carry three different milestone sequences: `INTENT.md` (M0–M3),
|
||||
`docs/TestDriverInitialMilestones.md` (M0–M10), and
|
||||
`docs/TestDriverConceptModel.md` §15 (a third M0–M3). This is a CONCEPT_DRIFT
|
||||
finding by the project's own taxonomy, present before any code exists.
|
||||
|
||||
Pick one canonical sequence, mark the others superseded in place, and commit
|
||||
`INTENT.md` and `docs/` to git (currently untracked). Record the drift as the
|
||||
first entry in the framework findings log — the project should catch this on
|
||||
itself before it catches anything else.
|
||||
|
||||
## Decide intent provenance and the classification mechanism on paper
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Two design questions gate everything downstream. Answer both in a written design
|
||||
note plus a State Hub decision record, **before** writing kernel code.
|
||||
|
||||
**(a) The classifier.** What *deterministic evidence* separates "the button
|
||||
moved" from "Bob can still read after revoke"? Specify the signal, not the
|
||||
intent. If the honest answer is "a model decides", that violates Concept Model
|
||||
§2.3 and must be resolved now rather than discovered at M6.
|
||||
|
||||
**(b) Intent provenance.** If an agent writes the implementation, an agent writes
|
||||
the use case, and an agent realises the test, then "oracles independent from
|
||||
actors" is procedurally true but epistemically thin. Record whether use cases are
|
||||
human-authored, spec-derived, or agent-generated, and what independence guarantee
|
||||
survives in each case.
|
||||
|
||||
Deliverable: `docs/TestDriverClassificationDesign.md` + one recorded decision.
|
||||
|
||||
## Minimal research control plane
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Timebox: half a day. Five hypothesis files (H-001…H-005) each with an explicit
|
||||
falsification condition, one Concept ↔ Implementation Fitness Map, and a stable
|
||||
identifier convention. Plain files under `research/`.
|
||||
|
||||
No CLI, no register tooling, no schema. The control plane is overhead until there
|
||||
are readings to record.
|
||||
|
||||
## Deterministic semantic kernel
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Minimal executable representations of UseCase, Actor, World, Scenario,
|
||||
SemanticAction, Observation, Evidence, Oracle, Verdict, VerificationAsset, Run.
|
||||
One deterministic driver (HTTP or direct adapter).
|
||||
|
||||
Run the reference use case end to end:
|
||||
|
||||
```
|
||||
Alice owns resource R. Alice grants Bob READ. Bob can read R.
|
||||
Carol cannot read R. Alice revokes Bob. Bob can no longer read R.
|
||||
```
|
||||
|
||||
Exit: Alice, Bob and Carol hold genuinely isolated identities and sessions; all
|
||||
outcomes are judged by independent deterministic oracles; a run emits a
|
||||
structured Evidence Pack; the scenario replays from known initial state.
|
||||
|
||||
Emit raw `EnergyEvent` records from this point onward. Implement no scoring.
|
||||
|
||||
## Test-driver lab with labelled ground truth
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
A deliberately small application: users, tenants, auth, resources, sharing,
|
||||
read/write permissions, revoke, audit history, HTTP API, minimal browser UI.
|
||||
|
||||
Build **15–20 labelled mutations**, not the six sketched in the milestones doc.
|
||||
Six cannot support any statement about precision or recall. Each mutation is a
|
||||
reproducible toggle carrying a recorded expected classification
|
||||
(`MECHANICAL` / `SEMANTIC` / `DEFECT`), with baseline and mutated versions
|
||||
carrying explicit version identifiers.
|
||||
|
||||
The lab is the measuring instrument for every claim the framework makes — a weak
|
||||
lab caps the credibility of all downstream results. It is also potentially the
|
||||
project's first publishable artefact in its own right.
|
||||
|
||||
## Out-of-band ground truth for self-verification
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Write the actor-isolation and oracle-independence checks as **plain pytest**
|
||||
against observable behaviour, outside the framework. Using test-driver to verify
|
||||
that test-driver's oracles are independent is a system certifying itself.
|
||||
|
||||
Exit: deliberately breaking actor isolation fails the out-of-band check;
|
||||
allowing an actor to determine its own verdict fails oracle-independence;
|
||||
each failure produces a Framework Finding.
|
||||
|
||||
## Agentic realisation of one semantic action, fully instrumented
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T07
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Browser driver plus one agentic actor runtime, realising exactly one semantic
|
||||
action — `grant_access(Bob, R, READ)` — from intent, against the lab UI. Strict
|
||||
per-actor context isolation; bounded navigation and tool permissions; full
|
||||
action and evidence recording; agent/model/configuration identity captured in
|
||||
evidence. Oracles stay deterministic.
|
||||
|
||||
**Instrument cost and nondeterminism from the very first run:** tokens, wall
|
||||
time, retry count, and run-to-run variance. If agentic realisation costs more per
|
||||
run than simply asking an agent to rewrite the broken test, the crystallization
|
||||
argument is an aesthetic preference rather than a value proposition. This data is
|
||||
free to collect from run one and impossible to backfill.
|
||||
|
||||
## Adaptation detection and the defect-vs-adaptation classifier
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T08
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
The centrepiece. Implement the mechanism designed in T02 and run it across the
|
||||
full labelled mutation set from T05.
|
||||
|
||||
Classification path: `IMPLEMENTATION CHANGE` / `INTENT CHANGE` /
|
||||
`PRODUCT DEFECT` / `AMBIGUOUS`, with escalation for semantic changes and
|
||||
ambiguity. Preserve semantic action identity across adaptation; record adaptation
|
||||
history on the verification asset; never modify claims or invariants to
|
||||
accommodate an observed behaviour.
|
||||
|
||||
Measure: Mechanical Recovery Rate, classification precision/recall, and
|
||||
**False Adaptation Rate**. The last is the project's existential safety metric —
|
||||
one publicly normalised authorization defect kills the thesis permanently. Target
|
||||
is zero across the set, and a non-zero result is a stop-and-redesign signal, not
|
||||
a tuning exercise.
|
||||
|
||||
## Crystallize one asset into deterministic regression
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T09
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Semantic action trajectory capture, stable-realisation detection, deterministic
|
||||
candidate generation, candidate comparison against the existing oracle set, and
|
||||
lineage from the agentic ancestor to the deterministic descendant. Manual
|
||||
acceptance step for now.
|
||||
|
||||
Emit ordinary pytest/Playwright code. Output that drops into a CI system which
|
||||
already exists lets a user adopt the result without adopting the framework.
|
||||
|
||||
Exit: the generated test runs with zero agentic involvement, preserves the
|
||||
relevant claims and oracles, retains visible lineage, and measurably costs less
|
||||
to execute than the agentic ancestor.
|
||||
|
||||
## Gate review and first compression pass
|
||||
|
||||
```task
|
||||
id: TD-WP-0002-T10
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Evaluate the four success-gate criteria against collected evidence and write the
|
||||
result up in `history/` regardless of outcome.
|
||||
|
||||
Then run the compression questions from Milestone 10 while the spike is still
|
||||
small: which concepts have no supporting evidence, which abstractions have no
|
||||
conceptual justification, which metadata never informed a decision, what did the
|
||||
experiments falsify. Mark rejected concepts as rejected rather than silently
|
||||
retaining them, and update the Concept ↔ Implementation Fitness Map.
|
||||
|
||||
Convert at least one framework finding discovered during T04–T09 into a permanent
|
||||
self-verification or deterministic regression — that closes the first genuine
|
||||
Concept–Implementation Fitness Loop.
|
||||
Loading…
Add table
Add a link
Reference in a new issue