Register with State Hub, persist concept assessment, seed first workplan
- history/2026-08-22-concept-assessment-swot.md: SWOT assessment of the concept corpus with recommendations for the first workplan - statehub register: infotech domain, TD-WP prefix, generated AGENTS.md, .custodian-brief.md and TD-WP-0001 bootstrap workplan - .repo-classification.yaml: category research, domain infotech - SCOPE.md rewritten with real repo boundaries - TD-WP-0002: vertical spike reordering M0-M10 into one end-to-end thread that can falsify the crystallization thesis early - commit previously untracked INTENT.md and docs/ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
b471bed707
commit
7249c6a403
13 changed files with 3900 additions and 0 deletions
338
docs/TestDriverResearchPrototype.md
Normal file
338
docs/TestDriverResearchPrototype.md
Normal file
|
|
@ -0,0 +1,338 @@
|
|||
# Stage 1 Test Driver Validation
|
||||
|
||||
The biggest risk is not technical feasibility. It is that **test-driver becomes conceptually elegant but too broad to prove itself quickly**.
|
||||
|
||||
I would improve the odds of success by treating the next phase as an experiment in whether three specific ideas actually work:
|
||||
|
||||
1. **A use case can survive implementation change better than a conventional test script.**
|
||||
2. **Agentic execution can bridge unstable implementation without corrupting the intended semantics.**
|
||||
3. **Successful agentic tests can crystallize into cheaper deterministic tests.**
|
||||
|
||||
If those three work, the rest—energy, campaigns, security mutation, multi-user orchestration—has a strong foundation.
|
||||
|
||||
### Narrow the first battlefield
|
||||
|
||||
Choose exactly one real application and perhaps three use cases. They should deliberately include the difficult characteristics test-driver is meant to solve: authentication, several users, state transitions, authorization, and an evolving UI/API.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
UC-01 Alice creates a workspace
|
||||
UC-02 Alice invites Bob and Bob joins
|
||||
UC-03 Alice revokes Bob and Bob loses access
|
||||
```
|
||||
|
||||
UC-03 already gives you functional, interaction, temporal and security semantics.
|
||||
|
||||
Avoid building a generic testing platform first. Make test-driver extraordinarily good at this one sequence.
|
||||
|
||||
### Make the semantic layer the core intellectual property
|
||||
|
||||
The critical interface is not the LLM integration or browser automation. It is:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
↓
|
||||
Semantic Goal
|
||||
↓
|
||||
Semantic Action
|
||||
↓
|
||||
Concrete realization
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
grant_access(Bob, resource, READ)
|
||||
```
|
||||
|
||||
may currently mean six browser interactions.
|
||||
|
||||
Later it may mean an API operation.
|
||||
|
||||
Test-driver should care about the semantic action. Drivers care about realization.
|
||||
|
||||
If this abstraction is good, the framework survives technology changes. If it is poor, agentic execution becomes sophisticated screen scraping.
|
||||
|
||||
### Force the framework to distinguish discovery from truth
|
||||
|
||||
One of the strongest architectural principles should be:
|
||||
|
||||
> **Agents discover paths. Oracles establish truth.**
|
||||
|
||||
For the first implementation, make every important oracle deterministic.
|
||||
|
||||
For example, let the agent discover how Alice invites Bob through the UI, but verify independently through an API or database-facing test interface that:
|
||||
|
||||
```text
|
||||
membership(Bob, Workspace) == MEMBER
|
||||
membership(Carol, Workspace) == NONE
|
||||
```
|
||||
|
||||
Do not let the actor agent conclude, “It looks like Bob joined.”
|
||||
|
||||
This separation will prevent many future problems.
|
||||
|
||||
### Build reproducibility before intelligence
|
||||
|
||||
For every run, capture at least:
|
||||
|
||||
```text
|
||||
use-case version
|
||||
scenario version
|
||||
application version/commit
|
||||
actor identities and roles
|
||||
initial world
|
||||
semantic actions
|
||||
actual actions
|
||||
random seed
|
||||
timestamps
|
||||
observations
|
||||
oracle results
|
||||
screenshots/traces where useful
|
||||
agent/model/config version
|
||||
```
|
||||
|
||||
An agentically discovered failure that cannot be reproduced is much less valuable.
|
||||
|
||||
The first impressive demonstration should therefore not be “the agent found a bug.”
|
||||
|
||||
It should be:
|
||||
|
||||
> “The agent found a bug, test-driver reduced it to this scenario, and the failure can now be replayed deterministically.”
|
||||
|
||||
### Treat crystallization as an explicit deliverable
|
||||
|
||||
Don't postpone crystallization until later.
|
||||
|
||||
Make the first milestone contain this lifecycle:
|
||||
|
||||
```text
|
||||
new use case
|
||||
↓
|
||||
agentic execution
|
||||
↓
|
||||
stable semantic trajectory
|
||||
↓
|
||||
candidate deterministic implementation
|
||||
↓
|
||||
deterministic regression
|
||||
```
|
||||
|
||||
You need to learn early whether this transition can actually be automated or assisted effectively.
|
||||
|
||||
A useful success metric might be:
|
||||
|
||||
> **How many agentic verification assets can be downgraded to deterministic execution without losing semantic coverage?**
|
||||
|
||||
That is much more meaningful than counting generated tests.
|
||||
|
||||
### Introduce energy only after you have event history
|
||||
|
||||
I like Test Energy a lot, but I would avoid optimizing its formula early.
|
||||
|
||||
Start by recording events:
|
||||
|
||||
```text
|
||||
found-defect
|
||||
false-positive
|
||||
mechanical-adaptation
|
||||
semantic-adaptation
|
||||
duplicate-detected
|
||||
usecase-changed
|
||||
crystallized
|
||||
regression-caught
|
||||
```
|
||||
|
||||
Then initially compute a crude score.
|
||||
|
||||
After a few hundred runs you can inspect whether the proposed energy changes actually correspond to human intuition about test value.
|
||||
|
||||
In other words:
|
||||
|
||||
> **Store the evidence first; invent the fitness function second.**
|
||||
|
||||
Otherwise you'll encode assumptions before you have data.
|
||||
|
||||
### Separate three kinds of change
|
||||
|
||||
This will probably become one of the framework's most important capabilities.
|
||||
|
||||
Whenever a test stops matching the system, classify the change as:
|
||||
|
||||
```text
|
||||
IMPLEMENTATION CHANGE
|
||||
same behavior, different realization
|
||||
|
||||
INTENT CHANGE
|
||||
the product is deliberately supposed to behave differently
|
||||
|
||||
DEFECT
|
||||
implementation no longer satisfies unchanged intent
|
||||
```
|
||||
|
||||
This classification is what makes adaptive tests safe.
|
||||
|
||||
A lot of test maintenance today implicitly assumes the first case. Test-driver must never make that assumption.
|
||||
|
||||
### Give humans a small number of meaningful decision points
|
||||
|
||||
Agentic systems fail when humans either have to approve everything or are expected to trust everything.
|
||||
|
||||
Aim for perhaps three escalation types:
|
||||
|
||||
```text
|
||||
SEMANTIC CHANGE
|
||||
"The expected workflow appears to have changed."
|
||||
|
||||
INVARIANT CONFLICT
|
||||
"The observed behavior conflicts with a protected invariant."
|
||||
|
||||
AMBIGUOUS VERDICT
|
||||
"Available evidence cannot establish correctness."
|
||||
```
|
||||
|
||||
Everything mechanical beneath that should eventually be autonomous.
|
||||
|
||||
That creates a good human/agent boundary.
|
||||
|
||||
### Make security an early proof point
|
||||
|
||||
Security testing is especially good for proving the framework because ordinary functional use cases naturally generate adversarial variants.
|
||||
|
||||
From:
|
||||
|
||||
```text
|
||||
Alice grants Bob access.
|
||||
```
|
||||
|
||||
derive:
|
||||
|
||||
```text
|
||||
Carol tries access.
|
||||
Bob attempts WRITE.
|
||||
Bob accesses before grant.
|
||||
Bob accesses after revoke.
|
||||
Bob substitutes another resource ID.
|
||||
Bob attempts cross-tenant access.
|
||||
```
|
||||
|
||||
If test-driver can systematically produce these from the ordinary use case, you already have something substantially more interesting than “an LLM controlling Playwright.”
|
||||
|
||||
### Use a deliberately boring implementation stack
|
||||
|
||||
The novelty should be in the verification model, not infrastructure.
|
||||
|
||||
For the first implementation, I'd prefer something like:
|
||||
|
||||
```text
|
||||
Python
|
||||
pytest
|
||||
Playwright
|
||||
Pydantic/dataclasses
|
||||
YAML
|
||||
SQLite
|
||||
OpenTelemetry where useful
|
||||
```
|
||||
|
||||
Maybe later a graph store, distributed workers, richer agent infrastructure, etc.
|
||||
|
||||
But initially:
|
||||
|
||||
> **One process, one database, one browser engine, one application under test.**
|
||||
|
||||
Complex infrastructure will hide conceptual flaws.
|
||||
|
||||
### Dogfood aggressively
|
||||
|
||||
Test-driver itself should become one of the systems tested by test-driver.
|
||||
|
||||
As soon as there is a usable CLI or web surface, introduce its own use cases:
|
||||
|
||||
```text
|
||||
define use case
|
||||
execute campaign
|
||||
inspect evidence
|
||||
accept finding
|
||||
crystallize test
|
||||
retire verification
|
||||
```
|
||||
|
||||
That gives you continuous pressure to make the framework usable rather than merely architecturally sophisticated.
|
||||
|
||||
### Define success quantitatively
|
||||
|
||||
I'd track a small scorecard from the first month:
|
||||
|
||||
| Measure | Why it matters |
|
||||
| -------------------------------------------- | ---------------- |
|
||||
| Time to express a new use case | usability |
|
||||
| Agentic scenario success rate | executability |
|
||||
| Mechanical adaptations handled automatically | fluidity benefit |
|
||||
| False semantic adaptations | safety |
|
||||
| Genuine defects discovered | effectiveness |
|
||||
| Defects reproducibly replayed | trustworthiness |
|
||||
| Agentic → deterministic crystallizations | core thesis |
|
||||
| Median execution cost per maturity level | economics |
|
||||
| Human interventions per 100 runs | autonomy |
|
||||
| Security variants derived per use case | leverage |
|
||||
|
||||
The key metric could eventually be something like:
|
||||
|
||||
> **Verified behavior per unit of human maintenance effort.**
|
||||
|
||||
That captures what test-driver is fundamentally trying to improve.
|
||||
|
||||
### Keep the architecture open, but the semantics opinionated
|
||||
|
||||
Drivers, agents and storage should be replaceable.
|
||||
|
||||
The following should not be casually replaceable:
|
||||
|
||||
```text
|
||||
UseCase
|
||||
Actor
|
||||
Scenario
|
||||
Claim
|
||||
Invariant
|
||||
SemanticAction
|
||||
Observation
|
||||
Oracle
|
||||
VerificationAsset
|
||||
Finding
|
||||
Adaptation
|
||||
Crystallization
|
||||
EnergyEvent
|
||||
Lineage
|
||||
```
|
||||
|
||||
Those are the conceptual backbone.
|
||||
|
||||
If they change constantly, you don't yet understand the domain.
|
||||
|
||||
### And establish a ruthless non-goal list
|
||||
|
||||
For v0.1 I would explicitly not try to become:
|
||||
|
||||
```text
|
||||
a generic unit-test framework
|
||||
a load-testing replacement
|
||||
a fuzzing engine
|
||||
a vulnerability scanner
|
||||
a test-management SaaS
|
||||
a CI/CD system
|
||||
an observability platform
|
||||
a universal browser agent
|
||||
```
|
||||
|
||||
Test-driver may eventually integrate all of those.
|
||||
|
||||
Its unique purpose should remain:
|
||||
|
||||
> **Maintain trustworthy verification of use cases while software changes rapidly, using agentic flexibility where necessary and deterministic verification wherever possible.**
|
||||
|
||||
If I were running the project, the next concrete milestone would be a **Test Driver Research Prototype**, not yet a product: one target application, three linked multi-user use cases, deterministic oracles, one agentic browser driver, complete evidence capture, one mutation mechanism, and one successful crystallization into a deterministic regression test.
|
||||
|
||||
If that demonstrably works, we will have validated the hardest and most original part of the idea.
|
||||
|
||||
xxx
|
||||
Loading…
Add table
Add a link
Reference in a new issue