coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
test-driver/research/findings
History
tegwick 44faf3de8e T07: agentic realization over a stdlib browser surface 
Two decisions taken with the operator: stdlib HTML driver instead of
Playwright (F-0004), and a deterministic discovery runtime instead of a live
model. Both sit behind interfaces so the alternatives drop in later.

- html.py: stdlib DOM parse and query
- agentic.py: DiscoveryRuntime (agentic arm, ignores data-td by construction)
  and RecordedSelectorRuntime (control arm, uses the strongest identifier the
  page offers)
- browser.py: per-actor sessions over real HTTP, constructed per call so no
  actor inherits another's connection state
- cost/nondeterminism metrics recorded from the first run

F-0005 (CONCEPT_DRIFT): the H-001 result is a narrowing. Where test ids are
preserved, discovery 9/9 and recorded selectors 9/9 - the semantic action buys
nothing. Where they are dropped, discovery 2/3 and recorded 0/3. The concept
model presents semantic actions as generally superior; the evidence says
conditionally superior.

M21 and M22 added mid-task: the deciding side of the axis was N=1. M22 (field
names renamed) defeats the heuristic and is the first concrete evidence that a
live model would add capability, not just cost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
2026-08-22 23:50:29 +02:00
..
F-0001-milestone-sequence-drift.md T01: establish canonical milestone sequence, record F-0001 2026-08-22 23:07:31 +02:00
F-0002-scenario-coverage-gap.md T05: the lab and its labelled mutation catalogue 2026-08-22 23:31:22 +02:00
F-0003-isolation-needs-canaries.md T06: out-of-band self-verification 2026-08-22 23:38:23 +02:00
F-0004-stdlib-driver-instead-of-playwright.md T07: agentic realization over a stdlib browser surface 2026-08-22 23:50:29 +02:00
F-0005-semantic-actions-earn-their-keep-narrowly.md T07: agentic realization over a stdlib browser surface 2026-08-22 23:50:29 +02:00
README.md T01: establish canonical milestone sequence, record F-0001 2026-08-22 23:07:31 +02:00

README.md

Framework Findings

Findings about test-driver itself, kept strictly separate from findings about any system under test. Classes are defined in docs/TestDriverImprovementLoop.md § 9:

PRODUCT_DEFECT · TEST_DEFECT · MECHANICAL_ADAPTATION · SEMANTIC_CHANGE · CONCEPT_DRIFT · FRAMEWORK_LIMITATION · EVIDENCE_FAILURE · UNNECESSARY_COMPLEXITY

File naming: F-NNNN-<slug>.md. Identifiers are stable and never reused.

Each finding carries frontmatter (id, class, status, dates, originating workplan/task) and, when resolved, states which of the three §5 resolution paths was taken. A finding that leaves an open question records it under "Residual" and names the task that will answer it — residuals are not parked in prose.

The wider research control plane (hypotheses/, experiments/, concepts/, decisions/) is built in TD-WP-0002-T03.