coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Complete local generalisation tasks and document remaining experiment blockers

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4
This commit is contained in:
tegwick 2026-09-28 12:06:24 +02:00
parent d54af628df
commit cf682281d7
36 changed files with 5394 additions and 279 deletions
Download patch file Download diff file Expand all files Collapse all files

37
lab/mutations.py
View file

@ -1,12 +1,13 @@
"""The labelled mutation catalogue — the project's measuring instrument.
Every claim test-driver makes is measured against this catalogue, so its quality
The structural adaptation claims are measured against this catalogue, so its quality
caps the credibility of every downstream result. Six mutations, as the milestones
document originally sketched, cannot support any statement about precision or
recall; there are twenty here.
recall; there are 38 here after TD-WP-0003-T03.
Each mutation carries a **ground-truth label**, decided by a human from the use
case and recorded before any run:
Each mutation carries a **ground-truth label** recorded before its measurement.
The original labels came with the use case; M25–M38 are explicitly agent-authored
synthetic extensions (not newly human-reviewed):
MECHANICAL the surface changed; protected semantics are identical.
test-driver should recover and report an adaptation.
@ -216,11 +217,37 @@ CATALOGUE: tuple[Mutation, ...] = (
"A user of another tenant reads the resource.", _m20),
)
# T03 extension: labels fixed from the synthetic contract before execution.
# These are agent-authored instrument variants, not newly human-reviewed labels.
# Paired variants isolate identifier loss from the structural mechanism.
EXTENDED_LAYOUTS = (
("M25", "M32", "Reordered fields", {"ui_field_order": "reversed"}),
("M26", "M33", "Relocated into an open dialog", {"ui_share_control": "modal"}),
("M27", "M34", "Anchor affordances", {"ui_button_element": "anchor"}),
("M28", "M35", "Fields grouped in a fieldset", {"ui_form_layout": "fieldset"}),
("M29", "M36", "Subject input becomes a textarea", {"ui_form_layout": "textarea"}),
("M30", "M37", "Unrelated form precedes the control", {"ui_form_layout": "decoy"}),
("M31", "M38", "Control moved after revoke into a sidebar", {"ui_form_layout": "sidebar"}),
)
for dropped_id, preserved_id, title, flags in EXTENDED_LAYOUTS:
for mutation_id, preserved in ((dropped_id, False), (preserved_id, True)):
CATALOGUE += (Mutation(
mutation_id, title + ("; ids preserved" if preserved else "; ids dropped"),
"MECHANICAL", "ui",
"Agent-authored synthetic variant: the same explicit grant form and "
"endpoint remain available; stored permissions and enforcement are unchanged.",
lambda app, flags=flags, preserved=preserved: _m(
app, **flags, ui_test_ids="stable" if preserved else "dropped"
),
preserves_test_ids=preserved,
),)
BY_ID: dict[str, Mutation] = {m.id: m for m in CATALOGUE}
def expected_classification(mutation_id: str) -> Label:
"""Ground truth. Recorded by a human before any run — never inferred."""
"""Pre-recorded instrument labels, never inferred from run outcomes."""
return BY_ID[mutation_id].label