test-driver/docs/TestDriverEvidenceAndVariants.md

Retained evidence and explicit adversarial variants

TD-WP-0004 adds two standard-library APIs. Use Python ≥3.11 from the repository; pytest remains the only dependency of the test suite.

Retain, load and compare runs

PYTHONPATH=src:. python3 - <<'PY'
from scenarios.alice_bob_carol import build
from testdriver import Runner
from testdriver.storage import EvidenceStore
from testdriver.classification import classify

store = EvidenceStore('/tmp/test-driver-evidence')
receipts = []
for _ in range(2):
    world, driver, observer, asset, oracle = build()
    result = Runner(world, driver, observer, oracle).run(asset, evidence_store=store)
    receipts.append(store.load(result.run_id))
    print(result.run_id, result.verdict.value)
print(classify(*receipts).classification.value)  # UNCHANGED
PY

EvidenceStore.write(pack) returns its path. load(run_id) returns the evidence mapping expected by classification/crystallization. The version-1 envelope wraps evidence with a SHA-256 checksum. Files publish atomically without overwrite, with mode 0600; newly created store directories request mode 0700. Concurrent publication of the same run yields one winner and FileExistsError for others. Directory fsync uses the local POSIX filesystem API. Storage failures propagate; a caller must not report retained evidence if a write failed.

Observations must use JSON-native dictionaries with string keys, lists, strings, integers, finite floats, booleans and null. Tuples and custom value/container types are rejected rather than coerced. Runner checks and detaches the snapshot before postconditions or oracles evaluate it; invalid evidence stops further actions, records an evidence failure and makes pending judgments INCONCLUSIVE. Already observed failures remain FAIL. Serialization also rejects unsupported values, non-string mapping keys, cycles and non-finite numbers. Observations detach nested data when recorded, so later domain mutations do not rewrite earlier snapshots. Receipts include the final run verdict and asset id, parent, maturity, variant and adaptation history. Independent claim revisions and scenario revisions bind evidence to the original run inputs.

Retention is opt-in. It covers finalized returned runs, including FAIL and guard aborts; unexpected driver/observer exceptions or process termination do not promise a complete receipt. The checksum detects damage, not malicious replacement with a recomputed checksum. The store is not encryption, credential redaction, access custody or a retention service. Select observations safe to retain before enabling it; existing driver mechanics can contain action arguments.

Derive a security question from the same intent

PYTHONPATH=src:. python3 - <<'PY'
from scenarios.alice_bob_carol import build
from testdriver import Runner
from testdriver.variants import substitute

world, driver, observer, parent, oracle = build()
variant = substitute(parent, variant_id='grant-to-carol', step_id='s2-grant',
                     arguments={'subject_id': 'carol'})
assert variant.scenario.use_case is parent.scenario.use_case
result = Runner(world, driver, observer, oracle).run(variant)
print(variant.parent_id, result.verdict.value)  # FAIL: original independent claims still apply
PY

substitute(..., actor_id='other') changes the scheduled actor; that actor must exist in the execution world's cast. Runner validates all scheduled actor references, cast key/id agreement and nonempty unique step ids before any action; invalid input raises ValueError with no execution or finalized receipt. arguments={...} changes only existing argument keys and covers resource, tenant or privilege substitution without new framework concepts. Mutable argument data is copied. Unknown/duplicate step ids, unknown keys, empty actor ids and invalid variant ids are rejected. Callers choose unique variant ids for distinct definitions; the API is not an asset registry.

The exact UseCase, claim/invariant predicates, postconditions, watches, step order and permitted surfaces remain unchanged. Parent identity and mutation metadata are retained without copying changed values into mutation history. A derived variant does not become an approved new requirement, and a denial is not a default PASS: it needs the independently authored assertions appropriate to the question.

Acceptance fingerprints include scheduled actor/action identities, argument values, order, surface permissions and postcondition definitions. Changed definitions require intent review, even when product verdicts happen to pass. Unsupported runtime dependencies or old packs without this revision cannot be accepted or crystallized automatically. Mechanical selector/endpoint choices remain outside scenario intent and can still qualify as mechanical adaptation.

This API deliberately does not generate new assertions, skip/reorder steps, schedule races or infer required evidence from a successful implementation run.

Evidence/preflight decision: aa6c3e31-614c-4030-990c-945acef08515.

Generated regression outcomes

Generated pytest modules embed the same predicate evaluator used by Oracle and continue to import the original independent predicates. They require no runtime test-driver machinery; the outcome adapter uses stdlib unittest.SkipTest.

True means PASS and False means FAIL. Empty snapshots, missing observation keys, predicate exceptions and non-boolean results mean INCONCLUSIVE. The module evaluates all supplied assertions: any FAIL takes precedence over INCONCLUSIVE, regardless of assertion order. With no failure but an inconclusive assertion, pytest reports a skip labeled INCONCLUSIVE, not a pass or product failure. Use pytest -rs to display those reasons. A zero pytest exit status alone is not proof that every generated assertion was verified; inspect skipped outcomes.

The checked-in descendant is regenerated with its original protected predicates and frozen path. Execution-based tests compare generated and checked-in judgments with Oracle, and a subprocess test verifies actual pytest pass/fail/skip reporting. Decision: c06c8752-80db-4458-9eaf-6321a0ca710c.