coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Preserve oracle semantics in generated regression judgments

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4
This commit is contained in:
tegwick 2026-09-28 15:57:45 +02:00
parent 10077edb8c
commit 3ce772466b
8 changed files with 224 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/TestDriverEvidenceAndVariants.md
View file

@ -92,3 +92,22 @@ This API deliberately does not generate new assertions, skip/reorder steps,
schedule races or infer required evidence from a successful implementation run.
Evidence/preflight decision: `aa6c3e31-614c-4030-990c-945acef08515`.
## Generated regression outcomes
Generated pytest modules embed the same predicate evaluator used by Oracle and
continue to import the original independent predicates. They require no runtime
test-driver machinery; the outcome adapter uses stdlib unittest.SkipTest.
True means PASS and False means FAIL. Empty snapshots, missing observation keys,
predicate exceptions and non-boolean results mean INCONCLUSIVE. The module
evaluates all supplied assertions: any FAIL takes precedence over INCONCLUSIVE,
regardless of assertion order. With no failure but an inconclusive assertion,
pytest reports a skip labeled INCONCLUSIVE, not a pass or product failure.
Use `pytest -rs` to display those reasons. A zero pytest exit status alone is not
proof that every generated assertion was verified; inspect skipped outcomes.
The checked-in descendant is regenerated with its original protected predicates
and frozen path. Execution-based tests compare generated and checked-in judgments
with Oracle, and a subprocess test verifies actual pytest pass/fail/skip reporting.
Decision: `c06c8752-80db-4458-9eaf-6321a0ca710c`.