EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s

Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness,
so they pick up the real Received: headers an MTA adds, and
harness.inject_maildir() as the byte-exact offline counterpart.

tests/test_evidence_realism.py asserts all ten recognized evidence classes
after passing through a real MTA, proves MTA delivery does not change which
evidence is produced, and covers Maildir injection offline.

Documents what no local server can honestly produce -- provider-generated DSNs,
real 4xx deferral and retry, ISP feedback loops, provider suppression behavior,
MX acceptance as distinct from provider acceptance -- and names SES simulator
addresses as the staging path. That tier stays out of the test run because it
needs real credentials.

Adds docs/test-harness-tutorial.md covering the three test tiers and the
start/send/scan/assert/reset walkthrough, with an explicit
assertable/not-assertable list so the evidence ceiling is stated where tests
get written. Adds a Maildir section to the mailbox report tutorial.

Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed +
21 skipped with it down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-14 02:00:17 +02:00
parent e9609b4024
commit 3497ca88bf
7 changed files with 431 additions and 4 deletions

View file

@ -4,7 +4,7 @@ type: workplan
title: "Test Mailbox Harness for Automated Test Environments"
domain: infotech
repo: email-connect
status: proposed
status: finished
owner: claude
topic_slug: custodian
created: "2026-08-14"
@ -322,7 +322,7 @@ Two production defects surfaced, both invisible to the fixture-only suite:
```task
id: EMAIL-WP-0005-T05
status: todo
status: done
priority: medium
state_hub_task_id: "48b8eaf9-5a4b-437b-ad2c-952907340ed7"
```
@ -345,11 +345,26 @@ path, and the limits of the local harness are stated explicitly rather than
papered over.
```
Done 2026-08-14:
* `harness.deliver_raw()` injects a crafted `.eml` through the harness, so the
message picks up real `Received:` headers on the way in.
`harness.inject_maildir()` is the offline counterpart, byte-exact with no MTA.
* `tests/test_evidence_realism.py`, 13 cases: all ten recognized classes are
asserted after passing through a real MTA, plus a parity case proving MTA
delivery does not change which evidence is produced, plus two offline Maildir
injection cases.
* Documented what no local server can honestly produce — provider-generated
DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression
behavior, MX acceptance as distinct from provider acceptance — and named SES
simulator addresses as the staging path. That tier is deliberately not wired
into the test run: it needs real credentials, which stay in OpenBao.
## T06 - Documentation and decision record
```task
id: EMAIL-WP-0005-T06
status: todo
status: done
priority: medium
state_hub_task_id: "0599de98-c0ff-4160-a2c8-54265b31ca5d"
```
@ -371,6 +386,21 @@ A new contributor can start the harness, run the integration tier, and
correctly state what harness-observed mail does and does not prove.
```
Done 2026-08-14:
* `docs/test-harness-tutorial.md`: three test tiers, start, mailbox, send, scan,
assert, reset, fixture injection, teardown. Includes an explicit
assertable/not-assertable list, so the evidence ceiling is stated where tests
get written rather than only in the canon.
* `docs/mailbox-report-tutorial.md` gained a Maildir section covering
`protocol: maildir`, `source.maildir_dir`, identity across `new/``cur/`, and
the read-only guarantees.
* `tests/harness/README.md` carries the reference material: ports, addressing
convention, reset contract, skip gating, injection, and the
provider-simulator table.
* Decisions were recorded as the work happened: s/qmail rejection and the
GreenMail plus loopback-plaintext choice are both in `DECISIONS.md`.
## 8. Completion Criteria
This workplan is complete when: