EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s

Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness,
so they pick up the real Received: headers an MTA adds, and
harness.inject_maildir() as the byte-exact offline counterpart.

tests/test_evidence_realism.py asserts all ten recognized evidence classes
after passing through a real MTA, proves MTA delivery does not change which
evidence is produced, and covers Maildir injection offline.

Documents what no local server can honestly produce -- provider-generated DSNs,
real 4xx deferral and retry, ISP feedback loops, provider suppression behavior,
MX acceptance as distinct from provider acceptance -- and names SES simulator
addresses as the staging path. That tier stays out of the test run because it
needs real credentials.

Adds docs/test-harness-tutorial.md covering the three test tiers and the
start/send/scan/assert/reset walkthrough, with an explicit
assertable/not-assertable list so the evidence ceiling is stated where tests
get written. Adds a Maildir section to the mailbox report tutorial.

Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed +
21 skipped with it down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-14 02:00:17 +02:00
parent e9609b4024
commit 3497ca88bf
7 changed files with 431 additions and 4 deletions

View file

@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
for any non-loopback host, so this setting cannot weaken a real deployment: it
fails at startup rather than sending credentials in the clear.
## Exercising bounces, complaints and deferrals
The harness accepts every message, so it cannot *generate* a bounce, complaint,
or deferral. Those classes are exercised by injecting the crafted fixtures in
`tests/fixtures/mailbox/` and letting them travel through the harness:
```python
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
```
This matters more than reading the fixture off disk. Delivery adds the real
`Received:` headers an MTA-handled message carries — the exact difference that
hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
writes the message straight into a Maildir tree, byte for byte, with no MTA
involved and no harness required.
`tests/test_evidence_realism.py` runs every recognized class both ways and
asserts the two agree.
## What is only reachable in a provider staging tier
Some behavior no local server can honestly reproduce, because it belongs to a
real provider and a real remote MX:
| Not reachable locally | Why |
| --- | --- |
| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
| Real 4xx deferral and retry over time | The harness never defers or retries |
| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
| Provider suppression-list behavior | Belongs to the provider account |
| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
For those, use a provider staging tier with simulator addresses rather than
inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
touching a real recipient.
That tier is deliberately **not** wired into this repo's test run: it needs real
credentials, which are routed through OpenBao and must never reach the harness.
Treat it as an operator-run check against a staging provider account.
## What harness mail proves
Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is