EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s

Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness,
so they pick up the real Received: headers an MTA adds, and
harness.inject_maildir() as the byte-exact offline counterpart.

tests/test_evidence_realism.py asserts all ten recognized evidence classes
after passing through a real MTA, proves MTA delivery does not change which
evidence is produced, and covers Maildir injection offline.

Documents what no local server can honestly produce -- provider-generated DSNs,
real 4xx deferral and retry, ISP feedback loops, provider suppression behavior,
MX acceptance as distinct from provider acceptance -- and names SES simulator
addresses as the staging path. That tier stays out of the test run because it
needs real credentials.

Adds docs/test-harness-tutorial.md covering the three test tiers and the
start/send/scan/assert/reset walkthrough, with an explicit
assertable/not-assertable list so the evidence ceiling is stated where tests
get written. Adds a Maildir section to the mailbox report tutorial.

Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed +
21 skipped with it down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-14 02:00:17 +02:00
parent e9609b4024
commit 3497ca88bf
7 changed files with 431 additions and 4 deletions

View file

@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
for any non-loopback host, so this setting cannot weaken a real deployment: it
fails at startup rather than sending credentials in the clear.
## Exercising bounces, complaints and deferrals
The harness accepts every message, so it cannot *generate* a bounce, complaint,
or deferral. Those classes are exercised by injecting the crafted fixtures in
`tests/fixtures/mailbox/` and letting them travel through the harness:
```python
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
```
This matters more than reading the fixture off disk. Delivery adds the real
`Received:` headers an MTA-handled message carries — the exact difference that
hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
writes the message straight into a Maildir tree, byte for byte, with no MTA
involved and no harness required.
`tests/test_evidence_realism.py` runs every recognized class both ways and
asserts the two agree.
## What is only reachable in a provider staging tier
Some behavior no local server can honestly reproduce, because it belongs to a
real provider and a real remote MX:
| Not reachable locally | Why |
| --- | --- |
| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
| Real 4xx deferral and retry over time | The harness never defers or retries |
| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
| Provider suppression-list behavior | Belongs to the provider account |
| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
For those, use a provider staging tier with simulator addresses rather than
inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
touching a real recipient.
That tier is deliberately **not** wired into this repo's test run: it needs real
credentials, which are routed through OpenBao and must never reach the harness.
Treat it as an operator-run check against a staging provider account.
## What harness mail proves
Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is

View file

@ -104,6 +104,40 @@ def smtp_provider(sender: str = DEFAULT_SENDER):
return SMTPProvider(HOST, SMTP_PORT, sender, PASSWORD, sender, security="plaintext")
def deliver_raw(recipient: str, raw_bytes: bytes, sender: str = DEFAULT_SENDER) -> None:
"""Deliver a crafted message verbatim to a harness mailbox.
The harness accepts everything, so it cannot *generate* a bounce, complaint,
or deferral. Injecting a crafted `.eml` is how those classes are exercised:
the message still travels through a real MTA and picks up real `Received:`
headers, which a fixture read off disk never does.
"""
import smtplib
with smtplib.SMTP(HOST, SMTP_PORT, timeout=10) as smtp:
smtp.sendmail(sender, [recipient], raw_bytes)
def inject_maildir(maildir_dir, raw_bytes: bytes, *, name: str | None = None, subdir: str = "new"):
"""Write a crafted message straight into a Maildir tree.
The offline counterpart to `deliver_raw`: no MTA involved, so the message
arrives byte-for-byte with no added headers.
"""
import time
from pathlib import Path
root = Path(maildir_dir)
for required in ("new", "cur", "tmp"):
(root / required).mkdir(parents=True, exist_ok=True)
filename = name or f"{int(time.time())}.M{len(raw_bytes)}P{id(raw_bytes) % 100000}.harness"
path = root / subdir / filename
path.write_bytes(raw_bytes)
return path
def mailbox_config(
user_address: str,
*,