EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness, so they pick up the real Received: headers an MTA adds, and harness.inject_maildir() as the byte-exact offline counterpart. tests/test_evidence_realism.py asserts all ten recognized evidence classes after passing through a real MTA, proves MTA delivery does not change which evidence is produced, and covers Maildir injection offline. Documents what no local server can honestly produce -- provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance -- and names SES simulator addresses as the staging path. That tier stays out of the test run because it needs real credentials. Adds docs/test-harness-tutorial.md covering the three test tiers and the start/send/scan/assert/reset walkthrough, with an explicit assertable/not-assertable list so the evidence ceiling is stated where tests get written. Adds a Maildir section to the mailbox report tutorial. Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed + 21 skipped with it down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e9609b4024
commit
3497ca88bf
7 changed files with 431 additions and 4 deletions
|
|
@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
|
|||
for any non-loopback host, so this setting cannot weaken a real deployment: it
|
||||
fails at startup rather than sending credentials in the clear.
|
||||
|
||||
## Exercising bounces, complaints and deferrals
|
||||
|
||||
The harness accepts every message, so it cannot *generate* a bounce, complaint,
|
||||
or deferral. Those classes are exercised by injecting the crafted fixtures in
|
||||
`tests/fixtures/mailbox/` and letting them travel through the harness:
|
||||
|
||||
```python
|
||||
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
|
||||
```
|
||||
|
||||
This matters more than reading the fixture off disk. Delivery adds the real
|
||||
`Received:` headers an MTA-handled message carries — the exact difference that
|
||||
hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
|
||||
|
||||
`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
|
||||
writes the message straight into a Maildir tree, byte for byte, with no MTA
|
||||
involved and no harness required.
|
||||
|
||||
`tests/test_evidence_realism.py` runs every recognized class both ways and
|
||||
asserts the two agree.
|
||||
|
||||
## What is only reachable in a provider staging tier
|
||||
|
||||
Some behavior no local server can honestly reproduce, because it belongs to a
|
||||
real provider and a real remote MX:
|
||||
|
||||
| Not reachable locally | Why |
|
||||
| --- | --- |
|
||||
| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
|
||||
| Real 4xx deferral and retry over time | The harness never defers or retries |
|
||||
| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
|
||||
| Provider suppression-list behavior | Belongs to the provider account |
|
||||
| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
|
||||
|
||||
For those, use a provider staging tier with simulator addresses rather than
|
||||
inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
|
||||
`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
|
||||
`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
|
||||
touching a real recipient.
|
||||
|
||||
That tier is deliberately **not** wired into this repo's test run: it needs real
|
||||
credentials, which are routed through OpenBao and must never reach the harness.
|
||||
Treat it as an operator-run check against a staging provider account.
|
||||
|
||||
## What harness mail proves
|
||||
|
||||
Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is
|
||||
|
|
|
|||
|
|
@ -104,6 +104,40 @@ def smtp_provider(sender: str = DEFAULT_SENDER):
|
|||
return SMTPProvider(HOST, SMTP_PORT, sender, PASSWORD, sender, security="plaintext")
|
||||
|
||||
|
||||
def deliver_raw(recipient: str, raw_bytes: bytes, sender: str = DEFAULT_SENDER) -> None:
|
||||
"""Deliver a crafted message verbatim to a harness mailbox.
|
||||
|
||||
The harness accepts everything, so it cannot *generate* a bounce, complaint,
|
||||
or deferral. Injecting a crafted `.eml` is how those classes are exercised:
|
||||
the message still travels through a real MTA and picks up real `Received:`
|
||||
headers, which a fixture read off disk never does.
|
||||
"""
|
||||
|
||||
import smtplib
|
||||
|
||||
with smtplib.SMTP(HOST, SMTP_PORT, timeout=10) as smtp:
|
||||
smtp.sendmail(sender, [recipient], raw_bytes)
|
||||
|
||||
|
||||
def inject_maildir(maildir_dir, raw_bytes: bytes, *, name: str | None = None, subdir: str = "new"):
|
||||
"""Write a crafted message straight into a Maildir tree.
|
||||
|
||||
The offline counterpart to `deliver_raw`: no MTA involved, so the message
|
||||
arrives byte-for-byte with no added headers.
|
||||
"""
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
root = Path(maildir_dir)
|
||||
for required in ("new", "cur", "tmp"):
|
||||
(root / required).mkdir(parents=True, exist_ok=True)
|
||||
filename = name or f"{int(time.time())}.M{len(raw_bytes)}P{id(raw_bytes) % 100000}.harness"
|
||||
path = root / subdir / filename
|
||||
path.write_bytes(raw_bytes)
|
||||
return path
|
||||
|
||||
|
||||
def mailbox_config(
|
||||
user_address: str,
|
||||
*,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue