EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness, so they pick up the real Received: headers an MTA adds, and harness.inject_maildir() as the byte-exact offline counterpart. tests/test_evidence_realism.py asserts all ten recognized evidence classes after passing through a real MTA, proves MTA delivery does not change which evidence is produced, and covers Maildir injection offline. Documents what no local server can honestly produce -- provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance -- and names SES simulator addresses as the staging path. That tier stays out of the test run because it needs real credentials. Adds docs/test-harness-tutorial.md covering the three test tiers and the start/send/scan/assert/reset walkthrough, with an explicit assertable/not-assertable list so the evidence ceiling is stated where tests get written. Adds a Maildir section to the mailbox report tutorial. Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed + 21 skipped with it down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e9609b4024
commit
3497ca88bf
7 changed files with 431 additions and 4 deletions
|
|
@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
|
|||
for any non-loopback host, so this setting cannot weaken a real deployment: it
|
||||
fails at startup rather than sending credentials in the clear.
|
||||
|
||||
## Exercising bounces, complaints and deferrals
|
||||
|
||||
The harness accepts every message, so it cannot *generate* a bounce, complaint,
|
||||
or deferral. Those classes are exercised by injecting the crafted fixtures in
|
||||
`tests/fixtures/mailbox/` and letting them travel through the harness:
|
||||
|
||||
```python
|
||||
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
|
||||
```
|
||||
|
||||
This matters more than reading the fixture off disk. Delivery adds the real
|
||||
`Received:` headers an MTA-handled message carries — the exact difference that
|
||||
hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
|
||||
|
||||
`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
|
||||
writes the message straight into a Maildir tree, byte for byte, with no MTA
|
||||
involved and no harness required.
|
||||
|
||||
`tests/test_evidence_realism.py` runs every recognized class both ways and
|
||||
asserts the two agree.
|
||||
|
||||
## What is only reachable in a provider staging tier
|
||||
|
||||
Some behavior no local server can honestly reproduce, because it belongs to a
|
||||
real provider and a real remote MX:
|
||||
|
||||
| Not reachable locally | Why |
|
||||
| --- | --- |
|
||||
| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
|
||||
| Real 4xx deferral and retry over time | The harness never defers or retries |
|
||||
| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
|
||||
| Provider suppression-list behavior | Belongs to the provider account |
|
||||
| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
|
||||
|
||||
For those, use a provider staging tier with simulator addresses rather than
|
||||
inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
|
||||
`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
|
||||
`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
|
||||
touching a real recipient.
|
||||
|
||||
That tier is deliberately **not** wired into this repo's test run: it needs real
|
||||
credentials, which are routed through OpenBao and must never reach the harness.
|
||||
Treat it as an operator-run check against a staging provider account.
|
||||
|
||||
## What harness mail proves
|
||||
|
||||
Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue