tegwick
3497ca88bf
Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness, so they pick up the real Received: headers an MTA adds, and harness.inject_maildir() as the byte-exact offline counterpart. tests/test_evidence_realism.py asserts all ten recognized evidence classes after passing through a real MTA, proves MTA delivery does not change which evidence is produced, and covers Maildir injection offline. Documents what no local server can honestly produce -- provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance -- and names SES simulator addresses as the staging path. That tier stays out of the test run because it needs real credentials. Adds docs/test-harness-tutorial.md covering the three test tiers and the start/send/scan/assert/reset walkthrough, with an explicit assertable/not-assertable list so the evidence ceiling is stated where tests get written. Adds a Maildir section to the mailbox report tutorial. Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed + 21 skipped with it down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
175 lines
6.2 KiB
Markdown
175 lines
6.2 KiB
Markdown
# Local mail test harness
|
|
|
|
A single GreenMail container serving SMTP and IMAP on loopback, so integration
|
|
tests can drive the transactional send path and read the result back through the
|
|
scanner without a live provider.
|
|
|
|
Part of `EMAIL-WP-0005`. Unit tests do not need this harness — they run against
|
|
fixture directories and temporary Maildir trees and stay offline.
|
|
|
|
## Start and stop
|
|
|
|
```bash
|
|
docker compose -f tests/harness/docker-compose.yml up -d
|
|
docker compose -f tests/harness/docker-compose.yml down -v
|
|
```
|
|
|
|
Readiness (the container healthcheck probes both mail ports; the API also
|
|
answers):
|
|
|
|
```bash
|
|
docker inspect --format '{{.State.Health.Status}}' email-connect-harness
|
|
curl -s http://127.0.0.1:8080/api/service/readiness
|
|
```
|
|
|
|
## Ports
|
|
|
|
| Port | Protocol | Notes |
|
|
| --- | --- | --- |
|
|
| 3025 | SMTP | plaintext, no STARTTLS |
|
|
| 3143 | IMAP | plaintext |
|
|
| 8080 | HTTP | GreenMail API — readiness, user and mail management |
|
|
|
|
All three bind `127.0.0.1` only. The harness accepts any credentials and must
|
|
never be exposed beyond the host.
|
|
|
|
## Accounts
|
|
|
|
The harness runs with `greenmail.auth.disabled`, so **any** login is accepted and
|
|
the mailbox is created on first use. No provisioning step is needed: pick an
|
|
address under `harness.email-connect.test` and log in.
|
|
|
|
```bash
|
|
export EMAIL_CONNECT_IMAP_USER=t01@harness.email-connect.test
|
|
export EMAIL_CONNECT_IMAP_PASSWORD=harness # ignored, but must be set
|
|
```
|
|
|
|
These credentials are deliberately non-secret and belong in the repo. Real
|
|
provider material is routed through OpenBao — see
|
|
`.claude/rules/credential-routing.md`. Never point this harness at real
|
|
credentials or a real mailbox.
|
|
|
|
### Addressing convention
|
|
|
|
Tests address mailboxes through the `harness` helper package rather than
|
|
hardcoding strings:
|
|
|
|
```python
|
|
import harness
|
|
|
|
recipient = harness.address("invitation resend")
|
|
# -> invitation-resend@harness.email-connect.test
|
|
```
|
|
|
|
`harness.address()` slugs the name, so each test picks its own address and two
|
|
tests cannot collide. `.test` is reserved by RFC 2606, so a stray send can never
|
|
leave the host.
|
|
|
|
### Reset contract
|
|
|
|
`harness.reset()` drops **every** user and message — GreenMail's reset clears
|
|
accounts along with their mail, and the next login recreates the mailbox empty.
|
|
|
|
Call it at the **start** of a test, not the end: a crashed or interrupted test
|
|
then cannot leave state behind for the next one. It is global, so tests that
|
|
reset cannot run in parallel against one harness.
|
|
|
|
```python
|
|
def test_something():
|
|
harness.reset()
|
|
...
|
|
```
|
|
|
|
### Skipping when the harness is down
|
|
|
|
Harness-dependent tests are gated so the default suite stays offline:
|
|
|
|
```python
|
|
requires_harness = pytest.mark.skipif(
|
|
not harness.available(),
|
|
reason="mail harness not running",
|
|
)
|
|
```
|
|
|
|
`harness.available()` is evaluated at import time, so start the harness before
|
|
invoking pytest.
|
|
|
|
## Scanning the harness
|
|
|
|
```bash
|
|
docker compose -f tests/harness/docker-compose.yml up -d
|
|
export EMAIL_CONNECT_IMAP_USER=t01@harness.email-connect.test
|
|
export EMAIL_CONNECT_IMAP_PASSWORD=harness
|
|
email-connect scan-mailbox --config config/harness-imap.yml --out reports/
|
|
```
|
|
|
|
## Sending through the harness
|
|
|
|
GreenMail offers no STARTTLS, only plaintext or implicit TLS with a self-signed
|
|
certificate. The transactional service therefore needs its transport mode set
|
|
explicitly:
|
|
|
|
```bash
|
|
export EMAIL_CONNECT_SMTP_HOST=127.0.0.1
|
|
export EMAIL_CONNECT_SMTP_PORT=3025
|
|
export EMAIL_CONNECT_SMTP_SECURITY=plaintext
|
|
export EMAIL_CONNECT_SMTP_USERNAME=sender@harness.email-connect.test
|
|
export EMAIL_CONNECT_SMTP_PASSWORD=harness
|
|
export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
|
|
```
|
|
|
|
`EMAIL_CONNECT_SMTP_SECURITY` defaults to `starttls`. `plaintext` is rejected
|
|
for any non-loopback host, so this setting cannot weaken a real deployment: it
|
|
fails at startup rather than sending credentials in the clear.
|
|
|
|
## Exercising bounces, complaints and deferrals
|
|
|
|
The harness accepts every message, so it cannot *generate* a bounce, complaint,
|
|
or deferral. Those classes are exercised by injecting the crafted fixtures in
|
|
`tests/fixtures/mailbox/` and letting them travel through the harness:
|
|
|
|
```python
|
|
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
|
|
```
|
|
|
|
This matters more than reading the fixture off disk. Delivery adds the real
|
|
`Received:` headers an MTA-handled message carries — the exact difference that
|
|
hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
|
|
|
|
`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
|
|
writes the message straight into a Maildir tree, byte for byte, with no MTA
|
|
involved and no harness required.
|
|
|
|
`tests/test_evidence_realism.py` runs every recognized class both ways and
|
|
asserts the two agree.
|
|
|
|
## What is only reachable in a provider staging tier
|
|
|
|
Some behavior no local server can honestly reproduce, because it belongs to a
|
|
real provider and a real remote MX:
|
|
|
|
| Not reachable locally | Why |
|
|
| --- | --- |
|
|
| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
|
|
| Real 4xx deferral and retry over time | The harness never defers or retries |
|
|
| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
|
|
| Provider suppression-list behavior | Belongs to the provider account |
|
|
| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
|
|
|
|
For those, use a provider staging tier with simulator addresses rather than
|
|
inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
|
|
`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
|
|
`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
|
|
touching a real recipient.
|
|
|
|
That tier is deliberately **not** wired into this repo's test run: it needs real
|
|
credentials, which are routed through OpenBao and must never reach the harness.
|
|
Treat it as an operator-run check against a staging provider account.
|
|
|
|
## What harness mail proves
|
|
|
|
Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is
|
|
not evidence of inbox placement, recipient awareness, identity, or
|
|
authorization. Bounces, complaints, and deferrals are **not** reproducible here
|
|
— GreenMail accepts everything. Those classes stay on crafted `.eml` fixtures
|
|
and an optional provider-simulator tier (`EMAIL-WP-0005-T05`).
|