EMAIL-WP-0005 T05/T06: evidence realism and harness documentation
Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness, so they pick up the real Received: headers an MTA adds, and harness.inject_maildir() as the byte-exact offline counterpart. tests/test_evidence_realism.py asserts all ten recognized evidence classes after passing through a real MTA, proves MTA delivery does not change which evidence is produced, and covers Maildir injection offline. Documents what no local server can honestly produce -- provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance -- and names SES simulator addresses as the staging path. That tier stays out of the test run because it needs real credentials. Adds docs/test-harness-tutorial.md covering the three test tiers and the start/send/scan/assert/reset walkthrough, with an explicit assertable/not-assertable list so the evidence ceiling is stated where tests get written. Adds a Maildir section to the mailbox report tutorial. Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed + 21 skipped with it down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e9609b4024
commit
3497ca88bf
7 changed files with 431 additions and 4 deletions
164
docs/test-harness-tutorial.md
Normal file
164
docs/test-harness-tutorial.md
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
# Test Harness Tutorial
|
||||
|
||||
How to run `email-connect` against a local mail server: start it, send through
|
||||
it, scan it back, assert on the result, and reset between tests.
|
||||
|
||||
Reference material lives in `tests/harness/README.md`. This page is the
|
||||
walkthrough.
|
||||
|
||||
## 1. The three test tiers
|
||||
|
||||
`email-connect` tests at three levels, and it matters which one a given
|
||||
assertion belongs to.
|
||||
|
||||
| Tier | Source | Needs | What it is for |
|
||||
| --- | --- | --- | --- |
|
||||
| Unit | `.eml` fixtures, temporary Maildir trees | nothing | Parsing, classification, reporting. Deterministic and offline. |
|
||||
| Integration | local GreenMail harness | Docker | Send-to-scan continuity through a real SMTP/IMAP server. |
|
||||
| Provider staging | real provider account | credentials, operator | Behavior only a real provider and remote MX produce. |
|
||||
|
||||
The default `pytest` run is the unit tier plus any integration tests whose
|
||||
harness happens to be up. Nothing in the default run requires a container: with
|
||||
the harness down the integration cases skip.
|
||||
|
||||
The staging tier is not wired into the test run at all. It needs real
|
||||
credentials, which are routed through OpenBao — see
|
||||
`.claude/rules/credential-routing.md` — and must never be pointed at the
|
||||
harness.
|
||||
|
||||
## 2. Start the harness
|
||||
|
||||
```bash
|
||||
docker compose -f tests/harness/docker-compose.yml up -d
|
||||
docker inspect --format '{{.State.Health.Status}}' email-connect-harness
|
||||
```
|
||||
|
||||
Wait for `healthy`. The container serves SMTP on 3025 and IMAP on 3143, both
|
||||
bound to `127.0.0.1` only.
|
||||
|
||||
## 3. Give the test a mailbox
|
||||
|
||||
Authentication is disabled and mailboxes are created on first login, so there is
|
||||
no provisioning step — pick an address and use it:
|
||||
|
||||
```python
|
||||
import harness
|
||||
|
||||
recipient = harness.address("invitation delivered")
|
||||
# -> invitation-delivered@harness.email-connect.test
|
||||
```
|
||||
|
||||
Slugging the test's own name keeps two tests from colliding. The domain is under
|
||||
`.test`, reserved by RFC 2606, so a stray send cannot leave the host.
|
||||
|
||||
## 4. Send through the service
|
||||
|
||||
```python
|
||||
from email_connect.transactional import SQLiteDeliveryStore, TransactionalApplication
|
||||
|
||||
app = TransactionalApplication(
|
||||
SQLiteDeliveryStore(str(tmp_path / "mail.db")),
|
||||
harness.smtp_provider(),
|
||||
"harness-ingest-token",
|
||||
"https://users.harness.email-connect.test",
|
||||
)
|
||||
```
|
||||
|
||||
`harness.smtp_provider()` returns a real `SMTPProvider` in plaintext transport
|
||||
mode. GreenMail offers no STARTTLS, and `SMTPProvider` permits plaintext only
|
||||
for loopback hosts, so this cannot be pointed at a real provider by accident.
|
||||
|
||||
## 5. Scan it back
|
||||
|
||||
```python
|
||||
from email_connect.scanner import scan_mailbox
|
||||
|
||||
config = harness.mailbox_config(
|
||||
recipient,
|
||||
storage_path=str(tmp_path / "state.sqlite"),
|
||||
reports_dir=str(tmp_path / "reports"),
|
||||
)
|
||||
result = scan_mailbox(config)
|
||||
```
|
||||
|
||||
From the command line:
|
||||
|
||||
```bash
|
||||
export EMAIL_CONNECT_IMAP_USER=invitation-delivered@harness.email-connect.test
|
||||
export EMAIL_CONNECT_IMAP_PASSWORD=harness
|
||||
email-connect scan-mailbox --config config/harness-imap.yml --out reports/
|
||||
```
|
||||
|
||||
## 6. Assert — and what you may not assert
|
||||
|
||||
An accepted send is `provider_accepted`. A message sitting in a harness mailbox
|
||||
is a message sitting in a harness mailbox. Neither is evidence of inbox
|
||||
placement, recipient awareness, identity, or authorization, and a test that
|
||||
asserts otherwise is wrong even when it passes.
|
||||
|
||||
Assertable:
|
||||
|
||||
```python
|
||||
assert body["evidence_ceiling"] == "provider_accepted"
|
||||
assert body["reference"] == str(messages[0]["Message-ID"]).strip()
|
||||
assert len(delivered_messages(recipient)) == 1
|
||||
```
|
||||
|
||||
Not assertable from harness mail — these are the claims the repo exists to
|
||||
avoid:
|
||||
|
||||
```text
|
||||
the recipient received it
|
||||
the recipient read it
|
||||
the recipient's identity is confirmed
|
||||
the recipient is authorized
|
||||
the intended result was satisfied
|
||||
```
|
||||
|
||||
`coordination-engine` evaluates results. `email-connect` reports what the email
|
||||
channel observed.
|
||||
|
||||
## 7. Reset between tests
|
||||
|
||||
```python
|
||||
def test_something():
|
||||
harness.reset()
|
||||
...
|
||||
```
|
||||
|
||||
`harness.reset()` drops every user and message; the next login recreates the
|
||||
mailbox empty. Call it at the **start** of a test so a crashed or interrupted
|
||||
run cannot leave state for the next one. It is global, so tests that reset
|
||||
cannot run in parallel against a single harness.
|
||||
|
||||
## 8. Bounces, complaints and deferrals
|
||||
|
||||
The harness accepts everything, so it never produces these on its own. Inject a
|
||||
crafted fixture instead:
|
||||
|
||||
```python
|
||||
harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
|
||||
```
|
||||
|
||||
Injected messages still travel through a real MTA and pick up real `Received:`
|
||||
headers — which is how `EMAIL-WP-0005-T04` caught a reply heuristic that was
|
||||
matching those headers and labelling ordinary mail a human reply.
|
||||
|
||||
`harness.inject_maildir()` is the offline counterpart, writing straight into a
|
||||
Maildir tree with no MTA and no harness.
|
||||
|
||||
Provider-generated DSNs, real deferral and retry, ISP feedback loops, and
|
||||
provider suppression behavior are **not** reachable locally. See the
|
||||
provider-simulator table in `tests/harness/README.md`.
|
||||
|
||||
## 9. Stop the harness
|
||||
|
||||
```bash
|
||||
docker compose -f tests/harness/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
Then confirm the default suite is still offline and green:
|
||||
|
||||
```bash
|
||||
pytest tests/ -q # integration cases skip
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue