email-connect/docs/test-harness-tutorial.md

Test Harness Tutorial

How to run email-connect against a local mail server: start it, send through it, scan it back, assert on the result, and reset between tests.

Reference material lives in tests/harness/README.md. This page is the walkthrough.

1. The three test tiers

email-connect tests at three levels, and it matters which one a given assertion belongs to.

Tier	Source	Needs	What it is for
Unit	.eml fixtures, temporary Maildir trees	nothing	Parsing, classification, reporting. Deterministic and offline.
Integration	local GreenMail harness	Docker	Send-to-scan continuity through a real SMTP/IMAP server.
Provider staging	real provider account	credentials, operator	Behavior only a real provider and remote MX produce.

The default pytest run is the unit tier plus any integration tests whose harness happens to be up. Nothing in the default run requires a container: with the harness down the integration cases skip.

The staging tier is not wired into the test run at all. It needs real credentials, which are routed through OpenBao — see .claude/rules/credential-routing.md — and must never be pointed at the harness.

2. Start the harness

docker compose -f tests/harness/docker-compose.yml up -d
docker inspect --format '{{.State.Health.Status}}' email-connect-harness

Wait for healthy. The container serves SMTP on 3025 and IMAP on 3143, both bound to 127.0.0.1 only.

3. Give the test a mailbox

Authentication is disabled and mailboxes are created on first login, so there is no provisioning step — pick an address and use it:

import harness

recipient = harness.address("invitation delivered")
# -> invitation-delivered@harness.email-connect.test

Slugging the test's own name keeps two tests from colliding. The domain is under .test, reserved by RFC 2606, so a stray send cannot leave the host.

4. Send through the service

from email_connect.transactional import SQLiteDeliveryStore, TransactionalApplication

app = TransactionalApplication(
    SQLiteDeliveryStore(str(tmp_path / "mail.db")),
    harness.smtp_provider(),
    "harness-ingest-token",
    "https://users.harness.email-connect.test",
)

harness.smtp_provider() returns a real SMTPProvider in plaintext transport mode. GreenMail offers no STARTTLS, and SMTPProvider permits plaintext only for loopback hosts, so this cannot be pointed at a real provider by accident.

5. Scan it back

from email_connect.scanner import scan_mailbox

config = harness.mailbox_config(
    recipient,
    storage_path=str(tmp_path / "state.sqlite"),
    reports_dir=str(tmp_path / "reports"),
)
result = scan_mailbox(config)

From the command line:

export EMAIL_CONNECT_IMAP_USER=invitation-delivered@harness.email-connect.test
export EMAIL_CONNECT_IMAP_PASSWORD=harness
email-connect scan-mailbox --config config/harness-imap.yml --out reports/

6. Assert — and what you may not assert

An accepted send is provider_accepted. A message sitting in a harness mailbox is a message sitting in a harness mailbox. Neither is evidence of inbox placement, recipient awareness, identity, or authorization, and a test that asserts otherwise is wrong even when it passes.

Assertable:

assert body["evidence_ceiling"] == "provider_accepted"
assert body["reference"] == str(messages[0]["Message-ID"]).strip()
assert len(delivered_messages(recipient)) == 1

Not assertable from harness mail — these are the claims the repo exists to avoid:

the recipient received it
the recipient read it
the recipient's identity is confirmed
the recipient is authorized
the intended result was satisfied

coordination-engine evaluates results. email-connect reports what the email channel observed.

7. Reset between tests

def test_something():
    harness.reset()
    ...

harness.reset() drops every user and message; the next login recreates the mailbox empty. Call it at the start of a test so a crashed or interrupted run cannot leave state for the next one. It is global, so tests that reset cannot run in parallel against a single harness.

8. Bounces, complaints and deferrals

The harness accepts everything, so it never produces these on its own. Inject a crafted fixture instead:

harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())

Injected messages still travel through a real MTA and pick up real Received: headers — which is how EMAIL-WP-0005-T04 caught a reply heuristic that was matching those headers and labelling ordinary mail a human reply.

harness.inject_maildir() is the offline counterpart, writing straight into a Maildir tree with no MTA and no harness.

Provider-generated DSNs, real deferral and retry, ISP feedback loops, and provider suppression behavior are not reachable locally. See the provider-simulator table in tests/harness/README.md.

9. Stop the harness

docker compose -f tests/harness/docker-compose.yml down -v

Then confirm the default suite is still offline and green:

pytest tests/ -q     # integration cases skip