email-connect/workplans/EMAIL-WP-0005-test-mailbox-harness.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	state_hub_workstream_id
EMAIL-WP-0005	workplan	Test Mailbox Harness for Automated Test Environments	infotech	email-connect	finished	claude	custodian	2026-08-14	2026-08-14	1ffd54d3-4fdf-4a66-86ba-36a5d8202f3c

EMAIL-WP-0005 - Test Mailbox Harness for Automated Test Environments

1. Purpose

Automated test environments need mail accounts for test users so that end-to-end flows (invitation send, verification, return-path evidence) can be exercised without a production provider and without hand-crafted fixtures for every case.

This workplan adds a test mailbox harness: a Maildir mailbox source, a containerized SMTP/IMAP test server, deterministic test-user accounts, and end-to-end tests that drive /v1/send into the harness and scan the resulting mailboxes back out through the existing scanner.

The harness is test infrastructure. It does not make email-connect a mail provider or an MTA operator, and it must not weaken the evidence discipline: mail observed in a test server is still provider/MX-level fact, never proof of delivery, awareness, identity, or authorization.

2. User Story

As a developer or CI job, I want to start one container, send invitation and verification mail through email-connect to per-test mailboxes, and then scan those mailboxes with the normal scanner path, so that I can assert on end-to-end behavior without a live provider and without network egress.

3. In Scope

• A maildir mailbox source alongside the existing fixture and imap sources.
• A containerized SMTP + IMAP test server for integration tests.
• Deterministic per-test-user account provisioning and reset.
• End-to-end tests: send through the transactional service, scan through the scanner, assert evidence rows.
• A documented strategy for bounce/complaint/deferral realism, which local servers cannot generate honestly.
• Documentation and a decision record for the chosen approach.

4. Out of Scope

• Operating a production or internet-facing MTA.
• Owning DNS, TLS certificates, rDNS, SPF/DKIM/DMARC, or spam filtering.
• Mailbox write-back actions (the scanner stays read-only; mark_seen remains rejected).
• A mailbox management UI or user-facing account provisioning.
• Replacing the fixture path as the default test source.
• Integrating s/qmail — see DECISIONS.md, "Test mailbox harness: purpose-built test server over s/qmail".

5. Design Constraints

The harness must respect the existing seams rather than introduce new ones:

mailbox.protocol     fixture | imap | maildir     (src/email_connect/mailbox.py)
SMTP provider        EMAIL_CONNECT_SMTP_HOST/PORT (src/email_connect/transactional.py)

Pointing tests at the harness must require configuration only, not code branching inside the scanner or the transactional application. No test-only code paths may exist in the production send or scan logic.

The fixture source remains the default for unit tests: deterministic, offline, no container required. The harness is the integration tier.

6. Evidence Semantics

Mail retrieved from a test server carries exactly the same evidence ceiling as mail retrieved from a real mailbox. Specifically, the harness must not be used to justify any of:

message was delivered to a real inbox
recipient became aware of the message
recipient identity was verified
recipient was authorized

A test-server acceptance is provider_accepted and nothing more. Tests that assert stronger claims are defects in the test, not features of the harness.

7. Work Packages

T01 - Maildir mailbox source

id: EMAIL-WP-0005-T01
status: done
priority: high
state_hub_task_id: "28856bdc-7ea2-44ce-8c06-88bf006e17fd"

Tasks:

Add protocol: maildir to config and AppConfig validation
Implement MaildirMailboxSource reading new/ and cur/ per Maildir spec
Derive stable message identity from Maildir filename plus existing dedup keys
Support incremental scans via scan_cursors for maildir sources
Preserve raw_message_ref as a maildir:// reference
Reject mark_seen for maildir the same way IMAP does
Wire the source into source_for_config
Add unit tests over a temporary Maildir tree

Acceptance:

The scanner reads a Maildir directory, classifies messages, and produces the
same evidence rows as the equivalent fixture directory, with deduplication and
incremental cursors working across repeated scans.

Done 2026-08-14:

• MaildirMailboxSource reads new/ and cur/, skips dotfiles, prefers cur/ when a base name appears in both, and emits maildir://<root>/<subdir>/<file> refs. mark_seen and a missing directory both raise, matching the read-only IMAP contract.
• Identity is the Maildir unique name minus the :2,FLAGS suffix, so it is stable across the new/→cur/ move. It is carried on a new MailboxSourceMessage.dedup_uid and appended to the message dedup key only when set, leaving existing fixture and IMAP state-store keys byte-identical.
• Cursor ordering parses the Maildir delivery time rather than comparing names lexically, so cursors stay correct as the time field changes width.
• Fixed alongside: the parse-failure path keyed identity on raw_message_ref, which a new/→cur/ move rewrites, so an unparseable message re-registered as new on every rescan. It now prefers the source uid when one exists. Behavior for fixture and IMAP sources is unchanged.
• source.maildir_dir added to config and the example file; source_for_config validates it. include_seen: false skips S-flagged messages.
• Tests: tests/test_maildir.py, 11 cases including maildir-vs-fixture evidence parity. Full suite 38 passed.

T02 - Containerized SMTP/IMAP test server

id: EMAIL-WP-0005-T02
status: done
priority: high
state_hub_task_id: "967d2bab-43e3-4240-8bd5-c657ba4178ff"

Tasks:

Select and pin a test mail server image providing SMTP and IMAP
Add a compose or container definition under a test harness directory
Expose SMTP and IMAP on fixed local ports with non-secret test credentials
Document startup, teardown, and health check
Add a config profile pointing mailbox.protocol imap at the harness
Add a config profile pointing the SMTP provider env vars at the harness
Ensure the harness never binds a routable interface by default

Acceptance:

One documented command starts a local mail server; the scanner can complete an
IMAP scan against it and the transactional service can complete a send to it,
using committed non-secret test credentials only.

Notes:

Preferred candidate is GreenMail, which serves SMTP and IMAP and creates
accounts on demand. Mailpit is the fallback for send-side-only inspection and
does not exercise the IMAP source. Record the choice in DECISIONS.md.

Done 2026-08-14:

• tests/harness/docker-compose.yml runs GreenMail 2.1.12, digest-pinned, SMTP 3025 / IMAP 3143 / API 8080, all bound to 127.0.0.1 only. The image ships no curl/wget/nc, so the healthcheck probes both mail ports with bash /dev/tcp. Verified healthy.
• greenmail.auth.disabled with no declared users: any login is accepted and the mailbox is created on first use, so T03 needs no provisioning step. Declaring users and enabling auth-disabled conflicts — GreenMail tries to auto-create the login and collides with the declared address.
• config/harness-imap.yml scanner profile; env-var recipe for the send side in tests/harness/README.md.
• GreenMail standalone has no STARTTLS support (confirmed against the shipped jar's property builder — only plain and implicit-TLS setups exist), and SMTPProvider hardcoded starttls(), so no send could reach it. Added an EMAIL_CONNECT_SMTP_SECURITY mode defaulting to starttls; plaintext is refused for any non-loopback host and hostnames are never resolved to decide it. See DECISIONS.md.
• Verified end to end: SMTPProvider.send → GreenMail → ImapMailboxSource fetch, then the documented CLI (scan-mailbox --config config/harness-imap.yml) against the live harness — 1 message seen, parsed, 1 evidence event.
• Tests: 14 offline cases for the transport-security guard in tests/test_transactional.py. Full suite 52 passed, still container-free.

T03 - Test-user account provisioning and reset

id: EMAIL-WP-0005-T03
status: done
priority: high
state_hub_task_id: "c761d26d-a8b0-4b66-ba18-8f22c6d59440"

Tasks:

Define the test-user address convention and test domain
Provide a helper to create or address per-test mailboxes deterministically
Provide per-test reset so state does not leak between tests
Keep test credentials non-secret and committed, never routed through OpenBao
Document the convention and the reset contract

Acceptance:

A test can obtain an isolated mailbox for a named test user, send to it, read
it back, and reset it, with no cross-test interference and no real credentials
involved.

Notes:

Test credentials are deliberately non-secret and belong in the repo. Per
.claude/rules/credential-routing.md, OpenBao and warden route are for real
provider material; the harness must not touch them.

Done 2026-08-14:

• tests/harness/__init__.py helper package: address(), available(), reset(), users(), smtp_provider(), mailbox_config().
• Convention is harness.address("<test name>") → slug@harness.email-connect.test. .test is RFC 2606 reserved, so a stray send cannot leave the host. No provisioning call is needed — T02's auth-disabled setup creates the mailbox on first login, so distinct names are automatically isolated.
• Reset contract: harness.reset() (GreenMail POST /api/service/reset) drops all users and mail; the next login recreates the mailbox empty. Documented to run at test start, so a crashed test cannot leak state forward. It is global, so resetting tests cannot run in parallel against one harness. Verified: send → 1 message and a live user → reset → no users, mailbox empty.
• Credentials stay non-secret placeholders in the repo; OpenBao is untouched.
• Tests: tests/test_harness_users.py, 12 cases. Slug and config cases run offline; the three harness-dependent cases skip when it is down. Suite: 64 passed with the harness up, 61 passed + 3 skipped with it down.

T04 - End-to-end send-and-scan integration tests

id: EMAIL-WP-0005-T04
status: done
priority: high
state_hub_task_id: "69655fad-97b8-4c17-bc54-af341aaaf38c"

Tasks:

Add an integration test tier that is skipped when the harness is unavailable
Drive /v1/send invitation and verification flows into harness mailboxes
Scan the resulting mailboxes through the normal scanner path
Assert evidence rows, evidence_ceiling, and absence of overclaiming
Cover idempotency and duplicate-request behavior end to end
Cover suppression behavior end to end
Ensure the default unit test run stays offline and container-free

Acceptance:

An integration run proves send-to-scan continuity against a live local server,
while the default test run remains offline, deterministic, and unchanged.

Done 2026-08-14:

• tests/test_integration_send_scan.py, 7 harness-gated cases: accepted invitation reaches the mailbox and correlates by Message-ID; scanner ingests it without a delivery claim; duplicate event sends once and returns the same reference; resend delivers a second distinct message; suppressed recipient receives nothing; unauthorized / template-denied / key-mismatch requests never reach the provider; verification mail delivers and reports authorization=false.
• Suite: 72 passed with the harness up, 62 passed + 10 skipped with it down.

Two production defects surfaced, both invisible to the fixture-only suite:

• Reply heuristic matched message headers. _looks_like_human_reply ran against headers plus body, so the Received: trace that every MTA-handled message carries matched its "received" keyword. Ordinary mail was classified human_reply with a success.reply_received assessment at medium confidence — exactly the overclaim this repo exists to prevent. Hand-written fixtures carry no Received: headers, which is why only real scanned mail exposed it. The heuristic now takes the body alone; DSN detection still sees headers, which it needs. Same message before/after: human_reply/medium → unknown_return_message/low. Regression test is offline (tests/fixtures/mailbox_transit/ordinary_transit.eml).
• Provider reference was not a real reference. SMTPProvider.send fell back to abs(hash((recipient, subject))) because it set no Message-ID. Python randomizes string hashing per process, and equal recipient/subject pairs collided. Outgoing mail now carries a proper RFC 5322 Message-ID, which is returned as the reference — so an accepted send can be tied to a message later observed in a mailbox. This is what makes correlation testable at all.

T05 - Bounce, complaint, and deferral realism

id: EMAIL-WP-0005-T05
status: done
priority: medium
state_hub_task_id: "48b8eaf9-5a4b-437b-ad2c-952907340ed7"

Tasks:

Document that local servers do not generate realistic DSNs, complaints, or deferrals
Keep crafted .eml fixtures as the deterministic path for those classes
Add a helper to inject fixture messages into a harness mailbox or Maildir
Define an optional staging tier using provider simulator addresses
Document which evidence classes are only reachable in the staging tier

Acceptance:

Every evidence class the parser recognizes has a documented, reachable test
path, and the limits of the local harness are stated explicitly rather than
papered over.

Done 2026-08-14:

• harness.deliver_raw() injects a crafted .eml through the harness, so the message picks up real Received: headers on the way in. harness.inject_maildir() is the offline counterpart, byte-exact with no MTA.
• tests/test_evidence_realism.py, 13 cases: all ten recognized classes are asserted after passing through a real MTA, plus a parity case proving MTA delivery does not change which evidence is produced, plus two offline Maildir injection cases.
• Documented what no local server can honestly produce — provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance — and named SES simulator addresses as the staging path. That tier is deliberately not wired into the test run: it needs real credentials, which stay in OpenBao.

T06 - Documentation and decision record

id: EMAIL-WP-0005-T06
status: done
priority: medium
state_hub_task_id: "0599de98-c0ff-4160-a2c8-54265b31ca5d"

Tasks:

Write a test harness tutorial covering start, send, scan, assert, reset
Extend docs/mailbox-report-tutorial.md with the maildir source
Record the DECISIONS.md entry for the harness approach and the s/qmail rejection
Document the three test tiers: fixture unit, harness integration, provider staging
Document the evidence ceiling that applies to harness-observed mail

Acceptance:

A new contributor can start the harness, run the integration tier, and
correctly state what harness-observed mail does and does not prove.

Done 2026-08-14:

• docs/test-harness-tutorial.md: three test tiers, start, mailbox, send, scan, assert, reset, fixture injection, teardown. Includes an explicit assertable/not-assertable list, so the evidence ceiling is stated where tests get written rather than only in the canon.
• docs/mailbox-report-tutorial.md gained a Maildir section covering protocol: maildir, source.maildir_dir, identity across new/→cur/, and the read-only guarantees.
• tests/harness/README.md carries the reference material: ports, addressing convention, reset contract, skip gating, injection, and the provider-simulator table.
• Decisions were recorded as the work happened: s/qmail rejection and the GreenMail plus loopback-plaintext choice are both in DECISIONS.md.

8. Completion Criteria

This workplan is complete when:

1. mailbox.protocol: maildir is supported, tested, and documented.
2. A pinned containerized SMTP/IMAP test server starts from one documented command with committed non-secret credentials.
3. Test users get isolated, resettable mailboxes by convention.
4. Integration tests prove send-to-scan continuity while the default test run stays offline and container-free.
5. Every recognized evidence class has a documented, reachable test path, with local-harness limits stated explicitly.
6. Documentation and the decision record are in place, including the evidence ceiling for harness-observed mail.