--- id: EMAIL-WP-0005 type: workplan title: "Test Mailbox Harness for Automated Test Environments" domain: infotech repo: email-connect status: finished owner: claude topic_slug: custodian created: "2026-08-14" updated: "2026-08-14" state_hub_workstream_id: "44ba2150-41a3-5931-890f-081aa2697e4f" --- # EMAIL-WP-0005 - Test Mailbox Harness for Automated Test Environments ## 1. Purpose Automated test environments need mail accounts for test users so that end-to-end flows (invitation send, verification, return-path evidence) can be exercised without a production provider and without hand-crafted fixtures for every case. This workplan adds a **test mailbox harness**: a Maildir mailbox source, a containerized SMTP/IMAP test server, deterministic test-user accounts, and end-to-end tests that drive `/v1/send` into the harness and scan the resulting mailboxes back out through the existing scanner. The harness is test infrastructure. It does not make `email-connect` a mail provider or an MTA operator, and it must not weaken the evidence discipline: mail observed in a test server is still provider/MX-level fact, never proof of delivery, awareness, identity, or authorization. ## 2. User Story As a developer or CI job, I want to start one container, send invitation and verification mail through `email-connect` to per-test mailboxes, and then scan those mailboxes with the normal scanner path, so that I can assert on end-to-end behavior without a live provider and without network egress. ## 3. In Scope - A `maildir` mailbox source alongside the existing `fixture` and `imap` sources. - A containerized SMTP + IMAP test server for integration tests. - Deterministic per-test-user account provisioning and reset. - End-to-end tests: send through the transactional service, scan through the scanner, assert evidence rows. - A documented strategy for bounce/complaint/deferral realism, which local servers cannot generate honestly. - Documentation and a decision record for the chosen approach. ## 4. Out of Scope - Operating a production or internet-facing MTA. - Owning DNS, TLS certificates, rDNS, SPF/DKIM/DMARC, or spam filtering. - Mailbox write-back actions (the scanner stays read-only; `mark_seen` remains rejected). - A mailbox management UI or user-facing account provisioning. - Replacing the fixture path as the default test source. - Integrating s/qmail — see `DECISIONS.md`, "Test mailbox harness: purpose-built test server over s/qmail". ## 5. Design Constraints The harness must respect the existing seams rather than introduce new ones: ```text mailbox.protocol fixture | imap | maildir (src/email_connect/mailbox.py) SMTP provider EMAIL_CONNECT_SMTP_HOST/PORT (src/email_connect/transactional.py) ``` Pointing tests at the harness must require configuration only, not code branching inside the scanner or the transactional application. No test-only code paths may exist in the production send or scan logic. The fixture source remains the default for unit tests: deterministic, offline, no container required. The harness is the *integration* tier. ## 6. Evidence Semantics Mail retrieved from a test server carries exactly the same evidence ceiling as mail retrieved from a real mailbox. Specifically, the harness must not be used to justify any of: ```text message was delivered to a real inbox recipient became aware of the message recipient identity was verified recipient was authorized ``` A test-server acceptance is `provider_accepted` and nothing more. Tests that assert stronger claims are defects in the test, not features of the harness. ## 7. Work Packages ## T01 - Maildir mailbox source ```task id: EMAIL-WP-0005-T01 status: done priority: high state_hub_task_id: "87b919f5-6fac-5112-998e-9c0f04bacf61" ``` Tasks: ```text Add protocol: maildir to config and AppConfig validation Implement MaildirMailboxSource reading new/ and cur/ per Maildir spec Derive stable message identity from Maildir filename plus existing dedup keys Support incremental scans via scan_cursors for maildir sources Preserve raw_message_ref as a maildir:// reference Reject mark_seen for maildir the same way IMAP does Wire the source into source_for_config Add unit tests over a temporary Maildir tree ``` Acceptance: ```text The scanner reads a Maildir directory, classifies messages, and produces the same evidence rows as the equivalent fixture directory, with deduplication and incremental cursors working across repeated scans. ``` Done 2026-08-14: * `MaildirMailboxSource` reads `new/` and `cur/`, skips dotfiles, prefers `cur/` when a base name appears in both, and emits `maildir:////` refs. `mark_seen` and a missing directory both raise, matching the read-only IMAP contract. * Identity is the Maildir unique name minus the `:2,FLAGS` suffix, so it is stable across the `new/`→`cur/` move. It is carried on a new `MailboxSourceMessage.dedup_uid` and appended to the message dedup key only when set, leaving existing fixture and IMAP state-store keys byte-identical. * Cursor ordering parses the Maildir delivery time rather than comparing names lexically, so cursors stay correct as the time field changes width. * Fixed alongside: the parse-failure path keyed identity on `raw_message_ref`, which a `new/`→`cur/` move rewrites, so an unparseable message re-registered as new on every rescan. It now prefers the source uid when one exists. Behavior for fixture and IMAP sources is unchanged. * `source.maildir_dir` added to config and the example file; `source_for_config` validates it. `include_seen: false` skips `S`-flagged messages. * Tests: `tests/test_maildir.py`, 11 cases including maildir-vs-fixture evidence parity. Full suite 38 passed. ## T02 - Containerized SMTP/IMAP test server ```task id: EMAIL-WP-0005-T02 status: done priority: high state_hub_task_id: "86727bd5-6738-5bcf-a255-af67065f753e" ``` Tasks: ```text Select and pin a test mail server image providing SMTP and IMAP Add a compose or container definition under a test harness directory Expose SMTP and IMAP on fixed local ports with non-secret test credentials Document startup, teardown, and health check Add a config profile pointing mailbox.protocol imap at the harness Add a config profile pointing the SMTP provider env vars at the harness Ensure the harness never binds a routable interface by default ``` Acceptance: ```text One documented command starts a local mail server; the scanner can complete an IMAP scan against it and the transactional service can complete a send to it, using committed non-secret test credentials only. ``` Notes: ```text Preferred candidate is GreenMail, which serves SMTP and IMAP and creates accounts on demand. Mailpit is the fallback for send-side-only inspection and does not exercise the IMAP source. Record the choice in DECISIONS.md. ``` Done 2026-08-14: * `tests/harness/docker-compose.yml` runs GreenMail `2.1.12`, digest-pinned, SMTP 3025 / IMAP 3143 / API 8080, all bound to `127.0.0.1` only. The image ships no curl/wget/nc, so the healthcheck probes both mail ports with bash `/dev/tcp`. Verified healthy. * `greenmail.auth.disabled` with no declared users: any login is accepted and the mailbox is created on first use, so T03 needs no provisioning step. Declaring users *and* enabling auth-disabled conflicts — GreenMail tries to auto-create the login and collides with the declared address. * `config/harness-imap.yml` scanner profile; env-var recipe for the send side in `tests/harness/README.md`. * GreenMail standalone has no STARTTLS support (confirmed against the shipped jar's property builder — only plain and implicit-TLS setups exist), and `SMTPProvider` hardcoded `starttls()`, so no send could reach it. Added an `EMAIL_CONNECT_SMTP_SECURITY` mode defaulting to `starttls`; `plaintext` is refused for any non-loopback host and hostnames are never resolved to decide it. See DECISIONS.md. * Verified end to end: `SMTPProvider.send` → GreenMail → `ImapMailboxSource` fetch, then the documented CLI (`scan-mailbox --config config/harness-imap.yml`) against the live harness — 1 message seen, parsed, 1 evidence event. * Tests: 14 offline cases for the transport-security guard in `tests/test_transactional.py`. Full suite 52 passed, still container-free. ## T03 - Test-user account provisioning and reset ```task id: EMAIL-WP-0005-T03 status: done priority: high state_hub_task_id: "64f76e62-94dd-5e94-be49-ae989fba1f6d" ``` Tasks: ```text Define the test-user address convention and test domain Provide a helper to create or address per-test mailboxes deterministically Provide per-test reset so state does not leak between tests Keep test credentials non-secret and committed, never routed through OpenBao Document the convention and the reset contract ``` Acceptance: ```text A test can obtain an isolated mailbox for a named test user, send to it, read it back, and reset it, with no cross-test interference and no real credentials involved. ``` Notes: ```text Test credentials are deliberately non-secret and belong in the repo. Per .claude/rules/credential-routing.md, OpenBao and warden route are for real provider material; the harness must not touch them. ``` Done 2026-08-14: * `tests/harness/__init__.py` helper package: `address()`, `available()`, `reset()`, `users()`, `smtp_provider()`, `mailbox_config()`. * Convention is `harness.address("")` → slug@`harness.email-connect.test`. `.test` is RFC 2606 reserved, so a stray send cannot leave the host. No provisioning call is needed — T02's auth-disabled setup creates the mailbox on first login, so distinct names are automatically isolated. * Reset contract: `harness.reset()` (GreenMail `POST /api/service/reset`) drops all users and mail; the next login recreates the mailbox empty. Documented to run at test *start*, so a crashed test cannot leak state forward. It is global, so resetting tests cannot run in parallel against one harness. Verified: send → 1 message and a live user → reset → no users, mailbox empty. * Credentials stay non-secret placeholders in the repo; OpenBao is untouched. * Tests: `tests/test_harness_users.py`, 12 cases. Slug and config cases run offline; the three harness-dependent cases skip when it is down. Suite: 64 passed with the harness up, 61 passed + 3 skipped with it down. ## T04 - End-to-end send-and-scan integration tests ```task id: EMAIL-WP-0005-T04 status: done priority: high state_hub_task_id: "e2470206-f166-5c39-a60c-84de8fdc3fba" ``` Tasks: ```text Add an integration test tier that is skipped when the harness is unavailable Drive /v1/send invitation and verification flows into harness mailboxes Scan the resulting mailboxes through the normal scanner path Assert evidence rows, evidence_ceiling, and absence of overclaiming Cover idempotency and duplicate-request behavior end to end Cover suppression behavior end to end Ensure the default unit test run stays offline and container-free ``` Acceptance: ```text An integration run proves send-to-scan continuity against a live local server, while the default test run remains offline, deterministic, and unchanged. ``` Done 2026-08-14: * `tests/test_integration_send_scan.py`, 7 harness-gated cases: accepted invitation reaches the mailbox and correlates by Message-ID; scanner ingests it without a delivery claim; duplicate event sends once and returns the same reference; resend delivers a second distinct message; suppressed recipient receives nothing; unauthorized / template-denied / key-mismatch requests never reach the provider; verification mail delivers and reports `authorization=false`. * Suite: 72 passed with the harness up, 62 passed + 10 skipped with it down. Two production defects surfaced, both invisible to the fixture-only suite: * **Reply heuristic matched message headers.** `_looks_like_human_reply` ran against headers plus body, so the `Received:` trace that every MTA-handled message carries matched its "received" keyword. Ordinary mail was classified `human_reply` with a `success.reply_received` assessment at medium confidence — exactly the overclaim this repo exists to prevent. Hand-written fixtures carry no `Received:` headers, which is why only real scanned mail exposed it. The heuristic now takes the body alone; DSN detection still sees headers, which it needs. Same message before/after: `human_reply`/medium → `unknown_return_message`/low. Regression test is offline (`tests/fixtures/mailbox_transit/ordinary_transit.eml`). * **Provider reference was not a real reference.** `SMTPProvider.send` fell back to `abs(hash((recipient, subject)))` because it set no `Message-ID`. Python randomizes string hashing per process, and equal recipient/subject pairs collided. Outgoing mail now carries a proper RFC 5322 `Message-ID`, which is returned as the reference — so an accepted send can be tied to a message later observed in a mailbox. This is what makes correlation testable at all. ## T05 - Bounce, complaint, and deferral realism ```task id: EMAIL-WP-0005-T05 status: done priority: medium state_hub_task_id: "87acf840-5d02-5055-b552-34979cb753af" ``` Tasks: ```text Document that local servers do not generate realistic DSNs, complaints, or deferrals Keep crafted .eml fixtures as the deterministic path for those classes Add a helper to inject fixture messages into a harness mailbox or Maildir Define an optional staging tier using provider simulator addresses Document which evidence classes are only reachable in the staging tier ``` Acceptance: ```text Every evidence class the parser recognizes has a documented, reachable test path, and the limits of the local harness are stated explicitly rather than papered over. ``` Done 2026-08-14: * `harness.deliver_raw()` injects a crafted `.eml` through the harness, so the message picks up real `Received:` headers on the way in. `harness.inject_maildir()` is the offline counterpart, byte-exact with no MTA. * `tests/test_evidence_realism.py`, 13 cases: all ten recognized classes are asserted after passing through a real MTA, plus a parity case proving MTA delivery does not change which evidence is produced, plus two offline Maildir injection cases. * Documented what no local server can honestly produce — provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance — and named SES simulator addresses as the staging path. That tier is deliberately not wired into the test run: it needs real credentials, which stay in OpenBao. ## T06 - Documentation and decision record ```task id: EMAIL-WP-0005-T06 status: done priority: medium state_hub_task_id: "61dd5431-8869-57d9-bbe8-14562618b014" ``` Tasks: ```text Write a test harness tutorial covering start, send, scan, assert, reset Extend docs/mailbox-report-tutorial.md with the maildir source Record the DECISIONS.md entry for the harness approach and the s/qmail rejection Document the three test tiers: fixture unit, harness integration, provider staging Document the evidence ceiling that applies to harness-observed mail ``` Acceptance: ```text A new contributor can start the harness, run the integration tier, and correctly state what harness-observed mail does and does not prove. ``` Done 2026-08-14: * `docs/test-harness-tutorial.md`: three test tiers, start, mailbox, send, scan, assert, reset, fixture injection, teardown. Includes an explicit assertable/not-assertable list, so the evidence ceiling is stated where tests get written rather than only in the canon. * `docs/mailbox-report-tutorial.md` gained a Maildir section covering `protocol: maildir`, `source.maildir_dir`, identity across `new/`→`cur/`, and the read-only guarantees. * `tests/harness/README.md` carries the reference material: ports, addressing convention, reset contract, skip gating, injection, and the provider-simulator table. * Decisions were recorded as the work happened: s/qmail rejection and the GreenMail plus loopback-plaintext choice are both in `DECISIONS.md`. ## 8. Completion Criteria This workplan is complete when: 1. `mailbox.protocol: maildir` is supported, tested, and documented. 2. A pinned containerized SMTP/IMAP test server starts from one documented command with committed non-secret credentials. 3. Test users get isolated, resettable mailboxes by convention. 4. Integration tests prove send-to-scan continuity while the default test run stays offline and container-free. 5. Every recognized evidence class has a documented, reachable test path, with local-harness limits stated explicitly. 6. Documentation and the decision record are in place, including the evidence ceiling for harness-observed mail.