Add EMAIL-WP-0005 test mailbox harness workplan
Plans a Maildir mailbox source, a containerized SMTP/IMAP test server, deterministic test-user mailboxes, and end-to-end send-and-scan integration tests, so automated test environments get mail accounts for test users without a live provider. Records the decision to use a purpose-built test server rather than integrating s/qmail, and keeps bounce/complaint/deferral evidence on fixtures plus an optional provider simulator tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
04897a149f
commit
9c0d6f1b40
2 changed files with 320 additions and 0 deletions
287
workplans/EMAIL-WP-0005-test-mailbox-harness.md
Normal file
287
workplans/EMAIL-WP-0005-test-mailbox-harness.md
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
---
|
||||
id: EMAIL-WP-0005
|
||||
type: workplan
|
||||
title: "Test Mailbox Harness for Automated Test Environments"
|
||||
domain: infotech
|
||||
repo: email-connect
|
||||
status: proposed
|
||||
owner: claude
|
||||
topic_slug: custodian
|
||||
created: "2026-08-14"
|
||||
updated: "2026-08-14"
|
||||
---
|
||||
|
||||
# EMAIL-WP-0005 - Test Mailbox Harness for Automated Test Environments
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Automated test environments need mail accounts for test users so that
|
||||
end-to-end flows (invitation send, verification, return-path evidence) can be
|
||||
exercised without a production provider and without hand-crafted fixtures for
|
||||
every case.
|
||||
|
||||
This workplan adds a **test mailbox harness**: a Maildir mailbox source, a
|
||||
containerized SMTP/IMAP test server, deterministic test-user accounts, and
|
||||
end-to-end tests that drive `/v1/send` into the harness and scan the resulting
|
||||
mailboxes back out through the existing scanner.
|
||||
|
||||
The harness is test infrastructure. It does not make `email-connect` a mail
|
||||
provider or an MTA operator, and it must not weaken the evidence discipline:
|
||||
mail observed in a test server is still provider/MX-level fact, never proof of
|
||||
delivery, awareness, identity, or authorization.
|
||||
|
||||
## 2. User Story
|
||||
|
||||
As a developer or CI job, I want to start one container, send invitation and
|
||||
verification mail through `email-connect` to per-test mailboxes, and then scan
|
||||
those mailboxes with the normal scanner path, so that I can assert on end-to-end
|
||||
behavior without a live provider and without network egress.
|
||||
|
||||
## 3. In Scope
|
||||
|
||||
- A `maildir` mailbox source alongside the existing `fixture` and `imap`
|
||||
sources.
|
||||
- A containerized SMTP + IMAP test server for integration tests.
|
||||
- Deterministic per-test-user account provisioning and reset.
|
||||
- End-to-end tests: send through the transactional service, scan through the
|
||||
scanner, assert evidence rows.
|
||||
- A documented strategy for bounce/complaint/deferral realism, which local
|
||||
servers cannot generate honestly.
|
||||
- Documentation and a decision record for the chosen approach.
|
||||
|
||||
## 4. Out of Scope
|
||||
|
||||
- Operating a production or internet-facing MTA.
|
||||
- Owning DNS, TLS certificates, rDNS, SPF/DKIM/DMARC, or spam filtering.
|
||||
- Mailbox write-back actions (the scanner stays read-only; `mark_seen` remains
|
||||
rejected).
|
||||
- A mailbox management UI or user-facing account provisioning.
|
||||
- Replacing the fixture path as the default test source.
|
||||
- Integrating s/qmail — see `DECISIONS.md`, "Test mailbox harness: purpose-built
|
||||
test server over s/qmail".
|
||||
|
||||
## 5. Design Constraints
|
||||
|
||||
The harness must respect the existing seams rather than introduce new ones:
|
||||
|
||||
```text
|
||||
mailbox.protocol fixture | imap | maildir (src/email_connect/mailbox.py)
|
||||
SMTP provider EMAIL_CONNECT_SMTP_HOST/PORT (src/email_connect/transactional.py)
|
||||
```
|
||||
|
||||
Pointing tests at the harness must require configuration only, not code
|
||||
branching inside the scanner or the transactional application. No test-only
|
||||
code paths may exist in the production send or scan logic.
|
||||
|
||||
The fixture source remains the default for unit tests: deterministic, offline,
|
||||
no container required. The harness is the *integration* tier.
|
||||
|
||||
## 6. Evidence Semantics
|
||||
|
||||
Mail retrieved from a test server carries exactly the same evidence ceiling as
|
||||
mail retrieved from a real mailbox. Specifically, the harness must not be used
|
||||
to justify any of:
|
||||
|
||||
```text
|
||||
message was delivered to a real inbox
|
||||
recipient became aware of the message
|
||||
recipient identity was verified
|
||||
recipient was authorized
|
||||
```
|
||||
|
||||
A test-server acceptance is `provider_accepted` and nothing more. Tests that
|
||||
assert stronger claims are defects in the test, not features of the harness.
|
||||
|
||||
## 7. Work Packages
|
||||
|
||||
## T01 - Maildir mailbox source
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Add protocol: maildir to config and AppConfig validation
|
||||
Implement MaildirMailboxSource reading new/ and cur/ per Maildir spec
|
||||
Derive stable message identity from Maildir filename plus existing dedup keys
|
||||
Support incremental scans via scan_cursors for maildir sources
|
||||
Preserve raw_message_ref as a maildir:// reference
|
||||
Reject mark_seen for maildir the same way IMAP does
|
||||
Wire the source into source_for_config
|
||||
Add unit tests over a temporary Maildir tree
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
The scanner reads a Maildir directory, classifies messages, and produces the
|
||||
same evidence rows as the equivalent fixture directory, with deduplication and
|
||||
incremental cursors working across repeated scans.
|
||||
```
|
||||
|
||||
## T02 - Containerized SMTP/IMAP test server
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Select and pin a test mail server image providing SMTP and IMAP
|
||||
Add a compose or container definition under a test harness directory
|
||||
Expose SMTP and IMAP on fixed local ports with non-secret test credentials
|
||||
Document startup, teardown, and health check
|
||||
Add a config profile pointing mailbox.protocol imap at the harness
|
||||
Add a config profile pointing the SMTP provider env vars at the harness
|
||||
Ensure the harness never binds a routable interface by default
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
One documented command starts a local mail server; the scanner can complete an
|
||||
IMAP scan against it and the transactional service can complete a send to it,
|
||||
using committed non-secret test credentials only.
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
```text
|
||||
Preferred candidate is GreenMail, which serves SMTP and IMAP and creates
|
||||
accounts on demand. Mailpit is the fallback for send-side-only inspection and
|
||||
does not exercise the IMAP source. Record the choice in DECISIONS.md.
|
||||
```
|
||||
|
||||
## T03 - Test-user account provisioning and reset
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Define the test-user address convention and test domain
|
||||
Provide a helper to create or address per-test mailboxes deterministically
|
||||
Provide per-test reset so state does not leak between tests
|
||||
Keep test credentials non-secret and committed, never routed through OpenBao
|
||||
Document the convention and the reset contract
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
A test can obtain an isolated mailbox for a named test user, send to it, read
|
||||
it back, and reset it, with no cross-test interference and no real credentials
|
||||
involved.
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
```text
|
||||
Test credentials are deliberately non-secret and belong in the repo. Per
|
||||
.claude/rules/credential-routing.md, OpenBao and warden route are for real
|
||||
provider material; the harness must not touch them.
|
||||
```
|
||||
|
||||
## T04 - End-to-end send-and-scan integration tests
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Add an integration test tier that is skipped when the harness is unavailable
|
||||
Drive /v1/send invitation and verification flows into harness mailboxes
|
||||
Scan the resulting mailboxes through the normal scanner path
|
||||
Assert evidence rows, evidence_ceiling, and absence of overclaiming
|
||||
Cover idempotency and duplicate-request behavior end to end
|
||||
Cover suppression behavior end to end
|
||||
Ensure the default unit test run stays offline and container-free
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
An integration run proves send-to-scan continuity against a live local server,
|
||||
while the default test run remains offline, deterministic, and unchanged.
|
||||
```
|
||||
|
||||
## T05 - Bounce, complaint, and deferral realism
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Document that local servers do not generate realistic DSNs, complaints, or deferrals
|
||||
Keep crafted .eml fixtures as the deterministic path for those classes
|
||||
Add a helper to inject fixture messages into a harness mailbox or Maildir
|
||||
Define an optional staging tier using provider simulator addresses
|
||||
Document which evidence classes are only reachable in the staging tier
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
Every evidence class the parser recognizes has a documented, reachable test
|
||||
path, and the limits of the local harness are stated explicitly rather than
|
||||
papered over.
|
||||
```
|
||||
|
||||
## T06 - Documentation and decision record
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Write a test harness tutorial covering start, send, scan, assert, reset
|
||||
Extend docs/mailbox-report-tutorial.md with the maildir source
|
||||
Record the DECISIONS.md entry for the harness approach and the s/qmail rejection
|
||||
Document the three test tiers: fixture unit, harness integration, provider staging
|
||||
Document the evidence ceiling that applies to harness-observed mail
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
A new contributor can start the harness, run the integration tier, and
|
||||
correctly state what harness-observed mail does and does not prove.
|
||||
```
|
||||
|
||||
## 8. Completion Criteria
|
||||
|
||||
This workplan is complete when:
|
||||
|
||||
1. `mailbox.protocol: maildir` is supported, tested, and documented.
|
||||
2. A pinned containerized SMTP/IMAP test server starts from one documented
|
||||
command with committed non-secret credentials.
|
||||
3. Test users get isolated, resettable mailboxes by convention.
|
||||
4. Integration tests prove send-to-scan continuity while the default test run
|
||||
stays offline and container-free.
|
||||
5. Every recognized evidence class has a documented, reachable test path, with
|
||||
local-harness limits stated explicitly.
|
||||
6. Documentation and the decision record are in place, including the evidence
|
||||
ceiling for harness-observed mail.
|
||||
Loading…
Add table
Add a link
Reference in a new issue