Add EMAIL-WP-0005 test mailbox harness workplan
Plans a Maildir mailbox source, a containerized SMTP/IMAP test server, deterministic test-user mailboxes, and end-to-end send-and-scan integration tests, so automated test environments get mail accounts for test users without a live provider. Records the decision to use a purpose-built test server rather than integrating s/qmail, and keeps bounce/complaint/deferral evidence on fixtures plus an optional provider simulator tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
04897a149f
commit
9c0d6f1b40
2 changed files with 320 additions and 0 deletions
33
DECISIONS.md
33
DECISIONS.md
|
|
@ -127,3 +127,36 @@ We will do C and allow for early customers to easy starting serving customere bu
|
|||
We will go for B as Coulomb Social is it's own Application. Managed by connecting it with our infrastructure but destinct in its purpose and operations model. It is an Application for the Binky Hedgehog GmbH as another App might be provided to other clients. It should not be relevant the the ecosystem as a whole is provided by Binky Hedgehog as the organization creating coulomb and holding the domain and forgejo repos.
|
||||
|
||||
---
|
||||
|
||||
## Test mailbox harness: purpose-built test server over s/qmail
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Decided by:** human
|
||||
|
||||
Automated test environments need mail accounts for test users. We will build a
|
||||
test harness (EMAIL-WP-0005) around a Maildir mailbox source plus a
|
||||
containerized SMTP/IMAP test server, rather than integrating s/qmail.
|
||||
|
||||
Reasons against s/qmail here:
|
||||
|
||||
* qmail lineage is an MTA — SMTP transport and Maildir delivery. Test users need
|
||||
IMAP mailboxes, which would require bolting on Dovecot or Courier anyway, so
|
||||
s/qmail solves the half we least need.
|
||||
* Running it means operating a real mail server for test purposes: DNS, TLS,
|
||||
rDNS, SPF/DKIM/DMARC, queue and spam handling. That is ongoing ops cost
|
||||
charged against test infrastructure.
|
||||
* It inverts the repo's scope boundary. `email-connect` is provider-neutral and
|
||||
deliberately does not own an MTA; absorbing one makes us the provider we are
|
||||
supposed to abstract over.
|
||||
* Configuration is idiosyncratic and container/CI recipes are thin compared to
|
||||
purpose-built test servers, which raises the debugging-budget risk.
|
||||
|
||||
If a long-lived, internet-facing test domain with many persistent accounts is
|
||||
ever needed, revisit with docker-mailserver (Postfix + Dovecot) or Stalwart
|
||||
before reconsidering s/qmail.
|
||||
|
||||
Bounce, complaint, and deferral evidence is explicitly not obtainable from a
|
||||
local server; those classes stay on crafted fixtures plus an optional provider
|
||||
simulator tier.
|
||||
|
||||
---
|
||||
|
|
|
|||
287
workplans/EMAIL-WP-0005-test-mailbox-harness.md
Normal file
287
workplans/EMAIL-WP-0005-test-mailbox-harness.md
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
---
|
||||
id: EMAIL-WP-0005
|
||||
type: workplan
|
||||
title: "Test Mailbox Harness for Automated Test Environments"
|
||||
domain: infotech
|
||||
repo: email-connect
|
||||
status: proposed
|
||||
owner: claude
|
||||
topic_slug: custodian
|
||||
created: "2026-08-14"
|
||||
updated: "2026-08-14"
|
||||
---
|
||||
|
||||
# EMAIL-WP-0005 - Test Mailbox Harness for Automated Test Environments
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Automated test environments need mail accounts for test users so that
|
||||
end-to-end flows (invitation send, verification, return-path evidence) can be
|
||||
exercised without a production provider and without hand-crafted fixtures for
|
||||
every case.
|
||||
|
||||
This workplan adds a **test mailbox harness**: a Maildir mailbox source, a
|
||||
containerized SMTP/IMAP test server, deterministic test-user accounts, and
|
||||
end-to-end tests that drive `/v1/send` into the harness and scan the resulting
|
||||
mailboxes back out through the existing scanner.
|
||||
|
||||
The harness is test infrastructure. It does not make `email-connect` a mail
|
||||
provider or an MTA operator, and it must not weaken the evidence discipline:
|
||||
mail observed in a test server is still provider/MX-level fact, never proof of
|
||||
delivery, awareness, identity, or authorization.
|
||||
|
||||
## 2. User Story
|
||||
|
||||
As a developer or CI job, I want to start one container, send invitation and
|
||||
verification mail through `email-connect` to per-test mailboxes, and then scan
|
||||
those mailboxes with the normal scanner path, so that I can assert on end-to-end
|
||||
behavior without a live provider and without network egress.
|
||||
|
||||
## 3. In Scope
|
||||
|
||||
- A `maildir` mailbox source alongside the existing `fixture` and `imap`
|
||||
sources.
|
||||
- A containerized SMTP + IMAP test server for integration tests.
|
||||
- Deterministic per-test-user account provisioning and reset.
|
||||
- End-to-end tests: send through the transactional service, scan through the
|
||||
scanner, assert evidence rows.
|
||||
- A documented strategy for bounce/complaint/deferral realism, which local
|
||||
servers cannot generate honestly.
|
||||
- Documentation and a decision record for the chosen approach.
|
||||
|
||||
## 4. Out of Scope
|
||||
|
||||
- Operating a production or internet-facing MTA.
|
||||
- Owning DNS, TLS certificates, rDNS, SPF/DKIM/DMARC, or spam filtering.
|
||||
- Mailbox write-back actions (the scanner stays read-only; `mark_seen` remains
|
||||
rejected).
|
||||
- A mailbox management UI or user-facing account provisioning.
|
||||
- Replacing the fixture path as the default test source.
|
||||
- Integrating s/qmail — see `DECISIONS.md`, "Test mailbox harness: purpose-built
|
||||
test server over s/qmail".
|
||||
|
||||
## 5. Design Constraints
|
||||
|
||||
The harness must respect the existing seams rather than introduce new ones:
|
||||
|
||||
```text
|
||||
mailbox.protocol fixture | imap | maildir (src/email_connect/mailbox.py)
|
||||
SMTP provider EMAIL_CONNECT_SMTP_HOST/PORT (src/email_connect/transactional.py)
|
||||
```
|
||||
|
||||
Pointing tests at the harness must require configuration only, not code
|
||||
branching inside the scanner or the transactional application. No test-only
|
||||
code paths may exist in the production send or scan logic.
|
||||
|
||||
The fixture source remains the default for unit tests: deterministic, offline,
|
||||
no container required. The harness is the *integration* tier.
|
||||
|
||||
## 6. Evidence Semantics
|
||||
|
||||
Mail retrieved from a test server carries exactly the same evidence ceiling as
|
||||
mail retrieved from a real mailbox. Specifically, the harness must not be used
|
||||
to justify any of:
|
||||
|
||||
```text
|
||||
message was delivered to a real inbox
|
||||
recipient became aware of the message
|
||||
recipient identity was verified
|
||||
recipient was authorized
|
||||
```
|
||||
|
||||
A test-server acceptance is `provider_accepted` and nothing more. Tests that
|
||||
assert stronger claims are defects in the test, not features of the harness.
|
||||
|
||||
## 7. Work Packages
|
||||
|
||||
## T01 - Maildir mailbox source
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Add protocol: maildir to config and AppConfig validation
|
||||
Implement MaildirMailboxSource reading new/ and cur/ per Maildir spec
|
||||
Derive stable message identity from Maildir filename plus existing dedup keys
|
||||
Support incremental scans via scan_cursors for maildir sources
|
||||
Preserve raw_message_ref as a maildir:// reference
|
||||
Reject mark_seen for maildir the same way IMAP does
|
||||
Wire the source into source_for_config
|
||||
Add unit tests over a temporary Maildir tree
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
The scanner reads a Maildir directory, classifies messages, and produces the
|
||||
same evidence rows as the equivalent fixture directory, with deduplication and
|
||||
incremental cursors working across repeated scans.
|
||||
```
|
||||
|
||||
## T02 - Containerized SMTP/IMAP test server
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Select and pin a test mail server image providing SMTP and IMAP
|
||||
Add a compose or container definition under a test harness directory
|
||||
Expose SMTP and IMAP on fixed local ports with non-secret test credentials
|
||||
Document startup, teardown, and health check
|
||||
Add a config profile pointing mailbox.protocol imap at the harness
|
||||
Add a config profile pointing the SMTP provider env vars at the harness
|
||||
Ensure the harness never binds a routable interface by default
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
One documented command starts a local mail server; the scanner can complete an
|
||||
IMAP scan against it and the transactional service can complete a send to it,
|
||||
using committed non-secret test credentials only.
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
```text
|
||||
Preferred candidate is GreenMail, which serves SMTP and IMAP and creates
|
||||
accounts on demand. Mailpit is the fallback for send-side-only inspection and
|
||||
does not exercise the IMAP source. Record the choice in DECISIONS.md.
|
||||
```
|
||||
|
||||
## T03 - Test-user account provisioning and reset
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Define the test-user address convention and test domain
|
||||
Provide a helper to create or address per-test mailboxes deterministically
|
||||
Provide per-test reset so state does not leak between tests
|
||||
Keep test credentials non-secret and committed, never routed through OpenBao
|
||||
Document the convention and the reset contract
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
A test can obtain an isolated mailbox for a named test user, send to it, read
|
||||
it back, and reset it, with no cross-test interference and no real credentials
|
||||
involved.
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
```text
|
||||
Test credentials are deliberately non-secret and belong in the repo. Per
|
||||
.claude/rules/credential-routing.md, OpenBao and warden route are for real
|
||||
provider material; the harness must not touch them.
|
||||
```
|
||||
|
||||
## T04 - End-to-end send-and-scan integration tests
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Add an integration test tier that is skipped when the harness is unavailable
|
||||
Drive /v1/send invitation and verification flows into harness mailboxes
|
||||
Scan the resulting mailboxes through the normal scanner path
|
||||
Assert evidence rows, evidence_ceiling, and absence of overclaiming
|
||||
Cover idempotency and duplicate-request behavior end to end
|
||||
Cover suppression behavior end to end
|
||||
Ensure the default unit test run stays offline and container-free
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
An integration run proves send-to-scan continuity against a live local server,
|
||||
while the default test run remains offline, deterministic, and unchanged.
|
||||
```
|
||||
|
||||
## T05 - Bounce, complaint, and deferral realism
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Document that local servers do not generate realistic DSNs, complaints, or deferrals
|
||||
Keep crafted .eml fixtures as the deterministic path for those classes
|
||||
Add a helper to inject fixture messages into a harness mailbox or Maildir
|
||||
Define an optional staging tier using provider simulator addresses
|
||||
Document which evidence classes are only reachable in the staging tier
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
Every evidence class the parser recognizes has a documented, reachable test
|
||||
path, and the limits of the local harness are stated explicitly rather than
|
||||
papered over.
|
||||
```
|
||||
|
||||
## T06 - Documentation and decision record
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0005-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tasks:
|
||||
|
||||
```text
|
||||
Write a test harness tutorial covering start, send, scan, assert, reset
|
||||
Extend docs/mailbox-report-tutorial.md with the maildir source
|
||||
Record the DECISIONS.md entry for the harness approach and the s/qmail rejection
|
||||
Document the three test tiers: fixture unit, harness integration, provider staging
|
||||
Document the evidence ceiling that applies to harness-observed mail
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
```text
|
||||
A new contributor can start the harness, run the integration tier, and
|
||||
correctly state what harness-observed mail does and does not prove.
|
||||
```
|
||||
|
||||
## 8. Completion Criteria
|
||||
|
||||
This workplan is complete when:
|
||||
|
||||
1. `mailbox.protocol: maildir` is supported, tested, and documented.
|
||||
2. A pinned containerized SMTP/IMAP test server starts from one documented
|
||||
command with committed non-secret credentials.
|
||||
3. Test users get isolated, resettable mailboxes by convention.
|
||||
4. Integration tests prove send-to-scan continuity while the default test run
|
||||
stays offline and container-free.
|
||||
5. Every recognized evidence class has a documented, reachable test path, with
|
||||
local-harness limits stated explicitly.
|
||||
6. Documentation and the decision record are in place, including the evidence
|
||||
ceiling for harness-observed mail.
|
||||
Loading…
Add table
Add a link
Reference in a new issue