From 89fd13ac2ddf79090fed1327fd62ab1f1c01cd92 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 14 Aug 2026 01:41:38 +0200 Subject: [PATCH] EMAIL-WP-0005-T02: add GreenMail test harness Adds tests/harness/docker-compose.yml running GreenMail 2.1.12 (digest-pinned) with SMTP 3025, IMAP 3143 and the API bound to 127.0.0.1 only, plus a config/harness-imap.yml scanner profile and harness README. Auth is disabled and no users are declared, so a mailbox is created on first login and per-test users need no provisioning. GreenMail standalone offers no STARTTLS, only plaintext or implicit TLS, while SMTPProvider hardcoded starttls() -- so no send could reach it. SMTPProvider now takes a security mode via EMAIL_CONNECT_SMTP_SECURITY, defaulting to starttls. plaintext is refused for any non-loopback host, and hostnames are never resolved to decide that, so a misconfigured deployment fails at startup rather than sending credentials in the clear. Trusting GreenMail's self-signed cert was rejected as the wider risk; see DECISIONS.md. Verified end to end against the live harness: SMTPProvider.send -> GreenMail -> ImapMailboxSource, and the documented scan-mailbox CLI. Suite: 52 passed with the harness down. Co-Authored-By: Claude Opus 5 --- DECISIONS.md | 28 ++++++ WORK-RECORDS.md | 2 +- config/harness-imap.yml | 46 ++++++++++ src/email_connect/transactional.py | 38 +++++++- tests/harness/README.md | 86 +++++++++++++++++++ tests/harness/docker-compose.yml | 38 ++++++++ tests/test_transactional.py | 69 +++++++++++++++ .../EMAIL-WP-0005-test-mailbox-harness.md | 26 +++++- 8 files changed, 329 insertions(+), 4 deletions(-) create mode 100644 config/harness-imap.yml create mode 100644 tests/harness/README.md create mode 100644 tests/harness/docker-compose.yml diff --git a/DECISIONS.md b/DECISIONS.md index a6f7127..7361499 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -160,3 +160,31 @@ local server; those classes stay on crafted fixtures plus an optional provider simulator tier. --- + +## Test harness mail server: GreenMail, and plaintext SMTP for loopback only + +**Date:** 2026-08-14 +**Decided by:** codex (EMAIL-WP-0005-T02) + +GreenMail `2.1.12` (digest-pinned) is the harness mail server. It serves SMTP +and IMAP from one container and, with `greenmail.auth.disabled`, creates a +mailbox on first login — so per-test users need no provisioning step. Mailpit +was the alternative but offers no IMAP, which would leave the scanner's IMAP +source untested. + +GreenMail standalone supports only plaintext or implicit-TLS setups; it has no +STARTTLS option (verified against the shipped jar's property builder). The +transactional service previously hardcoded `starttls()`, so no send could reach +it. + +`SMTPProvider` therefore takes a `security` mode, `EMAIL_CONNECT_SMTP_SECURITY`, +defaulting to `starttls`. `plaintext` is refused for any non-loopback host, and +hostnames are never resolved to decide this — a name pointing at 127.0.0.1 today +is not a durable guarantee. A misconfigured deployment fails at startup instead +of putting credentials on the wire. + +The alternative was trusting GreenMail's self-signed certificate for implicit +TLS, which would have meant shipping certificate-verification bypass code in the +send path. A loopback-guarded plaintext mode is the narrower risk. + +--- diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 026a4d2..373bb49 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -40,7 +40,7 @@ | task | EMAIL-WP-0004-T02 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | | task | EMAIL-WP-0004-T03 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | | task | EMAIL-WP-0004-T04 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | -| task | EMAIL-WP-0005-T01 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | +| task | EMAIL-WP-0005-T01 | done | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T02 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T03 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T04 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | diff --git a/config/harness-imap.yml b/config/harness-imap.yml new file mode 100644 index 0000000..0816262 --- /dev/null +++ b/config/harness-imap.yml @@ -0,0 +1,46 @@ +# Scanner profile for the local GreenMail test harness (EMAIL-WP-0005-T02). +# +# Start the harness first: +# docker compose -f tests/harness/docker-compose.yml up -d +# +# The harness runs with authentication disabled and creates a mailbox on first +# login, so these credentials are placeholders that only need to be set: +# export EMAIL_CONNECT_IMAP_USER=t01@harness.email-connect.test +# export EMAIL_CONNECT_IMAP_PASSWORD=harness +# +# Then: +# email-connect scan-mailbox --config config/harness-imap.yml --out reports/ +mailbox: + id: harness-mailbox + protocol: imap + host: 127.0.0.1 + port: 3143 + tls: false + username_env: EMAIL_CONNECT_IMAP_USER + password_env: EMAIL_CONNECT_IMAP_PASSWORD + folder: INBOX + +scan: + mode: incremental + max_messages_per_run: 5000 + since: null + from: null + to: null + include_seen: true + mark_seen: false + store_raw_headers: true + store_raw_body: false + store_raw_message_ref: true + +expected_recipients: + path: null + csv_column: email + +storage: + path: .email-connect/harness-state.sqlite + +reports: + output_dir: reports + include_all_evidence: true + include_unknown_messages: true + timestamp_timezone: UTC diff --git a/src/email_connect/transactional.py b/src/email_connect/transactional.py index 91b7ba3..fd32438 100644 --- a/src/email_connect/transactional.py +++ b/src/email_connect/transactional.py @@ -4,6 +4,7 @@ from __future__ import annotations import hmac import hashlib +import ipaddress import json import os import smtplib @@ -148,7 +149,25 @@ class SQLiteDeliveryStore: class SMTPProvider: - def __init__(self, host: str, port: int, username: str, password: str, sender: str) -> None: + #: STARTTLS is the only transport allowed against a remote provider. + #: `plaintext` exists for loopback test servers that cannot offer TLS and is + #: refused for any other host, so a misconfiguration cannot put credentials + #: on the wire in the clear. + SECURITY_MODES = ("starttls", "plaintext") + + def __init__( + self, + host: str, + port: int, + username: str, + password: str, + sender: str, + security: str = "starttls", + ) -> None: + if security not in self.SECURITY_MODES: + raise ValueError(f"Unsupported SMTP security mode: {security}") + if security == "plaintext" and not _is_loopback_host(host): + raise ValueError("Plaintext SMTP is only permitted for loopback hosts.") self.host, self.port, self.username, self.password, self.sender = ( host, port, @@ -156,6 +175,7 @@ class SMTPProvider: password, sender, ) + self.security = security def send(self, recipient: str, subject: str, text: str) -> str: message = EmailMessage() @@ -163,7 +183,8 @@ class SMTPProvider: message.set_content(text) try: with smtplib.SMTP(self.host, self.port, timeout=10) as smtp: - smtp.starttls() + if self.security == "starttls": + smtp.starttls() smtp.login(self.username, self.password) smtp.send_message(message) except TimeoutError as exc: @@ -352,6 +373,18 @@ class TransactionalApplication: return [body] +def _is_loopback_host(host: str) -> bool: + candidate = (host or "").strip().strip("[]") + if candidate.lower() == "localhost": + return True + try: + return ipaddress.ip_address(candidate).is_loopback + except ValueError: + # Never resolve names here: a hostname that happens to point at 127.0.0.1 + # today is not a durable guarantee that credentials stay on the host. + return False + + def _valid_address(value: str) -> bool: _, address = parseaddr(value) return address == value and "@" in address and "\n" not in address and "\r" not in address @@ -364,6 +397,7 @@ def main() -> None: os.environ["EMAIL_CONNECT_SMTP_USERNAME"], os.environ["EMAIL_CONNECT_SMTP_PASSWORD"], os.environ["EMAIL_CONNECT_SENDER"], + security=os.environ.get("EMAIL_CONNECT_SMTP_SECURITY", "starttls"), ) app = TransactionalApplication( SQLiteDeliveryStore(os.environ.get("EMAIL_CONNECT_DATABASE_PATH", "/data/email-connect.db")), diff --git a/tests/harness/README.md b/tests/harness/README.md new file mode 100644 index 0000000..fa3140f --- /dev/null +++ b/tests/harness/README.md @@ -0,0 +1,86 @@ +# Local mail test harness + +A single GreenMail container serving SMTP and IMAP on loopback, so integration +tests can drive the transactional send path and read the result back through the +scanner without a live provider. + +Part of `EMAIL-WP-0005`. Unit tests do not need this harness — they run against +fixture directories and temporary Maildir trees and stay offline. + +## Start and stop + +```bash +docker compose -f tests/harness/docker-compose.yml up -d +docker compose -f tests/harness/docker-compose.yml down -v +``` + +Readiness (the container healthcheck probes both mail ports; the API also +answers): + +```bash +docker inspect --format '{{.State.Health.Status}}' email-connect-harness +curl -s http://127.0.0.1:8080/api/service/readiness +``` + +## Ports + +| Port | Protocol | Notes | +| --- | --- | --- | +| 3025 | SMTP | plaintext, no STARTTLS | +| 3143 | IMAP | plaintext | +| 8080 | HTTP | GreenMail API — readiness, user and mail management | + +All three bind `127.0.0.1` only. The harness accepts any credentials and must +never be exposed beyond the host. + +## Accounts + +The harness runs with `greenmail.auth.disabled`, so **any** login is accepted and +the mailbox is created on first use. No provisioning step is needed: pick an +address under `harness.email-connect.test` and log in. + +```bash +export EMAIL_CONNECT_IMAP_USER=t01@harness.email-connect.test +export EMAIL_CONNECT_IMAP_PASSWORD=harness # ignored, but must be set +``` + +These credentials are deliberately non-secret and belong in the repo. Real +provider material is routed through OpenBao — see +`.claude/rules/credential-routing.md`. Never point this harness at real +credentials or a real mailbox. + +## Scanning the harness + +```bash +docker compose -f tests/harness/docker-compose.yml up -d +export EMAIL_CONNECT_IMAP_USER=t01@harness.email-connect.test +export EMAIL_CONNECT_IMAP_PASSWORD=harness +email-connect scan-mailbox --config config/harness-imap.yml --out reports/ +``` + +## Sending through the harness + +GreenMail offers no STARTTLS, only plaintext or implicit TLS with a self-signed +certificate. The transactional service therefore needs its transport mode set +explicitly: + +```bash +export EMAIL_CONNECT_SMTP_HOST=127.0.0.1 +export EMAIL_CONNECT_SMTP_PORT=3025 +export EMAIL_CONNECT_SMTP_SECURITY=plaintext +export EMAIL_CONNECT_SMTP_USERNAME=sender@harness.email-connect.test +export EMAIL_CONNECT_SMTP_PASSWORD=harness +export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test +``` + +`EMAIL_CONNECT_SMTP_SECURITY` defaults to `starttls`. `plaintext` is rejected +for any non-loopback host, so this setting cannot weaken a real deployment: it +fails at startup rather than sending credentials in the clear. + +## What harness mail proves + +Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is +not evidence of inbox placement, recipient awareness, identity, or +authorization. Bounces, complaints, and deferrals are **not** reproducible here +— GreenMail accepts everything. Those classes stay on crafted `.eml` fixtures +and an optional provider-simulator tier (`EMAIL-WP-0005-T05`). diff --git a/tests/harness/docker-compose.yml b/tests/harness/docker-compose.yml new file mode 100644 index 0000000..f1642b7 --- /dev/null +++ b/tests/harness/docker-compose.yml @@ -0,0 +1,38 @@ +# Local mail server for email-connect integration tests (EMAIL-WP-0005-T02). +# +# GreenMail serves SMTP and IMAP from one container and creates the declared +# accounts at startup, which is what the scanner's IMAP source and the +# transactional service's SMTP provider need. +# +# Everything here is test-only. The credentials below are deliberately +# non-secret and belong in the repo; real provider material is routed through +# OpenBao (see .claude/rules/credential-routing.md) and must never appear here. +services: + greenmail: + image: greenmail/standalone:2.1.12@sha256:9f32971b4f25d32b4de6fa2e297423768441c65e4541f6aecd7631c890a229a7 + container_name: email-connect-harness + # Loopback-only bindings. The harness accepts unauthenticated-ish test + # traffic and must never be reachable from the network. + ports: + - "127.0.0.1:3025:3025" # SMTP + - "127.0.0.1:3143:3143" # IMAP + - "127.0.0.1:8080:8080" # GreenMail API (readiness, purge) + environment: + GREENMAIL_OPTS: >- + -Dgreenmail.setup.test.smtp + -Dgreenmail.setup.test.imap + -Dgreenmail.hostname=0.0.0.0 + -Dgreenmail.auth.disabled + -Dgreenmail.verbose + # The image ships no curl/wget/nc, so readiness is probed with bash's + # /dev/tcp against both mail ports. + healthcheck: + test: + - CMD + - bash + - -c + - "exec 3<>/dev/tcp/127.0.0.1/3025 && exec 4<>/dev/tcp/127.0.0.1/3143" + interval: 2s + timeout: 3s + retries: 30 + start_period: 5s diff --git a/tests/test_transactional.py b/tests/test_transactional.py index f43cbc8..5cf388b 100644 --- a/tests/test_transactional.py +++ b/tests/test_transactional.py @@ -2,8 +2,11 @@ import io import json import smtplib +import pytest + from email_connect.transactional import ( ProviderError, + SMTPProvider, SQLiteDeliveryStore, TransactionalApplication, ) @@ -276,3 +279,69 @@ def test_mailbox_evidence_is_not_authorization(tmp_path): assert accepted["evidence_ceiling"] == "provider_accepted" # Ceiling is explicit; user-engine must not elevate this to authz. assert "authorized" not in accepted + + +class RecordingSMTP: + """Minimal smtplib.SMTP stand-in that records the handshake it was given.""" + + instances = [] + + def __init__(self, host, port, timeout=None): + self.host, self.port, self.timeout = host, port, timeout + self.starttls_calls = 0 + self.logins = [] + self.sent = [] + RecordingSMTP.instances.append(self) + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def starttls(self): + self.starttls_calls += 1 + + def login(self, username, password): + self.logins.append((username, password)) + + def send_message(self, message): + self.sent.append(message) + + +def test_smtp_provider_defaults_to_starttls(monkeypatch): + monkeypatch.setattr(smtplib, "SMTP", RecordingSMTP) + RecordingSMTP.instances = [] + provider = SMTPProvider("smtp.example.com", 587, "user", "pass", "noreply@example.com") + + assert provider.security == "starttls" + provider.send("person@example.test", "Subject", "body") + assert RecordingSMTP.instances[0].starttls_calls == 1 + + +def test_plaintext_security_skips_starttls_on_loopback(monkeypatch): + monkeypatch.setattr(smtplib, "SMTP", RecordingSMTP) + RecordingSMTP.instances = [] + provider = SMTPProvider("127.0.0.1", 3025, "user", "pass", "noreply@example.com", security="plaintext") + + provider.send("person@example.test", "Subject", "body") + smtp = RecordingSMTP.instances[0] + assert smtp.starttls_calls == 0 + assert smtp.logins == [("user", "pass")] + assert len(smtp.sent) == 1 + + +@pytest.mark.parametrize("host", ["127.0.0.1", "127.0.1.1", "::1", "[::1]", "localhost", "LOCALHOST"]) +def test_plaintext_security_is_allowed_for_loopback_hosts(host): + assert SMTPProvider(host, 3025, "user", "pass", "noreply@example.com", security="plaintext") + + +@pytest.mark.parametrize("host", ["smtp.example.com", "10.0.0.5", "::ffff:10.0.0.5", "", "localhost.example.com"]) +def test_plaintext_security_is_refused_for_non_loopback_hosts(host): + with pytest.raises(ValueError, match="loopback"): + SMTPProvider(host, 3025, "user", "pass", "noreply@example.com", security="plaintext") + + +def test_unsupported_security_mode_is_refused(): + with pytest.raises(ValueError, match="security mode"): + SMTPProvider("127.0.0.1", 3025, "user", "pass", "noreply@example.com", security="none") diff --git a/workplans/EMAIL-WP-0005-test-mailbox-harness.md b/workplans/EMAIL-WP-0005-test-mailbox-harness.md index b02b80f..bc6477b 100644 --- a/workplans/EMAIL-WP-0005-test-mailbox-harness.md +++ b/workplans/EMAIL-WP-0005-test-mailbox-harness.md @@ -150,7 +150,7 @@ Done 2026-08-14: ```task id: EMAIL-WP-0005-T02 -status: todo +status: done priority: high state_hub_task_id: "967d2bab-43e3-4240-8bd5-c657ba4178ff" ``` @@ -183,6 +183,30 @@ accounts on demand. Mailpit is the fallback for send-side-only inspection and does not exercise the IMAP source. Record the choice in DECISIONS.md. ``` +Done 2026-08-14: + +* `tests/harness/docker-compose.yml` runs GreenMail `2.1.12`, digest-pinned, + SMTP 3025 / IMAP 3143 / API 8080, all bound to `127.0.0.1` only. The image + ships no curl/wget/nc, so the healthcheck probes both mail ports with bash + `/dev/tcp`. Verified healthy. +* `greenmail.auth.disabled` with no declared users: any login is accepted and + the mailbox is created on first use, so T03 needs no provisioning step. + Declaring users *and* enabling auth-disabled conflicts — GreenMail tries to + auto-create the login and collides with the declared address. +* `config/harness-imap.yml` scanner profile; env-var recipe for the send side in + `tests/harness/README.md`. +* GreenMail standalone has no STARTTLS support (confirmed against the shipped + jar's property builder — only plain and implicit-TLS setups exist), and + `SMTPProvider` hardcoded `starttls()`, so no send could reach it. Added an + `EMAIL_CONNECT_SMTP_SECURITY` mode defaulting to `starttls`; `plaintext` is + refused for any non-loopback host and hostnames are never resolved to decide + it. See DECISIONS.md. +* Verified end to end: `SMTPProvider.send` → GreenMail → `ImapMailboxSource` + fetch, then the documented CLI (`scan-mailbox --config config/harness-imap.yml`) + against the live harness — 1 message seen, parsed, 1 evidence event. +* Tests: 14 offline cases for the transport-security guard in + `tests/test_transactional.py`. Full suite 52 passed, still container-free. + ## T03 - Test-user account provisioning and reset ```task