From 3497ca88bf446d34532bd25c3c5191d2d7af2de7 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 14 Aug 2026 02:00:17 +0200 Subject: [PATCH] EMAIL-WP-0005 T05/T06: evidence realism and harness documentation Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness, so they pick up the real Received: headers an MTA adds, and harness.inject_maildir() as the byte-exact offline counterpart. tests/test_evidence_realism.py asserts all ten recognized evidence classes after passing through a real MTA, proves MTA delivery does not change which evidence is produced, and covers Maildir injection offline. Documents what no local server can honestly produce -- provider-generated DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression behavior, MX acceptance as distinct from provider acceptance -- and names SES simulator addresses as the staging path. That tier stays out of the test run because it needs real credentials. Adds docs/test-harness-tutorial.md covering the three test tiers and the start/send/scan/assert/reset walkthrough, with an explicit assertable/not-assertable list so the evidence ceiling is stated where tests get written. Adds a Maildir section to the mailbox report tutorial. Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed + 21 skipped with it down. Co-Authored-By: Claude Opus 5 --- WORK-RECORDS.md | 2 +- docs/mailbox-report-tutorial.md | 26 +++ docs/test-harness-tutorial.md | 164 ++++++++++++++++++ tests/harness/README.md | 44 +++++ tests/harness/__init__.py | 34 ++++ tests/test_evidence_realism.py | 129 ++++++++++++++ .../EMAIL-WP-0005-test-mailbox-harness.md | 36 +++- 7 files changed, 431 insertions(+), 4 deletions(-) create mode 100644 docs/test-harness-tutorial.md create mode 100644 tests/test_evidence_realism.py diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 1c3958c..aae3185 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -43,6 +43,6 @@ | task | EMAIL-WP-0005-T01 | done | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T02 | done | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T03 | done | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | -| task | EMAIL-WP-0005-T04 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | +| task | EMAIL-WP-0005-T04 | done | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T05 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | | task | EMAIL-WP-0005-T06 | todo | — | workplans/EMAIL-WP-0005-test-mailbox-harness.md | diff --git a/docs/mailbox-report-tutorial.md b/docs/mailbox-report-tutorial.md index e7284ae..7e356ac 100644 --- a/docs/mailbox-report-tutorial.md +++ b/docs/mailbox-report-tutorial.md @@ -50,6 +50,32 @@ export EMAIL_CONNECT_IMAP_PASSWORD='app-password' IMAP scans select the folder read-only and fetch messages with `BODY.PEEK[]`. The scanner does not mark messages seen, move messages, or delete messages. +## 3b. Scan A Maildir Instead + +When mail is delivered locally — an MTA writing to disk, or the test harness — +point the scanner at the Maildir root instead of an IMAP server: + +```yaml +mailbox: + protocol: maildir + folder: INBOX + +source: + maildir_dir: /var/mail/returns/Maildir +``` + +The directory is the mailbox root containing `new/` and `cur/`; both are read. +No credentials are involved. Message identity is the Maildir unique name without +its `:2,FLAGS` suffix, so a message stays the same message when the MTA or a +mail client moves it from `new/` to `cur/`. + +Like IMAP, Maildir scans are read-only: `mark_seen` is rejected, and the scanner +never moves, flags, or deletes a message. `include_seen: false` skips messages +already flagged `S`. + +For a local Maildir or IMAP server to test against, see +`docs/test-harness-tutorial.md`. + ## 4. Add Expected Recipients Expected recipients are optional. A newline-separated file can look like: diff --git a/docs/test-harness-tutorial.md b/docs/test-harness-tutorial.md new file mode 100644 index 0000000..acf27ac --- /dev/null +++ b/docs/test-harness-tutorial.md @@ -0,0 +1,164 @@ +# Test Harness Tutorial + +How to run `email-connect` against a local mail server: start it, send through +it, scan it back, assert on the result, and reset between tests. + +Reference material lives in `tests/harness/README.md`. This page is the +walkthrough. + +## 1. The three test tiers + +`email-connect` tests at three levels, and it matters which one a given +assertion belongs to. + +| Tier | Source | Needs | What it is for | +| --- | --- | --- | --- | +| Unit | `.eml` fixtures, temporary Maildir trees | nothing | Parsing, classification, reporting. Deterministic and offline. | +| Integration | local GreenMail harness | Docker | Send-to-scan continuity through a real SMTP/IMAP server. | +| Provider staging | real provider account | credentials, operator | Behavior only a real provider and remote MX produce. | + +The default `pytest` run is the unit tier plus any integration tests whose +harness happens to be up. Nothing in the default run requires a container: with +the harness down the integration cases skip. + +The staging tier is not wired into the test run at all. It needs real +credentials, which are routed through OpenBao — see +`.claude/rules/credential-routing.md` — and must never be pointed at the +harness. + +## 2. Start the harness + +```bash +docker compose -f tests/harness/docker-compose.yml up -d +docker inspect --format '{{.State.Health.Status}}' email-connect-harness +``` + +Wait for `healthy`. The container serves SMTP on 3025 and IMAP on 3143, both +bound to `127.0.0.1` only. + +## 3. Give the test a mailbox + +Authentication is disabled and mailboxes are created on first login, so there is +no provisioning step — pick an address and use it: + +```python +import harness + +recipient = harness.address("invitation delivered") +# -> invitation-delivered@harness.email-connect.test +``` + +Slugging the test's own name keeps two tests from colliding. The domain is under +`.test`, reserved by RFC 2606, so a stray send cannot leave the host. + +## 4. Send through the service + +```python +from email_connect.transactional import SQLiteDeliveryStore, TransactionalApplication + +app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / "mail.db")), + harness.smtp_provider(), + "harness-ingest-token", + "https://users.harness.email-connect.test", +) +``` + +`harness.smtp_provider()` returns a real `SMTPProvider` in plaintext transport +mode. GreenMail offers no STARTTLS, and `SMTPProvider` permits plaintext only +for loopback hosts, so this cannot be pointed at a real provider by accident. + +## 5. Scan it back + +```python +from email_connect.scanner import scan_mailbox + +config = harness.mailbox_config( + recipient, + storage_path=str(tmp_path / "state.sqlite"), + reports_dir=str(tmp_path / "reports"), +) +result = scan_mailbox(config) +``` + +From the command line: + +```bash +export EMAIL_CONNECT_IMAP_USER=invitation-delivered@harness.email-connect.test +export EMAIL_CONNECT_IMAP_PASSWORD=harness +email-connect scan-mailbox --config config/harness-imap.yml --out reports/ +``` + +## 6. Assert — and what you may not assert + +An accepted send is `provider_accepted`. A message sitting in a harness mailbox +is a message sitting in a harness mailbox. Neither is evidence of inbox +placement, recipient awareness, identity, or authorization, and a test that +asserts otherwise is wrong even when it passes. + +Assertable: + +```python +assert body["evidence_ceiling"] == "provider_accepted" +assert body["reference"] == str(messages[0]["Message-ID"]).strip() +assert len(delivered_messages(recipient)) == 1 +``` + +Not assertable from harness mail — these are the claims the repo exists to +avoid: + +```text +the recipient received it +the recipient read it +the recipient's identity is confirmed +the recipient is authorized +the intended result was satisfied +``` + +`coordination-engine` evaluates results. `email-connect` reports what the email +channel observed. + +## 7. Reset between tests + +```python +def test_something(): + harness.reset() + ... +``` + +`harness.reset()` drops every user and message; the next login recreates the +mailbox empty. Call it at the **start** of a test so a crashed or interrupted +run cannot leave state for the next one. It is global, so tests that reset +cannot run in parallel against a single harness. + +## 8. Bounces, complaints and deferrals + +The harness accepts everything, so it never produces these on its own. Inject a +crafted fixture instead: + +```python +harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes()) +``` + +Injected messages still travel through a real MTA and pick up real `Received:` +headers — which is how `EMAIL-WP-0005-T04` caught a reply heuristic that was +matching those headers and labelling ordinary mail a human reply. + +`harness.inject_maildir()` is the offline counterpart, writing straight into a +Maildir tree with no MTA and no harness. + +Provider-generated DSNs, real deferral and retry, ISP feedback loops, and +provider suppression behavior are **not** reachable locally. See the +provider-simulator table in `tests/harness/README.md`. + +## 9. Stop the harness + +```bash +docker compose -f tests/harness/docker-compose.yml down -v +``` + +Then confirm the default suite is still offline and green: + +```bash +pytest tests/ -q # integration cases skip +``` diff --git a/tests/harness/README.md b/tests/harness/README.md index 9e9a313..2a72956 100644 --- a/tests/harness/README.md +++ b/tests/harness/README.md @@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test for any non-loopback host, so this setting cannot weaken a real deployment: it fails at startup rather than sending credentials in the clear. +## Exercising bounces, complaints and deferrals + +The harness accepts every message, so it cannot *generate* a bounce, complaint, +or deferral. Those classes are exercised by injecting the crafted fixtures in +`tests/fixtures/mailbox/` and letting them travel through the harness: + +```python +harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes()) +``` + +This matters more than reading the fixture off disk. Delivery adds the real +`Received:` headers an MTA-handled message carries — the exact difference that +hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`. + +`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it +writes the message straight into a Maildir tree, byte for byte, with no MTA +involved and no harness required. + +`tests/test_evidence_realism.py` runs every recognized class both ways and +asserts the two agree. + +## What is only reachable in a provider staging tier + +Some behavior no local server can honestly reproduce, because it belongs to a +real provider and a real remote MX: + +| Not reachable locally | Why | +| --- | --- | +| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves | +| Real 4xx deferral and retry over time | The harness never defers or retries | +| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription | +| Provider suppression-list behavior | Belongs to the provider account | +| MX acceptance as distinct from provider acceptance | Needs a real remote MX | + +For those, use a provider staging tier with simulator addresses rather than +inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`, +`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and +`suppressionlist@simulator.amazonses.com` drive the corresponding paths without +touching a real recipient. + +That tier is deliberately **not** wired into this repo's test run: it needs real +credentials, which are routed through OpenBao and must never reach the harness. +Treat it as an operator-run check against a staging provider account. + ## What harness mail proves Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is diff --git a/tests/harness/__init__.py b/tests/harness/__init__.py index c666335..0de5ff1 100644 --- a/tests/harness/__init__.py +++ b/tests/harness/__init__.py @@ -104,6 +104,40 @@ def smtp_provider(sender: str = DEFAULT_SENDER): return SMTPProvider(HOST, SMTP_PORT, sender, PASSWORD, sender, security="plaintext") +def deliver_raw(recipient: str, raw_bytes: bytes, sender: str = DEFAULT_SENDER) -> None: + """Deliver a crafted message verbatim to a harness mailbox. + + The harness accepts everything, so it cannot *generate* a bounce, complaint, + or deferral. Injecting a crafted `.eml` is how those classes are exercised: + the message still travels through a real MTA and picks up real `Received:` + headers, which a fixture read off disk never does. + """ + + import smtplib + + with smtplib.SMTP(HOST, SMTP_PORT, timeout=10) as smtp: + smtp.sendmail(sender, [recipient], raw_bytes) + + +def inject_maildir(maildir_dir, raw_bytes: bytes, *, name: str | None = None, subdir: str = "new"): + """Write a crafted message straight into a Maildir tree. + + The offline counterpart to `deliver_raw`: no MTA involved, so the message + arrives byte-for-byte with no added headers. + """ + + import time + from pathlib import Path + + root = Path(maildir_dir) + for required in ("new", "cur", "tmp"): + (root / required).mkdir(parents=True, exist_ok=True) + filename = name or f"{int(time.time())}.M{len(raw_bytes)}P{id(raw_bytes) % 100000}.harness" + path = root / subdir / filename + path.write_bytes(raw_bytes) + return path + + def mailbox_config( user_address: str, *, diff --git a/tests/test_evidence_realism.py b/tests/test_evidence_realism.py new file mode 100644 index 0000000..e83c829 --- /dev/null +++ b/tests/test_evidence_realism.py @@ -0,0 +1,129 @@ +"""Every recognized evidence class, exercised through a real MTA. + +The harness accepts everything, so it cannot generate a bounce, complaint, or +deferral on its own. Those classes are exercised by injecting the crafted +fixtures and letting them travel through GreenMail, which adds the real +`Received:` headers that a fixture read off disk never has — the difference that +hid the reply-heuristic overclaim until EMAIL-WP-0005-T04. + +Classes that no local server can produce are listed in +`tests/harness/README.md` under the provider-simulator tier. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +import harness +from email_connect.config import AppConfig, MailboxConfig, ReportsConfig, ScanConfig, SourceConfig, StorageConfig +from email_connect.scanner import scan_mailbox + +FIXTURES = Path(__file__).parent / "fixtures" / "mailbox" + +requires_harness = pytest.mark.skipif( + not harness.available(), + reason="mail harness not running: docker compose -f tests/harness/docker-compose.yml up -d", +) + +#: The evidence each crafted fixture must still yield after passing through an +#: MTA. Anything absent here is unreachable locally, not merely untested. +EXPECTED_EVENTS = { + "hard_bounce.eml": "notification.endpoint.rejected_permanent", + "soft_bounce.eml": "notification.endpoint.rejected_temporary", + "delayed_delivery.eml": "notification.endpoint.deferred", + "complaint.eml": "notification.channel.complaint_received", + "unsubscribe.eml": "notification.channel.unsubscribe_received", + "out_of_office.eml": "interaction.out_of_office_received", + "challenge_response.eml": "interaction.unverified_actor_interaction", + "human_reply.eml": "interaction.reply_received", + "final_failure.eml": "notification.endpoint.rejected_permanent", + "unknown_return.eml": "notification.endpoint.unknown", +} + + +def scan_events(config) -> set[str]: + result = scan_mailbox(config, full_rescan=True) + rows = result.report_path.read_text(encoding="utf-8").splitlines() + header = rows[0].split(",") + index = header.index("normalized_event_type") + return {line.split(",")[index] for line in rows[1:] if line.strip()} + + +@pytest.mark.parametrize(("fixture", "expected"), sorted(EXPECTED_EVENTS.items())) +@requires_harness +def test_crafted_evidence_survives_a_real_mta(fixture, expected, tmp_path): + harness.reset() + recipient = harness.address(f"realism {Path(fixture).stem}") + harness.deliver_raw(recipient, (FIXTURES / fixture).read_bytes()) + + config = harness.mailbox_config( + recipient, + storage_path=str(tmp_path / "state.sqlite"), + reports_dir=str(tmp_path / "reports"), + ) + + assert expected in scan_events(config) + + +@requires_harness +def test_injected_fixtures_match_the_offline_fixture_scan(tmp_path): + """Delivery through an MTA must not change which evidence is produced.""" + + harness.reset() + recipient = harness.address("realism parity") + for fixture in EXPECTED_EVENTS: + harness.deliver_raw(recipient, (FIXTURES / fixture).read_bytes()) + + delivered = scan_events( + harness.mailbox_config( + recipient, + storage_path=str(tmp_path / "delivered.sqlite"), + reports_dir=str(tmp_path / "delivered-reports"), + ) + ) + + offline_dir = tmp_path / "offline-fixtures" + offline_dir.mkdir() + for fixture in EXPECTED_EVENTS: + (offline_dir / fixture).write_bytes((FIXTURES / fixture).read_bytes()) + offline = scan_events( + AppConfig( + mailbox=MailboxConfig(id="offline", protocol="fixture"), + scan=ScanConfig(), + storage=StorageConfig(path=str(tmp_path / "offline.sqlite")), + reports=ReportsConfig(output_dir=str(tmp_path / "offline-reports")), + source=SourceConfig(fixture_dir=str(offline_dir)), + ) + ) + + assert delivered == offline + + +def test_maildir_injection_is_byte_exact(tmp_path): + """The offline injection path adds nothing to the message.""" + + raw = (FIXTURES / "hard_bounce.eml").read_bytes() + path = harness.inject_maildir(tmp_path / "Maildir", raw) + + assert path.read_bytes() == raw + assert path.parent.name == "new" + + +def test_maildir_injection_feeds_the_scanner(tmp_path): + maildir = tmp_path / "Maildir" + harness.inject_maildir(maildir, (FIXTURES / "hard_bounce.eml").read_bytes(), name="1749000001.M1P1.harness") + harness.inject_maildir(maildir, (FIXTURES / "complaint.eml").read_bytes(), name="1749000002.M2P2.harness") + + config = AppConfig( + mailbox=MailboxConfig(id="injected", protocol="maildir"), + scan=ScanConfig(), + storage=StorageConfig(path=str(tmp_path / "state.sqlite")), + reports=ReportsConfig(output_dir=str(tmp_path / "reports")), + source=SourceConfig(maildir_dir=str(maildir)), + ) + events = scan_events(config) + + assert "notification.endpoint.rejected_permanent" in events + assert "notification.channel.complaint_received" in events diff --git a/workplans/EMAIL-WP-0005-test-mailbox-harness.md b/workplans/EMAIL-WP-0005-test-mailbox-harness.md index a5dd411..3fcd6eb 100644 --- a/workplans/EMAIL-WP-0005-test-mailbox-harness.md +++ b/workplans/EMAIL-WP-0005-test-mailbox-harness.md @@ -4,7 +4,7 @@ type: workplan title: "Test Mailbox Harness for Automated Test Environments" domain: infotech repo: email-connect -status: proposed +status: finished owner: claude topic_slug: custodian created: "2026-08-14" @@ -322,7 +322,7 @@ Two production defects surfaced, both invisible to the fixture-only suite: ```task id: EMAIL-WP-0005-T05 -status: todo +status: done priority: medium state_hub_task_id: "48b8eaf9-5a4b-437b-ad2c-952907340ed7" ``` @@ -345,11 +345,26 @@ path, and the limits of the local harness are stated explicitly rather than papered over. ``` +Done 2026-08-14: + +* `harness.deliver_raw()` injects a crafted `.eml` through the harness, so the + message picks up real `Received:` headers on the way in. + `harness.inject_maildir()` is the offline counterpart, byte-exact with no MTA. +* `tests/test_evidence_realism.py`, 13 cases: all ten recognized classes are + asserted after passing through a real MTA, plus a parity case proving MTA + delivery does not change which evidence is produced, plus two offline Maildir + injection cases. +* Documented what no local server can honestly produce — provider-generated + DSNs, real 4xx deferral and retry, ISP feedback loops, provider suppression + behavior, MX acceptance as distinct from provider acceptance — and named SES + simulator addresses as the staging path. That tier is deliberately not wired + into the test run: it needs real credentials, which stay in OpenBao. + ## T06 - Documentation and decision record ```task id: EMAIL-WP-0005-T06 -status: todo +status: done priority: medium state_hub_task_id: "0599de98-c0ff-4160-a2c8-54265b31ca5d" ``` @@ -371,6 +386,21 @@ A new contributor can start the harness, run the integration tier, and correctly state what harness-observed mail does and does not prove. ``` +Done 2026-08-14: + +* `docs/test-harness-tutorial.md`: three test tiers, start, mailbox, send, scan, + assert, reset, fixture injection, teardown. Includes an explicit + assertable/not-assertable list, so the evidence ceiling is stated where tests + get written rather than only in the canon. +* `docs/mailbox-report-tutorial.md` gained a Maildir section covering + `protocol: maildir`, `source.maildir_dir`, identity across `new/`→`cur/`, and + the read-only guarantees. +* `tests/harness/README.md` carries the reference material: ports, addressing + convention, reset contract, skip gating, injection, and the + provider-simulator table. +* Decisions were recorded as the work happened: s/qmail rejection and the + GreenMail plus loopback-plaintext choice are both in `DECISIONS.md`. + ## 8. Completion Criteria This workplan is complete when: