EMAIL-WP-0005 T05/T06: evidence realism and harness documentation

Adds harness.deliver_raw() to inject crafted .eml fixtures through the harness,
so they pick up the real Received: headers an MTA adds, and
harness.inject_maildir() as the byte-exact offline counterpart.

tests/test_evidence_realism.py asserts all ten recognized evidence classes
after passing through a real MTA, proves MTA delivery does not change which
evidence is produced, and covers Maildir injection offline.

Documents what no local server can honestly produce -- provider-generated DSNs,
real 4xx deferral and retry, ISP feedback loops, provider suppression behavior,
MX acceptance as distinct from provider acceptance -- and names SES simulator
addresses as the staging path. That tier stays out of the test run because it
needs real credentials.

Adds docs/test-harness-tutorial.md covering the three test tiers and the
start/send/scan/assert/reset walkthrough, with an explicit
assertable/not-assertable list so the evidence ceiling is stated where tests
get written. Adds a Maildir section to the mailbox report tutorial.

Completes EMAIL-WP-0005. Suite: 85 passed with the harness up, 64 passed +
21 skipped with it down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/harness/README.md

@@ -122,6 +122,50 @@ export EMAIL_CONNECT_SENDER=noreply@harness.email-connect.test
 for any non-loopback host, so this setting cannot weaken a real deployment: it
 fails at startup rather than sending credentials in the clear.
 
+## Exercising bounces, complaints and deferrals
+
+The harness accepts every message, so it cannot *generate* a bounce, complaint,
+or deferral. Those classes are exercised by injecting the crafted fixtures in
+`tests/fixtures/mailbox/` and letting them travel through the harness:
+
+```python
+harness.deliver_raw(recipient, (FIXTURES / "hard_bounce.eml").read_bytes())
+```
+
+This matters more than reading the fixture off disk. Delivery adds the real
+`Received:` headers an MTA-handled message carries — the exact difference that
+hid an overclaiming reply heuristic until `EMAIL-WP-0005-T04`.
+
+`harness.inject_maildir(maildir_dir, raw_bytes)` is the offline counterpart: it
+writes the message straight into a Maildir tree, byte for byte, with no MTA
+involved and no harness required.
+
+`tests/test_evidence_realism.py` runs every recognized class both ways and
+asserts the two agree.
+
+## What is only reachable in a provider staging tier
+
+Some behavior no local server can honestly reproduce, because it belongs to a
+real provider and a real remote MX:
+
+| Not reachable locally | Why |
+| --- | --- |
+| Provider-generated DSNs | Local injection uses a DSN we wrote ourselves |
+| Real 4xx deferral and retry over time | The harness never defers or retries |
+| ISP complaint feedback loops (ARF) | Needs a real feedback-loop subscription |
+| Provider suppression-list behavior | Belongs to the provider account |
+| MX acceptance as distinct from provider acceptance | Needs a real remote MX |
+
+For those, use a provider staging tier with simulator addresses rather than
+inventing local equivalents. On SES, for example, `bounce@simulator.amazonses.com`,
+`complaint@simulator.amazonses.com`, `ooto@simulator.amazonses.com` and
+`suppressionlist@simulator.amazonses.com` drive the corresponding paths without
+touching a real recipient.
+
+That tier is deliberately **not** wired into this repo's test run: it needs real
+credentials, which are routed through OpenBao and must never reach the harness.
+Treat it as an operator-run check against a staging provider account.
+
 ## What harness mail proves
 
 Nothing beyond `provider_accepted`. A message sitting in a GreenMail mailbox is

tests/harness/__init__.py

@@ -104,6 +104,40 @@ def smtp_provider(sender: str = DEFAULT_SENDER):
     return SMTPProvider(HOST, SMTP_PORT, sender, PASSWORD, sender, security="plaintext")
 
 
+def deliver_raw(recipient: str, raw_bytes: bytes, sender: str = DEFAULT_SENDER) -> None:
+    """Deliver a crafted message verbatim to a harness mailbox.
+
+    The harness accepts everything, so it cannot *generate* a bounce, complaint,
+    or deferral. Injecting a crafted `.eml` is how those classes are exercised:
+    the message still travels through a real MTA and picks up real `Received:`
+    headers, which a fixture read off disk never does.
+    """
+
+    import smtplib
+
+    with smtplib.SMTP(HOST, SMTP_PORT, timeout=10) as smtp:
+        smtp.sendmail(sender, [recipient], raw_bytes)
+
+
+def inject_maildir(maildir_dir, raw_bytes: bytes, *, name: str | None = None, subdir: str = "new"):
+    """Write a crafted message straight into a Maildir tree.
+
+    The offline counterpart to `deliver_raw`: no MTA involved, so the message
+    arrives byte-for-byte with no added headers.
+    """
+
+    import time
+    from pathlib import Path
+
+    root = Path(maildir_dir)
+    for required in ("new", "cur", "tmp"):
+        (root / required).mkdir(parents=True, exist_ok=True)
+    filename = name or f"{int(time.time())}.M{len(raw_bytes)}P{id(raw_bytes) % 100000}.harness"
+    path = root / subdir / filename
+    path.write_bytes(raw_bytes)
+    return path
+
+
 def mailbox_config(
     user_address: str,
     *,

tests/test_evidence_realism.py

@@ -0,0 +1,129 @@
+"""Every recognized evidence class, exercised through a real MTA.
+
+The harness accepts everything, so it cannot generate a bounce, complaint, or
+deferral on its own. Those classes are exercised by injecting the crafted
+fixtures and letting them travel through GreenMail, which adds the real
+`Received:` headers that a fixture read off disk never has — the difference that
+hid the reply-heuristic overclaim until EMAIL-WP-0005-T04.
+
+Classes that no local server can produce are listed in
+`tests/harness/README.md` under the provider-simulator tier.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+import harness
+from email_connect.config import AppConfig, MailboxConfig, ReportsConfig, ScanConfig, SourceConfig, StorageConfig
+from email_connect.scanner import scan_mailbox
+
+FIXTURES = Path(__file__).parent / "fixtures" / "mailbox"
+
+requires_harness = pytest.mark.skipif(
+    not harness.available(),
+    reason="mail harness not running: docker compose -f tests/harness/docker-compose.yml up -d",
+)
+
+#: The evidence each crafted fixture must still yield after passing through an
+#: MTA. Anything absent here is unreachable locally, not merely untested.
+EXPECTED_EVENTS = {
+    "hard_bounce.eml": "notification.endpoint.rejected_permanent",
+    "soft_bounce.eml": "notification.endpoint.rejected_temporary",
+    "delayed_delivery.eml": "notification.endpoint.deferred",
+    "complaint.eml": "notification.channel.complaint_received",
+    "unsubscribe.eml": "notification.channel.unsubscribe_received",
+    "out_of_office.eml": "interaction.out_of_office_received",
+    "challenge_response.eml": "interaction.unverified_actor_interaction",
+    "human_reply.eml": "interaction.reply_received",
+    "final_failure.eml": "notification.endpoint.rejected_permanent",
+    "unknown_return.eml": "notification.endpoint.unknown",
+}
+
+
+def scan_events(config) -> set[str]:
+    result = scan_mailbox(config, full_rescan=True)
+    rows = result.report_path.read_text(encoding="utf-8").splitlines()
+    header = rows[0].split(",")
+    index = header.index("normalized_event_type")
+    return {line.split(",")[index] for line in rows[1:] if line.strip()}
+
+
+@pytest.mark.parametrize(("fixture", "expected"), sorted(EXPECTED_EVENTS.items()))
+@requires_harness
+def test_crafted_evidence_survives_a_real_mta(fixture, expected, tmp_path):
+    harness.reset()
+    recipient = harness.address(f"realism {Path(fixture).stem}")
+    harness.deliver_raw(recipient, (FIXTURES / fixture).read_bytes())
+
+    config = harness.mailbox_config(
+        recipient,
+        storage_path=str(tmp_path / "state.sqlite"),
+        reports_dir=str(tmp_path / "reports"),
+    )
+
+    assert expected in scan_events(config)
+
+
+@requires_harness
+def test_injected_fixtures_match_the_offline_fixture_scan(tmp_path):
+    """Delivery through an MTA must not change which evidence is produced."""
+
+    harness.reset()
+    recipient = harness.address("realism parity")
+    for fixture in EXPECTED_EVENTS:
+        harness.deliver_raw(recipient, (FIXTURES / fixture).read_bytes())
+
+    delivered = scan_events(
+        harness.mailbox_config(
+            recipient,
+            storage_path=str(tmp_path / "delivered.sqlite"),
+            reports_dir=str(tmp_path / "delivered-reports"),
+        )
+    )
+
+    offline_dir = tmp_path / "offline-fixtures"
+    offline_dir.mkdir()
+    for fixture in EXPECTED_EVENTS:
+        (offline_dir / fixture).write_bytes((FIXTURES / fixture).read_bytes())
+    offline = scan_events(
+        AppConfig(
+            mailbox=MailboxConfig(id="offline", protocol="fixture"),
+            scan=ScanConfig(),
+            storage=StorageConfig(path=str(tmp_path / "offline.sqlite")),
+            reports=ReportsConfig(output_dir=str(tmp_path / "offline-reports")),
+            source=SourceConfig(fixture_dir=str(offline_dir)),
+        )
+    )
+
+    assert delivered == offline
+
+
+def test_maildir_injection_is_byte_exact(tmp_path):
+    """The offline injection path adds nothing to the message."""
+
+    raw = (FIXTURES / "hard_bounce.eml").read_bytes()
+    path = harness.inject_maildir(tmp_path / "Maildir", raw)
+
+    assert path.read_bytes() == raw
+    assert path.parent.name == "new"
+
+
+def test_maildir_injection_feeds_the_scanner(tmp_path):
+    maildir = tmp_path / "Maildir"
+    harness.inject_maildir(maildir, (FIXTURES / "hard_bounce.eml").read_bytes(), name="1749000001.M1P1.harness")
+    harness.inject_maildir(maildir, (FIXTURES / "complaint.eml").read_bytes(), name="1749000002.M2P2.harness")
+
+    config = AppConfig(
+        mailbox=MailboxConfig(id="injected", protocol="maildir"),
+        scan=ScanConfig(),
+        storage=StorageConfig(path=str(tmp_path / "state.sqlite")),
+        reports=ReportsConfig(output_dir=str(tmp_path / "reports")),
+        source=SourceConfig(maildir_dir=str(maildir)),
+    )
+    events = scan_events(config)
+
+    assert "notification.endpoint.rejected_permanent" in events
+    assert "notification.channel.complaint_received" in events