diff --git a/deploy/k8s/railiance/README.md b/deploy/k8s/railiance/README.md index cafe6b2..4c00018 100644 --- a/deploy/k8s/railiance/README.md +++ b/deploy/k8s/railiance/README.md @@ -1,136 +1,44 @@ # email-connect transactional receiver — railiance01 -Deploy the narrow invitation/verification HTTP receiver so **user-engine** can -send mail without holding SMTP credentials, and so unrelated workloads cannot -reach the send API. +The receiver permits user-engine invitation and registration-verification +calls, using fixed templates and bounded SMTP/STARTTLS. Its persistent SQLite +store contains delivery/challenge evidence; SMTP acceptance is not inbox receipt +or authorization. Current release review: ../../../docs/verification-retry-review-2026-10-07.md. -**Target cluster:** railiance01 only (`KUBECONFIG=~/.kube/config-hosteurope`). +Target: railiance01, cluster kube-system UID +a553c742-0115-43d4-99a4-a5ca56fe0786. Observe through +~/.kube/config-railiance01; change the cluster through ssh railiance01 and the +founder's exact live go. config-hosteurope is the retired cluster and must not +be used. Read the-custodian/docs/agent-environment-orientation.md first. -Stable in-cluster URL after apply: +Platform owns OpenBao custody, Kubernetes-auth ClusterSecretStore +openbao-email-connect and its scoped ESO ServiceAccount (RPF-WP-0045). +ExternalSecret email-connect/email-connect-runtime delivers the existing +EMAIL_CONNECT_INGEST_TOKEN, EMAIL_CONNECT_SMTP_USERNAME and +EMAIL_CONNECT_SMTP_PASSWORD fields. Never read or print Secret objects/annotations. +Route needs through warden route find/show email-connect-transactional. -```text -http://email-connect.email-connect.svc.cluster.local:8080 -``` +The static ESO token bootstrap and duplicate store template are retired. +Do not recreate external-secrets/openbao-email-connect-eso-token or apply a +store copy from this repo. Custody and store mutations require the platform lane. -## Ownership split +Stable URL: http://email-connect.email-connect.svc.cluster.local:8080. +Default-deny NetworkPolicy permits only user-engine:8080 ingress, DNS and public +SMTP:587 egress. No additional caller admission is part of an image correction. +ConfigMap owns host, port, sender and portal URL; no provider credentials belong +in HTTP requests, Git, chat, State Hub or test fixtures. -| Concern | Owner | Where | -| --- | --- | --- | -| Package, manifests, probes, NetworkPolicy, rollback | `email-connect` | this directory | -| OpenBao path, ESO role/policy, ClusterSecretStore | `railiance-platform` | CCR-2026-0010, `openbao/policies/` | -| Credential routing catalog | `ops-warden` | `email-connect-transactional` | +After reviewing current live manifests and CPU headroom, perform client diff +and server dry-run for the exact digest change. Preserve live resource ownership; +apply only that reviewed image change from this owner repo after founder go. +Observe Ready/probes and an isolated no-send authenticated rejection before +calling the release successful. Do not recreate secrets as a workaround. -Never paste SMTP passwords, ingest tokens, or OpenBao tokens into Git, State -Hub, workplans, logs, or chat. Route first: +Before the verification correction, capture SQLite with its backup API into a +private mode-0600 artifact. Keep PVC evidence intact. Do not run rollout undo +with sending enabled: an older image ignores verification_attempts and may +resend an uncertain operation. Stop the receiver first and reconcile unknown +reservations; roll forward with the invariant preserved where possible. -```bash -warden route find "email-connect SMTP transactional" --json -warden route show email-connect-transactional --json -``` - -## Runtime secrets (OpenBao → ESO) - -| Item | Value | -| --- | --- | -| KV path | `platform/workloads/email-connect/transactional` | -| Fields | `EMAIL_CONNECT_INGEST_TOKEN`, `EMAIL_CONNECT_SMTP_USERNAME`, `EMAIL_CONNECT_SMTP_PASSWORD` | -| Read / ESO policy | `external-secrets-email-connect` (+ `workload-kv-read-email-connect-transactional` for operator OIDC) | -| ClusterSecretStore | `openbao-email-connect` (namespace-limited; token auth like activity-core/reuse) | -| ESO token Secret | `external-secrets/openbao-email-connect-eso-token` (via `openbao-eso-token-apply.sh`) | -| ExternalSecret | `email-connect/email-connect-runtime` → Secret `email-connect-runtime` | -| ops-warden catalog | `email-connect-transactional` (draft until provisioned) | - -Non-secret config (host, port, sender, portal URL) lives in ConfigMap -`email-connect-config`. - -### Operator provision (attended, after CCR approval) - -```bash -# Values from mode-0600 files only; shred after. -bao kv put platform/workloads/email-connect/transactional \ - EMAIL_CONNECT_INGEST_TOKEN=@/path/to/ingest.token \ - EMAIL_CONNECT_SMTP_USERNAME=@/path/to/smtp.user \ - EMAIL_CONNECT_SMTP_PASSWORD=@/path/to/smtp.pass -``` - -user-engine must receive the **same** ingest token under its own runtime secret -path so it can call `Authorization: Bearer …` without ever seeing SMTP fields. - -## Network posture - -- **Default deny** ingress and egress in `email-connect`. -- **Ingress:** only pods labeled `app.kubernetes.io/name=user-engine` in - namespace `user-engine`, TCP 8080. -- **Egress:** kube-system DNS + TCP 587 to the public internet (IONOS - STARTTLS). No cluster-lateral egress and no general HTTPS. - -Bearer auth on `/v1/*` is the second gate; NetworkPolicy is the outer gate. - -## Apply - -Prerequisites: - -1. CCR-2026-0010 approved. -2. KV secret provisioned at `platform/workloads/email-connect/transactional` - (fields non-empty; do not log values). -3. ESO delivery token + ClusterSecretStore on railiance01 (token pattern matches - `openbao-activity-core` / `openbao-reuse`, not Kubernetes auth yet). -4. Image available on the node/registry at the digest pinned in `deployment.yaml`. - -```bash -export KUBECONFIG=~/.kube/config-hosteurope - -# 1) Policy + limited ESO child token → external-secrets Secret -./deploy/k8s/railiance/openbao-eso-token-apply.sh - -# 2) Namespace-scoped ClusterSecretStore (platform add-on copy) -kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml - -# 3) Optional: load a freshly built image when the registry is not yet updated -# docker build -f Containerfile -t forgejo.coulomb.social/coulomb/email-connect:local . -# docker save … | ssh railiance01 sudo k3s ctr -n k8s.io images import - - -# 4) Package -kubectl apply -k deploy/k8s/railiance -kubectl -n email-connect rollout status deployment/email-connect -kubectl -n email-connect get pods,svc,networkpolicy,externalsecret,secret -``` - -Expect ExternalSecret `SecretSynced` and a Ready pod. If the Secret is missing, -do not create a bootstrap Secret with values from chat — fix OpenBao/ESO. - -## Smoke (no SMTP proof yet) - -From a **user-engine** pod (or a temporary pod in that namespace with the same -labels — NetworkPolicy will otherwise drop the packet): - -```bash -# Health only (no auth) -kubectl -n user-engine exec deploy/user-engine -- \ - wget -qO- http://email-connect.email-connect.svc.cluster.local:8080/healthz - -# Auth check: wrong token must be 401; correct token is exercised by T04. -``` - -Negative checks: - -- A pod in another namespace cannot reach port 8080 (connection timeout/reset). -- user-engine runtime must not contain `EMAIL_CONNECT_SMTP_*` keys. - -## Rollback - -```bash -kubectl -n email-connect rollout undo deployment/email-connect -kubectl -n email-connect rollout status deployment/email-connect -``` - -`revisionHistoryLimit: 5` keeps prior digests. SQLite on the PVC is additive; -rolling back does not clear delivery/idempotency rows (by design). - -## Done criteria (EMAIL-WP-0004-T03) - -- [ ] user-engine calls the cluster-local receiver without possessing SMTP credentials -- [ ] Unrelated workloads cannot open TCP 8080 on the receiver -- [ ] SMTP password and ingest token exist only in OpenBao + the scoped K8s Secret -- [ ] Probes green; NetworkPolicy default-deny + SMTP-only egress applied -- [ ] Documented rollback path works against the previous revision +Local regression harness: tests/harness/README.md. It uses synthetic addresses +and loopback SMTP/IMAP only; no production mailbox or message send is a test. diff --git a/deploy/k8s/railiance/clustersecretstore.yaml.template b/deploy/k8s/railiance/clustersecretstore.yaml.template index ac5f364..53fbae0 100644 --- a/deploy/k8s/railiance/clustersecretstore.yaml.template +++ b/deploy/k8s/railiance/clustersecretstore.yaml.template @@ -1,30 +1,5 @@ -# Template copy of the railiance-platform add-on store. Prefer applying from: -# ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml -# -# Do not apply until CCR-2026-0010 is approved and Secret -# external-secrets/openbao-email-connect-eso-token exists -# (./openbao-eso-token-apply.sh). ---- -apiVersion: external-secrets.io/v1beta1 -kind: ClusterSecretStore -metadata: - name: openbao-email-connect - labels: - app.kubernetes.io/part-of: railiance-gitops - railiance-platform/component: external-secrets - app.kubernetes.io/name: email-connect -spec: - provider: - vault: - # In-cluster OpenBao on railiance01 (not the public bao.coulomb.social UI). - server: http://openbao.openbao.svc:8200 - path: platform - version: v2 - auth: - tokenSecretRef: - name: openbao-email-connect-eso-token - namespace: external-secrets - key: token - conditions: - - namespaces: - - email-connect +# RETIRED COPY — intentionally contains no Kubernetes resource. +# The platform owns the Kubernetes-auth store and its scoped ServiceAccount: +# railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml +# Consult RPF-WP-0045 and the authoritative platform declarations. +# Do not recreate external-secrets/openbao-email-connect-eso-token. diff --git a/deploy/k8s/railiance/openbao-eso-token-apply.sh b/deploy/k8s/railiance/openbao-eso-token-apply.sh index 09a2207..4450591 100755 --- a/deploy/k8s/railiance/openbao-eso-token-apply.sh +++ b/deploy/k8s/railiance/openbao-eso-token-apply.sh @@ -1,83 +1,5 @@ #!/usr/bin/env bash -# Mint a read-limited OpenBao token and store it on railiance01 for -# ClusterSecretStore openbao-email-connect / ExternalSecret email-connect-runtime. -# -# Policies (files in railiance-platform/openbao/policies/): -# external-secrets-email-connect -# workload-kv-read-email-connect-transactional (optional operator OIDC sibling) -# -# Does not print secret values. Requires an attended operator OpenBao token. -# CCR-2026-0010 must be approved before production use. +# Retired after RPF-WP-0045: never recreate the static ESO token Secret. set -euo pipefail - -DEFAULT_POLICIES="external-secrets-email-connect" -POLICIES="${OPENBAO_EMAIL_CONNECT_POLICIES:-$DEFAULT_POLICIES}" -POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}" -BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}" -RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}" -SECRET_NAME="${OPENBAO_EMAIL_CONNECT_ESO_SECRET:-openbao-email-connect-eso-token}" -SECRET_NS="${OPENBAO_EMAIL_CONNECT_ESO_NAMESPACE:-external-secrets}" -TTL="${OPENBAO_EMAIL_CONNECT_ESO_TTL:-768h}" - -if ! command -v bao >/dev/null 2>&1; then - echo "ERROR: bao CLI not found" >&2 - exit 1 -fi -if ! command -v kubectl >/dev/null 2>&1; then - echo "ERROR: kubectl not found" >&2 - exit 1 -fi - -echo "OpenBao addr: $BAO_ADDR" -echo "Policies: $POLICIES" -echo "K8s secret: $SECRET_NS/$SECRET_NAME (railiance01)" - -if [[ -n "${BAO_TOKEN:-}" ]]; then - : -elif [[ -n "${OPENBAO_TOKEN_FILE:-}" && -f "${OPENBAO_TOKEN_FILE}" ]]; then - BAO_TOKEN="$(head -n 1 "${OPENBAO_TOKEN_FILE}")" -else - read -r -s -p "OpenBao operator token: " BAO_TOKEN - echo >&2 -fi -if [[ -z "${BAO_TOKEN:-}" ]]; then - echo "ERROR: empty OpenBao token" >&2 - exit 1 -fi - -export BAO_ADDR BAO_TOKEN - -health="$(curl -fsS "$BAO_ADDR/v1/sys/health")" -if echo "$health" | grep -q '"sealed":true'; then - echo "ERROR: OpenBao at $BAO_ADDR reports sealed" >&2 - exit 1 -fi - -for policy in $POLICIES; do - policy_file="$POLICY_DIR/${policy}.hcl" - if [[ -f "$policy_file" ]]; then - bao policy write "$policy" "$policy_file" - echo "policy written: $policy" - else - echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2 - fi -done - -# Child token: renewable, orphan so operator logout does not revoke delivery. -# shellcheck disable=SC2086 -token_json="$(bao token create -policy="$(echo $POLICIES | tr ' ' ',')" -ttl="$TTL" -renewable=true -orphan -format=json)" -child_token="$(printf '%s' "$token_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["auth"]["client_token"])')" -if [[ -z "$child_token" || ${#child_token} -lt 8 ]]; then - echo "ERROR: failed to mint child token" >&2 - exit 1 -fi -echo "minted child token length=${#child_token} (value not printed)" - -export KUBECONFIG="$RAILIANCE01_KUBECONFIG" -kubectl -n "$SECRET_NS" create secret generic "$SECRET_NAME" \ - --from-literal=token="$child_token" \ - --dry-run=client -o yaml | kubectl apply -f - - -unset child_token BAO_TOKEN -echo "Secret $SECRET_NS/$SECRET_NAME applied on railiance01." -echo "Next: apply ClusterSecretStore openbao-email-connect, then email-connect package." +printf '%s\n' 'Retired: openbao-email-connect uses platform-owned Kubernetes auth. See deploy/k8s/railiance/README.md.' >&2 +exit 1 diff --git a/docs/verification-retry-review-2026-10-07.md b/docs/verification-retry-review-2026-10-07.md new file mode 100644 index 0000000..9c6be7d --- /dev/null +++ b/docs/verification-retry-review-2026-10-07.md @@ -0,0 +1,35 @@ +# Verification retry correction — 2026-10-07 + +Source contract: POST /v1/registration-verifications keeps its 202 JSON +{request_id, accepted: true} response. Repeated tenant/client/registration +operations return the same receipt after provider acceptance without resending. +The correlation ID may change; changing recipient, username or display name +for the same operation is rejected. A new registration is a separate operation. + +A durable reservation precedes SMTP and keeps only the challenge digest. An +unknown send outcome returns redacted 422 delivery_outcome_unknown, +retryable=false, and keeps its usable challenge until expiry. Definitive +retryable non-acceptance returns 503 and atomically cancels the failed challenge +before retry. Permanent rejection stays terminal. Existing pre-upgrade +challenges have no recorded SMTP outcome, so their retries fail conservatively +while existing consume/cancel links keep working. There is no automatic resend +or expired-challenge renewal for the same registration. + +39 transactional tests passed. The full repository suite with the pinned +GreenMail harness running passed all 101 tests with zero skips. The revised +real SMTP/IMAP verification journey observed exactly one mailbox message after +store restart and HTTP retry. Harness container/network were removed afterward. +Synthetic fixture accounts only. Packaged image receipt follows separately. + +Release scope is the email-connect container image only, reusing the live PVC, +caller token, ConfigMap, ServiceAccount and NetworkPolicies. It does not authorize +another caller, sender, provider or credential delivery. Founder live go remains +separate from code preparation under the-custodian/docs/agent-environment-orientation.md +section 4. The pre-release SQLite backup must use the SQLite backup API while +consistent, with output kept private; never emit its rows. + +Rollback: block sending first (scale receiver to zero, preserving its PVC), +then compare/reconcile reservation outcomes. Old code ignores verification +reservations: an unrestricted rollout undo can duplicate verification mail. +Prefer rolling forward with the retry invariant preserved. Additive schema can +be read by the old image, but schema compatibility is not delivery safety. diff --git a/src/email_connect/transactional.py b/src/email_connect/transactional.py index 68c659f..e3103c6 100644 --- a/src/email_connect/transactional.py +++ b/src/email_connect/transactional.py @@ -57,8 +57,77 @@ class SQLiteDeliveryStore: self.db.execute( "CREATE TABLE IF NOT EXISTS send_attempts (event_id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, state TEXT NOT NULL)" ) + self.db.execute("""CREATE TABLE IF NOT EXISTS verification_attempts ( + operation_key TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, + request_id TEXT NOT NULL, state TEXT NOT NULL, + error TEXT, retryable INTEGER, provider_ref TEXT + )""") self.db.commit() + def reserve_verification(self, payload, handle_hash, expires_at): + # Registration identity, not a tracing ID, defines the logical operation. + identity = [payload[key] for key in ("tenant", "client_id", "registration_id")] + operation_key = hashlib.sha256(json.dumps(identity).encode()).hexdigest() + binding = {key: payload.get(key) for key in ( + "registration_id", "normalized_email", "preferred_username", + "client_id", "tenant", "display_name", + )} + fingerprint = hashlib.sha256( + json.dumps(binding, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + with self.lock, self.db: + self.db.execute("BEGIN IMMEDIATE") + row = self.db.execute( + "SELECT fingerprint,request_id,state,error,retryable " + "FROM verification_attempts WHERE operation_key=?", (operation_key,), + ).fetchone() + if not row: + legacy = self.db.execute( + "SELECT 1 FROM verifications WHERE registration_id=? AND client_id=? AND tenant=? LIMIT 1", + (payload["registration_id"], payload["client_id"], payload["tenant"]), + ).fetchone() + if legacy: + # Pre-upgrade challenges have no trustworthy send outcome. + # Preserve their links, but do not silently issue another. + raise ProviderError("delivery_outcome_unknown", retryable=False) + if row: + if row[0] != fingerprint: + raise ValueError("idempotency_payload_mismatch") + if row[2] == "accepted": + return row[1], False + if row[2] == "in_flight": + raise ProviderError("delivery_outcome_unknown", retryable=False) + if not row[4]: + raise ProviderError(row[3], retryable=False) + request_id = self._insert_verification(payload, handle_hash, expires_at) + self.db.execute( + "INSERT OR REPLACE INTO verification_attempts VALUES (?,?,?,?,NULL,NULL,NULL)", + (operation_key, fingerprint, request_id, "in_flight"), + ) + return request_id, True + + def verification_accepted(self, request_id, reference): + with self.lock, self.db: + self.db.execute( + "UPDATE verification_attempts SET state='accepted',provider_ref=? " + "WHERE request_id=? AND state='in_flight'", (reference, request_id), + ) + + def verification_failed(self, request_id, error): + # Called only when non-acceptance is known. Unknown outcomes retain the + # reservation and the delivered handle remains usable until expiry. + with self.lock, self.db: + self.db.execute("BEGIN IMMEDIATE") + self.db.execute( + "UPDATE verifications SET canceled_at=? WHERE request_id=?", + (datetime.now(timezone.utc).isoformat(), request_id), + ) + self.db.execute( + "UPDATE verification_attempts SET state='failed',error=?,retryable=? " + "WHERE request_id=? AND state='in_flight'", + (error.code, int(error.retryable), request_id), + ) + def reserve_send(self, event_id, fingerprint): with self.lock: self.db.execute("BEGIN IMMEDIATE") @@ -132,29 +201,33 @@ class SQLiteDeliveryStore: def create_verification( self, payload: dict, handle_hash: str, expires_at: str ) -> str: + with self.lock, self.db: + return self._insert_verification(payload, handle_hash, expires_at) + + def _insert_verification(self, payload, handle_hash, expires_at): request_id = f"vrq_{secrets.token_hex(12)}" - with self.db: - self.db.execute( - """INSERT INTO verifications - (request_id, handle_hash, registration_id, email, username, - client_id, tenant, display_name, expires_at, consumed_at, canceled_at) - VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""", - ( - request_id, - handle_hash, - payload["registration_id"], - payload["normalized_email"], - payload["preferred_username"], - payload["client_id"], - payload["tenant"], - payload.get("display_name"), - expires_at, - ), - ) + self.db.execute( + """INSERT INTO verifications + (request_id, handle_hash, registration_id, email, username, + client_id, tenant, display_name, expires_at, consumed_at, canceled_at) + VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""", + ( + request_id, + handle_hash, + payload["registration_id"], + payload["normalized_email"], + payload["preferred_username"], + payload["client_id"], + payload["tenant"], + payload.get("display_name"), + expires_at, + ), + ) return request_id def consume_verification(self, handle_hash: str) -> dict: with self.lock, self.db: + self.db.execute("BEGIN IMMEDIATE") row = self.db.execute( "SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,) ).fetchone() @@ -178,6 +251,7 @@ class SQLiteDeliveryStore: def cancel_verification(self, handle_hash: str) -> dict: with self.lock, self.db: + self.db.execute("BEGIN IMMEDIATE") row = self.db.execute( "SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,) ).fetchone() @@ -456,7 +530,7 @@ class TransactionalApplication: "tenant", "correlation_id", ) - if any(not payload.get(key) for key in required) or not _valid_address( + if any(not isinstance(payload.get(key), str) or not payload[key] for key in required) or not _valid_address( str(payload["normalized_email"]) ): raise ValueError("invalid_verification_request") @@ -465,16 +539,33 @@ class TransactionalApplication: raise ValueError("recipient_suppressed") handle = secrets.token_urlsafe(32) expires_at = datetime.now(timezone.utc) + timedelta(minutes=30) - request_id = self.store.create_verification( + request_id, send = self.store.reserve_verification( payload, hashlib.sha256(handle.encode()).hexdigest(), expires_at.isoformat() ) - self.provider.send( - email, - "Verify your NetKingdom registration", - f"Continue securely at {self.portal_url}/registration/verify?handle={handle}\n" - f"Cancel this request at {self.portal_url}/registration/cancel?handle={handle}\n" - "These links expire in 30 minutes.\n", - ) + if send: + try: + reference = self.provider.send( + email, + "Verify your NetKingdom registration", + f"Continue securely at {self.portal_url}/registration/verify?handle={handle}\n" + f"Cancel this request at {self.portal_url}/registration/cancel?handle={handle}\n" + "These links expire in 30 minutes.\n", + ) + self.store.verification_accepted(request_id, reference) + except ProviderError as exc: + definitive = getattr( + exc, "definitive", + exc.code in {"temporary_deferral", "permanent_rejection"}, + ) + if not definitive: + raise ProviderError("delivery_outcome_unknown", retryable=False) from None + try: + self.store.verification_failed(request_id, exc) + except Exception: + raise ProviderError("delivery_outcome_unknown", retryable=False) from None + raise + except Exception: + raise ProviderError("delivery_outcome_unknown", retryable=False) from None return self._json( start_response, HTTPStatus.ACCEPTED, diff --git a/tests/test_integration_send_scan.py b/tests/test_integration_send_scan.py index e68cbf9..a40db1b 100644 --- a/tests/test_integration_send_scan.py +++ b/tests/test_integration_send_scan.py @@ -206,18 +206,18 @@ def test_verification_mail_is_delivered_and_carries_no_authorization(tmp_path): recipient = harness.address("verification delivered") app = application(tmp_path) - status, body = invoke( - app, - { - "registration_id": "reg-1", - "normalized_email": recipient, - "preferred_username": "harness-user", - "client_id": "portal", - "tenant": "harness", - "correlation_id": "corr-1", - }, - path="/v1/registration-verifications", - ) + payload = { + "registration_id": "reg-1", "normalized_email": recipient, + "preferred_username": "harness-user", "client_id": "portal", + "tenant": "harness", "correlation_id": "corr-1", + } + path = "/v1/registration-verifications" + status, body = invoke(app, payload, path=path) + app.store.db.close() + # A real SMTP/IMAP journey includes service restart and a changed trace ID. + app = application(tmp_path) + replay_status, replay_body = invoke(app, {**payload, "correlation_id": "corr-retry"}, path=path) + assert replay_status.startswith("202") and replay_body == body assert status.startswith("2"), (status, body) messages = delivered_messages(recipient) diff --git a/tests/test_transactional.py b/tests/test_transactional.py index f252966..64822f1 100644 --- a/tests/test_transactional.py +++ b/tests/test_transactional.py @@ -495,3 +495,158 @@ def test_disconnect_after_smtp_acceptance_never_releases_reservation( assert body["error"] == "delivery_outcome_unknown" assert "private" not in str(body) assert SMTP.sends == 1 + + +VERIFICATION_PATH = "/v1/registration-verifications" + + +def verification_request(**changes): + return { + "registration_id": "reg-retry", "normalized_email": "person@example.test", + "preferred_username": "person", "client_id": "coulomb-social", + "tenant": "tenant:coulomb", "correlation_id": "corr-retry", **changes, + } + + +def verification_app(path, provider): + store = SQLiteDeliveryStore(str(path)) + return TransactionalApplication(store, provider, "opaque", "https://users.example") + + +def test_verification_retry_after_restart_returns_same_receipt(tmp_path): + path = tmp_path / "mail.db" + provider = Provider() + app = verification_app(path, provider) + status, receipt = invoke(app, verification_request(), path=VERIFICATION_PATH) + assert status.startswith("202") + handle = provider.calls[0][2].split("handle=", 1)[1].splitlines()[0] + app.store.db.close() + app = verification_app(path, provider) + # Correlation IDs can change between HTTP retries; they are tracing metadata. + assert invoke(app, verification_request(correlation_id="corr-new"), path=VERIFICATION_PATH)[1] == receipt + assert len(provider.calls) == 1 + assert app.store.db.execute("SELECT count(*) FROM verifications").fetchone()[0] == 1 + assert handle not in json.dumps(receipt) + assert handle.encode() not in path.read_bytes() + assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200") + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == receipt + assert len(provider.calls) == 1 # consumption does not turn a retry into a resend + + +@pytest.mark.parametrize("change", [ + {"normalized_email": "other@example.test"}, {"preferred_username": "other"}, + {"display_name": "changed"}, +]) +def test_verification_binding_cannot_change_on_retry(tmp_path, change): + provider = Provider() + app = verification_app(tmp_path / "mail.db", provider) + invoke(app, verification_request(), path=VERIFICATION_PATH) + status, body = invoke(app, verification_request(**change), path=VERIFICATION_PATH) + assert status.startswith("400") and body["error"] == "idempotency_payload_mismatch" + assert len(provider.calls) == 1 + + +@pytest.mark.parametrize("error", [ + OSError("private smtp credential"), ProviderError("provider_timeout", retryable=True), +]) +def test_verification_uncertain_send_is_not_repeated_after_restart(tmp_path, error): + provider = FailingProvider(error) + path = tmp_path / "mail.db" + app = verification_app(path, provider) + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == { + "error": "delivery_outcome_unknown", "retryable": False, + } + app.store.db.close() + app = verification_app(path, provider) + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown" + assert provider.calls == 1 + assert app.store.db.execute("SELECT count(*) FROM verifications").fetchone()[0] == 1 + + +def test_verification_acceptance_record_failure_does_not_resend(tmp_path, monkeypatch): + provider = Provider() + app = verification_app(tmp_path / "mail.db", provider) + def fail(*args): + raise OSError("private database path") + monkeypatch.setattr(app.store, "verification_accepted", fail) + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown" + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown" + assert len(provider.calls) == 1 + handle = provider.calls[0][2].split("handle=", 1)[1].splitlines()[0] + # A message accepted before evidence persistence failed still has a usable link. + assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200") + + +def test_verification_definitive_deferral_allows_retry_and_cancels_old_handle(tmp_path): + failing = FailingProvider(ProviderError("temporary_deferral", retryable=True)) + app = verification_app(tmp_path / "mail.db", failing) + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == { + "error": "temporary_deferral", "retryable": True, + } + old = app.store.db.execute("SELECT handle_hash,canceled_at FROM verifications").fetchone() + assert old[1] + with pytest.raises(ValueError, match="verification_invalid"): + app.store.consume_verification(old[0]) + provider = Provider() + app.provider = provider + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[0].startswith("202") + assert len(provider.calls) == 1 + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[0].startswith("202") + assert len(provider.calls) == 1 + + +def test_verification_permanent_failure_stays_terminal(tmp_path): + provider = FailingProvider(ProviderError("permanent_rejection", retryable=False)) + app = verification_app(tmp_path / "mail.db", provider) + for _ in range(2): + status, body = invoke(app, verification_request(), path=VERIFICATION_PATH) + assert status.startswith("422") and body["error"] == "permanent_rejection" + assert provider.calls == 1 + + +def test_verification_concurrent_connections_reserve_before_provider_call(tmp_path): + from concurrent.futures import ThreadPoolExecutor + from threading import Event + entered, release = Event(), Event() + class BlockingProvider(Provider): + def send(self, *args): + self.calls.append(args) + entered.set() + assert release.wait(5) + return "message:1" + provider = BlockingProvider() + first = verification_app(tmp_path / "mail.db", provider) + second = verification_app(tmp_path / "mail.db", provider) + with ThreadPoolExecutor(max_workers=1) as pool: + pending = pool.submit(invoke, first, verification_request(), path=VERIFICATION_PATH) + try: + assert entered.wait(5) + assert invoke(second, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown" + finally: + release.set() + receipt = pending.result()[1] + assert invoke(second, verification_request(), path=VERIFICATION_PATH)[1] == receipt + assert len(provider.calls) == 1 + + +def test_verification_tenant_and_client_operations_are_separate(tmp_path): + provider = Provider() + app = verification_app(tmp_path / "mail.db", provider) + for changes in ({}, {"tenant": "tenant:other"}, {"client_id": "other-client"}): + assert invoke(app, verification_request(**changes), path=VERIFICATION_PATH)[0].startswith("202") + assert len(provider.calls) == 3 + + +def test_legacy_verification_is_preserved_without_resending(tmp_path): + from datetime import datetime, timedelta, timezone + import hashlib + provider = Provider() + app = verification_app(tmp_path / "mail.db", provider) + handle = "legacy-handle-" + "x" * 32 + app.store.create_verification( + verification_request(), hashlib.sha256(handle.encode()).hexdigest(), + (datetime.now(timezone.utc) + timedelta(minutes=30)).isoformat(), + ) + assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown" + assert provider.calls == [] + assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200") diff --git a/workplans/EMAIL-WP-0006-verification-retry-safety.md b/workplans/EMAIL-WP-0006-verification-retry-safety.md new file mode 100644 index 0000000..87e651a --- /dev/null +++ b/workplans/EMAIL-WP-0006-verification-retry-safety.md @@ -0,0 +1,80 @@ +--- +id: EMAIL-WP-0006 +type: workplan +title: "Make registration verification delivery retry safe" +domain: infotech +repo: email-connect +status: active +owner: codex +topic_slug: netkingdom +created: "2026-10-07" +updated: "2026-10-07" +--- + +Repeated verification requests currently mint and send another challenge, even +when SMTP may already have accepted the previous message. CSOC-WP-0010-T04 +identified this on source 73702b7 using synthetic requests. This workplan owns +that correction; it does not admit new callers or change SMTP sender custody. + +## T01 — Reserve verification operations durably + +```task +id: EMAIL-WP-0006-T01 +status: done +priority: high +``` + +Implemented additive SQLite verification_attempts. Tenant/client/registration +identity defines the operation; immutable applicant fields define its fingerprint. +Correlation ID is tracing metadata and can change on retry. Reserve challenge +and operation atomically before SMTP. Accepted replay returns the same request +receipt. Unknown outcomes are terminal for automatic retry; only definitive +retryable non-acceptance permits a new challenge, invalidating the old one. +Legacy challenges remain usable and conservatively block another send. +Consume/cancel use SQLite write transactions across independent connections. +Plaintext handles remain only in mail, never in SQLite or HTTP receipts. + +## T02 — Prove failure, restart and SMTP mailbox behavior + +```task +id: EMAIL-WP-0006-T02 +status: done +priority: high +``` + +39 transactional cases pass, including separate-connection concurrency, +restart, post-acceptance evidence failure, binding mutation, legacy challenge, +definitive deferral/permanent refusal, consumption and cancellation. +The existing real loopback SMTP/IMAP verification journey now retries after a +store restart and requires exactly one mailbox message. Full repository suite with the pinned GreenMail harness running: 101 passed, +zero skips. The synthetic SMTP/IMAP journey confirmed one message after restart +and retry. Harness container/network were removed after validation. + +## T03 — Release the immutable candidate under a separate live gate + +```task +id: EMAIL-WP-0006-T03 +status: progress +priority: high +``` + +Prepare immutable image, live digest/headroom observation and exact image-only +diff; capture SQLite consistently before release; use deployment owner path and +preserve delivery/challenge records. Founder activation is required for this +new live change. Do not rollback to old verification code with sending enabled: +it ignores the new reservation table. Stop sending first and reconcile unknown +operations. Platform sender/caller admission remains CSOC-WP-0010-T04/T05. + +## T04 — Retire the obsolete static ESO bootstrap + +```task +id: EMAIL-WP-0006-T04 +status: done +priority: medium +``` + +Owner inbox records RPF-WP-0045 Kubernetes-auth recovery and retirement request +9d63cf96-333c-4142-a517-aa2633ae52eb. Bootstrap now fails closed without touching +credentials or Kubernetes; the obsolete template contains only its retirement +pointer. Documented package apply does not recreate the platform-owned store. +No live custody change was made.