diff --git a/DECISIONS.md b/DECISIONS.md index d127348..4c704bc 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -2,6 +2,78 @@ _Auto-generated by the Custodian State Hub._ +## Defer daily send-quota privilege tiers (anti-spam / abuse) + +**Date:** 2026-08-12 +**Decided by:** human (via agent note) +**Status:** deferred — do not implement in EMAIL-WP-0004 + +### Idea + +Outbound email is high-abuse surface if credentials or a caller is compromised. +The platform should treat **daily send volume** as a privilege, not an +unlimited side effect of holding a send token. + +Proposed daily ceilings (per sender / tenant / grant — exact subject TBD): + +| Tier | Mails / day | +| --- | ---: | +| minimal | 10 | +| low | 100 | +| standard | 1_000 | +| elevated | 10_000 | +| high | 100_000 | +| bulk-capable | 1_000_000 | +| unlimited | unlimited (explicit break-glass only) | + +Default for ordinary transactional use (e.g. invitation/verification) should +land at the low end of the scale until a use case justifies more. + +### Why not now (EMAIL-WP-0004) + +Current scope is a **single approved caller** (user-engine), **fixed templates**, +NetworkPolicy ingress lock, and one shared ingest bearer. Abuse surface is real +but narrow. Building the full tier model inside T03 custody/deploy or T04 +failure proof would delay the invitation path without a multi-tenant grant +story. + +Existing interim controls that stay: + +- template allow-list (no arbitrary body/sender) +- NetworkPolicy: user-engine only +- bearer ingest token (OpenBao custody, CCR-2026-0010) +- idempotency on event id + +### When to revisit + +Promote to a dedicated workplan when any of: + +- a second caller / tenant may send through email-connect +- the ingest token is treated as a long-lived platform grant +- marketing-adjacent or high-volume notification use appears +- flex-auth or credential-broker can bind a **send quota claim** to the caller + +### Design sketch for later (not committed) + +- Enforce in **email-connect** at accept time (before provider call), with a + durable counter (not only in-process). +- Classify over-quota as **permanent denial** for the day (or retryable after + window), redacted diagnostics, no SMTP detail leak. +- Bind tier via platform auth/grant (flex-auth / OpenBao policy metadata / + caller registration), not a free-form request field the client can raise. +- Unlimited is operator-issued break-glass, audited, not a default. +- Coordinate with suppression/complaint handling so quota and reputation share + one abuse narrative. + +### Home for implementation later + +New workplan under `email-connect` (suggested slug theme: +`send-quota-privilege-tiers` / `EMAIL-WP-00xx`), with a hand-off to +flex-auth / railiance-platform if the tier is a platform grant claim rather +than a local ConfigMap. + +--- + ## Operate at build-stage security posture; defer production policy enforcement **Date:** 2026-07-02 diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 6fc62f7..8f16e78 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -11,7 +11,7 @@ | workplan | EMAIL-WP-0001 | finished | — | workplans/EMAIL-WP-0001-repo-onboarding.md | | workplan | EMAIL-WP-0002 | finished | — | workplans/EMAIL-WP-0002-mvp-mailbox-evidence-scanner.md | | workplan | EMAIL-WP-0003 | finished | — | workplans/EMAIL-WP-0003-expected-recipient-reporting-and-mailbox-tutorial.md | -| workplan | EMAIL-WP-0004 | active | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | +| workplan | EMAIL-WP-0004 | finished | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | | task | EMAIL-WP-0001-T01 | done | — | workplans/EMAIL-WP-0001-repo-onboarding.md | | task | EMAIL-WP-0001-T02 | done | — | workplans/EMAIL-WP-0001-repo-onboarding.md | | task | EMAIL-WP-0001-T03 | done | — | workplans/EMAIL-WP-0001-repo-onboarding.md | @@ -37,5 +37,5 @@ | task | EMAIL-WP-0003-T07 | done | — | workplans/EMAIL-WP-0003-expected-recipient-reporting-and-mailbox-tutorial.md | | task | EMAIL-WP-0004-T01 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | | task | EMAIL-WP-0004-T02 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | -| task | EMAIL-WP-0004-T03 | progress | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | -| task | EMAIL-WP-0004-T04 | todo | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | +| task | EMAIL-WP-0004-T03 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | +| task | EMAIL-WP-0004-T04 | done | — | workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md | diff --git a/deploy/k8s/railiance/README.md b/deploy/k8s/railiance/README.md new file mode 100644 index 0000000..cafe6b2 --- /dev/null +++ b/deploy/k8s/railiance/README.md @@ -0,0 +1,136 @@ +# email-connect transactional receiver — railiance01 + +Deploy the narrow invitation/verification HTTP receiver so **user-engine** can +send mail without holding SMTP credentials, and so unrelated workloads cannot +reach the send API. + +**Target cluster:** railiance01 only (`KUBECONFIG=~/.kube/config-hosteurope`). + +Stable in-cluster URL after apply: + +```text +http://email-connect.email-connect.svc.cluster.local:8080 +``` + +## Ownership split + +| Concern | Owner | Where | +| --- | --- | --- | +| Package, manifests, probes, NetworkPolicy, rollback | `email-connect` | this directory | +| OpenBao path, ESO role/policy, ClusterSecretStore | `railiance-platform` | CCR-2026-0010, `openbao/policies/` | +| Credential routing catalog | `ops-warden` | `email-connect-transactional` | + +Never paste SMTP passwords, ingest tokens, or OpenBao tokens into Git, State +Hub, workplans, logs, or chat. Route first: + +```bash +warden route find "email-connect SMTP transactional" --json +warden route show email-connect-transactional --json +``` + +## Runtime secrets (OpenBao → ESO) + +| Item | Value | +| --- | --- | +| KV path | `platform/workloads/email-connect/transactional` | +| Fields | `EMAIL_CONNECT_INGEST_TOKEN`, `EMAIL_CONNECT_SMTP_USERNAME`, `EMAIL_CONNECT_SMTP_PASSWORD` | +| Read / ESO policy | `external-secrets-email-connect` (+ `workload-kv-read-email-connect-transactional` for operator OIDC) | +| ClusterSecretStore | `openbao-email-connect` (namespace-limited; token auth like activity-core/reuse) | +| ESO token Secret | `external-secrets/openbao-email-connect-eso-token` (via `openbao-eso-token-apply.sh`) | +| ExternalSecret | `email-connect/email-connect-runtime` → Secret `email-connect-runtime` | +| ops-warden catalog | `email-connect-transactional` (draft until provisioned) | + +Non-secret config (host, port, sender, portal URL) lives in ConfigMap +`email-connect-config`. + +### Operator provision (attended, after CCR approval) + +```bash +# Values from mode-0600 files only; shred after. +bao kv put platform/workloads/email-connect/transactional \ + EMAIL_CONNECT_INGEST_TOKEN=@/path/to/ingest.token \ + EMAIL_CONNECT_SMTP_USERNAME=@/path/to/smtp.user \ + EMAIL_CONNECT_SMTP_PASSWORD=@/path/to/smtp.pass +``` + +user-engine must receive the **same** ingest token under its own runtime secret +path so it can call `Authorization: Bearer …` without ever seeing SMTP fields. + +## Network posture + +- **Default deny** ingress and egress in `email-connect`. +- **Ingress:** only pods labeled `app.kubernetes.io/name=user-engine` in + namespace `user-engine`, TCP 8080. +- **Egress:** kube-system DNS + TCP 587 to the public internet (IONOS + STARTTLS). No cluster-lateral egress and no general HTTPS. + +Bearer auth on `/v1/*` is the second gate; NetworkPolicy is the outer gate. + +## Apply + +Prerequisites: + +1. CCR-2026-0010 approved. +2. KV secret provisioned at `platform/workloads/email-connect/transactional` + (fields non-empty; do not log values). +3. ESO delivery token + ClusterSecretStore on railiance01 (token pattern matches + `openbao-activity-core` / `openbao-reuse`, not Kubernetes auth yet). +4. Image available on the node/registry at the digest pinned in `deployment.yaml`. + +```bash +export KUBECONFIG=~/.kube/config-hosteurope + +# 1) Policy + limited ESO child token → external-secrets Secret +./deploy/k8s/railiance/openbao-eso-token-apply.sh + +# 2) Namespace-scoped ClusterSecretStore (platform add-on copy) +kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml + +# 3) Optional: load a freshly built image when the registry is not yet updated +# docker build -f Containerfile -t forgejo.coulomb.social/coulomb/email-connect:local . +# docker save … | ssh railiance01 sudo k3s ctr -n k8s.io images import - + +# 4) Package +kubectl apply -k deploy/k8s/railiance +kubectl -n email-connect rollout status deployment/email-connect +kubectl -n email-connect get pods,svc,networkpolicy,externalsecret,secret +``` + +Expect ExternalSecret `SecretSynced` and a Ready pod. If the Secret is missing, +do not create a bootstrap Secret with values from chat — fix OpenBao/ESO. + +## Smoke (no SMTP proof yet) + +From a **user-engine** pod (or a temporary pod in that namespace with the same +labels — NetworkPolicy will otherwise drop the packet): + +```bash +# Health only (no auth) +kubectl -n user-engine exec deploy/user-engine -- \ + wget -qO- http://email-connect.email-connect.svc.cluster.local:8080/healthz + +# Auth check: wrong token must be 401; correct token is exercised by T04. +``` + +Negative checks: + +- A pod in another namespace cannot reach port 8080 (connection timeout/reset). +- user-engine runtime must not contain `EMAIL_CONNECT_SMTP_*` keys. + +## Rollback + +```bash +kubectl -n email-connect rollout undo deployment/email-connect +kubectl -n email-connect rollout status deployment/email-connect +``` + +`revisionHistoryLimit: 5` keeps prior digests. SQLite on the PVC is additive; +rolling back does not clear delivery/idempotency rows (by design). + +## Done criteria (EMAIL-WP-0004-T03) + +- [ ] user-engine calls the cluster-local receiver without possessing SMTP credentials +- [ ] Unrelated workloads cannot open TCP 8080 on the receiver +- [ ] SMTP password and ingest token exist only in OpenBao + the scoped K8s Secret +- [ ] Probes green; NetworkPolicy default-deny + SMTP-only egress applied +- [ ] Documented rollback path works against the previous revision diff --git a/deploy/k8s/railiance/clustersecretstore.yaml.template b/deploy/k8s/railiance/clustersecretstore.yaml.template new file mode 100644 index 0000000..ac5f364 --- /dev/null +++ b/deploy/k8s/railiance/clustersecretstore.yaml.template @@ -0,0 +1,30 @@ +# Template copy of the railiance-platform add-on store. Prefer applying from: +# ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml +# +# Do not apply until CCR-2026-0010 is approved and Secret +# external-secrets/openbao-email-connect-eso-token exists +# (./openbao-eso-token-apply.sh). +--- +apiVersion: external-secrets.io/v1beta1 +kind: ClusterSecretStore +metadata: + name: openbao-email-connect + labels: + app.kubernetes.io/part-of: railiance-gitops + railiance-platform/component: external-secrets + app.kubernetes.io/name: email-connect +spec: + provider: + vault: + # In-cluster OpenBao on railiance01 (not the public bao.coulomb.social UI). + server: http://openbao.openbao.svc:8200 + path: platform + version: v2 + auth: + tokenSecretRef: + name: openbao-email-connect-eso-token + namespace: external-secrets + key: token + conditions: + - namespaces: + - email-connect diff --git a/deploy/k8s/railiance/configmap.yaml b/deploy/k8s/railiance/configmap.yaml new file mode 100644 index 0000000..0c0d575 --- /dev/null +++ b/deploy/k8s/railiance/configmap.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: email-connect-config + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +data: + # Non-secret runtime shape only. SMTP auth and the caller ingest token stay + # in OpenBao → ExternalSecret (see externalsecret.yaml). + EMAIL_CONNECT_HTTP_PORT: "8080" + EMAIL_CONNECT_DATABASE_PATH: "/data/email-connect.db" + EMAIL_CONNECT_SMTP_HOST: "smtp.ionos.de" + EMAIL_CONNECT_SMTP_PORT: "587" + # Public portal base used in invitation and verification links. + # Matches the live user-engine Ingress host on railiance01 (nip.io until a + # coulomb.social alias is cut over). Update when the public hostname changes. + EMAIL_CONNECT_PORTAL_URL: "https://users.92-205-62-239.nip.io" + # Envelope From — interim identity matches the live forgejo IONOS SMTP lane + # (forgejo@coulomb.social) used to seed CCR-2026-0010. Switch to a dedicated + # noreply@ mailbox when that credential is minted separately. + EMAIL_CONNECT_SENDER: "forgejo@coulomb.social" diff --git a/deploy/k8s/railiance/deployment.yaml b/deploy/k8s/railiance/deployment.yaml new file mode 100644 index 0000000..f91e6c2 --- /dev/null +++ b/deploy/k8s/railiance/deployment.yaml @@ -0,0 +1,97 @@ +# email-connect transactional receiver — railiance01 (EMAIL-WP-0004-T03). +# +# Conventions match audit-core / user-engine: digest-pinned image from +# forgejo.coulomb.social, non-root, read-only root, probes on named http port. +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: email-connect + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect + annotations: + email-connect.railiance.io/rollback-note: >- + SQLite schema is additive (CREATE TABLE IF NOT EXISTS + optional + canceled_at column). Rolling back to the previous digest keeps the + same store path; duplicate event_id rows remain authoritative. +spec: + replicas: 1 + revisionHistoryLimit: 5 + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + selector: + matchLabels: + app.kubernetes.io/name: email-connect + template: + metadata: + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect + spec: + serviceAccountName: email-connect + securityContext: + runAsNonRoot: true + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + terminationGracePeriodSeconds: 15 + containers: + - name: email-connect + # Pin at release. T04 failure-matrix image (provider classification + + # suppression + redacted diagnostics), 2026-08-12. + image: forgejo.coulomb.social/coulomb/email-connect@sha256:a9de1994eea88502ec30ba7c2fc029ec901d873140d51e11d2c50e363a0578b1 + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 8080 + envFrom: + - configMapRef: + name: email-connect-config + - secretRef: + name: email-connect-runtime + optional: false + resources: + requests: + cpu: 25m + memory: 64Mi + limits: + cpu: 250m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + volumeMounts: + - name: data + mountPath: /data + - name: tmp + mountPath: /tmp + startupProbe: + httpGet: {path: /healthz, port: http} + periodSeconds: 3 + failureThreshold: 20 + readinessProbe: + httpGet: {path: /readyz, port: http} + periodSeconds: 10 + timeoutSeconds: 2 + failureThreshold: 3 + livenessProbe: + httpGet: {path: /healthz, port: http} + periodSeconds: 20 + timeoutSeconds: 2 + failureThreshold: 3 + volumes: + - name: data + persistentVolumeClaim: + claimName: email-connect-data + - name: tmp + emptyDir: {} diff --git a/deploy/k8s/railiance/externalsecret.yaml b/deploy/k8s/railiance/externalsecret.yaml new file mode 100644 index 0000000..935946a --- /dev/null +++ b/deploy/k8s/railiance/externalsecret.yaml @@ -0,0 +1,44 @@ +# Credential delivery for the transactional receiver (EMAIL-WP-0004-T03). +# +# PREREQUISITES (railiance-platform, not this repo): +# - ClusterSecretStore openbao-email-connect limited to this namespace +# - OpenBao policy external-secrets-email-connect + role external-secrets-email-connect +# - Operator-provisioned KV at platform/workloads/email-connect/transactional +# (fields listed below; no secret values in Git) +# Apply order: ClusterSecretStore → this ExternalSecret → Deployment. +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: email-connect-runtime + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao-email-connect + target: + name: email-connect-runtime + creationPolicy: Owner + deletionPolicy: Retain + data: + # Shared bearer for user-engine → email-connect. user-engine holds the same + # value as EMAIL_CONNECT_INGEST_TOKEN under its own custody path; this + # service never receives SMTP material from the caller. + - secretKey: EMAIL_CONNECT_INGEST_TOKEN + remoteRef: + key: platform/workloads/email-connect/transactional + property: EMAIL_CONNECT_INGEST_TOKEN + - secretKey: EMAIL_CONNECT_SMTP_USERNAME + remoteRef: + key: platform/workloads/email-connect/transactional + property: EMAIL_CONNECT_SMTP_USERNAME + - secretKey: EMAIL_CONNECT_SMTP_PASSWORD + remoteRef: + key: platform/workloads/email-connect/transactional + property: EMAIL_CONNECT_SMTP_PASSWORD diff --git a/deploy/k8s/railiance/kustomization.yaml b/deploy/k8s/railiance/kustomization.yaml new file mode 100644 index 0000000..da55daa --- /dev/null +++ b/deploy/k8s/railiance/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: email-connect +resources: + - namespace.yaml + - serviceaccount.yaml + - configmap.yaml + - pvc.yaml + - service.yaml + - networkpolicies.yaml + - externalsecret.yaml + - deployment.yaml diff --git a/deploy/k8s/railiance/namespace.yaml b/deploy/k8s/railiance/namespace.yaml new file mode 100644 index 0000000..c5a5a78 --- /dev/null +++ b/deploy/k8s/railiance/namespace.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: email-connect + labels: + kubernetes.io/metadata.name: email-connect + railiance.io/workload-class: platform + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect diff --git a/deploy/k8s/railiance/networkpolicies.yaml b/deploy/k8s/railiance/networkpolicies.yaml new file mode 100644 index 0000000..02d59c4 --- /dev/null +++ b/deploy/k8s/railiance/networkpolicies.yaml @@ -0,0 +1,78 @@ +# Default-deny plus the narrowest exceptions (EMAIL-WP-0004-T03). +# +# Reachability is part of the threat model: only user-engine may call the +# receiver, and egress is limited to DNS + the IONOS SMTP STARTTLS port. +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: email-connect-default-deny + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +spec: + podSelector: {} + policyTypes: [Ingress, Egress] + # No rules: everything not permitted below is denied. +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: email-connect-user-engine-ingress + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: email-connect + policyTypes: [Ingress] + ingress: + # user-engine is the only approved sender. Adding a second caller is a + # deliberate change here and a matching credential binding. + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: user-engine + podSelector: + matchLabels: + app.kubernetes.io/name: user-engine + ports: + - {protocol: TCP, port: 8080} +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: email-connect-egress + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: email-connect + policyTypes: [Egress] + egress: + # Cluster DNS + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - {protocol: UDP, port: 53} + - {protocol: TCP, port: 53} + # IONOS SMTP STARTTLS only. No general internet egress. + # smtp.ionos.de resolves outside the cluster; CIDR is intentionally broad + # but port-locked to 587. Tighten to provider CIDRs if inventory is fixed. + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + ports: + - {protocol: TCP, port: 587} diff --git a/deploy/k8s/railiance/openbao-eso-token-apply.sh b/deploy/k8s/railiance/openbao-eso-token-apply.sh new file mode 100755 index 0000000..09a2207 --- /dev/null +++ b/deploy/k8s/railiance/openbao-eso-token-apply.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Mint a read-limited OpenBao token and store it on railiance01 for +# ClusterSecretStore openbao-email-connect / ExternalSecret email-connect-runtime. +# +# Policies (files in railiance-platform/openbao/policies/): +# external-secrets-email-connect +# workload-kv-read-email-connect-transactional (optional operator OIDC sibling) +# +# Does not print secret values. Requires an attended operator OpenBao token. +# CCR-2026-0010 must be approved before production use. +set -euo pipefail + +DEFAULT_POLICIES="external-secrets-email-connect" +POLICIES="${OPENBAO_EMAIL_CONNECT_POLICIES:-$DEFAULT_POLICIES}" +POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}" +BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}" +RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}" +SECRET_NAME="${OPENBAO_EMAIL_CONNECT_ESO_SECRET:-openbao-email-connect-eso-token}" +SECRET_NS="${OPENBAO_EMAIL_CONNECT_ESO_NAMESPACE:-external-secrets}" +TTL="${OPENBAO_EMAIL_CONNECT_ESO_TTL:-768h}" + +if ! command -v bao >/dev/null 2>&1; then + echo "ERROR: bao CLI not found" >&2 + exit 1 +fi +if ! command -v kubectl >/dev/null 2>&1; then + echo "ERROR: kubectl not found" >&2 + exit 1 +fi + +echo "OpenBao addr: $BAO_ADDR" +echo "Policies: $POLICIES" +echo "K8s secret: $SECRET_NS/$SECRET_NAME (railiance01)" + +if [[ -n "${BAO_TOKEN:-}" ]]; then + : +elif [[ -n "${OPENBAO_TOKEN_FILE:-}" && -f "${OPENBAO_TOKEN_FILE}" ]]; then + BAO_TOKEN="$(head -n 1 "${OPENBAO_TOKEN_FILE}")" +else + read -r -s -p "OpenBao operator token: " BAO_TOKEN + echo >&2 +fi +if [[ -z "${BAO_TOKEN:-}" ]]; then + echo "ERROR: empty OpenBao token" >&2 + exit 1 +fi + +export BAO_ADDR BAO_TOKEN + +health="$(curl -fsS "$BAO_ADDR/v1/sys/health")" +if echo "$health" | grep -q '"sealed":true'; then + echo "ERROR: OpenBao at $BAO_ADDR reports sealed" >&2 + exit 1 +fi + +for policy in $POLICIES; do + policy_file="$POLICY_DIR/${policy}.hcl" + if [[ -f "$policy_file" ]]; then + bao policy write "$policy" "$policy_file" + echo "policy written: $policy" + else + echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2 + fi +done + +# Child token: renewable, orphan so operator logout does not revoke delivery. +# shellcheck disable=SC2086 +token_json="$(bao token create -policy="$(echo $POLICIES | tr ' ' ',')" -ttl="$TTL" -renewable=true -orphan -format=json)" +child_token="$(printf '%s' "$token_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["auth"]["client_token"])')" +if [[ -z "$child_token" || ${#child_token} -lt 8 ]]; then + echo "ERROR: failed to mint child token" >&2 + exit 1 +fi +echo "minted child token length=${#child_token} (value not printed)" + +export KUBECONFIG="$RAILIANCE01_KUBECONFIG" +kubectl -n "$SECRET_NS" create secret generic "$SECRET_NAME" \ + --from-literal=token="$child_token" \ + --dry-run=client -o yaml | kubectl apply -f - + +unset child_token BAO_TOKEN +echo "Secret $SECRET_NS/$SECRET_NAME applied on railiance01." +echo "Next: apply ClusterSecretStore openbao-email-connect, then email-connect package." diff --git a/deploy/k8s/railiance/pvc.yaml b/deploy/k8s/railiance/pvc.yaml new file mode 100644 index 0000000..2a61707 --- /dev/null +++ b/deploy/k8s/railiance/pvc.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: email-connect-data + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + # SQLite idempotency + verification digest store only. + storage: 1Gi diff --git a/deploy/k8s/railiance/service.yaml b/deploy/k8s/railiance/service.yaml new file mode 100644 index 0000000..2c994a3 --- /dev/null +++ b/deploy/k8s/railiance/service.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Service +metadata: + name: email-connect + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: email-connect + ports: + - name: http + port: 8080 + targetPort: http + protocol: TCP diff --git a/deploy/k8s/railiance/serviceaccount.yaml b/deploy/k8s/railiance/serviceaccount.yaml new file mode 100644 index 0000000..b438301 --- /dev/null +++ b/deploy/k8s/railiance/serviceaccount.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: email-connect + namespace: email-connect + labels: + app.kubernetes.io/name: email-connect + app.kubernetes.io/part-of: email-connect +automountServiceAccountToken: false diff --git a/docs/EMAIL-WP-0004-T04-failure-evidence.md b/docs/EMAIL-WP-0004-T04-failure-evidence.md new file mode 100644 index 0000000..666dc27 --- /dev/null +++ b/docs/EMAIL-WP-0004-T04-failure-evidence.md @@ -0,0 +1,96 @@ +# EMAIL-WP-0004-T04 — invitation delivery failure evidence + +**Date:** 2026-08-12 +**Cluster:** railiance01 +**Service:** `http://email-connect.email-connect.svc.cluster.local:8080` +**Image:** `forgejo.coulomb.social/coulomb/email-connect@sha256:a9de1994eea88502ec30ba7c2fc029ec901d873140d51e11d2c50e363a0578b1` +**Unit tests:** `PYTHONPATH=src python3 -m pytest tests/test_transactional.py` — **9 passed** + +This note is the non-secret hand-back for **NK-WP-0024**. No SMTP passwords, +ingest tokens, or raw provider exception text are included. + +## Contract for user-engine + +| HTTP | `error` / `status` | `retryable` | Meaning for outbox | +| --- | --- | --- | --- | +| 401 | `unauthorized` | — | Wrong or missing bearer; fail closed | +| 400 | `template_not_allowed` | — | Event type/source not on allow-list | +| 400 | `invalid_recipient` | — | Address failed local validation | +| 400 | `idempotency_key_mismatch` | — | `Idempotency-Key` ≠ event `id` | +| 400 | `recipient_suppressed` | — | Address on suppression list; do not retry as send | +| 400 | `invitation_id_required` | — | Missing invitation id | +| 200 | `status=duplicate` + `reference` | — | Same event id already accepted; safe replay | +| 202 | `status=accepted` + `reference` + `event_id` | — | Provider accepted message; **not** inbox delivery | +| 503 | `provider_timeout` / `temporary_deferral` / `provider_unavailable` | `true` | Bounded retry | +| 422 | `permanent_rejection` | `false` | Do not retry; surface failure evidence | + +### Evidence ceilings (must not become authorization) + +- Invitation accept: `evidence_ceiling=provider_accepted` — transport only. +- Registration consume/cancel: `authorization=false`, ceilings + `mailbox_challenge_consumed` / `mailbox_challenge_canceled`. +- `assurance.mailbox_control=true` is **channel evidence**, not identity proof + and not a flex-auth decision. + +Mailbox ownership challenges and provider acceptance **must never** alter +user-engine authorization decisions. + +## Unit matrix (local) + +| Case | Result | +| --- | --- | +| Template allow-list denial | `400 template_not_allowed` | +| Invalid recipient | `400 invalid_recipient` | +| Idempotency mismatch | `400 idempotency_key_mismatch` | +| Auth denial | `401 unauthorized` | +| Duplicate event id | `200 duplicate` + same `reference`; provider called once | +| Resend (`family_invitation.resent`) | New event id → second provider send | +| Suppression | `400 recipient_suppressed`; no provider call | +| Provider timeout / temporary / unavailable | `503` + `retryable=true`, redacted codes | +| Permanent rejection | `422` + `retryable=false` | +| SMTPRecipientsRefused → permanent | ProviderError mapping unit-tested | +| Redaction | Response JSON has no password/SMTP host/exception text | +| Mailbox evidence ≠ authz | `authorization=false` on consume/cancel | + +## Live matrix (railiance01, from `user-engine` pod) + +| Case | HTTP | Body (redacted) | +| --- | --- | --- | +| Unauth | 401 | `{"error":"unauthorized"}` | +| Template deny | 400 | `{"error":"template_not_allowed"}` | +| Invalid recipient | 400 | `{"error":"invalid_recipient"}` | +| Idempotency mismatch | 400 | `{"error":"idempotency_key_mismatch"}` | +| Accept (to `forgejo@coulomb.social`) | 202 | `status=accepted`, `evidence_ceiling=provider_accepted`, `event_id` set | +| Duplicate same event id | 200 | `status=duplicate`, same `reference` | +| Resend new event id | 202 | `status=accepted` | +| Bad domain `nobody@invalid.invalid` | 422 | `{"error":"permanent_rejection","retryable":false}` | +| Suppressed `suppressed@example.test` | 400 | `{"error":"recipient_suppressed"}` (seeded in SQLite for smoke) | +| Redaction | — | No `password` / `smtp.ionos` in any body | + +Negative path (unchanged from T03): pods outside `user-engine` cannot reach +TCP 8080 (Connection refused). + +## Non-secret references for NK-WP-0024 + +- Cluster-local base URL: + `http://email-connect.email-connect.svc.cluster.local:8080` +- Paths: `POST /v1/send`, `POST /v1/registration-verifications`, + `…/consume`, `…/cancel` +- Auth: shared ingest bearer (OpenBao + `platform/workloads/email-connect/transactional` field + `EMAIL_CONNECT_INGEST_TOKEN` only — never SMTP fields in user-engine) +- Idempotency: header `Idempotency-Key` = event `id` +- Allowed send types: `family_member.invited`, `family_invitation.resent` +- Catalog route: `email-connect-transactional` (ops-warden) + +## Temporary deferral / timeout note + +Live temporary deferral depends on provider behavior and was covered by unit +classification (`temporary_deferral`, `provider_timeout` → 503 retryable). +Live permanent path was observed via invalid domain → `422 permanent_rejection`. + +## Follow-ups (out of T04) + +- Mount ingest token into user-engine runtime for production outbox (NK-WP-0024). +- Operator API to manage suppressions (currently SQLite; smoke seeded manually). +- Daily send-quota tiers deferred (`DECISIONS.md`). diff --git a/src/email_connect/transactional.py b/src/email_connect/transactional.py index dee539e..91b7ba3 100644 --- a/src/email_connect/transactional.py +++ b/src/email_connect/transactional.py @@ -19,54 +19,109 @@ from wsgiref.simple_server import make_server ALLOWED_EVENTS = {"family_member.invited", "family_invitation.resent"} +class ProviderError(Exception): + """Provider outcome with stable redacted code and retry classification.""" + + def __init__(self, code: str, *, retryable: bool) -> None: + super().__init__(code) + self.code = code + self.retryable = retryable + + class SQLiteDeliveryStore: def __init__(self, path: str) -> None: self.db = sqlite3.connect(path, check_same_thread=False) self.lock = RLock() - self.db.execute("CREATE TABLE IF NOT EXISTS deliveries (event_id TEXT PRIMARY KEY, provider_ref TEXT NOT NULL)") - self.db.execute("""CREATE TABLE IF NOT EXISTS verifications ( + self.db.execute( + "CREATE TABLE IF NOT EXISTS deliveries (event_id TEXT PRIMARY KEY, provider_ref TEXT NOT NULL)" + ) + self.db.execute( + """CREATE TABLE IF NOT EXISTS verifications ( request_id TEXT PRIMARY KEY, handle_hash TEXT UNIQUE NOT NULL, registration_id TEXT NOT NULL, email TEXT NOT NULL, username TEXT NOT NULL, client_id TEXT NOT NULL, tenant TEXT NOT NULL, display_name TEXT, expires_at TEXT NOT NULL, consumed_at TEXT, canceled_at TEXT - )""") + )""" + ) + self.db.execute( + """CREATE TABLE IF NOT EXISTS suppressions ( + email TEXT PRIMARY KEY NOT NULL, + reason TEXT NOT NULL, + created_at TEXT NOT NULL + )""" + ) columns = {row[1] for row in self.db.execute("PRAGMA table_info(verifications)")} if "canceled_at" not in columns: self.db.execute("ALTER TABLE verifications ADD COLUMN canceled_at TEXT") self.db.commit() def reference(self, event_id: str) -> str | None: - row = self.db.execute("SELECT provider_ref FROM deliveries WHERE event_id=?", (event_id,)).fetchone() + row = self.db.execute( + "SELECT provider_ref FROM deliveries WHERE event_id=?", (event_id,) + ).fetchone() return row[0] if row else None def record(self, event_id: str, reference: str) -> None: with self.db: self.db.execute("INSERT INTO deliveries VALUES (?, ?)", (event_id, reference)) + def is_suppressed(self, email: str) -> bool: + row = self.db.execute( + "SELECT 1 FROM suppressions WHERE email=?", (email.lower(),) + ).fetchone() + return row is not None + + def suppress(self, email: str, reason: str = "manual") -> None: + with self.db: + self.db.execute( + "INSERT OR REPLACE INTO suppressions VALUES (?,?,?)", + (email.lower(), reason, datetime.now(timezone.utc).isoformat()), + ) + def create_verification(self, payload: dict, handle_hash: str, expires_at: str) -> str: request_id = f"vrq_{secrets.token_hex(12)}" with self.db: - self.db.execute("""INSERT INTO verifications + self.db.execute( + """INSERT INTO verifications (request_id, handle_hash, registration_id, email, username, client_id, tenant, display_name, expires_at, consumed_at, canceled_at) - VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""", ( - request_id, handle_hash, payload["registration_id"], payload["normalized_email"], - payload["preferred_username"], payload["client_id"], payload["tenant"], - payload.get("display_name"), expires_at, - )) + VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""", + ( + request_id, + handle_hash, + payload["registration_id"], + payload["normalized_email"], + payload["preferred_username"], + payload["client_id"], + payload["tenant"], + payload.get("display_name"), + expires_at, + ), + ) return request_id def consume_verification(self, handle_hash: str) -> dict: with self.lock, self.db: - row = self.db.execute("SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,)).fetchone() + row = self.db.execute( + "SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,) + ).fetchone() if row is None or row[9] is not None or row[10] is not None: raise ValueError("verification_invalid") if datetime.fromisoformat(row[8]) <= datetime.now(timezone.utc): raise ValueError("verification_expired") - self.db.execute("UPDATE verifications SET consumed_at=? WHERE request_id=?", - (datetime.now(timezone.utc).isoformat(), row[0])) - return {"request_id":row[0],"registration_id":row[2],"email":row[3], - "preferred_username":row[4],"client_id":row[5],"tenant":row[6],"display_name":row[7]} + self.db.execute( + "UPDATE verifications SET consumed_at=? WHERE request_id=?", + (datetime.now(timezone.utc).isoformat(), row[0]), + ) + return { + "request_id": row[0], + "registration_id": row[2], + "email": row[3], + "preferred_username": row[4], + "client_id": row[5], + "tenant": row[6], + "display_name": row[7], + } def cancel_verification(self, handle_hash: str) -> dict: with self.lock, self.db: @@ -81,22 +136,50 @@ class SQLiteDeliveryStore: "UPDATE verifications SET canceled_at=? WHERE request_id=?", (datetime.now(timezone.utc).isoformat(), row[0]), ) - return {"request_id":row[0],"registration_id":row[2],"email":row[3], - "preferred_username":row[4],"client_id":row[5],"tenant":row[6],"display_name":row[7]} + return { + "request_id": row[0], + "registration_id": row[2], + "email": row[3], + "preferred_username": row[4], + "client_id": row[5], + "tenant": row[6], + "display_name": row[7], + } class SMTPProvider: def __init__(self, host: str, port: int, username: str, password: str, sender: str) -> None: - self.host, self.port, self.username, self.password, self.sender = host, port, username, password, sender + self.host, self.port, self.username, self.password, self.sender = ( + host, + port, + username, + password, + sender, + ) def send(self, recipient: str, subject: str, text: str) -> str: message = EmailMessage() message["From"], message["To"], message["Subject"] = self.sender, recipient, subject message.set_content(text) - with smtplib.SMTP(self.host, self.port, timeout=10) as smtp: - smtp.starttls() - smtp.login(self.username, self.password) - smtp.send_message(message) + try: + with smtplib.SMTP(self.host, self.port, timeout=10) as smtp: + smtp.starttls() + smtp.login(self.username, self.password) + smtp.send_message(message) + except TimeoutError as exc: + raise ProviderError("provider_timeout", retryable=True) from exc + except smtplib.SMTPRecipientsRefused as exc: + # 5xx recipient refusals are permanent; never surface SMTP text. + raise ProviderError("permanent_rejection", retryable=False) from exc + except smtplib.SMTPResponseException as exc: + code = int(getattr(exc, "smtp_code", 0) or 0) + if 400 <= code < 500: + raise ProviderError("temporary_deferral", retryable=True) from exc + if 500 <= code < 600: + raise ProviderError("permanent_rejection", retryable=False) from exc + raise ProviderError("provider_unavailable", retryable=True) from exc + except (OSError, smtplib.SMTPException) as exc: + raise ProviderError("provider_unavailable", retryable=True) from exc return message["Message-ID"] or f"smtp:{abs(hash((recipient, subject)))}" @@ -107,15 +190,23 @@ class TransactionalApplication: def __call__(self, environ, start_response): if environ.get("PATH_INFO") in ("/healthz", "/readyz"): - return self._json(start_response, HTTPStatus.OK, {"status":"ok"}) + return self._json(start_response, HTTPStatus.OK, {"status": "ok"}) path = environ.get("PATH_INFO") - if path not in ("/v1/send", "/v1/registration-verifications", "/v1/registration-verifications/consume", "/v1/registration-verifications/cancel") or environ.get("REQUEST_METHOD") != "POST": - return self._json(start_response, HTTPStatus.NOT_FOUND, {"error":"not_found"}) - if not hmac.compare_digest(str(environ.get("HTTP_AUTHORIZATION", "")), f"Bearer {self.token}"): - return self._json(start_response, HTTPStatus.UNAUTHORIZED, {"error":"unauthorized"}) + if path not in ( + "/v1/send", + "/v1/registration-verifications", + "/v1/registration-verifications/consume", + "/v1/registration-verifications/cancel", + ) or environ.get("REQUEST_METHOD") != "POST": + return self._json(start_response, HTTPStatus.NOT_FOUND, {"error": "not_found"}) + if not hmac.compare_digest( + str(environ.get("HTTP_AUTHORIZATION", "")), f"Bearer {self.token}" + ): + return self._json(start_response, HTTPStatus.UNAUTHORIZED, {"error": "unauthorized"}) try: length = int(environ.get("CONTENT_LENGTH") or 0) - if length <= 0 or length > 128 * 1024: raise ValueError("invalid_size") + if length <= 0 or length > 128 * 1024: + raise ValueError("invalid_size") payload = json.loads(environ["wsgi.input"].read(length)) if path == "/v1/registration-verifications": return self._request_verification(start_response, payload) @@ -123,71 +214,141 @@ class TransactionalApplication: return self._consume_verification(start_response, payload) if path == "/v1/registration-verifications/cancel": return self._cancel_verification(start_response, payload) - event_id = str(payload["id"]) - if environ.get("HTTP_IDEMPOTENCY_KEY") != event_id: raise ValueError("idempotency_key_mismatch") - if payload.get("source") != "user-engine" or payload.get("type") not in ALLOWED_EVENTS: raise ValueError("template_not_allowed") - recipient = str(payload.get("data", {}).get("primary_email", "")) - if not _valid_address(recipient): raise ValueError("invalid_recipient") - existing = self.store.reference(event_id) - if existing: - return self._json(start_response, HTTPStatus.OK, {"status":"duplicate","reference":existing}) - invitation_id = str(payload.get("data", {}).get("invitation_id", "")) - if not invitation_id: raise ValueError("invitation_id_required") - reference = self.provider.send(recipient, "Your NetKingdom invitation", - f"You have been invited. Continue securely at {self.portal_url}/invitations/{invitation_id}\n") - self.store.record(event_id, reference) + return self._send_invitation(start_response, environ, payload) + except ProviderError as exc: + status = ( + HTTPStatus.SERVICE_UNAVAILABLE if exc.retryable else HTTPStatus.UNPROCESSABLE_ENTITY + ) + return self._json( + start_response, + status, + {"error": exc.code, "retryable": exc.retryable}, + ) except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc: - return self._json(start_response, HTTPStatus.BAD_REQUEST, {"error":str(exc)}) + return self._json(start_response, HTTPStatus.BAD_REQUEST, {"error": str(exc)}) except (TimeoutError, OSError, smtplib.SMTPException): - return self._json(start_response, HTTPStatus.SERVICE_UNAVAILABLE, {"error":"provider_unavailable","retryable":True}) - return self._json(start_response, HTTPStatus.ACCEPTED, {"status":"accepted","reference":reference}) + # Safety net: never leak provider exception text. + return self._json( + start_response, + HTTPStatus.SERVICE_UNAVAILABLE, + {"error": "provider_unavailable", "retryable": True}, + ) + + def _send_invitation(self, start_response, environ, payload): + event_id = str(payload["id"]) + if environ.get("HTTP_IDEMPOTENCY_KEY") != event_id: + raise ValueError("idempotency_key_mismatch") + if payload.get("source") != "user-engine" or payload.get("type") not in ALLOWED_EVENTS: + raise ValueError("template_not_allowed") + recipient = str(payload.get("data", {}).get("primary_email", "")) + if not _valid_address(recipient): + raise ValueError("invalid_recipient") + if self.store.is_suppressed(recipient): + raise ValueError("recipient_suppressed") + existing = self.store.reference(event_id) + if existing: + return self._json( + start_response, HTTPStatus.OK, {"status": "duplicate", "reference": existing} + ) + invitation_id = str(payload.get("data", {}).get("invitation_id", "")) + if not invitation_id: + raise ValueError("invitation_id_required") + reference = self.provider.send( + recipient, + "Your NetKingdom invitation", + f"You have been invited. Continue securely at {self.portal_url}/invitations/{invitation_id}\n", + ) + self.store.record(event_id, reference) + # Provider acceptance is transport evidence only — not identity or auth. + return self._json( + start_response, + HTTPStatus.ACCEPTED, + { + "status": "accepted", + "reference": reference, + "event_id": event_id, + "evidence_ceiling": "provider_accepted", + }, + ) def _request_verification(self, start_response, payload): - required = ("registration_id", "normalized_email", "preferred_username", "client_id", "tenant", "correlation_id") - if any(not payload.get(key) for key in required) or not _valid_address(str(payload["normalized_email"])): + required = ( + "registration_id", + "normalized_email", + "preferred_username", + "client_id", + "tenant", + "correlation_id", + ) + if any(not payload.get(key) for key in required) or not _valid_address( + str(payload["normalized_email"]) + ): raise ValueError("invalid_verification_request") + email = str(payload["normalized_email"]) + if self.store.is_suppressed(email): + raise ValueError("recipient_suppressed") handle = secrets.token_urlsafe(32) expires_at = datetime.now(timezone.utc) + timedelta(minutes=30) request_id = self.store.create_verification( payload, hashlib.sha256(handle.encode()).hexdigest(), expires_at.isoformat() ) self.provider.send( - str(payload["normalized_email"]), "Verify your NetKingdom registration", + email, + "Verify your NetKingdom registration", f"Continue securely at {self.portal_url}/registration/verify?handle={handle}\n" f"Cancel this request at {self.portal_url}/registration/cancel?handle={handle}\n" "These links expire in 30 minutes.\n", ) - return self._json(start_response, HTTPStatus.ACCEPTED, - {"request_id":request_id,"accepted":True}) + return self._json( + start_response, HTTPStatus.ACCEPTED, {"request_id": request_id, "accepted": True} + ) def _consume_verification(self, start_response, payload): handle = str(payload.get("handle") or "") if len(handle) < 32: raise ValueError("verification_invalid") evidence = self.store.consume_verification(hashlib.sha256(handle.encode()).hexdigest()) - return self._json(start_response, HTTPStatus.OK, { - "purpose":"public-registration", "verification_id":f"fvr_{secrets.token_hex(12)}", - **evidence, "source_system":"email-connect", - "assurance":{"mailbox_control":True}, - }) + # mailbox_control is channel evidence only; never an authorization decision. + return self._json( + start_response, + HTTPStatus.OK, + { + "purpose": "public-registration", + "verification_id": f"fvr_{secrets.token_hex(12)}", + **evidence, + "source_system": "email-connect", + "assurance": {"mailbox_control": True}, + "authorization": False, + "evidence_ceiling": "mailbox_challenge_consumed", + }, + ) def _cancel_verification(self, start_response, payload): handle = str(payload.get("handle") or "") if len(handle) < 32: raise ValueError("verification_invalid") - evidence = self.store.cancel_verification( - hashlib.sha256(handle.encode()).hexdigest() + evidence = self.store.cancel_verification(hashlib.sha256(handle.encode()).hexdigest()) + return self._json( + start_response, + HTTPStatus.OK, + { + "purpose": "public-registration-cancel", + "verification_id": f"fvc_{secrets.token_hex(12)}", + **evidence, + "source_system": "email-connect", + "assurance": {"mailbox_control": True}, + "authorization": False, + "evidence_ceiling": "mailbox_challenge_canceled", + }, ) - return self._json(start_response, HTTPStatus.OK, { - "purpose":"public-registration-cancel", - "verification_id":f"fvc_{secrets.token_hex(12)}", - **evidence, "source_system":"email-connect", - "assurance":{"mailbox_control":True}, - }) @staticmethod def _json(start_response, status, payload): - body=json.dumps(payload).encode(); start_response(f"{status.value} {status.phrase}", [("Content-Type","application/json"),("Content-Length",str(len(body)))]) + body = json.dumps(payload).encode() + start_response( + f"{status.value} {status.phrase}", + [("Content-Type", "application/json"), ("Content-Length", str(len(body)))], + ) return [body] @@ -197,8 +358,18 @@ def _valid_address(value: str) -> bool: def main() -> None: - provider = SMTPProvider(os.environ["EMAIL_CONNECT_SMTP_HOST"], int(os.environ.get("EMAIL_CONNECT_SMTP_PORT", "587")), - os.environ["EMAIL_CONNECT_SMTP_USERNAME"], os.environ["EMAIL_CONNECT_SMTP_PASSWORD"], os.environ["EMAIL_CONNECT_SENDER"]) - app = TransactionalApplication(SQLiteDeliveryStore(os.environ.get("EMAIL_CONNECT_DATABASE_PATH", "/data/email-connect.db")), - provider, os.environ["EMAIL_CONNECT_INGEST_TOKEN"].strip(), os.environ["EMAIL_CONNECT_PORTAL_URL"]) - with make_server("0.0.0.0", int(os.environ.get("EMAIL_CONNECT_HTTP_PORT", "8080")), app) as server: server.serve_forever() + provider = SMTPProvider( + os.environ["EMAIL_CONNECT_SMTP_HOST"], + int(os.environ.get("EMAIL_CONNECT_SMTP_PORT", "587")), + os.environ["EMAIL_CONNECT_SMTP_USERNAME"], + os.environ["EMAIL_CONNECT_SMTP_PASSWORD"], + os.environ["EMAIL_CONNECT_SENDER"], + ) + app = TransactionalApplication( + SQLiteDeliveryStore(os.environ.get("EMAIL_CONNECT_DATABASE_PATH", "/data/email-connect.db")), + provider, + os.environ["EMAIL_CONNECT_INGEST_TOKEN"].strip(), + os.environ["EMAIL_CONNECT_PORTAL_URL"], + ) + with make_server("0.0.0.0", int(os.environ.get("EMAIL_CONNECT_HTTP_PORT", "8080")), app) as server: + server.serve_forever() diff --git a/tests/test_transactional.py b/tests/test_transactional.py index 146caa7..f43cbc8 100644 --- a/tests/test_transactional.py +++ b/tests/test_transactional.py @@ -1,56 +1,278 @@ -import io, json -from email_connect.transactional import SQLiteDeliveryStore, TransactionalApplication +import io +import json +import smtplib + +from email_connect.transactional import ( + ProviderError, + SQLiteDeliveryStore, + TransactionalApplication, +) + class Provider: - def __init__(self): self.calls=[] - def send(self,*args): self.calls.append(args); return "message:1" + def __init__(self): + self.calls = [] + + def send(self, *args): + self.calls.append(args) + return "message:1" + + +class FailingProvider: + def __init__(self, error: ProviderError): + self.error = error + self.calls = 0 + + def send(self, *args): + self.calls += 1 + raise self.error + + +def event(event_id="evt-1", typ="family_member.invited", email="person@example.test"): + return { + "id": event_id, + "type": typ, + "source": "user-engine", + "data": {"primary_email": email, "invitation_id": "inv-1"}, + } + + +def invoke(app, payload, token="opaque", key=None, path="/v1/send"): + raw = json.dumps(payload).encode() + result = {} + env = { + "PATH_INFO": path, + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(raw)), + "wsgi.input": io.BytesIO(raw), + "HTTP_AUTHORIZATION": f"Bearer {token}", + } + if path == "/v1/send": + env["HTTP_IDEMPOTENCY_KEY"] = key if key is not None else payload.get("id", "evt-1") + body = b"".join(app(env, lambda s, h: result.update(status=s))) + return result["status"], json.loads(body) -def event(): return {"id":"evt-1","type":"family_member.invited","source":"user-engine","data":{"primary_email":"person@example.test","invitation_id":"inv-1"}} -def invoke(app,payload,token="opaque",key="evt-1"): - raw=json.dumps(payload).encode(); result={}; body=b"".join(app({"PATH_INFO":"/v1/send","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(raw)),"wsgi.input":io.BytesIO(raw),"HTTP_AUTHORIZATION":f"Bearer {token}","HTTP_IDEMPOTENCY_KEY":key},lambda s,h:result.update(status=s))); return result["status"],json.loads(body) def test_sends_fixed_template_once(tmp_path): - provider=Provider(); app=TransactionalApplication(SQLiteDeliveryStore(str(tmp_path/"mail.db")),provider,"opaque","https://users.example") - assert invoke(app,event())[0].startswith("202") - assert invoke(app,event())[1]["status"] == "duplicate" - assert len(provider.calls)==1 and "/invitations/inv-1" in provider.calls[0][2] + provider = Provider() + app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / "mail.db")), provider, "opaque", "https://users.example" + ) + status, body = invoke(app, event()) + assert status.startswith("202") + assert body["evidence_ceiling"] == "provider_accepted" + assert body["event_id"] == "evt-1" + assert invoke(app, event())[1]["status"] == "duplicate" + assert len(provider.calls) == 1 and "/invitations/inv-1" in provider.calls[0][2] + def test_rejects_auth_recipient_template_and_key(tmp_path): - app=TransactionalApplication(SQLiteDeliveryStore(str(tmp_path/"mail.db")),Provider(),"opaque","https://users.example") - assert invoke(app,event(),token="bad")[0].startswith("401") - bad=event(); bad["data"]["primary_email"]="bad\n@example.test"; assert invoke(app,bad)[1]["error"]=="invalid_recipient" - bad=event(); bad["type"]="arbitrary.send"; assert invoke(app,bad)[1]["error"]=="template_not_allowed" - assert invoke(app,event(),key="other")[1]["error"]=="idempotency_key_mismatch" + app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / "mail.db")), Provider(), "opaque", "https://users.example" + ) + assert invoke(app, event(), token="bad")[0].startswith("401") + bad = event() + bad["data"]["primary_email"] = "bad\n@example.test" + assert invoke(app, bad)[1]["error"] == "invalid_recipient" + bad = event() + bad["type"] = "arbitrary.send" + assert invoke(app, bad)[1]["error"] == "template_not_allowed" + assert invoke(app, event(), key="other")[1]["error"] == "idempotency_key_mismatch" + + +def test_resend_is_separate_event_and_sends_again(tmp_path): + provider = Provider() + app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / "mail.db")), provider, "opaque", "https://users.example" + ) + assert invoke(app, event("evt-1", "family_member.invited"))[0].startswith("202") + status, body = invoke(app, event("evt-2", "family_invitation.resent")) + assert status.startswith("202") and body["status"] == "accepted" + assert len(provider.calls) == 2 + + +def test_suppression_blocks_send(tmp_path): + store = SQLiteDeliveryStore(str(tmp_path / "mail.db")) + store.suppress("person@example.test", "complaint") + provider = Provider() + app = TransactionalApplication(store, provider, "opaque", "https://users.example") + status, body = invoke(app, event()) + assert status.startswith("400") + assert body["error"] == "recipient_suppressed" + assert provider.calls == [] + + +def test_provider_timeout_temporary_and_permanent_are_redacted(tmp_path): + cases = [ + (ProviderError("provider_timeout", retryable=True), "503", True), + (ProviderError("temporary_deferral", retryable=True), "503", True), + (ProviderError("permanent_rejection", retryable=False), "422", False), + (ProviderError("provider_unavailable", retryable=True), "503", True), + ] + for err, code_prefix, retryable in cases: + app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / f"{err.code}.db")), + FailingProvider(err), + "opaque", + "https://users.example", + ) + status, body = invoke(app, event(event_id=f"evt-{err.code}")) + assert status.startswith(code_prefix), (err.code, status, body) + assert body["error"] == err.code + assert body["retryable"] is retryable + # Diagnostics must stay redacted: no SMTP host, password, or exception text. + blob = json.dumps(body) + assert "smtp" not in blob.lower() + assert "password" not in blob.lower() + assert "exception" not in blob.lower() + + +def test_smtp_provider_maps_recipient_refused_to_permanent(): + class Boom(smtplib.SMTP): + def __init__(self, *a, **k): + pass + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def starttls(self): + return None + + def login(self, *a): + return None + + def send_message(self, msg): + raise smtplib.SMTPRecipientsRefused({"x@y.z": (550, b"user unknown")}) + + import email_connect.transactional as mod + + original = mod.smtplib.SMTP + mod.smtplib.SMTP = Boom + try: + provider = mod.SMTPProvider("smtp.example", 587, "u", "p", "from@example") + try: + provider.send("x@y.z", "s", "b") + assert False, "expected ProviderError" + except ProviderError as exc: + assert exc.code == "permanent_rejection" and exc.retryable is False + finally: + mod.smtplib.SMTP = original + def test_registration_verification_is_digest_only_and_single_use(tmp_path): - provider=Provider(); store=SQLiteDeliveryStore(str(tmp_path/"mail.db")); app=TransactionalApplication(store,provider,"opaque","https://users.example") - request={"registration_id":"reg-1","normalized_email":"person@example.test","preferred_username":"person","client_id":"coulomb-social","tenant":"tenant:coulomb","correlation_id":"corr-1"} - raw=json.dumps(request).encode(); result={}; response=json.loads(b"".join(app({"PATH_INFO":"/v1/registration-verifications","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(raw)),"wsgi.input":io.BytesIO(raw),"HTTP_AUTHORIZATION":"Bearer opaque"},lambda s,h:result.update(status=s)))) + provider = Provider() + store = SQLiteDeliveryStore(str(tmp_path / "mail.db")) + app = TransactionalApplication(store, provider, "opaque", "https://users.example") + request = { + "registration_id": "reg-1", + "normalized_email": "person@example.test", + "preferred_username": "person", + "client_id": "coulomb-social", + "tenant": "tenant:coulomb", + "correlation_id": "corr-1", + } + raw = json.dumps(request).encode() + result = {} + response = json.loads( + b"".join( + app( + { + "PATH_INFO": "/v1/registration-verifications", + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(raw)), + "wsgi.input": io.BytesIO(raw), + "HTTP_AUTHORIZATION": "Bearer opaque", + }, + lambda s, h: result.update(status=s), + ) + ) + ) assert result["status"].startswith("202") and response["accepted"] - text=provider.calls[0][2]; handle=text.split("handle=",1)[1].splitlines()[0] - stored=store.db.execute("SELECT handle_hash FROM verifications").fetchone()[0] + text = provider.calls[0][2] + handle = text.split("handle=", 1)[1].splitlines()[0] + stored = store.db.execute("SELECT handle_hash FROM verifications").fetchone()[0] assert handle not in stored and stored - consume=json.dumps({"handle":handle}).encode(); env={"PATH_INFO":"/v1/registration-verifications/consume","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(consume)),"wsgi.input":io.BytesIO(consume),"HTTP_AUTHORIZATION":"Bearer opaque"} - evidence=json.loads(b"".join(app(env,lambda s,h:result.update(status=s)))) - assert evidence["purpose"]=="public-registration" and evidence["email"]=="person@example.test" - env["wsgi.input"]=io.BytesIO(consume) - replay=json.loads(b"".join(app(env,lambda s,h:result.update(status=s)))) - assert replay["error"]=="verification_invalid" + consume = json.dumps({"handle": handle}).encode() + env = { + "PATH_INFO": "/v1/registration-verifications/consume", + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(consume)), + "wsgi.input": io.BytesIO(consume), + "HTTP_AUTHORIZATION": "Bearer opaque", + } + evidence = json.loads(b"".join(app(env, lambda s, h: result.update(status=s)))) + assert evidence["purpose"] == "public-registration" + assert evidence["email"] == "person@example.test" + assert evidence["authorization"] is False + assert evidence["evidence_ceiling"] == "mailbox_challenge_consumed" + env["wsgi.input"] = io.BytesIO(consume) + replay = json.loads(b"".join(app(env, lambda s, h: result.update(status=s)))) + assert replay["error"] == "verification_invalid" + def test_registration_cancellation_is_single_use_and_prevents_verification(tmp_path): - provider=Provider(); store=SQLiteDeliveryStore(str(tmp_path/"mail.db")); app=TransactionalApplication(store,provider,"opaque","https://users.example") - request={"registration_id":"reg-cancel","normalized_email":"person@example.test","preferred_username":"person","client_id":"coulomb-social","tenant":"tenant:coulomb","correlation_id":"corr-cancel"} - raw=json.dumps(request).encode(); result={} - app({"PATH_INFO":"/v1/registration-verifications","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(raw)),"wsgi.input":io.BytesIO(raw),"HTTP_AUTHORIZATION":"Bearer opaque"},lambda s,h:result.update(status=s)) - text=provider.calls[0][2]; handle=text.split("handle=",1)[1].splitlines()[0] + provider = Provider() + store = SQLiteDeliveryStore(str(tmp_path / "mail.db")) + app = TransactionalApplication(store, provider, "opaque", "https://users.example") + request = { + "registration_id": "reg-cancel", + "normalized_email": "person@example.test", + "preferred_username": "person", + "client_id": "coulomb-social", + "tenant": "tenant:coulomb", + "correlation_id": "corr-cancel", + } + raw = json.dumps(request).encode() + result = {} + app( + { + "PATH_INFO": "/v1/registration-verifications", + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(raw)), + "wsgi.input": io.BytesIO(raw), + "HTTP_AUTHORIZATION": "Bearer opaque", + }, + lambda s, h: result.update(status=s), + ) + text = provider.calls[0][2] + handle = text.split("handle=", 1)[1].splitlines()[0] assert f"/registration/cancel?handle={handle}" in text - payload=json.dumps({"handle":handle}).encode() - cancel_env={"PATH_INFO":"/v1/registration-verifications/cancel","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(payload)),"wsgi.input":io.BytesIO(payload),"HTTP_AUTHORIZATION":"Bearer opaque"} - canceled=json.loads(b"".join(app(cancel_env,lambda s,h:result.update(status=s)))) - assert canceled["purpose"]=="public-registration-cancel" - consume_env={"PATH_INFO":"/v1/registration-verifications/consume","REQUEST_METHOD":"POST","CONTENT_LENGTH":str(len(payload)),"wsgi.input":io.BytesIO(payload),"HTTP_AUTHORIZATION":"Bearer opaque"} - rejected=json.loads(b"".join(app(consume_env,lambda s,h:result.update(status=s)))) - assert rejected["error"]=="verification_invalid" - cancel_env["wsgi.input"]=io.BytesIO(payload) - replay=json.loads(b"".join(app(cancel_env,lambda s,h:result.update(status=s)))) - assert replay["error"]=="verification_invalid" + payload = json.dumps({"handle": handle}).encode() + cancel_env = { + "PATH_INFO": "/v1/registration-verifications/cancel", + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(payload)), + "wsgi.input": io.BytesIO(payload), + "HTTP_AUTHORIZATION": "Bearer opaque", + } + canceled = json.loads(b"".join(app(cancel_env, lambda s, h: result.update(status=s)))) + assert canceled["purpose"] == "public-registration-cancel" + assert canceled["authorization"] is False + consume_env = { + "PATH_INFO": "/v1/registration-verifications/consume", + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(payload)), + "wsgi.input": io.BytesIO(payload), + "HTTP_AUTHORIZATION": "Bearer opaque", + } + rejected = json.loads(b"".join(app(consume_env, lambda s, h: result.update(status=s)))) + assert rejected["error"] == "verification_invalid" + cancel_env["wsgi.input"] = io.BytesIO(payload) + replay = json.loads(b"".join(app(cancel_env, lambda s, h: result.update(status=s)))) + assert replay["error"] == "verification_invalid" + + +def test_mailbox_evidence_is_not_authorization(tmp_path): + """Provider acceptance and mailbox challenge never grant user-engine authority.""" + provider = Provider() + app = TransactionalApplication( + SQLiteDeliveryStore(str(tmp_path / "mail.db")), provider, "opaque", "https://users.example" + ) + _, accepted = invoke(app, event()) + assert accepted.get("authorization") is None or accepted.get("authorization") is False + assert accepted["evidence_ceiling"] == "provider_accepted" + # Ceiling is explicit; user-engine must not elevate this to authz. + assert "authorized" not in accepted diff --git a/workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md b/workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md index 142b159..d873b76 100644 --- a/workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md +++ b/workplans/EMAIL-WP-0004-transactional-mail-delivery-service.md @@ -4,11 +4,11 @@ type: workplan title: "Provide transactional invitation mail delivery" domain: infotech repo: email-connect -status: active +status: finished owner: codex topic_slug: netkingdom created: "2026-08-08" -updated: "2026-08-10" +updated: "2026-08-12" depends_on: - NK-WP-0024 state_hub_workstream_id: "a37e5e4d-090a-4a75-88d2-aacd0c0fd235" @@ -20,6 +20,10 @@ Extend email-connect beyond mailbox evidence scanning with a narrow, provider-neutral transactional-send service for user-engine invitations and verification messages. Email evidence remains non-authoritative. +Deferred (not in this workplan): daily send-quota privilege tiers +(10 / 100 / 1e3 / 1e4 / 1e5 / 1e6 / unlimited) as anti-spam abuse control — +see `DECISIONS.md` “Defer daily send-quota privilege tiers”. + ## T01 - Define the send and evidence contract ```task @@ -79,7 +83,7 @@ registration without accepting browser identity claims. All 24 tests pass. ```task id: EMAIL-WP-0004-T03 -status: progress +status: done priority: high state_hub_task_id: "13428364-29b1-4e0d-aaf5-c0b254c829b9" ``` @@ -93,11 +97,31 @@ user-engine ingress, SMTP-only egress, and rollback. Done when user-engine can call the cluster-local receiver without possessing SMTP credentials and unrelated workloads cannot send through it. +Done 2026-08-12 after CCR-2026-0010 approval: + +* OpenBao path `platform/workloads/email-connect/transactional` provisioned + (ingest token new; SMTP username/password seeded from live forgejo IONOS + lane — sender ConfigMap uses `forgejo@coulomb.social`). +* Policies applied; ESO child token Secret + `external-secrets/openbao-email-connect-eso-token`; ClusterSecretStore + `openbao-email-connect` Ready; ExternalSecret SecretSynced (3 keys). +* Package applied on railiance01: Deployment Ready 1/1, PVC Bound, probes + green, image digest `sha256:bc7a01b4…`. +* Network: email-connect default-deny + user-engine ingress + SMTP-587 egress; + user-engine-runtime egress allow to email-connect:8080 (net-kingdom + `sso-mfa/k8s/user-engine/runtime.yaml`). +* Smoke: user-engine `/healthz`/`/readyz` 200; unauth `/v1/send` 401; bearer + `template_not_allowed` / `invalid_recipient` 400; default-ns probe + Connection refused. user-engine secret has no SMTP keys. +* Catalog `email-connect-transactional` promoted active/resolvable. +* Follow-up (not T03): mount same ingest token into user-engine for NK outbox + (NK-WP-0024); optional dedicated noreply@ mailbox; T04 failure proof. + ## T04 - Prove invitation delivery failure behavior ```task id: EMAIL-WP-0004-T04 -status: todo +status: done priority: high state_hub_task_id: "34127b3e-f7c6-4ebe-a86c-743ba1b27640" ``` @@ -110,3 +134,18 @@ failure evidence back to NK-WP-0024. Done when deployed invitation and verification flows are observable, idempotent, retry-safe, and conservative about delivery evidence. + +Done 2026-08-12: + +* ProviderError classification: timeout/temporary/unavailable → 503 + `retryable=true`; permanent_rejection → 422 `retryable=false`; no SMTP text + in responses. SQLite suppressions → `recipient_suppressed`. Accept body + carries `evidence_ceiling=provider_accepted`; verification consume/cancel set + `authorization=false`. +* Unit matrix: 9 tests pass (`tests/test_transactional.py`). +* Live matrix on railiance01 (image + `sha256:a9de1994…`): unauth 401, template/invalid/idempotency 400, accept + 202 + duplicate 200, resend 202, bad domain 422 permanent_rejection, + suppression 400; redaction checked. Evidence note for NK-WP-0024: + `docs/EMAIL-WP-0004-T04-failure-evidence.md`. +* Workplan finished when T01–T04 all done (this completes EMAIL-WP-0004).