Reserve verification sends durably across retries and restart
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a11538-05ea-7b63-b3c7-a2df5f38898d
This commit is contained in:
parent
73702b7e1a
commit
7258505917
8 changed files with 443 additions and 277 deletions
|
|
@ -1,136 +1,44 @@
|
|||
# email-connect transactional receiver — railiance01
|
||||
|
||||
Deploy the narrow invitation/verification HTTP receiver so **user-engine** can
|
||||
send mail without holding SMTP credentials, and so unrelated workloads cannot
|
||||
reach the send API.
|
||||
The receiver permits user-engine invitation and registration-verification
|
||||
calls, using fixed templates and bounded SMTP/STARTTLS. Its persistent SQLite
|
||||
store contains delivery/challenge evidence; SMTP acceptance is not inbox receipt
|
||||
or authorization. Current release review: ../../../docs/verification-retry-review-2026-10-07.md.
|
||||
|
||||
**Target cluster:** railiance01 only (`KUBECONFIG=~/.kube/config-hosteurope`).
|
||||
Target: railiance01, cluster kube-system UID
|
||||
a553c742-0115-43d4-99a4-a5ca56fe0786. Observe through
|
||||
~/.kube/config-railiance01; change the cluster through ssh railiance01 and the
|
||||
founder's exact live go. config-hosteurope is the retired cluster and must not
|
||||
be used. Read the-custodian/docs/agent-environment-orientation.md first.
|
||||
|
||||
Stable in-cluster URL after apply:
|
||||
Platform owns OpenBao custody, Kubernetes-auth ClusterSecretStore
|
||||
openbao-email-connect and its scoped ESO ServiceAccount (RPF-WP-0045).
|
||||
ExternalSecret email-connect/email-connect-runtime delivers the existing
|
||||
EMAIL_CONNECT_INGEST_TOKEN, EMAIL_CONNECT_SMTP_USERNAME and
|
||||
EMAIL_CONNECT_SMTP_PASSWORD fields. Never read or print Secret objects/annotations.
|
||||
Route needs through warden route find/show email-connect-transactional.
|
||||
|
||||
```text
|
||||
http://email-connect.email-connect.svc.cluster.local:8080
|
||||
```
|
||||
The static ESO token bootstrap and duplicate store template are retired.
|
||||
Do not recreate external-secrets/openbao-email-connect-eso-token or apply a
|
||||
store copy from this repo. Custody and store mutations require the platform lane.
|
||||
|
||||
## Ownership split
|
||||
Stable URL: http://email-connect.email-connect.svc.cluster.local:8080.
|
||||
Default-deny NetworkPolicy permits only user-engine:8080 ingress, DNS and public
|
||||
SMTP:587 egress. No additional caller admission is part of an image correction.
|
||||
ConfigMap owns host, port, sender and portal URL; no provider credentials belong
|
||||
in HTTP requests, Git, chat, State Hub or test fixtures.
|
||||
|
||||
| Concern | Owner | Where |
|
||||
| --- | --- | --- |
|
||||
| Package, manifests, probes, NetworkPolicy, rollback | `email-connect` | this directory |
|
||||
| OpenBao path, ESO role/policy, ClusterSecretStore | `railiance-platform` | CCR-2026-0010, `openbao/policies/` |
|
||||
| Credential routing catalog | `ops-warden` | `email-connect-transactional` |
|
||||
After reviewing current live manifests and CPU headroom, perform client diff
|
||||
and server dry-run for the exact digest change. Preserve live resource ownership;
|
||||
apply only that reviewed image change from this owner repo after founder go.
|
||||
Observe Ready/probes and an isolated no-send authenticated rejection before
|
||||
calling the release successful. Do not recreate secrets as a workaround.
|
||||
|
||||
Never paste SMTP passwords, ingest tokens, or OpenBao tokens into Git, State
|
||||
Hub, workplans, logs, or chat. Route first:
|
||||
Before the verification correction, capture SQLite with its backup API into a
|
||||
private mode-0600 artifact. Keep PVC evidence intact. Do not run rollout undo
|
||||
with sending enabled: an older image ignores verification_attempts and may
|
||||
resend an uncertain operation. Stop the receiver first and reconcile unknown
|
||||
reservations; roll forward with the invariant preserved where possible.
|
||||
|
||||
```bash
|
||||
warden route find "email-connect SMTP transactional" --json
|
||||
warden route show email-connect-transactional --json
|
||||
```
|
||||
|
||||
## Runtime secrets (OpenBao → ESO)
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| KV path | `platform/workloads/email-connect/transactional` |
|
||||
| Fields | `EMAIL_CONNECT_INGEST_TOKEN`, `EMAIL_CONNECT_SMTP_USERNAME`, `EMAIL_CONNECT_SMTP_PASSWORD` |
|
||||
| Read / ESO policy | `external-secrets-email-connect` (+ `workload-kv-read-email-connect-transactional` for operator OIDC) |
|
||||
| ClusterSecretStore | `openbao-email-connect` (namespace-limited; token auth like activity-core/reuse) |
|
||||
| ESO token Secret | `external-secrets/openbao-email-connect-eso-token` (via `openbao-eso-token-apply.sh`) |
|
||||
| ExternalSecret | `email-connect/email-connect-runtime` → Secret `email-connect-runtime` |
|
||||
| ops-warden catalog | `email-connect-transactional` (draft until provisioned) |
|
||||
|
||||
Non-secret config (host, port, sender, portal URL) lives in ConfigMap
|
||||
`email-connect-config`.
|
||||
|
||||
### Operator provision (attended, after CCR approval)
|
||||
|
||||
```bash
|
||||
# Values from mode-0600 files only; shred after.
|
||||
bao kv put platform/workloads/email-connect/transactional \
|
||||
EMAIL_CONNECT_INGEST_TOKEN=@/path/to/ingest.token \
|
||||
EMAIL_CONNECT_SMTP_USERNAME=@/path/to/smtp.user \
|
||||
EMAIL_CONNECT_SMTP_PASSWORD=@/path/to/smtp.pass
|
||||
```
|
||||
|
||||
user-engine must receive the **same** ingest token under its own runtime secret
|
||||
path so it can call `Authorization: Bearer …` without ever seeing SMTP fields.
|
||||
|
||||
## Network posture
|
||||
|
||||
- **Default deny** ingress and egress in `email-connect`.
|
||||
- **Ingress:** only pods labeled `app.kubernetes.io/name=user-engine` in
|
||||
namespace `user-engine`, TCP 8080.
|
||||
- **Egress:** kube-system DNS + TCP 587 to the public internet (IONOS
|
||||
STARTTLS). No cluster-lateral egress and no general HTTPS.
|
||||
|
||||
Bearer auth on `/v1/*` is the second gate; NetworkPolicy is the outer gate.
|
||||
|
||||
## Apply
|
||||
|
||||
Prerequisites:
|
||||
|
||||
1. CCR-2026-0010 approved.
|
||||
2. KV secret provisioned at `platform/workloads/email-connect/transactional`
|
||||
(fields non-empty; do not log values).
|
||||
3. ESO delivery token + ClusterSecretStore on railiance01 (token pattern matches
|
||||
`openbao-activity-core` / `openbao-reuse`, not Kubernetes auth yet).
|
||||
4. Image available on the node/registry at the digest pinned in `deployment.yaml`.
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=~/.kube/config-hosteurope
|
||||
|
||||
# 1) Policy + limited ESO child token → external-secrets Secret
|
||||
./deploy/k8s/railiance/openbao-eso-token-apply.sh
|
||||
|
||||
# 2) Namespace-scoped ClusterSecretStore (platform add-on copy)
|
||||
kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
|
||||
|
||||
# 3) Optional: load a freshly built image when the registry is not yet updated
|
||||
# docker build -f Containerfile -t forgejo.coulomb.social/coulomb/email-connect:local .
|
||||
# docker save … | ssh railiance01 sudo k3s ctr -n k8s.io images import -
|
||||
|
||||
# 4) Package
|
||||
kubectl apply -k deploy/k8s/railiance
|
||||
kubectl -n email-connect rollout status deployment/email-connect
|
||||
kubectl -n email-connect get pods,svc,networkpolicy,externalsecret,secret
|
||||
```
|
||||
|
||||
Expect ExternalSecret `SecretSynced` and a Ready pod. If the Secret is missing,
|
||||
do not create a bootstrap Secret with values from chat — fix OpenBao/ESO.
|
||||
|
||||
## Smoke (no SMTP proof yet)
|
||||
|
||||
From a **user-engine** pod (or a temporary pod in that namespace with the same
|
||||
labels — NetworkPolicy will otherwise drop the packet):
|
||||
|
||||
```bash
|
||||
# Health only (no auth)
|
||||
kubectl -n user-engine exec deploy/user-engine -- \
|
||||
wget -qO- http://email-connect.email-connect.svc.cluster.local:8080/healthz
|
||||
|
||||
# Auth check: wrong token must be 401; correct token is exercised by T04.
|
||||
```
|
||||
|
||||
Negative checks:
|
||||
|
||||
- A pod in another namespace cannot reach port 8080 (connection timeout/reset).
|
||||
- user-engine runtime must not contain `EMAIL_CONNECT_SMTP_*` keys.
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
kubectl -n email-connect rollout undo deployment/email-connect
|
||||
kubectl -n email-connect rollout status deployment/email-connect
|
||||
```
|
||||
|
||||
`revisionHistoryLimit: 5` keeps prior digests. SQLite on the PVC is additive;
|
||||
rolling back does not clear delivery/idempotency rows (by design).
|
||||
|
||||
## Done criteria (EMAIL-WP-0004-T03)
|
||||
|
||||
- [ ] user-engine calls the cluster-local receiver without possessing SMTP credentials
|
||||
- [ ] Unrelated workloads cannot open TCP 8080 on the receiver
|
||||
- [ ] SMTP password and ingest token exist only in OpenBao + the scoped K8s Secret
|
||||
- [ ] Probes green; NetworkPolicy default-deny + SMTP-only egress applied
|
||||
- [ ] Documented rollback path works against the previous revision
|
||||
Local regression harness: tests/harness/README.md. It uses synthetic addresses
|
||||
and loopback SMTP/IMAP only; no production mailbox or message send is a test.
|
||||
|
|
|
|||
|
|
@ -1,30 +1,5 @@
|
|||
# Template copy of the railiance-platform add-on store. Prefer applying from:
|
||||
# ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
|
||||
#
|
||||
# Do not apply until CCR-2026-0010 is approved and Secret
|
||||
# external-secrets/openbao-email-connect-eso-token exists
|
||||
# (./openbao-eso-token-apply.sh).
|
||||
---
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ClusterSecretStore
|
||||
metadata:
|
||||
name: openbao-email-connect
|
||||
labels:
|
||||
app.kubernetes.io/part-of: railiance-gitops
|
||||
railiance-platform/component: external-secrets
|
||||
app.kubernetes.io/name: email-connect
|
||||
spec:
|
||||
provider:
|
||||
vault:
|
||||
# In-cluster OpenBao on railiance01 (not the public bao.coulomb.social UI).
|
||||
server: http://openbao.openbao.svc:8200
|
||||
path: platform
|
||||
version: v2
|
||||
auth:
|
||||
tokenSecretRef:
|
||||
name: openbao-email-connect-eso-token
|
||||
namespace: external-secrets
|
||||
key: token
|
||||
conditions:
|
||||
- namespaces:
|
||||
- email-connect
|
||||
# RETIRED COPY — intentionally contains no Kubernetes resource.
|
||||
# The platform owns the Kubernetes-auth store and its scoped ServiceAccount:
|
||||
# railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-email-connect.clustersecretstore.yaml
|
||||
# Consult RPF-WP-0045 and the authoritative platform declarations.
|
||||
# Do not recreate external-secrets/openbao-email-connect-eso-token.
|
||||
|
|
|
|||
|
|
@ -1,83 +1,5 @@
|
|||
#!/usr/bin/env bash
|
||||
# Mint a read-limited OpenBao token and store it on railiance01 for
|
||||
# ClusterSecretStore openbao-email-connect / ExternalSecret email-connect-runtime.
|
||||
#
|
||||
# Policies (files in railiance-platform/openbao/policies/):
|
||||
# external-secrets-email-connect
|
||||
# workload-kv-read-email-connect-transactional (optional operator OIDC sibling)
|
||||
#
|
||||
# Does not print secret values. Requires an attended operator OpenBao token.
|
||||
# CCR-2026-0010 must be approved before production use.
|
||||
# Retired after RPF-WP-0045: never recreate the static ESO token Secret.
|
||||
set -euo pipefail
|
||||
|
||||
DEFAULT_POLICIES="external-secrets-email-connect"
|
||||
POLICIES="${OPENBAO_EMAIL_CONNECT_POLICIES:-$DEFAULT_POLICIES}"
|
||||
POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}"
|
||||
BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}"
|
||||
RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}"
|
||||
SECRET_NAME="${OPENBAO_EMAIL_CONNECT_ESO_SECRET:-openbao-email-connect-eso-token}"
|
||||
SECRET_NS="${OPENBAO_EMAIL_CONNECT_ESO_NAMESPACE:-external-secrets}"
|
||||
TTL="${OPENBAO_EMAIL_CONNECT_ESO_TTL:-768h}"
|
||||
|
||||
if ! command -v bao >/dev/null 2>&1; then
|
||||
echo "ERROR: bao CLI not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v kubectl >/dev/null 2>&1; then
|
||||
echo "ERROR: kubectl not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OpenBao addr: $BAO_ADDR"
|
||||
echo "Policies: $POLICIES"
|
||||
echo "K8s secret: $SECRET_NS/$SECRET_NAME (railiance01)"
|
||||
|
||||
if [[ -n "${BAO_TOKEN:-}" ]]; then
|
||||
:
|
||||
elif [[ -n "${OPENBAO_TOKEN_FILE:-}" && -f "${OPENBAO_TOKEN_FILE}" ]]; then
|
||||
BAO_TOKEN="$(head -n 1 "${OPENBAO_TOKEN_FILE}")"
|
||||
else
|
||||
read -r -s -p "OpenBao operator token: " BAO_TOKEN
|
||||
echo >&2
|
||||
fi
|
||||
if [[ -z "${BAO_TOKEN:-}" ]]; then
|
||||
echo "ERROR: empty OpenBao token" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export BAO_ADDR BAO_TOKEN
|
||||
|
||||
health="$(curl -fsS "$BAO_ADDR/v1/sys/health")"
|
||||
if echo "$health" | grep -q '"sealed":true'; then
|
||||
echo "ERROR: OpenBao at $BAO_ADDR reports sealed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for policy in $POLICIES; do
|
||||
policy_file="$POLICY_DIR/${policy}.hcl"
|
||||
if [[ -f "$policy_file" ]]; then
|
||||
bao policy write "$policy" "$policy_file"
|
||||
echo "policy written: $policy"
|
||||
else
|
||||
echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
# Child token: renewable, orphan so operator logout does not revoke delivery.
|
||||
# shellcheck disable=SC2086
|
||||
token_json="$(bao token create -policy="$(echo $POLICIES | tr ' ' ',')" -ttl="$TTL" -renewable=true -orphan -format=json)"
|
||||
child_token="$(printf '%s' "$token_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["auth"]["client_token"])')"
|
||||
if [[ -z "$child_token" || ${#child_token} -lt 8 ]]; then
|
||||
echo "ERROR: failed to mint child token" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "minted child token length=${#child_token} (value not printed)"
|
||||
|
||||
export KUBECONFIG="$RAILIANCE01_KUBECONFIG"
|
||||
kubectl -n "$SECRET_NS" create secret generic "$SECRET_NAME" \
|
||||
--from-literal=token="$child_token" \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
unset child_token BAO_TOKEN
|
||||
echo "Secret $SECRET_NS/$SECRET_NAME applied on railiance01."
|
||||
echo "Next: apply ClusterSecretStore openbao-email-connect, then email-connect package."
|
||||
printf '%s\n' 'Retired: openbao-email-connect uses platform-owned Kubernetes auth. See deploy/k8s/railiance/README.md.' >&2
|
||||
exit 1
|
||||
|
|
|
|||
35
docs/verification-retry-review-2026-10-07.md
Normal file
35
docs/verification-retry-review-2026-10-07.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Verification retry correction — 2026-10-07
|
||||
|
||||
Source contract: POST /v1/registration-verifications keeps its 202 JSON
|
||||
{request_id, accepted: true} response. Repeated tenant/client/registration
|
||||
operations return the same receipt after provider acceptance without resending.
|
||||
The correlation ID may change; changing recipient, username or display name
|
||||
for the same operation is rejected. A new registration is a separate operation.
|
||||
|
||||
A durable reservation precedes SMTP and keeps only the challenge digest. An
|
||||
unknown send outcome returns redacted 422 delivery_outcome_unknown,
|
||||
retryable=false, and keeps its usable challenge until expiry. Definitive
|
||||
retryable non-acceptance returns 503 and atomically cancels the failed challenge
|
||||
before retry. Permanent rejection stays terminal. Existing pre-upgrade
|
||||
challenges have no recorded SMTP outcome, so their retries fail conservatively
|
||||
while existing consume/cancel links keep working. There is no automatic resend
|
||||
or expired-challenge renewal for the same registration.
|
||||
|
||||
39 transactional tests passed. The full repository suite with the pinned
|
||||
GreenMail harness running passed all 101 tests with zero skips. The revised
|
||||
real SMTP/IMAP verification journey observed exactly one mailbox message after
|
||||
store restart and HTTP retry. Harness container/network were removed afterward.
|
||||
Synthetic fixture accounts only. Packaged image receipt follows separately.
|
||||
|
||||
Release scope is the email-connect container image only, reusing the live PVC,
|
||||
caller token, ConfigMap, ServiceAccount and NetworkPolicies. It does not authorize
|
||||
another caller, sender, provider or credential delivery. Founder live go remains
|
||||
separate from code preparation under the-custodian/docs/agent-environment-orientation.md
|
||||
section 4. The pre-release SQLite backup must use the SQLite backup API while
|
||||
consistent, with output kept private; never emit its rows.
|
||||
|
||||
Rollback: block sending first (scale receiver to zero, preserving its PVC),
|
||||
then compare/reconcile reservation outcomes. Old code ignores verification
|
||||
reservations: an unrestricted rollout undo can duplicate verification mail.
|
||||
Prefer rolling forward with the retry invariant preserved. Additive schema can
|
||||
be read by the old image, but schema compatibility is not delivery safety.
|
||||
|
|
@ -57,8 +57,77 @@ class SQLiteDeliveryStore:
|
|||
self.db.execute(
|
||||
"CREATE TABLE IF NOT EXISTS send_attempts (event_id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, state TEXT NOT NULL)"
|
||||
)
|
||||
self.db.execute("""CREATE TABLE IF NOT EXISTS verification_attempts (
|
||||
operation_key TEXT PRIMARY KEY, fingerprint TEXT NOT NULL,
|
||||
request_id TEXT NOT NULL, state TEXT NOT NULL,
|
||||
error TEXT, retryable INTEGER, provider_ref TEXT
|
||||
)""")
|
||||
self.db.commit()
|
||||
|
||||
def reserve_verification(self, payload, handle_hash, expires_at):
|
||||
# Registration identity, not a tracing ID, defines the logical operation.
|
||||
identity = [payload[key] for key in ("tenant", "client_id", "registration_id")]
|
||||
operation_key = hashlib.sha256(json.dumps(identity).encode()).hexdigest()
|
||||
binding = {key: payload.get(key) for key in (
|
||||
"registration_id", "normalized_email", "preferred_username",
|
||||
"client_id", "tenant", "display_name",
|
||||
)}
|
||||
fingerprint = hashlib.sha256(
|
||||
json.dumps(binding, sort_keys=True, separators=(",", ":")).encode()
|
||||
).hexdigest()
|
||||
with self.lock, self.db:
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
row = self.db.execute(
|
||||
"SELECT fingerprint,request_id,state,error,retryable "
|
||||
"FROM verification_attempts WHERE operation_key=?", (operation_key,),
|
||||
).fetchone()
|
||||
if not row:
|
||||
legacy = self.db.execute(
|
||||
"SELECT 1 FROM verifications WHERE registration_id=? AND client_id=? AND tenant=? LIMIT 1",
|
||||
(payload["registration_id"], payload["client_id"], payload["tenant"]),
|
||||
).fetchone()
|
||||
if legacy:
|
||||
# Pre-upgrade challenges have no trustworthy send outcome.
|
||||
# Preserve their links, but do not silently issue another.
|
||||
raise ProviderError("delivery_outcome_unknown", retryable=False)
|
||||
if row:
|
||||
if row[0] != fingerprint:
|
||||
raise ValueError("idempotency_payload_mismatch")
|
||||
if row[2] == "accepted":
|
||||
return row[1], False
|
||||
if row[2] == "in_flight":
|
||||
raise ProviderError("delivery_outcome_unknown", retryable=False)
|
||||
if not row[4]:
|
||||
raise ProviderError(row[3], retryable=False)
|
||||
request_id = self._insert_verification(payload, handle_hash, expires_at)
|
||||
self.db.execute(
|
||||
"INSERT OR REPLACE INTO verification_attempts VALUES (?,?,?,?,NULL,NULL,NULL)",
|
||||
(operation_key, fingerprint, request_id, "in_flight"),
|
||||
)
|
||||
return request_id, True
|
||||
|
||||
def verification_accepted(self, request_id, reference):
|
||||
with self.lock, self.db:
|
||||
self.db.execute(
|
||||
"UPDATE verification_attempts SET state='accepted',provider_ref=? "
|
||||
"WHERE request_id=? AND state='in_flight'", (reference, request_id),
|
||||
)
|
||||
|
||||
def verification_failed(self, request_id, error):
|
||||
# Called only when non-acceptance is known. Unknown outcomes retain the
|
||||
# reservation and the delivered handle remains usable until expiry.
|
||||
with self.lock, self.db:
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
self.db.execute(
|
||||
"UPDATE verifications SET canceled_at=? WHERE request_id=?",
|
||||
(datetime.now(timezone.utc).isoformat(), request_id),
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE verification_attempts SET state='failed',error=?,retryable=? "
|
||||
"WHERE request_id=? AND state='in_flight'",
|
||||
(error.code, int(error.retryable), request_id),
|
||||
)
|
||||
|
||||
def reserve_send(self, event_id, fingerprint):
|
||||
with self.lock:
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
|
|
@ -132,29 +201,33 @@ class SQLiteDeliveryStore:
|
|||
def create_verification(
|
||||
self, payload: dict, handle_hash: str, expires_at: str
|
||||
) -> str:
|
||||
with self.lock, self.db:
|
||||
return self._insert_verification(payload, handle_hash, expires_at)
|
||||
|
||||
def _insert_verification(self, payload, handle_hash, expires_at):
|
||||
request_id = f"vrq_{secrets.token_hex(12)}"
|
||||
with self.db:
|
||||
self.db.execute(
|
||||
"""INSERT INTO verifications
|
||||
(request_id, handle_hash, registration_id, email, username,
|
||||
client_id, tenant, display_name, expires_at, consumed_at, canceled_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""",
|
||||
(
|
||||
request_id,
|
||||
handle_hash,
|
||||
payload["registration_id"],
|
||||
payload["normalized_email"],
|
||||
payload["preferred_username"],
|
||||
payload["client_id"],
|
||||
payload["tenant"],
|
||||
payload.get("display_name"),
|
||||
expires_at,
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
"""INSERT INTO verifications
|
||||
(request_id, handle_hash, registration_id, email, username,
|
||||
client_id, tenant, display_name, expires_at, consumed_at, canceled_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,NULL,NULL)""",
|
||||
(
|
||||
request_id,
|
||||
handle_hash,
|
||||
payload["registration_id"],
|
||||
payload["normalized_email"],
|
||||
payload["preferred_username"],
|
||||
payload["client_id"],
|
||||
payload["tenant"],
|
||||
payload.get("display_name"),
|
||||
expires_at,
|
||||
),
|
||||
)
|
||||
return request_id
|
||||
|
||||
def consume_verification(self, handle_hash: str) -> dict:
|
||||
with self.lock, self.db:
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
row = self.db.execute(
|
||||
"SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,)
|
||||
).fetchone()
|
||||
|
|
@ -178,6 +251,7 @@ class SQLiteDeliveryStore:
|
|||
|
||||
def cancel_verification(self, handle_hash: str) -> dict:
|
||||
with self.lock, self.db:
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
row = self.db.execute(
|
||||
"SELECT * FROM verifications WHERE handle_hash=?", (handle_hash,)
|
||||
).fetchone()
|
||||
|
|
@ -456,7 +530,7 @@ class TransactionalApplication:
|
|||
"tenant",
|
||||
"correlation_id",
|
||||
)
|
||||
if any(not payload.get(key) for key in required) or not _valid_address(
|
||||
if any(not isinstance(payload.get(key), str) or not payload[key] for key in required) or not _valid_address(
|
||||
str(payload["normalized_email"])
|
||||
):
|
||||
raise ValueError("invalid_verification_request")
|
||||
|
|
@ -465,16 +539,33 @@ class TransactionalApplication:
|
|||
raise ValueError("recipient_suppressed")
|
||||
handle = secrets.token_urlsafe(32)
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(minutes=30)
|
||||
request_id = self.store.create_verification(
|
||||
request_id, send = self.store.reserve_verification(
|
||||
payload, hashlib.sha256(handle.encode()).hexdigest(), expires_at.isoformat()
|
||||
)
|
||||
self.provider.send(
|
||||
email,
|
||||
"Verify your NetKingdom registration",
|
||||
f"Continue securely at {self.portal_url}/registration/verify?handle={handle}\n"
|
||||
f"Cancel this request at {self.portal_url}/registration/cancel?handle={handle}\n"
|
||||
"These links expire in 30 minutes.\n",
|
||||
)
|
||||
if send:
|
||||
try:
|
||||
reference = self.provider.send(
|
||||
email,
|
||||
"Verify your NetKingdom registration",
|
||||
f"Continue securely at {self.portal_url}/registration/verify?handle={handle}\n"
|
||||
f"Cancel this request at {self.portal_url}/registration/cancel?handle={handle}\n"
|
||||
"These links expire in 30 minutes.\n",
|
||||
)
|
||||
self.store.verification_accepted(request_id, reference)
|
||||
except ProviderError as exc:
|
||||
definitive = getattr(
|
||||
exc, "definitive",
|
||||
exc.code in {"temporary_deferral", "permanent_rejection"},
|
||||
)
|
||||
if not definitive:
|
||||
raise ProviderError("delivery_outcome_unknown", retryable=False) from None
|
||||
try:
|
||||
self.store.verification_failed(request_id, exc)
|
||||
except Exception:
|
||||
raise ProviderError("delivery_outcome_unknown", retryable=False) from None
|
||||
raise
|
||||
except Exception:
|
||||
raise ProviderError("delivery_outcome_unknown", retryable=False) from None
|
||||
return self._json(
|
||||
start_response,
|
||||
HTTPStatus.ACCEPTED,
|
||||
|
|
|
|||
|
|
@ -206,18 +206,18 @@ def test_verification_mail_is_delivered_and_carries_no_authorization(tmp_path):
|
|||
recipient = harness.address("verification delivered")
|
||||
app = application(tmp_path)
|
||||
|
||||
status, body = invoke(
|
||||
app,
|
||||
{
|
||||
"registration_id": "reg-1",
|
||||
"normalized_email": recipient,
|
||||
"preferred_username": "harness-user",
|
||||
"client_id": "portal",
|
||||
"tenant": "harness",
|
||||
"correlation_id": "corr-1",
|
||||
},
|
||||
path="/v1/registration-verifications",
|
||||
)
|
||||
payload = {
|
||||
"registration_id": "reg-1", "normalized_email": recipient,
|
||||
"preferred_username": "harness-user", "client_id": "portal",
|
||||
"tenant": "harness", "correlation_id": "corr-1",
|
||||
}
|
||||
path = "/v1/registration-verifications"
|
||||
status, body = invoke(app, payload, path=path)
|
||||
app.store.db.close()
|
||||
# A real SMTP/IMAP journey includes service restart and a changed trace ID.
|
||||
app = application(tmp_path)
|
||||
replay_status, replay_body = invoke(app, {**payload, "correlation_id": "corr-retry"}, path=path)
|
||||
assert replay_status.startswith("202") and replay_body == body
|
||||
|
||||
assert status.startswith("2"), (status, body)
|
||||
messages = delivered_messages(recipient)
|
||||
|
|
|
|||
|
|
@ -495,3 +495,158 @@ def test_disconnect_after_smtp_acceptance_never_releases_reservation(
|
|||
assert body["error"] == "delivery_outcome_unknown"
|
||||
assert "private" not in str(body)
|
||||
assert SMTP.sends == 1
|
||||
|
||||
|
||||
VERIFICATION_PATH = "/v1/registration-verifications"
|
||||
|
||||
|
||||
def verification_request(**changes):
|
||||
return {
|
||||
"registration_id": "reg-retry", "normalized_email": "person@example.test",
|
||||
"preferred_username": "person", "client_id": "coulomb-social",
|
||||
"tenant": "tenant:coulomb", "correlation_id": "corr-retry", **changes,
|
||||
}
|
||||
|
||||
|
||||
def verification_app(path, provider):
|
||||
store = SQLiteDeliveryStore(str(path))
|
||||
return TransactionalApplication(store, provider, "opaque", "https://users.example")
|
||||
|
||||
|
||||
def test_verification_retry_after_restart_returns_same_receipt(tmp_path):
|
||||
path = tmp_path / "mail.db"
|
||||
provider = Provider()
|
||||
app = verification_app(path, provider)
|
||||
status, receipt = invoke(app, verification_request(), path=VERIFICATION_PATH)
|
||||
assert status.startswith("202")
|
||||
handle = provider.calls[0][2].split("handle=", 1)[1].splitlines()[0]
|
||||
app.store.db.close()
|
||||
app = verification_app(path, provider)
|
||||
# Correlation IDs can change between HTTP retries; they are tracing metadata.
|
||||
assert invoke(app, verification_request(correlation_id="corr-new"), path=VERIFICATION_PATH)[1] == receipt
|
||||
assert len(provider.calls) == 1
|
||||
assert app.store.db.execute("SELECT count(*) FROM verifications").fetchone()[0] == 1
|
||||
assert handle not in json.dumps(receipt)
|
||||
assert handle.encode() not in path.read_bytes()
|
||||
assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200")
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == receipt
|
||||
assert len(provider.calls) == 1 # consumption does not turn a retry into a resend
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", [
|
||||
{"normalized_email": "other@example.test"}, {"preferred_username": "other"},
|
||||
{"display_name": "changed"},
|
||||
])
|
||||
def test_verification_binding_cannot_change_on_retry(tmp_path, change):
|
||||
provider = Provider()
|
||||
app = verification_app(tmp_path / "mail.db", provider)
|
||||
invoke(app, verification_request(), path=VERIFICATION_PATH)
|
||||
status, body = invoke(app, verification_request(**change), path=VERIFICATION_PATH)
|
||||
assert status.startswith("400") and body["error"] == "idempotency_payload_mismatch"
|
||||
assert len(provider.calls) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("error", [
|
||||
OSError("private smtp credential"), ProviderError("provider_timeout", retryable=True),
|
||||
])
|
||||
def test_verification_uncertain_send_is_not_repeated_after_restart(tmp_path, error):
|
||||
provider = FailingProvider(error)
|
||||
path = tmp_path / "mail.db"
|
||||
app = verification_app(path, provider)
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == {
|
||||
"error": "delivery_outcome_unknown", "retryable": False,
|
||||
}
|
||||
app.store.db.close()
|
||||
app = verification_app(path, provider)
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown"
|
||||
assert provider.calls == 1
|
||||
assert app.store.db.execute("SELECT count(*) FROM verifications").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_verification_acceptance_record_failure_does_not_resend(tmp_path, monkeypatch):
|
||||
provider = Provider()
|
||||
app = verification_app(tmp_path / "mail.db", provider)
|
||||
def fail(*args):
|
||||
raise OSError("private database path")
|
||||
monkeypatch.setattr(app.store, "verification_accepted", fail)
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown"
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown"
|
||||
assert len(provider.calls) == 1
|
||||
handle = provider.calls[0][2].split("handle=", 1)[1].splitlines()[0]
|
||||
# A message accepted before evidence persistence failed still has a usable link.
|
||||
assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200")
|
||||
|
||||
|
||||
def test_verification_definitive_deferral_allows_retry_and_cancels_old_handle(tmp_path):
|
||||
failing = FailingProvider(ProviderError("temporary_deferral", retryable=True))
|
||||
app = verification_app(tmp_path / "mail.db", failing)
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1] == {
|
||||
"error": "temporary_deferral", "retryable": True,
|
||||
}
|
||||
old = app.store.db.execute("SELECT handle_hash,canceled_at FROM verifications").fetchone()
|
||||
assert old[1]
|
||||
with pytest.raises(ValueError, match="verification_invalid"):
|
||||
app.store.consume_verification(old[0])
|
||||
provider = Provider()
|
||||
app.provider = provider
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[0].startswith("202")
|
||||
assert len(provider.calls) == 1
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[0].startswith("202")
|
||||
assert len(provider.calls) == 1
|
||||
|
||||
|
||||
def test_verification_permanent_failure_stays_terminal(tmp_path):
|
||||
provider = FailingProvider(ProviderError("permanent_rejection", retryable=False))
|
||||
app = verification_app(tmp_path / "mail.db", provider)
|
||||
for _ in range(2):
|
||||
status, body = invoke(app, verification_request(), path=VERIFICATION_PATH)
|
||||
assert status.startswith("422") and body["error"] == "permanent_rejection"
|
||||
assert provider.calls == 1
|
||||
|
||||
|
||||
def test_verification_concurrent_connections_reserve_before_provider_call(tmp_path):
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from threading import Event
|
||||
entered, release = Event(), Event()
|
||||
class BlockingProvider(Provider):
|
||||
def send(self, *args):
|
||||
self.calls.append(args)
|
||||
entered.set()
|
||||
assert release.wait(5)
|
||||
return "message:1"
|
||||
provider = BlockingProvider()
|
||||
first = verification_app(tmp_path / "mail.db", provider)
|
||||
second = verification_app(tmp_path / "mail.db", provider)
|
||||
with ThreadPoolExecutor(max_workers=1) as pool:
|
||||
pending = pool.submit(invoke, first, verification_request(), path=VERIFICATION_PATH)
|
||||
try:
|
||||
assert entered.wait(5)
|
||||
assert invoke(second, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown"
|
||||
finally:
|
||||
release.set()
|
||||
receipt = pending.result()[1]
|
||||
assert invoke(second, verification_request(), path=VERIFICATION_PATH)[1] == receipt
|
||||
assert len(provider.calls) == 1
|
||||
|
||||
|
||||
def test_verification_tenant_and_client_operations_are_separate(tmp_path):
|
||||
provider = Provider()
|
||||
app = verification_app(tmp_path / "mail.db", provider)
|
||||
for changes in ({}, {"tenant": "tenant:other"}, {"client_id": "other-client"}):
|
||||
assert invoke(app, verification_request(**changes), path=VERIFICATION_PATH)[0].startswith("202")
|
||||
assert len(provider.calls) == 3
|
||||
|
||||
|
||||
def test_legacy_verification_is_preserved_without_resending(tmp_path):
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
provider = Provider()
|
||||
app = verification_app(tmp_path / "mail.db", provider)
|
||||
handle = "legacy-handle-" + "x" * 32
|
||||
app.store.create_verification(
|
||||
verification_request(), hashlib.sha256(handle.encode()).hexdigest(),
|
||||
(datetime.now(timezone.utc) + timedelta(minutes=30)).isoformat(),
|
||||
)
|
||||
assert invoke(app, verification_request(), path=VERIFICATION_PATH)[1]["error"] == "delivery_outcome_unknown"
|
||||
assert provider.calls == []
|
||||
assert invoke(app, {"handle": handle}, path=VERIFICATION_PATH + "/consume")[0].startswith("200")
|
||||
|
|
|
|||
80
workplans/EMAIL-WP-0006-verification-retry-safety.md
Normal file
80
workplans/EMAIL-WP-0006-verification-retry-safety.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
---
|
||||
id: EMAIL-WP-0006
|
||||
type: workplan
|
||||
title: "Make registration verification delivery retry safe"
|
||||
domain: infotech
|
||||
repo: email-connect
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: netkingdom
|
||||
created: "2026-10-07"
|
||||
updated: "2026-10-07"
|
||||
---
|
||||
|
||||
Repeated verification requests currently mint and send another challenge, even
|
||||
when SMTP may already have accepted the previous message. CSOC-WP-0010-T04
|
||||
identified this on source 73702b7 using synthetic requests. This workplan owns
|
||||
that correction; it does not admit new callers or change SMTP sender custody.
|
||||
|
||||
## T01 — Reserve verification operations durably
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0006-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Implemented additive SQLite verification_attempts. Tenant/client/registration
|
||||
identity defines the operation; immutable applicant fields define its fingerprint.
|
||||
Correlation ID is tracing metadata and can change on retry. Reserve challenge
|
||||
and operation atomically before SMTP. Accepted replay returns the same request
|
||||
receipt. Unknown outcomes are terminal for automatic retry; only definitive
|
||||
retryable non-acceptance permits a new challenge, invalidating the old one.
|
||||
Legacy challenges remain usable and conservatively block another send.
|
||||
Consume/cancel use SQLite write transactions across independent connections.
|
||||
Plaintext handles remain only in mail, never in SQLite or HTTP receipts.
|
||||
|
||||
## T02 — Prove failure, restart and SMTP mailbox behavior
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0006-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
39 transactional cases pass, including separate-connection concurrency,
|
||||
restart, post-acceptance evidence failure, binding mutation, legacy challenge,
|
||||
definitive deferral/permanent refusal, consumption and cancellation.
|
||||
The existing real loopback SMTP/IMAP verification journey now retries after a
|
||||
store restart and requires exactly one mailbox message. Full repository suite with the pinned GreenMail harness running: 101 passed,
|
||||
zero skips. The synthetic SMTP/IMAP journey confirmed one message after restart
|
||||
and retry. Harness container/network were removed after validation.
|
||||
|
||||
## T03 — Release the immutable candidate under a separate live gate
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0006-T03
|
||||
status: progress
|
||||
priority: high
|
||||
```
|
||||
|
||||
Prepare immutable image, live digest/headroom observation and exact image-only
|
||||
diff; capture SQLite consistently before release; use deployment owner path and
|
||||
preserve delivery/challenge records. Founder activation is required for this
|
||||
new live change. Do not rollback to old verification code with sending enabled:
|
||||
it ignores the new reservation table. Stop sending first and reconcile unknown
|
||||
operations. Platform sender/caller admission remains CSOC-WP-0010-T04/T05.
|
||||
|
||||
## T04 — Retire the obsolete static ESO bootstrap
|
||||
|
||||
```task
|
||||
id: EMAIL-WP-0006-T04
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Owner inbox records RPF-WP-0045 Kubernetes-auth recovery and retirement request
|
||||
9d63cf96-333c-4142-a517-aa2633ae52eb. Bootstrap now fails closed without touching
|
||||
credentials or Kubernetes; the obsolete template contains only its retirement
|
||||
pointer. Documented package apply does not recreate the platform-owned store.
|
||||
No live custody change was made.
|
||||
Loading…
Add table
Add a link
Reference in a new issue