email-connect/docs/ledger-backup-review-2026-10-08.md

Email ledger backup/recovery candidate — 2026-10-08

EMAIL-WP-0007 implements the independent transactional SQLite recovery scope requested by CSOC-WP-0010-T04/T05. The verification retry release remains live; this candidate does not change that deployment, SMTP/caller credentials or inbox access. The standalone node scripts and four systemd units are prepared for platform admission. No live capture, primary upload or timer activation occurred.

Scope and coherence

The ledger contains deliveries, send_attempts, verification_attempts, verifications and suppressions. Preserve every row, including uncertain outcomes, consumed/cancelled links and failed reservation history. Scanner databases, provider mailboxes/message bodies, SMTP passwords and ingest tokens are separate scopes and are not included. Application database/content backup cannot recover this ledger.

capture-ledger.py is default server dry-run. It checks exact cluster UID, current immutable receiver image, single Ready replica, Bound email-connect-data PVC and /data/email-connect.db configuration. The helper runs UID10001, mounts only the read-only PVC and bounded scratch space, has no environment credential or ServiceAccount token, and must be selected by no-egress NetworkPolicies. No SMTP code is started in the helper.

Under admitted --execute, a guarded replica patch pauses the receiver, waits for its pods to disappear and rejects any other ledger PVC consumer. SQLite's backup API reads committed data including WAL; copying only the main file would miss uncheckpointed records. Integrity checks, exact five-table inventory and schema/ row fingerprints validate the private download. The default 128MiB SQLite ceiling is a reviewed candidate budget, not a promise of unlimited ledger growth.

Finally removes the helper and resumes the unchanged receiver. Replica patches test the original deployment template and expected count; a foreign rollout is never overwritten or silently resumed. Failed cleanup/resume cannot declare a successful capture. A nonblocking lock prevents overlapping captures. SIGTERM attempts the same cleanup; host loss/SIGKILL still needs an explicit replica/helper check. Receipts contain counts/hashes/booleans, never row contents or credentials.

Publication and recovery

backup-ledger.py validates the capture, packages only database.sqlite3 and receipt.json, then seals once using the platform public age recipient. A failed publication preserves one private pending directory and blocks another capture; reviewed retry reuses the ciphertext and object key without another pause. Partial encryption remains for review. Successful publication clears only its own staging directory; historical private pre-release copies remain untouched.

The platform primary_application_archive.py email-connect profile uses platform-pg/application-archives/email-connect/ in the existing primary bucket. Conditional multipart completion, returned version GET and ciphertext size/hash verification gate last-success.json and inventory.json. Independent retrieval requires only a separately retained version/hash/size receipt and recovery caller. No publisher may delete retained archives, access other app prefixes, or hold an age private identity. The dedicated hash-locked platform SDK runtime is reused.

The candidate node worker expects owner-delivered AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in root-owned mode 0600 /etc/email-connect-backup/primary.env, parent mode 0700. A metadata-only node check found that file absent. Existing SMTP custody and the databases backup Secret do not admit this consumer. RPF-WP-0053-T05 tracks its separate prefix/caller, retention/ capacity, independently retained receipt and recovery identity admission. No Secret was fetched/copied, ESO scope widened or provider key minted in this work.

After retrieving ciphertext, the attended custodian runs:

python3 scripts/backup-ledger.py recover --execute \
  --archive /private/retrieved.tar.age --transfer-receipt /private/last-success.json \
  --identity /private/recovery-identity --destination /private/recovered-ledger
python3 scripts/rehearse-ledger.py --execute \
  --recovered /private/recovered-ledger --receipt /private/recovery-proof.json

Recovery verifies exact provider key/version/hash/size evidence, age-decrypts privately, inspects the entire packet before extraction, accepts exactly two regular files and rejects traversal, links, duplicates, extras and oversized entries. It checks the inner image/capture/database/schema/row bindings. Files are mode 0600 and directories mode 0700. A failed drill preserves private evidence; no production database path is modified.

Rehearsal uses the already-cached pinned receiver image under UID10001, read-only root filesystem, bounded tmpfs, dropped capabilities and --network none. It compares schema/all-table row fingerprints before and after opening the current SQLiteDeliveryStore, probes accepted/unknown reservations without a provider, checks invalidated handles/suppression, and verifies the ledger remains unchanged. No SMTP configuration is supplied. Only its own named disposable container is removed on completion/failure. application_restore_verified means this isolated ledger rehearsal; production_restored remains false and recipient inbox/reply operations are not inferred from it.

Historical-point recovery gate

A backup preserves state at its capture point. Operations after that point may include new provider submissions, suppressions, or challenge consumption/ cancellation. Restoring an older point alone can lose a reservation and revive a handle consumed later. Therefore an isolated successful drill never releases production traffic automatically. Incident recovery must first fence callers and outbox replay, keep sending/challenge consumption disabled, reconcile post-capture operations with user-engine/provider evidence and the identity owner, then invalidate or reconcile recovered challenges under an explicit incident decision. Only that owner-approved reconciliation may release the receiver. Unknown outcomes must remain unknown; no automatic resend or inference of non-delivery is allowed. An RPO 24h proposal needs explicit acceptance of this loss window; consider a shorter schedule if the ledger owner requires it. Daily timing is not a guarantee of acceptable authentication recovery.

Scheduling and operating acceptance

Proposed daily 04:15 UTC (+up to 5min jitter), hourly freshness threshold 26h and provisional attended RTO 60min are pending targets. Install reviewed scripts in /opt/email-connect-backup/ and four deploy/backup units disabled. Use root-owned mode 0700 /var/lib/email-connect-backup. Record source and lock hashes and custody delivery reference. Update both capture/rehearsal image pins under a release review.

Publisher local encrypted inventory grows until separate owner retention acts. The platform retention preview can propose 14 daily/4 weekly distinct generations and protects newest plus the exact recovered version/hash, but owns no deletion. Admit capacity/headroom and compatibility with the bucket's documented 30-day lifecycle before enabling a schedule. Connect service/freshness failures to the established evidence owner and prove reception/response. Retain public recovery receipts and the recovery image/key independently of node/workstation/provider loss. Production restore and private-copy expiration are separate owner actions.

Evidence

• Full email suite: 120 passed, zero skips, with the local GreenMail SMTP/IMAP harness; 19 new synthetic backup/recovery cases. No production account, URL or mail send is used by those tests. Existing 39 transactional regression cases pass.
• Actual age synthetic seal/decrypt and packaged rehearsal pass: all five tables, schema and rows preserved; accepted verification replay, unknown verification, invalidated handles, suppression and invitation reservations checked without sending. This is local synthetic evidence, no provider retrieval claim.
• Node default server dry-run passes: current pinned image/PVC, no-egress helper, and unchanged receiver specification. No pause, helper creation or upload.
• All four systemd units pass systemd-analyze verify. Dedicated primary SDK source is already hash-locked and offline-verified by RPF-WP-0053; actual email caller admission remains missing.

Machine-readable evidence: ledger-backup-rehearsal-2026-10-08.json. The actual synthetic packaged drill checked 3 accepted verification operations, 1 unknown verification, 2 invalidated handles, 1 suppression, 1 accepted invitation and 1 unknown invitation reservation; isolated startup/checks took 0.86s. This excludes capture/upload/retrieval/decryption and is not full RTO. Private fixture paths and all row contents are omitted from the evidence file.

T01/T02 implementation/rehearsal can close on this evidence; T03/T04 remain open for actual custody, offsite capture/recovery, post-capture reconciliation contract, monitored operation and founder activation. CSOC-WP-0010 remains active for those and caller/sender/rate-limit, reply/bounce/inbox and cross-owner journey acceptance.