email-connect/docs/ledger-backup-review-2026-10-08.md

Email ledger backup/recovery candidate — 2026-10-08

Historical candidate review. The admitted first run and monitoring proof on 2026-10-09 are recorded below; scheduling and independent custody remain open.

EMAIL-WP-0007 implements the independent transactional SQLite recovery scope requested by CSOC-WP-0010-T04/T05. The verification retry release remains live; this candidate does not change that deployment, SMTP/caller credentials or inbox access. The standalone node scripts and four systemd units are prepared for platform admission. No live capture, primary upload or timer activation occurred.

Scope and coherence

The ledger contains deliveries, send_attempts, verification_attempts, verifications and suppressions. Preserve every row, including uncertain outcomes, consumed/cancelled links and failed reservation history. Scanner databases, provider mailboxes/message bodies, SMTP passwords and ingest tokens are separate scopes and are not included. Application database/content backup cannot recover this ledger.

capture-ledger.py is default server dry-run. It checks exact cluster UID, current immutable receiver image, single Ready replica, Bound email-connect-data PVC and /data/email-connect.db configuration. The helper runs UID10001, mounts only the read-only PVC and bounded scratch space, has no environment credential or ServiceAccount token, and must be selected by no-egress NetworkPolicies. No SMTP code is started in the helper.

Under admitted --execute, a guarded replica patch pauses the receiver, waits for its pods to disappear and rejects any other ledger PVC consumer. SQLite's backup API reads committed data including WAL; copying only the main file would miss uncheckpointed records. Integrity checks, exact five-table inventory and schema/ row fingerprints validate the private download. The default 128MiB SQLite ceiling is a reviewed candidate budget, not a promise of unlimited ledger growth.

Finally removes the helper and resumes the unchanged receiver. Replica patches test the original deployment template and expected count; a foreign rollout is never overwritten or silently resumed. Failed cleanup/resume cannot declare a successful capture. A nonblocking lock prevents overlapping captures. SIGTERM attempts the same cleanup; host loss/SIGKILL still needs an explicit replica/helper check. Receipts contain counts/hashes/booleans, never row contents or credentials.

Publication and recovery

backup-ledger.py validates the capture, packages only database.sqlite3 and receipt.json, then seals once using the platform public age recipient. A failed publication preserves one private pending directory and blocks another capture; reviewed retry reuses the ciphertext and object key without another pause. Partial encryption remains for review. Successful publication clears only its own staging directory; historical private pre-release copies remain untouched.

The platform primary_application_archive.py email-connect profile uses platform-pg/application-archives/email-connect/ in the existing primary bucket. Conditional multipart completion, returned version GET and ciphertext size/hash verification gate last-success.json and inventory.json. Independent retrieval requires only a separately retained version/hash/size receipt and recovery caller. No publisher may delete retained archives, access other app prefixes, or hold an age private identity. The dedicated hash-locked platform SDK runtime is reused.

The candidate node worker expects owner-delivered AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in root-owned mode 0600 /etc/email-connect-backup/primary.env, parent mode 0700. A metadata-only node check found that file absent. Existing SMTP custody and the databases backup Secret do not admit this consumer. RPF-WP-0053-T05 tracks its separate prefix/caller, retention/ capacity, independently retained receipt and recovery identity admission. No Secret was fetched/copied, ESO scope widened or provider key minted in this work.

After retrieving ciphertext, the attended custodian runs:

python3 scripts/backup-ledger.py recover --execute \
  --archive /private/retrieved.tar.age --transfer-receipt /private/last-success.json \
  --identity /private/recovery-identity --destination /private/recovered-ledger
python3 scripts/rehearse-ledger.py --execute \
  --recovered /private/recovered-ledger --receipt /private/recovery-proof.json

Recovery verifies exact provider key/version/hash/size evidence, age-decrypts privately, inspects the entire packet before extraction, accepts exactly two regular files and rejects traversal, links, duplicates, extras and oversized entries. It checks the inner image/capture/database/schema/row bindings. Files are mode 0600 and directories mode 0700. A failed drill preserves private evidence; no production database path is modified.

Rehearsal uses the already-cached pinned receiver image under UID10001, read-only root filesystem, bounded tmpfs, dropped capabilities and --network none. It compares schema/all-table row fingerprints before and after opening the current SQLiteDeliveryStore, probes accepted/unknown reservations without a provider, checks invalidated handles/suppression, and verifies the ledger remains unchanged. No SMTP configuration is supplied. Only its own named disposable container is removed on completion/failure. application_restore_verified means this isolated ledger rehearsal; production_restored remains false and recipient inbox/reply operations are not inferred from it.

Historical-point recovery gate

A backup preserves state at its capture point. Operations after that point may include new provider submissions, suppressions, or challenge consumption/ cancellation. Restoring an older point alone can lose a reservation and revive a handle consumed later. Therefore an isolated successful drill never releases production traffic automatically. Incident recovery must first fence callers and outbox replay, keep sending/challenge consumption disabled, reconcile post-capture operations with user-engine/provider evidence and the identity owner, then invalidate or reconcile recovered challenges under an explicit incident decision. Only that owner-approved reconciliation may release the receiver. Unknown outcomes must remain unknown; no automatic resend or inference of non-delivery is allowed. An RPO 24h proposal needs explicit acceptance of this loss window; consider a shorter schedule if the ledger owner requires it. Daily timing is not a guarantee of acceptable authentication recovery.

Scheduling and operating acceptance

Proposed daily 04:15 UTC (+up to 5min jitter), hourly freshness threshold 26h and provisional attended RTO 60min are pending targets. Install reviewed scripts in /opt/email-connect-backup/ and four deploy/backup units disabled. Use root-owned mode 0700 /var/lib/email-connect-backup. Record source and lock hashes and custody delivery reference. Update both capture/rehearsal image pins under a release review.

Publisher local encrypted inventory grows until separate owner retention acts. The platform retention preview can propose 14 daily/4 weekly distinct generations and protects newest plus the exact recovered version/hash, but owns no deletion. Admit capacity/headroom and compatibility with the bucket's documented 30-day lifecycle before enabling a schedule. Connect service/freshness failures to the established evidence owner and prove reception/response. Retain public recovery receipts and the recovery image/key independently of node/workstation/provider loss. Production restore and private-copy expiration are separate owner actions.

Evidence

• Full email suite: 120 passed, zero skips, with the local GreenMail SMTP/IMAP harness; 19 new synthetic backup/recovery cases. No production account, URL or mail send is used by those tests. Existing 39 transactional regression cases pass.
• Actual age synthetic seal/decrypt and packaged rehearsal pass: all five tables, schema and rows preserved; accepted verification replay, unknown verification, invalidated handles, suppression and invitation reservations checked without sending. This is local synthetic evidence, no provider retrieval claim.
• Node default server dry-run passes: current pinned image/PVC, no-egress helper, and unchanged receiver specification. No pause, helper creation or upload.
• All four systemd units pass systemd-analyze verify. Dedicated primary SDK source is already hash-locked and offline-verified by RPF-WP-0053; actual email caller admission remains missing.

Machine-readable evidence: ledger-backup-rehearsal-2026-10-08.json. The actual synthetic packaged drill checked 3 accepted verification operations, 1 unknown verification, 2 invalidated handles, 1 suppression, 1 accepted invitation and 1 unknown invitation reservation; isolated startup/checks took 0.86s. This excludes capture/upload/retrieval/decryption and is not full RTO. Private fixture paths and all row contents are omitted from the evidence file.

T01/T02 implementation/rehearsal can close on this evidence; T03/T04 remain open for actual custody, offsite capture/recovery, post-capture reconciliation contract, monitored operation and founder activation. CSOC-WP-0010 remains active for those and caller/sender/rate-limit, reply/bounce/inbox and cross-owner journey acceptance.

Admitted first run — 2026-10-09

The user authorized one first capture/upload/retrieval/isolated recovery with schedules disabled (decision 0dfcdd94-50ee-4219-9659-5c4c445721f0). Installed source and hash-locked SDK pass hardened node preflight. The receiver resumed after 56.41 seconds; primary version 1791537739370570 passed size/hash readback. A separate version-pinned GET and workstation decrypt/application rehearsal preserve all six tables, two accepted deliveries and one suppression, under UID10001/network-none with no provider calls. Other state tables are empty in this live snapshot; their invariants retain synthetic recovery evidence. The isolated drill interval was 636.29 seconds including intervening operator work, rather than a production restore/RTO measurement.

Public transfer/recovery evidence is docs/ledger-first-backup-2026-10-09.json. The existing mode-0600 workstation identity matches the configured public age recipient; separate escrow admission is still open. The admitted node publisher credential performed retrieval and was not copied to the workstation. This does not prove a separately scoped recovery caller or node/workstation-loss custody.

User selected the platform-root operations inbox. OnFailure hooks now deliver redacted notices to its State Hub inbox. A synthetic failed unit delivered adcebf95-26c4-4adb-99b0-4446c8505a42, verified by an independent inbox read. Failures to deliver retain private pending records for at-least-once replay with stable event ids; duplicates are possible if acceptance succeeded but its reply was lost. Successful freshness checks replay pending notices, while failures invoke the same delivery hook. No notification includes journal output, credentials, message contents or ledger rows. Human response is not claimed. Both timers are installed and disabled; one attended freshness check passed.

Full email regression after caller/notification changes: 178 passed, zero skips using the loopback GreenMail harness; focused syntax checks pass. Retention, capacity, independent receipt/key/image custody, separate recovery caller and recurring schedule/RPO admission retain EMAIL-WP-0007-T03/T04. Production restore still requires fencing and post-capture reconciliation before traffic release.