Prepare independent ledger backup and isolated no-send recovery

Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a11538-05ea-7b63-b3c7-a2df5f38898d

docs/ledger-backup-review-2026-10-08.md

@@ -0,0 +1,142 @@
+# Email ledger backup/recovery candidate — 2026-10-08
+
+EMAIL-WP-0007 implements the independent transactional SQLite recovery scope
+requested by CSOC-WP-0010-T04/T05. The verification retry release remains live;
+this candidate does not change that deployment, SMTP/caller credentials or inbox
+access. The standalone node scripts and four systemd units are prepared for
+platform admission. No live capture, primary upload or timer activation occurred.
+
+## Scope and coherence
+
+The ledger contains deliveries, send_attempts, verification_attempts,
+verifications and suppressions. Preserve every row, including uncertain outcomes,
+consumed/cancelled links and failed reservation history. Scanner databases,
+provider mailboxes/message bodies, SMTP passwords and ingest tokens are separate
+scopes and are not included. Application database/content backup cannot recover
+this ledger.
+
+capture-ledger.py is default server dry-run. It checks exact cluster UID,
+current immutable receiver image, single Ready replica, Bound email-connect-data
+PVC and /data/email-connect.db configuration. The helper runs UID10001, mounts
+only the read-only PVC and bounded scratch space, has no environment credential
+or ServiceAccount token, and must be selected by no-egress NetworkPolicies.
+No SMTP code is started in the helper.
+
+Under admitted --execute, a guarded replica patch pauses the receiver, waits for
+its pods to disappear and rejects any other ledger PVC consumer. SQLite's backup
+API reads committed data including WAL; copying only the main file would miss
+uncheckpointed records. Integrity checks, exact five-table inventory and schema/
+row fingerprints validate the private download. The default 128MiB SQLite ceiling
+is a reviewed candidate budget, not a promise of unlimited ledger growth.
+
+Finally removes the helper and resumes the unchanged receiver. Replica patches
+test the original deployment template and expected count; a foreign rollout is
+never overwritten or silently resumed. Failed cleanup/resume cannot declare a
+successful capture. A nonblocking lock prevents overlapping captures. SIGTERM
+attempts the same cleanup; host loss/SIGKILL still needs an explicit replica/helper
+check. Receipts contain counts/hashes/booleans, never row contents or credentials.
+
+## Publication and recovery
+
+backup-ledger.py validates the capture, packages only database.sqlite3 and
+receipt.json, then seals once using the platform public age recipient. A failed
+publication preserves one private pending directory and blocks another capture;
+reviewed retry reuses the ciphertext and object key without another pause.
+Partial encryption remains for review. Successful publication clears only its
+own staging directory; historical private pre-release copies remain untouched.
+
+The platform primary_application_archive.py email-connect profile uses
+platform-pg/application-archives/email-connect/ in the existing primary bucket.
+Conditional multipart completion, returned version GET and ciphertext size/hash
+verification gate last-success.json and inventory.json. Independent retrieval
+requires only a separately retained version/hash/size receipt and recovery caller.
+No publisher may delete retained archives, access other app prefixes, or hold an
+age private identity. The dedicated hash-locked platform SDK runtime is reused.
+
+The candidate node worker expects owner-delivered AWS_ACCESS_KEY_ID and
+AWS_SECRET_ACCESS_KEY in root-owned mode 0600
+/etc/email-connect-backup/primary.env, parent mode 0700. A metadata-only node check found
+that file absent. Existing SMTP custody and the databases backup Secret do not
+admit this consumer. RPF-WP-0053-T05 tracks its separate prefix/caller, retention/
+capacity, independently retained receipt and recovery identity admission. No
+Secret was fetched/copied, ESO scope widened or provider key minted in this work.
+
+After retrieving ciphertext, the attended custodian runs:
+
+```bash
+python3 scripts/backup-ledger.py recover --execute \
+  --archive /private/retrieved.tar.age --transfer-receipt /private/last-success.json \
+  --identity /private/recovery-identity --destination /private/recovered-ledger
+python3 scripts/rehearse-ledger.py --execute \
+  --recovered /private/recovered-ledger --receipt /private/recovery-proof.json
+```
+
+Recovery verifies exact provider key/version/hash/size evidence, age-decrypts
+privately, inspects the entire packet before extraction, accepts exactly two
+regular files and rejects traversal, links, duplicates, extras and oversized
+entries. It checks the inner image/capture/database/schema/row bindings. Files
+are mode 0600 and directories mode 0700. A failed drill preserves private evidence; no
+production database path is modified.
+
+Rehearsal uses the already-cached pinned receiver image under UID10001, read-only
+root filesystem, bounded tmpfs, dropped capabilities and --network none. It
+compares schema/all-table row fingerprints before and after opening the current
+SQLiteDeliveryStore, probes accepted/unknown reservations without a provider,
+checks invalidated handles/suppression, and verifies the ledger remains unchanged.
+No SMTP configuration is supplied. Only its own named disposable container is
+removed on completion/failure. application_restore_verified means this isolated
+ledger rehearsal; production_restored remains false and recipient inbox/reply
+operations are not inferred from it.
+
+## Historical-point recovery gate
+
+A backup preserves state at its capture point. Operations after that point may
+include new provider submissions, suppressions, or challenge consumption/
+cancellation. Restoring an older point alone can lose a reservation and revive a
+handle consumed later. Therefore an isolated successful drill never releases
+production traffic automatically. Incident recovery must first fence callers and
+outbox replay, keep sending/challenge consumption disabled, reconcile post-capture
+operations with user-engine/provider evidence and the identity owner, then
+invalidate or reconcile recovered challenges under an explicit incident decision.
+Only that owner-approved reconciliation may release the receiver. Unknown outcomes
+must remain unknown; no automatic resend or inference of non-delivery is allowed.
+An RPO 24h proposal needs explicit acceptance of this loss window; consider a
+shorter schedule if the ledger owner requires it. Daily timing is not a guarantee
+of acceptable authentication recovery.
+
+## Scheduling and operating acceptance
+
+Proposed daily 04:15 UTC (+up to 5min jitter), hourly freshness threshold 26h and
+provisional attended RTO 60min are pending targets. Install reviewed scripts in
+/opt/email-connect-backup/ and four deploy/backup units disabled. Use root-owned mode 0700
+/var/lib/email-connect-backup. Record source and lock hashes and custody delivery
+reference. Update both capture/rehearsal image pins under a release review.
+
+Publisher local encrypted inventory grows until separate owner retention acts.
+The platform retention preview can propose 14 daily/4 weekly distinct generations
+and protects newest plus the exact recovered version/hash, but owns no deletion.
+Admit capacity/headroom and compatibility with the bucket's documented 30-day
+lifecycle before enabling a schedule. Connect service/freshness failures to the
+established evidence owner and prove reception/response. Retain public recovery
+receipts and the recovery image/key independently of node/workstation/provider
+loss. Production restore and private-copy expiration are separate owner actions.
+
+## Evidence
+
+- Full email suite: 120 passed, zero skips, with the local GreenMail SMTP/IMAP harness;
+ 19 new synthetic backup/recovery cases. No production account, URL or mail send
+ is used by those tests. Existing 39 transactional regression cases pass.
+- Actual age synthetic seal/decrypt and packaged rehearsal pass: all five tables,
+ schema and rows preserved; accepted verification replay, unknown verification,
+ invalidated handles, suppression and invitation reservations checked without
+ sending. This is local synthetic evidence, no provider retrieval claim.
+- Node default server dry-run passes: current pinned image/PVC, no-egress helper,
+ and unchanged receiver specification. No pause, helper creation or upload.
+- All four systemd units pass systemd-analyze verify. Dedicated primary SDK source
+ is already hash-locked and offline-verified by RPF-WP-0053; actual email caller
+ admission remains missing.
+
+T01/T02 implementation/rehearsal can close on this evidence; T03/T04 remain open
+for actual custody, offsite capture/recovery, post-capture reconciliation contract,
+monitored operation and founder activation. CSOC-WP-0010 remains active for those
+and caller/sender/rate-limit, reply/bounce/inbox and cross-owner journey acceptance.