email-connect/workplans/EMAIL-WP-0007-ledger-backup-recovery.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	origin	origin_ref	state_hub_workstream_id
EMAIL-WP-0007	workplan	Back up and recover the transactional delivery ledger	infotech	email-connect	finished	codex	email-connect	2026-10-07	2026-10-10	residual	EMAIL-WP-0006	35be21a2-343b-52e1-ba5e-32fead7a7831

CSOC-WP-0010-T04/T05 request durable recovery of the SQLite delivery/challenge ledger independently of Coulomb's metadata/content pair. EMAIL-WP-0006 completed the verification retry release; its temporary private pre-release copies are not a recurring backup. This work does not change SMTP/caller custody or send mail.

T01 — Implement coherent ledger capture and verified publication

id: EMAIL-WP-0007-T01
status: done
priority: high
state_hub_task_id: "20b72599-da99-5d66-a0ab-8d946e837a2f"

Prepare a default-dry-run node capture, pinned to the current cluster/image/PVC. Quiesce the single receiver, use SQLite's backup API against the read-only PVC, verify integrity/all five transactional tables and resume the unchanged receiver in finally. Lock overlap and preserve private failure evidence. Package SQLite and its capture receipt only, age-encrypt once, publish through platform's immutable email-connect archive profile, and require version/hash readback before advancing success. Reviewed retries reuse ciphertext without another pause. No SMTP password, ingest token, scanner mailbox or provider message body belongs in this archive. Tests use only synthetic records and no production endpoints.

T02 — Prove isolated recovery preserves delivery and challenge state

id: EMAIL-WP-0007-T02
status: done
priority: high
state_hub_task_id: "c8acba6d-1087-5ffd-a7e6-59608c07c1ff"

Add attended guarded decrypt/unpack and all-table/schema/row fingerprint checks. Exercise current packaged service under UID10001/network-disabled conditions with synthetic recovered SQLite. Accepted sends/verifications replay without a provider call; unknown outcomes remain blocked; suppression and consumed/cancelled links remain effective. A historical snapshot can lose newer reservations or revive a handle consumed after capture: production recovery must freeze delivery, reconcile post-capture operations/issuer state and explicitly invalidate/reconcile challenges before releasing traffic. Never auto-restore into a running service.

T03 — Admit the independent primary caller and retention custody

id: EMAIL-WP-0007-T03
status: done
priority: high
needs_human: false
state_hub_task_id: "680fe4d6-12af-5296-acf9-15c9a01a0db3"

Platform profile: platform-pg/application-archives/email-connect/ in the current Scaleway primary bucket. Request a separately scoped caller; do not copy another workload's Secret or inherit its write/delete authority. Worker has only public age recipient, no escrow identity or retained-object deletion rights. Bind the platform hash-locked SDK runtime and separate recovery/retention owner. RPF-WP-0053-T05 tracks admission; founder continuation authorizes source preparation, not an invented owner review or credential grant.

T04 — Complete the admitted first run, recovery and monitoring

id: EMAIL-WP-0007-T04
status: done
priority: high
needs_human: false
state_hub_task_id: "66401e5c-a5cf-5e67-beab-e503ae079964"

Present pinned source hashes, dry-run evidence, coherent ledger scope, bounded pause and rollback/resume protocol before exact activation. First capture must resume current service before encryption/upload. Independently retrieve/decrypt with escrow and validate all rows/schema and no-send recovery behavior. Record RPO/RTO and retained recovery point, then admit schedule and monitored failure response. Temporary private release copies retain their existing custody until the owner explicitly disposes of them; no expiry is implied here.

Implementation evidence — 2026-10-08

Standalone scripts/capture-ledger.py, ledger_archive.py, backup-ledger.py and rehearse-ledger.py plus four disabled unit candidates implement T01/T02. Full email suite 120 passed with the local GreenMail harness, zero skips, including 19 new synthetic ledger cases. Actual age roundtrip and pinned receiver-image rehearsal under UID10001/network none preserve all five tables/schema/rows and verify accepted/unknown reservation and consumed/cancelled/suppression behavior without provider calls. Isolated rehearsal is not live recovery acceptance.

Node server dry-run verifies current cluster/image/PVC, no-egress/no-secret helper and replica-only guarded pause projection; receiver remains unchanged, no helper created. All four units verify. Metadata-only check: email primary.env absent; no credentials fetched/copied, live pause/capture/provider upload, deploy or timer activation. Complete review: docs/ledger-backup-review-2026-10-08.md. Platform admission handoff RPF-WP-0053-T05 is a live record, not an owner approval. T03/T04 remain wait for custody, retention/capacity, actual offsite/recovery and monitoring plus the incident post-capture reconciliation gate. The workplan remains active.

Receiver image/schema alignment — 2026-10-08

EMAIL-WP-0008-T04 deploys receiver eb3e7b68 (sourcec9756b0) in the admitted single-caller mode. The ledger has caller_id/send-failure fields and caller_buckets. Capture/rehearsal source helpers are repinned together to this digest; fingerprints cover all six tables. Actual pre-release five-table backup migrates privately with every existing row preserved, UID10001/network-none and no send. See docs/caller-deployment-2026-10-08.md. This is image/schema alignment only: T03/T04 archive custody, first admitted offsite run/retrieval/application recovery and monitoring stay waiting. CSOC-WP-0015-T03 is the product acceptance follow-up.

Pair-custody preview — 2026-10-08

Founder admitted backup-pair-writer on 2026-10-08. The OpenBao policy is installed and matches its source file. Version 2 is current on both backup-primary paths and the placeholders are gone. The bucket policy now also names the email-connect publisher, limited to platform-pg/application-archives/email-connect/, without object delete. A prefix proof aborted its own upload and denied the other prefixes. No primary.env, receiver pause, upload or timer. T03/T04 stay wait. Review: railiance-platform/docs/credential-lane-designs/application-backup-pair-custody-2026-10-08.md.

Reader delivery — 2026-10-08

Founder acceptance recorded on CCR-2026-0036 and CCR-2026-0037. Both reader policies and Kubernetes roles are installed. email-connect /etc/email-connect-backup/primary.env is root:root mode 0600 in a mode 0700 directory, KV version 2, token revoked before publication, readback equal. Coulomb's reader is applied and its node file was not created. No receiver pause, upload, escrow, retention decision or timer. T04 stays wait.

First admitted offsite recovery and failure delivery — 2026-10-09

User authorized one guarded capture, encrypted primary upload, separate provider retrieval and isolated workstation recovery; schedules explicitly remain disabled. Decision: 0dfcdd94-50ee-4219-9659-5c4c445721f0. Runtime scripts and the platform hash-locked SDK are installed under /opt; actual credential presence/model check and capture preview pass under the service filesystem restrictions. File hashes: deploy/backup/runtime-manifest-2026-10-09.json.

Capture/resume took 56.41 seconds. Provider version 1791537739370570 and ciphertext size/hash readback pass. A separate GET retrieves that version with the admitted node publisher credential, then ciphertext and the public receipt move to the workstation; no credential or private identity moves to the node. The existing private workstation identity matches the configured public recipient. Attended decrypt plus UID10001/network-none application rehearsal preserve all six tables/schema/rows, two accepted deliveries and one suppression without sending. Other transactional state and quota tables are empty in this live snapshot; their recovery behavior remains supported by the earlier synthetic tests, not a claim of populated live recovery coverage. Production restore/RTO are not measured. Public proof: docs/ledger-first-backup-2026-10-09.json. Private recovery artifacts remain mode 0700/0600 under /tmp/email-connect-first-recovery-20261009.

Both units now have OnFailure hooks to the platform-root State Hub inbox, selected by the user. Notification failures retain private pending records with stable event ids; freshness runs replay them. A synthetic systemd failure delivered message adcebf95-26c4-4adb-99b0-4446c8505a42, independently found in that inbox. Human acknowledgement/response is not claimed. Installed freshness passes and both timers are disabled. T04 is progress, with first-run/recovery/delivery proved. T03 retains generation retention/capacity, independently retained receipt/key/image escrow and a separate recovery caller. Schedule/RPO admission and incident post-capture reconciliation remain required before workplan completion.

Closure package and Scaleway custody — 2026-10-09

User selected Scaleway, then named email-connect-backup-primary. Retain that existing IAM application (0eb0f958-77a4-4c70-9aa7-14ba46411415) for routine archive publication; CCR-2026-0037 admits only the email-connect archive prefix and excludes escrow and administration. This selection is not evidence of additional grants. Actual native metadata shows versioning Enabled and a bucket-wide 30-day expiry for current and noncurrent objects. No provider configuration was changed.

The public closure contract is deploy/closure/plan.json; preparation evidence is deploy/closure/evidence-2026-10-09.json. Proposed schedule is daily 04:15 UTC with five-minute jitter, 24-hour RPO target and 26-hour freshness threshold. Proposed local capacity floor/budget are both 8 GiB, with 14 daily/4 weekly generations and weekly owner-operated expiry by platform-root. The new staged capacity guard refuses before capture at low free space or when local retained files approach that budget; it never deletes retained objects. These targets are prepared, not measured production RTO or enabled operation. Node-installed helpers are unchanged.

A private 47,541,825-byte recovery kit binds the existing protected first backup, receiver image, recovery tools and public receipts. It excludes the private age identity. SHA256: 97d1a4d4b55729dd226277601637d64b086de21b21c9951c1d7da05c914c350b. Rehearsal using its hash-bound bundled image passes without pulling an image, under UID10001/network-none, preserving all six tables; elapsed 2.31 seconds is application rehearsal only. The kit remains local and has not been uploaded.

Prepared Scaleway custody uses a protected opaque Secret Manager container email-connect-ledger-age-recovery in nl-ams with no ephemeral policy, and private versioned bucket railiance-email-connect-recovery, prefix kits/, with no expiry. This preserves the protected recovery point beyond the primary archive lifecycle. Provider containers, project/secret references, separate recovery application and effective grants still need owner admission. Existing publisher access cannot create these resources. A different durable arrangement is acceptable after its actual effective permissions and retention are reconciled with the contract.

scripts/scaleway-recovery-custody.py defaults to preview. Attended execution takes --secret-id, --project-id, --identity and a fresh --receipt; SCW_SECRET_KEY stays in the operator environment. It checks exact protected metadata, writes/reuses only revision 1 and silently verifies the matching age recipient by readback. scripts/publish-ledger-recovery-kit.py defaults to preview; execution accepts private owner/reader credential files, --kit, --kit-receipt, --output and --receipt. It refuses disabled versioning or lifecycle rules, publishes conditionally and checks size/hash through a separate reader's version-pinned GET. Different key IDs do not prove different application bindings or denied writes. Receipts keep those fields false pending native proof, independent account entry and a drill from the provider-retrieved kit and escrow identity. No secret values belong in chat, argv, evidence, Git or the operations inbox.

Full email regression: 193 passed, zero skips; focused syntax checks pass. Timers remain disabled. T03/T04 stay progress until durable independent custody, retention/schedule admission and actual recurring operation are evidenced.

Runtime guard installed — 2026-10-09

User instructed continuation. Installed the reviewed capacity guard and offline image-binding rehearsal helper on railiance01 after comparing existing hashes with the first-run runtime manifest. Readback hashes match the closure contract. Actual pre-capture capacity check passes: retained local files 148,994 bytes, 8 GiB minimum free space and 8 GiB local budget. No credential read or backup run. Receiver image/pins remain eb3e7b68 and both timers remain disabled. Receipt: deploy/backup/runtime-update-2026-10-09.json. Earlier preparation evidence retains its historical statement that node helpers had not yet changed.

Scoped receiver recovery point — 2026-10-09

Capture/rehearsal source and installed runtime now bind the live scoped receiver 5d79fd1f. Actual new offsite version 1791569930973785 has ciphertext SHA256 1a7ae09129d8252053ea210cfe37d8a27d25bc4ee615bfe85f65c207b3e5f38e. Capture/resume took 57.47 seconds. Private retrieval to the workstation, existing age identity decrypt and UID10001/network-none application rehearsal preserve all six tables, including one caller quota bucket, two accepted deliveries and one suppression. Other transactional tables remain empty in this live point. Proof: docs/ledger-scoped-image-recovery-2026-10-09.json.

Refreshed private recovery kit: 47,624,219 bytes, SHA256 3ce8788375640398cd464996ea371a3d3d61a9252b2fb06a2f4192ad70f3811b, with the current receiver image/tools and ciphertext, excluding the private age identity. It has not been uploaded into independent durable custody. Existing publisher readback is not a separate recovery IAM caller. Both deployments resume Ready. EMAIL-WP-0008 is finished; T03/T04 here remain progress for independent Scaleway custody/grants, retention/schedule and recurring operation. Timers remain disabled. The user was asked for an existing Scaleway operator lane or Console custody setup, since the existing publisher cannot administer escrow resources.

Completed independent custody and recurring operation — 2026-10-10

The user completed the attended Scaleway operator login and authorized finishing the workplans. T03/T04 are done. Protected, nonexpiring age revision 1 and a private, versioned, nonexpiring recovery kit now reside in independent project email-connect-recovery (1d37ea63-8c8f-44e1-80e1-2de862579ba7), nl-ams. The separate IAM reader is natively bound and tested; scheduled-worker kit/key reads and retained-object deletes return 403. Recovery using only provider-retrieved kit, bundled image/tools and restored escrow identity passes without network or mail. Public kit library permissions were corrected after the cold drill found UID10001 could not read its original 0600 bind mount; a regression covers it.

A temporary original-project reader grant admitted metadata reads from an unrelated bucket. It was removed and original bucket grants restored. Native unrelated GET/LIST now deny 403. Daily archives and complete recovery receipts are copied to private, versioned railiance-email-connect-archives in the isolated project with 30-day expiry. The existing primary publisher remains in use. Backup success/freshness require hash/version-verified mirror publication. Failures preserve a pending capture for retry without another receiver pause.

The final installed-code run captured/resumed in 57.55 seconds, independently published/read back both copies, and restored through the separate recovery reader and provider receipt. All six tables match, with one quota bucket, two accepted deliveries and one suppression. Remaining live transactional tables are empty; their behavior remains covered synthetically. Both deployments are Ready, and daily 04:15 UTC (+ up to five-minute jitter) backup and hourly freshness timers are enabled/active. Freshness passes; failures route to the user-selected platform-root inbox, with pending delivery replay. Platform-root owns weekly 14-daily/4-weekly retention selection, local ciphertext expiry/capacity review and retained-kit refresh. Worker deletion remains prohibited; 8 GiB local budget/free floor guards remain installed. RPO target is 24 hours with a 26-hour stale alert. 60-minute attended RTO is a target, not a measured production restore. Production recovery still requires fencing and post-capture/challenge reconciliation.

Evidence: docs/ledger-scaleway-custody-2026-10-10.json, deploy/backup/operating-acceptance-2026-10-10.json and updated docs/ledger-backup-review-2026-10-08.md. Focused regression: 47 passed, zero skipped. Syntax/undefined-name lint passes. The broader lint profile reports existing shell/rehearsal exceptions and the deliberate credential-safe top-level exception boundary; no broad clean-lint claim is made. Temporary operator/drill API key records were deleted after proofs; the permanent reader application and operator-account bootstrap path survive workstation/node loss. EMAIL-WP-0008 remains finished. Earlier waiting/local-only entries above are historical.