coulomb-social/docs/release/paired-backup-operations-2026-10-07.md

Paired backup operations candidate — 2026-10-07

CSOC-WP-0008-T02 has a coherent snapshot, encrypted publication, independent retrieval and guarded recovery implementation. Admission remains proposed in RPF-WP-0053. The previous Nextcloud overwrite-slot candidate was never activated and is withdrawn: its governed workload uploader is create-only. The revised candidate uses immutable Scaleway primary archives, matching the platform's current storage contract.

Product source owns coherent capture and app checks. Railiance-platform owns execution admission, primary custody, retention, escrow and monitoring. The exact consumer request is in its docs/credential-lane-designs/coulomb-paired-backup-primary.md. It records the absent caller credential file and stale historical routing id; no secret retrieval or cross-workload Secret copy was attempted. No production archive has been uploaded and no timer has been installed or enabled.

Proposed contract

Item	Candidate contract
Target	railiance01; kube-system UID a553c742-0115-43d4-99a4-a5ca56fe0786
Data	coulomb_social_db on apps-pg and entire coulomb-social-content PVC
Coherence	Exact app digest, one replica, scale to zero, capture both stores, resume before encryption/upload
Schedule	Daily 03:15 UTC with up to five minutes jitter; persistent timer
Targets	Proposed RPO 24h; provisional attended RTO 60min; full measurement/acceptance open
Primary	s3://railiance-platform-pg-backup/platform-pg/application-archives/coulomb-social/; nl-ams Standard Multi-AZ
Publication	Timestamp/random immutable key, conditional multipart completion, returned version GET and ciphertext SHA-256 check
Retention	Proposed 14 daily/4 weekly generations, no duplicate weekly uploads; separate owner admission/executor, currently not enforced
Health	Hourly check fails at age >26h, missing version/readback evidence or local encrypted hash mismatch; evidence reception unproven
Escrow	Worker gets public age recipient; attended recovery custodian holds private identity and independent retrieval authority

No scheduled publisher deletes retained objects. The bucket's documented 30-day current/noncurrent lifecycle must be checked for prefix coverage and compatibility with weekly/protected points. Platform application_archive_retention_plan.py proposes 14 distinct daily and4 weekly points without duplicate copies, protecting the newest and exact application-restored version/hash. It refuses missing/changed recovery proof and ignores foreign/unverified inventory. It has no deletion capability. Local encrypted inventory also grows until the separate retention owner acts. Capacity/headroom and a bounded retention contract are activation prerequisites; candidate code alone does not establish them. The platform's independent Nextcloud secondary remains separately governed.

The node executor candidate uses root/k3s admin with exact app/database/cluster checks; platform may select a bounded activity-core executor instead. This remains pending owner admission. No live control has been relaxed. The email-connect SQLite delivery/challenge ledger is a separate recovery scope tracked by CSOC-WP-0010-T04/T05; an application DB/content pair does not recover that ledger.

Publication and failure behavior

scripts/paired-backup.py validates capture completeness, app resume, both hashes and sizes, then wraps database.dump/content.tar.gz/receipt.json into a private tar and seals it with age. The platform helper receives only ciphertext through owner-delivered AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. It does not read Kubernetes Secrets or fall back to another workload's credentials.

Multipart completion uses IfNoneMatch='*'. The readback uses the exact returned VersionId, checks length and hashes downloaded bytes. Only that proof advances last-success.json and inventory.json. application_restore_verified=false remains explicit: ciphertext retrieval does not prove application recovery. The root receipt carries object key/version/hash/size and paired image/hash bindings in platform.application-primary-archive.v1 format for independent retrieval.

Both publication and direct capture take nonblocking locks. Snapshot termination attempts cleanup/resume; the parent permits the cleanup window before killing a stuck child. Host loss/SIGKILL still requires an operator replica check. Failure retains one private pending directory and blocks further unattended capture. Reviewed --execute --resume-pending reuses the same sealed ciphertext and object key without another pause/encryption. An uncertain completion retries HEAD/hash binding plus version GET rather than creating another recovery point. Changed bytes/versions refuse. Partial encryption/readback is preserved for review. Success removes only its own private staging tree and retains downloaded encrypted archives. Historical private capture/drill copies remain under their existing custody; this candidate does not expire them.

Installation and activation

RPF-WP-0053-T01/T02 must admit execution identity, exact primary prefix/caller and retention. Root-owned mode 0600 /etc/coulomb-social-backup/primary.env is the candidate delivery surface (parent mode 0700), containing AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. No AGE_PRIVATE_KEY belongs in that file or scheduled worker. Existing provider keys may have broader rights; minting another key under the same IAM application does not narrow that application's policy. Owner must prove the admitted caller's rights and negative tests, not rely on client code to constrain a broad key.

Install pinned consumer scripts in /opt/coulomb-social-backup/ and the platform primary_application_archive.py in /opt/railiance-platform/scripts/. Install age, kubectl and the dedicated platform backup-venv. Its requirements/backup-primary.lock pins all seven packages and wheel/source hashes; install with a hash-enforcing sync, rather than resolving dependencies at each scheduled run. boto3 1.40.18 / botocore 1.40.76 expose conditional multipart completion. No SDK is added to the application image. Install four deploy/backup units disabled and use root-owned mode 0700 /var/lib/coulomb-social-backup. Record source hashes, dependency lock hash, owner delivery/grant reference, cluster UID and exact app digest. Release changes must refresh the snapshot helper's expected image; mismatch refuses capture.

Default consumer invocation performs a server dry-run. Platform transfer/retrieve commands preview without contacting the provider unless --execute is present. Platform check validates SDK/custody presence offline; it does not prove provider rights. Live transfer must demonstrate caller prefix isolation, versioning, conditional create and retained-object deletion denial. First approved service run must leave the app Ready before any offsite transfer. Preserve failed evidence; never auto-restore production after a backup failure.

Independent retrieval and attended recovery

Retain the public version/hash/size receipt independently of the node/workstation. Node-loss retrieval uses only that receipt, the recovery caller and a fresh private output directory; no original ciphertext/source pair is needed:

/opt/railiance-platform/backup-venv/bin/python \
  /opt/railiance-platform/scripts/primary_application_archive.py retrieve \
  --application coulomb-social --receipt /private/last-success.json \
  --output /private/retrieved.tar.age --execute
python3 scripts/recover-paired-backup.py \
  --archive /private/retrieved.tar.age --transfer-receipt /private/last-success.json \
  --identity /private/recovery-identity --destination /private/recovered-pair --execute
python3 scripts/restore-paired-snapshot.py /private/recovered-pair

The attended recovery helper verifies ciphertext hash/size/version evidence, decrypts into a fresh private directory, inspects the entire tar before extraction and admits only the three expected regular files. It rejects links, duplicates, traversal, extras, sparse/oversized entries and mismatched inner pair/capture/image bindings. Extracted files are mode 0600 and the directory mode 0700. Recovery credentials and age identity are owner-supplied private files/environment; values never enter commands, Git, State Hub or logs. Failure preserves private evidence for review.

The existing restore helper creates only an internal Docker PostgreSQL environment with no published ports, compares COPY rows/sequences and all safe content files. Then rehearse migrations, actual rendering, owner/reader behavior and denied ungranted reads. Record retrieval/decryption/restore/app-ready elapsed time against the proposed RTO. Production restoration remains a separately authorized action. A successful drill must pin its recovery point before retention expires older data.

Enable timers only after the admitted first run/recovery and operating contract. Connect failed/freshness units to the evidence owner and prove reception of a synthetic stale signal; journald alone is not monitored acceptance. Stop by disabling the timer and stopping the service with its five-minute cleanup window; confirm one Ready replica and no helper pod. Preserve pending pairs and recovery receipts.

Validation

Full make check passes Ruff, 150 tests and all mandatory Chromium journeys. The new 19 consumer backup/recovery cases plus 30 synthetic provider/retention cases cover failures, stable retry, conditional creation, version/hash binding, lost acknowledgements, node-loss retrieval and unsafe packet refusal. Real age roundtrip uses an ephemeral synthetic identity/data; no production accounts or content enter tests. All four units pass systemd-analyze verification.

Earlier node server dry-run passed against the current app digest/PVC without quiescence, capture, upload or helper pod creation. Earlier production-pair isolated restore matched 13 tables, 104 rows, sequences and 75 files; its 4.14s is restoration only, not full RTO. This revised transport has no live provider/recovery claim yet. T02 stays progress for custody, retention, offsite/application recovery, monitored operation and service-contract acceptance.

Current admission and explicit recipient guard — 2026-10-28

The scoped node executor and publisher pair are admitted/delivered under decision 9557e2b8-2b37-5b94-a5e6-e5fe08d0351e. Independent Coulomb recovery custody is now approved under 27bbc1ac-e275-42be-ba34-6cf16eae4e78; native resource setup needs the platform-owned attended operator entry point. Existing publisher application coulomb-social-backup-primary is correctly prefix-bounded and denied IAM administration; it cannot provision the independent reader and escrow.

The backup consumer no longer defaults to a shared recipient. Before execution, the owner must deliver only the dedicated public age recipient in root-owned 0600 /etc/coulomb-social-backup/recipient.env beneath the existing 0700 parent: COULOMB_SOCIAL_BACKUP_AGE_RECIPIENT=<approved-public-recipient>. The paired service requires that file; no private identity belongs there or on the worker. An attended invocation can use --age-recipient explicitly. Missing/malformed configuration refuses before commands/state/capture; age validates the actual recipient/checksum by encrypting empty input before provider auth or app pause. Default server dry-run and freshness checks do not require escrow access.

Sealed retry metadata and final receipts bind the public recipient. A changed or absent pending binding refuses before publication; do not relabel old ciphertext or automatically re-encrypt/re-capture it. Preserve any such private evidence for custodian review. Source approval/configuration and a publisher GET remain separate from independent kit/key readback and application recovery acceptance.

First independent application recovery — 2026-10-29

The admitted worker now holds its own publisher pair and only the dedicated public recipient. The independent API custody front door and provider coordinates are published by railiance-platform in docs/credential-lane-designs/coulomb-recovery-api.{json,md}. Temporary construction authority and personal browser state have been retired.

First coherent capture/publication succeeded; exact-version primary and isolated mirror readbacks match ciphertext hash/size. Reader API retrieval and custodian API key retrieval independently supplied a verified pair. Isolated recovery matches all13tables/112rows/sequences and82files, and the actual packaged app renders all39pages with restored-owner access and viewer/ungranted denials. See first-*-2026-10-29.json receipts in this directory. The4.23s DB/content restore is not a full RTO measurement; real-browser/identity-owner acceptance remains separate. No production restore, lifecycle expiry or timer activation occurred.

scripts/verify-restored-application.py accepts --restore-receipt, --snapshot and a fresh --output; it verifies the real isolated network/container/image/ports, uses only that recovered DB/content with read-only content, and rolls back its synthetic permission fixtures. It prints counts/booleans, never page/principal values. A complete independently retrieved protected image/tools/receipt kit, full recovery timing, retention/headroom and received monitoring proofs remain required before schedule activation.

Protected provider-only recovery proof — 2026-10-29

The complete protected kit and last recovered point are now independently held outside rolling expiry. The277,585,920byte immutable kit contains exact app and PostgreSQL images, encrypted data, recovery tools/locked SDK and receipts, with no private identity/API pair. The separate reader verifies its exact version/hash; all2098member hashes are checked before executing any recovered tool or image.

Platform application_recovery_drill.py automates retrieval, separate age escrow, image loading, decryption, isolated SQL/content restore and packaged app/view permission checks. Native end-to-end monotonic time30.232s, including12.373s kit download. All13tables/112rows/sequences,82files and39rendered pages/access denials pass. This uses held API callers and the existing Docker daemon; operator account recovery latency and a fresh daemon are excluded, and it is not a real-browser identity journey. See provider-kit-recovery-2026-10-29.json. No production restore or timer activation is implied. Retention/copy disposition, received monitoring and service-contract acceptance still gate scheduling and workplan completion.

For offline restoration, restore-paired-snapshot.py accepts --postgres-image with the exact retained image ID; verify-restored-application.py accepts --kit-manifest to bind both image IDs to that captured point. This avoids depending on registry availability or preexisting image tags. The public platform contract records the protected kit key/version/hash and exact last recovered primary point.

Current validation after offline image binding: make check passes246tests and the mandatory browser registry gate. Platform's70targeted synthetic cases cover identity/custody isolation, multipart mirror retries, kit corruption/path refusal, nonexpiring publication and independent proof retention. Those synthetic results supplement the native recovery receipt; they do not replace its application/data checks or the still-open service and productive user verdicts.

Latest verified protected recovery point (2026-10-31): capture07:13:52Z, kit4166eb8b, version1793432303941465, protected proof1793432409572654. Full provider-kit/escrow recovery passes in46.692s with13tables/114rows/86files/40pages. See provider-kit-recovery-post-date-2026-10-31.md and the platform public custody contract for exact references and scope. Do not substitute older kit pointers for this current point; historical kits stay retained pending disposition. Timers remain off.