diff --git a/docs/credential-lane-designs/coulomb-paired-backup-primary.md b/docs/credential-lane-designs/coulomb-paired-backup-primary.md new file mode 100644 index 0000000..2795ec8 --- /dev/null +++ b/docs/credential-lane-designs/coulomb-paired-backup-primary.md @@ -0,0 +1,75 @@ +# Coulomb paired-backup primary custody review — 2026-10-07 + +Pending admission under RPF-WP-0053-T01/T02; this is a concrete consumer request, +not an approved credential grant. No existing Secret, policy or grant is changed. + +The revised consumer uses Scaleway primary storage, separate from native Barman +objects. The earlier Nextcloud overwrite-slot proposal is withdrawn: the current +Nextcloud workload uploader is create-only and remains the independent secondary. + +| Coordinate | Requested contract | +| --- | --- | +| Consumer | Node-side coulomb-social-paired-backup.service; proposed root executor guarded by current cluster UID/app digest, subject to T01 review | +| Cluster | railiance01; kube-system UID a553c742-0115-43d4-99a4-a5ca56fe0786 | +| Destination | s3://railiance-platform-pg-backup/platform-pg/application-archives/coulomb-social/ | +| Provider | Scaleway Standard Multi-AZ, nl-ams; https://s3.nl-ams.scw.cloud | +| Source scope | coulomb_social_db on apps-pg plus the coulomb-social-content PVC, one quiesced capture | +| Publisher rights | Multipart creation/parts/completion, abort its own unfinished uploads, HEAD and version-pinned GET within the exact archive prefix; prefix-constrained listing if required for HEAD missing-object semantics | +| Publisher exclusions | No retained-object/version deletion, native Barman writes, other application prefixes, bucket/IAM configuration, escrow identity or bootstrap provider key | +| Worker delivery | Owner writes /etc/coulomb-social-backup/primary.env, root:root0600, parent0700; AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY only | +| Recovery custody | Independent read authority plus the age identity; public object key/version/hash/size receipt retained outside the node/workstation | +| Retention | Separate owner; proposed 14 daily/4 weekly recovery generations, deduplicated and protecting newest plus last successfully recovered point; not activated | +| Runtime | Dedicated /opt/railiance-platform/backup-venv, hash-locked requirements/backup-primary.lock; SDK conditional-completion field verified offline | + +Existing CCR-2026-0012 identifies the custody owner and source +platform/workloads/railiance/backup/object-storage fields ACCESS_KEY/SECRET_KEY. +It authorizes the existing databases/platform-pg-backup-s3 delivery, not this +consumer. Do not copy that Secret, broaden its ESO namespace conditions, reuse the +bootstrap key or manufacture a successor grant. Owner must determine whether a +separate scoped IAM application/key is needed: Scaleway policies attach to the +application, so merely minting another key under a broad application does not +establish prefix isolation. A worker intentionally has no delete authority even +if an existing provider identity has wider capabilities. + +Routing check: warden route show backup-object-storage returns unknown on this +workstation, despite that historical catalog id in CCR-2026-0012. Broad route +lookup returns the Nextcloud lane and generic OpenBao pointer. No secret retrieval +was attempted. Owner must reconcile the current primary front door and delivery; +record its review/grant reference here before execution. Node metadata-only check +on 2026-10-07: primary.env absent. Source presence is not custody admission. + +Before first publication, prove bucket versioning and conditional create behavior +under this caller, missing-object HEAD behavior, denied foreign-prefix access and +retained-object deletion denial. Metadata/code tests do not prove those live +rights. The worker does not contact a provider during default preview. Its check +requires the SDK model and delivered fields, without printing values or contacting +Scaleway. Live transfer needs --execute; it never reads Kubernetes Secrets. + +The platform documents a 30-day current/noncurrent lifecycle; exact prefix coverage +must be checked before accepting a 14/4 calendar contract. Four weekly selections, +protected recovery points and monthly drills can outlive an age-based expiration. +Do not activate an incompatible lifecycle or promise protected points survive it. +Owner must bind capacity/headroom, local encrypted-copy retention and independent +receipt retention before enabling an unattended timer. No live deletion/lifecycle +change is included in this request. + +First run: approved current-image capture, app resumes, age sealing, immutable +conditional completion, version-pinned GET and ciphertext SHA-256 check. Receipt +explicitly says application_restore_verified=false. Recovery on a different +workspace uses the retained receipt with primary_application_archive.py retrieve, +then consumer recover-paired-backup.py and restore-paired-snapshot.py. Retrieve does +only version-pinned GET and needs no original archive. Decryption belongs to the +attended custodian, never the scheduled publisher. Complete rendering and denied +access checks and record full elapsed time before calling this recoverable service. + +Provider contract reference: [Scaleway conditional writes](https://www.scaleway.com/en/docs/object-storage/api-cli/using-conditional-writes/). +If-None-Match is applied at multipart completion. The lock pins boto3 1.40.18 and +botocore 1.40.76; the actual botocore service model exposes IfNoneMatch. Synthetic +provider tests cover lost completion response, abort of own pending upload, +changed versions, corrupted readback, scope/hash binding, and node-loss retrieval. + +Email-connect has a separate archive profile in the platform helper. That is a +transport capability only: this request neither captures its SQLite ledger nor +admits an email worker. EMAIL-WP-0006 deployment evidence is complete; ongoing +ledger recovery stays tracked by CSOC-WP-0010-T04/T05 and requires a separate +producer/custody/rehearsal contract. diff --git a/requirements/README.md b/requirements/README.md new file mode 100644 index 0000000..5b31855 --- /dev/null +++ b/requirements/README.md @@ -0,0 +1,27 @@ +# Primary application archive runtime + +Use a dedicated platform-managed backup virtual environment. The app image does +not need boto3. backup-primary.in pins the boto3/botocore pair needed for conditional +multipart completion; backup-primary.lock also freezes all transitive dependencies +and accepted package hashes. Reviewed pair: boto3 1.40.18 / botocore 1.40.76. + +After admission, prepare the runtime before a capture window: + +```bash +uv venv /opt/railiance-platform/backup-venv +uv pip sync --python /opt/railiance-platform/backup-venv/bin/python \ + --require-hashes requirements/backup-primary.lock +``` + +These are installation instructions, not an instruction to alter a running +workload or enable a schedule. Do not resolve dependencies during a scheduled +backup. Node/tool/actor admission remains RPF-WP-0053. Deploy these requirements +with the reviewed scripts and record installed hashes. Credentials are delivered +separately by the owner; no values belong in this directory. + +An offline check with synthetic environment fields on 2026-10-07 proved the exact +installed botocore model exposes CompleteMultipartUpload.IfNoneMatch, without +provider contact. A presence check does not prove actual custody or provider +permissions. Requirements were compiled in /tmp; the comments preserve that +command's provenance. Regenerate from this repository's requirements/*.in with +uv pip compile --generate-hashes and review the new lock before installation. diff --git a/requirements/backup-primary.in b/requirements/backup-primary.in new file mode 100644 index 0000000..915fb0e --- /dev/null +++ b/requirements/backup-primary.in @@ -0,0 +1,2 @@ +boto3==1.40.18 +botocore==1.40.76 diff --git a/requirements/backup-primary.lock b/requirements/backup-primary.lock new file mode 100644 index 0000000..ff0e6ad --- /dev/null +++ b/requirements/backup-primary.lock @@ -0,0 +1,35 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile --generate-hashes --output-file /tmp/backup-primary.lock /tmp/backup-primary.in +boto3==1.40.18 \ + --hash=sha256:64301d39adecc154e3e595eaf0d4f28998ef0a5551f1d033aeac51a9e1a688e5 \ + --hash=sha256:daa776ba1251a7458c9d6c7627873d0c2460c8e8272d35759065580e9193700a + # via -r /tmp/backup-primary.in +botocore==1.40.76 \ + --hash=sha256:2b16024d68b29b973005adfb5039adfe9099ebe772d40a90ca89f2e165c495dc \ + --hash=sha256:fe425d386e48ac64c81cbb4a7181688d813df2e2b4c78b95ebe833c9e868c6f4 + # via + # -r /tmp/backup-primary.in + # boto3 + # s3transfer +jmespath==1.1.0 \ + --hash=sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d \ + --hash=sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64 + # via + # boto3 + # botocore +python-dateutil==2.9.0.post0 \ + --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \ + --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 + # via botocore +s3transfer==0.13.1 \ + --hash=sha256:a981aa7429be23fe6dfc13e80e4020057cbab622b08c0315288758d67cabc724 \ + --hash=sha256:c3fdba22ba1bd367922f27ec8032d6a1cf5f10c934fb5d68cf60fd5a23d936cf + # via boto3 +six==1.17.0 \ + --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \ + --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81 + # via python-dateutil +urllib3==2.8.0 \ + --hash=sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3 \ + --hash=sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63 + # via botocore diff --git a/scripts/application_archive_retention_plan.py b/scripts/application_archive_retention_plan.py new file mode 100644 index 0000000..ac84281 --- /dev/null +++ b/scripts/application_archive_retention_plan.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""Review primary archive generations; no provider or local deletion capability.""" + +from __future__ import annotations + +import argparse +import json +import re +from datetime import UTC, datetime +from pathlib import Path + +from primary_application_archive import PROFILES, require_version, validate_key + + +def identity(receipt): + return { + "object_key": receipt["object_key"], + "version_id": receipt["version_id"], + "ciphertext_sha256": receipt["ciphertext_sha256"], + "ciphertext_bytes": receipt["ciphertext_bytes"], + } + + +def plan(application, inventory, recovery, now=None): + now = now or datetime.now(UTC) + if ( + recovery.get("status") != "application_restored_verified" + or recovery.get("application") != application + or recovery.get("application_restore_verified") is not True + ): + raise ValueError("attended_application_recovery_proof_required") + protected = recovery.get("object_key", "") + validate_key(application, protected) + require_version(recovery.get("version_id")) + items, ignored = {}, [] + for receipt in inventory: + key = receipt.get("object_key", "") + if ( + receipt.get("application") != application + or receipt.get("status") != "offsite_fetched_verified" + or receipt.get("remote_retrieval_verified") is not True + or not isinstance(key, str) + or not key.startswith(PROFILES[application]) + ): + ignored.append(key) + continue + validate_key(application, key) + require_version(receipt.get("version_id")) + captured = datetime.fromisoformat(receipt["captured_at"].replace("Z", "+00:00")) + if ( + captured.tzinfo is None + or captured > now + or key in items + or not re.fullmatch(r"[a-f0-9]{64}", receipt.get("ciphertext_sha256", "")) + or not isinstance(receipt.get("ciphertext_bytes"), int) + or receipt["ciphertext_bytes"] <= 0 + ): + raise ValueError("unverified_archive_inventory") + items[key] = (captured.astimezone(UTC), receipt) + if ( + protected not in items + or items[protected][1]["version_id"] != recovery["version_id"] + or items[protected][1]["ciphertext_sha256"] != recovery.get("ciphertext_sha256") + ): + raise ValueError("protected_recovery_point_missing_or_changed") + ordered = sorted(items, key=lambda key: (items[key][0], key), reverse=True) + daily, weekly = {}, {} + for key in ordered: + stamp = items[key][0] + daily.setdefault(stamp.date(), key) + weekly.setdefault(stamp.isocalendar()[:2], key) + keep = {protected, ordered[0]} | set(list(daily.values())[:14]) | set(list(weekly.values())[:4]) + return { + "schema": "platform.application-retention-review.v1", + "status": "reviewed", + "application": application, + "daily_generations": 14, + "weekly_generations": 4, + "protected": identity(items[protected][1]), + "keep": [identity(items[key][1]) for key in ordered if key in keep], + "expiration_candidates": [identity(items[key][1]) for key in ordered if key not in keep], + "ignored": ignored, + "provider_mutations": False, + "local_mutations": False, + "activation_gate": "Separate owner must admit lifecycle/capacity, revalidate protected " + "recovery and exact versions before any expiration.", + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--application", choices=PROFILES, required=True) + parser.add_argument("--inventory", type=Path, required=True) + parser.add_argument("--recovery-receipt", type=Path, required=True) + args = parser.parse_args() + print( + json.dumps( + plan( + args.application, + json.loads(args.inventory.read_text()), + json.loads(args.recovery_receipt.read_text()), + ), + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except Exception as error: + print(json.dumps({"status": "failed", "failure_type": type(error).__name__})) + raise SystemExit(1) diff --git a/scripts/primary_application_archive.py b/scripts/primary_application_archive.py new file mode 100644 index 0000000..3976511 --- /dev/null +++ b/scripts/primary_application_archive.py @@ -0,0 +1,375 @@ +#!/usr/bin/env python3 +"""Immutable application archives through the platform's primary storage. + +Credentials are owner-delivered environment fields, never read from another +workload's Secret. CLI transfer previews by default. Output is redacted. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import secrets +import stat +import tempfile +from datetime import UTC, datetime +from pathlib import Path + +ENDPOINT = "https://s3.nl-ams.scw.cloud" +BUCKET = "railiance-platform-pg-backup" +PROFILES = { + "coulomb-social": "platform-pg/application-archives/coulomb-social/", + "email-connect": "platform-pg/application-archives/email-connect/", +} +PART_SIZE = 64 * 1024**2 +MAX_BYTES = 20 * 1024**3 + + +def private_directory(path): + info = path.lstat() + if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.geteuid() or info.st_mode & 0o077: + raise ValueError("private_owner_directory_required") + + +def regular_file(path): + info = path.lstat() + if not stat.S_ISREG(info.st_mode) or info.st_uid != os.geteuid() or info.st_mode & 0o077: + raise ValueError("private_owner_file_required") + + +def digest(path): + with path.open("rb") as source: + return hashlib.file_digest(source, "sha256").hexdigest() + + +def checkpoint(path, receipt): + fd, name = tempfile.mkstemp(prefix="archive-receipt-", dir=path.parent) + with os.fdopen(fd, "w") as output: + json.dump(receipt, output, indent=2) + output.flush() + os.fsync(output.fileno()) + Path(name).replace(path) + + +def key_for(application, captured_at): + stamp = datetime.fromisoformat(captured_at.replace("Z", "+00:00")) + if stamp.tzinfo is None: + raise ValueError("capture_timezone_required") + stamp = stamp.astimezone(UTC) + return ( + PROFILES[application] + + stamp.strftime("%Y/%m/%d/%Y%m%dT%H%M%SZ-") + + secrets.token_hex(12) + + ".tar.age" + ) + + +def validate_key(application, key): + suffix = key.removeprefix(PROFILES[application]) + if not key.startswith(PROFILES[application]) or not re.fullmatch( + r"\d{4}/\d{2}/\d{2}/\d{8}T\d{6}Z-[a-f0-9]{24}\.tar\.age", suffix + ): + raise ValueError("archive_key_outside_contract") + + +def missing(error): + response = getattr(error, "response", {}) + return response.get("ResponseMetadata", {}).get("HTTPStatusCode") == 404 + + +def require_version(value): + if ( + not isinstance(value, str) + or not value + or value == "null" + or any(c in value for c in "\r\n") + ): + raise ValueError("versioned_archive_required") + return value + + +def existing(client, key, expected, size): + try: + head = client.head_object(Bucket=BUCKET, Key=key) + except Exception as error: + if missing(error): + return None + raise + if head.get("ContentLength") != size or head.get("Metadata", {}).get("sha256") != expected: + raise ValueError("existing_archive_binding_mismatch") + return require_version(head.get("VersionId")) + + +def transfer(client, application, source, output, receipt, save=lambda: None): + regular_file(source) + private_directory(output.parent) + size = source.stat().st_size + if not 0 < size <= MAX_BYTES or output.exists() or output.is_symlink(): + raise ValueError("invalid_archive_source_or_output") + with source.open("rb") as stream: + if stream.read(22) != b"age-encryption.org/v1\n": + raise ValueError("encrypted_archive_required") + expected = digest(source) + key = receipt["object_key"] + validate_key(application, key) + if receipt.get("ciphertext_sha256", expected) != expected: + raise ValueError("archive_receipt_binding_mismatch") + receipt.update( + application=application, + destination="s3://" + BUCKET + "/" + key, + ciphertext_sha256=expected, + ciphertext_bytes=size, + remote_retrieval_verified=False, + stage="existing_object_check", + ) + save() + version = existing(client, key, expected, size) + if receipt.get("version_id") and receipt["version_id"] != version: + raise ValueError("existing_archive_version_changed") + upload_id = None + if version is None: + try: + receipt["stage"] = "multipart_upload" + save() + upload_id = client.create_multipart_upload( + Bucket=BUCKET, + Key=key, + ContentType="application/octet-stream", + Metadata={"sha256": expected, "application": application}, + )["UploadId"] + parts = [] + with source.open("rb") as stream: + while block := stream.read(PART_SIZE): + number = len(parts) + 1 + result = client.upload_part( + Bucket=BUCKET, Key=key, UploadId=upload_id, PartNumber=number, Body=block + ) + parts.append({"PartNumber": number, "ETag": result["ETag"]}) + completed = client.complete_multipart_upload( + Bucket=BUCKET, + Key=key, + UploadId=upload_id, + MultipartUpload={"Parts": parts}, + IfNoneMatch="*", + ) + upload_id = None + version = require_version(completed.get("VersionId")) + receipt["multipart_completed"] = True + finally: + if upload_id: + try: + client.abort_multipart_upload(Bucket=BUCKET, Key=key, UploadId=upload_id) + receipt["multipart_aborted"] = True + except Exception: + receipt["multipart_aborted"] = False + save() + else: + receipt["reused_verified_binding"] = True + receipt.update(stage="version_pinned_readback", version_id=version) + save() + download(client, key, version, size, expected, output) + receipt.update( + status="offsite_fetched_verified", + stage="transfer_verified", + remote_retrieval_verified=True, + immutable_create=True, + ) + save() + return receipt + + +def download(client, key, version, size, expected, output): + private_directory(output.parent) + if output.exists() or output.is_symlink() or not 0 < size <= MAX_BYTES: + raise ValueError("fresh_bounded_readback_required") + response = client.get_object(Bucket=BUCKET, Key=key, VersionId=version) + if response["ContentLength"] != size or response.get("VersionId") != version: + response["Body"].close() + raise ValueError("readback_version_or_length_mismatch") + read_bytes = 0 + with response["Body"] as body, output.open("xb") as target: + output.chmod(0o600) + while block := body.read(8 * 1024**2): + read_bytes += len(block) + if read_bytes > size: + raise ValueError("readback_exceeds_expected_size") + target.write(block) + if read_bytes != size or digest(output) != expected: + raise ValueError("readback_digest_mismatch") + + +def retrieve(client, application, output, receipt): + """Node-loss recovery needs only the independently retained public receipt.""" + key = receipt.get("object_key", "") + validate_key(application, key) + version = require_version(receipt.get("version_id")) + expected = receipt.get("ciphertext_sha256", "") + size = receipt.get("ciphertext_bytes", 0) + if ( + receipt.get("schema") != "platform.application-primary-archive.v1" + or receipt.get("application") != application + or receipt.get("status") != "offsite_fetched_verified" + or receipt.get("remote_retrieval_verified") is not True + or receipt.get("destination") != "s3://" + BUCKET + "/" + key + or not re.fullmatch(r"[a-f0-9]{64}", expected) + or not isinstance(size, int) + or isinstance(size, bool) + or not 0 < size <= MAX_BYTES + ): + raise ValueError("verified_recovery_receipt_required") + download(client, key, version, size, expected, output) + return { + "status": "retrieved_verified", + "application": application, + "object_key": key, + "version_id": version, + "ciphertext_sha256": expected, + "ciphertext_bytes": size, + "application_restore_verified": False, + "provider_mutations": False, + } + + +def runtime_client(): + import boto3 + from botocore.config import Config + + key = os.environ.get("AWS_ACCESS_KEY_ID", "").strip() + secret = os.environ.get("AWS_SECRET_ACCESS_KEY", "").strip() + if not key or not secret: + raise ValueError("owner_delivered_primary_custody_required") + return boto3.client( + "s3", + endpoint_url=ENDPOINT, + region_name="nl-ams", + aws_access_key_id=key, + aws_secret_access_key=secret, + config=Config( + signature_version="s3v4", + connect_timeout=15, + read_timeout=180, + retries={"max_attempts": 3}, + s3={"addressing_style": "path"}, + request_checksum_calculation="when_required", + response_checksum_validation="when_required", + ), + ) + + +def check_runtime(): + from botocore.session import get_session + + model = get_session().get_service_model("s3").operation_model("CompleteMultipartUpload") + if "IfNoneMatch" not in model.input_shape.members: + raise ValueError("conditional_multipart_sdk_required") + if not all( + os.environ.get(name, "").strip() for name in ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] + ): + raise ValueError("owner_delivered_primary_custody_required") + return { + "status": "configured", + "credential_values_printed": False, + "provider_contacted": False, + "conditional_multipart_sdk": True, + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=["check", "transfer", "retrieve"]) + parser.add_argument("--application", choices=PROFILES, required=True) + parser.add_argument("--source", type=Path) + parser.add_argument("--output", type=Path) + parser.add_argument("--receipt", type=Path) + parser.add_argument("--captured-at") + parser.add_argument("--execute", action="store_true") + args = parser.parse_args() + if args.action == "check": + print(json.dumps(check_runtime())) + return 0 + if not args.execute: + print( + json.dumps( + { + "status": "reviewed", + "provider_contacted": False, + "bucket": BUCKET, + "prefix": PROFILES[args.application], + "immutable_create": True, + } + ) + ) + return 0 + if args.action == "retrieve": + if not args.output or not args.receipt: + raise ValueError("retrieval_arguments_required") + regular_file(args.receipt) + receipt = json.loads(args.receipt.read_text()) + print(json.dumps(retrieve(runtime_client(), args.application, args.output, receipt))) + return 0 + if not all([args.source, args.output, args.receipt, args.captured_at]): + raise ValueError("transfer_arguments_required") + regular_file(args.source) + private_directory(args.output.parent) + private_directory(args.receipt.parent) + if args.output.is_symlink() or args.receipt.is_symlink(): + raise ValueError("linked_transfer_path_refused") + if args.receipt.exists(): + regular_file(args.receipt) + receipt = json.loads(args.receipt.read_text()) + if ( + receipt.get("schema") != "platform.application-primary-archive.v1" + or receipt.get("application") != args.application + or receipt.get("captured_at") != args.captured_at + ): + raise ValueError("transfer_receipt_scope_mismatch") + else: + receipt = { + "schema": "platform.application-primary-archive.v1", + "status": "pending", + "application": args.application, + "captured_at": args.captured_at, + "object_key": key_for(args.application, args.captured_at), + } + validate_key(args.application, receipt["object_key"]) + if args.output.exists(): + regular_file(args.output) + if ( + receipt.get("status") == "offsite_fetched_verified" + and receipt.get("remote_retrieval_verified") + and receipt.get("immutable_create") + and require_version(receipt.get("version_id")) + and args.source.stat().st_size == receipt.get("ciphertext_bytes") + and digest(args.output) == receipt.get("ciphertext_sha256") + and digest(args.source) == receipt.get("ciphertext_sha256") + ): + print(json.dumps(receipt)) + return 0 + raise ValueError("unverified_existing_readback_requires_review") + try: + checkpoint(args.receipt, receipt) + transfer( + runtime_client(), + args.application, + args.source, + args.output, + receipt, + lambda: checkpoint(args.receipt, receipt), + ) + except Exception as error: + receipt.update(status="failed", failure_type=type(error).__name__) + checkpoint(args.receipt, receipt) + raise + print(json.dumps(receipt)) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except Exception as error: + print(json.dumps({"status": "failed", "failure_type": type(error).__name__})) + raise SystemExit(1) diff --git a/tests/test_application_archive_retention_plan.py b/tests/test_application_archive_retention_plan.py new file mode 100644 index 0000000..766e43b --- /dev/null +++ b/tests/test_application_archive_retention_plan.py @@ -0,0 +1,90 @@ +"""Retention proposals preserve known recovered points and never own expiration.""" + +import importlib.util +import sys +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + + +@pytest.fixture +def planner(monkeypatch): + scripts = Path(__file__).resolve().parents[1] / "scripts" + monkeypatch.syspath_prepend(str(scripts)) + spec = importlib.util.spec_from_file_location( + "application_archive_retention_plan", scripts / "application_archive_retention_plan.py" + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + sys.modules.pop("primary_application_archive", None) + return module + + +def inventory(planner): + now = datetime(2027, 1, 12, 3, 15, tzinfo=UTC) + result = [] + for day in range(70): + stamp = now - timedelta(days=day) + result.append( + { + "application": "coulomb-social", + "status": "offsite_fetched_verified", + "remote_retrieval_verified": True, + "captured_at": stamp.isoformat(), + "object_key": planner.PROFILES["coulomb-social"] + + stamp.strftime("%Y/%m/%d/%Y%m%dT%H%M%SZ-") + + f"{day:024x}" + + ".tar.age", + "version_id": "version-" + str(day), + "ciphertext_sha256": "a" * 64, + "ciphertext_bytes": 5000, + } + ) + recovery = dict( + result[-1], status="application_restored_verified", application_restore_verified=True + ) + return now, result, recovery + + +def test_protects_old_recovered_point_and_deduplicates_daily_weekly(planner): + now, records, recovery = inventory(planner) + records.append(dict(records[0], application="email-connect", object_key="foreign")) + records.append(dict(records[0], status="failed", object_key="failed")) + result = planner.plan("coulomb-social", records, recovery, now) + keys = [item["object_key"] for item in result["keep"]] + assert len(keys) == len(set(keys)) <= 19 + assert records[0]["object_key"] in keys and recovery["object_key"] in keys + assert all(item["object_key"] in keys for item in records[:14]) + assert result["ignored"] == ["foreign", "failed"] + assert result["expiration_candidates"] + assert not result["provider_mutations"] and not result["local_mutations"] + assert result["protected"]["version_id"] == recovery["version_id"] + + +@pytest.mark.parametrize("change", ["missing", "version", "hash", "unproved"]) +def test_refuses_missing_changed_or_unproved_application_recovery(planner, change): + now, records, recovery = inventory(planner) + if change == "missing": + records.pop() + elif change == "version": + recovery["version_id"] = "different-version" + elif change == "hash": + recovery["ciphertext_sha256"] = "b" * 64 + else: + recovery["application_restore_verified"] = False + with pytest.raises(ValueError): + planner.plan("coulomb-social", records, recovery, now) + + +@pytest.mark.parametrize("change", ["duplicate", "future", "unversioned"]) +def test_refuses_ambiguous_inventory(planner, change): + now, records, recovery = inventory(planner) + if change == "duplicate": + records.append(dict(records[0])) + elif change == "future": + records[0]["captured_at"] = (now + timedelta(days=1)).isoformat() + else: + records[0]["version_id"] = "null" + with pytest.raises(ValueError): + planner.plan("coulomb-social", records, recovery, now) diff --git a/tests/test_primary_application_archive.py b/tests/test_primary_application_archive.py new file mode 100644 index 0000000..3702adc --- /dev/null +++ b/tests/test_primary_application_archive.py @@ -0,0 +1,204 @@ +"""Immutable archive transfer with a synthetic provider; no live credentials.""" + +import hashlib +import importlib.util +import io +from pathlib import Path + +import pytest + + +@pytest.fixture +def archive(): + spec = importlib.util.spec_from_file_location( + "primary_application_archive", + Path(__file__).resolve().parents[1] / "scripts/primary_application_archive.py", + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class NotFound(Exception): + response = {"ResponseMetadata": {"HTTPStatusCode": 404}} + + +class Provider: + def __init__(self, data, failure=None, exists=False): + self.data, self.failure, self.exists = data, failure, exists + self.calls = [] + self.version = "synthetic-version-1" + + def head_object(self, **kwargs): + self.calls.append(("head", kwargs)) + if self.failure == "permission": + raise PermissionError("synthetic provider failure") + if not self.exists: + raise NotFound() + return { + "ContentLength": len(self.data), + "VersionId": self.version, + "Metadata": {"sha256": hashlib.sha256(self.data).hexdigest()}, + } + + def create_multipart_upload(self, **kwargs): + self.calls.append(("start", kwargs)) + return {"UploadId": "own-unfinished-upload"} + + def upload_part(self, **kwargs): + self.calls.append(("part", kwargs)) + if self.failure == "part": + raise TimeoutError("synthetic timeout") + assert kwargs["Body"] == self.data + return {"ETag": "synthetic-part-etag"} + + def complete_multipart_upload(self, **kwargs): + self.calls.append(("complete", kwargs)) + assert kwargs["IfNoneMatch"] == "*" + self.exists = True + if self.failure == "lost-response": + raise TimeoutError("synthetic lost acknowledgement") + return {"VersionId": None if self.failure == "unversioned" else self.version} + + def abort_multipart_upload(self, **kwargs): + self.calls.append(("abort", kwargs)) + assert kwargs["UploadId"] == "own-unfinished-upload" + + def get_object(self, **kwargs): + self.calls.append(("get", kwargs)) + assert kwargs["VersionId"] == self.version + data = b"X" * len(self.data) if self.failure == "corrupt" else self.data + if self.failure == "oversize": + data += b"extra" + return { + "ContentLength": len(self.data), + "VersionId": "other-version" if self.failure == "wrong-version" else self.version, + "Body": io.BytesIO(data), + } + + +def fixture(archive, tmp_path, failure=None, exists=False): + tmp_path.chmod(0o700) + source = tmp_path / "packet.tar.age" + source.write_bytes(b"age-encryption.org/v1\nsynthetic-encrypted-pair") + source.chmod(0o600) + receipt = {"object_key": archive.key_for("coulomb-social", "2026-10-07T20:00:00Z")} + return source, tmp_path / "fetched.age", receipt, Provider(source.read_bytes(), failure, exists) + + +def test_conditional_create_and_version_pinned_readback(archive, tmp_path): + source, output, receipt, client = fixture(archive, tmp_path) + saved = [] + result = archive.transfer( + client, "coulomb-social", source, output, receipt, lambda: saved.append(dict(receipt)) + ) + assert result["status"] == "offsite_fetched_verified" + assert result["remote_retrieval_verified"] and result["immutable_create"] + assert result["version_id"] == client.version + assert output.read_bytes() == source.read_bytes() + assert output.stat().st_mode & 0o777 == 0o600 + assert [call[0] for call in client.calls] == ["head", "start", "part", "complete", "get"] + assert not saved[0]["remote_retrieval_verified"] + assert saved[-1]["remote_retrieval_verified"] + + +@pytest.mark.parametrize( + "failure", + ["part", "lost-response", "unversioned", "corrupt", "oversize", "wrong-version", "permission"], +) +def test_failure_never_declares_retrieval_or_deletes_retained_objects(archive, tmp_path, failure): + source, output, receipt, client = fixture(archive, tmp_path, failure) + with pytest.raises((ValueError, TimeoutError, PermissionError)): + archive.transfer(client, "coulomb-social", source, output, receipt) + assert not receipt["remote_retrieval_verified"] + assert not any(name == "delete" for name, _ in client.calls) + if failure in {"part", "lost-response"}: + assert len([call for call in client.calls if call[0] == "abort"]) == 1 + else: + assert not any(name == "abort" for name, _ in client.calls) + + +def test_uncertain_completion_retries_readback_without_another_upload(archive, tmp_path): + source, output, receipt, client = fixture(archive, tmp_path, "lost-response") + with pytest.raises(TimeoutError): + archive.transfer(client, "coulomb-social", source, output, receipt) + key = receipt["object_key"] + client.failure = None + archive.transfer(client, "coulomb-social", source, output, receipt) + assert receipt["object_key"] == key and receipt["reused_verified_binding"] + assert len([call for call in client.calls if call[0] == "start"]) == 1 + assert receipt["remote_retrieval_verified"] + + +def test_existing_key_must_bind_bytes_and_previous_version(archive, tmp_path): + source, output, receipt, client = fixture(archive, tmp_path, exists=True) + client.data = b"other ciphertext" + with pytest.raises(ValueError, match="existing_archive_binding_mismatch"): + archive.transfer(client, "coulomb-social", source, output, receipt) + assert [name for name, _ in client.calls] == ["head"] + client.data = source.read_bytes() + receipt["version_id"] = "previous-version" + with pytest.raises(ValueError, match="existing_archive_version_changed"): + archive.transfer(client, "coulomb-social", source, output, receipt) + assert not output.exists() + + +@pytest.mark.parametrize( + "invalid", ["plaintext", "scope", "receipt-hash", "linked-source", "linked-output"] +) +def test_invalid_packet_refuses_before_provider_contact(archive, tmp_path, invalid): + source, output, receipt, client = fixture(archive, tmp_path) + if invalid == "plaintext": + source.write_bytes(b"private unencrypted data") + elif invalid == "scope": + receipt["object_key"] = archive.key_for("email-connect", "2026-10-07T20:00:00Z") + elif invalid == "receipt-hash": + receipt["ciphertext_sha256"] = "bad" + elif invalid == "linked-source": + target = tmp_path / "real.age" + source.rename(target) + source.symlink_to(target) + else: + output.symlink_to(tmp_path / "absent") + with pytest.raises(ValueError): + archive.transfer(client, "coulomb-social", source, output, receipt) + assert client.calls == [] + + +def test_retrieve_after_source_loss_uses_saved_version_without_provider_writes(archive, tmp_path): + source, output, receipt, client = fixture(archive, tmp_path) + receipt.update( + schema="platform.application-primary-archive.v1", captured_at="2026-10-07T20:00:00Z" + ) + archive.transfer(client, "coulomb-social", source, output, receipt) + source.unlink() + output.unlink() + client.calls.clear() + result = archive.retrieve(client, "coulomb-social", output, receipt) + assert result["status"] == "retrieved_verified" + assert not result["provider_mutations"] and not result["application_restore_verified"] + assert output.read_bytes() == client.data + assert [name for name, _ in client.calls] == ["get"] + + +@pytest.mark.parametrize( + "field,value", + [ + ("remote_retrieval_verified", False), + ("destination", "s3://another-bucket/key"), + ("ciphertext_sha256", "bad"), + ("version_id", "null"), + ("ciphertext_bytes", -1), + ("application", "email-connect"), + ], +) +def test_retrieval_rejects_unbound_receipt_before_contact(archive, tmp_path, field, value): + source, output, receipt, client = fixture(archive, tmp_path) + receipt.update(schema="platform.application-primary-archive.v1") + archive.transfer(client, "coulomb-social", source, output, receipt) + output.unlink() + client.calls.clear() + receipt[field] = value + with pytest.raises(ValueError): + archive.retrieve(client, "coulomb-social", output, receipt) + assert client.calls == [] diff --git a/workplans/RPF-WP-0053-coulomb-paired-backup-admission.md b/workplans/RPF-WP-0053-coulomb-paired-backup-admission.md index e1bae93..e5e5539 100644 --- a/workplans/RPF-WP-0053-coulomb-paired-backup-admission.md +++ b/workplans/RPF-WP-0053-coulomb-paired-backup-admission.md @@ -27,7 +27,7 @@ No credential was fetched, no timer enabled, no live upload/overwrite performed. ```task id: RPF-WP-0053-T01 -status: todo +status: progress priority: high state_hub_task_id: "ceb58abf-77ef-5d75-a65b-fc67bfd9aa54" ``` @@ -48,19 +48,41 @@ id: RPF-WP-0053-T02 status: wait priority: high needs_human: true -intervention_note: "Platform must admit the backup workload and exact Nextcloud prefix/slot overwrite contract, and deliver bounded caller custody before live activation." +intervention_note: "Platform must admit the exact Scaleway primary archive caller/prefix, reconcile its current custody front door and retention/capacity, and deliver bounded credentials before activation." state_hub_task_id: "b06f1a44-1a55-588c-9c6c-3c1e290544df" ``` -Reuse governed railiance-backup-offsite-lane, OpenBao path -platform/workloads/railiance/backup/offsite-lane. Admit independent prefix -coulomb-social and 14 daily + 4 weekly encrypted fixed slots; prove directory -existence, PUT replacement and recoverable retrieval. Prepared node consumer -expects root-owned mode-0600 /etc/coulomb-social-backup/offsite.env with -RAILIANCE_BACKUP_NC_TOKEN and RAILIANCE_BACKUP_NC_WEBDAV_URL delivered by the -custody owner; public age recipient is inherited from the platform helper. -Do not put AGE_PRIVATE_KEY in the scheduled worker. Decide delivery/rotation -using the owner contract, rather than copying another workload's Secret. +The original Nextcloud overwrite-slot proposal is withdrawn; its current workload +uploader is create-only and secondary. Reuse the platform primary contract: +s3://railiance-platform-pg-backup/platform-pg/application-archives/coulomb-social/, +Standard Multi-AZ nl-ams. Exact review package is +docs/credential-lane-designs/coulomb-paired-backup-primary.md. + +scripts/primary_application_archive.py implements conditional immutable multipart +completion, own unfinished-upload abort, version-pinned GET and ciphertext hash +verification. Independent retrieve needs only a retained public receipt, no +original archive/node. requirements/backup-primary.lock pins seven packages with +hashes in a dedicated backup-venv; SDK conditional-completion shape is verified. +Synthetic provider tests cover uncertain completion, replay without new upload, +corrupted bytes, changed versions, caller scope and node-loss retrieval. No live +provider rights or custody are inferred from these tests. + +CCR-2026-0012 identifies existing primary custody coordinates and its approved +rapp-postgres/databases delivery, not a new consumer. Its historical catalog id +backup-object-storage is unknown to the current local warden CLI. Owner must +reconcile the front door, choose/admit appropriately prefix-scoped IAM authority +and deliver AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY to root-owned mode 0600 +/etc/coulomb-social-backup/primary.env (parent mode 0700). Metadata-only node check found +that file absent. No Secret fetch/copy or namespace expansion occurred. +No bootstrap key or age identity belongs in the scheduled worker. + +Separate non-mutating application_archive_retention_plan.py proposes 14 daily/4 +weekly distinct generations and protects newest plus the exact last application- +recovered version/hash. A missing/changed proof refuses; foreign/unverified points +are ignored. This is a review, not a deletion executor or activated lifecycle. +Bind local/provider headroom, independently retained metadata receipts, protected +recovery and compatibility with the existing 30-day bucket lifecycle before +activation. No retained-object/version-delete rights belong in the publisher. ## T03 — Activate and verify retrieval and application recovery @@ -74,13 +96,14 @@ state_hub_task_id: "871c14da-2940-5be6-8ce8-953973422064" ``` Founder ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED, plus -explicit offsite prefix write/overwrite authority. Proposed UTC daily03:15 +explicit primary archive creation/version-read authority. Proposed UTC daily03:15 (+up to5min jitter), RPO target24h, hourly stale threshold26h, provisional RTO target60min for an admitted attended restore. These are proposal targets, not measured promises. First capture/upload is bounded and app resumes before -encryption/upload. Retrieve ciphertext independently, compare encrypted hash, -decrypt through recovery escrow into a private isolated workspace, unpack only -database.dump/content.tar.gz/receipt.json, run app paired restore and compare +encryption/upload. Retrieve using the independently retained public receipt, +compare encrypted hash, decrypt through recovery +escrow with consumer recover-paired-backup.py (strict three-file packet/hash and +capture/image binding), then run restore-paired-snapshot.py and compare row/sequence/file hashes, actual rendering and denial. Record receipt paths, counts/timing only; preserve the source recovery point on any failed check. No production data restore is part of backup schedule activation. @@ -99,5 +122,16 @@ established evidence owner (RPF-WP-0036-T03/T04), prove reception and failure response; local journald failure alone is not monitored acceptance. Name the recovery custodian and approve a monthly isolated drill with offsite readback. Verify key recovery survives node/workstation loss. Record release-triggered -backup helper digest refresh and retention-slot policy. CSOC-WP-0008-T02 stays -progress until operating evidence and service contract are accepted. +backup helper digest refresh and admitted generation/capacity/retention policy. +CSOC-WP-0008-T02 stays progress until operating evidence and service contract are accepted. + +## Implementation evidence — 2026-10-07 + +New primary/retention 30 cases and existing backup tier/archive lifecycle 23 cases +pass (53 total); no live provider/custody interaction. Consumer full make check +passes 150 tests/mandatory browser journeys and Ruff; real synthetic age recovery +roundtrip passes and all four systemd units verify. T01 is progress for the +prepared execution package; owner identity acceptance remains open. T02/T03 remain +wait for custody/admission and exact activation; no timer installed/enabled and +no productive ciphertext uploaded. Email-connect's separate transport profile +is not ledger capture/recovery coverage; CSOC-WP-0010-T04/T05 own that residual.