diff --git a/docs/release/backup-source-installation-execution-review-2026-10-31.md b/docs/release/backup-source-installation-execution-review-2026-10-31.md new file mode 100644 index 0000000..b003566 --- /dev/null +++ b/docs/release/backup-source-installation-execution-review-2026-10-31.md @@ -0,0 +1,112 @@ +# Supervised inactive backup-source installation + +The final wrapper is prepared for CSOC-WP-0012-T08. It installs the previously +sealed22-member archive into its exact19 target paths, using the existing +transaction library. It neither captures data nor starts/enables a service or +timer. Current native preflight **refuses**; do not execute this packet yet. + +Proposal: +`sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42`. + +The digest binds launcher/runtime/transaction/owner-verifier sources, archive and +manifest, native target preimages, cluster identity, the fixed capacity profile +and the existing telemetry-route execution packet4d848106. No owner route source +or frozen SMTP admission packet was changed. + +## Native findings and disposition + +Read-only preflight holds the existing backup and snapshot locks without replacing +or truncating them; verifies primary cluster identity, inactive/disabled units, +absence of queued starts, and the exact19 target preimages. It then refuses +`unsafe_state_entry` in the existing capacity guard. No installation attempt or +target replacement was created. + +Metadata inspection found eight exceptions inside the backup state tree: +four root-owned0755 release/capsule directories, and four date-operation private +entries owned by helper UID/GID10001. The latter are the private directory, +result.json, journal.json and execute.lock. Their bodies were not read. These +are historical operation evidence, not newly detected backup corruption. +The private ancestors matter to exposure; this observation alone is not a claim +that their contents were publicly accessible. + +Exact metadata is in `backup-state-custody-observation-2026-10-31.json`. +T04 owns reviewed custody/disposition compatible with the capacity guard. Do not +chmod/chown/delete/move these historical records opportunistically, skip them +from capacity accounting, or loosen the guard to force installation through. +Retention, original evidence paths and the terminated helper's ownership contract +must be considered in that disposition. No disposition was executed here. + +The separate telemetry-route receipt is still absent at this observation. +T07's existing founder command remains unchanged. Successful route execution and +independent native verification are required before this installation can pass. +The installer reuses the pinned owner verifier to compare the full release, +resources, runtime config, Ready image and preserved PVC, both before and after +installation. A past receipt without the current native checks is insufficient. + +## Reviewed operation and future terminal entry + +Offline proposal (no node access): + +```bash +python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py +``` + +Read-only preflight: + +```bash +python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py --preflight +``` + +Only after T04's custody disposition and T07's verified route, refresh preflight +and review any changed source/preimage digest. The founder's future installation +command is: + +```bash +python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py \ + --execute --expect-digest sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42 +``` + +It requires a terminal and `INSTALL` followed by the full digest. That exact +confirmation includes admission of the displayed1GiB new-file limit,8GiB local +state budget,8GiB free-space floor,1024-inode minimum and16MiB control margin. +Capacity admission here does not resolve rolling provider retention, expiry or +historical-copy deletion. Existing node/custody/destination/schedule permissions +are preserved; no new schedule or notification destination is being requested. + +The transaction preserves shared platform helpers and the current snapshot pin. +It saves preimages, stages files and journals identities before replacing targets. +After daemon reload, every target is read back and units must remain inactive, +disabled/static and free of queued jobs. Dependency SDK versions and credential +file custody are checked without reading credential values. The outer receipt is +at `/var/lib/coulomb-social-backup-installations/20261031bacc0001/receipt.json`; +the nested `files/receipt.json` and preimages describe the file transaction. + +After interruption or a lost connection, inspect both receipts, process state and +native files before any further operation. Existing attempts refuse installation; +there is no automatic retry or rollback. The explicit founder rollback uses +`--rollback --expect-digest ` and confirmation `ROLLBACK `. +It takes the same locks, checks cluster/inactive state, refuses interfering file +changes, restores only its own journaled targets, reloads and rechecks inactivity. +Rollback retains evidence and does not require an operational email route. + +## Validation and acceptance boundary + +The disposable-filesystem tests exercise the real file transaction plus controller +ordering, locked preflight, source/preimage/refusal gates, interrupted reload, +preserved preimages, explicit rollback, route loss after installation, enabled/ +failed/foreign units, queued starts and nonterminal refusal. Real file-lock tests +prove both locks are exclusive and released after partial acquisition, preserve +their bytes/inodes, and reject symlinks. + +The full regression result and exact source hashes are in +`backup-supervised-installer-validation-2026-10-31.json`. The gate also exposed +formatting errors in two previously added mail preparation scripts; these were +corrected. The sender preparer's embedded remote Python AST is unchanged, and +the send-free return-mail rehearsal passed again. Historical evidence keeps its +original source hashes; the new validation binds the current formatted sources. + +T08 remains progress until actual installation and independent inactive-state +readback. T09 still needs native service/failure/retry/stale/producer-death proof; +T10 still needs activation and observation after its existing prerequisites. +No full service acceptance, mail delivery or migration verdict follows from +source preparation alone. diff --git a/docs/release/backup-source-installer-review-2026-10-31.md b/docs/release/backup-source-installer-review-2026-10-31.md index d71cc04..d01e56a 100644 --- a/docs/release/backup-source-installer-review-2026-10-31.md +++ b/docs/release/backup-source-installer-review-2026-10-31.md @@ -2,9 +2,10 @@ `scripts/backup_install.py` implements the file transaction needed to install the reviewed operating sources with units inactive. It is a library, with no CLI, -provider call, capture or timer activation. The exact supervised calling wrapper -and final installation approval are still required; importing this module does -not authorize a node mutation. +provider call, capture or timer activation. Its supervised calling wrapper is now +prepared in `backup-source-installation-execution-review-2026-10-31.md`; +prerequisite disposition and final attended installation remain pending. +Importing this module does not authorize a node mutation. The current sealed archiveeb5dd9ba / manifest84f67186 passes its strict reader: 22 regular tar members, including manifest and review evidence, with exactly19 @@ -58,7 +59,12 @@ preimages/staging references, symlink/hardlink refusal and unsafe parent modes. No production URL, identity, account or email send participates in these tests. Final source hashes and full regression results are recorded alongside this review. -The exact terminal wrapper must enforce the caller contract, journal a unique +The exact terminal wrapper is now implemented and reviewed in +`backup-source-installation-execution-review-2026-10-31.md`. Its native preflight +refuses historical state-tree custody exceptions; the owner route receipt also +remains absent. Neither prerequisite is bypassed and no installation occurred. + +The wrapper enforces the caller contract, journals a unique attempt, and verify disabled native units after reload. The first admitted native service/capture and received failure/pending/stale signals, restart/replay and producer-death detection remain follow-through. Existing capacity, retention, diff --git a/docs/release/backup-state-custody-observation-2026-10-31.json b/docs/release/backup-state-custody-observation-2026-10-31.json new file mode 100644 index 0000000..9302736 --- /dev/null +++ b/docs/release/backup-state-custody-observation-2026-10-31.json @@ -0,0 +1,89 @@ +{ + "schema": "coulomb.backup-state-custody-observation.v1", + "observed_at": "2026-10-31T11:11:43.981497+00:00", + "unsafe_entries": [ + { + "relative_path": "releases", + "kind": "directory", + "mode": "0o755", + "uid": 0, + "gid": 0, + "nlink": 4, + "bytes": 4096, + "inode": 543834 + }, + { + "relative_path": "releases/9f1daf6c5fb56b2b4fc9817f28f5ad090c884b0295f7d6f4469e42d48fa8d59f/capsule/charts", + "kind": "directory", + "mode": "0o755", + "uid": 0, + "gid": 0, + "nlink": 3, + "bytes": 4096, + "inode": 549330 + }, + { + "relative_path": "releases/9f1daf6c5fb56b2b4fc9817f28f5ad090c884b0295f7d6f4469e42d48fa8d59f/capsule/release", + "kind": "directory", + "mode": "0o755", + "uid": 0, + "gid": 0, + "nlink": 3, + "bytes": 4096, + "inode": 549341 + }, + { + "relative_path": "releases/5fd7ccd8f68057346367d81efb82b48280a16dd31d87683a5bd83e60e3d1837e/capsule/charts", + "kind": "directory", + "mode": "0o755", + "uid": 0, + "gid": 0, + "nlink": 3, + "bytes": 4096, + "inode": 549219 + }, + { + "relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private", + "kind": "directory", + "mode": "0o700", + "uid": 10001, + "gid": 10001, + "nlink": 2, + "bytes": 4096, + "inode": 549263 + }, + { + "relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/result.json", + "kind": "regular", + "mode": "0o600", + "uid": 10001, + "gid": 10001, + "nlink": 1, + "bytes": 455, + "inode": 549311 + }, + { + "relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/journal.json", + "kind": "regular", + "mode": "0o600", + "uid": 10001, + "gid": 10001, + "nlink": 1, + "bytes": 12930, + "inode": 549315 + }, + { + "relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/execute.lock", + "kind": "regular", + "mode": "0o600", + "uid": 10001, + "gid": 10001, + "nlink": 1, + "bytes": 0, + "inode": 549310 + } + ], + "route_receipt_present": false, + "data_read": false, + "changed": false +} diff --git a/docs/release/backup-supervised-installer-validation-2026-10-31.json b/docs/release/backup-supervised-installer-validation-2026-10-31.json new file mode 100644 index 0000000..5b12dfe --- /dev/null +++ b/docs/release/backup-supervised-installer-validation-2026-10-31.json @@ -0,0 +1,99 @@ +{ + "schema": "coulomb.backup-supervised-installer-validation.v1", + "completed_at": "2026-10-31T11:15:54.935514+00:00", + "proposal": { + "status": "prepared", + "proposal_digest": "sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42", + "capacity_profile": { + "maximum_new_file_bytes": 1073741824, + "state_budget_bytes": 8589934592, + "minimum_free_bytes": 8589934592, + "minimum_free_inodes": 1024, + "control_margin_bytes": 16777216 + }, + "targets": 19, + "execution_requires_verified_route": true, + "installed": false + }, + "native_preflight": { + "status": "preflight_refused", + "proposal_digest": "sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42", + "failure_type": "Refused", + "retry_automatically": false, + "reason": "unsafe_state_entry" + }, + "source_files": [ + { + "path": "scripts/backup_install.py", + "sha256": "e9128c59e612d2147c8ab5dd54b35805cf0dcbc2b4691c9d4763ae826d2912ca" + }, + { + "path": "scripts/backup_install_runtime.py", + "sha256": "7e103cf690647b455bf132df7a58fc22e23732f05c1e0889873afdcc591e7097" + }, + { + "path": "scripts/install-backup-sources.py", + "sha256": "7e5c418ca919b182377efcd501408692c1a03482bb606e03249ec2d52f055afe" + }, + { + "path": "tests/test_backup_install.py", + "sha256": "2a31df2471918143fc64f7b48d58d303c943679122c336f4d74441d3237a446e" + }, + { + "path": "tests/test_backup_install_runtime.py", + "sha256": "3888113705cf936072b161fc5e4808221a0bc79a7e9fa18a73b47376a99c7ab1" + }, + { + "path": "scripts/prepare-platform-mail-sender.py", + "sha256": "eebbeed10af5e2dd82691580078d74d4d3b44ddbd52f7b84035ffb14f8df1889" + }, + { + "path": "scripts/rehearse-mail-return-boundary.py", + "sha256": "a5b60246bd3b1577ed4064623aa3a39150396192a2e6f6c160af641be50c7edc" + } + ], + "full_gate": { + "command": "make check", + "tests": 490, + "errors": 0, + "failures": 0, + "skipped": 0, + "warnings": 69, + "log_sha256": "e8c4654a1a14ee85438723c80eb183bcb879740577925c21d40a2e24fc6881b4", + "junit_sha256": "59fb491523bf3ce5bac36fe56a8b51ab5e807042ef66c46b0f2edd0b0155179c" + }, + "installer_tests": { + "library": 15, + "controller": 15 + }, + "native_custody_evidence": "backup-state-custody-observation-2026-10-31.json", + "mail_lint_followup": { + "embedded_remote_preparer_ast_unchanged": true, + "owner_sources_unchanged": true, + "rehearsal": { + "schema": "coulomb.mail-return-boundary-rehearsal.v1", + "completed_at": "2026-10-31T11:11:10.314527+00:00", + "probe_sha256": "a5b60246bd3b1577ed4064623aa3a39150396192a2e6f6c160af641be50c7edc", + "transactional_source_matches_native": true, + "smtp_transport": "in-process synthetic replacement; no SMTP connection", + "message_id_correlates_with_ledger_and_synthetic_bounce": true, + "explicit_reply_to_or_return_path_headers": false, + "scanner_deduplicates_evidence": true, + "scanner_updates_transactional_suppression": false, + "ordinary_sender_suppress_and_unsuppress_denied": true, + "synthetic_operator_suppression_blocks_send_and_survives_restart": true, + "suppression_scope": "recipient-global across callers and tenants", + "explicit_synthetic_operator_unsuppression_verified": true, + "native_operator_admission_proved": false, + "received_mail_authenticity_or_correlation_authority_proved": false, + "python_network_operations_blocked": true, + "temporary_ledgers_removed": true, + "mail_sent": false, + "production_mailbox_accessed": false + } + }, + "installed": false, + "timers_activated": false, + "production_capture_executed": false, + "historical_metadata_modified": false +} diff --git a/scripts/backup_install_runtime.py b/scripts/backup_install_runtime.py new file mode 100644 index 0000000..0a10ce0 --- /dev/null +++ b/scripts/backup_install_runtime.py @@ -0,0 +1,339 @@ +"""Remote, attended source installation; never captures, sends or starts a unit.""" + +from __future__ import annotations + +import base64 +import fcntl +import hashlib +import json +import os +import stat +import subprocess +import types +from contextlib import contextmanager +from datetime import UTC, datetime +from pathlib import Path + +ROOT = Path("/") +STATE = Path("/var/lib/coulomb-social-backup") +ATTEMPTS = Path("/var/lib/coulomb-social-backup-installations") +ROUTE_ROOT = Path("/var/lib/rapp-telemetry/backup-route-releases") +CLUSTER_UID = "a553c742-0115-43d4-99a4-a5ca56fe0786" +UNITS = ( + "coulomb-social-paired-backup.service", + "coulomb-social-paired-backup.timer", + "coulomb-social-backup-freshness.service", + "coulomb-social-backup-freshness.timer", + "coulomb-social-backup-alert@coulomb-social-paired-backup.service.service", +) +PROFILE = dict( + maximum_new_file_bytes=1024**3, + state_budget_bytes=8 * 1024**3, + minimum_free_bytes=8 * 1024**3, + minimum_free_inodes=1024, + control_margin_bytes=16 * 1024**2, +) + + +class Refused(ValueError): + pass + + +def require(condition, reason): + if not condition: + raise Refused(reason) + + +def sha(raw): + return hashlib.sha256(raw).hexdigest() + + +def canonical(value): + return json.dumps(value, sort_keys=True, separators=(",", ":")).encode() + + +def module(source, name): + result = types.ModuleType(name) + exec(compile(source, "", "exec"), result.__dict__) + return result + + +def command(args, timeout=30): + result = subprocess.run( + args, + capture_output=True, + timeout=timeout, + env=dict(os.environ, KUBECONFIG="/etc/rancher/k3s/k3s.yaml"), + ) + require( + result.returncode == 0 and len(result.stdout) < 16 * 1024 * 1024, "bounded_command_failed" + ) + return result.stdout + + +def cluster_identity(): + namespace = json.loads( + command(["k3s", "kubectl", "get", "namespace", "kube-system", "-o", "json"]) + ) + require(namespace["metadata"]["uid"] == CLUSTER_UID, "primary_cluster_mismatch") + + +def inactive_units(): + rows = {} + # This host's list-jobs does not implement JSON output; no-legend emits + # empty output for no jobs, otherwise JOB UNIT TYPE STATE rows. + jobs = command(["systemctl", "list-jobs", "--no-legend", "--no-pager", "--plain"]) + jobs = [line.split() for line in jobs.decode().splitlines() if line.strip()] + require(all(len(row) == 4 and row[0].isdecimal() for row in jobs), "job_listing_unrecognized") + require( + not any(row[1].startswith("coulomb-social-") for row in jobs), + "queued_coulomb_unit_job", + ) + # Also reject an already running alternative instance of the alert template. + listing = json.loads( + command( + ["systemctl", "list-units", "--all", "--output=json", "--no-pager", "coulomb-social-*"] + ) + ) + require(all(x.get("active") == "inactive" for x in listing), "coulomb_unit_active_or_failed") + for unit in UNITS: + raw = command( + [ + "systemctl", + "show", + unit, + "--property=LoadState,ActiveState,SubState,UnitFileState,FragmentPath", + ] + ) + row = dict(line.split("=", 1) for line in raw.decode().splitlines() if "=" in line) + require(row["ActiveState"] == "inactive" and row["SubState"] == "dead", "unit_not_inactive") + require( + row["LoadState"] in ("loaded", "not-found") + and row.get("UnitFileState", "") in ("", "disabled", "static"), + "unit_enabled_or_unexpected", + ) + if row["LoadState"] == "loaded": + fragment = unit if "@" not in unit else "coulomb-social-backup-alert@.service" + require( + row["FragmentPath"] == "/etc/systemd/system/" + fragment, "foreign_unit_fragment" + ) + rows[unit] = row + return rows + + +@contextmanager +def locks(): + handles = [] + try: + info = STATE.lstat() + require( + stat.S_ISDIR(info.st_mode) and info.st_uid == os.geteuid() and not info.st_mode & 0o077, + "private_backup_state_required", + ) + for name in ("backup.lock", "snapshot.lock"): + # Existing lock only; no truncation, replacement or lock-file creation. + fd = os.open(STATE / name, os.O_RDONLY | os.O_NOFOLLOW) + handles.append(fd) + info = os.fstat(fd) + require( + stat.S_ISREG(info.st_mode) + and info.st_uid == os.geteuid() + and info.st_nlink == 1 + and not info.st_mode & 0o077, + "unsafe_lock_file", + ) + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + yield + finally: + for fd in reversed(handles): + os.close(fd) + + +def verified_payload(packet): + require(__debug__, "optimized_python_refused") + plan = packet["plan"] + require( + packet["proposal_digest"] == "sha256:" + sha(canonical(plan)), "proposal_digest_differs" + ) + require( + plan["capacity_profile"] == PROFILE and plan["cluster_uid"] == CLUSTER_UID, + "operating_scope_differs", + ) + sources = {} + for key in ("installer", "route_verifier"): + raw = base64.b64decode(packet[key], validate=True) + require(sha(raw) == plan[key + "_sha256"], "source_digest_differs") + sources[key] = raw + archive = base64.b64decode(packet["archive"], validate=True) + installer = module(sources["installer"], "backup_install") + manifest, contents = installer.read_bundle( + archive, plan["archive_sha256"], plan["manifest_sha256"] + ) + require( + plan["route_packet"]["execution_digest"] == plan["route_execution_digest"], + "route_digest_differs", + ) + return installer, manifest, contents, sources + + +def route_verified(plan, source): + path = ROUTE_ROOT / plan["route_execution_digest"].split(":")[1] / "receipt.json" + require(path.is_file() and not path.is_symlink(), "route_receipt_missing") + info = path.stat() + require( + info.st_uid == 0 and not info.st_mode & 0o077 and info.st_size < 65536, + "route_receipt_not_private", + ) + receipt = json.loads(path.read_bytes()) + require( + receipt["status"] == "backup_email_route_deployed_verified" + and receipt["execution_digest"] == plan["route_execution_digest"], + "route_not_verified", + ) + verifier = module(source, "backup_route_runtime") + revision = verifier.verify_return(plan["route_packet"], after=True) + require(revision == receipt["helm_revision"], "route_revision_differs") + return {"helm_revision": revision, "runtime_and_complete_resources_verified": True} + + +def capacity(contents): + guard = module(contents["consumer/backup_capacity.py"], "backup_capacity") + require( + ( + guard.FILE_LIMIT, + guard.STATE_BUDGET, + guard.MINIMUM_FREE, + guard.MINIMUM_INODES, + guard.CONTROL_MARGIN, + ) + == tuple(PROFILE.values()), + "capacity_profile_differs", + ) + try: + report = guard.reserve_plan(STATE, require_file_limit=False) + except guard.Refused as error: + raise Refused(str(error)) from None + require(not report["reasons"], "capacity_headroom_refused") + return { + k: report[k] + for k in ( + "state_bytes", + "state_reserve_bytes", + "disk_reserve_bytes", + "free_bytes", + "free_inodes", + "filesystem_blocks_reserved", + ) + } + + +def runtime_dependencies(): + code = ( + "import json,boto3,botocore; print(json.dumps(dict(boto3=boto3.__version__," + "botocore=botocore.__version__)))" + ) + actual = json.loads(command(["/opt/railiance-platform/backup-venv/bin/python", "-c", code])) + require(actual == {"boto3": "1.40.18", "botocore": "1.40.76"}, "sdk_dependency_drift") + # Check credential references' custody, without reading their contents. + for name in ("primary.env", "recipient.env"): + path = Path("/etc/coulomb-social-backup") / name + info = path.lstat() + require( + stat.S_ISREG(info.st_mode) + and info.st_uid == 0 + and info.st_nlink == 1 + and not info.st_mode & 0o077, + "credential_reference_custody_differs", + ) + return actual + + +def preflight(plan, installer, contents, source): + cluster_identity() + units = inactive_units() + before = installer.observe(ROOT) + require(before == plan["expected_before"], "native_preimage_changed") + require(not (ATTEMPTS / plan["attempt_id"]).exists(), "existing_attempt_requires_inspection") + profile = capacity(contents) + sdk = runtime_dependencies() + route = route_verified(plan, source) + return dict(units=units, capacity=profile, sdk=sdk, route=route) + + +def run(packet, action): + require(action in ("preflight", "install", "rollback"), "unknown_action") + require(os.geteuid() == 0, "root_owner_execution_required") + installer, manifest, contents, sources = verified_payload(packet) + plan = packet["plan"] + directory = ATTEMPTS / plan["attempt_id"] + require( + len(plan["attempt_id"]) == 16 and all(c in "0123456789abcdef" for c in plan["attempt_id"]), + "invalid_attempt_id", + ) + installer.trusted_parents(ROOT, directory / "receipt.json") + with locks(): + if action == "rollback": + cluster_identity() + inactive_units() + require(directory.is_dir() and not directory.is_symlink(), "attempt_missing") + prior = json.loads((directory / "receipt.json").read_text()) + require( + prior["proposal_digest"] == packet["proposal_digest"], "attempt_proposal_differs" + ) + record = json.loads((directory / "files/receipt.json").read_text()) + prior.update(status="rollback_in_progress", updated_at=datetime.now(UTC).isoformat()) + installer.journal(directory, prior) + installer.restore( + ROOT, directory / "files", record, lambda: command(["systemctl", "daemon-reload"]) + ) + inactive_units() + prior.update(status="prior_files_restored_units_inactive_verified") + installer.journal(directory, prior) + return prior + result = preflight(plan, installer, contents, sources["route_verifier"]) + receipt = dict( + schema="coulomb.backup-source-supervised-installation.v1", + proposal_digest=packet["proposal_digest"], + status="preflight_verified", + preflight=result, + timers_activated=False, + capture_executed=False, + email_sent=False, + retained_data_deleted=False, + ) + if action == "preflight": + return receipt + directory.mkdir(mode=0o700, parents=True, exist_ok=False) + receipt.update( + status="installing", + started_at=datetime.now(UTC).isoformat(), + capacity_profile=PROFILE, + founder_confirmed_profile_and_installation=True, + ) + installer.journal(directory, receipt) + try: + files = installer.install( + manifest, + contents, + plan["expected_before"], + directory / "files", + root=ROOT, + reload_manager=lambda: command(["systemctl", "daemon-reload"]), + verify_inactive=inactive_units, + ) + # Recheck the external dependency after installation, without a send. + receipt["route_postflight"] = route_verified(plan, sources["route_verifier"]) + receipt.update( + status="sources_installed_units_inactive_verified", + verified_targets=files["verified_targets"], + units=inactive_units(), + ) + except Exception as error: + receipt.update( + status="installation_requires_inspection", failure_type=type(error).__name__ + ) + raise + finally: + receipt["updated_at"] = datetime.now(UTC).isoformat() + installer.journal(directory, receipt) + return receipt diff --git a/scripts/install-backup-sources.py b/scripts/install-backup-sources.py new file mode 100644 index 0000000..0f80470 --- /dev/null +++ b/scripts/install-backup-sources.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Review and attend the fixed inactive backup-source installation or rollback.""" + +from __future__ import annotations + +import argparse +import base64 +import hashlib +import importlib.util +import json +import shlex +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +OWNER = Path("/home/worsch/rapp-telemetry") +DELIVERY = ROOT / "docs/release/backup-operating-delivery-alerts-2026-10-31.json" +OBSERVATION = ROOT / "docs/release/backup-install-target-observation-2026-10-31.json" +ROUTE_DIGEST = "sha256:4d8481061dc9ce6835169defc96887945052332a0db8d4234ae84032d63cc328" + + +def module(name, path): + spec = importlib.util.spec_from_file_location(name, path) + loaded = importlib.util.module_from_spec(spec) + spec.loader.exec_module(loaded) + return loaded + + +def prepare(): + if not __debug__: + raise ValueError("optimized_python_refused") + runtime = module("backup_install_runtime", ROOT / "scripts/backup_install_runtime.py") + installer = (ROOT / "scripts/backup_install.py").read_bytes() + implementation = module("backup_install", ROOT / "scripts/backup_install.py") + delivery = json.loads(DELIVERY.read_text()) + archive = Path(delivery["private_archive"]).read_bytes() + implementation.read_bundle(archive, delivery["archive_sha256"], delivery["manifest_sha256"]) + route = module("reviewed_backup_route", OWNER / "tools/deploy_coulomb_backup_route.py") + route_packet, route_source = route.prepare(route.DEFAULT) + if route_packet["execution_digest"] != ROUTE_DIGEST: + raise ValueError("reviewed_owner_route_changed") + # Verification needs no chart upload and does not perform an upgrade. + route_packet.pop("chart") + source = (ROOT / "scripts/backup_install_runtime.py").read_text() + plan = dict( + schema="coulomb.backup-source-installation-proposal.v1", + attempt_id="20261031bacc0001", + cluster_uid=runtime.CLUSTER_UID, + archive_sha256=delivery["archive_sha256"], + manifest_sha256=delivery["manifest_sha256"], + capacity_profile=runtime.PROFILE, + expected_before=json.loads(OBSERVATION.read_text())["targets"], + route_execution_digest=ROUTE_DIGEST, + route_packet=route_packet, + installer_sha256=hashlib.sha256(installer).hexdigest(), + route_verifier_sha256=hashlib.sha256(route_source.encode()).hexdigest(), + runtime_sha256=hashlib.sha256(source.encode()).hexdigest(), + launcher_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + ) + packet = dict( + plan=plan, + proposal_digest="sha256:" + runtime.sha(runtime.canonical(plan)), + archive=base64.b64encode(archive).decode(), + installer=base64.b64encode(installer).decode(), + route_verifier=base64.b64encode(route_source.encode()).decode(), + ) + return packet, source + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + actions = parser.add_mutually_exclusive_group() + actions.add_argument("--preflight", action="store_true") + actions.add_argument("--execute", action="store_true") + actions.add_argument("--rollback", action="store_true") + parser.add_argument("--expect-digest") + args = parser.parse_args(argv) + packet, source = prepare() + digest = packet["proposal_digest"] + if args.expect_digest and args.expect_digest != digest: + raise SystemExit("Reviewed source installation proposal changed; no action") + if not (args.preflight or args.execute or args.rollback): + print( + json.dumps( + dict( + status="prepared", + proposal_digest=digest, + capacity_profile=packet["plan"]["capacity_profile"], + targets=len(packet["plan"]["expected_before"]), + execution_requires_verified_route=True, + installed=False, + ) + ) + ) + return 0 + if args.execute or args.rollback: + if not args.expect_digest or not sys.stdin.isatty(): + raise SystemExit("Founder terminal and reviewed digest required") + verb = "ROLLBACK" if args.rollback else "INSTALL" + print( + "Restore this installation only, with units inactive." + if args.rollback + else "Admit the displayed capacity profile and install the reviewed sources " + "with units inactive." + ) + print(json.dumps(packet["plan"]["capacity_profile"], sort_keys=True)) + if input("Type " + verb + " followed by the full digest: ").strip() != verb + " " + digest: + raise SystemExit("Not confirmed; no action") + packet["action"] = "rollback" if args.rollback else "install" if args.execute else "preflight" + bootstrap = ( + source + + """ +import sys +packet=json.load(sys.stdin) +try: + result=run(packet,packet['action']) +except Exception as error: + status='preflight_refused' if packet['action']=='preflight' else 'installation_requires_inspection' + result=dict(status=status, + proposal_digest=packet['proposal_digest'],failure_type=type(error).__name__,retry_automatically=False) + if isinstance(error,Refused):result['reason']=str(error) +print(json.dumps(result)) +sys.exit(0 if result['status'] in ('preflight_verified','sources_installed_units_inactive_verified', + 'prior_files_restored_units_inactive_verified') else 1) +""" + ) + try: + response = subprocess.run( + [ + "ssh", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=10", + "railiance01", + "sudo -n python3 -c " + shlex.quote(bootstrap), + ], + input=json.dumps(packet).encode(), + capture_output=True, + timeout=600, + ) + result = json.loads(response.stdout) + except (subprocess.TimeoutExpired, ValueError): + print( + json.dumps( + dict( + status="remote_outcome_unconfirmed", + proposal_digest=digest, + retry_automatically=False, + ) + ) + ) + return 1 + print(json.dumps(result)) + return int(response.returncode != 0) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/prepare-platform-mail-sender.py b/scripts/prepare-platform-mail-sender.py index e634f93..68da53f 100644 --- a/scripts/prepare-platform-mail-sender.py +++ b/scripts/prepare-platform-mail-sender.py @@ -4,63 +4,303 @@ No create/apply/patch is executed without --dry-run=server. This does not admit OpenBao access, install a resource, scan a mailbox or send an email. """ -import argparse, os, subprocess,shlex,json,hashlib + +import argparse +import hashlib +import json +import os +import shlex +import subprocess from pathlib import Path -from datetime import datetime,timezone + parser = argparse.ArgumentParser(description=__doc__) -parser.add_argument('--output', type=Path, required=True) +parser.add_argument("--output", type=Path, required=True) args = parser.parse_args() if not __debug__: - raise SystemExit('Assertions must be enabled') + raise SystemExit("Assertions must be enabled") if args.output.exists(): - raise SystemExit('Choose a new candidate output; existing evidence is retained') -code=r''' -import subprocess,json,hashlib -from datetime import datetime,timezone -K=['k3s','kubectl'] -def get(kind,name): - return json.loads(subprocess.check_output(K+['get',kind,name,'-n','email-connect','-o','json'],timeout=20)) -d=get('deployment','email-connect');cm=get('configmap','email-connect-config') -assert d['metadata']['uid']=='2e98e52a-a2d1-4c52-9ac8-6efbc0a18e72' -assert d['status']['observedGeneration']==d['metadata']['generation'] and d['status']['readyReplicas']==1 -c=d['spec']['template']['spec']['containers'][0] -assert c['name']=='email-connect' and len(d['spec']['template']['spec']['containers'])==1 -assert c['image']=='forgejo.coulomb.social/coulomb/email-connect@sha256:5d79fd1fec15d3a279383820f170d263de3a55f5308df01a0a34732ebaea6d3d' -assert c['envFrom']==[{'configMapRef':{'name':'email-connect-config'}},{'secretRef':{'name':'email-connect-runtime','optional':False}}] -assert cm['data']['EMAIL_CONNECT_SENDER']=='forgejo@coulomb.social' -assert all(e['name'] not in {'EMAIL_CONNECT_SMTP_USERNAME','EMAIL_CONNECT_SMTP_PASSWORD','EMAIL_CONNECT_SENDER'} for e in c['env']) -assert c['env']==[{'name':'EMAIL_CONNECT_CALLERS_FILE','value':'/etc/email-connect-callers/callers.json'}], 'unexpected environment; refuse to copy unknown values' -role='email-connect-platform-smtp-eso';store='openbao-email-connect-platform-smtp';secret='email-connect-platform-smtp' -def obj(api,kind,name,**extra):return {'apiVersion':api,'kind':kind,'metadata':{'name':name,'namespace':'email-connect'},**extra} -resources=[obj('v1','ServiceAccount',role,automountServiceAccountToken=False),obj('external-secrets.io/v1','SecretStore',store,spec={'provider':{'vault':{'server':'http://openbao.openbao.svc:8200','path':'platform','version':'v2','auth':{'kubernetes':{'mountPath':'kubernetes','role':role,'serviceAccountRef':{'name':role,'audiences':['openbao']}}}}}}),obj('external-secrets.io/v1','ExternalSecret',secret,spec={'refreshInterval':'1h','secretStoreRef':{'kind':'SecretStore','name':store},'target':{'name':secret,'creationPolicy':'Owner','deletionPolicy':'Retain','template':{'type':'Opaque'}},'data':[{'secretKey':'password','remoteRef':{'key':'workloads/railiance-telemetry/smtp','property':'SMTP_PASSWORD'}}]})] + raise SystemExit("Choose a new candidate output; existing evidence is retained") +code = r""" +import subprocess, json, hashlib +from datetime import datetime, timezone + +K = ["k3s", "kubectl"] + + +def get(kind, name): + return json.loads( + subprocess.check_output( + K + ["get", kind, name, "-n", "email-connect", "-o", "json"], timeout=20 + ) + ) + + +d = get("deployment", "email-connect") +cm = get("configmap", "email-connect-config") +assert d["metadata"]["uid"] == "2e98e52a-a2d1-4c52-9ac8-6efbc0a18e72" +assert ( + d["status"]["observedGeneration"] == d["metadata"]["generation"] + and d["status"]["readyReplicas"] == 1 +) +c = d["spec"]["template"]["spec"]["containers"][0] +assert c["name"] == "email-connect" and len(d["spec"]["template"]["spec"]["containers"]) == 1 +assert ( + c["image"] + == "forgejo.coulomb.social/coulomb/email-connect@sha256:" + "5d79fd1fec15d3a279383820f170d263de3a55f5308df01a0a34732ebaea6d3d" +) +assert c["envFrom"] == [ + {"configMapRef": {"name": "email-connect-config"}}, + {"secretRef": {"name": "email-connect-runtime", "optional": False}}, +] +assert cm["data"]["EMAIL_CONNECT_SENDER"] == "forgejo@coulomb.social" +assert all( + e["name"] + not in {"EMAIL_CONNECT_SMTP_USERNAME", "EMAIL_CONNECT_SMTP_PASSWORD", "EMAIL_CONNECT_SENDER"} + for e in c["env"] +) +assert c["env"] == [ + {"name": "EMAIL_CONNECT_CALLERS_FILE", "value": "/etc/email-connect-callers/callers.json"} +], "unexpected environment; refuse to copy unknown values" +role = "email-connect-platform-smtp-eso" +store = "openbao-email-connect-platform-smtp" +secret = "email-connect-platform-smtp" + + +def obj(api, kind, name, **extra): + return { + "apiVersion": api, + "kind": kind, + "metadata": {"name": name, "namespace": "email-connect"}, + **extra, + } + + +resources = [ + obj("v1", "ServiceAccount", role, automountServiceAccountToken=False), + obj( + "external-secrets.io/v1", + "SecretStore", + store, + spec={ + "provider": { + "vault": { + "server": "http://openbao.openbao.svc:8200", + "path": "platform", + "version": "v2", + "auth": { + "kubernetes": { + "mountPath": "kubernetes", + "role": role, + "serviceAccountRef": {"name": role, "audiences": ["openbao"]}, + } + }, + } + } + }, + ), + obj( + "external-secrets.io/v1", + "ExternalSecret", + secret, + spec={ + "refreshInterval": "1h", + "secretStoreRef": {"kind": "SecretStore", "name": store}, + "target": { + "name": secret, + "creationPolicy": "Owner", + "deletionPolicy": "Retain", + "template": {"type": "Opaque"}, + }, + "data": [ + { + "secretKey": "password", + "remoteRef": { + "key": "workloads/railiance-telemetry/smtp", + "property": "SMTP_PASSWORD", + }, + } + ], + }, + ), +] for x in resources: - out=subprocess.check_output(K+['get',x['kind'],x['metadata']['name'],'-n','email-connect','--ignore-not-found','-o','name'],timeout=20) - assert not out.strip(),'candidate resource already exists' -base='/spec/template/spec/containers/0' -dpatch=[{'op':'test','path':'/metadata/uid','value':d['metadata']['uid']},{'op':'test','path':'/metadata/resourceVersion','value':d['metadata']['resourceVersion']},{'op':'test','path':base+'/image','value':c['image']},{'op':'test','path':base+'/envFrom','value':c['envFrom']},{'op':'test','path':base+'/env','value':c['env']},{'op':'replace','path':base+'/envFrom','value':[{'configMapRef':{'name':'email-connect-config'}}]},{'op':'replace','path':base+'/env','value':c['env']+[{'name':'EMAIL_CONNECT_SMTP_USERNAME','value':'platform@coulomb.social'},{'name':'EMAIL_CONNECT_SMTP_PASSWORD','valueFrom':{'secretKeyRef':{'name':secret,'key':'password','optional':False}}}]}] -cmpatch=[{'op':'test','path':'/metadata/uid','value':cm['metadata']['uid']},{'op':'test','path':'/data/EMAIL_CONNECT_SENDER','value':'forgejo@coulomb.social'},{'op':'replace','path':'/data/EMAIL_CONNECT_SENDER','value':'platform@coulomb.social'}] -def patchdry(kind,name,patch): - return json.loads(subprocess.check_output(K+['patch',kind,name,'-n','email-connect','--type=json','--dry-run=server','-p',json.dumps(patch),'-o','json'],timeout=30)) -checks=[] + out = subprocess.check_output( + K + + [ + "get", + x["kind"], + x["metadata"]["name"], + "-n", + "email-connect", + "--ignore-not-found", + "-o", + "name", + ], + timeout=20, + ) + assert not out.strip(), "candidate resource already exists" +base = "/spec/template/spec/containers/0" +dpatch = [ + {"op": "test", "path": "/metadata/uid", "value": d["metadata"]["uid"]}, + {"op": "test", "path": "/metadata/resourceVersion", "value": d["metadata"]["resourceVersion"]}, + {"op": "test", "path": base + "/image", "value": c["image"]}, + {"op": "test", "path": base + "/envFrom", "value": c["envFrom"]}, + {"op": "test", "path": base + "/env", "value": c["env"]}, + { + "op": "replace", + "path": base + "/envFrom", + "value": [{"configMapRef": {"name": "email-connect-config"}}], + }, + { + "op": "replace", + "path": base + "/env", + "value": c["env"] + + [ + {"name": "EMAIL_CONNECT_SMTP_USERNAME", "value": "platform@coulomb.social"}, + { + "name": "EMAIL_CONNECT_SMTP_PASSWORD", + "valueFrom": { + "secretKeyRef": {"name": secret, "key": "password", "optional": False} + }, + }, + ], + }, +] +cmpatch = [ + {"op": "test", "path": "/metadata/uid", "value": cm["metadata"]["uid"]}, + {"op": "test", "path": "/data/EMAIL_CONNECT_SENDER", "value": "forgejo@coulomb.social"}, + {"op": "replace", "path": "/data/EMAIL_CONNECT_SENDER", "value": "platform@coulomb.social"}, +] + + +def patchdry(kind, name, patch): + return json.loads( + subprocess.check_output( + K + + [ + "patch", + kind, + name, + "-n", + "email-connect", + "--type=json", + "--dry-run=server", + "-p", + json.dumps(patch), + "-o", + "json", + ], + timeout=30, + ) + ) + + +checks = [] for x in resources: - p=subprocess.run(K+['create','--dry-run=server','-f','-','-o','json'],input=json.dumps(x).encode(),capture_output=True,timeout=30) - if p.returncode:raise RuntimeError('resource dry-run failed: '+x['kind']) - y=json.loads(p.stdout);checks.append({'kind':y['kind'],'name':y['metadata']['name'],'accepted':True}) -da=patchdry('deployment','email-connect',dpatch);ca=patchdry('configmap','email-connect-config',cmpatch) -expected=json.loads(json.dumps(d['spec']));expected['template']['spec']['containers'][0]['envFrom']=dpatch[-2]['value'];expected['template']['spec']['containers'][0]['env']=dpatch[-1]['value'] -assert da['spec']==expected,'unexpected deployment spec change' -expectedcm=dict(cm['data'],EMAIL_CONNECT_SENDER='platform@coulomb.social');assert ca['data']==expectedcm -checks.extend([{'kind':'Deployment','name':'email-connect','accepted':True,'only_envFrom_and_env_changed':True},{'kind':'ConfigMap','name':'email-connect-config','accepted':True,'only_sender_changed':True}]) -volumes=d['spec']['template']['spec']['volumes']; policy_name=next(v['configMap']['name'] for v in volumes if v['name']=='caller-policy') -pol=get('configmap',policy_name)['data']['callers.json'];parsed=json.loads(pol) -publiccallers=[{k:x.get(k) for k in ('caller_id','operations','templates')} for x in parsed['callers']] -print(json.dumps({'schema':'coulomb.platform-mail-sender-candidate.v1','observed_at':datetime.now(timezone.utc).isoformat(),'deployment_uid':d['metadata']['uid'],'deployment_generation':d['metadata']['generation'],'image':c['image'],'new_resources':resources,'deployment_json_patch':dpatch,'configmap_json_patch':cmpatch,'native_server_dry_run':checks,'live_caller_policy':{'name':policy_name,'sha256':hashlib.sha256(pol.encode()).hexdigest(),'callers':publiccallers},'platform_mailbox_admission_executed':False,'resources_installed':False,'mail_sent':False,'secrets_read':False,'execution_ready':False})) -''' -r=subprocess.run(['ssh','-o','BatchMode=yes','-o','ConnectTimeout=10','railiance01','sudo -n python3 -c '+shlex.quote(code)],capture_output=True,text=True,timeout=180) -if r.returncode:raise SystemExit('read/dry-run preparation failed: '+r.stderr[-1500:]) -j=json.loads(r.stdout) -j['preparer_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest() -fd=os.open(args.output,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600) -with os.fdopen(fd,'w') as target: - json.dump(j,target,indent=2);target.write('\n');target.flush();os.fsync(target.fileno()) -print(json.dumps({k:j[k] for k in ('observed_at','deployment_generation','native_server_dry_run','live_caller_policy','execution_ready')},indent=2)) + p = subprocess.run( + K + ["create", "--dry-run=server", "-f", "-", "-o", "json"], + input=json.dumps(x).encode(), + capture_output=True, + timeout=30, + ) + if p.returncode: + raise RuntimeError("resource dry-run failed: " + x["kind"]) + y = json.loads(p.stdout) + checks.append({"kind": y["kind"], "name": y["metadata"]["name"], "accepted": True}) +da = patchdry("deployment", "email-connect", dpatch) +ca = patchdry("configmap", "email-connect-config", cmpatch) +expected = json.loads(json.dumps(d["spec"])) +expected["template"]["spec"]["containers"][0]["envFrom"] = dpatch[-2]["value"] +expected["template"]["spec"]["containers"][0]["env"] = dpatch[-1]["value"] +assert da["spec"] == expected, "unexpected deployment spec change" +expectedcm = dict(cm["data"], EMAIL_CONNECT_SENDER="platform@coulomb.social") +assert ca["data"] == expectedcm +checks.extend( + [ + { + "kind": "Deployment", + "name": "email-connect", + "accepted": True, + "only_envFrom_and_env_changed": True, + }, + { + "kind": "ConfigMap", + "name": "email-connect-config", + "accepted": True, + "only_sender_changed": True, + }, + ] +) +volumes = d["spec"]["template"]["spec"]["volumes"] +policy_name = next(v["configMap"]["name"] for v in volumes if v["name"] == "caller-policy") +pol = get("configmap", policy_name)["data"]["callers.json"] +parsed = json.loads(pol) +publiccallers = [ + {k: x.get(k) for k in ("caller_id", "operations", "templates")} for x in parsed["callers"] +] +print( + json.dumps( + { + "schema": "coulomb.platform-mail-sender-candidate.v1", + "observed_at": datetime.now(timezone.utc).isoformat(), + "deployment_uid": d["metadata"]["uid"], + "deployment_generation": d["metadata"]["generation"], + "image": c["image"], + "new_resources": resources, + "deployment_json_patch": dpatch, + "configmap_json_patch": cmpatch, + "native_server_dry_run": checks, + "live_caller_policy": { + "name": policy_name, + "sha256": hashlib.sha256(pol.encode()).hexdigest(), + "callers": publiccallers, + }, + "platform_mailbox_admission_executed": False, + "resources_installed": False, + "mail_sent": False, + "secrets_read": False, + "execution_ready": False, + } + ) +) +""" +r = subprocess.run( + [ + "ssh", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=10", + "railiance01", + "sudo -n python3 -c " + shlex.quote(code), + ], + capture_output=True, + text=True, + timeout=180, +) +if r.returncode: + raise SystemExit("read/dry-run preparation failed: " + r.stderr[-1500:]) +j = json.loads(r.stdout) +j["preparer_sha256"] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest() +fd = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) +with os.fdopen(fd, "w") as target: + json.dump(j, target, indent=2) + target.write("\n") + target.flush() + os.fsync(target.fileno()) +print( + json.dumps( + { + k: j[k] + for k in ( + "observed_at", + "deployment_generation", + "native_server_dry_run", + "live_caller_policy", + "execution_ready", + ) + }, + indent=2, + ) +) diff --git a/scripts/rehearse-mail-return-boundary.py b/scripts/rehearse-mail-return-boundary.py index 6506930..237279a 100644 --- a/scripts/rehearse-mail-return-boundary.py +++ b/scripts/rehearse-mail-return-boundary.py @@ -4,6 +4,7 @@ This probe documents present capabilities and missing integration. It is not an inbound worker, sender deployment, credential admission or suppression operator. """ + from __future__ import annotations import argparse @@ -12,158 +13,220 @@ import io import json import sys import tempfile -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from unittest.mock import patch def main(): if not __debug__: - raise SystemExit('Assertions must be enabled') + raise SystemExit("Assertions must be enabled") parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--owner-repo', type=Path, required=True) - parser.add_argument('--native-evidence', type=Path, required=True) + parser.add_argument("--owner-repo", type=Path, required=True) + parser.add_argument("--native-evidence", type=Path, required=True) args = parser.parse_args() - source = args.owner_repo.resolve() / 'src' + source = args.owner_repo.resolve() / "src" native = json.loads(args.native_evidence.read_text()) - module = source / 'email_connect/transactional.py' - assert hashlib.sha256(module.read_bytes()).hexdigest() == native[ - 'running_process_configuration']['transactional_source_sha256'] + module = source / "email_connect/transactional.py" + assert ( + hashlib.sha256(module.read_bytes()).hexdigest() + == native["running_process_configuration"]["transactional_source_sha256"] + ) + # Fail any attempted Python network connection/bind, even if a future source # change accidentally substitutes a real transport for these fixtures. def no_network(event, _args): - if event in {'socket.connect', 'socket.bind', 'socket.getaddrinfo', 'socket.sendto'}: - raise RuntimeError('Offline rehearsal forbids network access') + if event in {"socket.connect", "socket.bind", "socket.getaddrinfo", "socket.sendto"}: + raise RuntimeError("Offline rehearsal forbids network access") + sys.addaudithook(no_network) sys.path.insert(0, str(source)) from email_connect.callers import CallerPolicy, ScopedTransactionalApplication - from email_connect.config import (AppConfig, MailboxConfig, ReportsConfig, - ScanConfig, SourceConfig, StorageConfig) + from email_connect.config import ( + AppConfig, + MailboxConfig, + ReportsConfig, + ScanConfig, + SourceConfig, + StorageConfig, + ) from email_connect.scanner import scan_mailbox from email_connect.storage import StateStore from email_connect.transactional import SMTPProvider, SQLiteDeliveryStore captured = [] + class SyntheticSMTP: def __init__(self, *args, **kwargs): self.tls = False + def __enter__(self): return self + def __exit__(self, *args): pass + def starttls(self, context): assert context.check_hostname self.tls = True + def login(self, username, password): - assert self.tls and username == 'fixture' and password == 'fixture-only' + assert self.tls and username == "fixture" and password == "fixture-only" + def send_message(self, message): - assert message['To'] == 'recipient@example.test' + assert message["To"] == "recipient@example.test" captured.append(message) - def policy(name, operations, tenant='tenant:fixture'): - return CallerPolicy(caller_id=name, token='fixture-credential-' + name, - source=name, bindings=frozenset({(tenant, 'client:fixture')}), - operations=frozenset(operations), - templates=frozenset({'family_member.invited'}), - rate_per_minute=60, burst=30, max_payload_bytes=8192) - sender = policy('fixture-sender', {'send', 'delivery-status'}) - operator = policy('fixture-operator', {'suppress', 'unsuppress'}) - other = policy('fixture-other', {'send'}, 'tenant:other') - def event(name, caller=sender, tenant='tenant:fixture'): - return {'id': name, 'source': caller.source, 'type': 'family_member.invited', - 'tenant': tenant, 'client_id': 'client:fixture', - 'data': {'primary_email': 'recipient@example.test', 'invitation_id': name}} - def invoke(app, payload, path='/v1/send', caller=sender): - raw = json.dumps(payload).encode(); result = {} - env = {'PATH_INFO': path, 'REQUEST_METHOD': 'POST', 'CONTENT_LENGTH': str(len(raw)), - 'wsgi.input': io.BytesIO(raw), 'HTTP_AUTHORIZATION': 'Bearer ' + caller.token, - 'HTTP_IDEMPOTENCY_KEY': payload.get('id', '')} - body = b''.join(app(env, lambda status, headers: result.update(status=int(status[:3])))) - return result['status'], json.loads(body) + def policy(name, operations, tenant="tenant:fixture"): + return CallerPolicy( + caller_id=name, + token="fixture-credential-" + name, + source=name, + bindings=frozenset({(tenant, "client:fixture")}), + operations=frozenset(operations), + templates=frozenset({"family_member.invited"}), + rate_per_minute=60, + burst=30, + max_payload_bytes=8192, + ) - with tempfile.TemporaryDirectory(prefix='csoc-mail-return-') as tmp, patch( - 'email_connect.transactional.smtplib.SMTP', SyntheticSMTP): + sender = policy("fixture-sender", {"send", "delivery-status"}) + operator = policy("fixture-operator", {"suppress", "unsuppress"}) + other = policy("fixture-other", {"send"}, "tenant:other") + + def event(name, caller=sender, tenant="tenant:fixture"): + return { + "id": name, + "source": caller.source, + "type": "family_member.invited", + "tenant": tenant, + "client_id": "client:fixture", + "data": {"primary_email": "recipient@example.test", "invitation_id": name}, + } + + def invoke(app, payload, path="/v1/send", caller=sender): + raw = json.dumps(payload).encode() + result = {} + env = { + "PATH_INFO": path, + "REQUEST_METHOD": "POST", + "CONTENT_LENGTH": str(len(raw)), + "wsgi.input": io.BytesIO(raw), + "HTTP_AUTHORIZATION": "Bearer " + caller.token, + "HTTP_IDEMPOTENCY_KEY": payload.get("id", ""), + } + body = b"".join(app(env, lambda status, headers: result.update(status=int(status[:3])))) + return result["status"], json.loads(body) + + with ( + tempfile.TemporaryDirectory(prefix="csoc-mail-return-") as tmp, + patch("email_connect.transactional.smtplib.SMTP", SyntheticSMTP), + ): root = Path(tmp) - ledger = SQLiteDeliveryStore(str(root / 'transactional.sqlite')) - provider = SMTPProvider('smtp.example.test', 587, 'fixture', 'fixture-only', - 'platform@example.test') - app = ScopedTransactionalApplication(ledger, provider, 'https://issuer.example.test', - [sender, operator, other]) - assert invoke(app, event('invitation-1'))[0] == 202 + ledger = SQLiteDeliveryStore(str(root / "transactional.sqlite")) + provider = SMTPProvider( + "smtp.example.test", 587, "fixture", "fixture-only", "platform@example.test" + ) + app = ScopedTransactionalApplication( + ledger, provider, "https://issuer.example.test", [sender, operator, other] + ) + assert invoke(app, event("invitation-1"))[0] == 202 assert len(captured) == 1 message = captured[0] - reference = str(message['Message-ID']) - assert message['From'] == 'platform@example.test' - assert message['Reply-To'] is None and message['Return-Path'] is None - assert ledger.db.execute('SELECT count(*) FROM deliveries WHERE provider_ref=?', - (reference,)).fetchone()[0] == 1 - fixture = root / 'mailbox'; fixture.mkdir() - (fixture / 'bounce.eml').write_text( - 'From: mailer-daemon@example.test\nTo: platform@example.test\n' - 'Subject: Delivery Status Notification (Failure)\n' - 'Date: Sat, 31 Oct 2026 10:00:00 +0000\n' - 'Message-ID: \n' - 'Content-Type: text/plain; charset=utf-8\n\n' - 'Delivery failure.\nFinal-Recipient: rfc822; recipient@example.test\n' - 'Action: failed\nStatus: 5.1.1\nDiagnostic-Code: smtp; 550 User unknown\n' - 'Original-Message-ID: ' + reference + '\n') - config = AppConfig(mailbox=MailboxConfig(id='fixture-mailbox', protocol='fixture'), - scan=ScanConfig(), storage=StorageConfig(str(root / 'scanner.sqlite')), - reports=ReportsConfig(str(root / 'reports')), - source=SourceConfig(fixture_dir=str(fixture))) + reference = str(message["Message-ID"]) + assert message["From"] == "platform@example.test" + assert message["Reply-To"] is None and message["Return-Path"] is None + assert ( + ledger.db.execute( + "SELECT count(*) FROM deliveries WHERE provider_ref=?", (reference,) + ).fetchone()[0] + == 1 + ) + fixture = root / "mailbox" + fixture.mkdir() + (fixture / "bounce.eml").write_text( + "From: mailer-daemon@example.test\nTo: platform@example.test\n" + "Subject: Delivery Status Notification (Failure)\n" + "Date: Sat, 31 Oct 2026 10:00:00 +0000\n" + "Message-ID: \n" + "Content-Type: text/plain; charset=utf-8\n\n" + "Delivery failure.\nFinal-Recipient: rfc822; recipient@example.test\n" + "Action: failed\nStatus: 5.1.1\nDiagnostic-Code: smtp; 550 User unknown\n" + "Original-Message-ID: " + reference + "\n" + ) + config = AppConfig( + mailbox=MailboxConfig(id="fixture-mailbox", protocol="fixture"), + scan=ScanConfig(), + storage=StorageConfig(str(root / "scanner.sqlite")), + reports=ReportsConfig(str(root / "reports")), + source=SourceConfig(fixture_dir=str(fixture)), + ) scan = scan_mailbox(config) evidence = StateStore(config.storage.path) rows = evidence.evidence_rows() assert scan.scan.messages_seen == 1 and len(rows) == 1 - assert rows[0]['original_message_id'] == reference - assert rows[0]['affected_email_address'] == 'recipient@example.test' - assert rows[0]['assessment_subclass'] == 'fail.hard_bounce' - assert not ledger.is_suppressed('recipient@example.test') + assert rows[0]["original_message_id"] == reference + assert rows[0]["affected_email_address"] == "recipient@example.test" + assert rows[0]["assessment_subclass"] == "fail.hard_bounce" + assert not ledger.is_suppressed("recipient@example.test") repeat = scan_mailbox(config) assert repeat.scan.messages_new == 0 and repeat.scan.evidence_events_created == 0 - payload = {'recipient': 'recipient@example.test', 'reason': 'hard_bounce'} - assert invoke(app, payload, '/v1/suppressions')[0] == 403 - assert not ledger.is_suppressed('recipient@example.test') - assert invoke(app, payload, '/v1/suppressions', operator)[0] == 200 - assert invoke(app, event('invitation-2')) == (400, {'error': 'recipient_suppressed'}) - assert invoke(app, event('invitation-3', other, 'tenant:other'), caller=other) == ( - 400, {'error': 'recipient_suppressed'}) + payload = {"recipient": "recipient@example.test", "reason": "hard_bounce"} + assert invoke(app, payload, "/v1/suppressions")[0] == 403 + assert not ledger.is_suppressed("recipient@example.test") + assert invoke(app, payload, "/v1/suppressions", operator)[0] == 200 + assert invoke(app, event("invitation-2")) == (400, {"error": "recipient_suppressed"}) + assert invoke(app, event("invitation-3", other, "tenant:other"), caller=other) == ( + 400, + {"error": "recipient_suppressed"}, + ) assert len(captured) == 1 ledger.db.close() - ledger = SQLiteDeliveryStore(str(root / 'transactional.sqlite')) - assert ledger.is_suppressed('recipient@example.test') - app = ScopedTransactionalApplication(ledger, provider, 'https://issuer.example.test', - [sender, operator, other]) - assert invoke(app, payload, '/v1/suppressions/remove')[0] == 403 - assert invoke(app, payload, '/v1/suppressions/remove', operator)[0] == 200 - assert not ledger.is_suppressed('recipient@example.test') + ledger = SQLiteDeliveryStore(str(root / "transactional.sqlite")) + assert ledger.is_suppressed("recipient@example.test") + app = ScopedTransactionalApplication( + ledger, provider, "https://issuer.example.test", [sender, operator, other] + ) + assert invoke(app, payload, "/v1/suppressions/remove")[0] == 403 + assert invoke(app, payload, "/v1/suppressions/remove", operator)[0] == 200 + assert not ledger.is_suppressed("recipient@example.test") assert len(captured) == 1 - ledger.db.close(); evidence.close() + ledger.db.close() + evidence.close() assert not root.exists() - print(json.dumps({ - 'schema': 'coulomb.mail-return-boundary-rehearsal.v1', - 'completed_at': datetime.now(timezone.utc).isoformat(), - 'source_sha256': {str(p.relative_to(args.owner_repo.resolve())): - hashlib.sha256(p.read_bytes()).hexdigest() - for p in sorted((source / 'email_connect').glob('*.py'))}, - 'probe_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), - 'transactional_source_matches_native': True, - 'smtp_transport': 'in-process synthetic replacement; no SMTP connection', - 'message_id_correlates_with_ledger_and_synthetic_bounce': True, - 'explicit_reply_to_or_return_path_headers': False, - 'scanner_deduplicates_evidence': True, - 'scanner_updates_transactional_suppression': False, - 'ordinary_sender_suppress_and_unsuppress_denied': True, - 'synthetic_operator_suppression_blocks_send_and_survives_restart': True, - 'suppression_scope': 'recipient-global across callers and tenants', - 'explicit_synthetic_operator_unsuppression_verified': True, - 'native_operator_admission_proved': False, - 'received_mail_authenticity_or_correlation_authority_proved': False, - 'python_network_operations_blocked': True, - 'temporary_ledgers_removed': True, - 'mail_sent': False, 'production_mailbox_accessed': False, - }, indent=2)) + print( + json.dumps( + { + "schema": "coulomb.mail-return-boundary-rehearsal.v1", + "completed_at": datetime.now(UTC).isoformat(), + "source_sha256": { + str(p.relative_to(args.owner_repo.resolve())): hashlib.sha256( + p.read_bytes() + ).hexdigest() + for p in sorted((source / "email_connect").glob("*.py")) + }, + "probe_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "transactional_source_matches_native": True, + "smtp_transport": "in-process synthetic replacement; no SMTP connection", + "message_id_correlates_with_ledger_and_synthetic_bounce": True, + "explicit_reply_to_or_return_path_headers": False, + "scanner_deduplicates_evidence": True, + "scanner_updates_transactional_suppression": False, + "ordinary_sender_suppress_and_unsuppress_denied": True, + "synthetic_operator_suppression_blocks_send_and_survives_restart": True, + "suppression_scope": "recipient-global across callers and tenants", + "explicit_synthetic_operator_unsuppression_verified": True, + "native_operator_admission_proved": False, + "received_mail_authenticity_or_correlation_authority_proved": False, + "python_network_operations_blocked": True, + "temporary_ledgers_removed": True, + "mail_sent": False, + "production_mailbox_accessed": False, + }, + indent=2, + ) + ) -if __name__ == '__main__': +if __name__ == "__main__": main() diff --git a/tests/test_backup_install_runtime.py b/tests/test_backup_install_runtime.py new file mode 100644 index 0000000..53d0499 --- /dev/null +++ b/tests/test_backup_install_runtime.py @@ -0,0 +1,285 @@ +"""Attended controller transactions against disposable files and fake host boundaries.""" + +import base64 +import fcntl +import importlib.util +import json +import os +from contextlib import contextmanager +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from tests import test_backup_install as fixtures + +bundle = fixtures.bundle +install = fixtures.install +prepared = fixtures.prepared + +ROOT = Path(__file__).parents[1] +spec = importlib.util.spec_from_file_location( + "backup_install_runtime", ROOT / "scripts/backup_install_runtime.py" +) +runtime = importlib.util.module_from_spec(spec) +spec.loader.exec_module(runtime) + + +@pytest.fixture +def controller(prepared, monkeypatch): + root, manifest, contents, before, _ = prepared + archive, manifest_hash = bundle() + source = (ROOT / "scripts/backup_install.py").read_bytes() + route = b"# synthetic verifier\n" + plan = dict( + attempt_id="20261031bacc0001", + cluster_uid=runtime.CLUSTER_UID, + archive_sha256=runtime.sha(archive), + manifest_sha256=manifest_hash, + capacity_profile=runtime.PROFILE, + expected_before=before, + installer_sha256=runtime.sha(source), + route_verifier_sha256=runtime.sha(route), + route_execution_digest="sha256:" + "a" * 64, + route_packet={"execution_digest": "sha256:" + "a" * 64}, + ) + packet = dict( + plan=plan, + proposal_digest="sha256:" + runtime.sha(runtime.canonical(plan)), + installer=base64.b64encode(source).decode(), + archive=base64.b64encode(archive).decode(), + route_verifier=base64.b64encode(route).decode(), + ) + monkeypatch.setattr(runtime, "ROOT", root) + monkeypatch.setattr(runtime, "ATTEMPTS", root / "var/lib/install-attempts") + # Controller owner guard is synthetic; filesystem library still sees real ownership. + fake_os = SimpleNamespace(**vars(os)) + fake_os.geteuid = lambda: 0 + monkeypatch.setattr(runtime, "os", fake_os) + calls = [] + + @contextmanager + def locked(): + calls.append("lock") + try: + yield + finally: + calls.append("unlock") + + monkeypatch.setattr(runtime, "locks", locked) + monkeypatch.setattr(runtime, "cluster_identity", lambda: calls.append("cluster")) + monkeypatch.setattr(runtime, "inactive_units", lambda: calls.append("inactive") or {}) + monkeypatch.setattr(runtime, "capacity", lambda _: {"fixture": True}) + monkeypatch.setattr(runtime, "runtime_dependencies", lambda: {"fixture": True}) + monkeypatch.setattr( + runtime, "route_verified", lambda *_: calls.append("route") or {"helm_revision": 6} + ) + + def command(args, **kwargs): + assert args == ["systemctl", "daemon-reload"] + calls.append("reload") + return b"" + + monkeypatch.setattr(runtime, "command", command) + return packet, root, calls + + +def test_preflight_holds_both_locks_without_writing(controller): + packet, root, calls = controller + result = runtime.run(packet, "preflight") + assert result["status"] == "preflight_verified" + assert install.observe(root) == packet["plan"]["expected_before"] + assert not runtime.ATTEMPTS.exists() + assert calls[0] == "lock" and calls[-1] == "unlock" + assert "reload" not in calls + + +def test_install_and_explicit_rollback_use_durable_receipts(controller): + packet, root, calls = controller + result = runtime.run(packet, "install") + assert result["status"] == "sources_installed_units_inactive_verified" + assert result["verified_targets"] == 19 + assert result["timers_activated"] is False and result["capture_executed"] is False + directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"] + assert json.loads((directory / "receipt.json").read_text()) == result + assert (directory / "files/receipt.json").is_file() + assert calls.count("reload") == 1 + with pytest.raises(runtime.Refused): + runtime.run(packet, "install") + restored = runtime.run(packet, "rollback") + assert restored["status"] == "prior_files_restored_units_inactive_verified" + now = install.observe(root) + for target, before in packet["plan"]["expected_before"].items(): + assert now[target]["exists"] == before["exists"] + if before["exists"]: + assert all(now[target][key] == before[key] for key in ("sha256", "mode", "uid", "gid")) + assert calls.count("reload") == 2 + + +@pytest.mark.parametrize("boundary", ["cluster", "route", "capacity", "units", "sdk"]) +def test_missing_prerequisite_refuses_before_attempt_and_replacement( + controller, monkeypatch, boundary +): + packet, root, _ = controller + names = dict( + cluster="cluster_identity", + route="route_verified", + capacity="capacity", + units="inactive_units", + sdk="runtime_dependencies", + ) + + def refuse(*_): + raise runtime.Refused("synthetic_refusal") + + monkeypatch.setattr(runtime, names[boundary], refuse) + with pytest.raises(runtime.Refused): + runtime.run(packet, "install") + assert not runtime.ATTEMPTS.exists() + assert install.observe(root) == packet["plan"]["expected_before"] + + +def test_changed_bytes_or_digest_refuse_before_locks(controller): + packet, _, calls = controller + packet["installer"] = base64.b64encode(b"raise RuntimeError('must not execute')").decode() + with pytest.raises(runtime.Refused, match="source_digest_differs"): + runtime.run(packet, "install") + assert calls == [] + + +def test_failed_reload_keeps_preimages_and_requires_explicit_rollback(controller, monkeypatch): + packet, _, _ = controller + + def fail(*_, **__): + raise RuntimeError("synthetic provider text") + + monkeypatch.setattr(runtime, "command", fail) + with pytest.raises(RuntimeError): + runtime.run(packet, "install") + directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"] + receipt = json.loads((directory / "receipt.json").read_text()) + assert receipt["status"] == "installation_requires_inspection" + assert "provider text" not in (directory / "receipt.json").read_text() + files = json.loads((directory / "files/receipt.json").read_text()) + assert files["status"] == "installation_requires_review" + assert files["changes"] + + +def test_postflight_route_failure_is_not_installation_success(controller, monkeypatch): + packet, _, _ = controller + count = 0 + + def route(*_): + nonlocal count + count += 1 + if count == 2: + raise runtime.Refused("route_changed") + return {} + + monkeypatch.setattr(runtime, "route_verified", route) + with pytest.raises(runtime.Refused): + runtime.run(packet, "install") + directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"] + assert ( + json.loads((directory / "receipt.json").read_text())["status"] + == "installation_requires_inspection" + ) + assert ( + json.loads((directory / "files/receipt.json").read_text())["status"] + == "sources_installed_units_inactive_verified" + ) + + +def test_unit_guard_rejects_enabled_failed_or_foreign_units(monkeypatch): + def check(**changes): + row = dict( + LoadState="loaded", ActiveState="inactive", SubState="dead", UnitFileState="disabled" + ) + row.update(changes) + + def command(args, **_): + if "list-jobs" in args: + return b"" + if "list-units" in args: + return b"[]" + fragment = args[2] if "@" not in args[2] else "coulomb-social-backup-alert@.service" + values = dict(row, FragmentPath="/etc/systemd/system/" + fragment) + if changes.get("FragmentPath"): + values["FragmentPath"] = changes["FragmentPath"] + return "\n".join(k + "=" + v for k, v in values.items()).encode() + + monkeypatch.setattr(runtime, "command", command) + return runtime.inactive_units() + + assert len(check()) == 5 + for change in [ + dict(ActiveState="active"), + dict(UnitFileState="enabled"), + dict(LoadState="error"), + dict(FragmentPath="/tmp/foreign.service"), + ]: + with pytest.raises(runtime.Refused): + check(**change) + + +def test_launcher_refuses_nonterminal_without_ssh(monkeypatch): + spec = importlib.util.spec_from_file_location( + "install_backup_launcher", ROOT / "scripts/install-backup-sources.py" + ) + launcher = importlib.util.module_from_spec(spec) + spec.loader.exec_module(launcher) + monkeypatch.setattr(launcher, "prepare", lambda: ({"proposal_digest": "fixture"}, "")) + monkeypatch.setattr(launcher.sys, "stdin", SimpleNamespace(isatty=lambda: False)) + monkeypatch.setattr( + launcher.subprocess, "run", lambda *_args, **_kwargs: pytest.fail("SSH forbidden") + ) + with pytest.raises(SystemExit, match="Founder terminal"): + launcher.main(["--execute", "--expect-digest", "fixture"]) + + +def test_real_locks_are_exclusive_preserved_and_released_on_partial_acquire(tmp_path, monkeypatch): + state = tmp_path / "state" + state.mkdir(mode=0o700) + paths = [state / name for name in ("backup.lock", "snapshot.lock")] + for path in paths: + path.write_bytes(b"retained lock bytes") + path.chmod(0o600) + before = [install.fingerprint(p) for p in paths] + monkeypatch.setattr(runtime, "STATE", state) + with runtime.locks(): + for path in paths: + with path.open("rb") as stream: + with pytest.raises(BlockingIOError): + fcntl.flock(stream, fcntl.LOCK_EX | fcntl.LOCK_NB) + with paths[1].open("rb") as busy: + fcntl.flock(busy, fcntl.LOCK_EX | fcntl.LOCK_NB) + with pytest.raises(BlockingIOError), runtime.locks(): + pytest.fail("second lock must refuse") + with paths[0].open("rb") as first: + fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB) + assert [install.fingerprint(p) for p in paths] == before + + +def test_lock_symlink_refuses_without_replacement(tmp_path, monkeypatch): + state = tmp_path / "state" + state.mkdir(mode=0o700) + target = state / "unrelated" + target.write_bytes(b"unchanged") + (state / "backup.lock").symlink_to(target) + monkeypatch.setattr(runtime, "STATE", state) + with pytest.raises(OSError), runtime.locks(): + pytest.fail("symlink lock must refuse") + assert (state / "backup.lock").is_symlink() and target.read_bytes() == b"unchanged" + + +def test_queued_start_refuses_even_before_inactive_unit_checks(monkeypatch): + calls = [] + + def command(args, **_): + calls.append(args) + return b"123 coulomb-social-paired-backup.service start waiting\n" + + monkeypatch.setattr(runtime, "command", command) + with pytest.raises(runtime.Refused, match="queued_coulomb_unit_job"): + runtime.inactive_units() + assert len(calls) == 1 diff --git a/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md b/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md index 1bc536d..c128bc9 100644 --- a/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md +++ b/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md @@ -558,11 +558,19 @@ status: progress priority: high state_hub_task_id: "a36722a5-5f13-5728-9d58-24f030c2bb00" needs_human: true -intervention_note: "Historical private production-pair copies still need explicit owner disposition; retention/capacity admission remains open. Existing custody and schedule approval are settled." +intervention_note: "Historical private production-pair copies still need owner disposition. Native installer preflight also found eight historical release/date-evidence ownership/mode exceptions rejected by the capacity guard; see backup-state-custody-observation-2026-10-31.json. Preserve evidence; no automatic permission change or deletion. Existing custody and schedule approval are settled." ``` Remaining scope from T01. Close with owner-admitted capacity/headroom limits, a current generation/expiry plan, and explicit retention or deletion disposition for historical private production-pair copies. Execute only admitted expiry and retain native receipts. Proposed limits and a deletion candidate list are not admission. Evidence: `docs/release/backup-capacity-review-2026-10-30.md` and `generation-retention-post-date-2026-10-31.json`. No deletion has occurred. Existing RPF-WP-0053 custody/expiry ownership remains canonical. +Native installer preflight now identifies an additional concrete custody return: +four root-owned0755 release/capsule directories and four helper-owned UID10001 +date-operation entries are inside the capacity-accounted state tree. The existing +strict guard refuses unsafe_state_entry. Metadata-only evidence is retained in +docs/release/backup-state-custody-observation-2026-10-31.json. Prepare a reviewed +disposition preserving historical evidence and its ownership/access requirements; +do not bypass accounting or silently change/delete it. No body was read or changed. + ## T05 — Verify both approved backup alert destinations ```task @@ -607,7 +615,16 @@ priority: high state_hub_task_id: "ca41b874-5e09-538e-9692-d31d990d679b" ``` -The installation/rollback library and 15 disposable-filesystem cases pass; the full gate has 475 passes with zero skips. The final supervised wrapper must bind sources, bundle, preimages, locks, route/profile and inactive-unit guards. Then obtain the attended execution return and verify installed hashes and disabled units. Evidence: `docs/release/backup-source-installer-review-2026-10-31.md` and its validation JSON. Three target files exist; other 16 targets and all five units were absent in the observation. No installation has occurred. +The installation/rollback library and final supervised wrapper are implemented. +The wrapper binds sources, archive/manifest, native preimages, both exclusive +locks, full owner route verification, capacity/profile, SDK/custody and inactive/ +disabled/no-queued-job guards. Explicit rollback retains preimages and refuses +interference. Current native preflight reaches and refuses the capacity guard's +historical custody exceptions; T04 now records that exact blocker. T07's route +receipt is also still absent. No installation attempt or target change occurred. +Then obtain the attended return and independently verify installed hashes and +disabled units. Review: docs/release/backup-source-installation-execution-review-2026-10-31.md; +source/gate evidence: backup-supervised-installer-validation-2026-10-31.json. ## T09 — Prove native paired service and recurring failure reception