Prepare supervised inactive backup installation and record custody blockers
Some checks failed
Use-case regression gate / usecase-regressions (push) Failing after 2m27s
Some checks failed
Use-case regression gate / usecase-regressions (push) Failing after 2m27s
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a18e2d-f513-7742-b8e7-12af0b744ff3
This commit is contained in:
parent
39f39b5ff6
commit
7077ca9db0
10 changed files with 1573 additions and 163 deletions
|
|
@ -0,0 +1,112 @@
|
|||
# Supervised inactive backup-source installation
|
||||
|
||||
The final wrapper is prepared for CSOC-WP-0012-T08. It installs the previously
|
||||
sealed22-member archive into its exact19 target paths, using the existing
|
||||
transaction library. It neither captures data nor starts/enables a service or
|
||||
timer. Current native preflight **refuses**; do not execute this packet yet.
|
||||
|
||||
Proposal:
|
||||
`sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42`.
|
||||
|
||||
The digest binds launcher/runtime/transaction/owner-verifier sources, archive and
|
||||
manifest, native target preimages, cluster identity, the fixed capacity profile
|
||||
and the existing telemetry-route execution packet4d848106. No owner route source
|
||||
or frozen SMTP admission packet was changed.
|
||||
|
||||
## Native findings and disposition
|
||||
|
||||
Read-only preflight holds the existing backup and snapshot locks without replacing
|
||||
or truncating them; verifies primary cluster identity, inactive/disabled units,
|
||||
absence of queued starts, and the exact19 target preimages. It then refuses
|
||||
`unsafe_state_entry` in the existing capacity guard. No installation attempt or
|
||||
target replacement was created.
|
||||
|
||||
Metadata inspection found eight exceptions inside the backup state tree:
|
||||
four root-owned0755 release/capsule directories, and four date-operation private
|
||||
entries owned by helper UID/GID10001. The latter are the private directory,
|
||||
result.json, journal.json and execute.lock. Their bodies were not read. These
|
||||
are historical operation evidence, not newly detected backup corruption.
|
||||
The private ancestors matter to exposure; this observation alone is not a claim
|
||||
that their contents were publicly accessible.
|
||||
|
||||
Exact metadata is in `backup-state-custody-observation-2026-10-31.json`.
|
||||
T04 owns reviewed custody/disposition compatible with the capacity guard. Do not
|
||||
chmod/chown/delete/move these historical records opportunistically, skip them
|
||||
from capacity accounting, or loosen the guard to force installation through.
|
||||
Retention, original evidence paths and the terminated helper's ownership contract
|
||||
must be considered in that disposition. No disposition was executed here.
|
||||
|
||||
The separate telemetry-route receipt is still absent at this observation.
|
||||
T07's existing founder command remains unchanged. Successful route execution and
|
||||
independent native verification are required before this installation can pass.
|
||||
The installer reuses the pinned owner verifier to compare the full release,
|
||||
resources, runtime config, Ready image and preserved PVC, both before and after
|
||||
installation. A past receipt without the current native checks is insufficient.
|
||||
|
||||
## Reviewed operation and future terminal entry
|
||||
|
||||
Offline proposal (no node access):
|
||||
|
||||
```bash
|
||||
python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py
|
||||
```
|
||||
|
||||
Read-only preflight:
|
||||
|
||||
```bash
|
||||
python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py --preflight
|
||||
```
|
||||
|
||||
Only after T04's custody disposition and T07's verified route, refresh preflight
|
||||
and review any changed source/preimage digest. The founder's future installation
|
||||
command is:
|
||||
|
||||
```bash
|
||||
python3 /home/worsch/coulomb-social/scripts/install-backup-sources.py \
|
||||
--execute --expect-digest sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42
|
||||
```
|
||||
|
||||
It requires a terminal and `INSTALL` followed by the full digest. That exact
|
||||
confirmation includes admission of the displayed1GiB new-file limit,8GiB local
|
||||
state budget,8GiB free-space floor,1024-inode minimum and16MiB control margin.
|
||||
Capacity admission here does not resolve rolling provider retention, expiry or
|
||||
historical-copy deletion. Existing node/custody/destination/schedule permissions
|
||||
are preserved; no new schedule or notification destination is being requested.
|
||||
|
||||
The transaction preserves shared platform helpers and the current snapshot pin.
|
||||
It saves preimages, stages files and journals identities before replacing targets.
|
||||
After daemon reload, every target is read back and units must remain inactive,
|
||||
disabled/static and free of queued jobs. Dependency SDK versions and credential
|
||||
file custody are checked without reading credential values. The outer receipt is
|
||||
at `/var/lib/coulomb-social-backup-installations/20261031bacc0001/receipt.json`;
|
||||
the nested `files/receipt.json` and preimages describe the file transaction.
|
||||
|
||||
After interruption or a lost connection, inspect both receipts, process state and
|
||||
native files before any further operation. Existing attempts refuse installation;
|
||||
there is no automatic retry or rollback. The explicit founder rollback uses
|
||||
`--rollback --expect-digest <same-digest>` and confirmation `ROLLBACK <same-digest>`.
|
||||
It takes the same locks, checks cluster/inactive state, refuses interfering file
|
||||
changes, restores only its own journaled targets, reloads and rechecks inactivity.
|
||||
Rollback retains evidence and does not require an operational email route.
|
||||
|
||||
## Validation and acceptance boundary
|
||||
|
||||
The disposable-filesystem tests exercise the real file transaction plus controller
|
||||
ordering, locked preflight, source/preimage/refusal gates, interrupted reload,
|
||||
preserved preimages, explicit rollback, route loss after installation, enabled/
|
||||
failed/foreign units, queued starts and nonterminal refusal. Real file-lock tests
|
||||
prove both locks are exclusive and released after partial acquisition, preserve
|
||||
their bytes/inodes, and reject symlinks.
|
||||
|
||||
The full regression result and exact source hashes are in
|
||||
`backup-supervised-installer-validation-2026-10-31.json`. The gate also exposed
|
||||
formatting errors in two previously added mail preparation scripts; these were
|
||||
corrected. The sender preparer's embedded remote Python AST is unchanged, and
|
||||
the send-free return-mail rehearsal passed again. Historical evidence keeps its
|
||||
original source hashes; the new validation binds the current formatted sources.
|
||||
|
||||
T08 remains progress until actual installation and independent inactive-state
|
||||
readback. T09 still needs native service/failure/retry/stale/producer-death proof;
|
||||
T10 still needs activation and observation after its existing prerequisites.
|
||||
No full service acceptance, mail delivery or migration verdict follows from
|
||||
source preparation alone.
|
||||
|
|
@ -2,9 +2,10 @@
|
|||
|
||||
`scripts/backup_install.py` implements the file transaction needed to install the
|
||||
reviewed operating sources with units inactive. It is a library, with no CLI,
|
||||
provider call, capture or timer activation. The exact supervised calling wrapper
|
||||
and final installation approval are still required; importing this module does
|
||||
not authorize a node mutation.
|
||||
provider call, capture or timer activation. Its supervised calling wrapper is now
|
||||
prepared in `backup-source-installation-execution-review-2026-10-31.md`;
|
||||
prerequisite disposition and final attended installation remain pending.
|
||||
Importing this module does not authorize a node mutation.
|
||||
|
||||
The current sealed archiveeb5dd9ba / manifest84f67186 passes its strict reader:
|
||||
22 regular tar members, including manifest and review evidence, with exactly19
|
||||
|
|
@ -58,7 +59,12 @@ preimages/staging references, symlink/hardlink refusal and unsafe parent modes.
|
|||
No production URL, identity, account or email send participates in these tests.
|
||||
Final source hashes and full regression results are recorded alongside this review.
|
||||
|
||||
The exact terminal wrapper must enforce the caller contract, journal a unique
|
||||
The exact terminal wrapper is now implemented and reviewed in
|
||||
`backup-source-installation-execution-review-2026-10-31.md`. Its native preflight
|
||||
refuses historical state-tree custody exceptions; the owner route receipt also
|
||||
remains absent. Neither prerequisite is bypassed and no installation occurred.
|
||||
|
||||
The wrapper enforces the caller contract, journals a unique
|
||||
attempt, and verify disabled native units after reload. The first admitted native
|
||||
service/capture and received failure/pending/stale signals, restart/replay and
|
||||
producer-death detection remain follow-through. Existing capacity, retention,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,89 @@
|
|||
{
|
||||
"schema": "coulomb.backup-state-custody-observation.v1",
|
||||
"observed_at": "2026-10-31T11:11:43.981497+00:00",
|
||||
"unsafe_entries": [
|
||||
{
|
||||
"relative_path": "releases",
|
||||
"kind": "directory",
|
||||
"mode": "0o755",
|
||||
"uid": 0,
|
||||
"gid": 0,
|
||||
"nlink": 4,
|
||||
"bytes": 4096,
|
||||
"inode": 543834
|
||||
},
|
||||
{
|
||||
"relative_path": "releases/9f1daf6c5fb56b2b4fc9817f28f5ad090c884b0295f7d6f4469e42d48fa8d59f/capsule/charts",
|
||||
"kind": "directory",
|
||||
"mode": "0o755",
|
||||
"uid": 0,
|
||||
"gid": 0,
|
||||
"nlink": 3,
|
||||
"bytes": 4096,
|
||||
"inode": 549330
|
||||
},
|
||||
{
|
||||
"relative_path": "releases/9f1daf6c5fb56b2b4fc9817f28f5ad090c884b0295f7d6f4469e42d48fa8d59f/capsule/release",
|
||||
"kind": "directory",
|
||||
"mode": "0o755",
|
||||
"uid": 0,
|
||||
"gid": 0,
|
||||
"nlink": 3,
|
||||
"bytes": 4096,
|
||||
"inode": 549341
|
||||
},
|
||||
{
|
||||
"relative_path": "releases/5fd7ccd8f68057346367d81efb82b48280a16dd31d87683a5bd83e60e3d1837e/capsule/charts",
|
||||
"kind": "directory",
|
||||
"mode": "0o755",
|
||||
"uid": 0,
|
||||
"gid": 0,
|
||||
"nlink": 3,
|
||||
"bytes": 4096,
|
||||
"inode": 549219
|
||||
},
|
||||
{
|
||||
"relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private",
|
||||
"kind": "directory",
|
||||
"mode": "0o700",
|
||||
"uid": 10001,
|
||||
"gid": 10001,
|
||||
"nlink": 2,
|
||||
"bytes": 4096,
|
||||
"inode": 549263
|
||||
},
|
||||
{
|
||||
"relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/result.json",
|
||||
"kind": "regular",
|
||||
"mode": "0o600",
|
||||
"uid": 10001,
|
||||
"gid": 10001,
|
||||
"nlink": 1,
|
||||
"bytes": 455,
|
||||
"inode": 549311
|
||||
},
|
||||
{
|
||||
"relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/journal.json",
|
||||
"kind": "regular",
|
||||
"mode": "0o600",
|
||||
"uid": 10001,
|
||||
"gid": 10001,
|
||||
"nlink": 1,
|
||||
"bytes": 12930,
|
||||
"inode": 549315
|
||||
},
|
||||
{
|
||||
"relative_path": "date-operations/47829c7ac2e6fd2c81fde1c914aec7c6e433098fa66985db6cfb9b34069942dc/private/execute.lock",
|
||||
"kind": "regular",
|
||||
"mode": "0o600",
|
||||
"uid": 10001,
|
||||
"gid": 10001,
|
||||
"nlink": 1,
|
||||
"bytes": 0,
|
||||
"inode": 549310
|
||||
}
|
||||
],
|
||||
"route_receipt_present": false,
|
||||
"data_read": false,
|
||||
"changed": false
|
||||
}
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
{
|
||||
"schema": "coulomb.backup-supervised-installer-validation.v1",
|
||||
"completed_at": "2026-10-31T11:15:54.935514+00:00",
|
||||
"proposal": {
|
||||
"status": "prepared",
|
||||
"proposal_digest": "sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42",
|
||||
"capacity_profile": {
|
||||
"maximum_new_file_bytes": 1073741824,
|
||||
"state_budget_bytes": 8589934592,
|
||||
"minimum_free_bytes": 8589934592,
|
||||
"minimum_free_inodes": 1024,
|
||||
"control_margin_bytes": 16777216
|
||||
},
|
||||
"targets": 19,
|
||||
"execution_requires_verified_route": true,
|
||||
"installed": false
|
||||
},
|
||||
"native_preflight": {
|
||||
"status": "preflight_refused",
|
||||
"proposal_digest": "sha256:8f442bf0ebed8bbf3b7f227b52081df52613f3f5f6874c4a59511cfc2ba06d42",
|
||||
"failure_type": "Refused",
|
||||
"retry_automatically": false,
|
||||
"reason": "unsafe_state_entry"
|
||||
},
|
||||
"source_files": [
|
||||
{
|
||||
"path": "scripts/backup_install.py",
|
||||
"sha256": "e9128c59e612d2147c8ab5dd54b35805cf0dcbc2b4691c9d4763ae826d2912ca"
|
||||
},
|
||||
{
|
||||
"path": "scripts/backup_install_runtime.py",
|
||||
"sha256": "7e103cf690647b455bf132df7a58fc22e23732f05c1e0889873afdcc591e7097"
|
||||
},
|
||||
{
|
||||
"path": "scripts/install-backup-sources.py",
|
||||
"sha256": "7e5c418ca919b182377efcd501408692c1a03482bb606e03249ec2d52f055afe"
|
||||
},
|
||||
{
|
||||
"path": "tests/test_backup_install.py",
|
||||
"sha256": "2a31df2471918143fc64f7b48d58d303c943679122c336f4d74441d3237a446e"
|
||||
},
|
||||
{
|
||||
"path": "tests/test_backup_install_runtime.py",
|
||||
"sha256": "3888113705cf936072b161fc5e4808221a0bc79a7e9fa18a73b47376a99c7ab1"
|
||||
},
|
||||
{
|
||||
"path": "scripts/prepare-platform-mail-sender.py",
|
||||
"sha256": "eebbeed10af5e2dd82691580078d74d4d3b44ddbd52f7b84035ffb14f8df1889"
|
||||
},
|
||||
{
|
||||
"path": "scripts/rehearse-mail-return-boundary.py",
|
||||
"sha256": "a5b60246bd3b1577ed4064623aa3a39150396192a2e6f6c160af641be50c7edc"
|
||||
}
|
||||
],
|
||||
"full_gate": {
|
||||
"command": "make check",
|
||||
"tests": 490,
|
||||
"errors": 0,
|
||||
"failures": 0,
|
||||
"skipped": 0,
|
||||
"warnings": 69,
|
||||
"log_sha256": "e8c4654a1a14ee85438723c80eb183bcb879740577925c21d40a2e24fc6881b4",
|
||||
"junit_sha256": "59fb491523bf3ce5bac36fe56a8b51ab5e807042ef66c46b0f2edd0b0155179c"
|
||||
},
|
||||
"installer_tests": {
|
||||
"library": 15,
|
||||
"controller": 15
|
||||
},
|
||||
"native_custody_evidence": "backup-state-custody-observation-2026-10-31.json",
|
||||
"mail_lint_followup": {
|
||||
"embedded_remote_preparer_ast_unchanged": true,
|
||||
"owner_sources_unchanged": true,
|
||||
"rehearsal": {
|
||||
"schema": "coulomb.mail-return-boundary-rehearsal.v1",
|
||||
"completed_at": "2026-10-31T11:11:10.314527+00:00",
|
||||
"probe_sha256": "a5b60246bd3b1577ed4064623aa3a39150396192a2e6f6c160af641be50c7edc",
|
||||
"transactional_source_matches_native": true,
|
||||
"smtp_transport": "in-process synthetic replacement; no SMTP connection",
|
||||
"message_id_correlates_with_ledger_and_synthetic_bounce": true,
|
||||
"explicit_reply_to_or_return_path_headers": false,
|
||||
"scanner_deduplicates_evidence": true,
|
||||
"scanner_updates_transactional_suppression": false,
|
||||
"ordinary_sender_suppress_and_unsuppress_denied": true,
|
||||
"synthetic_operator_suppression_blocks_send_and_survives_restart": true,
|
||||
"suppression_scope": "recipient-global across callers and tenants",
|
||||
"explicit_synthetic_operator_unsuppression_verified": true,
|
||||
"native_operator_admission_proved": false,
|
||||
"received_mail_authenticity_or_correlation_authority_proved": false,
|
||||
"python_network_operations_blocked": true,
|
||||
"temporary_ledgers_removed": true,
|
||||
"mail_sent": false,
|
||||
"production_mailbox_accessed": false
|
||||
}
|
||||
},
|
||||
"installed": false,
|
||||
"timers_activated": false,
|
||||
"production_capture_executed": false,
|
||||
"historical_metadata_modified": false
|
||||
}
|
||||
339
scripts/backup_install_runtime.py
Normal file
339
scripts/backup_install_runtime.py
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
"""Remote, attended source installation; never captures, sends or starts a unit."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import subprocess
|
||||
import types
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path("/")
|
||||
STATE = Path("/var/lib/coulomb-social-backup")
|
||||
ATTEMPTS = Path("/var/lib/coulomb-social-backup-installations")
|
||||
ROUTE_ROOT = Path("/var/lib/rapp-telemetry/backup-route-releases")
|
||||
CLUSTER_UID = "a553c742-0115-43d4-99a4-a5ca56fe0786"
|
||||
UNITS = (
|
||||
"coulomb-social-paired-backup.service",
|
||||
"coulomb-social-paired-backup.timer",
|
||||
"coulomb-social-backup-freshness.service",
|
||||
"coulomb-social-backup-freshness.timer",
|
||||
"coulomb-social-backup-alert@coulomb-social-paired-backup.service.service",
|
||||
)
|
||||
PROFILE = dict(
|
||||
maximum_new_file_bytes=1024**3,
|
||||
state_budget_bytes=8 * 1024**3,
|
||||
minimum_free_bytes=8 * 1024**3,
|
||||
minimum_free_inodes=1024,
|
||||
control_margin_bytes=16 * 1024**2,
|
||||
)
|
||||
|
||||
|
||||
class Refused(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def require(condition, reason):
|
||||
if not condition:
|
||||
raise Refused(reason)
|
||||
|
||||
|
||||
def sha(raw):
|
||||
return hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":")).encode()
|
||||
|
||||
|
||||
def module(source, name):
|
||||
result = types.ModuleType(name)
|
||||
exec(compile(source, "<reviewed-" + name + ">", "exec"), result.__dict__)
|
||||
return result
|
||||
|
||||
|
||||
def command(args, timeout=30):
|
||||
result = subprocess.run(
|
||||
args,
|
||||
capture_output=True,
|
||||
timeout=timeout,
|
||||
env=dict(os.environ, KUBECONFIG="/etc/rancher/k3s/k3s.yaml"),
|
||||
)
|
||||
require(
|
||||
result.returncode == 0 and len(result.stdout) < 16 * 1024 * 1024, "bounded_command_failed"
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def cluster_identity():
|
||||
namespace = json.loads(
|
||||
command(["k3s", "kubectl", "get", "namespace", "kube-system", "-o", "json"])
|
||||
)
|
||||
require(namespace["metadata"]["uid"] == CLUSTER_UID, "primary_cluster_mismatch")
|
||||
|
||||
|
||||
def inactive_units():
|
||||
rows = {}
|
||||
# This host's list-jobs does not implement JSON output; no-legend emits
|
||||
# empty output for no jobs, otherwise JOB UNIT TYPE STATE rows.
|
||||
jobs = command(["systemctl", "list-jobs", "--no-legend", "--no-pager", "--plain"])
|
||||
jobs = [line.split() for line in jobs.decode().splitlines() if line.strip()]
|
||||
require(all(len(row) == 4 and row[0].isdecimal() for row in jobs), "job_listing_unrecognized")
|
||||
require(
|
||||
not any(row[1].startswith("coulomb-social-") for row in jobs),
|
||||
"queued_coulomb_unit_job",
|
||||
)
|
||||
# Also reject an already running alternative instance of the alert template.
|
||||
listing = json.loads(
|
||||
command(
|
||||
["systemctl", "list-units", "--all", "--output=json", "--no-pager", "coulomb-social-*"]
|
||||
)
|
||||
)
|
||||
require(all(x.get("active") == "inactive" for x in listing), "coulomb_unit_active_or_failed")
|
||||
for unit in UNITS:
|
||||
raw = command(
|
||||
[
|
||||
"systemctl",
|
||||
"show",
|
||||
unit,
|
||||
"--property=LoadState,ActiveState,SubState,UnitFileState,FragmentPath",
|
||||
]
|
||||
)
|
||||
row = dict(line.split("=", 1) for line in raw.decode().splitlines() if "=" in line)
|
||||
require(row["ActiveState"] == "inactive" and row["SubState"] == "dead", "unit_not_inactive")
|
||||
require(
|
||||
row["LoadState"] in ("loaded", "not-found")
|
||||
and row.get("UnitFileState", "") in ("", "disabled", "static"),
|
||||
"unit_enabled_or_unexpected",
|
||||
)
|
||||
if row["LoadState"] == "loaded":
|
||||
fragment = unit if "@" not in unit else "coulomb-social-backup-alert@.service"
|
||||
require(
|
||||
row["FragmentPath"] == "/etc/systemd/system/" + fragment, "foreign_unit_fragment"
|
||||
)
|
||||
rows[unit] = row
|
||||
return rows
|
||||
|
||||
|
||||
@contextmanager
|
||||
def locks():
|
||||
handles = []
|
||||
try:
|
||||
info = STATE.lstat()
|
||||
require(
|
||||
stat.S_ISDIR(info.st_mode) and info.st_uid == os.geteuid() and not info.st_mode & 0o077,
|
||||
"private_backup_state_required",
|
||||
)
|
||||
for name in ("backup.lock", "snapshot.lock"):
|
||||
# Existing lock only; no truncation, replacement or lock-file creation.
|
||||
fd = os.open(STATE / name, os.O_RDONLY | os.O_NOFOLLOW)
|
||||
handles.append(fd)
|
||||
info = os.fstat(fd)
|
||||
require(
|
||||
stat.S_ISREG(info.st_mode)
|
||||
and info.st_uid == os.geteuid()
|
||||
and info.st_nlink == 1
|
||||
and not info.st_mode & 0o077,
|
||||
"unsafe_lock_file",
|
||||
)
|
||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
yield
|
||||
finally:
|
||||
for fd in reversed(handles):
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def verified_payload(packet):
|
||||
require(__debug__, "optimized_python_refused")
|
||||
plan = packet["plan"]
|
||||
require(
|
||||
packet["proposal_digest"] == "sha256:" + sha(canonical(plan)), "proposal_digest_differs"
|
||||
)
|
||||
require(
|
||||
plan["capacity_profile"] == PROFILE and plan["cluster_uid"] == CLUSTER_UID,
|
||||
"operating_scope_differs",
|
||||
)
|
||||
sources = {}
|
||||
for key in ("installer", "route_verifier"):
|
||||
raw = base64.b64decode(packet[key], validate=True)
|
||||
require(sha(raw) == plan[key + "_sha256"], "source_digest_differs")
|
||||
sources[key] = raw
|
||||
archive = base64.b64decode(packet["archive"], validate=True)
|
||||
installer = module(sources["installer"], "backup_install")
|
||||
manifest, contents = installer.read_bundle(
|
||||
archive, plan["archive_sha256"], plan["manifest_sha256"]
|
||||
)
|
||||
require(
|
||||
plan["route_packet"]["execution_digest"] == plan["route_execution_digest"],
|
||||
"route_digest_differs",
|
||||
)
|
||||
return installer, manifest, contents, sources
|
||||
|
||||
|
||||
def route_verified(plan, source):
|
||||
path = ROUTE_ROOT / plan["route_execution_digest"].split(":")[1] / "receipt.json"
|
||||
require(path.is_file() and not path.is_symlink(), "route_receipt_missing")
|
||||
info = path.stat()
|
||||
require(
|
||||
info.st_uid == 0 and not info.st_mode & 0o077 and info.st_size < 65536,
|
||||
"route_receipt_not_private",
|
||||
)
|
||||
receipt = json.loads(path.read_bytes())
|
||||
require(
|
||||
receipt["status"] == "backup_email_route_deployed_verified"
|
||||
and receipt["execution_digest"] == plan["route_execution_digest"],
|
||||
"route_not_verified",
|
||||
)
|
||||
verifier = module(source, "backup_route_runtime")
|
||||
revision = verifier.verify_return(plan["route_packet"], after=True)
|
||||
require(revision == receipt["helm_revision"], "route_revision_differs")
|
||||
return {"helm_revision": revision, "runtime_and_complete_resources_verified": True}
|
||||
|
||||
|
||||
def capacity(contents):
|
||||
guard = module(contents["consumer/backup_capacity.py"], "backup_capacity")
|
||||
require(
|
||||
(
|
||||
guard.FILE_LIMIT,
|
||||
guard.STATE_BUDGET,
|
||||
guard.MINIMUM_FREE,
|
||||
guard.MINIMUM_INODES,
|
||||
guard.CONTROL_MARGIN,
|
||||
)
|
||||
== tuple(PROFILE.values()),
|
||||
"capacity_profile_differs",
|
||||
)
|
||||
try:
|
||||
report = guard.reserve_plan(STATE, require_file_limit=False)
|
||||
except guard.Refused as error:
|
||||
raise Refused(str(error)) from None
|
||||
require(not report["reasons"], "capacity_headroom_refused")
|
||||
return {
|
||||
k: report[k]
|
||||
for k in (
|
||||
"state_bytes",
|
||||
"state_reserve_bytes",
|
||||
"disk_reserve_bytes",
|
||||
"free_bytes",
|
||||
"free_inodes",
|
||||
"filesystem_blocks_reserved",
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def runtime_dependencies():
|
||||
code = (
|
||||
"import json,boto3,botocore; print(json.dumps(dict(boto3=boto3.__version__,"
|
||||
"botocore=botocore.__version__)))"
|
||||
)
|
||||
actual = json.loads(command(["/opt/railiance-platform/backup-venv/bin/python", "-c", code]))
|
||||
require(actual == {"boto3": "1.40.18", "botocore": "1.40.76"}, "sdk_dependency_drift")
|
||||
# Check credential references' custody, without reading their contents.
|
||||
for name in ("primary.env", "recipient.env"):
|
||||
path = Path("/etc/coulomb-social-backup") / name
|
||||
info = path.lstat()
|
||||
require(
|
||||
stat.S_ISREG(info.st_mode)
|
||||
and info.st_uid == 0
|
||||
and info.st_nlink == 1
|
||||
and not info.st_mode & 0o077,
|
||||
"credential_reference_custody_differs",
|
||||
)
|
||||
return actual
|
||||
|
||||
|
||||
def preflight(plan, installer, contents, source):
|
||||
cluster_identity()
|
||||
units = inactive_units()
|
||||
before = installer.observe(ROOT)
|
||||
require(before == plan["expected_before"], "native_preimage_changed")
|
||||
require(not (ATTEMPTS / plan["attempt_id"]).exists(), "existing_attempt_requires_inspection")
|
||||
profile = capacity(contents)
|
||||
sdk = runtime_dependencies()
|
||||
route = route_verified(plan, source)
|
||||
return dict(units=units, capacity=profile, sdk=sdk, route=route)
|
||||
|
||||
|
||||
def run(packet, action):
|
||||
require(action in ("preflight", "install", "rollback"), "unknown_action")
|
||||
require(os.geteuid() == 0, "root_owner_execution_required")
|
||||
installer, manifest, contents, sources = verified_payload(packet)
|
||||
plan = packet["plan"]
|
||||
directory = ATTEMPTS / plan["attempt_id"]
|
||||
require(
|
||||
len(plan["attempt_id"]) == 16 and all(c in "0123456789abcdef" for c in plan["attempt_id"]),
|
||||
"invalid_attempt_id",
|
||||
)
|
||||
installer.trusted_parents(ROOT, directory / "receipt.json")
|
||||
with locks():
|
||||
if action == "rollback":
|
||||
cluster_identity()
|
||||
inactive_units()
|
||||
require(directory.is_dir() and not directory.is_symlink(), "attempt_missing")
|
||||
prior = json.loads((directory / "receipt.json").read_text())
|
||||
require(
|
||||
prior["proposal_digest"] == packet["proposal_digest"], "attempt_proposal_differs"
|
||||
)
|
||||
record = json.loads((directory / "files/receipt.json").read_text())
|
||||
prior.update(status="rollback_in_progress", updated_at=datetime.now(UTC).isoformat())
|
||||
installer.journal(directory, prior)
|
||||
installer.restore(
|
||||
ROOT, directory / "files", record, lambda: command(["systemctl", "daemon-reload"])
|
||||
)
|
||||
inactive_units()
|
||||
prior.update(status="prior_files_restored_units_inactive_verified")
|
||||
installer.journal(directory, prior)
|
||||
return prior
|
||||
result = preflight(plan, installer, contents, sources["route_verifier"])
|
||||
receipt = dict(
|
||||
schema="coulomb.backup-source-supervised-installation.v1",
|
||||
proposal_digest=packet["proposal_digest"],
|
||||
status="preflight_verified",
|
||||
preflight=result,
|
||||
timers_activated=False,
|
||||
capture_executed=False,
|
||||
email_sent=False,
|
||||
retained_data_deleted=False,
|
||||
)
|
||||
if action == "preflight":
|
||||
return receipt
|
||||
directory.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
receipt.update(
|
||||
status="installing",
|
||||
started_at=datetime.now(UTC).isoformat(),
|
||||
capacity_profile=PROFILE,
|
||||
founder_confirmed_profile_and_installation=True,
|
||||
)
|
||||
installer.journal(directory, receipt)
|
||||
try:
|
||||
files = installer.install(
|
||||
manifest,
|
||||
contents,
|
||||
plan["expected_before"],
|
||||
directory / "files",
|
||||
root=ROOT,
|
||||
reload_manager=lambda: command(["systemctl", "daemon-reload"]),
|
||||
verify_inactive=inactive_units,
|
||||
)
|
||||
# Recheck the external dependency after installation, without a send.
|
||||
receipt["route_postflight"] = route_verified(plan, sources["route_verifier"])
|
||||
receipt.update(
|
||||
status="sources_installed_units_inactive_verified",
|
||||
verified_targets=files["verified_targets"],
|
||||
units=inactive_units(),
|
||||
)
|
||||
except Exception as error:
|
||||
receipt.update(
|
||||
status="installation_requires_inspection", failure_type=type(error).__name__
|
||||
)
|
||||
raise
|
||||
finally:
|
||||
receipt["updated_at"] = datetime.now(UTC).isoformat()
|
||||
installer.journal(directory, receipt)
|
||||
return receipt
|
||||
160
scripts/install-backup-sources.py
Normal file
160
scripts/install-backup-sources.py
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Review and attend the fixed inactive backup-source installation or rollback."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OWNER = Path("/home/worsch/rapp-telemetry")
|
||||
DELIVERY = ROOT / "docs/release/backup-operating-delivery-alerts-2026-10-31.json"
|
||||
OBSERVATION = ROOT / "docs/release/backup-install-target-observation-2026-10-31.json"
|
||||
ROUTE_DIGEST = "sha256:4d8481061dc9ce6835169defc96887945052332a0db8d4234ae84032d63cc328"
|
||||
|
||||
|
||||
def module(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
loaded = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(loaded)
|
||||
return loaded
|
||||
|
||||
|
||||
def prepare():
|
||||
if not __debug__:
|
||||
raise ValueError("optimized_python_refused")
|
||||
runtime = module("backup_install_runtime", ROOT / "scripts/backup_install_runtime.py")
|
||||
installer = (ROOT / "scripts/backup_install.py").read_bytes()
|
||||
implementation = module("backup_install", ROOT / "scripts/backup_install.py")
|
||||
delivery = json.loads(DELIVERY.read_text())
|
||||
archive = Path(delivery["private_archive"]).read_bytes()
|
||||
implementation.read_bundle(archive, delivery["archive_sha256"], delivery["manifest_sha256"])
|
||||
route = module("reviewed_backup_route", OWNER / "tools/deploy_coulomb_backup_route.py")
|
||||
route_packet, route_source = route.prepare(route.DEFAULT)
|
||||
if route_packet["execution_digest"] != ROUTE_DIGEST:
|
||||
raise ValueError("reviewed_owner_route_changed")
|
||||
# Verification needs no chart upload and does not perform an upgrade.
|
||||
route_packet.pop("chart")
|
||||
source = (ROOT / "scripts/backup_install_runtime.py").read_text()
|
||||
plan = dict(
|
||||
schema="coulomb.backup-source-installation-proposal.v1",
|
||||
attempt_id="20261031bacc0001",
|
||||
cluster_uid=runtime.CLUSTER_UID,
|
||||
archive_sha256=delivery["archive_sha256"],
|
||||
manifest_sha256=delivery["manifest_sha256"],
|
||||
capacity_profile=runtime.PROFILE,
|
||||
expected_before=json.loads(OBSERVATION.read_text())["targets"],
|
||||
route_execution_digest=ROUTE_DIGEST,
|
||||
route_packet=route_packet,
|
||||
installer_sha256=hashlib.sha256(installer).hexdigest(),
|
||||
route_verifier_sha256=hashlib.sha256(route_source.encode()).hexdigest(),
|
||||
runtime_sha256=hashlib.sha256(source.encode()).hexdigest(),
|
||||
launcher_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
|
||||
)
|
||||
packet = dict(
|
||||
plan=plan,
|
||||
proposal_digest="sha256:" + runtime.sha(runtime.canonical(plan)),
|
||||
archive=base64.b64encode(archive).decode(),
|
||||
installer=base64.b64encode(installer).decode(),
|
||||
route_verifier=base64.b64encode(route_source.encode()).decode(),
|
||||
)
|
||||
return packet, source
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
actions = parser.add_mutually_exclusive_group()
|
||||
actions.add_argument("--preflight", action="store_true")
|
||||
actions.add_argument("--execute", action="store_true")
|
||||
actions.add_argument("--rollback", action="store_true")
|
||||
parser.add_argument("--expect-digest")
|
||||
args = parser.parse_args(argv)
|
||||
packet, source = prepare()
|
||||
digest = packet["proposal_digest"]
|
||||
if args.expect_digest and args.expect_digest != digest:
|
||||
raise SystemExit("Reviewed source installation proposal changed; no action")
|
||||
if not (args.preflight or args.execute or args.rollback):
|
||||
print(
|
||||
json.dumps(
|
||||
dict(
|
||||
status="prepared",
|
||||
proposal_digest=digest,
|
||||
capacity_profile=packet["plan"]["capacity_profile"],
|
||||
targets=len(packet["plan"]["expected_before"]),
|
||||
execution_requires_verified_route=True,
|
||||
installed=False,
|
||||
)
|
||||
)
|
||||
)
|
||||
return 0
|
||||
if args.execute or args.rollback:
|
||||
if not args.expect_digest or not sys.stdin.isatty():
|
||||
raise SystemExit("Founder terminal and reviewed digest required")
|
||||
verb = "ROLLBACK" if args.rollback else "INSTALL"
|
||||
print(
|
||||
"Restore this installation only, with units inactive."
|
||||
if args.rollback
|
||||
else "Admit the displayed capacity profile and install the reviewed sources "
|
||||
"with units inactive."
|
||||
)
|
||||
print(json.dumps(packet["plan"]["capacity_profile"], sort_keys=True))
|
||||
if input("Type " + verb + " followed by the full digest: ").strip() != verb + " " + digest:
|
||||
raise SystemExit("Not confirmed; no action")
|
||||
packet["action"] = "rollback" if args.rollback else "install" if args.execute else "preflight"
|
||||
bootstrap = (
|
||||
source
|
||||
+ """
|
||||
import sys
|
||||
packet=json.load(sys.stdin)
|
||||
try:
|
||||
result=run(packet,packet['action'])
|
||||
except Exception as error:
|
||||
status='preflight_refused' if packet['action']=='preflight' else 'installation_requires_inspection'
|
||||
result=dict(status=status,
|
||||
proposal_digest=packet['proposal_digest'],failure_type=type(error).__name__,retry_automatically=False)
|
||||
if isinstance(error,Refused):result['reason']=str(error)
|
||||
print(json.dumps(result))
|
||||
sys.exit(0 if result['status'] in ('preflight_verified','sources_installed_units_inactive_verified',
|
||||
'prior_files_restored_units_inactive_verified') else 1)
|
||||
"""
|
||||
)
|
||||
try:
|
||||
response = subprocess.run(
|
||||
[
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=10",
|
||||
"railiance01",
|
||||
"sudo -n python3 -c " + shlex.quote(bootstrap),
|
||||
],
|
||||
input=json.dumps(packet).encode(),
|
||||
capture_output=True,
|
||||
timeout=600,
|
||||
)
|
||||
result = json.loads(response.stdout)
|
||||
except (subprocess.TimeoutExpired, ValueError):
|
||||
print(
|
||||
json.dumps(
|
||||
dict(
|
||||
status="remote_outcome_unconfirmed",
|
||||
proposal_digest=digest,
|
||||
retry_automatically=False,
|
||||
)
|
||||
)
|
||||
)
|
||||
return 1
|
||||
print(json.dumps(result))
|
||||
return int(response.returncode != 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -4,63 +4,303 @@
|
|||
No create/apply/patch is executed without --dry-run=server. This does not admit
|
||||
OpenBao access, install a resource, scan a mailbox or send an email.
|
||||
"""
|
||||
import argparse, os, subprocess,shlex,json,hashlib
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from datetime import datetime,timezone
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
if not __debug__:
|
||||
raise SystemExit('Assertions must be enabled')
|
||||
raise SystemExit("Assertions must be enabled")
|
||||
if args.output.exists():
|
||||
raise SystemExit('Choose a new candidate output; existing evidence is retained')
|
||||
code=r'''
|
||||
import subprocess,json,hashlib
|
||||
from datetime import datetime,timezone
|
||||
K=['k3s','kubectl']
|
||||
def get(kind,name):
|
||||
return json.loads(subprocess.check_output(K+['get',kind,name,'-n','email-connect','-o','json'],timeout=20))
|
||||
d=get('deployment','email-connect');cm=get('configmap','email-connect-config')
|
||||
assert d['metadata']['uid']=='2e98e52a-a2d1-4c52-9ac8-6efbc0a18e72'
|
||||
assert d['status']['observedGeneration']==d['metadata']['generation'] and d['status']['readyReplicas']==1
|
||||
c=d['spec']['template']['spec']['containers'][0]
|
||||
assert c['name']=='email-connect' and len(d['spec']['template']['spec']['containers'])==1
|
||||
assert c['image']=='forgejo.coulomb.social/coulomb/email-connect@sha256:5d79fd1fec15d3a279383820f170d263de3a55f5308df01a0a34732ebaea6d3d'
|
||||
assert c['envFrom']==[{'configMapRef':{'name':'email-connect-config'}},{'secretRef':{'name':'email-connect-runtime','optional':False}}]
|
||||
assert cm['data']['EMAIL_CONNECT_SENDER']=='forgejo@coulomb.social'
|
||||
assert all(e['name'] not in {'EMAIL_CONNECT_SMTP_USERNAME','EMAIL_CONNECT_SMTP_PASSWORD','EMAIL_CONNECT_SENDER'} for e in c['env'])
|
||||
assert c['env']==[{'name':'EMAIL_CONNECT_CALLERS_FILE','value':'/etc/email-connect-callers/callers.json'}], 'unexpected environment; refuse to copy unknown values'
|
||||
role='email-connect-platform-smtp-eso';store='openbao-email-connect-platform-smtp';secret='email-connect-platform-smtp'
|
||||
def obj(api,kind,name,**extra):return {'apiVersion':api,'kind':kind,'metadata':{'name':name,'namespace':'email-connect'},**extra}
|
||||
resources=[obj('v1','ServiceAccount',role,automountServiceAccountToken=False),obj('external-secrets.io/v1','SecretStore',store,spec={'provider':{'vault':{'server':'http://openbao.openbao.svc:8200','path':'platform','version':'v2','auth':{'kubernetes':{'mountPath':'kubernetes','role':role,'serviceAccountRef':{'name':role,'audiences':['openbao']}}}}}}),obj('external-secrets.io/v1','ExternalSecret',secret,spec={'refreshInterval':'1h','secretStoreRef':{'kind':'SecretStore','name':store},'target':{'name':secret,'creationPolicy':'Owner','deletionPolicy':'Retain','template':{'type':'Opaque'}},'data':[{'secretKey':'password','remoteRef':{'key':'workloads/railiance-telemetry/smtp','property':'SMTP_PASSWORD'}}]})]
|
||||
raise SystemExit("Choose a new candidate output; existing evidence is retained")
|
||||
code = r"""
|
||||
import subprocess, json, hashlib
|
||||
from datetime import datetime, timezone
|
||||
|
||||
K = ["k3s", "kubectl"]
|
||||
|
||||
|
||||
def get(kind, name):
|
||||
return json.loads(
|
||||
subprocess.check_output(
|
||||
K + ["get", kind, name, "-n", "email-connect", "-o", "json"], timeout=20
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
d = get("deployment", "email-connect")
|
||||
cm = get("configmap", "email-connect-config")
|
||||
assert d["metadata"]["uid"] == "2e98e52a-a2d1-4c52-9ac8-6efbc0a18e72"
|
||||
assert (
|
||||
d["status"]["observedGeneration"] == d["metadata"]["generation"]
|
||||
and d["status"]["readyReplicas"] == 1
|
||||
)
|
||||
c = d["spec"]["template"]["spec"]["containers"][0]
|
||||
assert c["name"] == "email-connect" and len(d["spec"]["template"]["spec"]["containers"]) == 1
|
||||
assert (
|
||||
c["image"]
|
||||
== "forgejo.coulomb.social/coulomb/email-connect@sha256:"
|
||||
"5d79fd1fec15d3a279383820f170d263de3a55f5308df01a0a34732ebaea6d3d"
|
||||
)
|
||||
assert c["envFrom"] == [
|
||||
{"configMapRef": {"name": "email-connect-config"}},
|
||||
{"secretRef": {"name": "email-connect-runtime", "optional": False}},
|
||||
]
|
||||
assert cm["data"]["EMAIL_CONNECT_SENDER"] == "forgejo@coulomb.social"
|
||||
assert all(
|
||||
e["name"]
|
||||
not in {"EMAIL_CONNECT_SMTP_USERNAME", "EMAIL_CONNECT_SMTP_PASSWORD", "EMAIL_CONNECT_SENDER"}
|
||||
for e in c["env"]
|
||||
)
|
||||
assert c["env"] == [
|
||||
{"name": "EMAIL_CONNECT_CALLERS_FILE", "value": "/etc/email-connect-callers/callers.json"}
|
||||
], "unexpected environment; refuse to copy unknown values"
|
||||
role = "email-connect-platform-smtp-eso"
|
||||
store = "openbao-email-connect-platform-smtp"
|
||||
secret = "email-connect-platform-smtp"
|
||||
|
||||
|
||||
def obj(api, kind, name, **extra):
|
||||
return {
|
||||
"apiVersion": api,
|
||||
"kind": kind,
|
||||
"metadata": {"name": name, "namespace": "email-connect"},
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
resources = [
|
||||
obj("v1", "ServiceAccount", role, automountServiceAccountToken=False),
|
||||
obj(
|
||||
"external-secrets.io/v1",
|
||||
"SecretStore",
|
||||
store,
|
||||
spec={
|
||||
"provider": {
|
||||
"vault": {
|
||||
"server": "http://openbao.openbao.svc:8200",
|
||||
"path": "platform",
|
||||
"version": "v2",
|
||||
"auth": {
|
||||
"kubernetes": {
|
||||
"mountPath": "kubernetes",
|
||||
"role": role,
|
||||
"serviceAccountRef": {"name": role, "audiences": ["openbao"]},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
},
|
||||
),
|
||||
obj(
|
||||
"external-secrets.io/v1",
|
||||
"ExternalSecret",
|
||||
secret,
|
||||
spec={
|
||||
"refreshInterval": "1h",
|
||||
"secretStoreRef": {"kind": "SecretStore", "name": store},
|
||||
"target": {
|
||||
"name": secret,
|
||||
"creationPolicy": "Owner",
|
||||
"deletionPolicy": "Retain",
|
||||
"template": {"type": "Opaque"},
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"secretKey": "password",
|
||||
"remoteRef": {
|
||||
"key": "workloads/railiance-telemetry/smtp",
|
||||
"property": "SMTP_PASSWORD",
|
||||
},
|
||||
}
|
||||
],
|
||||
},
|
||||
),
|
||||
]
|
||||
for x in resources:
|
||||
out=subprocess.check_output(K+['get',x['kind'],x['metadata']['name'],'-n','email-connect','--ignore-not-found','-o','name'],timeout=20)
|
||||
assert not out.strip(),'candidate resource already exists'
|
||||
base='/spec/template/spec/containers/0'
|
||||
dpatch=[{'op':'test','path':'/metadata/uid','value':d['metadata']['uid']},{'op':'test','path':'/metadata/resourceVersion','value':d['metadata']['resourceVersion']},{'op':'test','path':base+'/image','value':c['image']},{'op':'test','path':base+'/envFrom','value':c['envFrom']},{'op':'test','path':base+'/env','value':c['env']},{'op':'replace','path':base+'/envFrom','value':[{'configMapRef':{'name':'email-connect-config'}}]},{'op':'replace','path':base+'/env','value':c['env']+[{'name':'EMAIL_CONNECT_SMTP_USERNAME','value':'platform@coulomb.social'},{'name':'EMAIL_CONNECT_SMTP_PASSWORD','valueFrom':{'secretKeyRef':{'name':secret,'key':'password','optional':False}}}]}]
|
||||
cmpatch=[{'op':'test','path':'/metadata/uid','value':cm['metadata']['uid']},{'op':'test','path':'/data/EMAIL_CONNECT_SENDER','value':'forgejo@coulomb.social'},{'op':'replace','path':'/data/EMAIL_CONNECT_SENDER','value':'platform@coulomb.social'}]
|
||||
def patchdry(kind,name,patch):
|
||||
return json.loads(subprocess.check_output(K+['patch',kind,name,'-n','email-connect','--type=json','--dry-run=server','-p',json.dumps(patch),'-o','json'],timeout=30))
|
||||
checks=[]
|
||||
out = subprocess.check_output(
|
||||
K
|
||||
+ [
|
||||
"get",
|
||||
x["kind"],
|
||||
x["metadata"]["name"],
|
||||
"-n",
|
||||
"email-connect",
|
||||
"--ignore-not-found",
|
||||
"-o",
|
||||
"name",
|
||||
],
|
||||
timeout=20,
|
||||
)
|
||||
assert not out.strip(), "candidate resource already exists"
|
||||
base = "/spec/template/spec/containers/0"
|
||||
dpatch = [
|
||||
{"op": "test", "path": "/metadata/uid", "value": d["metadata"]["uid"]},
|
||||
{"op": "test", "path": "/metadata/resourceVersion", "value": d["metadata"]["resourceVersion"]},
|
||||
{"op": "test", "path": base + "/image", "value": c["image"]},
|
||||
{"op": "test", "path": base + "/envFrom", "value": c["envFrom"]},
|
||||
{"op": "test", "path": base + "/env", "value": c["env"]},
|
||||
{
|
||||
"op": "replace",
|
||||
"path": base + "/envFrom",
|
||||
"value": [{"configMapRef": {"name": "email-connect-config"}}],
|
||||
},
|
||||
{
|
||||
"op": "replace",
|
||||
"path": base + "/env",
|
||||
"value": c["env"]
|
||||
+ [
|
||||
{"name": "EMAIL_CONNECT_SMTP_USERNAME", "value": "platform@coulomb.social"},
|
||||
{
|
||||
"name": "EMAIL_CONNECT_SMTP_PASSWORD",
|
||||
"valueFrom": {
|
||||
"secretKeyRef": {"name": secret, "key": "password", "optional": False}
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
cmpatch = [
|
||||
{"op": "test", "path": "/metadata/uid", "value": cm["metadata"]["uid"]},
|
||||
{"op": "test", "path": "/data/EMAIL_CONNECT_SENDER", "value": "forgejo@coulomb.social"},
|
||||
{"op": "replace", "path": "/data/EMAIL_CONNECT_SENDER", "value": "platform@coulomb.social"},
|
||||
]
|
||||
|
||||
|
||||
def patchdry(kind, name, patch):
|
||||
return json.loads(
|
||||
subprocess.check_output(
|
||||
K
|
||||
+ [
|
||||
"patch",
|
||||
kind,
|
||||
name,
|
||||
"-n",
|
||||
"email-connect",
|
||||
"--type=json",
|
||||
"--dry-run=server",
|
||||
"-p",
|
||||
json.dumps(patch),
|
||||
"-o",
|
||||
"json",
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
checks = []
|
||||
for x in resources:
|
||||
p=subprocess.run(K+['create','--dry-run=server','-f','-','-o','json'],input=json.dumps(x).encode(),capture_output=True,timeout=30)
|
||||
if p.returncode:raise RuntimeError('resource dry-run failed: '+x['kind'])
|
||||
y=json.loads(p.stdout);checks.append({'kind':y['kind'],'name':y['metadata']['name'],'accepted':True})
|
||||
da=patchdry('deployment','email-connect',dpatch);ca=patchdry('configmap','email-connect-config',cmpatch)
|
||||
expected=json.loads(json.dumps(d['spec']));expected['template']['spec']['containers'][0]['envFrom']=dpatch[-2]['value'];expected['template']['spec']['containers'][0]['env']=dpatch[-1]['value']
|
||||
assert da['spec']==expected,'unexpected deployment spec change'
|
||||
expectedcm=dict(cm['data'],EMAIL_CONNECT_SENDER='platform@coulomb.social');assert ca['data']==expectedcm
|
||||
checks.extend([{'kind':'Deployment','name':'email-connect','accepted':True,'only_envFrom_and_env_changed':True},{'kind':'ConfigMap','name':'email-connect-config','accepted':True,'only_sender_changed':True}])
|
||||
volumes=d['spec']['template']['spec']['volumes']; policy_name=next(v['configMap']['name'] for v in volumes if v['name']=='caller-policy')
|
||||
pol=get('configmap',policy_name)['data']['callers.json'];parsed=json.loads(pol)
|
||||
publiccallers=[{k:x.get(k) for k in ('caller_id','operations','templates')} for x in parsed['callers']]
|
||||
print(json.dumps({'schema':'coulomb.platform-mail-sender-candidate.v1','observed_at':datetime.now(timezone.utc).isoformat(),'deployment_uid':d['metadata']['uid'],'deployment_generation':d['metadata']['generation'],'image':c['image'],'new_resources':resources,'deployment_json_patch':dpatch,'configmap_json_patch':cmpatch,'native_server_dry_run':checks,'live_caller_policy':{'name':policy_name,'sha256':hashlib.sha256(pol.encode()).hexdigest(),'callers':publiccallers},'platform_mailbox_admission_executed':False,'resources_installed':False,'mail_sent':False,'secrets_read':False,'execution_ready':False}))
|
||||
'''
|
||||
r=subprocess.run(['ssh','-o','BatchMode=yes','-o','ConnectTimeout=10','railiance01','sudo -n python3 -c '+shlex.quote(code)],capture_output=True,text=True,timeout=180)
|
||||
if r.returncode:raise SystemExit('read/dry-run preparation failed: '+r.stderr[-1500:])
|
||||
j=json.loads(r.stdout)
|
||||
j['preparer_sha256']=hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
|
||||
fd=os.open(args.output,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600)
|
||||
with os.fdopen(fd,'w') as target:
|
||||
json.dump(j,target,indent=2);target.write('\n');target.flush();os.fsync(target.fileno())
|
||||
print(json.dumps({k:j[k] for k in ('observed_at','deployment_generation','native_server_dry_run','live_caller_policy','execution_ready')},indent=2))
|
||||
p = subprocess.run(
|
||||
K + ["create", "--dry-run=server", "-f", "-", "-o", "json"],
|
||||
input=json.dumps(x).encode(),
|
||||
capture_output=True,
|
||||
timeout=30,
|
||||
)
|
||||
if p.returncode:
|
||||
raise RuntimeError("resource dry-run failed: " + x["kind"])
|
||||
y = json.loads(p.stdout)
|
||||
checks.append({"kind": y["kind"], "name": y["metadata"]["name"], "accepted": True})
|
||||
da = patchdry("deployment", "email-connect", dpatch)
|
||||
ca = patchdry("configmap", "email-connect-config", cmpatch)
|
||||
expected = json.loads(json.dumps(d["spec"]))
|
||||
expected["template"]["spec"]["containers"][0]["envFrom"] = dpatch[-2]["value"]
|
||||
expected["template"]["spec"]["containers"][0]["env"] = dpatch[-1]["value"]
|
||||
assert da["spec"] == expected, "unexpected deployment spec change"
|
||||
expectedcm = dict(cm["data"], EMAIL_CONNECT_SENDER="platform@coulomb.social")
|
||||
assert ca["data"] == expectedcm
|
||||
checks.extend(
|
||||
[
|
||||
{
|
||||
"kind": "Deployment",
|
||||
"name": "email-connect",
|
||||
"accepted": True,
|
||||
"only_envFrom_and_env_changed": True,
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "email-connect-config",
|
||||
"accepted": True,
|
||||
"only_sender_changed": True,
|
||||
},
|
||||
]
|
||||
)
|
||||
volumes = d["spec"]["template"]["spec"]["volumes"]
|
||||
policy_name = next(v["configMap"]["name"] for v in volumes if v["name"] == "caller-policy")
|
||||
pol = get("configmap", policy_name)["data"]["callers.json"]
|
||||
parsed = json.loads(pol)
|
||||
publiccallers = [
|
||||
{k: x.get(k) for k in ("caller_id", "operations", "templates")} for x in parsed["callers"]
|
||||
]
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"schema": "coulomb.platform-mail-sender-candidate.v1",
|
||||
"observed_at": datetime.now(timezone.utc).isoformat(),
|
||||
"deployment_uid": d["metadata"]["uid"],
|
||||
"deployment_generation": d["metadata"]["generation"],
|
||||
"image": c["image"],
|
||||
"new_resources": resources,
|
||||
"deployment_json_patch": dpatch,
|
||||
"configmap_json_patch": cmpatch,
|
||||
"native_server_dry_run": checks,
|
||||
"live_caller_policy": {
|
||||
"name": policy_name,
|
||||
"sha256": hashlib.sha256(pol.encode()).hexdigest(),
|
||||
"callers": publiccallers,
|
||||
},
|
||||
"platform_mailbox_admission_executed": False,
|
||||
"resources_installed": False,
|
||||
"mail_sent": False,
|
||||
"secrets_read": False,
|
||||
"execution_ready": False,
|
||||
}
|
||||
)
|
||||
)
|
||||
"""
|
||||
r = subprocess.run(
|
||||
[
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=10",
|
||||
"railiance01",
|
||||
"sudo -n python3 -c " + shlex.quote(code),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=180,
|
||||
)
|
||||
if r.returncode:
|
||||
raise SystemExit("read/dry-run preparation failed: " + r.stderr[-1500:])
|
||||
j = json.loads(r.stdout)
|
||||
j["preparer_sha256"] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
|
||||
fd = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(fd, "w") as target:
|
||||
json.dump(j, target, indent=2)
|
||||
target.write("\n")
|
||||
target.flush()
|
||||
os.fsync(target.fileno())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
k: j[k]
|
||||
for k in (
|
||||
"observed_at",
|
||||
"deployment_generation",
|
||||
"native_server_dry_run",
|
||||
"live_caller_policy",
|
||||
"execution_ready",
|
||||
)
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
This probe documents present capabilities and missing integration. It is not an
|
||||
inbound worker, sender deployment, credential admission or suppression operator.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
|
@ -12,158 +13,220 @@ import io
|
|||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
def main():
|
||||
if not __debug__:
|
||||
raise SystemExit('Assertions must be enabled')
|
||||
raise SystemExit("Assertions must be enabled")
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--owner-repo', type=Path, required=True)
|
||||
parser.add_argument('--native-evidence', type=Path, required=True)
|
||||
parser.add_argument("--owner-repo", type=Path, required=True)
|
||||
parser.add_argument("--native-evidence", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
source = args.owner_repo.resolve() / 'src'
|
||||
source = args.owner_repo.resolve() / "src"
|
||||
native = json.loads(args.native_evidence.read_text())
|
||||
module = source / 'email_connect/transactional.py'
|
||||
assert hashlib.sha256(module.read_bytes()).hexdigest() == native[
|
||||
'running_process_configuration']['transactional_source_sha256']
|
||||
module = source / "email_connect/transactional.py"
|
||||
assert (
|
||||
hashlib.sha256(module.read_bytes()).hexdigest()
|
||||
== native["running_process_configuration"]["transactional_source_sha256"]
|
||||
)
|
||||
|
||||
# Fail any attempted Python network connection/bind, even if a future source
|
||||
# change accidentally substitutes a real transport for these fixtures.
|
||||
def no_network(event, _args):
|
||||
if event in {'socket.connect', 'socket.bind', 'socket.getaddrinfo', 'socket.sendto'}:
|
||||
raise RuntimeError('Offline rehearsal forbids network access')
|
||||
if event in {"socket.connect", "socket.bind", "socket.getaddrinfo", "socket.sendto"}:
|
||||
raise RuntimeError("Offline rehearsal forbids network access")
|
||||
|
||||
sys.addaudithook(no_network)
|
||||
sys.path.insert(0, str(source))
|
||||
from email_connect.callers import CallerPolicy, ScopedTransactionalApplication
|
||||
from email_connect.config import (AppConfig, MailboxConfig, ReportsConfig,
|
||||
ScanConfig, SourceConfig, StorageConfig)
|
||||
from email_connect.config import (
|
||||
AppConfig,
|
||||
MailboxConfig,
|
||||
ReportsConfig,
|
||||
ScanConfig,
|
||||
SourceConfig,
|
||||
StorageConfig,
|
||||
)
|
||||
from email_connect.scanner import scan_mailbox
|
||||
from email_connect.storage import StateStore
|
||||
from email_connect.transactional import SMTPProvider, SQLiteDeliveryStore
|
||||
|
||||
captured = []
|
||||
|
||||
class SyntheticSMTP:
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.tls = False
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
pass
|
||||
|
||||
def starttls(self, context):
|
||||
assert context.check_hostname
|
||||
self.tls = True
|
||||
|
||||
def login(self, username, password):
|
||||
assert self.tls and username == 'fixture' and password == 'fixture-only'
|
||||
assert self.tls and username == "fixture" and password == "fixture-only"
|
||||
|
||||
def send_message(self, message):
|
||||
assert message['To'] == 'recipient@example.test'
|
||||
assert message["To"] == "recipient@example.test"
|
||||
captured.append(message)
|
||||
|
||||
def policy(name, operations, tenant='tenant:fixture'):
|
||||
return CallerPolicy(caller_id=name, token='fixture-credential-' + name,
|
||||
source=name, bindings=frozenset({(tenant, 'client:fixture')}),
|
||||
operations=frozenset(operations),
|
||||
templates=frozenset({'family_member.invited'}),
|
||||
rate_per_minute=60, burst=30, max_payload_bytes=8192)
|
||||
sender = policy('fixture-sender', {'send', 'delivery-status'})
|
||||
operator = policy('fixture-operator', {'suppress', 'unsuppress'})
|
||||
other = policy('fixture-other', {'send'}, 'tenant:other')
|
||||
def event(name, caller=sender, tenant='tenant:fixture'):
|
||||
return {'id': name, 'source': caller.source, 'type': 'family_member.invited',
|
||||
'tenant': tenant, 'client_id': 'client:fixture',
|
||||
'data': {'primary_email': 'recipient@example.test', 'invitation_id': name}}
|
||||
def invoke(app, payload, path='/v1/send', caller=sender):
|
||||
raw = json.dumps(payload).encode(); result = {}
|
||||
env = {'PATH_INFO': path, 'REQUEST_METHOD': 'POST', 'CONTENT_LENGTH': str(len(raw)),
|
||||
'wsgi.input': io.BytesIO(raw), 'HTTP_AUTHORIZATION': 'Bearer ' + caller.token,
|
||||
'HTTP_IDEMPOTENCY_KEY': payload.get('id', '')}
|
||||
body = b''.join(app(env, lambda status, headers: result.update(status=int(status[:3]))))
|
||||
return result['status'], json.loads(body)
|
||||
def policy(name, operations, tenant="tenant:fixture"):
|
||||
return CallerPolicy(
|
||||
caller_id=name,
|
||||
token="fixture-credential-" + name,
|
||||
source=name,
|
||||
bindings=frozenset({(tenant, "client:fixture")}),
|
||||
operations=frozenset(operations),
|
||||
templates=frozenset({"family_member.invited"}),
|
||||
rate_per_minute=60,
|
||||
burst=30,
|
||||
max_payload_bytes=8192,
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix='csoc-mail-return-') as tmp, patch(
|
||||
'email_connect.transactional.smtplib.SMTP', SyntheticSMTP):
|
||||
sender = policy("fixture-sender", {"send", "delivery-status"})
|
||||
operator = policy("fixture-operator", {"suppress", "unsuppress"})
|
||||
other = policy("fixture-other", {"send"}, "tenant:other")
|
||||
|
||||
def event(name, caller=sender, tenant="tenant:fixture"):
|
||||
return {
|
||||
"id": name,
|
||||
"source": caller.source,
|
||||
"type": "family_member.invited",
|
||||
"tenant": tenant,
|
||||
"client_id": "client:fixture",
|
||||
"data": {"primary_email": "recipient@example.test", "invitation_id": name},
|
||||
}
|
||||
|
||||
def invoke(app, payload, path="/v1/send", caller=sender):
|
||||
raw = json.dumps(payload).encode()
|
||||
result = {}
|
||||
env = {
|
||||
"PATH_INFO": path,
|
||||
"REQUEST_METHOD": "POST",
|
||||
"CONTENT_LENGTH": str(len(raw)),
|
||||
"wsgi.input": io.BytesIO(raw),
|
||||
"HTTP_AUTHORIZATION": "Bearer " + caller.token,
|
||||
"HTTP_IDEMPOTENCY_KEY": payload.get("id", ""),
|
||||
}
|
||||
body = b"".join(app(env, lambda status, headers: result.update(status=int(status[:3]))))
|
||||
return result["status"], json.loads(body)
|
||||
|
||||
with (
|
||||
tempfile.TemporaryDirectory(prefix="csoc-mail-return-") as tmp,
|
||||
patch("email_connect.transactional.smtplib.SMTP", SyntheticSMTP),
|
||||
):
|
||||
root = Path(tmp)
|
||||
ledger = SQLiteDeliveryStore(str(root / 'transactional.sqlite'))
|
||||
provider = SMTPProvider('smtp.example.test', 587, 'fixture', 'fixture-only',
|
||||
'platform@example.test')
|
||||
app = ScopedTransactionalApplication(ledger, provider, 'https://issuer.example.test',
|
||||
[sender, operator, other])
|
||||
assert invoke(app, event('invitation-1'))[0] == 202
|
||||
ledger = SQLiteDeliveryStore(str(root / "transactional.sqlite"))
|
||||
provider = SMTPProvider(
|
||||
"smtp.example.test", 587, "fixture", "fixture-only", "platform@example.test"
|
||||
)
|
||||
app = ScopedTransactionalApplication(
|
||||
ledger, provider, "https://issuer.example.test", [sender, operator, other]
|
||||
)
|
||||
assert invoke(app, event("invitation-1"))[0] == 202
|
||||
assert len(captured) == 1
|
||||
message = captured[0]
|
||||
reference = str(message['Message-ID'])
|
||||
assert message['From'] == 'platform@example.test'
|
||||
assert message['Reply-To'] is None and message['Return-Path'] is None
|
||||
assert ledger.db.execute('SELECT count(*) FROM deliveries WHERE provider_ref=?',
|
||||
(reference,)).fetchone()[0] == 1
|
||||
fixture = root / 'mailbox'; fixture.mkdir()
|
||||
(fixture / 'bounce.eml').write_text(
|
||||
'From: mailer-daemon@example.test\nTo: platform@example.test\n'
|
||||
'Subject: Delivery Status Notification (Failure)\n'
|
||||
'Date: Sat, 31 Oct 2026 10:00:00 +0000\n'
|
||||
'Message-ID: <fixture-bounce@example.test>\n'
|
||||
'Content-Type: text/plain; charset=utf-8\n\n'
|
||||
'Delivery failure.\nFinal-Recipient: rfc822; recipient@example.test\n'
|
||||
'Action: failed\nStatus: 5.1.1\nDiagnostic-Code: smtp; 550 User unknown\n'
|
||||
'Original-Message-ID: ' + reference + '\n')
|
||||
config = AppConfig(mailbox=MailboxConfig(id='fixture-mailbox', protocol='fixture'),
|
||||
scan=ScanConfig(), storage=StorageConfig(str(root / 'scanner.sqlite')),
|
||||
reports=ReportsConfig(str(root / 'reports')),
|
||||
source=SourceConfig(fixture_dir=str(fixture)))
|
||||
reference = str(message["Message-ID"])
|
||||
assert message["From"] == "platform@example.test"
|
||||
assert message["Reply-To"] is None and message["Return-Path"] is None
|
||||
assert (
|
||||
ledger.db.execute(
|
||||
"SELECT count(*) FROM deliveries WHERE provider_ref=?", (reference,)
|
||||
).fetchone()[0]
|
||||
== 1
|
||||
)
|
||||
fixture = root / "mailbox"
|
||||
fixture.mkdir()
|
||||
(fixture / "bounce.eml").write_text(
|
||||
"From: mailer-daemon@example.test\nTo: platform@example.test\n"
|
||||
"Subject: Delivery Status Notification (Failure)\n"
|
||||
"Date: Sat, 31 Oct 2026 10:00:00 +0000\n"
|
||||
"Message-ID: <fixture-bounce@example.test>\n"
|
||||
"Content-Type: text/plain; charset=utf-8\n\n"
|
||||
"Delivery failure.\nFinal-Recipient: rfc822; recipient@example.test\n"
|
||||
"Action: failed\nStatus: 5.1.1\nDiagnostic-Code: smtp; 550 User unknown\n"
|
||||
"Original-Message-ID: " + reference + "\n"
|
||||
)
|
||||
config = AppConfig(
|
||||
mailbox=MailboxConfig(id="fixture-mailbox", protocol="fixture"),
|
||||
scan=ScanConfig(),
|
||||
storage=StorageConfig(str(root / "scanner.sqlite")),
|
||||
reports=ReportsConfig(str(root / "reports")),
|
||||
source=SourceConfig(fixture_dir=str(fixture)),
|
||||
)
|
||||
scan = scan_mailbox(config)
|
||||
evidence = StateStore(config.storage.path)
|
||||
rows = evidence.evidence_rows()
|
||||
assert scan.scan.messages_seen == 1 and len(rows) == 1
|
||||
assert rows[0]['original_message_id'] == reference
|
||||
assert rows[0]['affected_email_address'] == 'recipient@example.test'
|
||||
assert rows[0]['assessment_subclass'] == 'fail.hard_bounce'
|
||||
assert not ledger.is_suppressed('recipient@example.test')
|
||||
assert rows[0]["original_message_id"] == reference
|
||||
assert rows[0]["affected_email_address"] == "recipient@example.test"
|
||||
assert rows[0]["assessment_subclass"] == "fail.hard_bounce"
|
||||
assert not ledger.is_suppressed("recipient@example.test")
|
||||
repeat = scan_mailbox(config)
|
||||
assert repeat.scan.messages_new == 0 and repeat.scan.evidence_events_created == 0
|
||||
payload = {'recipient': 'recipient@example.test', 'reason': 'hard_bounce'}
|
||||
assert invoke(app, payload, '/v1/suppressions')[0] == 403
|
||||
assert not ledger.is_suppressed('recipient@example.test')
|
||||
assert invoke(app, payload, '/v1/suppressions', operator)[0] == 200
|
||||
assert invoke(app, event('invitation-2')) == (400, {'error': 'recipient_suppressed'})
|
||||
assert invoke(app, event('invitation-3', other, 'tenant:other'), caller=other) == (
|
||||
400, {'error': 'recipient_suppressed'})
|
||||
payload = {"recipient": "recipient@example.test", "reason": "hard_bounce"}
|
||||
assert invoke(app, payload, "/v1/suppressions")[0] == 403
|
||||
assert not ledger.is_suppressed("recipient@example.test")
|
||||
assert invoke(app, payload, "/v1/suppressions", operator)[0] == 200
|
||||
assert invoke(app, event("invitation-2")) == (400, {"error": "recipient_suppressed"})
|
||||
assert invoke(app, event("invitation-3", other, "tenant:other"), caller=other) == (
|
||||
400,
|
||||
{"error": "recipient_suppressed"},
|
||||
)
|
||||
assert len(captured) == 1
|
||||
ledger.db.close()
|
||||
ledger = SQLiteDeliveryStore(str(root / 'transactional.sqlite'))
|
||||
assert ledger.is_suppressed('recipient@example.test')
|
||||
app = ScopedTransactionalApplication(ledger, provider, 'https://issuer.example.test',
|
||||
[sender, operator, other])
|
||||
assert invoke(app, payload, '/v1/suppressions/remove')[0] == 403
|
||||
assert invoke(app, payload, '/v1/suppressions/remove', operator)[0] == 200
|
||||
assert not ledger.is_suppressed('recipient@example.test')
|
||||
ledger = SQLiteDeliveryStore(str(root / "transactional.sqlite"))
|
||||
assert ledger.is_suppressed("recipient@example.test")
|
||||
app = ScopedTransactionalApplication(
|
||||
ledger, provider, "https://issuer.example.test", [sender, operator, other]
|
||||
)
|
||||
assert invoke(app, payload, "/v1/suppressions/remove")[0] == 403
|
||||
assert invoke(app, payload, "/v1/suppressions/remove", operator)[0] == 200
|
||||
assert not ledger.is_suppressed("recipient@example.test")
|
||||
assert len(captured) == 1
|
||||
ledger.db.close(); evidence.close()
|
||||
ledger.db.close()
|
||||
evidence.close()
|
||||
assert not root.exists()
|
||||
print(json.dumps({
|
||||
'schema': 'coulomb.mail-return-boundary-rehearsal.v1',
|
||||
'completed_at': datetime.now(timezone.utc).isoformat(),
|
||||
'source_sha256': {str(p.relative_to(args.owner_repo.resolve())):
|
||||
hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
for p in sorted((source / 'email_connect').glob('*.py'))},
|
||||
'probe_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
|
||||
'transactional_source_matches_native': True,
|
||||
'smtp_transport': 'in-process synthetic replacement; no SMTP connection',
|
||||
'message_id_correlates_with_ledger_and_synthetic_bounce': True,
|
||||
'explicit_reply_to_or_return_path_headers': False,
|
||||
'scanner_deduplicates_evidence': True,
|
||||
'scanner_updates_transactional_suppression': False,
|
||||
'ordinary_sender_suppress_and_unsuppress_denied': True,
|
||||
'synthetic_operator_suppression_blocks_send_and_survives_restart': True,
|
||||
'suppression_scope': 'recipient-global across callers and tenants',
|
||||
'explicit_synthetic_operator_unsuppression_verified': True,
|
||||
'native_operator_admission_proved': False,
|
||||
'received_mail_authenticity_or_correlation_authority_proved': False,
|
||||
'python_network_operations_blocked': True,
|
||||
'temporary_ledgers_removed': True,
|
||||
'mail_sent': False, 'production_mailbox_accessed': False,
|
||||
}, indent=2))
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"schema": "coulomb.mail-return-boundary-rehearsal.v1",
|
||||
"completed_at": datetime.now(UTC).isoformat(),
|
||||
"source_sha256": {
|
||||
str(p.relative_to(args.owner_repo.resolve())): hashlib.sha256(
|
||||
p.read_bytes()
|
||||
).hexdigest()
|
||||
for p in sorted((source / "email_connect").glob("*.py"))
|
||||
},
|
||||
"probe_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
|
||||
"transactional_source_matches_native": True,
|
||||
"smtp_transport": "in-process synthetic replacement; no SMTP connection",
|
||||
"message_id_correlates_with_ledger_and_synthetic_bounce": True,
|
||||
"explicit_reply_to_or_return_path_headers": False,
|
||||
"scanner_deduplicates_evidence": True,
|
||||
"scanner_updates_transactional_suppression": False,
|
||||
"ordinary_sender_suppress_and_unsuppress_denied": True,
|
||||
"synthetic_operator_suppression_blocks_send_and_survives_restart": True,
|
||||
"suppression_scope": "recipient-global across callers and tenants",
|
||||
"explicit_synthetic_operator_unsuppression_verified": True,
|
||||
"native_operator_admission_proved": False,
|
||||
"received_mail_authenticity_or_correlation_authority_proved": False,
|
||||
"python_network_operations_blocked": True,
|
||||
"temporary_ledgers_removed": True,
|
||||
"mail_sent": False,
|
||||
"production_mailbox_accessed": False,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
285
tests/test_backup_install_runtime.py
Normal file
285
tests/test_backup_install_runtime.py
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
"""Attended controller transactions against disposable files and fake host boundaries."""
|
||||
|
||||
import base64
|
||||
import fcntl
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from tests import test_backup_install as fixtures
|
||||
|
||||
bundle = fixtures.bundle
|
||||
install = fixtures.install
|
||||
prepared = fixtures.prepared
|
||||
|
||||
ROOT = Path(__file__).parents[1]
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"backup_install_runtime", ROOT / "scripts/backup_install_runtime.py"
|
||||
)
|
||||
runtime = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(runtime)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def controller(prepared, monkeypatch):
|
||||
root, manifest, contents, before, _ = prepared
|
||||
archive, manifest_hash = bundle()
|
||||
source = (ROOT / "scripts/backup_install.py").read_bytes()
|
||||
route = b"# synthetic verifier\n"
|
||||
plan = dict(
|
||||
attempt_id="20261031bacc0001",
|
||||
cluster_uid=runtime.CLUSTER_UID,
|
||||
archive_sha256=runtime.sha(archive),
|
||||
manifest_sha256=manifest_hash,
|
||||
capacity_profile=runtime.PROFILE,
|
||||
expected_before=before,
|
||||
installer_sha256=runtime.sha(source),
|
||||
route_verifier_sha256=runtime.sha(route),
|
||||
route_execution_digest="sha256:" + "a" * 64,
|
||||
route_packet={"execution_digest": "sha256:" + "a" * 64},
|
||||
)
|
||||
packet = dict(
|
||||
plan=plan,
|
||||
proposal_digest="sha256:" + runtime.sha(runtime.canonical(plan)),
|
||||
installer=base64.b64encode(source).decode(),
|
||||
archive=base64.b64encode(archive).decode(),
|
||||
route_verifier=base64.b64encode(route).decode(),
|
||||
)
|
||||
monkeypatch.setattr(runtime, "ROOT", root)
|
||||
monkeypatch.setattr(runtime, "ATTEMPTS", root / "var/lib/install-attempts")
|
||||
# Controller owner guard is synthetic; filesystem library still sees real ownership.
|
||||
fake_os = SimpleNamespace(**vars(os))
|
||||
fake_os.geteuid = lambda: 0
|
||||
monkeypatch.setattr(runtime, "os", fake_os)
|
||||
calls = []
|
||||
|
||||
@contextmanager
|
||||
def locked():
|
||||
calls.append("lock")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
calls.append("unlock")
|
||||
|
||||
monkeypatch.setattr(runtime, "locks", locked)
|
||||
monkeypatch.setattr(runtime, "cluster_identity", lambda: calls.append("cluster"))
|
||||
monkeypatch.setattr(runtime, "inactive_units", lambda: calls.append("inactive") or {})
|
||||
monkeypatch.setattr(runtime, "capacity", lambda _: {"fixture": True})
|
||||
monkeypatch.setattr(runtime, "runtime_dependencies", lambda: {"fixture": True})
|
||||
monkeypatch.setattr(
|
||||
runtime, "route_verified", lambda *_: calls.append("route") or {"helm_revision": 6}
|
||||
)
|
||||
|
||||
def command(args, **kwargs):
|
||||
assert args == ["systemctl", "daemon-reload"]
|
||||
calls.append("reload")
|
||||
return b""
|
||||
|
||||
monkeypatch.setattr(runtime, "command", command)
|
||||
return packet, root, calls
|
||||
|
||||
|
||||
def test_preflight_holds_both_locks_without_writing(controller):
|
||||
packet, root, calls = controller
|
||||
result = runtime.run(packet, "preflight")
|
||||
assert result["status"] == "preflight_verified"
|
||||
assert install.observe(root) == packet["plan"]["expected_before"]
|
||||
assert not runtime.ATTEMPTS.exists()
|
||||
assert calls[0] == "lock" and calls[-1] == "unlock"
|
||||
assert "reload" not in calls
|
||||
|
||||
|
||||
def test_install_and_explicit_rollback_use_durable_receipts(controller):
|
||||
packet, root, calls = controller
|
||||
result = runtime.run(packet, "install")
|
||||
assert result["status"] == "sources_installed_units_inactive_verified"
|
||||
assert result["verified_targets"] == 19
|
||||
assert result["timers_activated"] is False and result["capture_executed"] is False
|
||||
directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"]
|
||||
assert json.loads((directory / "receipt.json").read_text()) == result
|
||||
assert (directory / "files/receipt.json").is_file()
|
||||
assert calls.count("reload") == 1
|
||||
with pytest.raises(runtime.Refused):
|
||||
runtime.run(packet, "install")
|
||||
restored = runtime.run(packet, "rollback")
|
||||
assert restored["status"] == "prior_files_restored_units_inactive_verified"
|
||||
now = install.observe(root)
|
||||
for target, before in packet["plan"]["expected_before"].items():
|
||||
assert now[target]["exists"] == before["exists"]
|
||||
if before["exists"]:
|
||||
assert all(now[target][key] == before[key] for key in ("sha256", "mode", "uid", "gid"))
|
||||
assert calls.count("reload") == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("boundary", ["cluster", "route", "capacity", "units", "sdk"])
|
||||
def test_missing_prerequisite_refuses_before_attempt_and_replacement(
|
||||
controller, monkeypatch, boundary
|
||||
):
|
||||
packet, root, _ = controller
|
||||
names = dict(
|
||||
cluster="cluster_identity",
|
||||
route="route_verified",
|
||||
capacity="capacity",
|
||||
units="inactive_units",
|
||||
sdk="runtime_dependencies",
|
||||
)
|
||||
|
||||
def refuse(*_):
|
||||
raise runtime.Refused("synthetic_refusal")
|
||||
|
||||
monkeypatch.setattr(runtime, names[boundary], refuse)
|
||||
with pytest.raises(runtime.Refused):
|
||||
runtime.run(packet, "install")
|
||||
assert not runtime.ATTEMPTS.exists()
|
||||
assert install.observe(root) == packet["plan"]["expected_before"]
|
||||
|
||||
|
||||
def test_changed_bytes_or_digest_refuse_before_locks(controller):
|
||||
packet, _, calls = controller
|
||||
packet["installer"] = base64.b64encode(b"raise RuntimeError('must not execute')").decode()
|
||||
with pytest.raises(runtime.Refused, match="source_digest_differs"):
|
||||
runtime.run(packet, "install")
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_failed_reload_keeps_preimages_and_requires_explicit_rollback(controller, monkeypatch):
|
||||
packet, _, _ = controller
|
||||
|
||||
def fail(*_, **__):
|
||||
raise RuntimeError("synthetic provider text")
|
||||
|
||||
monkeypatch.setattr(runtime, "command", fail)
|
||||
with pytest.raises(RuntimeError):
|
||||
runtime.run(packet, "install")
|
||||
directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"]
|
||||
receipt = json.loads((directory / "receipt.json").read_text())
|
||||
assert receipt["status"] == "installation_requires_inspection"
|
||||
assert "provider text" not in (directory / "receipt.json").read_text()
|
||||
files = json.loads((directory / "files/receipt.json").read_text())
|
||||
assert files["status"] == "installation_requires_review"
|
||||
assert files["changes"]
|
||||
|
||||
|
||||
def test_postflight_route_failure_is_not_installation_success(controller, monkeypatch):
|
||||
packet, _, _ = controller
|
||||
count = 0
|
||||
|
||||
def route(*_):
|
||||
nonlocal count
|
||||
count += 1
|
||||
if count == 2:
|
||||
raise runtime.Refused("route_changed")
|
||||
return {}
|
||||
|
||||
monkeypatch.setattr(runtime, "route_verified", route)
|
||||
with pytest.raises(runtime.Refused):
|
||||
runtime.run(packet, "install")
|
||||
directory = runtime.ATTEMPTS / packet["plan"]["attempt_id"]
|
||||
assert (
|
||||
json.loads((directory / "receipt.json").read_text())["status"]
|
||||
== "installation_requires_inspection"
|
||||
)
|
||||
assert (
|
||||
json.loads((directory / "files/receipt.json").read_text())["status"]
|
||||
== "sources_installed_units_inactive_verified"
|
||||
)
|
||||
|
||||
|
||||
def test_unit_guard_rejects_enabled_failed_or_foreign_units(monkeypatch):
|
||||
def check(**changes):
|
||||
row = dict(
|
||||
LoadState="loaded", ActiveState="inactive", SubState="dead", UnitFileState="disabled"
|
||||
)
|
||||
row.update(changes)
|
||||
|
||||
def command(args, **_):
|
||||
if "list-jobs" in args:
|
||||
return b""
|
||||
if "list-units" in args:
|
||||
return b"[]"
|
||||
fragment = args[2] if "@" not in args[2] else "coulomb-social-backup-alert@.service"
|
||||
values = dict(row, FragmentPath="/etc/systemd/system/" + fragment)
|
||||
if changes.get("FragmentPath"):
|
||||
values["FragmentPath"] = changes["FragmentPath"]
|
||||
return "\n".join(k + "=" + v for k, v in values.items()).encode()
|
||||
|
||||
monkeypatch.setattr(runtime, "command", command)
|
||||
return runtime.inactive_units()
|
||||
|
||||
assert len(check()) == 5
|
||||
for change in [
|
||||
dict(ActiveState="active"),
|
||||
dict(UnitFileState="enabled"),
|
||||
dict(LoadState="error"),
|
||||
dict(FragmentPath="/tmp/foreign.service"),
|
||||
]:
|
||||
with pytest.raises(runtime.Refused):
|
||||
check(**change)
|
||||
|
||||
|
||||
def test_launcher_refuses_nonterminal_without_ssh(monkeypatch):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"install_backup_launcher", ROOT / "scripts/install-backup-sources.py"
|
||||
)
|
||||
launcher = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(launcher)
|
||||
monkeypatch.setattr(launcher, "prepare", lambda: ({"proposal_digest": "fixture"}, ""))
|
||||
monkeypatch.setattr(launcher.sys, "stdin", SimpleNamespace(isatty=lambda: False))
|
||||
monkeypatch.setattr(
|
||||
launcher.subprocess, "run", lambda *_args, **_kwargs: pytest.fail("SSH forbidden")
|
||||
)
|
||||
with pytest.raises(SystemExit, match="Founder terminal"):
|
||||
launcher.main(["--execute", "--expect-digest", "fixture"])
|
||||
|
||||
|
||||
def test_real_locks_are_exclusive_preserved_and_released_on_partial_acquire(tmp_path, monkeypatch):
|
||||
state = tmp_path / "state"
|
||||
state.mkdir(mode=0o700)
|
||||
paths = [state / name for name in ("backup.lock", "snapshot.lock")]
|
||||
for path in paths:
|
||||
path.write_bytes(b"retained lock bytes")
|
||||
path.chmod(0o600)
|
||||
before = [install.fingerprint(p) for p in paths]
|
||||
monkeypatch.setattr(runtime, "STATE", state)
|
||||
with runtime.locks():
|
||||
for path in paths:
|
||||
with path.open("rb") as stream:
|
||||
with pytest.raises(BlockingIOError):
|
||||
fcntl.flock(stream, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
with paths[1].open("rb") as busy:
|
||||
fcntl.flock(busy, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
with pytest.raises(BlockingIOError), runtime.locks():
|
||||
pytest.fail("second lock must refuse")
|
||||
with paths[0].open("rb") as first:
|
||||
fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
assert [install.fingerprint(p) for p in paths] == before
|
||||
|
||||
|
||||
def test_lock_symlink_refuses_without_replacement(tmp_path, monkeypatch):
|
||||
state = tmp_path / "state"
|
||||
state.mkdir(mode=0o700)
|
||||
target = state / "unrelated"
|
||||
target.write_bytes(b"unchanged")
|
||||
(state / "backup.lock").symlink_to(target)
|
||||
monkeypatch.setattr(runtime, "STATE", state)
|
||||
with pytest.raises(OSError), runtime.locks():
|
||||
pytest.fail("symlink lock must refuse")
|
||||
assert (state / "backup.lock").is_symlink() and target.read_bytes() == b"unchanged"
|
||||
|
||||
|
||||
def test_queued_start_refuses_even_before_inactive_unit_checks(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def command(args, **_):
|
||||
calls.append(args)
|
||||
return b"123 coulomb-social-paired-backup.service start waiting\n"
|
||||
|
||||
monkeypatch.setattr(runtime, "command", command)
|
||||
with pytest.raises(runtime.Refused, match="queued_coulomb_unit_job"):
|
||||
runtime.inactive_units()
|
||||
assert len(calls) == 1
|
||||
|
|
@ -558,11 +558,19 @@ status: progress
|
|||
priority: high
|
||||
state_hub_task_id: "a36722a5-5f13-5728-9d58-24f030c2bb00"
|
||||
needs_human: true
|
||||
intervention_note: "Historical private production-pair copies still need explicit owner disposition; retention/capacity admission remains open. Existing custody and schedule approval are settled."
|
||||
intervention_note: "Historical private production-pair copies still need owner disposition. Native installer preflight also found eight historical release/date-evidence ownership/mode exceptions rejected by the capacity guard; see backup-state-custody-observation-2026-10-31.json. Preserve evidence; no automatic permission change or deletion. Existing custody and schedule approval are settled."
|
||||
```
|
||||
|
||||
Remaining scope from T01. Close with owner-admitted capacity/headroom limits, a current generation/expiry plan, and explicit retention or deletion disposition for historical private production-pair copies. Execute only admitted expiry and retain native receipts. Proposed limits and a deletion candidate list are not admission. Evidence: `docs/release/backup-capacity-review-2026-10-30.md` and `generation-retention-post-date-2026-10-31.json`. No deletion has occurred. Existing RPF-WP-0053 custody/expiry ownership remains canonical.
|
||||
|
||||
Native installer preflight now identifies an additional concrete custody return:
|
||||
four root-owned0755 release/capsule directories and four helper-owned UID10001
|
||||
date-operation entries are inside the capacity-accounted state tree. The existing
|
||||
strict guard refuses unsafe_state_entry. Metadata-only evidence is retained in
|
||||
docs/release/backup-state-custody-observation-2026-10-31.json. Prepare a reviewed
|
||||
disposition preserving historical evidence and its ownership/access requirements;
|
||||
do not bypass accounting or silently change/delete it. No body was read or changed.
|
||||
|
||||
## T05 — Verify both approved backup alert destinations
|
||||
|
||||
```task
|
||||
|
|
@ -607,7 +615,16 @@ priority: high
|
|||
state_hub_task_id: "ca41b874-5e09-538e-9692-d31d990d679b"
|
||||
```
|
||||
|
||||
The installation/rollback library and 15 disposable-filesystem cases pass; the full gate has 475 passes with zero skips. The final supervised wrapper must bind sources, bundle, preimages, locks, route/profile and inactive-unit guards. Then obtain the attended execution return and verify installed hashes and disabled units. Evidence: `docs/release/backup-source-installer-review-2026-10-31.md` and its validation JSON. Three target files exist; other 16 targets and all five units were absent in the observation. No installation has occurred.
|
||||
The installation/rollback library and final supervised wrapper are implemented.
|
||||
The wrapper binds sources, archive/manifest, native preimages, both exclusive
|
||||
locks, full owner route verification, capacity/profile, SDK/custody and inactive/
|
||||
disabled/no-queued-job guards. Explicit rollback retains preimages and refuses
|
||||
interference. Current native preflight reaches and refuses the capacity guard's
|
||||
historical custody exceptions; T04 now records that exact blocker. T07's route
|
||||
receipt is also still absent. No installation attempt or target change occurred.
|
||||
Then obtain the attended return and independently verify installed hashes and
|
||||
disabled units. Review: docs/release/backup-source-installation-execution-review-2026-10-31.md;
|
||||
source/gate evidence: backup-supervised-installer-validation-2026-10-31.json.
|
||||
|
||||
## T09 — Prove native paired service and recurring failure reception
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue