Prove installed Railiance runtime recovery under worker failures

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e387-534d-70e3-ad53-4ea05676db8c

docs/evidence/2026-09-27-installed-runtime-recovery.json

@@ -0,0 +1,110 @@
+{
+  "ok": true,
+  "runtime_sha256": "b6e4e8a429393d68831c996a65c0489664205df969ac9882e581983ec2da4969",
+  "proof_script_sha256": "6abd0212255e217b55780163bb6ddfb025341026c29afb3ad9853b485dca1fbd",
+  "scope": "installed worker/Glas/bwrap; synthetic queue and deterministic rein",
+  "profile_ref": "harness.agent-dev-local@1.1.1",
+  "imports": {
+    "rein_aharness": "lib/python3.12/site-packages/rein_aharness/__init__.py",
+    "glas_harness": "lib/python3.12/site-packages/glas_harness/__init__.py",
+    "sandboxer": "lib/python3.12/site-packages/sandboxer/__init__.py",
+    "llm_connect": "lib/python3.12/site-packages/llm_connect/__init__.py"
+  },
+  "cases": [
+    {
+      "case": "lost-close",
+      "ok": true,
+      "dispatches": 1,
+      "close_attempts": 2,
+      "outbox": {
+        "pending": 0,
+        "delivered": 1,
+        "quarantined": 0
+      },
+      "sandbox_destroyed": true,
+      "workspace_removed": true,
+      "lock_reacquired": true,
+      "source_clean": true,
+      "source_commit_count": 2,
+      "metrics_records": 1,
+      "metrics_success": true
+    },
+    {
+      "case": "execution-failure",
+      "ok": true,
+      "dispatches": 1,
+      "close_attempts": 1,
+      "outbox": {
+        "pending": 0,
+        "delivered": 1,
+        "quarantined": 0
+      },
+      "sandbox_destroyed": true,
+      "workspace_removed": true,
+      "lock_reacquired": true,
+      "source_clean": true,
+      "source_commit_count": 1,
+      "metrics_records": 1,
+      "metrics_success": false
+    },
+    {
+      "case": "lease-loss",
+      "ok": true,
+      "dispatches": 1,
+      "close_attempts": 0,
+      "outbox": {
+        "pending": 0,
+        "delivered": 0,
+        "quarantined": 0
+      },
+      "sandbox_destroyed": true,
+      "workspace_removed": true,
+      "lock_reacquired": true,
+      "source_clean": true,
+      "source_commit_count": 1,
+      "metrics_records": 1,
+      "metrics_success": false
+    },
+    {
+      "case": "sigterm-before-import",
+      "ok": true,
+      "dispatches": 1,
+      "close_attempts": 1,
+      "outbox": {
+        "pending": 0,
+        "delivered": 1,
+        "quarantined": 0
+      },
+      "sandbox_destroyed": true,
+      "workspace_removed": true,
+      "lock_reacquired": true,
+      "source_clean": true,
+      "source_commit_count": 1,
+      "metrics_records": 1,
+      "metrics_success": false
+    }
+  ],
+  "packaged_definitions": true,
+  "killed_lock_owner": {
+    "ok": true,
+    "held_lock_refused": true,
+    "sigkill_lock_reacquired": true
+  },
+  "artifact_unchanged": true,
+  "killed_worker": {
+    "ok": true,
+    "worker_sigkill": true,
+    "owner_cleanup": "explicit",
+    "sandbox_destroyed": true,
+    "workspace_removed": true,
+    "lock_reacquired": true,
+    "source_clean": true,
+    "accepted_commits": 0
+  },
+  "paid_requests": 0,
+  "production_queue_calls": 0,
+  "limitations": [
+    "not a live Activity Core expiry proof",
+    "hard-crash sandbox recovery is explicit owner cleanup"
+  ]
+}

docs/ops-run-claim-loop.md

@@ -170,6 +170,43 @@ every permanent Activity Core close code (including `expired_lease`), and Glas
 cancellation. It does not replace the isolated expired-row API smoke or the
 natural-run and sandbox-cleanup observations required from the deployed host.
 
+### Installed-runtime recovery drill
+
+Run `scripts/prove-runtime-recovery.py` with the selected artifact's interpreter:
+
+```bash
+"${RUNTIME}/bin/python3" -I -B scripts/prove-runtime-recovery.py \
+  --runtime "${RUNTIME}" --sha256 "${RUNTIME_SHA256}" \
+  --profile-ref harness.agent-dev-local@1.1.1
+```
+
+The script verifies the complete artifact digest and that all four runtime
+packages load from that artifact. It uses private temporary repositories,
+sandbox stores and worker state. The queue and rein authoring are fixtures;
+the installed worker, Glas lifecycle, bwrap execution, repository acceptance,
+external metrics and close outbox are real. No provider or production queue
+client is invoked. The selected profile is made ready only in an in-memory
+fixture catalog; no installed profile or standing service is changed.
+
+The receipt covers response-lost close replay with one execution/commit,
+execution failure, periodic lease rejection, a real SIGTERM to the disposable
+worker, a killed lock holder, and SIGKILL of a disposable worker after sandbox
+creation. Cancellation deliberately returns a late sandbox commit to prove
+that it is not imported into the source. Every case checks lock reacquisition,
+source state, sandbox/workspace cleanup and applicable durable evidence.
+
+After abrupt worker death, cleanup is **explicit owner recovery**, not an
+automatic startup sweep. Recover only the sandbox id attributed to the failed
+run, using sand-boxer's `get`/`destroy` with the same owner state directory.
+Verify destruction and workspace removal before resuming. Do not delete lock
+files to release a kernel lock, erase pending close evidence, or replay the
+workload to reconcile a close. Use `close-outbox status|replay` for that last
+step. Activity Core remains responsible for expiring the abandoned queue lease.
+
+Railiance receipt: `docs/evidence/2026-09-27-installed-runtime-recovery.json`.
+It proves these host mechanisms against runtime `b6e4e8a4`; it does not prove
+live Activity Core expiry, credential delivery, or a paid production run.
+
 ### Host access to cluster services (no port-forward)
 
 On railiance01 (single-node k3s), set **k8s://** pseudo-URLs in