coulomb/rapp-playwright
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Prove abrupt recovery with live jobs in released and managed containers

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13590-a16f-7f60-be60-fd71627bc99d
This commit is contained in:
tegwick 2026-10-13 22:32:10 +02:00
parent 5b8e798419
commit 2cb2b654e0
25 changed files with 1954 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/operations.md
View file

@ -111,3 +111,33 @@ Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream
termination and cross-version rollback require their separate evidence. The
command's local validation is retained in
[evidence](evidence/2026-10-13-consumer-verifier.json).
## Abrupt interruption verification
`tools/verify_crash_recovery.py --image <published-immutable-reference> --out
.local/new-crash-check` creates its own bounded, read-only Docker container and
private data directory. It verifies a completed trial, captures a partial PNG in
a real active job, queues another job, sends SIGKILL to the container, and starts
it on the same data. It requires interrupted receipts, orphan cleanup, preserved
completed hashes and a fresh trial; it removes only its own container. Run it
as workstation UID 1000 to match the released image's data-directory ownership.
The separately approved managed check is `tools/verify_managed_recovery.py`.
It is an ADMINISTER operation through existing SSH, never a consumer permission.
Use `--url <operator-loopback-forward> --image <published-reference> --out
.local/new-managed-crash --approval-reference <specific-founder-approval>
--execute-crash`. Merely supplying that flag/reference does not grant approval.
The tool pins railiance01's node address, one Ready pilot pod and its release,
requires an idle service, creates its own jobs, and guards the supervisor SIGKILL
on the expected active/queued jobs and partial screenshot. It verifies container
exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files,
completed hashes and a fresh trial. No replicas or manifests are changed.
The port-forward may exit when the container restarts. Re-establish the dedicated
operator connection after the container is Ready; the verification waits up to
180 seconds for it. Do not retry the crash command solely because the tunnel
was lost: inspect the existing verification process and pod restart state first.
Keep every `run.json` and the local data directory private. Share only selected
receipts under `docs/evidence`. These tools do not prove real consumer identity,
revocation or cross-version rollback. [Executed local and managed evidence](evidence/2026-10-13-crash-recovery.md).