coulomb/rapp-playwright
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
rapp-playwright/docs/evidence/2026-10-13-crash-recovery.md
tegwick 2cb2b654e0 Prove abrupt recovery with live jobs in released and managed containers 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13590-a16f-7f60-be60-fd71627bc99d
2026-10-13 22:32:10 +02:00

3.1 KiB
Raw Permalink Blame History

Real abrupt-interruption recovery — 2026-10-13

Both the isolated published container and the managed railiance01 pilot passed an actual SIGKILL recovery test. No browser implementation or image changed. The user approved the remaining recovery work with “Ok, lets do that.” Managed execution was ADMINISTER @ realm:kubernetes/railiance01, activation APPROVED; existing SSH access was used as an operator path, not consumer identity proof.

Local released container

python3 tools/verify_crash_recovery.py --image forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441 --out .local/crash-recovery-01 passed. Docker kill produced exit137 without OOM; the same container/data restarted. Its temporary container was removed. Test handles/data remain private under ignored .local/.

Managed pilot

Ran tools/verify_managed_recovery.py with the same immutable image, operator forward http://127.0.0.1:18087, output .local/managed-crash-recovery-02, specific approval reference and --execute-crash. Before starting, the service reported zero active and queued jobs. Guarded termination targeted only the verified node src/server.mjs supervisor after our active run wrote a 25,740-byte PNG and our second run was queued. It was not a fabricated persisted receipt.

At 20:28:49Z, the container exited137; its replacement started at 20:28:51Z. Restart count changed 0 → 1, container identity changed, and pod UID a22030a4-c12d-4e97-be8d-98a8d80808c2, PVC UID da36929a-1d69-457a-ad99-7580ef86ab18 and release digest remained unchanged. The pilot returned Ready. The dedicated forward was re-established after restart and stopped after verification. No grants, manifests or shared controls changed.

Both tests proved:

  • Actual running and queued jobs recovered as interrupted/service_restarted.
  • Uncommitted PNG and other orphan files were removed; interrupted receipts expose no artifacts. The local check also verified HTTP404 for the former PNG.
  • Completed receipt and all four downloaded PNG hashes remained identical.
  • A fresh responsive/keyboard trial completed all 14 steps and four downloads.

An earlier managed attempt lost its workstation Kubernetes port-forward before the crash step. It made no service interruption: restart count remained zero, and its baseline trial completed. That failed attempt is retained separately. The successful run used an SSH-local forward to a node-local Kubernetes forward. No observer timeout was treated as evidence that a running process had stopped.

Receipts: local, managed, first attempt. Selected before/after, interrupted and fresh-run receipts accompany those files; run capability handles and storage token hashes are excluded.

make test passed all three unit cases and syntax checks. The two released-image recovery tests above passed against real Chromium. These establish same-image abrupt recovery, not cross-version rollback or real human/agent authentication.