diff --git a/Makefile b/Makefile index f4fa059..65baeef 100644 --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ install: npm ci --ignore-scripts check: npm run check - python3 -m py_compile client/rapp_playwright.py tools/render.py tools/verify_consumer.py + python3 -m py_compile client/rapp_playwright.py tools/render.py tools/verify_consumer.py tools/verify_crash_recovery.py tools/verify_managed_recovery.py test: check npm test test-integration: diff --git a/SCOPE.md b/SCOPE.md index ade054f..2391f33 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -48,7 +48,7 @@ artifacts expire; sanitized evidence under `docs/evidence/` is retained. | --- | --- | | WP-0002 | Finished: consumption contract, lifecycle and trial/pilot design | | WP-0003 | Finished: tested image published at its exact digest and pulled on railiance01; ingress upload deadline fixed. | -| WP-0004 | T02 provider trial/pilot done. T01 awaits actual consumer identity/access and remaining recovery evidence. T03 has published A5 local-provider discovery; final managed availability update awaits T01. | +| WP-0004 | T02 provider trial/pilot done. T01 awaits actual consumer identity/access and cross-version rollback evidence. T03 has published A5 local-provider discovery; final managed availability update awaits T01. | Platform admitted and deployed the reviewed package under RPF-WP-0056. The `playwright-consumers` group is bound to the namespace consumer Role. Initial @@ -72,7 +72,10 @@ Platform's [consumer access instructions](https://forgejo.coulomb.social/coulomb are prepared but conditional on identity-owner activation. The earlier [admission review](docs/platform-admission.md) is retained as history; package admission, deployment and the binding are no longer the outstanding decision. -Abrupt-interruption recovery and actual cross-version rollback remain unverified. +Actual abrupt-interruption recovery is now verified locally and on the managed +pilot: active/queued jobs become interrupted, partial artifacts disappear, completed +hashes survive, and a fresh trial passes. See [crash evidence](docs/evidence/2026-10-13-crash-recovery.md). +Actual cross-version rollback remains unverified; there is only one released version. The repeatable [consumer API acceptance command](docs/operations.md#repeatable-consumer-api-acceptance) now passes six local API checks and refuses release/location mismatches. It is @@ -88,5 +91,5 @@ sandbox is disabled; hostile pages and mutually untrusted tenants are unsupporte Target failures identify the category but not detailed DNS/TLS/HTTP causes. The OnCo pilot covers the home page, not all consumer routes or accessibility. -See the [timestamped intent assessment](history/2026-10-13-2211-intent-scope-assessment.md) +See the [timestamped intent assessment](history/2026-10-13-2231-intent-scope-assessment.md) for the prioritized gaps and their acceptance evidence. diff --git a/docs/evidence/2026-10-13-crash-recovery.md b/docs/evidence/2026-10-13-crash-recovery.md new file mode 100644 index 0000000..bc00780 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery.md @@ -0,0 +1,55 @@ +# Real abrupt-interruption recovery — 2026-10-13 + +Both the isolated published container and the managed railiance01 pilot passed +an actual SIGKILL recovery test. No browser implementation or image changed. +The user approved the remaining recovery work with “Ok, lets do that.” Managed +execution was ADMINISTER @ realm:kubernetes/railiance01, activation APPROVED; +existing SSH access was used as an operator path, not consumer identity proof. + +## Local released container + +`python3 tools/verify_crash_recovery.py --image +forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441 +--out .local/crash-recovery-01` passed. Docker kill produced exit137 without OOM; +the same container/data restarted. Its temporary container was removed. Test +handles/data remain private under ignored `.local/`. + +## Managed pilot + +Ran `tools/verify_managed_recovery.py` with the same immutable image, operator +forward `http://127.0.0.1:18087`, output `.local/managed-crash-recovery-02`, specific +approval reference and `--execute-crash`. Before starting, the service reported +zero active and queued jobs. Guarded termination targeted only the verified +`node src/server.mjs` supervisor after our active run wrote a 25,740-byte PNG and +our second run was queued. It was not a fabricated persisted receipt. + +At 20:28:49Z, the container exited137; its replacement started at 20:28:51Z. +Restart count changed 0 → 1, container identity changed, and pod UID +`a22030a4-c12d-4e97-be8d-98a8d80808c2`, PVC UID +`da36929a-1d69-457a-ad99-7580ef86ab18` and release digest remained unchanged. +The pilot returned Ready. The dedicated forward was re-established after restart +and stopped after verification. No grants, manifests or shared controls changed. + +Both tests proved: + +- Actual running and queued jobs recovered as `interrupted/service_restarted`. +- Uncommitted PNG and other orphan files were removed; interrupted receipts expose + no artifacts. The local check also verified HTTP404 for the former PNG. +- Completed receipt and all four downloaded PNG hashes remained identical. +- A fresh responsive/keyboard trial completed all 14 steps and four downloads. + +An earlier managed attempt lost its workstation Kubernetes port-forward before +the crash step. It made no service interruption: restart count remained zero, +and its baseline trial completed. That failed attempt is retained separately. +The successful run used an SSH-local forward to a node-local Kubernetes forward. +No observer timeout was treated as evidence that a running process had stopped. + +Receipts: [local](2026-10-13-crash-recovery/local/verification.json), +[managed](2026-10-13-crash-recovery/managed/verification.json), +[first attempt](2026-10-13-crash-recovery/managed-first-attempt.json). +Selected before/after, interrupted and fresh-run receipts accompany those files; +run capability handles and storage token hashes are excluded. + +`make test` passed all three unit cases and syntax checks. The two released-image +recovery tests above passed against real Chromium. These establish same-image +abrupt recovery, not cross-version rollback or real human/agent authentication. diff --git a/docs/evidence/2026-10-13-crash-recovery/local/active-receipt.json b/docs/evidence/2026-10-13-crash-recovery/local/active-receipt.json new file mode 100644 index 0000000..f453a29 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/active-receipt.json @@ -0,0 +1,29 @@ +{ + "id": "2665c7cd-c707-4a7d-91eb-c51c91462410", + "api_version": "1", + "status": "interrupted", + "error": "service_restarted", + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:23:29.324Z", + "expires_at": "2026-10-13T21:23:29.324Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "isolated-crash-verification", + "browser_version": null, + "completed_steps": 0, + "steps": [], + "artifacts": [], + "started_at": "2026-10-13T20:23:29.334Z", + "finished_at": "2026-10-13T20:23:46.954Z", + "downloads": [] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/after-preflight.json b/docs/evidence/2026-10-13-crash-recovery/local/after-preflight.json new file mode 100644 index 0000000..5c942f7 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/after-preflight.json @@ -0,0 +1,22 @@ +{ + "available": true, + "error": null, + "checked_at": "2026-10-13T20:23:54.299Z", + "browser_version": "156.0.8078.4", + "api_version": "1", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "execution_location": "isolated-crash-verification", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "allowed_origins": [], + "fixture": true, + "active": 0, + "queued": 0, + "limits": { + "concurrency": 1, + "max_queue": 4, + "max_timeout_ms": 120000, + "retention_ms": 3600000, + "max_runs": 64 + } +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/before-preflight.json b/docs/evidence/2026-10-13-crash-recovery/local/before-preflight.json new file mode 100644 index 0000000..997cd95 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/before-preflight.json @@ -0,0 +1,22 @@ +{ + "available": true, + "error": null, + "checked_at": "2026-10-13T20:23:15.520Z", + "browser_version": "156.0.8078.4", + "api_version": "1", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "execution_location": "isolated-crash-verification", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "allowed_origins": [], + "fixture": true, + "active": 0, + "queued": 0, + "limits": { + "concurrency": 1, + "max_queue": 4, + "max_timeout_ms": 120000, + "retention_ms": 3600000, + "max_runs": 64 + } +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/completed-after-receipt.json b/docs/evidence/2026-10-13-crash-recovery/local/completed-after-receipt.json new file mode 100644 index 0000000..396a07e --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/completed-after-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:23:15.654Z", + "expires_at": "2026-10-13T21:23:15.654Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "isolated-crash-verification", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:23:21.899Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:22.384Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:22.587Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:23:22.602Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:22.690Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:22.708Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:23:22.794Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:23:26.394Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:26.919Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:27.208Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:23:27.295Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:27.391Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:27.483Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:23:27.510Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:23:15.663Z", + "blocked_requests": 13, + "finished_at": "2026-10-13T20:23:29.212Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/completed-before-receipt.json b/docs/evidence/2026-10-13-crash-recovery/local/completed-before-receipt.json new file mode 100644 index 0000000..396a07e --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/completed-before-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:23:15.654Z", + "expires_at": "2026-10-13T21:23:15.654Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "isolated-crash-verification", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:23:21.899Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:22.384Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:22.587Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:23:22.602Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:22.690Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:22.708Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:23:22.794Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:23:26.394Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:26.919Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:27.208Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:23:27.295Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:23:27.391Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:23:27.483Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:23:27.510Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:23:15.663Z", + "blocked_requests": 13, + "finished_at": "2026-10-13T20:23:29.212Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/new-trial-receipt.json b/docs/evidence/2026-10-13-crash-recovery/local/new-trial-receipt.json new file mode 100644 index 0000000..584c8d0 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/new-trial-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "263a5639-e17a-4828-b33d-9434cb0a9c6b", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:23:54.497Z", + "expires_at": "2026-10-13T21:23:54.497Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "isolated-crash-verification", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:24:03.160Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:24:03.858Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:24:04.139Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:24:04.162Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:24:04.250Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:24:04.264Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:24:04.347Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:24:07.458Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:24:07.760Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:24:07.954Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:24:08.042Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:24:08.059Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:24:08.073Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:24:08.165Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:23:54.510Z", + "blocked_requests": 13, + "finished_at": "2026-10-13T20:24:09.482Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/queued-receipt.json b/docs/evidence/2026-10-13-crash-recovery/local/queued-receipt.json new file mode 100644 index 0000000..ee3a454 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/queued-receipt.json @@ -0,0 +1,33 @@ +{ + "id": "6a0b47e4-2b47-4d77-9d4b-2f1615d849ed", + "api_version": "1", + "status": "interrupted", + "error": "service_restarted", + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:23:43.568Z", + "expires_at": "2026-10-13T21:23:43.568Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "isolated-crash-verification", + "browser_version": null, + "completed_steps": 0, + "steps": [], + "artifacts": [], + "finished_at": "2026-10-13T20:23:46.987Z", + "downloads": [] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/local/verification.json b/docs/evidence/2026-10-13-crash-recovery/local/verification.json new file mode 100644 index 0000000..b1d59ac --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/local/verification.json @@ -0,0 +1,45 @@ +{ + "started_at": "2026-10-13T20:23:09.595214+00:00", + "image": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "location": "isolated-local-docker", + "signal": "SIGKILL", + "checks": [ + "completed_trial_and_four_downloads", + "real_active_and_queued_jobs_killed_without_shutdown", + "active_and_queued_interrupted_orphans_removed", + "completed_receipt_and_hashes_preserved", + "fresh_keyboard_screenshot_trial_after_recovery" + ], + "status": "passed", + "managed_recovery_proven": false, + "persisted_before_kill": { + "active": "running", + "queued": "queued", + "partial_png_bytes": 25740 + }, + "exit_code_after_kill": 137, + "completed_downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ], + "test_container_removed": true, + "finished_at": "2026-10-13T20:24:11.347753+00:00" +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed-first-attempt.json b/docs/evidence/2026-10-13-crash-recovery/managed-first-attempt.json new file mode 100644 index 0000000..f68146f --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed-first-attempt.json @@ -0,0 +1,20 @@ +{ + "started_at": "2026-10-13T20:26:47.877053+00:00", + "status": "failed", + "authority": "ADMINISTER @ realm:kubernetes/railiance01", + "activation": "APPROVED", + "approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)", + "identity_acceptance": false, + "cross_version_rollback": false, + "checks": [], + "before": { + "pod": "playwright-c5b9c8bd7-fbfqv", + "pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2", + "restart_count": 0, + "container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce", + "image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18" + }, + "error": "service_unreachable: establish the documented tunnel and retry", + "finished_at": "2026-10-13T20:27:39.311029+00:00" +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/active-receipt.json b/docs/evidence/2026-10-13-crash-recovery/managed/active-receipt.json new file mode 100644 index 0000000..55ac6cd --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/active-receipt.json @@ -0,0 +1,29 @@ +{ + "id": "6440ad74-27d1-4daf-a768-768e041f6741", + "api_version": "1", + "status": "interrupted", + "error": "service_restarted", + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:28:46.325Z", + "expires_at": "2026-10-13T21:28:46.325Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "railiance01/playwright", + "browser_version": null, + "completed_steps": 0, + "steps": [], + "artifacts": [], + "started_at": "2026-10-13T20:28:46.328Z", + "finished_at": "2026-10-13T20:28:51.153Z", + "downloads": [] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/after-preflight.json b/docs/evidence/2026-10-13-crash-recovery/managed/after-preflight.json new file mode 100644 index 0000000..56df598 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/after-preflight.json @@ -0,0 +1,24 @@ +{ + "available": true, + "error": null, + "checked_at": "2026-10-13T20:28:52.784Z", + "browser_version": "156.0.8078.4", + "api_version": "1", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "execution_location": "railiance01/playwright", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "allowed_origins": [ + "https://onco.cc" + ], + "fixture": true, + "active": 0, + "queued": 0, + "limits": { + "concurrency": 1, + "max_queue": 4, + "max_timeout_ms": 120000, + "retention_ms": 3600000, + "max_runs": 64 + } +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/completed-after-receipt.json b/docs/evidence/2026-10-13-crash-recovery/managed/completed-after-receipt.json new file mode 100644 index 0000000..22ff444 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/completed-after-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:28:41.609Z", + "expires_at": "2026-10-13T21:28:41.609Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "railiance01/playwright", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:28:43.700Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:43.839Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:43.907Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:28:43.920Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:43.944Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:43.952Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:28:43.961Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:28:44.858Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:45.052Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:45.105Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:28:45.119Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:45.137Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:45.147Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:28:45.183Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:28:41.611Z", + "blocked_requests": 12, + "finished_at": "2026-10-13T20:28:45.914Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/completed-before-receipt.json b/docs/evidence/2026-10-13-crash-recovery/managed/completed-before-receipt.json new file mode 100644 index 0000000..22ff444 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/completed-before-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:28:41.609Z", + "expires_at": "2026-10-13T21:28:41.609Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "railiance01/playwright", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:28:43.700Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:43.839Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:43.907Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:28:43.920Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:43.944Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:43.952Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:28:43.961Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:28:44.858Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:45.052Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:45.105Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:28:45.119Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:28:45.137Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:28:45.147Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:28:45.183Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:28:41.611Z", + "blocked_requests": 12, + "finished_at": "2026-10-13T20:28:45.914Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/new-trial-receipt.json b/docs/evidence/2026-10-13-crash-recovery/managed/new-trial-receipt.json new file mode 100644 index 0000000..6fc2636 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/new-trial-receipt.json @@ -0,0 +1,184 @@ +{ + "id": "f76f4662-03cd-4596-928e-ee34eeaa4057", + "api_version": "1", + "status": "passed", + "error": null, + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:29:17.318Z", + "expires_at": "2026-10-13T21:29:17.318Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "railiance01/playwright", + "browser_version": "156.0.8078.4", + "completed_steps": 14, + "steps": [ + { + "viewport": "desktop", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:29:19.494Z", + "selector": "h1" + }, + { + "viewport": "desktop", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:29:19.657Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:29:19.745Z", + "key": "Tab" + }, + { + "viewport": "desktop", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:29:19.760Z", + "selector": "#increment" + }, + { + "viewport": "desktop", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:29:19.797Z", + "key": "Enter" + }, + { + "viewport": "desktop", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:29:19.802Z", + "selector": "#count" + }, + { + "viewport": "desktop", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:29:19.809Z" + }, + { + "viewport": "narrow", + "index": 0, + "action": "visible", + "status": "passed", + "at": "2026-10-13T20:29:20.740Z", + "selector": "h1" + }, + { + "viewport": "narrow", + "index": 1, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:29:20.862Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 2, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:29:20.930Z", + "key": "Tab" + }, + { + "viewport": "narrow", + "index": 3, + "action": "focused", + "status": "passed", + "at": "2026-10-13T20:29:20.958Z", + "selector": "#increment" + }, + { + "viewport": "narrow", + "index": 4, + "action": "press", + "status": "passed", + "at": "2026-10-13T20:29:20.974Z", + "key": "Enter" + }, + { + "viewport": "narrow", + "index": 5, + "action": "text", + "status": "passed", + "at": "2026-10-13T20:29:20.981Z", + "selector": "#count" + }, + { + "viewport": "narrow", + "index": 6, + "action": "screenshot", + "status": "passed", + "at": "2026-10-13T20:29:20.989Z" + } + ], + "artifacts": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622 + }, + { + "name": "desktop-final.png", + "bytes": 26622 + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280 + }, + { + "name": "narrow-final.png", + "bytes": 24280 + } + ], + "started_at": "2026-10-13T20:29:17.319Z", + "blocked_requests": 12, + "finished_at": "2026-10-13T20:29:21.401Z", + "downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/preflight.json b/docs/evidence/2026-10-13-crash-recovery/managed/preflight.json new file mode 100644 index 0000000..18861dc --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/preflight.json @@ -0,0 +1,24 @@ +{ + "available": true, + "error": null, + "checked_at": "2026-10-13T20:28:18.565Z", + "browser_version": "156.0.8078.4", + "api_version": "1", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "execution_location": "railiance01/playwright", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "allowed_origins": [ + "https://onco.cc" + ], + "fixture": true, + "active": 0, + "queued": 0, + "limits": { + "concurrency": 1, + "max_queue": 4, + "max_timeout_ms": 120000, + "retention_ms": 3600000, + "max_runs": 64 + } +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/queued-receipt.json b/docs/evidence/2026-10-13-crash-recovery/managed/queued-receipt.json new file mode 100644 index 0000000..ea49533 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/queued-receipt.json @@ -0,0 +1,33 @@ +{ + "id": "74e433ef-aee7-4c74-afe5-3171214f4014", + "api_version": "1", + "status": "interrupted", + "error": "service_restarted", + "label": "independent-browser-trial", + "target": "fixture", + "viewports": [ + { + "name": "desktop", + "width": 1280, + "height": 800 + }, + { + "name": "narrow", + "width": 390, + "height": 844 + } + ], + "timeout_ms": 60000, + "created_at": "2026-10-13T20:28:46.380Z", + "expires_at": "2026-10-13T21:28:46.380Z", + "service_version": "0.1.0", + "playwright_version": "1.64.0", + "release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "execution_location": "railiance01/playwright", + "browser_version": null, + "completed_steps": 0, + "steps": [], + "artifacts": [], + "finished_at": "2026-10-13T20:28:51.157Z", + "downloads": [] +} diff --git a/docs/evidence/2026-10-13-crash-recovery/managed/verification.json b/docs/evidence/2026-10-13-crash-recovery/managed/verification.json new file mode 100644 index 0000000..040f474 --- /dev/null +++ b/docs/evidence/2026-10-13-crash-recovery/managed/verification.json @@ -0,0 +1,64 @@ +{ + "started_at": "2026-10-13T20:28:36.758090+00:00", + "status": "passed", + "authority": "ADMINISTER @ realm:kubernetes/railiance01", + "activation": "APPROVED", + "approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)", + "identity_acceptance": false, + "cross_version_rollback": false, + "checks": [ + "baseline_trial", + "guarded_sigkill_of_active_service", + "interrupted_jobs_and_orphan_cleanup", + "completed_artifact_hashes_preserved", + "fresh_trial_after_container_restart" + ], + "before": { + "pod": "playwright-c5b9c8bd7-fbfqv", + "pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2", + "restart_count": 0, + "container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce", + "image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18", + "last_exit_code": null + }, + "crash": { + "signal": "SIGKILL", + "pid": 7, + "active_status": "running", + "queued_status": "queued", + "partial_png_bytes": 25740 + }, + "after": { + "pod": "playwright-c5b9c8bd7-fbfqv", + "pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2", + "restart_count": 1, + "container_id": "containerd://abd1f7081045b1b4505e8045bf298e0a10485834d760d6dbf8db73accdb3da08", + "image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441", + "pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18", + "last_exit_code": 137 + }, + "completed_downloads": [ + { + "name": "desktop-6-keyboard-result.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "desktop-final.png", + "bytes": 26622, + "sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad" + }, + { + "name": "narrow-6-keyboard-result.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + }, + { + "name": "narrow-final.png", + "bytes": 24280, + "sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8" + } + ], + "finished_at": "2026-10-13T20:29:20.202087+00:00" +} diff --git a/docs/operations.md b/docs/operations.md index 1e58820..15c3908 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -111,3 +111,33 @@ Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream termination and cross-version rollback require their separate evidence. The command's local validation is retained in [evidence](evidence/2026-10-13-consumer-verifier.json). + + +## Abrupt interruption verification + +`tools/verify_crash_recovery.py --image --out +.local/new-crash-check` creates its own bounded, read-only Docker container and +private data directory. It verifies a completed trial, captures a partial PNG in +a real active job, queues another job, sends SIGKILL to the container, and starts +it on the same data. It requires interrupted receipts, orphan cleanup, preserved +completed hashes and a fresh trial; it removes only its own container. Run it +as workstation UID 1000 to match the released image's data-directory ownership. + +The separately approved managed check is `tools/verify_managed_recovery.py`. +It is an ADMINISTER operation through existing SSH, never a consumer permission. +Use `--url --image --out +.local/new-managed-crash --approval-reference +--execute-crash`. Merely supplying that flag/reference does not grant approval. +The tool pins railiance01's node address, one Ready pilot pod and its release, +requires an idle service, creates its own jobs, and guards the supervisor SIGKILL +on the expected active/queued jobs and partial screenshot. It verifies container +exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files, +completed hashes and a fresh trial. No replicas or manifests are changed. + +The port-forward may exit when the container restarts. Re-establish the dedicated +operator connection after the container is Ready; the verification waits up to +180 seconds for it. Do not retry the crash command solely because the tunnel +was lost: inspect the existing verification process and pod restart state first. +Keep every `run.json` and the local data directory private. Share only selected +receipts under `docs/evidence`. These tools do not prove real consumer identity, +revocation or cross-version rollback. [Executed local and managed evidence](evidence/2026-10-13-crash-recovery.md). diff --git a/history/2026-10-13-2231-intent-scope-assessment.md b/history/2026-10-13-2231-intent-scope-assessment.md new file mode 100644 index 0000000..8813341 --- /dev/null +++ b/history/2026-10-13-2231-intent-scope-assessment.md @@ -0,0 +1,37 @@ +# INTENT against SCOPE — 2026-10-13 22:31 Europe/Berlin + +The abrupt-interruption gap is now closed with real local and managed execution. +Published release 0.1.0 recovered after SIGKILL during an actual browser job; +a queued job was also preserved as interrupted. Partial artifacts were removed, +completed hashes survived, and a fresh 14-step screenshot/keyboard trial passed. +Managed recovery used the same pod, PVC and image with a new container. See +[execution evidence](../docs/evidence/2026-10-13-crash-recovery.md). + +Most important remaining gaps: + +1. **Real self-service human and coding-agent access.** Identity registration, + governed agent credentials, native API authentication and per-caller transport + still need owner activation and native acceptance receipts. The completed + recovery check used an existing operator identity. It does not satisfy this + gap. RPF-WP-0056-T03 owns delivery; CUST-WP-0080-T05 coordinates it. Membership + is decided, so no additional eligibility question is needed. +2. **Cross-version rollback.** This is the first release. A same-image restart + proves crash recovery, not a version transition. Exercise actual rollback + when a second reviewed compatible release exists; retain the open requirement. +3. **Final managed availability and discovery.** A5 local-container guidance is + published. Real consumer receipts must precede final readiness and managed + discovery promotion under WP-0004-T01/T03. +4. **Consumer-specific acceptance.** Content-explorer owns its broader route, + keyboard and visual requirements. Existing provider evidence cannot close + that consumer task. + +WP-0001–0003 remain finished. WP-0004-T02 is done; T01/T03 retain their narrower +external waits. SCOPE now includes the managed abrupt-recovery proof. The repo +contains repeatable consumer API and crash-recovery commands and sanitized +receipts. The managed pilot is Ready; temporary verification connections were +stopped. No runtime code, image version, grants or shared controls changed. + +Validation: three unit tests and syntax checks passed; both real released-image +crash checks passed. An initial managed transport failure stopped before the +crash step and is retained explicitly. The overall implementation is not complete +until the outstanding consumer-access and acceptance requirements are satisfied. diff --git a/tools/verify_crash_recovery.py b/tools/verify_crash_recovery.py new file mode 100644 index 0000000..7af24d6 --- /dev/null +++ b/tools/verify_crash_recovery.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Kill an isolated released Docker container during a real browser job and verify recovery.""" +import argparse +from datetime import datetime, timezone +import json +from pathlib import Path +import re +import subprocess +import sys +import time +import uuid + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'client')) +from rapp_playwright import Client, ClientError, collect, save + + +def require(ok, message): + if not ok: + raise ClientError(message) + + +def docker(*args): + result = subprocess.run(['docker', *args], capture_output=True, text=True, timeout=90) + if result.returncode: + raise ClientError('docker_' + args[0] + '_failed (raw output suppressed)') + return result.stdout.strip() + + +def poll(check, description, timeout=60): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = check() + if value: + return value + time.sleep(0.1) + raise ClientError('timeout: ' + description) + + +def verify(args): + require(bool(re.fullmatch(r'forgejo\.coulomb\.social/coulomb/rapp-playwright@sha256:[a-f0-9]{64}', args.image)), + 'published immutable image reference required') + root = args.out.resolve() + root.mkdir(parents=True, mode=0o700, exist_ok=False) + data = root / 'data' + data.mkdir(mode=0o700) + name = 'rapp-crash-check-' + uuid.uuid4().hex[:12] + cid = None + report = {'started_at': datetime.now(timezone.utc).isoformat(), 'image': args.image, + 'location': 'isolated-local-docker', 'signal': 'SIGKILL', 'checks': [], + 'status': 'failed', 'managed_recovery_proven': False} + try: + cid = docker('run', '-d', '--name', name, '--read-only', '--user', '1000:1000', + '--tmpfs', '/tmp:rw,size=512m', '--shm-size=256m', '--memory=1g', '--cpus=1', + '--cap-drop', 'ALL', '--security-opt', 'no-new-privileges', + '--mount', 'type=bind,src=' + str(data) + ',dst=/data', + '-p', '127.0.0.1::8080', '-e', 'HOST=0.0.0.0', + '-e', 'RELEASE_ID=' + args.image, + '-e', 'EXECUTION_LOCATION=isolated-crash-verification', args.image) + require(bool(re.fullmatch('[a-f0-9]{64}', cid)), 'unexpected container identity') + + def connect(): + binding = json.loads(docker('inspect', '--format', '{{json .NetworkSettings.Ports}}', cid))['8080/tcp'][0] + require(binding['HostIp'] == '127.0.0.1', 'non_loopback_binding') + return Client('http://127.0.0.1:' + binding['HostPort']) + + def ready(client): + def probe(): + try: + result = client.request('/v1/preflight') + return result if result.get('available') else None + except ClientError: + return None + result = poll(probe, 'browser readiness') + require(result['release'] == args.image, 'unexpected release') + return result + + client = connect() + save(root / 'before-preflight.json', ready(client)) + spec = client.request('/v1/trial') + + def submit(specification, folder): + folder.mkdir(mode=0o700) + run = client.request('/v1/runs', 'POST', specification) + save(folder / 'run.json', run) + return run + + def terminal(run): + def check(): + result = client.request('/v1/runs/' + run['id'], token=run['token']) + return result if result['status'] in ('passed', 'failed', 'cancelled', 'interrupted') else None + return poll(check, 'terminal run') + + completed = submit(spec, root / 'completed-before') + require(terminal(completed)['status'] == 'passed', 'baseline trial failed') + before = collect(client, completed, root / 'completed-before') + require(len(before['downloads']) == 4, 'baseline screenshot count') + report['checks'].append('completed_trial_and_four_downloads') + slow = {**spec, 'viewports': [{'name': 'desktop', 'width': 1280, 'height': 800}], + 'steps': [{'action': 'screenshot', 'name': 'uncommitted'}, + {'action': 'visible', 'selector': '#never-present-crash-test'}]} + active = submit(slow, root / 'active') + partial = data / active['id'] / 'desktop-0-uncommitted.png' + + def screenshot_in_flight(): + result = client.request('/v1/runs/' + active['id'], token=active['token']) + require(result['status'] not in ('passed', 'failed', 'cancelled', 'interrupted'), 'active job ended before kill') + return result['status'] == 'running' and partial.is_file() and partial.stat().st_size > 0 + + poll(screenshot_in_flight, 'in-flight browser screenshot', timeout=30) + queued = submit(spec, root / 'queued') + require(client.request('/v1/runs/' + queued['id'], token=queued['token'])['status'] == 'queued', 'expected queued job') + # Read actual persisted state without exposing its capability hashes. + report['persisted_before_kill'] = { + 'active': json.loads((data / active['id'] / 'receipt.json').read_text())['status'], + 'queued': json.loads((data / queued['id'] / 'receipt.json').read_text())['status'], + 'partial_png_bytes': partial.stat().st_size} + require(report['persisted_before_kill']['active'] == 'running', 'active persistence missing') + require(report['persisted_before_kill']['queued'] == 'queued', 'queued persistence missing') + docker('kill', '--signal=KILL', cid) + state = json.loads(docker('inspect', '--format', '{{json .State}}', cid)) + require(not state['Running'] and state['ExitCode'] == 137 and not state['OOMKilled'], 'SIGKILL not established') + report['exit_code_after_kill'] = state['ExitCode'] + report['checks'].append('real_active_and_queued_jobs_killed_without_shutdown') + docker('start', cid) + client = connect() + save(root / 'after-preflight.json', ready(client)) + for label, run in [('active', active), ('queued', queued)]: + result = collect(client, run, root / label) + require(result['status'] == 'interrupted' and result['error'] == 'service_restarted', label + '_not_interrupted') + require(result['artifacts'] == [] and result['downloads'] == [], label + '_artifacts_exposed') + require(sorted(p.name for p in (data / run['id']).iterdir()) == ['receipt.json'], label + '_orphan_files') + report['checks'].append('active_and_queued_interrupted_orphans_removed') + try: + client.request('/v1/runs/' + active['id'] + '/artifacts/desktop-0-uncommitted.png', token=active['token'], binary=True) + except ClientError as exc: + require(str(exc) == 'HTTP 404: not_found', 'unexpected orphan denial') + else: + raise ClientError('uncommitted artifact still downloadable') + after_dir = root / 'completed-after' + after_dir.mkdir(mode=0o700) + after = collect(client, completed, after_dir) + require(after['status'] == 'passed' and before['downloads'] == after['downloads'], 'completed artifact mismatch') + report['completed_downloads'] = after['downloads'] + report['checks'].append('completed_receipt_and_hashes_preserved') + fresh = submit(spec, root / 'new-trial') + require(terminal(fresh)['status'] == 'passed', 'fresh trial failed after restart') + fresh_receipt = collect(client, fresh, root / 'new-trial') + require(fresh_receipt['completed_steps'] == 14 and len(fresh_receipt['downloads']) == 4, 'fresh trial incomplete') + report['checks'].append('fresh_keyboard_screenshot_trial_after_recovery') + report['status'] = 'passed' + except KeyboardInterrupt: + report['error'] = 'interrupted_by_caller' + except (ClientError, OSError, ValueError, KeyError, subprocess.TimeoutExpired) as exc: + report['error'] = str(exc) + finally: + if cid: + try: + docker('rm', '-f', cid) + report['test_container_removed'] = True + except ClientError: + report['test_container_removed'] = False + report['cleanup_container_name'] = name + report['status'] = 'failed' + report['finished_at'] = datetime.now(timezone.utc).isoformat() + save(root / 'verification.json', report) + print(json.dumps(report, indent=2)) + return 0 if report['status'] == 'passed' else 3 + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--image', required=True) + parser.add_argument('--out', required=True, type=Path) + try: + sys.exit(verify(parser.parse_args())) + except (ClientError, OSError, ValueError) as exc: + print(json.dumps({'error': str(exc)}), file=sys.stderr) + sys.exit(3) diff --git a/tools/verify_managed_recovery.py b/tools/verify_managed_recovery.py new file mode 100644 index 0000000..94b52e8 --- /dev/null +++ b/tools/verify_managed_recovery.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Approved operator-only crash check for the fixed railiance01 Playwright pilot. + +Requires --execute-crash; no identity acceptance or cross-version rollback is implied. +""" +import argparse +from datetime import datetime, timezone +import json +from pathlib import Path +import shlex +import subprocess +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'client')) +from rapp_playwright import Client, ClientError, collect, save +from verify_crash_recovery import poll, require + + +def remote(*args): + r = subprocess.run(['ssh', 'railiance01', shlex.join(list(args))], capture_output=True, text=True, timeout=30) + require(r.returncode == 0, 'remote check failed; raw output suppressed') + return json.loads(r.stdout) + + +def snapshot(expected): + nodes = remote('kubectl', 'get', 'nodes', '-o', 'json')['items'] + require(any(a['address'] == '92.205.62.239' for n in nodes for a in n['status']['addresses']), 'wrong node') + pods = remote('kubectl', '-n', 'playwright', 'get', 'pods', '-l', 'app.kubernetes.io/name=playwright', '-o', 'json')['items'] + require(len(pods) == 1, 'expected exactly one pilot pod') + p = pods[0] + c = next(c for c in p['status']['containerStatuses'] if c['name'] == 'service') + require(c['ready'] and c['imageID'] == expected, 'pilot not ready on expected image') + pvc = remote('kubectl', '-n', 'playwright', 'get', 'pvc', 'playwright-artifacts', '-o', 'json') + require(pvc['status']['phase'] == 'Bound', 'PVC not bound') + return {'pod': p['metadata']['name'], 'pod_uid': p['metadata']['uid'], 'restart_count': c['restartCount'], + 'container_id': c['containerID'], 'image_id': c['imageID'], 'pvc_uid': pvc['metadata']['uid'], + 'last_exit_code': c.get('lastState', {}).get('terminated', {}).get('exitCode')} + + +def verify(args): + require(args.execute_crash, 'explicit --execute-crash is required after owner/founder approval') + args.out.mkdir(mode=0o700, parents=True, exist_ok=False) + report = {'started_at': datetime.now(timezone.utc).isoformat(), 'status': 'failed', + 'authority': 'ADMINISTER @ realm:kubernetes/railiance01', 'activation': 'APPROVED', + 'approval_basis': args.approval_reference, 'identity_acceptance': False, + 'cross_version_rollback': False, 'checks': []} + client = Client(args.url) + runs = [] + try: + before = snapshot(args.image) + report['before'] = before + ready = client.request('/v1/preflight') + require(ready['available'] and ready['release'] == args.image and ready['execution_location'] == 'railiance01/playwright', 'provider mismatch') + require(ready['active'] == 0 and ready['queued'] == 0, 'pilot busy: do not interrupt unrelated jobs') + save(args.out / 'preflight.json', ready) + spec = client.request('/v1/trial') + + def submit(specification, label): + folder = args.out / label + folder.mkdir(mode=0o700) + run = client.request('/v1/runs', 'POST', specification) + runs.append(run) + save(folder / 'run.json', run) + return run, folder + + def finish(run): + def check(): + r = client.request('/v1/runs/' + run['id'], token=run['token']) + return r if r['status'] in ('passed', 'failed', 'cancelled', 'interrupted') else None + return poll(check, 'job completion') + + completed, folder = submit(spec, 'completed-before') + require(finish(completed)['status'] == 'passed', 'baseline trial failed') + original = collect(client, completed, folder) + require(len(original['downloads']) == 4, 'four baseline screenshots required') + report['checks'].append('baseline_trial') + slow = {**spec, 'viewports': [{'name': 'desktop', 'width': 1280, 'height': 800}], + 'steps': [{'action': 'screenshot', 'name': 'uncommitted'}, + {'action': 'visible', 'selector': '#never-present-crash-test'}]} + active, active_dir = submit(slow, 'active') + queued, queued_dir = submit(spec, 'queued') + # Run the guard and signal together on the pod to avoid a multi-command race. + ids = json.dumps([active['id'], queued['id']]) + code = r'''const fs=require('fs'); +(async()=>{ +const ids=IDS; +const a=JSON.parse(fs.readFileSync('/data/'+ids[0]+'/receipt.json')); +const q=JSON.parse(fs.readFileSync('/data/'+ids[1]+'/receipt.json')); +if(a.status!=='running'||q.status!=='queued')throw Error('test jobs not active/queued'); +const partial='/data/'+ids[0]+'/desktop-0-uncommitted.png'; +if(!fs.existsSync(partial)){console.log(JSON.stringify({waiting:true}));return;} +const pf=await (await fetch('http://127.0.0.1:8080/v1/preflight',{headers:{'X-Rapp-Protocol':'1'}})).json(); +if(pf.active!==1||pf.queued!==1||pf.release!==IMAGE)throw Error('unexpected concurrent work or release'); +const servers=fs.readdirSync('/proc').filter(x=>/^[0-9]+$/.test(x)).filter(x=>{ +try{const c=fs.readFileSync('/proc/'+x+'/cmdline','utf8').split(String.fromCharCode(0));return c[0]==='node'&&c[1]==='src/server.mjs';}catch{return false;}}); +if(servers.length!==1)throw Error('expected one service supervisor'); +const result={signal:'SIGKILL',pid:Number(servers[0]),active_status:a.status,queued_status:q.status,partial_png_bytes:fs.statSync(partial).size}; +fs.writeSync(1,JSON.stringify(result));process.kill(Number(servers[0]),'SIGKILL'); +})().catch(()=>{process.exitCode=2;});'''.replace('IDS', ids).replace('IMAGE', json.dumps(args.image)) + + def crash_when_ready(): + r = subprocess.run(['ssh', 'railiance01', shlex.join(['kubectl', '-n', 'playwright', 'exec', before['pod'], '--', 'node', '-e', code])], + capture_output=True, text=True, timeout=30) + # Killing the supervisor also terminates exec; consume only our JSON receipt. + try: + result = json.loads(r.stdout) + except ValueError: + raise ClientError('crash guard failed; raw output suppressed') from None + if result.get('waiting'): + require(r.returncode == 0, 'crash guard failed while waiting') + return None + require(result.get('signal') == 'SIGKILL', 'crash signal receipt absent') + return result + + report['crash'] = poll(crash_when_ready, 'real in-flight screenshot and guarded crash', timeout=30) + report['checks'].append('guarded_sigkill_of_active_service') + # Existing port-forward may end on container restart; the caller restores it. + def recovered(): + try: + pf = client.request('/v1/preflight') + return pf if pf.get('available') and pf.get('release') == args.image else None + except ClientError: + return None + save(args.out / 'after-preflight.json', poll(recovered, 'restore consumer tunnel after container restart', timeout=180)) + def ready_snapshot(): + try: + return snapshot(args.image) + except ClientError: + return None + after = poll(ready_snapshot, 'ready container after recovery', timeout=60) + report['after'] = after + require(after['last_exit_code'] == 137, 'expected SIGKILL termination code') + require(after['pod_uid'] == before['pod_uid'] and after['pvc_uid'] == before['pvc_uid'], 'pod or storage changed') + require(after['restart_count'] > before['restart_count'] and after['container_id'] != before['container_id'], 'container restart not observed') + for run, folder in [(active, active_dir), (queued, queued_dir)]: + r = collect(client, run, folder) + require(r['status'] == 'interrupted' and r['error'] == 'service_restarted' and r['artifacts'] == [], 'interruption receipt mismatch') + check = "const fs=require('fs');console.log(JSON.stringify(" + ids + ".map(id=>fs.readdirSync('/data/'+id))))" + files = remote('kubectl', '-n', 'playwright', 'exec', before['pod'], '--', 'node', '-e', check) + require(files == [['receipt.json'], ['receipt.json']], 'orphan artifacts remain') + report['checks'].append('interrupted_jobs_and_orphan_cleanup') + folder = args.out / 'completed-after' + folder.mkdir(mode=0o700) + restored = collect(client, completed, folder) + require(restored['status'] == 'passed' and restored['downloads'] == original['downloads'], 'completed artifacts changed') + report['completed_downloads'] = restored['downloads'] + report['checks'].append('completed_artifact_hashes_preserved') + fresh, folder = submit(spec, 'new-trial') + require(finish(fresh)['status'] == 'passed', 'new trial failed') + r = collect(client, fresh, folder) + require(r['completed_steps'] == 14 and len(r['downloads']) == 4, 'new trial incomplete') + report['checks'].append('fresh_trial_after_container_restart') + report['status'] = 'passed' + except (ClientError, OSError, ValueError, KeyError, subprocess.TimeoutExpired) as exc: + report['error'] = str(exc) + except KeyboardInterrupt: + report['error'] = 'interrupted_by_caller' + finally: + for run in runs: + try: + client.request('/v1/runs/' + run['id'], 'DELETE', token=run['token']) + except ClientError: + pass + report['finished_at'] = datetime.now(timezone.utc).isoformat() + save(args.out / 'verification.json', report) + print(json.dumps(report, indent=2)) + return 0 if report['status'] == 'passed' else 3 + + +if __name__ == '__main__': + p = argparse.ArgumentParser(description=__doc__) + p.add_argument('--url', required=True) + p.add_argument('--image', required=True) + p.add_argument('--out', type=Path, required=True) + p.add_argument('--approval-reference', required=True) + p.add_argument('--execute-crash', action='store_true') + try: + sys.exit(verify(p.parse_args())) + except (ClientError, OSError, ValueError) as exc: + print(json.dumps({'error': str(exc)}), file=sys.stderr) + sys.exit(3) diff --git a/workplans/RAPP-PLAYWRIGHT-WP-0004-consumer-verification.md b/workplans/RAPP-PLAYWRIGHT-WP-0004-consumer-verification.md index fe0345d..1f50555 100644 --- a/workplans/RAPP-PLAYWRIGHT-WP-0004-consumer-verification.md +++ b/workplans/RAPP-PLAYWRIGHT-WP-0004-consumer-verification.md @@ -165,3 +165,20 @@ cancellation. Wrong release and wrong execution location each fail before job submission. Retained [verification](../docs/evidence/2026-10-13-consumer-verifier.json). The command does not establish authentication, restart workloads or close T01; pair it with native identity and permission receipts plus remaining recovery tests. + + +## Abrupt recovery passed — 2026-10-13 22:29 Europe/Berlin + +User authorized the remaining recovery verification. Added and executed isolated +released-container and approved managed-pilot crash verifiers. A real screenshot +was present in an active job and another job was queued before SIGKILL. Managed +container exited137 and restarted on the same pod/PVC/image; both runs became +interrupted, partial files disappeared, completed PNG hashes matched and a fresh +14-step/four-PNG trial passed. Dedicated test connections were stopped. See +[executed recovery evidence](../docs/evidence/2026-10-13-crash-recovery.md). + +The earlier abrupt-interruption gap is resolved. T01 remains wait for actual +human/agent identity/transport receipts and cross-version rollback; no second +release was invented solely to claim rollback. T03 remains dependent on managed +consumer acceptance. No new identity-owner implementation return was available +at this execution; CUST-WP-0080-T05 retains coordination.