Prove abrupt recovery with live jobs in released and managed containers

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13590-a16f-7f60-be60-fd71627bc99d
This commit is contained in:
tegwick 2026-10-13 22:32:10 +02:00
parent 5b8e798419
commit 2cb2b654e0
25 changed files with 1954 additions and 4 deletions

View file

@ -0,0 +1,55 @@
# Real abrupt-interruption recovery — 2026-10-13
Both the isolated published container and the managed railiance01 pilot passed
an actual SIGKILL recovery test. No browser implementation or image changed.
The user approved the remaining recovery work with “Ok, lets do that.” Managed
execution was ADMINISTER @ realm:kubernetes/railiance01, activation APPROVED;
existing SSH access was used as an operator path, not consumer identity proof.
## Local released container
`python3 tools/verify_crash_recovery.py --image
forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441
--out .local/crash-recovery-01` passed. Docker kill produced exit137 without OOM;
the same container/data restarted. Its temporary container was removed. Test
handles/data remain private under ignored `.local/`.
## Managed pilot
Ran `tools/verify_managed_recovery.py` with the same immutable image, operator
forward `http://127.0.0.1:18087`, output `.local/managed-crash-recovery-02`, specific
approval reference and `--execute-crash`. Before starting, the service reported
zero active and queued jobs. Guarded termination targeted only the verified
`node src/server.mjs` supervisor after our active run wrote a 25,740-byte PNG and
our second run was queued. It was not a fabricated persisted receipt.
At 20:28:49Z, the container exited137; its replacement started at 20:28:51Z.
Restart count changed 0 → 1, container identity changed, and pod UID
`a22030a4-c12d-4e97-be8d-98a8d80808c2`, PVC UID
`da36929a-1d69-457a-ad99-7580ef86ab18` and release digest remained unchanged.
The pilot returned Ready. The dedicated forward was re-established after restart
and stopped after verification. No grants, manifests or shared controls changed.
Both tests proved:
- Actual running and queued jobs recovered as `interrupted/service_restarted`.
- Uncommitted PNG and other orphan files were removed; interrupted receipts expose
no artifacts. The local check also verified HTTP404 for the former PNG.
- Completed receipt and all four downloaded PNG hashes remained identical.
- A fresh responsive/keyboard trial completed all 14 steps and four downloads.
An earlier managed attempt lost its workstation Kubernetes port-forward before
the crash step. It made no service interruption: restart count remained zero,
and its baseline trial completed. That failed attempt is retained separately.
The successful run used an SSH-local forward to a node-local Kubernetes forward.
No observer timeout was treated as evidence that a running process had stopped.
Receipts: [local](2026-10-13-crash-recovery/local/verification.json),
[managed](2026-10-13-crash-recovery/managed/verification.json),
[first attempt](2026-10-13-crash-recovery/managed-first-attempt.json).
Selected before/after, interrupted and fresh-run receipts accompany those files;
run capability handles and storage token hashes are excluded.
`make test` passed all three unit cases and syntax checks. The two released-image
recovery tests above passed against real Chromium. These establish same-image
abrupt recovery, not cross-version rollback or real human/agent authentication.

View file

@ -0,0 +1,29 @@
{
"id": "2665c7cd-c707-4a7d-91eb-c51c91462410",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:29.324Z",
"expires_at": "2026-10-13T21:23:29.324Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"started_at": "2026-10-13T20:23:29.334Z",
"finished_at": "2026-10-13T20:23:46.954Z",
"downloads": []
}

View file

@ -0,0 +1,22 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:23:54.299Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "isolated-crash-verification",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

View file

@ -0,0 +1,22 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:23:15.520Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "isolated-crash-verification",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

View file

@ -0,0 +1,184 @@
{
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:15.654Z",
"expires_at": "2026-10-13T21:23:15.654Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:21.899Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.384Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.587Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:22.602Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.690Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.708Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:22.794Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:26.394Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:26.919Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.208Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:27.295Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.391Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:27.483Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:27.510Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:15.663Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:23:29.212Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,184 @@
{
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:15.654Z",
"expires_at": "2026-10-13T21:23:15.654Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:21.899Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.384Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.587Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:22.602Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.690Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.708Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:22.794Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:26.394Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:26.919Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.208Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:27.295Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.391Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:27.483Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:27.510Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:15.663Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:23:29.212Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,184 @@
{
"id": "263a5639-e17a-4828-b33d-9434cb0a9c6b",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:54.497Z",
"expires_at": "2026-10-13T21:23:54.497Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:24:03.160Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:03.858Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:04.139Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:24:04.162Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:04.250Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:04.264Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:24:04.347Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:24:07.458Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:07.760Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:07.954Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:24:08.042Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:08.059Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:08.073Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:24:08.165Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:54.510Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:24:09.482Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,33 @@
{
"id": "6a0b47e4-2b47-4d77-9d4b-2f1615d849ed",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:43.568Z",
"expires_at": "2026-10-13T21:23:43.568Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"finished_at": "2026-10-13T20:23:46.987Z",
"downloads": []
}

View file

@ -0,0 +1,45 @@
{
"started_at": "2026-10-13T20:23:09.595214+00:00",
"image": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"location": "isolated-local-docker",
"signal": "SIGKILL",
"checks": [
"completed_trial_and_four_downloads",
"real_active_and_queued_jobs_killed_without_shutdown",
"active_and_queued_interrupted_orphans_removed",
"completed_receipt_and_hashes_preserved",
"fresh_keyboard_screenshot_trial_after_recovery"
],
"status": "passed",
"managed_recovery_proven": false,
"persisted_before_kill": {
"active": "running",
"queued": "queued",
"partial_png_bytes": 25740
},
"exit_code_after_kill": 137,
"completed_downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
],
"test_container_removed": true,
"finished_at": "2026-10-13T20:24:11.347753+00:00"
}

View file

@ -0,0 +1,20 @@
{
"started_at": "2026-10-13T20:26:47.877053+00:00",
"status": "failed",
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
"activation": "APPROVED",
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
"identity_acceptance": false,
"cross_version_rollback": false,
"checks": [],
"before": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 0,
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18"
},
"error": "service_unreachable: establish the documented tunnel and retry",
"finished_at": "2026-10-13T20:27:39.311029+00:00"
}

View file

@ -0,0 +1,29 @@
{
"id": "6440ad74-27d1-4daf-a768-768e041f6741",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:46.325Z",
"expires_at": "2026-10-13T21:28:46.325Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"started_at": "2026-10-13T20:28:46.328Z",
"finished_at": "2026-10-13T20:28:51.153Z",
"downloads": []
}

View file

@ -0,0 +1,24 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:28:52.784Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "railiance01/playwright",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [
"https://onco.cc"
],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

View file

@ -0,0 +1,184 @@
{
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:41.609Z",
"expires_at": "2026-10-13T21:28:41.609Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:43.700Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.839Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.907Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:43.920Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.944Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.952Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:43.961Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:44.858Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.052Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.105Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:45.119Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.137Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.147Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:45.183Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:28:41.611Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:28:45.914Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,184 @@
{
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:41.609Z",
"expires_at": "2026-10-13T21:28:41.609Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:43.700Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.839Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.907Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:43.920Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.944Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.952Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:43.961Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:44.858Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.052Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.105Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:45.119Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.137Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.147Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:45.183Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:28:41.611Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:28:45.914Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,184 @@
{
"id": "f76f4662-03cd-4596-928e-ee34eeaa4057",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:29:17.318Z",
"expires_at": "2026-10-13T21:29:17.318Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:29:19.494Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:19.657Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:19.745Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:29:19.760Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:19.797Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:19.802Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:29:19.809Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:29:20.740Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:20.862Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:20.930Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:29:20.958Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:20.974Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:20.981Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:29:20.989Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:29:17.319Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:29:21.401Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

View file

@ -0,0 +1,24 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:28:18.565Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "railiance01/playwright",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [
"https://onco.cc"
],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

View file

@ -0,0 +1,33 @@
{
"id": "74e433ef-aee7-4c74-afe5-3171214f4014",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:46.380Z",
"expires_at": "2026-10-13T21:28:46.380Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"finished_at": "2026-10-13T20:28:51.157Z",
"downloads": []
}

View file

@ -0,0 +1,64 @@
{
"started_at": "2026-10-13T20:28:36.758090+00:00",
"status": "passed",
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
"activation": "APPROVED",
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
"identity_acceptance": false,
"cross_version_rollback": false,
"checks": [
"baseline_trial",
"guarded_sigkill_of_active_service",
"interrupted_jobs_and_orphan_cleanup",
"completed_artifact_hashes_preserved",
"fresh_trial_after_container_restart"
],
"before": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 0,
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
"last_exit_code": null
},
"crash": {
"signal": "SIGKILL",
"pid": 7,
"active_status": "running",
"queued_status": "queued",
"partial_png_bytes": 25740
},
"after": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 1,
"container_id": "containerd://abd1f7081045b1b4505e8045bf298e0a10485834d760d6dbf8db73accdb3da08",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
"last_exit_code": 137
},
"completed_downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
],
"finished_at": "2026-10-13T20:29:20.202087+00:00"
}

View file

@ -111,3 +111,33 @@ Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream
termination and cross-version rollback require their separate evidence. The
command's local validation is retained in
[evidence](evidence/2026-10-13-consumer-verifier.json).
## Abrupt interruption verification
`tools/verify_crash_recovery.py --image <published-immutable-reference> --out
.local/new-crash-check` creates its own bounded, read-only Docker container and
private data directory. It verifies a completed trial, captures a partial PNG in
a real active job, queues another job, sends SIGKILL to the container, and starts
it on the same data. It requires interrupted receipts, orphan cleanup, preserved
completed hashes and a fresh trial; it removes only its own container. Run it
as workstation UID 1000 to match the released image's data-directory ownership.
The separately approved managed check is `tools/verify_managed_recovery.py`.
It is an ADMINISTER operation through existing SSH, never a consumer permission.
Use `--url <operator-loopback-forward> --image <published-reference> --out
.local/new-managed-crash --approval-reference <specific-founder-approval>
--execute-crash`. Merely supplying that flag/reference does not grant approval.
The tool pins railiance01's node address, one Ready pilot pod and its release,
requires an idle service, creates its own jobs, and guards the supervisor SIGKILL
on the expected active/queued jobs and partial screenshot. It verifies container
exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files,
completed hashes and a fresh trial. No replicas or manifests are changed.
The port-forward may exit when the container restarts. Re-establish the dedicated
operator connection after the container is Ready; the verification waits up to
180 seconds for it. Do not retry the crash command solely because the tunnel
was lost: inspect the existing verification process and pod restart state first.
Keep every `run.json` and the local data directory private. Share only selected
receipts under `docs/evidence`. These tools do not prove real consumer identity,
revocation or cross-version rollback. [Executed local and managed evidence](evidence/2026-10-13-crash-recovery.md).