coulomb/rapp-playwright
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Prove abrupt recovery with live jobs in released and managed containers

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13590-a16f-7f60-be60-fd71627bc99d
This commit is contained in:
tegwick 2026-10-13 22:32:10 +02:00
parent 5b8e798419
commit 2cb2b654e0
25 changed files with 1954 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

55
docs/evidence/2026-10-13-crash-recovery.md Normal file
View file

@ -0,0 +1,55 @@
# Real abrupt-interruption recovery — 2026-10-13
Both the isolated published container and the managed railiance01 pilot passed
an actual SIGKILL recovery test. No browser implementation or image changed.
The user approved the remaining recovery work with “Ok, lets do that.” Managed
execution was ADMINISTER @ realm:kubernetes/railiance01, activation APPROVED;
existing SSH access was used as an operator path, not consumer identity proof.
## Local released container
`python3 tools/verify_crash_recovery.py --image
forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441
--out .local/crash-recovery-01` passed. Docker kill produced exit137 without OOM;
the same container/data restarted. Its temporary container was removed. Test
handles/data remain private under ignored `.local/`.
## Managed pilot
Ran `tools/verify_managed_recovery.py` with the same immutable image, operator
forward `http://127.0.0.1:18087`, output `.local/managed-crash-recovery-02`, specific
approval reference and `--execute-crash`. Before starting, the service reported
zero active and queued jobs. Guarded termination targeted only the verified
`node src/server.mjs` supervisor after our active run wrote a 25,740-byte PNG and
our second run was queued. It was not a fabricated persisted receipt.
At 20:28:49Z, the container exited137; its replacement started at 20:28:51Z.
Restart count changed 0 → 1, container identity changed, and pod UID
`a22030a4-c12d-4e97-be8d-98a8d80808c2`, PVC UID
`da36929a-1d69-457a-ad99-7580ef86ab18` and release digest remained unchanged.
The pilot returned Ready. The dedicated forward was re-established after restart
and stopped after verification. No grants, manifests or shared controls changed.
Both tests proved:
- Actual running and queued jobs recovered as `interrupted/service_restarted`.
- Uncommitted PNG and other orphan files were removed; interrupted receipts expose
no artifacts. The local check also verified HTTP404 for the former PNG.
- Completed receipt and all four downloaded PNG hashes remained identical.
- A fresh responsive/keyboard trial completed all 14 steps and four downloads.
An earlier managed attempt lost its workstation Kubernetes port-forward before
the crash step. It made no service interruption: restart count remained zero,
and its baseline trial completed. That failed attempt is retained separately.
The successful run used an SSH-local forward to a node-local Kubernetes forward.
No observer timeout was treated as evidence that a running process had stopped.
Receipts: [local](2026-10-13-crash-recovery/local/verification.json),
[managed](2026-10-13-crash-recovery/managed/verification.json),
[first attempt](2026-10-13-crash-recovery/managed-first-attempt.json).
Selected before/after, interrupted and fresh-run receipts accompany those files;
run capability handles and storage token hashes are excluded.
`make test` passed all three unit cases and syntax checks. The two released-image
recovery tests above passed against real Chromium. These establish same-image
abrupt recovery, not cross-version rollback or real human/agent authentication.

29
docs/evidence/2026-10-13-crash-recovery/local/active-receipt.json Normal file
View file

@ -0,0 +1,29 @@
{
"id": "2665c7cd-c707-4a7d-91eb-c51c91462410",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:29.324Z",
"expires_at": "2026-10-13T21:23:29.324Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"started_at": "2026-10-13T20:23:29.334Z",
"finished_at": "2026-10-13T20:23:46.954Z",
"downloads": []
}

22
docs/evidence/2026-10-13-crash-recovery/local/after-preflight.json Normal file
View file

@ -0,0 +1,22 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:23:54.299Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "isolated-crash-verification",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

22
docs/evidence/2026-10-13-crash-recovery/local/before-preflight.json Normal file
View file

@ -0,0 +1,22 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:23:15.520Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "isolated-crash-verification",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

184
docs/evidence/2026-10-13-crash-recovery/local/completed-after-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:15.654Z",
"expires_at": "2026-10-13T21:23:15.654Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:21.899Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.384Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.587Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:22.602Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.690Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.708Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:22.794Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:26.394Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:26.919Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.208Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:27.295Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.391Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:27.483Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:27.510Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:15.663Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:23:29.212Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

184
docs/evidence/2026-10-13-crash-recovery/local/completed-before-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:15.654Z",
"expires_at": "2026-10-13T21:23:15.654Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:21.899Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.384Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.587Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:22.602Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:22.690Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:22.708Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:22.794Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:23:26.394Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:26.919Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.208Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:23:27.295Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:23:27.391Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:23:27.483Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:23:27.510Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:15.663Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:23:29.212Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

184
docs/evidence/2026-10-13-crash-recovery/local/new-trial-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "263a5639-e17a-4828-b33d-9434cb0a9c6b",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:54.497Z",
"expires_at": "2026-10-13T21:23:54.497Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:24:03.160Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:03.858Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:04.139Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:24:04.162Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:04.250Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:04.264Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:24:04.347Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:24:07.458Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:07.760Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:07.954Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:24:08.042Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:24:08.059Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:24:08.073Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:24:08.165Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:23:54.510Z",
"blocked_requests": 13,
"finished_at": "2026-10-13T20:24:09.482Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

33
docs/evidence/2026-10-13-crash-recovery/local/queued-receipt.json Normal file
View file

@ -0,0 +1,33 @@
{
"id": "6a0b47e4-2b47-4d77-9d4b-2f1615d849ed",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:23:43.568Z",
"expires_at": "2026-10-13T21:23:43.568Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "isolated-crash-verification",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"finished_at": "2026-10-13T20:23:46.987Z",
"downloads": []
}

45
docs/evidence/2026-10-13-crash-recovery/local/verification.json Normal file
View file

@ -0,0 +1,45 @@
{
"started_at": "2026-10-13T20:23:09.595214+00:00",
"image": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"location": "isolated-local-docker",
"signal": "SIGKILL",
"checks": [
"completed_trial_and_four_downloads",
"real_active_and_queued_jobs_killed_without_shutdown",
"active_and_queued_interrupted_orphans_removed",
"completed_receipt_and_hashes_preserved",
"fresh_keyboard_screenshot_trial_after_recovery"
],
"status": "passed",
"managed_recovery_proven": false,
"persisted_before_kill": {
"active": "running",
"queued": "queued",
"partial_png_bytes": 25740
},
"exit_code_after_kill": 137,
"completed_downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
],
"test_container_removed": true,
"finished_at": "2026-10-13T20:24:11.347753+00:00"
}

20
docs/evidence/2026-10-13-crash-recovery/managed-first-attempt.json Normal file
View file

@ -0,0 +1,20 @@
{
"started_at": "2026-10-13T20:26:47.877053+00:00",
"status": "failed",
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
"activation": "APPROVED",
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
"identity_acceptance": false,
"cross_version_rollback": false,
"checks": [],
"before": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 0,
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18"
},
"error": "service_unreachable: establish the documented tunnel and retry",
"finished_at": "2026-10-13T20:27:39.311029+00:00"
}

29
docs/evidence/2026-10-13-crash-recovery/managed/active-receipt.json Normal file
View file

@ -0,0 +1,29 @@
{
"id": "6440ad74-27d1-4daf-a768-768e041f6741",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:46.325Z",
"expires_at": "2026-10-13T21:28:46.325Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"started_at": "2026-10-13T20:28:46.328Z",
"finished_at": "2026-10-13T20:28:51.153Z",
"downloads": []
}

24
docs/evidence/2026-10-13-crash-recovery/managed/after-preflight.json Normal file
View file

@ -0,0 +1,24 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:28:52.784Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "railiance01/playwright",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [
"https://onco.cc"
],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

184
docs/evidence/2026-10-13-crash-recovery/managed/completed-after-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:41.609Z",
"expires_at": "2026-10-13T21:28:41.609Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:43.700Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.839Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.907Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:43.920Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.944Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.952Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:43.961Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:44.858Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.052Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.105Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:45.119Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.137Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.147Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:45.183Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:28:41.611Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:28:45.914Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

184
docs/evidence/2026-10-13-crash-recovery/managed/completed-before-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:41.609Z",
"expires_at": "2026-10-13T21:28:41.609Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:43.700Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.839Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.907Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:43.920Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:43.944Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:43.952Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:43.961Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:28:44.858Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.052Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.105Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:28:45.119Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:28:45.137Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:28:45.147Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:28:45.183Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:28:41.611Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:28:45.914Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

184
docs/evidence/2026-10-13-crash-recovery/managed/new-trial-receipt.json Normal file
View file

@ -0,0 +1,184 @@
{
"id": "f76f4662-03cd-4596-928e-ee34eeaa4057",
"api_version": "1",
"status": "passed",
"error": null,
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:29:17.318Z",
"expires_at": "2026-10-13T21:29:17.318Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": "156.0.8078.4",
"completed_steps": 14,
"steps": [
{
"viewport": "desktop",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:29:19.494Z",
"selector": "h1"
},
{
"viewport": "desktop",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:19.657Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:19.745Z",
"key": "Tab"
},
{
"viewport": "desktop",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:29:19.760Z",
"selector": "#increment"
},
{
"viewport": "desktop",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:19.797Z",
"key": "Enter"
},
{
"viewport": "desktop",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:19.802Z",
"selector": "#count"
},
{
"viewport": "desktop",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:29:19.809Z"
},
{
"viewport": "narrow",
"index": 0,
"action": "visible",
"status": "passed",
"at": "2026-10-13T20:29:20.740Z",
"selector": "h1"
},
{
"viewport": "narrow",
"index": 1,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:20.862Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 2,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:20.930Z",
"key": "Tab"
},
{
"viewport": "narrow",
"index": 3,
"action": "focused",
"status": "passed",
"at": "2026-10-13T20:29:20.958Z",
"selector": "#increment"
},
{
"viewport": "narrow",
"index": 4,
"action": "press",
"status": "passed",
"at": "2026-10-13T20:29:20.974Z",
"key": "Enter"
},
{
"viewport": "narrow",
"index": 5,
"action": "text",
"status": "passed",
"at": "2026-10-13T20:29:20.981Z",
"selector": "#count"
},
{
"viewport": "narrow",
"index": 6,
"action": "screenshot",
"status": "passed",
"at": "2026-10-13T20:29:20.989Z"
}
],
"artifacts": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622
},
{
"name": "desktop-final.png",
"bytes": 26622
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280
},
{
"name": "narrow-final.png",
"bytes": 24280
}
],
"started_at": "2026-10-13T20:29:17.319Z",
"blocked_requests": 12,
"finished_at": "2026-10-13T20:29:21.401Z",
"downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
]
}

24
docs/evidence/2026-10-13-crash-recovery/managed/preflight.json Normal file
View file

@ -0,0 +1,24 @@
{
"available": true,
"error": null,
"checked_at": "2026-10-13T20:28:18.565Z",
"browser_version": "156.0.8078.4",
"api_version": "1",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"execution_location": "railiance01/playwright",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"allowed_origins": [
"https://onco.cc"
],
"fixture": true,
"active": 0,
"queued": 0,
"limits": {
"concurrency": 1,
"max_queue": 4,
"max_timeout_ms": 120000,
"retention_ms": 3600000,
"max_runs": 64
}
}

33
docs/evidence/2026-10-13-crash-recovery/managed/queued-receipt.json Normal file
View file

@ -0,0 +1,33 @@
{
"id": "74e433ef-aee7-4c74-afe5-3171214f4014",
"api_version": "1",
"status": "interrupted",
"error": "service_restarted",
"label": "independent-browser-trial",
"target": "fixture",
"viewports": [
{
"name": "desktop",
"width": 1280,
"height": 800
},
{
"name": "narrow",
"width": 390,
"height": 844
}
],
"timeout_ms": 60000,
"created_at": "2026-10-13T20:28:46.380Z",
"expires_at": "2026-10-13T21:28:46.380Z",
"service_version": "0.1.0",
"playwright_version": "1.64.0",
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"execution_location": "railiance01/playwright",
"browser_version": null,
"completed_steps": 0,
"steps": [],
"artifacts": [],
"finished_at": "2026-10-13T20:28:51.157Z",
"downloads": []
}

64
docs/evidence/2026-10-13-crash-recovery/managed/verification.json Normal file
View file

@ -0,0 +1,64 @@
{
"started_at": "2026-10-13T20:28:36.758090+00:00",
"status": "passed",
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
"activation": "APPROVED",
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
"identity_acceptance": false,
"cross_version_rollback": false,
"checks": [
"baseline_trial",
"guarded_sigkill_of_active_service",
"interrupted_jobs_and_orphan_cleanup",
"completed_artifact_hashes_preserved",
"fresh_trial_after_container_restart"
],
"before": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 0,
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
"last_exit_code": null
},
"crash": {
"signal": "SIGKILL",
"pid": 7,
"active_status": "running",
"queued_status": "queued",
"partial_png_bytes": 25740
},
"after": {
"pod": "playwright-c5b9c8bd7-fbfqv",
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
"restart_count": 1,
"container_id": "containerd://abd1f7081045b1b4505e8045bf298e0a10485834d760d6dbf8db73accdb3da08",
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
"last_exit_code": 137
},
"completed_downloads": [
{
"name": "desktop-6-keyboard-result.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "desktop-final.png",
"bytes": 26622,
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
},
{
"name": "narrow-6-keyboard-result.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
},
{
"name": "narrow-final.png",
"bytes": 24280,
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
}
],
"finished_at": "2026-10-13T20:29:20.202087+00:00"
}