Prove abrupt recovery with live jobs in released and managed containers
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a13590-a16f-7f60-be60-fd71627bc99d
This commit is contained in:
parent
5b8e798419
commit
2cb2b654e0
25 changed files with 1954 additions and 4 deletions
2
Makefile
2
Makefile
|
|
@ -10,7 +10,7 @@ install:
|
|||
npm ci --ignore-scripts
|
||||
check:
|
||||
npm run check
|
||||
python3 -m py_compile client/rapp_playwright.py tools/render.py tools/verify_consumer.py
|
||||
python3 -m py_compile client/rapp_playwright.py tools/render.py tools/verify_consumer.py tools/verify_crash_recovery.py tools/verify_managed_recovery.py
|
||||
test: check
|
||||
npm test
|
||||
test-integration:
|
||||
|
|
|
|||
9
SCOPE.md
9
SCOPE.md
|
|
@ -48,7 +48,7 @@ artifacts expire; sanitized evidence under `docs/evidence/` is retained.
|
|||
| --- | --- |
|
||||
| WP-0002 | Finished: consumption contract, lifecycle and trial/pilot design |
|
||||
| WP-0003 | Finished: tested image published at its exact digest and pulled on railiance01; ingress upload deadline fixed. |
|
||||
| WP-0004 | T02 provider trial/pilot done. T01 awaits actual consumer identity/access and remaining recovery evidence. T03 has published A5 local-provider discovery; final managed availability update awaits T01. |
|
||||
| WP-0004 | T02 provider trial/pilot done. T01 awaits actual consumer identity/access and cross-version rollback evidence. T03 has published A5 local-provider discovery; final managed availability update awaits T01. |
|
||||
|
||||
Platform admitted and deployed the reviewed package under RPF-WP-0056. The
|
||||
`playwright-consumers` group is bound to the namespace consumer Role. Initial
|
||||
|
|
@ -72,7 +72,10 @@ Platform's [consumer access instructions](https://forgejo.coulomb.social/coulomb
|
|||
are prepared but conditional on identity-owner activation. The earlier
|
||||
[admission review](docs/platform-admission.md) is retained as history; package
|
||||
admission, deployment and the binding are no longer the outstanding decision.
|
||||
Abrupt-interruption recovery and actual cross-version rollback remain unverified.
|
||||
Actual abrupt-interruption recovery is now verified locally and on the managed
|
||||
pilot: active/queued jobs become interrupted, partial artifacts disappear, completed
|
||||
hashes survive, and a fresh trial passes. See [crash evidence](docs/evidence/2026-10-13-crash-recovery.md).
|
||||
Actual cross-version rollback remains unverified; there is only one released version.
|
||||
|
||||
The repeatable [consumer API acceptance command](docs/operations.md#repeatable-consumer-api-acceptance)
|
||||
now passes six local API checks and refuses release/location mismatches. It is
|
||||
|
|
@ -88,5 +91,5 @@ sandbox is disabled; hostile pages and mutually untrusted tenants are unsupporte
|
|||
Target failures identify the category but not detailed DNS/TLS/HTTP causes.
|
||||
The OnCo pilot covers the home page, not all consumer routes or accessibility.
|
||||
|
||||
See the [timestamped intent assessment](history/2026-10-13-2211-intent-scope-assessment.md)
|
||||
See the [timestamped intent assessment](history/2026-10-13-2231-intent-scope-assessment.md)
|
||||
for the prioritized gaps and their acceptance evidence.
|
||||
|
|
|
|||
55
docs/evidence/2026-10-13-crash-recovery.md
Normal file
55
docs/evidence/2026-10-13-crash-recovery.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
# Real abrupt-interruption recovery — 2026-10-13
|
||||
|
||||
Both the isolated published container and the managed railiance01 pilot passed
|
||||
an actual SIGKILL recovery test. No browser implementation or image changed.
|
||||
The user approved the remaining recovery work with “Ok, lets do that.” Managed
|
||||
execution was ADMINISTER @ realm:kubernetes/railiance01, activation APPROVED;
|
||||
existing SSH access was used as an operator path, not consumer identity proof.
|
||||
|
||||
## Local released container
|
||||
|
||||
`python3 tools/verify_crash_recovery.py --image
|
||||
forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441
|
||||
--out .local/crash-recovery-01` passed. Docker kill produced exit137 without OOM;
|
||||
the same container/data restarted. Its temporary container was removed. Test
|
||||
handles/data remain private under ignored `.local/`.
|
||||
|
||||
## Managed pilot
|
||||
|
||||
Ran `tools/verify_managed_recovery.py` with the same immutable image, operator
|
||||
forward `http://127.0.0.1:18087`, output `.local/managed-crash-recovery-02`, specific
|
||||
approval reference and `--execute-crash`. Before starting, the service reported
|
||||
zero active and queued jobs. Guarded termination targeted only the verified
|
||||
`node src/server.mjs` supervisor after our active run wrote a 25,740-byte PNG and
|
||||
our second run was queued. It was not a fabricated persisted receipt.
|
||||
|
||||
At 20:28:49Z, the container exited137; its replacement started at 20:28:51Z.
|
||||
Restart count changed 0 → 1, container identity changed, and pod UID
|
||||
`a22030a4-c12d-4e97-be8d-98a8d80808c2`, PVC UID
|
||||
`da36929a-1d69-457a-ad99-7580ef86ab18` and release digest remained unchanged.
|
||||
The pilot returned Ready. The dedicated forward was re-established after restart
|
||||
and stopped after verification. No grants, manifests or shared controls changed.
|
||||
|
||||
Both tests proved:
|
||||
|
||||
- Actual running and queued jobs recovered as `interrupted/service_restarted`.
|
||||
- Uncommitted PNG and other orphan files were removed; interrupted receipts expose
|
||||
no artifacts. The local check also verified HTTP404 for the former PNG.
|
||||
- Completed receipt and all four downloaded PNG hashes remained identical.
|
||||
- A fresh responsive/keyboard trial completed all 14 steps and four downloads.
|
||||
|
||||
An earlier managed attempt lost its workstation Kubernetes port-forward before
|
||||
the crash step. It made no service interruption: restart count remained zero,
|
||||
and its baseline trial completed. That failed attempt is retained separately.
|
||||
The successful run used an SSH-local forward to a node-local Kubernetes forward.
|
||||
No observer timeout was treated as evidence that a running process had stopped.
|
||||
|
||||
Receipts: [local](2026-10-13-crash-recovery/local/verification.json),
|
||||
[managed](2026-10-13-crash-recovery/managed/verification.json),
|
||||
[first attempt](2026-10-13-crash-recovery/managed-first-attempt.json).
|
||||
Selected before/after, interrupted and fresh-run receipts accompany those files;
|
||||
run capability handles and storage token hashes are excluded.
|
||||
|
||||
`make test` passed all three unit cases and syntax checks. The two released-image
|
||||
recovery tests above passed against real Chromium. These establish same-image
|
||||
abrupt recovery, not cross-version rollback or real human/agent authentication.
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
{
|
||||
"id": "2665c7cd-c707-4a7d-91eb-c51c91462410",
|
||||
"api_version": "1",
|
||||
"status": "interrupted",
|
||||
"error": "service_restarted",
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:23:29.324Z",
|
||||
"expires_at": "2026-10-13T21:23:29.324Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"browser_version": null,
|
||||
"completed_steps": 0,
|
||||
"steps": [],
|
||||
"artifacts": [],
|
||||
"started_at": "2026-10-13T20:23:29.334Z",
|
||||
"finished_at": "2026-10-13T20:23:46.954Z",
|
||||
"downloads": []
|
||||
}
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"available": true,
|
||||
"error": null,
|
||||
"checked_at": "2026-10-13T20:23:54.299Z",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"api_version": "1",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"allowed_origins": [],
|
||||
"fixture": true,
|
||||
"active": 0,
|
||||
"queued": 0,
|
||||
"limits": {
|
||||
"concurrency": 1,
|
||||
"max_queue": 4,
|
||||
"max_timeout_ms": 120000,
|
||||
"retention_ms": 3600000,
|
||||
"max_runs": 64
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"available": true,
|
||||
"error": null,
|
||||
"checked_at": "2026-10-13T20:23:15.520Z",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"api_version": "1",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"allowed_origins": [],
|
||||
"fixture": true,
|
||||
"active": 0,
|
||||
"queued": 0,
|
||||
"limits": {
|
||||
"concurrency": 1,
|
||||
"max_queue": 4,
|
||||
"max_timeout_ms": 120000,
|
||||
"retention_ms": 3600000,
|
||||
"max_runs": 64
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:23:15.654Z",
|
||||
"expires_at": "2026-10-13T21:23:15.654Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:21.899Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.384Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.587Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.602Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.690Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.708Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.794Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:26.394Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:26.919Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.208Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.295Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.391Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.483Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.510Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:23:15.663Z",
|
||||
"blocked_requests": 13,
|
||||
"finished_at": "2026-10-13T20:23:29.212Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "c4a3f9be-b860-4b02-be59-1a4a65102d2a",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:23:15.654Z",
|
||||
"expires_at": "2026-10-13T21:23:15.654Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:21.899Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.384Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.587Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.602Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.690Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.708Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:22.794Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:26.394Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:26.919Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.208Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.295Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.391Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.483Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:23:27.510Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:23:15.663Z",
|
||||
"blocked_requests": 13,
|
||||
"finished_at": "2026-10-13T20:23:29.212Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "263a5639-e17a-4828-b33d-9434cb0a9c6b",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:23:54.497Z",
|
||||
"expires_at": "2026-10-13T21:23:54.497Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:03.160Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:03.858Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:04.139Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:04.162Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:04.250Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:04.264Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:04.347Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:07.458Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:07.760Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:07.954Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:08.042Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:08.059Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:08.073Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:24:08.165Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:23:54.510Z",
|
||||
"blocked_requests": 13,
|
||||
"finished_at": "2026-10-13T20:24:09.482Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"id": "6a0b47e4-2b47-4d77-9d4b-2f1615d849ed",
|
||||
"api_version": "1",
|
||||
"status": "interrupted",
|
||||
"error": "service_restarted",
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:23:43.568Z",
|
||||
"expires_at": "2026-10-13T21:23:43.568Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "isolated-crash-verification",
|
||||
"browser_version": null,
|
||||
"completed_steps": 0,
|
||||
"steps": [],
|
||||
"artifacts": [],
|
||||
"finished_at": "2026-10-13T20:23:46.987Z",
|
||||
"downloads": []
|
||||
}
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
{
|
||||
"started_at": "2026-10-13T20:23:09.595214+00:00",
|
||||
"image": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"location": "isolated-local-docker",
|
||||
"signal": "SIGKILL",
|
||||
"checks": [
|
||||
"completed_trial_and_four_downloads",
|
||||
"real_active_and_queued_jobs_killed_without_shutdown",
|
||||
"active_and_queued_interrupted_orphans_removed",
|
||||
"completed_receipt_and_hashes_preserved",
|
||||
"fresh_keyboard_screenshot_trial_after_recovery"
|
||||
],
|
||||
"status": "passed",
|
||||
"managed_recovery_proven": false,
|
||||
"persisted_before_kill": {
|
||||
"active": "running",
|
||||
"queued": "queued",
|
||||
"partial_png_bytes": 25740
|
||||
},
|
||||
"exit_code_after_kill": 137,
|
||||
"completed_downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
],
|
||||
"test_container_removed": true,
|
||||
"finished_at": "2026-10-13T20:24:11.347753+00:00"
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"started_at": "2026-10-13T20:26:47.877053+00:00",
|
||||
"status": "failed",
|
||||
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
|
||||
"activation": "APPROVED",
|
||||
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
|
||||
"identity_acceptance": false,
|
||||
"cross_version_rollback": false,
|
||||
"checks": [],
|
||||
"before": {
|
||||
"pod": "playwright-c5b9c8bd7-fbfqv",
|
||||
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
|
||||
"restart_count": 0,
|
||||
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
|
||||
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18"
|
||||
},
|
||||
"error": "service_unreachable: establish the documented tunnel and retry",
|
||||
"finished_at": "2026-10-13T20:27:39.311029+00:00"
|
||||
}
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
{
|
||||
"id": "6440ad74-27d1-4daf-a768-768e041f6741",
|
||||
"api_version": "1",
|
||||
"status": "interrupted",
|
||||
"error": "service_restarted",
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:28:46.325Z",
|
||||
"expires_at": "2026-10-13T21:28:46.325Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"browser_version": null,
|
||||
"completed_steps": 0,
|
||||
"steps": [],
|
||||
"artifacts": [],
|
||||
"started_at": "2026-10-13T20:28:46.328Z",
|
||||
"finished_at": "2026-10-13T20:28:51.153Z",
|
||||
"downloads": []
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"available": true,
|
||||
"error": null,
|
||||
"checked_at": "2026-10-13T20:28:52.784Z",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"api_version": "1",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"allowed_origins": [
|
||||
"https://onco.cc"
|
||||
],
|
||||
"fixture": true,
|
||||
"active": 0,
|
||||
"queued": 0,
|
||||
"limits": {
|
||||
"concurrency": 1,
|
||||
"max_queue": 4,
|
||||
"max_timeout_ms": 120000,
|
||||
"retention_ms": 3600000,
|
||||
"max_runs": 64
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:28:41.609Z",
|
||||
"expires_at": "2026-10-13T21:28:41.609Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.700Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.839Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.907Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.920Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.944Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.952Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.961Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:44.858Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.052Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.105Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.119Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.137Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.147Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.183Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:28:41.611Z",
|
||||
"blocked_requests": 12,
|
||||
"finished_at": "2026-10-13T20:28:45.914Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "7643cd14-8627-42e1-9cc4-4a798b1ae7c1",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:28:41.609Z",
|
||||
"expires_at": "2026-10-13T21:28:41.609Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.700Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.839Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.907Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.920Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.944Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.952Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:43.961Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:44.858Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.052Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.105Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.119Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.137Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.147Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:28:45.183Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:28:41.611Z",
|
||||
"blocked_requests": 12,
|
||||
"finished_at": "2026-10-13T20:28:45.914Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"id": "f76f4662-03cd-4596-928e-ee34eeaa4057",
|
||||
"api_version": "1",
|
||||
"status": "passed",
|
||||
"error": null,
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:29:17.318Z",
|
||||
"expires_at": "2026-10-13T21:29:17.318Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"completed_steps": 14,
|
||||
"steps": [
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.494Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.657Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.745Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.760Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.797Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.802Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "desktop",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:19.809Z"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 0,
|
||||
"action": "visible",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.740Z",
|
||||
"selector": "h1"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 1,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.862Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 2,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.930Z",
|
||||
"key": "Tab"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 3,
|
||||
"action": "focused",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.958Z",
|
||||
"selector": "#increment"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 4,
|
||||
"action": "press",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.974Z",
|
||||
"key": "Enter"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 5,
|
||||
"action": "text",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.981Z",
|
||||
"selector": "#count"
|
||||
},
|
||||
{
|
||||
"viewport": "narrow",
|
||||
"index": 6,
|
||||
"action": "screenshot",
|
||||
"status": "passed",
|
||||
"at": "2026-10-13T20:29:20.989Z"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280
|
||||
}
|
||||
],
|
||||
"started_at": "2026-10-13T20:29:17.319Z",
|
||||
"blocked_requests": 12,
|
||||
"finished_at": "2026-10-13T20:29:21.401Z",
|
||||
"downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"available": true,
|
||||
"error": null,
|
||||
"checked_at": "2026-10-13T20:28:18.565Z",
|
||||
"browser_version": "156.0.8078.4",
|
||||
"api_version": "1",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"allowed_origins": [
|
||||
"https://onco.cc"
|
||||
],
|
||||
"fixture": true,
|
||||
"active": 0,
|
||||
"queued": 0,
|
||||
"limits": {
|
||||
"concurrency": 1,
|
||||
"max_queue": 4,
|
||||
"max_timeout_ms": 120000,
|
||||
"retention_ms": 3600000,
|
||||
"max_runs": 64
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"id": "74e433ef-aee7-4c74-afe5-3171214f4014",
|
||||
"api_version": "1",
|
||||
"status": "interrupted",
|
||||
"error": "service_restarted",
|
||||
"label": "independent-browser-trial",
|
||||
"target": "fixture",
|
||||
"viewports": [
|
||||
{
|
||||
"name": "desktop",
|
||||
"width": 1280,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"name": "narrow",
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"timeout_ms": 60000,
|
||||
"created_at": "2026-10-13T20:28:46.380Z",
|
||||
"expires_at": "2026-10-13T21:28:46.380Z",
|
||||
"service_version": "0.1.0",
|
||||
"playwright_version": "1.64.0",
|
||||
"release": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"execution_location": "railiance01/playwright",
|
||||
"browser_version": null,
|
||||
"completed_steps": 0,
|
||||
"steps": [],
|
||||
"artifacts": [],
|
||||
"finished_at": "2026-10-13T20:28:51.157Z",
|
||||
"downloads": []
|
||||
}
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
{
|
||||
"started_at": "2026-10-13T20:28:36.758090+00:00",
|
||||
"status": "passed",
|
||||
"authority": "ADMINISTER @ realm:kubernetes/railiance01",
|
||||
"activation": "APPROVED",
|
||||
"approval_basis": "User approved remaining recovery verification: Ok, lets do that (2026-10-13)",
|
||||
"identity_acceptance": false,
|
||||
"cross_version_rollback": false,
|
||||
"checks": [
|
||||
"baseline_trial",
|
||||
"guarded_sigkill_of_active_service",
|
||||
"interrupted_jobs_and_orphan_cleanup",
|
||||
"completed_artifact_hashes_preserved",
|
||||
"fresh_trial_after_container_restart"
|
||||
],
|
||||
"before": {
|
||||
"pod": "playwright-c5b9c8bd7-fbfqv",
|
||||
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
|
||||
"restart_count": 0,
|
||||
"container_id": "containerd://13ed9eda6b44bb2768835968691a63c397cda96a3aaf1189ba184987e79e0dce",
|
||||
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
|
||||
"last_exit_code": null
|
||||
},
|
||||
"crash": {
|
||||
"signal": "SIGKILL",
|
||||
"pid": 7,
|
||||
"active_status": "running",
|
||||
"queued_status": "queued",
|
||||
"partial_png_bytes": 25740
|
||||
},
|
||||
"after": {
|
||||
"pod": "playwright-c5b9c8bd7-fbfqv",
|
||||
"pod_uid": "a22030a4-c12d-4e97-be8d-98a8d80808c2",
|
||||
"restart_count": 1,
|
||||
"container_id": "containerd://abd1f7081045b1b4505e8045bf298e0a10485834d760d6dbf8db73accdb3da08",
|
||||
"image_id": "forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441",
|
||||
"pvc_uid": "da36929a-1d69-457a-ad99-7580ef86ab18",
|
||||
"last_exit_code": 137
|
||||
},
|
||||
"completed_downloads": [
|
||||
{
|
||||
"name": "desktop-6-keyboard-result.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "desktop-final.png",
|
||||
"bytes": 26622,
|
||||
"sha256": "b15998824256b86c74f2bbb75f061061be65db0b4f3bb5444529ac69a723cdad"
|
||||
},
|
||||
{
|
||||
"name": "narrow-6-keyboard-result.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
},
|
||||
{
|
||||
"name": "narrow-final.png",
|
||||
"bytes": 24280,
|
||||
"sha256": "49d27cb169b8e180ce93969c6d2b8619a7684cf035bb9dcae9cd91d8bd9952e8"
|
||||
}
|
||||
],
|
||||
"finished_at": "2026-10-13T20:29:20.202087+00:00"
|
||||
}
|
||||
|
|
@ -111,3 +111,33 @@ Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream
|
|||
termination and cross-version rollback require their separate evidence. The
|
||||
command's local validation is retained in
|
||||
[evidence](evidence/2026-10-13-consumer-verifier.json).
|
||||
|
||||
|
||||
## Abrupt interruption verification
|
||||
|
||||
`tools/verify_crash_recovery.py --image <published-immutable-reference> --out
|
||||
.local/new-crash-check` creates its own bounded, read-only Docker container and
|
||||
private data directory. It verifies a completed trial, captures a partial PNG in
|
||||
a real active job, queues another job, sends SIGKILL to the container, and starts
|
||||
it on the same data. It requires interrupted receipts, orphan cleanup, preserved
|
||||
completed hashes and a fresh trial; it removes only its own container. Run it
|
||||
as workstation UID 1000 to match the released image's data-directory ownership.
|
||||
|
||||
The separately approved managed check is `tools/verify_managed_recovery.py`.
|
||||
It is an ADMINISTER operation through existing SSH, never a consumer permission.
|
||||
Use `--url <operator-loopback-forward> --image <published-reference> --out
|
||||
.local/new-managed-crash --approval-reference <specific-founder-approval>
|
||||
--execute-crash`. Merely supplying that flag/reference does not grant approval.
|
||||
The tool pins railiance01's node address, one Ready pilot pod and its release,
|
||||
requires an idle service, creates its own jobs, and guards the supervisor SIGKILL
|
||||
on the expected active/queued jobs and partial screenshot. It verifies container
|
||||
exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files,
|
||||
completed hashes and a fresh trial. No replicas or manifests are changed.
|
||||
|
||||
The port-forward may exit when the container restarts. Re-establish the dedicated
|
||||
operator connection after the container is Ready; the verification waits up to
|
||||
180 seconds for it. Do not retry the crash command solely because the tunnel
|
||||
was lost: inspect the existing verification process and pod restart state first.
|
||||
Keep every `run.json` and the local data directory private. Share only selected
|
||||
receipts under `docs/evidence`. These tools do not prove real consumer identity,
|
||||
revocation or cross-version rollback. [Executed local and managed evidence](evidence/2026-10-13-crash-recovery.md).
|
||||
|
|
|
|||
37
history/2026-10-13-2231-intent-scope-assessment.md
Normal file
37
history/2026-10-13-2231-intent-scope-assessment.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# INTENT against SCOPE — 2026-10-13 22:31 Europe/Berlin
|
||||
|
||||
The abrupt-interruption gap is now closed with real local and managed execution.
|
||||
Published release 0.1.0 recovered after SIGKILL during an actual browser job;
|
||||
a queued job was also preserved as interrupted. Partial artifacts were removed,
|
||||
completed hashes survived, and a fresh 14-step screenshot/keyboard trial passed.
|
||||
Managed recovery used the same pod, PVC and image with a new container. See
|
||||
[execution evidence](../docs/evidence/2026-10-13-crash-recovery.md).
|
||||
|
||||
Most important remaining gaps:
|
||||
|
||||
1. **Real self-service human and coding-agent access.** Identity registration,
|
||||
governed agent credentials, native API authentication and per-caller transport
|
||||
still need owner activation and native acceptance receipts. The completed
|
||||
recovery check used an existing operator identity. It does not satisfy this
|
||||
gap. RPF-WP-0056-T03 owns delivery; CUST-WP-0080-T05 coordinates it. Membership
|
||||
is decided, so no additional eligibility question is needed.
|
||||
2. **Cross-version rollback.** This is the first release. A same-image restart
|
||||
proves crash recovery, not a version transition. Exercise actual rollback
|
||||
when a second reviewed compatible release exists; retain the open requirement.
|
||||
3. **Final managed availability and discovery.** A5 local-container guidance is
|
||||
published. Real consumer receipts must precede final readiness and managed
|
||||
discovery promotion under WP-0004-T01/T03.
|
||||
4. **Consumer-specific acceptance.** Content-explorer owns its broader route,
|
||||
keyboard and visual requirements. Existing provider evidence cannot close
|
||||
that consumer task.
|
||||
|
||||
WP-0001–0003 remain finished. WP-0004-T02 is done; T01/T03 retain their narrower
|
||||
external waits. SCOPE now includes the managed abrupt-recovery proof. The repo
|
||||
contains repeatable consumer API and crash-recovery commands and sanitized
|
||||
receipts. The managed pilot is Ready; temporary verification connections were
|
||||
stopped. No runtime code, image version, grants or shared controls changed.
|
||||
|
||||
Validation: three unit tests and syntax checks passed; both real released-image
|
||||
crash checks passed. An initial managed transport failure stopped before the
|
||||
crash step and is retained explicitly. The overall implementation is not complete
|
||||
until the outstanding consumer-access and acceptance requirements are satisfied.
|
||||
178
tools/verify_crash_recovery.py
Normal file
178
tools/verify_crash_recovery.py
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Kill an isolated released Docker container during a real browser job and verify recovery."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'client'))
|
||||
from rapp_playwright import Client, ClientError, collect, save
|
||||
|
||||
|
||||
def require(ok, message):
|
||||
if not ok:
|
||||
raise ClientError(message)
|
||||
|
||||
|
||||
def docker(*args):
|
||||
result = subprocess.run(['docker', *args], capture_output=True, text=True, timeout=90)
|
||||
if result.returncode:
|
||||
raise ClientError('docker_' + args[0] + '_failed (raw output suppressed)')
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def poll(check, description, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
value = check()
|
||||
if value:
|
||||
return value
|
||||
time.sleep(0.1)
|
||||
raise ClientError('timeout: ' + description)
|
||||
|
||||
|
||||
def verify(args):
|
||||
require(bool(re.fullmatch(r'forgejo\.coulomb\.social/coulomb/rapp-playwright@sha256:[a-f0-9]{64}', args.image)),
|
||||
'published immutable image reference required')
|
||||
root = args.out.resolve()
|
||||
root.mkdir(parents=True, mode=0o700, exist_ok=False)
|
||||
data = root / 'data'
|
||||
data.mkdir(mode=0o700)
|
||||
name = 'rapp-crash-check-' + uuid.uuid4().hex[:12]
|
||||
cid = None
|
||||
report = {'started_at': datetime.now(timezone.utc).isoformat(), 'image': args.image,
|
||||
'location': 'isolated-local-docker', 'signal': 'SIGKILL', 'checks': [],
|
||||
'status': 'failed', 'managed_recovery_proven': False}
|
||||
try:
|
||||
cid = docker('run', '-d', '--name', name, '--read-only', '--user', '1000:1000',
|
||||
'--tmpfs', '/tmp:rw,size=512m', '--shm-size=256m', '--memory=1g', '--cpus=1',
|
||||
'--cap-drop', 'ALL', '--security-opt', 'no-new-privileges',
|
||||
'--mount', 'type=bind,src=' + str(data) + ',dst=/data',
|
||||
'-p', '127.0.0.1::8080', '-e', 'HOST=0.0.0.0',
|
||||
'-e', 'RELEASE_ID=' + args.image,
|
||||
'-e', 'EXECUTION_LOCATION=isolated-crash-verification', args.image)
|
||||
require(bool(re.fullmatch('[a-f0-9]{64}', cid)), 'unexpected container identity')
|
||||
|
||||
def connect():
|
||||
binding = json.loads(docker('inspect', '--format', '{{json .NetworkSettings.Ports}}', cid))['8080/tcp'][0]
|
||||
require(binding['HostIp'] == '127.0.0.1', 'non_loopback_binding')
|
||||
return Client('http://127.0.0.1:' + binding['HostPort'])
|
||||
|
||||
def ready(client):
|
||||
def probe():
|
||||
try:
|
||||
result = client.request('/v1/preflight')
|
||||
return result if result.get('available') else None
|
||||
except ClientError:
|
||||
return None
|
||||
result = poll(probe, 'browser readiness')
|
||||
require(result['release'] == args.image, 'unexpected release')
|
||||
return result
|
||||
|
||||
client = connect()
|
||||
save(root / 'before-preflight.json', ready(client))
|
||||
spec = client.request('/v1/trial')
|
||||
|
||||
def submit(specification, folder):
|
||||
folder.mkdir(mode=0o700)
|
||||
run = client.request('/v1/runs', 'POST', specification)
|
||||
save(folder / 'run.json', run)
|
||||
return run
|
||||
|
||||
def terminal(run):
|
||||
def check():
|
||||
result = client.request('/v1/runs/' + run['id'], token=run['token'])
|
||||
return result if result['status'] in ('passed', 'failed', 'cancelled', 'interrupted') else None
|
||||
return poll(check, 'terminal run')
|
||||
|
||||
completed = submit(spec, root / 'completed-before')
|
||||
require(terminal(completed)['status'] == 'passed', 'baseline trial failed')
|
||||
before = collect(client, completed, root / 'completed-before')
|
||||
require(len(before['downloads']) == 4, 'baseline screenshot count')
|
||||
report['checks'].append('completed_trial_and_four_downloads')
|
||||
slow = {**spec, 'viewports': [{'name': 'desktop', 'width': 1280, 'height': 800}],
|
||||
'steps': [{'action': 'screenshot', 'name': 'uncommitted'},
|
||||
{'action': 'visible', 'selector': '#never-present-crash-test'}]}
|
||||
active = submit(slow, root / 'active')
|
||||
partial = data / active['id'] / 'desktop-0-uncommitted.png'
|
||||
|
||||
def screenshot_in_flight():
|
||||
result = client.request('/v1/runs/' + active['id'], token=active['token'])
|
||||
require(result['status'] not in ('passed', 'failed', 'cancelled', 'interrupted'), 'active job ended before kill')
|
||||
return result['status'] == 'running' and partial.is_file() and partial.stat().st_size > 0
|
||||
|
||||
poll(screenshot_in_flight, 'in-flight browser screenshot', timeout=30)
|
||||
queued = submit(spec, root / 'queued')
|
||||
require(client.request('/v1/runs/' + queued['id'], token=queued['token'])['status'] == 'queued', 'expected queued job')
|
||||
# Read actual persisted state without exposing its capability hashes.
|
||||
report['persisted_before_kill'] = {
|
||||
'active': json.loads((data / active['id'] / 'receipt.json').read_text())['status'],
|
||||
'queued': json.loads((data / queued['id'] / 'receipt.json').read_text())['status'],
|
||||
'partial_png_bytes': partial.stat().st_size}
|
||||
require(report['persisted_before_kill']['active'] == 'running', 'active persistence missing')
|
||||
require(report['persisted_before_kill']['queued'] == 'queued', 'queued persistence missing')
|
||||
docker('kill', '--signal=KILL', cid)
|
||||
state = json.loads(docker('inspect', '--format', '{{json .State}}', cid))
|
||||
require(not state['Running'] and state['ExitCode'] == 137 and not state['OOMKilled'], 'SIGKILL not established')
|
||||
report['exit_code_after_kill'] = state['ExitCode']
|
||||
report['checks'].append('real_active_and_queued_jobs_killed_without_shutdown')
|
||||
docker('start', cid)
|
||||
client = connect()
|
||||
save(root / 'after-preflight.json', ready(client))
|
||||
for label, run in [('active', active), ('queued', queued)]:
|
||||
result = collect(client, run, root / label)
|
||||
require(result['status'] == 'interrupted' and result['error'] == 'service_restarted', label + '_not_interrupted')
|
||||
require(result['artifacts'] == [] and result['downloads'] == [], label + '_artifacts_exposed')
|
||||
require(sorted(p.name for p in (data / run['id']).iterdir()) == ['receipt.json'], label + '_orphan_files')
|
||||
report['checks'].append('active_and_queued_interrupted_orphans_removed')
|
||||
try:
|
||||
client.request('/v1/runs/' + active['id'] + '/artifacts/desktop-0-uncommitted.png', token=active['token'], binary=True)
|
||||
except ClientError as exc:
|
||||
require(str(exc) == 'HTTP 404: not_found', 'unexpected orphan denial')
|
||||
else:
|
||||
raise ClientError('uncommitted artifact still downloadable')
|
||||
after_dir = root / 'completed-after'
|
||||
after_dir.mkdir(mode=0o700)
|
||||
after = collect(client, completed, after_dir)
|
||||
require(after['status'] == 'passed' and before['downloads'] == after['downloads'], 'completed artifact mismatch')
|
||||
report['completed_downloads'] = after['downloads']
|
||||
report['checks'].append('completed_receipt_and_hashes_preserved')
|
||||
fresh = submit(spec, root / 'new-trial')
|
||||
require(terminal(fresh)['status'] == 'passed', 'fresh trial failed after restart')
|
||||
fresh_receipt = collect(client, fresh, root / 'new-trial')
|
||||
require(fresh_receipt['completed_steps'] == 14 and len(fresh_receipt['downloads']) == 4, 'fresh trial incomplete')
|
||||
report['checks'].append('fresh_keyboard_screenshot_trial_after_recovery')
|
||||
report['status'] = 'passed'
|
||||
except KeyboardInterrupt:
|
||||
report['error'] = 'interrupted_by_caller'
|
||||
except (ClientError, OSError, ValueError, KeyError, subprocess.TimeoutExpired) as exc:
|
||||
report['error'] = str(exc)
|
||||
finally:
|
||||
if cid:
|
||||
try:
|
||||
docker('rm', '-f', cid)
|
||||
report['test_container_removed'] = True
|
||||
except ClientError:
|
||||
report['test_container_removed'] = False
|
||||
report['cleanup_container_name'] = name
|
||||
report['status'] = 'failed'
|
||||
report['finished_at'] = datetime.now(timezone.utc).isoformat()
|
||||
save(root / 'verification.json', report)
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0 if report['status'] == 'passed' else 3
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--image', required=True)
|
||||
parser.add_argument('--out', required=True, type=Path)
|
||||
try:
|
||||
sys.exit(verify(parser.parse_args()))
|
||||
except (ClientError, OSError, ValueError) as exc:
|
||||
print(json.dumps({'error': str(exc)}), file=sys.stderr)
|
||||
sys.exit(3)
|
||||
181
tools/verify_managed_recovery.py
Normal file
181
tools/verify_managed_recovery.py
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Approved operator-only crash check for the fixed railiance01 Playwright pilot.
|
||||
|
||||
Requires --execute-crash; no identity acceptance or cross-version rollback is implied.
|
||||
"""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'client'))
|
||||
from rapp_playwright import Client, ClientError, collect, save
|
||||
from verify_crash_recovery import poll, require
|
||||
|
||||
|
||||
def remote(*args):
|
||||
r = subprocess.run(['ssh', 'railiance01', shlex.join(list(args))], capture_output=True, text=True, timeout=30)
|
||||
require(r.returncode == 0, 'remote check failed; raw output suppressed')
|
||||
return json.loads(r.stdout)
|
||||
|
||||
|
||||
def snapshot(expected):
|
||||
nodes = remote('kubectl', 'get', 'nodes', '-o', 'json')['items']
|
||||
require(any(a['address'] == '92.205.62.239' for n in nodes for a in n['status']['addresses']), 'wrong node')
|
||||
pods = remote('kubectl', '-n', 'playwright', 'get', 'pods', '-l', 'app.kubernetes.io/name=playwright', '-o', 'json')['items']
|
||||
require(len(pods) == 1, 'expected exactly one pilot pod')
|
||||
p = pods[0]
|
||||
c = next(c for c in p['status']['containerStatuses'] if c['name'] == 'service')
|
||||
require(c['ready'] and c['imageID'] == expected, 'pilot not ready on expected image')
|
||||
pvc = remote('kubectl', '-n', 'playwright', 'get', 'pvc', 'playwright-artifacts', '-o', 'json')
|
||||
require(pvc['status']['phase'] == 'Bound', 'PVC not bound')
|
||||
return {'pod': p['metadata']['name'], 'pod_uid': p['metadata']['uid'], 'restart_count': c['restartCount'],
|
||||
'container_id': c['containerID'], 'image_id': c['imageID'], 'pvc_uid': pvc['metadata']['uid'],
|
||||
'last_exit_code': c.get('lastState', {}).get('terminated', {}).get('exitCode')}
|
||||
|
||||
|
||||
def verify(args):
|
||||
require(args.execute_crash, 'explicit --execute-crash is required after owner/founder approval')
|
||||
args.out.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
report = {'started_at': datetime.now(timezone.utc).isoformat(), 'status': 'failed',
|
||||
'authority': 'ADMINISTER @ realm:kubernetes/railiance01', 'activation': 'APPROVED',
|
||||
'approval_basis': args.approval_reference, 'identity_acceptance': False,
|
||||
'cross_version_rollback': False, 'checks': []}
|
||||
client = Client(args.url)
|
||||
runs = []
|
||||
try:
|
||||
before = snapshot(args.image)
|
||||
report['before'] = before
|
||||
ready = client.request('/v1/preflight')
|
||||
require(ready['available'] and ready['release'] == args.image and ready['execution_location'] == 'railiance01/playwright', 'provider mismatch')
|
||||
require(ready['active'] == 0 and ready['queued'] == 0, 'pilot busy: do not interrupt unrelated jobs')
|
||||
save(args.out / 'preflight.json', ready)
|
||||
spec = client.request('/v1/trial')
|
||||
|
||||
def submit(specification, label):
|
||||
folder = args.out / label
|
||||
folder.mkdir(mode=0o700)
|
||||
run = client.request('/v1/runs', 'POST', specification)
|
||||
runs.append(run)
|
||||
save(folder / 'run.json', run)
|
||||
return run, folder
|
||||
|
||||
def finish(run):
|
||||
def check():
|
||||
r = client.request('/v1/runs/' + run['id'], token=run['token'])
|
||||
return r if r['status'] in ('passed', 'failed', 'cancelled', 'interrupted') else None
|
||||
return poll(check, 'job completion')
|
||||
|
||||
completed, folder = submit(spec, 'completed-before')
|
||||
require(finish(completed)['status'] == 'passed', 'baseline trial failed')
|
||||
original = collect(client, completed, folder)
|
||||
require(len(original['downloads']) == 4, 'four baseline screenshots required')
|
||||
report['checks'].append('baseline_trial')
|
||||
slow = {**spec, 'viewports': [{'name': 'desktop', 'width': 1280, 'height': 800}],
|
||||
'steps': [{'action': 'screenshot', 'name': 'uncommitted'},
|
||||
{'action': 'visible', 'selector': '#never-present-crash-test'}]}
|
||||
active, active_dir = submit(slow, 'active')
|
||||
queued, queued_dir = submit(spec, 'queued')
|
||||
# Run the guard and signal together on the pod to avoid a multi-command race.
|
||||
ids = json.dumps([active['id'], queued['id']])
|
||||
code = r'''const fs=require('fs');
|
||||
(async()=>{
|
||||
const ids=IDS;
|
||||
const a=JSON.parse(fs.readFileSync('/data/'+ids[0]+'/receipt.json'));
|
||||
const q=JSON.parse(fs.readFileSync('/data/'+ids[1]+'/receipt.json'));
|
||||
if(a.status!=='running'||q.status!=='queued')throw Error('test jobs not active/queued');
|
||||
const partial='/data/'+ids[0]+'/desktop-0-uncommitted.png';
|
||||
if(!fs.existsSync(partial)){console.log(JSON.stringify({waiting:true}));return;}
|
||||
const pf=await (await fetch('http://127.0.0.1:8080/v1/preflight',{headers:{'X-Rapp-Protocol':'1'}})).json();
|
||||
if(pf.active!==1||pf.queued!==1||pf.release!==IMAGE)throw Error('unexpected concurrent work or release');
|
||||
const servers=fs.readdirSync('/proc').filter(x=>/^[0-9]+$/.test(x)).filter(x=>{
|
||||
try{const c=fs.readFileSync('/proc/'+x+'/cmdline','utf8').split(String.fromCharCode(0));return c[0]==='node'&&c[1]==='src/server.mjs';}catch{return false;}});
|
||||
if(servers.length!==1)throw Error('expected one service supervisor');
|
||||
const result={signal:'SIGKILL',pid:Number(servers[0]),active_status:a.status,queued_status:q.status,partial_png_bytes:fs.statSync(partial).size};
|
||||
fs.writeSync(1,JSON.stringify(result));process.kill(Number(servers[0]),'SIGKILL');
|
||||
})().catch(()=>{process.exitCode=2;});'''.replace('IDS', ids).replace('IMAGE', json.dumps(args.image))
|
||||
|
||||
def crash_when_ready():
|
||||
r = subprocess.run(['ssh', 'railiance01', shlex.join(['kubectl', '-n', 'playwright', 'exec', before['pod'], '--', 'node', '-e', code])],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
# Killing the supervisor also terminates exec; consume only our JSON receipt.
|
||||
try:
|
||||
result = json.loads(r.stdout)
|
||||
except ValueError:
|
||||
raise ClientError('crash guard failed; raw output suppressed') from None
|
||||
if result.get('waiting'):
|
||||
require(r.returncode == 0, 'crash guard failed while waiting')
|
||||
return None
|
||||
require(result.get('signal') == 'SIGKILL', 'crash signal receipt absent')
|
||||
return result
|
||||
|
||||
report['crash'] = poll(crash_when_ready, 'real in-flight screenshot and guarded crash', timeout=30)
|
||||
report['checks'].append('guarded_sigkill_of_active_service')
|
||||
# Existing port-forward may end on container restart; the caller restores it.
|
||||
def recovered():
|
||||
try:
|
||||
pf = client.request('/v1/preflight')
|
||||
return pf if pf.get('available') and pf.get('release') == args.image else None
|
||||
except ClientError:
|
||||
return None
|
||||
save(args.out / 'after-preflight.json', poll(recovered, 'restore consumer tunnel after container restart', timeout=180))
|
||||
def ready_snapshot():
|
||||
try:
|
||||
return snapshot(args.image)
|
||||
except ClientError:
|
||||
return None
|
||||
after = poll(ready_snapshot, 'ready container after recovery', timeout=60)
|
||||
report['after'] = after
|
||||
require(after['last_exit_code'] == 137, 'expected SIGKILL termination code')
|
||||
require(after['pod_uid'] == before['pod_uid'] and after['pvc_uid'] == before['pvc_uid'], 'pod or storage changed')
|
||||
require(after['restart_count'] > before['restart_count'] and after['container_id'] != before['container_id'], 'container restart not observed')
|
||||
for run, folder in [(active, active_dir), (queued, queued_dir)]:
|
||||
r = collect(client, run, folder)
|
||||
require(r['status'] == 'interrupted' and r['error'] == 'service_restarted' and r['artifacts'] == [], 'interruption receipt mismatch')
|
||||
check = "const fs=require('fs');console.log(JSON.stringify(" + ids + ".map(id=>fs.readdirSync('/data/'+id))))"
|
||||
files = remote('kubectl', '-n', 'playwright', 'exec', before['pod'], '--', 'node', '-e', check)
|
||||
require(files == [['receipt.json'], ['receipt.json']], 'orphan artifacts remain')
|
||||
report['checks'].append('interrupted_jobs_and_orphan_cleanup')
|
||||
folder = args.out / 'completed-after'
|
||||
folder.mkdir(mode=0o700)
|
||||
restored = collect(client, completed, folder)
|
||||
require(restored['status'] == 'passed' and restored['downloads'] == original['downloads'], 'completed artifacts changed')
|
||||
report['completed_downloads'] = restored['downloads']
|
||||
report['checks'].append('completed_artifact_hashes_preserved')
|
||||
fresh, folder = submit(spec, 'new-trial')
|
||||
require(finish(fresh)['status'] == 'passed', 'new trial failed')
|
||||
r = collect(client, fresh, folder)
|
||||
require(r['completed_steps'] == 14 and len(r['downloads']) == 4, 'new trial incomplete')
|
||||
report['checks'].append('fresh_trial_after_container_restart')
|
||||
report['status'] = 'passed'
|
||||
except (ClientError, OSError, ValueError, KeyError, subprocess.TimeoutExpired) as exc:
|
||||
report['error'] = str(exc)
|
||||
except KeyboardInterrupt:
|
||||
report['error'] = 'interrupted_by_caller'
|
||||
finally:
|
||||
for run in runs:
|
||||
try:
|
||||
client.request('/v1/runs/' + run['id'], 'DELETE', token=run['token'])
|
||||
except ClientError:
|
||||
pass
|
||||
report['finished_at'] = datetime.now(timezone.utc).isoformat()
|
||||
save(args.out / 'verification.json', report)
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0 if report['status'] == 'passed' else 3
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--url', required=True)
|
||||
p.add_argument('--image', required=True)
|
||||
p.add_argument('--out', type=Path, required=True)
|
||||
p.add_argument('--approval-reference', required=True)
|
||||
p.add_argument('--execute-crash', action='store_true')
|
||||
try:
|
||||
sys.exit(verify(p.parse_args()))
|
||||
except (ClientError, OSError, ValueError) as exc:
|
||||
print(json.dumps({'error': str(exc)}), file=sys.stderr)
|
||||
sys.exit(3)
|
||||
|
|
@ -165,3 +165,20 @@ cancellation. Wrong release and wrong execution location each fail before job
|
|||
submission. Retained [verification](../docs/evidence/2026-10-13-consumer-verifier.json).
|
||||
The command does not establish authentication, restart workloads or close T01;
|
||||
pair it with native identity and permission receipts plus remaining recovery tests.
|
||||
|
||||
|
||||
## Abrupt recovery passed — 2026-10-13 22:29 Europe/Berlin
|
||||
|
||||
User authorized the remaining recovery verification. Added and executed isolated
|
||||
released-container and approved managed-pilot crash verifiers. A real screenshot
|
||||
was present in an active job and another job was queued before SIGKILL. Managed
|
||||
container exited137 and restarted on the same pod/PVC/image; both runs became
|
||||
interrupted, partial files disappeared, completed PNG hashes matched and a fresh
|
||||
14-step/four-PNG trial passed. Dedicated test connections were stopped. See
|
||||
[executed recovery evidence](../docs/evidence/2026-10-13-crash-recovery.md).
|
||||
|
||||
The earlier abrupt-interruption gap is resolved. T01 remains wait for actual
|
||||
human/agent identity/transport receipts and cross-version rollback; no second
|
||||
release was invented solely to claim rollback. T03 remains dependent on managed
|
||||
consumer acceptance. No new identity-owner implementation return was available
|
||||
at this execution; CUST-WP-0080-T05 retains coordination.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue