rapp-playwright/docs/operations.md

Service operations

Build with make install test build. make test-integration launches real Chromium inside the pinned upstream image; it needs Docker, not a host browser. Publish the tested image through the existing Forgejo OCI credential lane and read its digest back. make render IMAGE_REFERENCE=forgejo.coulomb.social/coulomb/rapp-playwright@sha256:… rejects floating tags. Never record a placeholder as a published digest.

The initial package uses a single replica, Recreate updates, 100m CPU/256Mi requests, 1 CPU/1Gi limits, and a 2Gi local-path PVC. Artifacts expire after one hour; they are transient evidence, not a backup service. Consumers download what they need to retain. Recreate avoids a surge on the capacity-constrained single-node cluster and preserves the RWO volume.

Before activation, confirm the node identity and current resource headroom, platform lifecycle ownership/admission, publication/pull access, and private consumer connectivity. Run make server-dry-run IMAGE_REFERENCE=…, inspect kubectl diff -f .rendered/workload.yaml on railiance01, then deploy the reviewed manifest with make deploy IMAGE_REFERENCE=…. No existing workload or policy needs relaxation. Platform confirmation is tracked in WP-0004-T01.

For the first installation, after admission, run make bootstrap-namespace IMAGE_REFERENCE=… before the full server dry run. Kubernetes does not persist a dry-run Namespace, so namespace-scoped objects cannot be server-validated until the namespace exists. This bootstrap dry-runs and diffs only the reviewed Namespace before creating it. The full deployment also diffs before applying.

The playwright-consumer Role grants only pod discovery and port-forward. Bind an approved existing user/group through the identity/platform owner. This repo does not mint credentials or silently bind all authenticated users. Callers with existing authorized Kubernetes access can use the documented port-forward. The pod has no service-account token and no cluster permissions.

Readiness performs the authenticated-protocol preflight over pod loopback. It requires a successful browser launch; liveness checks the process only. The pod uses non-root UID/GID 1000, read-only root, dropped capabilities and RuntimeDefault seccomp. Tini reaps orphaned browser processes. Chromium's OS sandbox is disabled for this trusted-target pilot; it is not approved for hostile websites or mutually untrusted tenants. The HTTP/CONNECT origin gate, private-address checks, blocked workers/WebSockets, per-run process/context and resource limits cover the documented application-level boundary, not browser-kernel exploits.

Verify and recover

After establishing the consumer tunnel, make verify-live TRIAL_OUT=.local/live-N must produce four PNGs, focus/keyboard assertions and a receipt. Check the live container image and imageID against the published digest, PVC binding, deny-all ingress and ready replica. Exercise wrong run token, deliberate UI failure, unreachable target, cancellation and service restart; a passing health endpoint alone is insufficient.

After restart, accepted nonterminal runs become interrupted; completed runs remain retrievable until expiry. A client disconnect does not cancel an accepted job, so all runs have enforced deadlines. Supervisor cancellation closes the browser; after two seconds it kills the recorded worker/browser process groups. Service shutdown drains/cancels bounded work. Corrupt receipts cause startup to fail closed; restore or quarantine the affected transient run directory through the owner, without exposing capability hashes or screenshots in logs.

Rollback: retain the previous immutable image reference and rendered revision, review its schema compatibility, then run make rollback IMAGE_REFERENCE=…. Verify the replacement's imageID and repeat the trial. For the first release, test recovery by reapplying the same verified digest; do not call that proof of cross-version rollback. Once a second release exists, exercise an actual digest change and return to the reviewed release. No automated rollback migrates or deletes the PVC. V1 receipts are JSON; future incompatible formats require an explicit migration/rollback decision.

Upgrade Playwright and its base image together, update the lockfile, repeat the unit/browser/client tests and consumer pilot, then publish a new immutable image. The Python client checks protocol v1 and does not install Playwright.

Published release

Release 0.1.0 is published and pull-verified. Use its immutable reference for rendering; registry publication is no longer blocked. The cluster-owned ingress now allows a bounded 600-second HTTPS request body; see cause, rollout and verification.

For subsequent publications, fetch the remote index with docker buildx imagetools inspect --raw <image-reference> and hash the exact returned bytes. Require the expected digest, then pull that immutable reference with sudo k3s crictl pull <image-reference> on railiance01 and verify its recorded repo digest. A successful upload alone does not establish node pull access.

Repeatable consumer API acceptance

After Platform establishes and verifies the caller's own identity, permission boundary and port-forward, run this from the provider checkout with Python 3.10+:

python3 tools/verify_consumer.py --url http://127.0.0.1:18081 \
  --expected-release forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441 \
  --expected-location railiance01/playwright --out .local/consumer-acceptance-01

Use a fresh output directory for each caller/run. Exit 0 means browser API checks passed; exit 3 means verification failed. verification.json lists checks and errors. The command verifies exact release/location, desktop/narrow PNG dimensions and hashed downloads, keyboard assertions, foreign-token receipt/artifact denial, failed-assertion evidence, private-target refusal and running-job cancellation. It submits only fixture jobs and one denied private-target request, cancels its own runs on exit and does not restart the service or alter access grants.

Keep all run.json files private. Pair the result with Platform's native consumer identity/permission receipt for each real human and registered coding agent. The result deliberately says identity_authentication_verified: false: neither operator impersonation nor this browser check alone proves self-service access. Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream termination and cross-version rollback require their separate evidence. The command's local validation is retained in evidence.

Abrupt interruption verification

tools/verify_crash_recovery.py --image <published-immutable-reference> --out .local/new-crash-check creates its own bounded, read-only Docker container and private data directory. It verifies a completed trial, captures a partial PNG in a real active job, queues another job, sends SIGKILL to the container, and starts it on the same data. It requires interrupted receipts, orphan cleanup, preserved completed hashes and a fresh trial; it removes only its own container. Run it as workstation UID 1000 to match the released image's data-directory ownership.

The separately approved managed check is tools/verify_managed_recovery.py. It is an ADMINISTER operation through existing SSH, never a consumer permission. Use --url <operator-loopback-forward> --image <published-reference> --out .local/new-managed-crash --approval-reference <specific-founder-approval> --execute-crash. Merely supplying that flag/reference does not grant approval. The tool pins railiance01's node address, one Ready pilot pod and its release, requires an idle service, creates its own jobs, and guards the supervisor SIGKILL on the expected active/queued jobs and partial screenshot. It verifies container exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files, completed hashes and a fresh trial. No replicas or manifests are changed.

The port-forward may exit when the container restarts. Re-establish the dedicated operator connection after the container is Ready; the verification waits up to 180 seconds for it. Do not retry the crash command solely because the tunnel was lost: inspect the existing verification process and pod restart state first. Keep every run.json and the local data directory private. Share only selected receipts under docs/evidence. These tools do not prove real consumer identity, revocation or cross-version rollback. Executed local and managed evidence.