rapp-playwright/docs/operations.md

# Service operations

Build with `make install test build`. `make test-integration` launches real Chromium
inside the pinned upstream image; it needs Docker, not a host browser. Publish
the tested image through the existing Forgejo OCI credential lane and read its
digest back. `make render IMAGE_REFERENCE=forgejo.coulomb.social/coulomb/rapp-playwright@sha256:…`
rejects floating tags. Never record a placeholder as a published digest.

The initial package uses a single replica, Recreate updates, 100m CPU/256Mi
requests, 1 CPU/1Gi limits, and a 2Gi local-path PVC. Artifacts expire after one
hour; they are transient evidence, not a backup service. Consumers download
what they need to retain. Recreate avoids a surge on the capacity-constrained
single-node cluster and preserves the RWO volume.

Before activation, confirm the node identity and current resource headroom,
platform lifecycle ownership/admission, publication/pull access, and private
consumer connectivity. Run `make server-dry-run IMAGE_REFERENCE=…`, inspect
`kubectl diff -f .rendered/workload.yaml` on railiance01, then deploy the reviewed
manifest with `make deploy IMAGE_REFERENCE=…`. No existing workload or policy
needs relaxation. Platform confirmation is tracked in WP-0004-T01.

For the first installation, after admission, run `make bootstrap-namespace
IMAGE_REFERENCE=…` before the full server dry run. Kubernetes does not persist
a dry-run Namespace, so namespace-scoped objects cannot be server-validated
until the namespace exists. This bootstrap dry-runs and diffs only the reviewed
Namespace before creating it. The full deployment also diffs before applying.

The `playwright-consumer` Role grants only pod discovery and port-forward. Bind
an approved existing user/group through the identity/platform owner. This repo
does not mint credentials or silently bind all authenticated users. Callers with
existing authorized Kubernetes access can use the documented port-forward.
The pod has no service-account token and no cluster permissions.

Readiness performs the authenticated-protocol preflight over pod loopback. It
requires a successful browser launch; liveness checks the process only. The pod
uses non-root UID/GID 1000, read-only root, dropped capabilities and RuntimeDefault
seccomp. Tini reaps orphaned browser processes. Chromium's OS sandbox is disabled
for this trusted-target pilot; it is not approved for hostile websites or
mutually untrusted tenants. The HTTP/CONNECT origin gate, private-address checks,
blocked workers/WebSockets, per-run process/context and resource limits cover
the documented application-level boundary, not browser-kernel exploits.

## Verify and recover

After establishing the consumer tunnel, `make verify-live TRIAL_OUT=.local/live-N`
must produce four PNGs, focus/keyboard assertions and a receipt. Check the live
container image and imageID against the published digest, PVC binding, deny-all
ingress and ready replica. Exercise wrong run token, deliberate UI failure,
unreachable target, cancellation and service restart; a passing health endpoint
alone is insufficient.

After restart, accepted nonterminal runs become `interrupted`; completed runs
remain retrievable until expiry. A client disconnect does not cancel an accepted
job, so all runs have enforced deadlines. Supervisor cancellation closes the
browser; after two seconds it kills the recorded worker/browser process groups.
Service shutdown drains/cancels bounded work. Corrupt receipts cause startup to
fail closed; restore or quarantine the affected transient run directory through
the owner, without exposing capability hashes or screenshots in logs.

Rollback: retain the previous immutable image reference and rendered revision,
review its schema compatibility, then run `make rollback IMAGE_REFERENCE=…`.
Verify the replacement's imageID and repeat the trial. For the first release,
test recovery by reapplying the same verified digest; do not call that proof of
cross-version rollback. Once a second release exists, exercise an actual digest
change and return to the reviewed release. No automated rollback migrates or
deletes the PVC. V1 receipts are JSON; future incompatible formats require an
explicit migration/rollback decision.

Upgrade Playwright and its base image together, update the lockfile, repeat the
unit/browser/client tests and consumer pilot, then publish a new immutable image.
The Python client checks protocol v1 and does not install Playwright.

## Published release

Release [0.1.0](../releases/0.1.0.json) is published and pull-verified. Use its
immutable reference for rendering; registry publication is no longer blocked.
The cluster-owned ingress now allows a bounded 600-second HTTPS request body;
see [cause, rollout and verification](evidence/2026-10-13-publication.md).

For subsequent publications, fetch the remote index with `docker buildx
imagetools inspect --raw <image-reference>` and hash the exact returned bytes.
Require the expected digest, then pull that immutable reference with
`sudo k3s crictl pull <image-reference>` on railiance01 and verify its recorded
repo digest. A successful upload alone does not establish node pull access.


## Repeatable consumer API acceptance

After Platform establishes and verifies the caller's own identity, permission
boundary and port-forward, run this from the provider checkout with Python 3.10+:

```sh
python3 tools/verify_consumer.py --url http://127.0.0.1:18081 \
  --expected-release forgejo.coulomb.social/coulomb/rapp-playwright@sha256:8d6fd5a8172cf906bb503642ed620761945c5138b86f6396c36f65cbe6def441 \
  --expected-location railiance01/playwright --out .local/consumer-acceptance-01
```

Use a fresh output directory for each caller/run. Exit 0 means browser API checks
passed; exit 3 means verification failed. `verification.json` lists checks and
errors. The command verifies exact release/location, desktop/narrow PNG dimensions
and hashed downloads, keyboard assertions, foreign-token receipt/artifact denial,
failed-assertion evidence, private-target refusal and running-job cancellation.
It submits only fixture jobs and one denied private-target request, cancels its
own runs on exit and does not restart the service or alter access grants.

Keep all `run.json` files private. Pair the result with Platform's native consumer
identity/permission receipt for each real human and registered coding agent.
The result deliberately says `identity_authentication_verified: false`: neither
operator impersonation nor this browser check alone proves self-service access.
Target-unreachable diagnostics, abrupt interruption/recovery, revocation/stream
termination and cross-version rollback require their separate evidence. The
command's local validation is retained in
[evidence](evidence/2026-10-13-consumer-verifier.json).


## Abrupt interruption verification

`tools/verify_crash_recovery.py --image <published-immutable-reference> --out
.local/new-crash-check` creates its own bounded, read-only Docker container and
private data directory. It verifies a completed trial, captures a partial PNG in
a real active job, queues another job, sends SIGKILL to the container, and starts
it on the same data. It requires interrupted receipts, orphan cleanup, preserved
completed hashes and a fresh trial; it removes only its own container. Run it
as workstation UID 1000 to match the released image's data-directory ownership.

The separately approved managed check is `tools/verify_managed_recovery.py`.
It is an ADMINISTER operation through existing SSH, never a consumer permission.
Use `--url <operator-loopback-forward> --image <published-reference> --out
.local/new-managed-crash --approval-reference <specific-founder-approval>
--execute-crash`. Merely supplying that flag/reference does not grant approval.
The tool pins railiance01's node address, one Ready pilot pod and its release,
requires an idle service, creates its own jobs, and guards the supervisor SIGKILL
on the expected active/queued jobs and partial screenshot. It verifies container
exit137/restart, unchanged pod/PVC/image, interrupted runs, removed partial files,
completed hashes and a fresh trial. No replicas or manifests are changed.

The port-forward may exit when the container restarts. Re-establish the dedicated
operator connection after the container is Ready; the verification waits up to
180 seconds for it. Do not retry the crash command solely because the tunnel
was lost: inspect the existing verification process and pod restart state first.
Keep every `run.json` and the local data directory private. Share only selected
receipts under `docs/evidence`. These tools do not prove real consumer identity,
revocation or cross-version rollback. [Executed local and managed evidence](evidence/2026-10-13-crash-recovery.md).