tenant-engine/deploy/README.md

tenant-engine production deployment

Apply path: kubectl apply -k deploy/ (TEN-WP-0008). deploy/base/tenant-engine.yaml is the document kustomize includes, and the file make verify-pin reads for the intended digest. Isolated canary: kubectl apply -k deploy/canary/.

Staged-promotion contract: railiance/app.toml. Recorded stable: tenant-engine@sha256:a8e8086ffc5b772c1391b166f5e1884b90f7d327b152c205eceae129df555c24.

The original manifests were recovered from live objects on 2026-08-14. The repository now carries the reviewed post-cutover desired state from TEN-WP-0009. It is not evidence that PostgreSQL has been provisioned or that the live SQLite deployment has moved.

File	Deployment	Service DNS
tenant-engine.yaml	tenant-engine	tenant-engine.tenant-engine.svc.cluster.local:8090

tenant-engine.yaml declares Namespace, rolling Deployment, Service, ServiceAccount, and least-privilege NetworkPolicy. Ingress is restricted to user-engine; egress is restricted to platform-pg in databases, flex-auth-tenant-engine, and cluster DNS. PostgreSQL and flex-auth caller credentials are projected as files so clients can re-read them on rotation.

tenant-engine-migration.yaml is the separate schema-migration Job and its egress policy. The pinned image in both files must contain the PostgreSQL code and migration before either manifest is applied.

The Deployment uses /live for process liveness and /health for readiness. /health checks the selected store, so a database or lease outage removes the pod from service without turning an external dependency failure into a restart loop.

Rolling out an image

Do not build images on a workstation. .forgejo/workflows/image.yaml builds from a pushed forge commit on the container-build runner.

# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
git push origin main

# 2. Take the immutable digest from the workflow's "Report immutable digest"
#    step -- deploy by digest, never by tag

# 3. Provision the rapp-postgres consumer and credential projections.
# 4. Run the schema migration Job with the migration-role lease.
# 5. Follow the stopped-write transfer below; only then apply the runtime.
kubectl apply -k deploy/
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s

The desired Deployment uses RollingUpdate; schema migration is never a runtime startup side effect. Keep replicas at one until recovery and caller latency evidence justifies otherwise.

Stopped-write SQLite transfer

This is an operator cutover, not an application startup step:

1. Scale the live SQLite Deployment to zero and verify no pod holds the PVC.

2. Snapshot or copy the SQLite file without modifying it.

3. Apply deploy/tenant-engine-migration.yaml with the new immutable image and wait for success. The Job sets role to the stable tenant_engine_migrate group before DDL; short-lived OpenBao login roles must never own schema objects that outlive their lease.

4. Run the transfer under the migration-role URL:

   tenant-engine-transfer \
     --sqlite /path/to/stopped/tenant-engine.db \
     --url-file /path/to/migration-url \
     --expect-tenant tenant:trial:portalcheck=active:1 \
     --expect-tenant tenant:trial:ten-wp-0005-t05=retired:5 \
     --evidence-output docs/evidence/TEN-WP-0009-cutover.json
   

   The target must be empty. The command transfers every table in one transaction, preserves event sequence values and nullable legacy timestamps, then compares every row and emits only counts, digests, and the requested lifecycle/version spot checks.

5. Apply the runtime desired state, require /health to report store_backend: postgresql, and execute the TEN-WP-0007 live verification list.

6. Retain the stopped SQLite PVC, access-restricted, through the recorded soak. Do not run two writable authorities.

Detecting pin drift

KUBECONFIG=~/.kube/config-railiance01 make verify-pin

Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout, and periodically — drift is not an event you get told about.

It compares four things that are supposed to agree:

1. the digest this repo intends to run (deploy/base/tenant-engine.yaml);
2. the digest the Deployment's spec asks for;
3. the digest the running pod actually resolved — a spec can be correct while the pod answering traffic is an older ReplicaSet that never finished rolling, and consumers talk to the pod;
4. the routes the live service actually serves.

Why this exists. Between 2026-08-13 and 2026-08-16 production was silently rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on 2026-08-13 were not being served. Nothing alerted. user-engine saw 404, and because its conformance evidence is contract-level, its tests passed throughout. flex-auth suffered a parallel rollback in the same window that surfaced as 403. Two different symptoms, one cause, no signal either time.

The route check is not redundant with the digest check. A digest comparison catches a changed pin; it is blind to whether the workload behind a correct digest still serves the contract. Both failures we have actually seen were invisible because they produced ordinary-looking responses rather than errors.

If it reports Unauthorized, check the cluster before the credential — KUBECONFIG defaulting to another cluster produces an identical message, and that cost us a round trip.

Rollback

kubectl -n tenant-engine rollout undo deploy/tenant-engine

Before the first PostgreSQL write, reattaching the stopped SQLite PVC is a recoverable rollback. After any PostgreSQL write, that file is stale and must not be made authoritative; rollback means restore or reverse-transfer from PostgreSQL under a new operator plan. The repository desired manifest does not carry the PVC, so pre-cutover rollback also requires the retained previous manifest/revision.

If the ReplicaSet history has been pruned, re-apply the manifest with the last-known-good digest below.

Deployment	Digest	State
tenant-engine	sha256:a8e8086ffc5b772c1391b166f5e1884b90f7d327b152c205eceae129df555c24	current — PostgreSQL runtime plus stable migration-role ownership, CI-built from d2a7fe3
tenant-engine (previous)	sha256:023ac49f5c5c0927aff685455626215f9daccd0eac34899a87c43b1d96fe4b63	initial PostgreSQL cutover image, replaced after the leased-login ownership defect was repaired
tenant-engine (pre-cutover)	sha256:7dc9d7bf1fcbb3f3ad7035366f4cd44038322995a126b4423142e446ca5c9cd5	final SQLite/PVC image; data authority stopped 2026-08-21

Rolling back to 2249e8c6… removes the lifecycle routes (GET /tenants/{id}, PATCH, retire, reactivate). Create, role grant/revoke, and plan assign keep working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are forward-only and stay in place; the older image ignores them.