tenant-engine/deploy

tenant-engine production deployment

Apply home moving to rapp-tenant-engine. These files remain the source of the absorbed manifests until that package is the operator apply path.

The original manifests were recovered from live objects on 2026-08-14. The repository now carries the reviewed post-cutover desired state from TEN-WP-0009. It is not evidence that PostgreSQL has been provisioned or that the live SQLite deployment has moved.

File	Deployment	Service DNS
tenant-engine.yaml	tenant-engine	tenant-engine.tenant-engine.svc.cluster.local:8090

tenant-engine.yaml declares Namespace, rolling Deployment, Service, ServiceAccount, and least-privilege NetworkPolicy. Ingress is restricted to user-engine; egress is restricted to platform-pg in databases, flex-auth-tenant-engine, and cluster DNS. PostgreSQL and flex-auth caller credentials are projected as files so clients can re-read them on rotation.

tenant-engine-migration.yaml is the separate schema-migration Job and its egress policy. The pinned image in both files must contain the PostgreSQL code and migration before either manifest is applied.

The Deployment uses /live for process liveness and /health for readiness. /health checks the selected store, so a database or lease outage removes the pod from service without turning an external dependency failure into a restart loop.

Rolling out an image

Do not build images on a workstation. .forgejo/workflows/image.yaml builds from a pushed forge commit on the container-build runner.

# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
git push origin main

# 2. Take the immutable digest from the workflow's "Report immutable digest"
#    step -- deploy by digest, never by tag

# 3. Provision the rapp-postgres consumer and credential projections.
# 4. Run the schema migration Job with the migration-role lease.
# 5. Follow the stopped-write transfer below; only then apply the runtime.
kubectl apply -f deploy/tenant-engine.yaml
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s

The desired Deployment uses RollingUpdate; schema migration is never a runtime startup side effect. Keep replicas at one until recovery and caller latency evidence justifies otherwise.

Stopped-write SQLite transfer

This is an operator cutover, not an application startup step:

1. Scale the live SQLite Deployment to zero and verify no pod holds the PVC.

2. Snapshot or copy the SQLite file without modifying it.

3. Apply deploy/tenant-engine-migration.yaml with the new immutable image and wait for success.

4. Run the transfer under the migration-role URL:

   tenant-engine-transfer \
     --sqlite /path/to/stopped/tenant-engine.db \
     --url-file /path/to/migration-url \
     --expect-tenant tenant:trial:portalcheck=active:1 \
     --expect-tenant tenant:trial:ten-wp-0005-t05=retired:5 \
     --evidence-output docs/evidence/TEN-WP-0009-cutover.json
   

   The target must be empty. The command transfers every table in one transaction, preserves event sequence values and nullable legacy timestamps, then compares every row and emits only counts, digests, and the requested lifecycle/version spot checks.

5. Apply the runtime desired state, require /health to report store_backend: postgresql, and execute the TEN-WP-0007 live verification list.

6. Retain the stopped SQLite PVC, access-restricted, through the recorded soak. Do not run two writable authorities.

Detecting pin drift

KUBECONFIG=~/.kube/config-railiance01 make verify-pin

Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout, and periodically — drift is not an event you get told about.

It compares four things that are supposed to agree:

1. the digest this repo intends to run (deploy/tenant-engine.yaml);
2. the digest the Deployment's spec asks for;
3. the digest the running pod actually resolved — a spec can be correct while the pod answering traffic is an older ReplicaSet that never finished rolling, and consumers talk to the pod;
4. the routes the live service actually serves.

Why this exists. Between 2026-08-13 and 2026-08-16 production was silently rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on 2026-08-13 were not being served. Nothing alerted. user-engine saw 404, and because its conformance evidence is contract-level, its tests passed throughout. flex-auth suffered a parallel rollback in the same window that surfaced as 403. Two different symptoms, one cause, no signal either time.

The route check is not redundant with the digest check. A digest comparison catches a changed pin; it is blind to whether the workload behind a correct digest still serves the contract. Both failures we have actually seen were invisible because they produced ordinary-looking responses rather than errors.

If it reports Unauthorized, check the cluster before the credential — KUBECONFIG defaulting to another cluster produces an identical message, and that cost us a round trip.

Rollback

kubectl -n tenant-engine rollout undo deploy/tenant-engine

Before the first PostgreSQL write, reattaching the stopped SQLite PVC is a recoverable rollback. After any PostgreSQL write, that file is stale and must not be made authoritative; rollback means restore or reverse-transfer from PostgreSQL under a new operator plan. The repository desired manifest does not carry the PVC, so pre-cutover rollback also requires the retained previous manifest/revision.

If the ReplicaSet history has been pruned, re-apply the manifest with the last-known-good digest below.

Deployment	Digest	State
tenant-engine	sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612	current — TEN-WP-0005 lifecycle API, CI-built from 7e68cc8, live 2026-08-14
tenant-engine (previous)	sha256:2249e8c6ee44ae36081cddc52daf9c3f63acd18a95a5d620ab4fa7ac85149207	TEN-WP-0004 create/role/plan API, live 2026-08-09 to 2026-08-14
tenant-engine (earlier)	sha256:33c5dd84eaf1c2f5e067c04931219e13f9348a764a153d641035a6d019095d4f	first TEN-WP-0004 revision

Rolling back to 2249e8c6… removes the lifecycle routes (GET /tenants/{id}, PATCH, retire, reactivate). Create, role grant/revoke, and plan assign keep working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are forward-only and stay in place; the older image ignores them.