tenant-engine/deploy/README.md

tenant-engine production deployment

Manifests recovered from the live objects' kubectl.kubernetes.io/last-applied-configuration on 2026-08-14 so that a rollback does not depend on a cluster annotation. Applied originally during TEN-WP-0004; TEN-WP-0005 keeps them in-repo and pins by digest.

File	Deployment	Service DNS
tenant-engine.yaml	tenant-engine	tenant-engine.tenant-engine.svc.cluster.local:8090

The file is a five-document manifest: Namespace, PersistentVolumeClaim, Deployment, Service, and a least-privilege NetworkPolicy. Ingress is restricted to the user-engine workload; egress is restricted to flex-auth-tenant-engine on 8080 plus cluster DNS.

Rolling out an image

Do not build images on a workstation. .forgejo/workflows/image.yaml builds from a pushed forge commit on the container-build runner.

# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
git push origin main

# 2. Take the immutable digest from the workflow's "Report immutable digest"
#    step -- deploy by digest, never by tag

# 3. Edit the image digest in deploy/tenant-engine.yaml, then apply
kubectl apply -f deploy/tenant-engine.yaml
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s

The Deployment uses Recreate because the SQLite PVC is ReadWriteOnce. A new pod applies the forward-only lifecycle migration on startup against the existing database.

Detecting pin drift

KUBECONFIG=~/.kube/config-railiance01 make verify-pin

Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout, and periodically — drift is not an event you get told about.

It compares four things that are supposed to agree:

1. the digest this repo intends to run (deploy/tenant-engine.yaml);
2. the digest the Deployment's spec asks for;
3. the digest the running pod actually resolved — a spec can be correct while the pod answering traffic is an older ReplicaSet that never finished rolling, and consumers talk to the pod;
4. the routes the live service actually serves.

Why this exists. Between 2026-08-13 and 2026-08-16 production was silently rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on 2026-08-13 were not being served. Nothing alerted. user-engine saw 404, and because its conformance evidence is contract-level, its tests passed throughout. flex-auth suffered a parallel rollback in the same window that surfaced as 403. Two different symptoms, one cause, no signal either time.

The route check is not redundant with the digest check. A digest comparison catches a changed pin; it is blind to whether the workload behind a correct digest still serves the contract. Both failures we have actually seen were invisible because they produced ordinary-looking responses rather than errors.

If it reports Unauthorized, check the cluster before the credential — KUBECONFIG defaulting to another cluster produces an identical message, and that cost us a round trip.

Rollback

kubectl -n tenant-engine rollout undo deploy/tenant-engine

If the ReplicaSet history has been pruned, re-apply the manifest with the last-known-good digest below.

Deployment	Digest	State
tenant-engine	sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612	current — TEN-WP-0005 lifecycle API, CI-built from 7e68cc8, live 2026-08-14
tenant-engine (previous)	sha256:2249e8c6ee44ae36081cddc52daf9c3f63acd18a95a5d620ab4fa7ac85149207	TEN-WP-0004 create/role/plan API, live 2026-08-09 to 2026-08-14
tenant-engine (earlier)	sha256:33c5dd84eaf1c2f5e067c04931219e13f9348a764a153d641035a6d019095d4f	first TEN-WP-0004 revision

Rolling back to 2249e8c6… removes the lifecycle routes (GET /tenants/{id}, PATCH, retire, reactivate). Create, role grant/revoke, and plan assign keep working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are forward-only and stay in place; the older image ignores them.