# tenant-engine production deployment

Manifests recovered from the live objects'
`kubectl.kubernetes.io/last-applied-configuration` on 2026-08-14 so that a
rollback does not depend on a cluster annotation. Applied originally during
TEN-WP-0004; TEN-WP-0005 keeps them in-repo and pins by digest.

| File | Deployment | Service DNS |
| --- | --- | --- |
| `tenant-engine.yaml` | `tenant-engine` | `tenant-engine.tenant-engine.svc.cluster.local:8090` |

The file is a five-document manifest: `Namespace`, `PersistentVolumeClaim`,
`Deployment`, `Service`, and a least-privilege `NetworkPolicy`. Ingress is
restricted to the `user-engine` workload; egress is restricted to
`flex-auth-tenant-engine` on 8080 plus cluster DNS.

## Rolling out an image

**Do not build images on a workstation.** `.forgejo/workflows/image.yaml`
builds from a pushed forge commit on the `container-build` runner.

```bash
# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
git push origin main

# 2. Take the immutable digest from the workflow's "Report immutable digest"
#    step -- deploy by digest, never by tag

# 3. Edit the image digest in deploy/tenant-engine.yaml, then apply
kubectl apply -f deploy/tenant-engine.yaml
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s
```

The Deployment uses `Recreate` because the SQLite PVC is `ReadWriteOnce`.
A new pod applies the forward-only lifecycle migration on startup against
the existing database.

## Detecting pin drift

```bash
KUBECONFIG=~/.kube/config-railiance01 make verify-pin
```

Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout,
and periodically — drift is not an event you get told about.

It compares four things that are supposed to agree:

1. the digest this repo intends to run (`deploy/tenant-engine.yaml`);
2. the digest the Deployment's spec asks for;
3. the digest the **running pod** actually resolved — a spec can be correct
   while the pod answering traffic is an older ReplicaSet that never finished
   rolling, and consumers talk to the pod;
4. the routes the live service actually serves.

**Why this exists.** Between 2026-08-13 and 2026-08-16 production was silently
rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on
2026-08-13 were not being served. Nothing alerted. `user-engine` saw `404`, and
because its conformance evidence is contract-level, its tests passed throughout.
flex-auth suffered a parallel rollback in the same window that surfaced as
`403`. Two different symptoms, one cause, no signal either time.

The route check is not redundant with the digest check. A digest comparison
catches a changed pin; it is blind to whether the workload behind a correct
digest still serves the contract. Both failures we have actually seen were
invisible because they produced *ordinary-looking* responses rather than errors.

If it reports `Unauthorized`, check the cluster before the credential —
`KUBECONFIG` defaulting to another cluster produces an identical message, and
that cost us a round trip.

## Rollback

```bash
kubectl -n tenant-engine rollout undo deploy/tenant-engine
```

If the ReplicaSet history has been pruned, re-apply the manifest with the
last-known-good digest below.

| Deployment | Digest | State |
| --- | --- | --- |
| `tenant-engine` | `sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612` | **current** — TEN-WP-0005 lifecycle API, CI-built from `7e68cc8`, live 2026-08-14 |
| `tenant-engine` *(previous)* | `sha256:2249e8c6ee44ae36081cddc52daf9c3f63acd18a95a5d620ab4fa7ac85149207` | TEN-WP-0004 create/role/plan API, live 2026-08-09 to 2026-08-14 |
| `tenant-engine` *(earlier)* | `sha256:33c5dd84eaf1c2f5e067c04931219e13f9348a764a153d641035a6d019095d4f` | first TEN-WP-0004 revision |

Rolling back to `2249e8c6…` removes the lifecycle routes (`GET /tenants/{id}`,
`PATCH`, retire, reactivate). Create, role grant/revoke, and plan assign keep
working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are
forward-only and stay in place; the older image ignores them.