90 lines
4 KiB
Markdown
90 lines
4 KiB
Markdown
# tenant-engine production deployment
|
|
|
|
Manifests recovered from the live objects'
|
|
`kubectl.kubernetes.io/last-applied-configuration` on 2026-08-14 so that a
|
|
rollback does not depend on a cluster annotation. Applied originally during
|
|
TEN-WP-0004; TEN-WP-0005 keeps them in-repo and pins by digest.
|
|
|
|
| File | Deployment | Service DNS |
|
|
| --- | --- | --- |
|
|
| `tenant-engine.yaml` | `tenant-engine` | `tenant-engine.tenant-engine.svc.cluster.local:8090` |
|
|
|
|
The file is a five-document manifest: `Namespace`, `PersistentVolumeClaim`,
|
|
`Deployment`, `Service`, and a least-privilege `NetworkPolicy`. Ingress is
|
|
restricted to the `user-engine` workload; egress is restricted to
|
|
`flex-auth-tenant-engine` on 8080 plus cluster DNS.
|
|
|
|
## Rolling out an image
|
|
|
|
**Do not build images on a workstation.** `.forgejo/workflows/image.yaml`
|
|
builds from a pushed forge commit on the `container-build` runner.
|
|
|
|
```bash
|
|
# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
|
|
git push origin main
|
|
|
|
# 2. Take the immutable digest from the workflow's "Report immutable digest"
|
|
# step -- deploy by digest, never by tag
|
|
|
|
# 3. Edit the image digest in deploy/tenant-engine.yaml, then apply
|
|
kubectl apply -f deploy/tenant-engine.yaml
|
|
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s
|
|
```
|
|
|
|
The Deployment uses `Recreate` because the SQLite PVC is `ReadWriteOnce`.
|
|
A new pod applies the forward-only lifecycle migration on startup against
|
|
the existing database.
|
|
|
|
## Detecting pin drift
|
|
|
|
```bash
|
|
KUBECONFIG=~/.kube/config-railiance01 make verify-pin
|
|
```
|
|
|
|
Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout,
|
|
and periodically — drift is not an event you get told about.
|
|
|
|
It compares four things that are supposed to agree:
|
|
|
|
1. the digest this repo intends to run (`deploy/tenant-engine.yaml`);
|
|
2. the digest the Deployment's spec asks for;
|
|
3. the digest the **running pod** actually resolved — a spec can be correct
|
|
while the pod answering traffic is an older ReplicaSet that never finished
|
|
rolling, and consumers talk to the pod;
|
|
4. the routes the live service actually serves.
|
|
|
|
**Why this exists.** Between 2026-08-13 and 2026-08-16 production was silently
|
|
rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on
|
|
2026-08-13 were not being served. Nothing alerted. `user-engine` saw `404`, and
|
|
because its conformance evidence is contract-level, its tests passed throughout.
|
|
flex-auth suffered a parallel rollback in the same window that surfaced as
|
|
`403`. Two different symptoms, one cause, no signal either time.
|
|
|
|
The route check is not redundant with the digest check. A digest comparison
|
|
catches a changed pin; it is blind to whether the workload behind a correct
|
|
digest still serves the contract. Both failures we have actually seen were
|
|
invisible because they produced *ordinary-looking* responses rather than errors.
|
|
|
|
If it reports `Unauthorized`, check the cluster before the credential —
|
|
`KUBECONFIG` defaulting to another cluster produces an identical message, and
|
|
that cost us a round trip.
|
|
|
|
## Rollback
|
|
|
|
```bash
|
|
kubectl -n tenant-engine rollout undo deploy/tenant-engine
|
|
```
|
|
|
|
If the ReplicaSet history has been pruned, re-apply the manifest with the
|
|
last-known-good digest below.
|
|
|
|
| Deployment | Digest | State |
|
|
| --- | --- | --- |
|
|
| `tenant-engine` | `sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612` | **current** — TEN-WP-0005 lifecycle API, CI-built from `7e68cc8`, live 2026-08-14 |
|
|
| `tenant-engine` *(previous)* | `sha256:2249e8c6ee44ae36081cddc52daf9c3f63acd18a95a5d620ab4fa7ac85149207` | TEN-WP-0004 create/role/plan API, live 2026-08-09 to 2026-08-14 |
|
|
| `tenant-engine` *(earlier)* | `sha256:33c5dd84eaf1c2f5e067c04931219e13f9348a764a153d641035a6d019095d4f` | first TEN-WP-0004 revision |
|
|
|
|
Rolling back to `2249e8c6…` removes the lifecycle routes (`GET /tenants/{id}`,
|
|
`PATCH`, retire, reactivate). Create, role grant/revoke, and plan assign keep
|
|
working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are
|
|
forward-only and stay in place; the older image ignores them.
|