143 lines
6.7 KiB
Markdown
143 lines
6.7 KiB
Markdown
# tenant-engine production deployment
|
|
|
|
**Apply home moving to `rapp-tenant-engine`.** These files remain the source of the absorbed manifests until that package is the operator apply path.
|
|
|
|
|
|
The original manifests were recovered from live objects on 2026-08-14. The
|
|
repository now carries the reviewed **post-cutover desired state** from
|
|
TEN-WP-0009. It is not evidence that PostgreSQL has been provisioned or that
|
|
the live SQLite deployment has moved.
|
|
|
|
| File | Deployment | Service DNS |
|
|
| --- | --- | --- |
|
|
| `tenant-engine.yaml` | `tenant-engine` | `tenant-engine.tenant-engine.svc.cluster.local:8090` |
|
|
|
|
`tenant-engine.yaml` declares `Namespace`, rolling `Deployment`, `Service`,
|
|
`ServiceAccount`, and least-privilege `NetworkPolicy`. Ingress is restricted
|
|
to `user-engine`; egress is restricted to `platform-pg` in `databases`,
|
|
`flex-auth-tenant-engine`, and cluster DNS. PostgreSQL and flex-auth caller
|
|
credentials are projected as files so clients can re-read them on rotation.
|
|
|
|
`tenant-engine-migration.yaml` is the separate schema-migration Job and its
|
|
egress policy. The pinned image in both files must contain the PostgreSQL code
|
|
and migration before either manifest is applied.
|
|
|
|
The Deployment uses `/live` for process liveness and `/health` for readiness.
|
|
`/health` checks the selected store, so a database or lease outage removes the
|
|
pod from service without turning an external dependency failure into a restart
|
|
loop.
|
|
|
|
## Rolling out an image
|
|
|
|
**Do not build images on a workstation.** `.forgejo/workflows/image.yaml`
|
|
builds from a pushed forge commit on the `container-build` runner.
|
|
|
|
```bash
|
|
# 1. Push the commit you intend to ship; CI builds :latest and :main-<short-sha>
|
|
git push origin main
|
|
|
|
# 2. Take the immutable digest from the workflow's "Report immutable digest"
|
|
# step -- deploy by digest, never by tag
|
|
|
|
# 3. Provision the rapp-postgres consumer and credential projections.
|
|
# 4. Run the schema migration Job with the migration-role lease.
|
|
# 5. Follow the stopped-write transfer below; only then apply the runtime.
|
|
kubectl apply -f deploy/tenant-engine.yaml
|
|
kubectl -n tenant-engine rollout status deploy/tenant-engine --timeout=120s
|
|
```
|
|
|
|
The desired Deployment uses `RollingUpdate`; schema migration is never a
|
|
runtime startup side effect. Keep replicas at one until recovery and caller
|
|
latency evidence justifies otherwise.
|
|
|
|
## Stopped-write SQLite transfer
|
|
|
|
This is an operator cutover, not an application startup step:
|
|
|
|
1. Scale the live SQLite Deployment to zero and verify no pod holds the PVC.
|
|
2. Snapshot or copy the SQLite file without modifying it.
|
|
3. Apply `deploy/tenant-engine-migration.yaml` with the new immutable image and
|
|
wait for success. The Job sets role to the stable
|
|
`tenant_engine_migrate` group before DDL; short-lived OpenBao login roles
|
|
must never own schema objects that outlive their lease.
|
|
4. Run the transfer under the migration-role URL:
|
|
|
|
```bash
|
|
tenant-engine-transfer \
|
|
--sqlite /path/to/stopped/tenant-engine.db \
|
|
--url-file /path/to/migration-url \
|
|
--expect-tenant tenant:trial:portalcheck=active:1 \
|
|
--expect-tenant tenant:trial:ten-wp-0005-t05=retired:5 \
|
|
--evidence-output docs/evidence/TEN-WP-0009-cutover.json
|
|
```
|
|
|
|
The target must be empty. The command transfers every table in one
|
|
transaction, preserves event sequence values and nullable legacy
|
|
timestamps, then compares every row and emits only counts, digests, and the
|
|
requested lifecycle/version spot checks.
|
|
5. Apply the runtime desired state, require `/health` to report
|
|
`store_backend: postgresql`, and execute the TEN-WP-0007 live verification
|
|
list.
|
|
6. Retain the stopped SQLite PVC, access-restricted, through the recorded soak.
|
|
Do not run two writable authorities.
|
|
|
|
## Detecting pin drift
|
|
|
|
```bash
|
|
KUBECONFIG=~/.kube/config-railiance01 make verify-pin
|
|
```
|
|
|
|
Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout,
|
|
and periodically — drift is not an event you get told about.
|
|
|
|
It compares four things that are supposed to agree:
|
|
|
|
1. the digest this repo intends to run (`deploy/tenant-engine.yaml`);
|
|
2. the digest the Deployment's spec asks for;
|
|
3. the digest the **running pod** actually resolved — a spec can be correct
|
|
while the pod answering traffic is an older ReplicaSet that never finished
|
|
rolling, and consumers talk to the pod;
|
|
4. the routes the live service actually serves.
|
|
|
|
**Why this exists.** Between 2026-08-13 and 2026-08-16 production was silently
|
|
rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on
|
|
2026-08-13 were not being served. Nothing alerted. `user-engine` saw `404`, and
|
|
because its conformance evidence is contract-level, its tests passed throughout.
|
|
flex-auth suffered a parallel rollback in the same window that surfaced as
|
|
`403`. Two different symptoms, one cause, no signal either time.
|
|
|
|
The route check is not redundant with the digest check. A digest comparison
|
|
catches a changed pin; it is blind to whether the workload behind a correct
|
|
digest still serves the contract. Both failures we have actually seen were
|
|
invisible because they produced *ordinary-looking* responses rather than errors.
|
|
|
|
If it reports `Unauthorized`, check the cluster before the credential —
|
|
`KUBECONFIG` defaulting to another cluster produces an identical message, and
|
|
that cost us a round trip.
|
|
|
|
## Rollback
|
|
|
|
```bash
|
|
kubectl -n tenant-engine rollout undo deploy/tenant-engine
|
|
```
|
|
|
|
Before the first PostgreSQL write, reattaching the stopped SQLite PVC is a
|
|
recoverable rollback. After any PostgreSQL write, that file is stale and must
|
|
not be made authoritative; rollback means restore or reverse-transfer from
|
|
PostgreSQL under a new operator plan. The repository desired manifest does not
|
|
carry the PVC, so pre-cutover rollback also requires the retained previous
|
|
manifest/revision.
|
|
|
|
If the ReplicaSet history has been pruned, re-apply the manifest with the
|
|
last-known-good digest below.
|
|
|
|
| Deployment | Digest | State |
|
|
| --- | --- | --- |
|
|
| `tenant-engine` | `sha256:a8e8086ffc5b772c1391b166f5e1884b90f7d327b152c205eceae129df555c24` | **current** — PostgreSQL runtime plus stable migration-role ownership, CI-built from `d2a7fe3` |
|
|
| `tenant-engine` *(previous)* | `sha256:023ac49f5c5c0927aff685455626215f9daccd0eac34899a87c43b1d96fe4b63` | initial PostgreSQL cutover image, replaced after the leased-login ownership defect was repaired |
|
|
| `tenant-engine` *(pre-cutover)* | `sha256:7dc9d7bf1fcbb3f3ad7035366f4cd44038322995a126b4423142e446ca5c9cd5` | final SQLite/PVC image; data authority stopped 2026-08-21 |
|
|
|
|
Rolling back to `2249e8c6…` removes the lifecycle routes (`GET /tenants/{id}`,
|
|
`PATCH`, retire, reactivate). Create, role grant/revoke, and plan assign keep
|
|
working. The SQLite lifecycle columns added by the TEN-WP-0005 migration are
|
|
forward-only and stay in place; the older image ignores them.
|