Add deployment pin-drift check (make verify-pin)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-16 10:46:26 +02:00
parent 28539bfce2
commit a69adb6498
4 changed files with 194 additions and 4 deletions

View file

@ -35,6 +35,40 @@ The Deployment uses `Recreate` because the SQLite PVC is `ReadWriteOnce`.
A new pod applies the forward-only lifecycle migration on startup against
the existing database.
## Detecting pin drift
```bash
KUBECONFIG=~/.kube/config-railiance01 make verify-pin
```
Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout,
and periodically — drift is not an event you get told about.
It compares four things that are supposed to agree:
1. the digest this repo intends to run (`deploy/tenant-engine.yaml`);
2. the digest the Deployment's spec asks for;
3. the digest the **running pod** actually resolved — a spec can be correct
while the pod answering traffic is an older ReplicaSet that never finished
rolling, and consumers talk to the pod;
4. the routes the live service actually serves.
**Why this exists.** Between 2026-08-13 and 2026-08-16 production was silently
rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on
2026-08-13 were not being served. Nothing alerted. `user-engine` saw `404`, and
because its conformance evidence is contract-level, its tests passed throughout.
flex-auth suffered a parallel rollback in the same window that surfaced as
`403`. Two different symptoms, one cause, no signal either time.
The route check is not redundant with the digest check. A digest comparison
catches a changed pin; it is blind to whether the workload behind a correct
digest still serves the contract. Both failures we have actually seen were
invisible because they produced *ordinary-looking* responses rather than errors.
If it reports `Unauthorized`, check the cluster before the credential —
`KUBECONFIG` defaulting to another cluster produces an identical message, and
that cost us a round trip.
## Rollback
```bash