Add deployment pin-drift check (make verify-pin)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
28539bfce2
commit
a69adb6498
4 changed files with 194 additions and 4 deletions
|
|
@ -35,6 +35,40 @@ The Deployment uses `Recreate` because the SQLite PVC is `ReadWriteOnce`.
|
|||
A new pod applies the forward-only lifecycle migration on startup against
|
||||
the existing database.
|
||||
|
||||
## Detecting pin drift
|
||||
|
||||
```bash
|
||||
KUBECONFIG=~/.kube/config-railiance01 make verify-pin
|
||||
```
|
||||
|
||||
Exit 0 in sync, 1 on drift, 2 if it could not tell. Run it after any rollout,
|
||||
and periodically — drift is not an event you get told about.
|
||||
|
||||
It compares four things that are supposed to agree:
|
||||
|
||||
1. the digest this repo intends to run (`deploy/tenant-engine.yaml`);
|
||||
2. the digest the Deployment's spec asks for;
|
||||
3. the digest the **running pod** actually resolved — a spec can be correct
|
||||
while the pod answering traffic is an older ReplicaSet that never finished
|
||||
rolling, and consumers talk to the pod;
|
||||
4. the routes the live service actually serves.
|
||||
|
||||
**Why this exists.** Between 2026-08-13 and 2026-08-16 production was silently
|
||||
rolled back to the TEN-WP-0004 image, so the lifecycle routes verified live on
|
||||
2026-08-13 were not being served. Nothing alerted. `user-engine` saw `404`, and
|
||||
because its conformance evidence is contract-level, its tests passed throughout.
|
||||
flex-auth suffered a parallel rollback in the same window that surfaced as
|
||||
`403`. Two different symptoms, one cause, no signal either time.
|
||||
|
||||
The route check is not redundant with the digest check. A digest comparison
|
||||
catches a changed pin; it is blind to whether the workload behind a correct
|
||||
digest still serves the contract. Both failures we have actually seen were
|
||||
invisible because they produced *ordinary-looking* responses rather than errors.
|
||||
|
||||
If it reports `Unauthorized`, check the cluster before the credential —
|
||||
`KUBECONFIG` defaulting to another cluster produces an identical message, and
|
||||
that cost us a round trip.
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue