Roll back Authelia to pinned 4.38 after 4.39.28 broke KeyCape token exchange
4.39 rejects KeyCape's in-cluster http back-channel token call (invalid X-Forwarded-Proto 'http'). Operator restored the pre-upgrade DB and the 4.38 digest; incident recorded in NK-WP-0041-T02. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 299762@bnt-lap001 Assistant-Session: d3d3cea1-869c-44f1-be2a-3d6d3550e72e
This commit is contained in:
parent
cc8ded3f88
commit
329a03d149
2 changed files with 38 additions and 1 deletions
|
|
@ -108,3 +108,40 @@ These belong to user-engine:
|
|||
- There is no sign-in link from the user entry once a password has been set.
|
||||
|
||||
Send these to user-engine and record the reply.
|
||||
|
||||
### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back
|
||||
|
||||
Timeline (UTC, 2026-09-23 on the server clock):
|
||||
|
||||
- **23:34.** A pre-upgrade database copy was taken
|
||||
(`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes).
|
||||
- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29.
|
||||
The first start failed the LDAP startup check (network not ready) and
|
||||
restarted clean. Health, discovery and the probes passed; the `+` address
|
||||
became a clean not-found. KeyCape's redirect checks passed.
|
||||
- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then
|
||||
Authelia rejected KeyCape's back-channel token request: "Error occurred
|
||||
determining the effective issuer … invalid X-Forwarded-Proto header value
|
||||
'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the
|
||||
portal, Vergabe and the OpenBao browser login. The pre-rollout checks were
|
||||
redirect-only and could not see this. The 23:43 attempts had failed
|
||||
separately, because of a leading space in the pasted username (4.39 does
|
||||
not trim it).
|
||||
- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0,
|
||||
restore the pre-upgrade database via a helper pod (the migrated copy is
|
||||
kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest
|
||||
`sha256:46021dc2…`. The schema is "already up to date" (15), and health
|
||||
returned 200. Two restarts come from the same LDAP startup race.
|
||||
|
||||
Exposure: about 18 minutes in which KeyCape sign-ins failed.
|
||||
|
||||
Before retrying: KeyCape calls Authelia's token endpoint in-cluster over
|
||||
plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded
|
||||
scheme. Resolve that first, either with KeyCape sending
|
||||
`X-Forwarded-Proto: https` and the public host, or through an Authelia
|
||||
4.39 setting for the in-cluster endpoint. Also add a real back-channel token
|
||||
exchange to the upgrade acceptance, because redirect-only checks miss it.
|
||||
The repository now pins the exact 4.38 digest instead of the floating tag.
|
||||
|
||||
Separate finding: Authelia's LDAP startup check fails on the first start
|
||||
after a pod is scheduled, then passes on restart.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue