Roll back Authelia to pinned 4.38 after 4.39.28 broke KeyCape token exchange
4.39 rejects KeyCape's in-cluster http back-channel token call (invalid X-Forwarded-Proto 'http'). Operator restored the pre-upgrade DB and the 4.38 digest; incident recorded in NK-WP-0041-T02. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 299762@bnt-lap001 Assistant-Session: d3d3cea1-869c-44f1-be2a-3d6d3550e72e
This commit is contained in:
parent
cc8ded3f88
commit
329a03d149
2 changed files with 38 additions and 1 deletions
|
|
@ -45,7 +45,7 @@ spec:
|
||||||
containers:
|
containers:
|
||||||
- name: authelia
|
- name: authelia
|
||||||
# Pin to a specific 4.x release. Check https://hub.docker.com/r/authelia/authelia
|
# Pin to a specific 4.x release. Check https://hub.docker.com/r/authelia/authelia
|
||||||
image: docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54 # v4.39.28
|
image: docker.io/authelia/authelia@sha256:46021dc20efdcc5cdc38a29e3050b8835429a155ae6215388ed3b793a02eb0ab # 4.38 (rolled back from 4.39.28, NK-WP-0041-T02)
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
|
|
||||||
ports:
|
ports:
|
||||||
|
|
|
||||||
|
|
@ -108,3 +108,40 @@ These belong to user-engine:
|
||||||
- There is no sign-in link from the user entry once a password has been set.
|
- There is no sign-in link from the user entry once a password has been set.
|
||||||
|
|
||||||
Send these to user-engine and record the reply.
|
Send these to user-engine and record the reply.
|
||||||
|
|
||||||
|
### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back
|
||||||
|
|
||||||
|
Timeline (UTC, 2026-09-23 on the server clock):
|
||||||
|
|
||||||
|
- **23:34.** A pre-upgrade database copy was taken
|
||||||
|
(`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes).
|
||||||
|
- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29.
|
||||||
|
The first start failed the LDAP startup check (network not ready) and
|
||||||
|
restarted clean. Health, discovery and the probes passed; the `+` address
|
||||||
|
became a clean not-found. KeyCape's redirect checks passed.
|
||||||
|
- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then
|
||||||
|
Authelia rejected KeyCape's back-channel token request: "Error occurred
|
||||||
|
determining the effective issuer … invalid X-Forwarded-Proto header value
|
||||||
|
'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the
|
||||||
|
portal, Vergabe and the OpenBao browser login. The pre-rollout checks were
|
||||||
|
redirect-only and could not see this. The 23:43 attempts had failed
|
||||||
|
separately, because of a leading space in the pasted username (4.39 does
|
||||||
|
not trim it).
|
||||||
|
- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0,
|
||||||
|
restore the pre-upgrade database via a helper pod (the migrated copy is
|
||||||
|
kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest
|
||||||
|
`sha256:46021dc2…`. The schema is "already up to date" (15), and health
|
||||||
|
returned 200. Two restarts come from the same LDAP startup race.
|
||||||
|
|
||||||
|
Exposure: about 18 minutes in which KeyCape sign-ins failed.
|
||||||
|
|
||||||
|
Before retrying: KeyCape calls Authelia's token endpoint in-cluster over
|
||||||
|
plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded
|
||||||
|
scheme. Resolve that first, either with KeyCape sending
|
||||||
|
`X-Forwarded-Proto: https` and the public host, or through an Authelia
|
||||||
|
4.39 setting for the in-cluster endpoint. Also add a real back-channel token
|
||||||
|
exchange to the upgrade acceptance, because redirect-only checks miss it.
|
||||||
|
The repository now pins the exact 4.38 digest instead of the floating tag.
|
||||||
|
|
||||||
|
Separate finding: Authelia's LDAP startup check fails on the first start
|
||||||
|
after a pod is scheduled, then passes on restart.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue