Roll back Authelia to pinned 4.38 after 4.39.28 broke KeyCape token exchange
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

4.39 rejects KeyCape's in-cluster http back-channel token call
(invalid X-Forwarded-Proto 'http'). Operator restored the pre-upgrade DB
and the 4.38 digest; incident recorded in NK-WP-0041-T02.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 299762@bnt-lap001
Assistant-Session: d3d3cea1-869c-44f1-be2a-3d6d3550e72e
This commit is contained in:
tegwick 2026-09-24 01:54:12 +02:00
parent cc8ded3f88
commit 329a03d149
2 changed files with 38 additions and 1 deletions

View file

@ -108,3 +108,40 @@ These belong to user-engine:
- There is no sign-in link from the user entry once a password has been set.
Send these to user-engine and record the reply.
### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back
Timeline (UTC, 2026-09-23 on the server clock):
- **23:34.** A pre-upgrade database copy was taken
(`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes).
- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29.
The first start failed the LDAP startup check (network not ready) and
restarted clean. Health, discovery and the probes passed; the `+` address
became a clean not-found. KeyCape's redirect checks passed.
- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then
Authelia rejected KeyCape's back-channel token request: "Error occurred
determining the effective issuer … invalid X-Forwarded-Proto header value
'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the
portal, Vergabe and the OpenBao browser login. The pre-rollout checks were
redirect-only and could not see this. The 23:43 attempts had failed
separately, because of a leading space in the pasted username (4.39 does
not trim it).
- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0,
restore the pre-upgrade database via a helper pod (the migrated copy is
kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest
`sha256:46021dc2…`. The schema is "already up to date" (15), and health
returned 200. Two restarts come from the same LDAP startup race.
Exposure: about 18 minutes in which KeyCape sign-ins failed.
Before retrying: KeyCape calls Authelia's token endpoint in-cluster over
plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded
scheme. Resolve that first, either with KeyCape sending
`X-Forwarded-Proto: https` and the public host, or through an Authelia
4.39 setting for the in-cluster endpoint. Also add a real back-channel token
exchange to the upgrade acceptance, because redirect-only checks miss it.
The repository now pins the exact 4.38 digest instead of the floating tag.
Separate finding: Authelia's LDAP startup check fails on the first start
after a pod is scheduled, then passes on restart.