diff --git a/sso-mfa/k8s/authelia/deployment.yaml b/sso-mfa/k8s/authelia/deployment.yaml index 246964d..1b23085 100644 --- a/sso-mfa/k8s/authelia/deployment.yaml +++ b/sso-mfa/k8s/authelia/deployment.yaml @@ -45,7 +45,7 @@ spec: containers: - name: authelia # Pin to a specific 4.x release. Check https://hub.docker.com/r/authelia/authelia - image: docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54 # v4.39.28 + image: docker.io/authelia/authelia@sha256:46021dc20efdcc5cdc38a29e3050b8835429a155ae6215388ed3b793a02eb0ab # 4.38 (rolled back from 4.39.28, NK-WP-0041-T02) imagePullPolicy: IfNotPresent ports: diff --git a/workplans/NK-WP-0041-onboarding-journey-usability.md b/workplans/NK-WP-0041-onboarding-journey-usability.md index db861e7..90cfd2f 100644 --- a/workplans/NK-WP-0041-onboarding-journey-usability.md +++ b/workplans/NK-WP-0041-onboarding-journey-usability.md @@ -108,3 +108,40 @@ These belong to user-engine: - There is no sign-in link from the user entry once a password has been set. Send these to user-engine and record the reply. + +### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back + +Timeline (UTC, 2026-09-23 on the server clock): + +- **23:34.** A pre-upgrade database copy was taken + (`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes). +- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29. + The first start failed the LDAP startup check (network not ready) and + restarted clean. Health, discovery and the probes passed; the `+` address + became a clean not-found. KeyCape's redirect checks passed. +- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then + Authelia rejected KeyCape's back-channel token request: "Error occurred + determining the effective issuer … invalid X-Forwarded-Proto header value + 'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the + portal, Vergabe and the OpenBao browser login. The pre-rollout checks were + redirect-only and could not see this. The 23:43 attempts had failed + separately, because of a leading space in the pasted username (4.39 does + not trim it). +- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0, + restore the pre-upgrade database via a helper pod (the migrated copy is + kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest + `sha256:46021dc2…`. The schema is "already up to date" (15), and health + returned 200. Two restarts come from the same LDAP startup race. + +Exposure: about 18 minutes in which KeyCape sign-ins failed. + +Before retrying: KeyCape calls Authelia's token endpoint in-cluster over +plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded +scheme. Resolve that first, either with KeyCape sending +`X-Forwarded-Proto: https` and the public host, or through an Authelia +4.39 setting for the in-cluster endpoint. Also add a real back-channel token +exchange to the upgrade acceptance, because redirect-only checks miss it. +The repository now pins the exact 4.38 digest instead of the floating tag. + +Separate finding: Authelia's LDAP startup check fails on the first start +after a pod is scheduled, then passes on restart.