2026-09-23 21:48:29 +02:00
|
|
|
---
|
|
|
|
|
id: NK-WP-0041
|
|
|
|
|
type: workplan
|
|
|
|
|
title: "Fix onboarding-journey defects found in the 2026-09-23 human run"
|
|
|
|
|
domain: infotech
|
|
|
|
|
repo: net-kingdom
|
|
|
|
|
status: active
|
|
|
|
|
flavor: implementation
|
|
|
|
|
owner: claude-code
|
|
|
|
|
topic_slug: netkingdom
|
|
|
|
|
created: "2026-09-23"
|
2026-09-27 13:07:31 +02:00
|
|
|
updated: "2026-09-27"
|
2026-09-23 21:48:29 +02:00
|
|
|
related: [NK-WP-0036, NK-WP-0037, KEY-WP-0033]
|
2026-09-23 21:49:34 +02:00
|
|
|
state_hub_workstream_id: "98168f50-7a4d-5bb5-a462-1e031563b89f"
|
2026-09-23 21:48:29 +02:00
|
|
|
---
|
|
|
|
|
|
|
|
|
|
Found by the operator's native onboarding run (NK-WP-0036-T05) and the
|
|
|
|
|
blocked Vergabe sign-in (NK-WP-0037-T02). Each item names its owner, and only
|
|
|
|
|
items in this repository are implemented here.
|
|
|
|
|
|
|
|
|
|
## Password managers and sign-in after password setup
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
id: NK-WP-0041-T01
|
2026-09-24 01:19:48 +02:00
|
|
|
status: done
|
2026-09-23 21:48:29 +02:00
|
|
|
priority: medium
|
2026-09-23 21:49:34 +02:00
|
|
|
state_hub_task_id: "82ce581a-b635-5b5b-9b08-91b61e7eb9bd"
|
2026-09-23 21:48:29 +02:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The setup form had no username field, so browsers stored the new password
|
|
|
|
|
without a login. After setup there was no way on to the sign-in page. The
|
|
|
|
|
source is fixed in `identity-provisioner`:
|
|
|
|
|
|
|
|
|
|
- The setup form shows the grant's login name as a read-only
|
|
|
|
|
`autocomplete="username"` field. The submitted value is ignored, and the
|
|
|
|
|
form no longer sets `autocomplete="off"`.
|
|
|
|
|
- The completion page links to `PASSWORD_SETUP_SIGNIN_URL` (HTTPS only)
|
|
|
|
|
when there is no company return. The deployment declares
|
|
|
|
|
`https://users.coulomb.social/`.
|
|
|
|
|
- The completion page still does not show the login name, as an existing
|
|
|
|
|
test requires.
|
|
|
|
|
|
2026-09-24 01:19:48 +02:00
|
|
|
Four new HTTP tests cover this, and the 32 provisioner tests pass.
|
|
|
|
|
|
|
|
|
|
Released on 2026-09-24. CI published `main-6c4fcaf` as
|
|
|
|
|
`identity-provisioner@sha256:ffacd5d7…`, and the operator applied the
|
|
|
|
|
declaration on railiance01. Only the image and the new env changed; the
|
|
|
|
|
other resources were unchanged. The new pod checks out:
|
|
|
|
|
|
|
|
|
|
- `/healthz` returns 200 and `/readyz` returns 200 (`dependency: directory`).
|
|
|
|
|
- An invalid setup link returns 400.
|
|
|
|
|
- The served code carries the `autocomplete="username"` field.
|
|
|
|
|
- `PASSWORD_SETUP_SIGNIN_URL` is set.
|
|
|
|
|
|
|
|
|
|
Rollback digest: `3317a261…`. The next real setup link will show the field
|
|
|
|
|
to a user; note it here if the password manager still misbehaves.
|
2026-09-23 21:48:29 +02:00
|
|
|
|
|
|
|
|
## Plus-addressed email sign-in fails with an LDAP filter error
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
id: NK-WP-0041-T02
|
2026-09-24 01:57:20 +02:00
|
|
|
status: wait
|
2026-09-23 21:48:29 +02:00
|
|
|
priority: medium
|
2026-09-23 21:49:34 +02:00
|
|
|
state_hub_task_id: "83633649-3e5c-57e7-8207-609380a29c25"
|
2026-09-23 21:48:29 +02:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Authelia's `users_filter` accepts `uid` or `mail`. Sign-in as
|
|
|
|
|
`bernd.worsch+99@gmail.com` failed with "LDAP Result Code 201 Filter Compile
|
|
|
|
|
Error: invalid characters for escape", not a clean result. A probe with fake
|
|
|
|
|
addresses (`nk-probe@example.invalid` compared with `nk-probe+x@…`) showed
|
|
|
|
|
that only the `+` form triggers the error. Establish whether Authelia 4.38
|
|
|
|
|
escapes `+` DN-style inside the filter, and fix it through the reference
|
|
|
|
|
configuration or the version. Users who use plus-addressing cannot sign in
|
|
|
|
|
by email until then.
|
|
|
|
|
|
2026-09-24 01:26:57 +02:00
|
|
|
Diagnosis, 2026-09-24. This is an upstream Authelia defect.
|
|
|
|
|
`internal/authentication/ldap_util.go` `ldapEscape()` in v4.38.0 through
|
|
|
|
|
v4.38.19 (the last 4.38 release) and in v4.39.0 applies `ldap.EscapeFilter`
|
|
|
|
|
and then DN-escapes `, # + < > ; " =` as `\c`. Inside a filter, an escape
|
|
|
|
|
must be `\XX` hex, so the go-ldap filter compile fails for any username or
|
|
|
|
|
email containing one of those eight characters. v4.39.28 (2026-09-17) builds
|
|
|
|
|
the filter with `ldap.EscapeFilter(input)` only (`ldap_user_provider.go:720`).
|
|
|
|
|
No configuration workaround exists.
|
|
|
|
|
|
|
|
|
|
Upgrade prepared. The target is `authelia/authelia:4.39.28`, pinned as
|
|
|
|
|
`sha256:bd97cff4…`. The live image is the floating tag `4.38`
|
|
|
|
|
(`sha256:46021dc2…`). The live `authelia-config` was validated locally with
|
|
|
|
|
placeholder secrets under both versions: each returned exit 0 with no errors
|
|
|
|
|
and the same set of auto-mapped deprecation warnings. Storage is SQLite on
|
|
|
|
|
the PVC, and 4.39 migrates the schema on start. The rollback therefore needs
|
|
|
|
|
the pre-upgrade copy (`backups/db.sqlite3.pre-4.39.28`) as well as the old
|
|
|
|
|
digest. The daily backups continue.
|
|
|
|
|
|
2026-09-23 21:48:29 +02:00
|
|
|
## Route the portal findings to user-engine
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
id: NK-WP-0041-T03
|
2026-09-27 13:07:31 +02:00
|
|
|
status: done
|
2026-09-23 21:48:29 +02:00
|
|
|
priority: low
|
2026-09-23 21:49:34 +02:00
|
|
|
state_hub_task_id: "7350f35a-6893-5ac8-bfc5-fb2e40b3cf5d"
|
2026-09-23 21:48:29 +02:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
These belong to user-engine:
|
|
|
|
|
|
|
|
|
|
- The setup link is shown to the provider instead of being delivered to the
|
|
|
|
|
recipient (journey U04/T03: mail delivery unresolved).
|
|
|
|
|
- The derived login name (`bernd.worsch-99`) is not obvious to the recipient.
|
|
|
|
|
- There is no sign-in link from the user entry once a password has been set.
|
|
|
|
|
|
2026-09-27 13:07:31 +02:00
|
|
|
Sent, and the reply is recorded (user-engine, 2026-09-23, hub message
|
|
|
|
|
`a500cc67`, thread `21b0b7e4`). user-engine opened USER-WP-0035
|
|
|
|
|
(`35aa6adf-255a-5705-9294-a50d98e45f00`).
|
|
|
|
|
|
|
|
|
|
Findings 2 and 3 are implemented in source as USER-WP-0035-T01
|
|
|
|
|
(`a26b22ae-05fd-52f4-accc-068ccba7bbda`): the tenant-admin user entry shows
|
|
|
|
|
the sign-in address beside the login name and states that the email address
|
|
|
|
|
is not the login name, and the password-setup handoff repeats the sign-in
|
|
|
|
|
address and says the portal does not deliver the link. Their regression is
|
|
|
|
|
`test_journey_roles.UserJourneys.test_password_handoff_names_actual_login_and_failure_can_retry`.
|
|
|
|
|
That is source, not a live release. They will not suggest that email sign-in
|
|
|
|
|
works while NK-WP-0041-T02 is open.
|
|
|
|
|
|
|
|
|
|
Finding 1 stays USER-WP-0035-T02
|
|
|
|
|
(`fae614ba-ffc0-511d-9d22-2d91fbab057f`, status wait). Delivery needs the
|
|
|
|
|
governed transactional mail lane, which is operator-owned. A portal-rendered
|
|
|
|
|
link is not delivery evidence.
|
2026-09-24 01:54:12 +02:00
|
|
|
|
|
|
|
|
### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back
|
|
|
|
|
|
|
|
|
|
Timeline (UTC, 2026-09-23 on the server clock):
|
|
|
|
|
|
|
|
|
|
- **23:34.** A pre-upgrade database copy was taken
|
|
|
|
|
(`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes).
|
|
|
|
|
- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29.
|
|
|
|
|
The first start failed the LDAP startup check (network not ready) and
|
|
|
|
|
restarted clean. Health, discovery and the probes passed; the `+` address
|
|
|
|
|
became a clean not-found. KeyCape's redirect checks passed.
|
|
|
|
|
- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then
|
|
|
|
|
Authelia rejected KeyCape's back-channel token request: "Error occurred
|
|
|
|
|
determining the effective issuer … invalid X-Forwarded-Proto header value
|
|
|
|
|
'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the
|
|
|
|
|
portal, Vergabe and the OpenBao browser login. The pre-rollout checks were
|
|
|
|
|
redirect-only and could not see this. The 23:43 attempts had failed
|
|
|
|
|
separately, because of a leading space in the pasted username (4.39 does
|
|
|
|
|
not trim it).
|
|
|
|
|
- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0,
|
|
|
|
|
restore the pre-upgrade database via a helper pod (the migrated copy is
|
|
|
|
|
kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest
|
|
|
|
|
`sha256:46021dc2…`. The schema is "already up to date" (15), and health
|
|
|
|
|
returned 200. Two restarts come from the same LDAP startup race.
|
|
|
|
|
|
|
|
|
|
Exposure: about 18 minutes in which KeyCape sign-ins failed.
|
|
|
|
|
|
|
|
|
|
Before retrying: KeyCape calls Authelia's token endpoint in-cluster over
|
|
|
|
|
plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded
|
|
|
|
|
scheme. Resolve that first, either with KeyCape sending
|
|
|
|
|
`X-Forwarded-Proto: https` and the public host, or through an Authelia
|
|
|
|
|
4.39 setting for the in-cluster endpoint. Also add a real back-channel token
|
|
|
|
|
exchange to the upgrade acceptance, because redirect-only checks miss it.
|
|
|
|
|
The repository now pins the exact 4.38 digest instead of the floating tag.
|
|
|
|
|
|
|
|
|
|
Separate finding: Authelia's LDAP startup check fails on the first start
|
|
|
|
|
after a pod is scheduled, then passes on restart.
|
2026-09-24 01:57:20 +02:00
|
|
|
|
|
|
|
|
Operator confirmation after the rollback, 2026-09-24: sign-in as
|
|
|
|
|
`bernd.worsch-99` works again. `bernd.worsch+99@gmail.com` still fails on
|
|
|
|
|
4.38, as expected.
|
|
|
|
|
|
|
|
|
|
Root cause of the 4.39 break. KeyCape's `tokenBaseURL` is the in-cluster
|
|
|
|
|
`http://authelia.sso.svc.cluster.local:9091` (declared in
|
|
|
|
|
`sso-mfa/k8s/keycape/create-secrets.sh`). The token request sets no
|
|
|
|
|
forwarded headers (key-cape `adapter.go`), so Authelia sees an `http`
|
|
|
|
|
scheme. 4.38 accepts that; 4.39 will not derive an issuer from it.
|
|
|
|
|
|
|
|
|
|
Path forward. Stage the fix on 4.38 first, then upgrade with only one
|
|
|
|
|
variable changing:
|
|
|
|
|
|
|
|
|
|
- **A (preferred, config only).** Point `tokenBaseURL` at
|
|
|
|
|
`https://auth.coulomb.social`, so the call goes through the ingress with
|
|
|
|
|
`X-Forwarded-Proto: https`. First check that the KeyCape pod can reach the
|
|
|
|
|
public hostname in-cluster (hairpin). Apply through the guarded KeyCape
|
|
|
|
|
config lane with unrelated bytes preserved, then prove it with a real
|
|
|
|
|
login on 4.38.
|
|
|
|
|
- **B (code).** key-cape sends `X-Forwarded-Proto: https` and
|
|
|
|
|
`X-Forwarded-Host` on back-channel calls. Whether 4.39 trusts forwarded
|
|
|
|
|
headers from a pod is unverified.
|
|
|
|
|
|
|
|
|
|
Then retry 4.39.28. The acceptance must include a real human sign-in
|
|
|
|
|
through KeyCape, and the `+` email sign-in. The task waits on the choice
|
|
|
|
|
of A or B, with key-cape consulted.
|
2026-09-27 13:07:31 +02:00
|
|
|
|
|
|
|
|
2026-09-27, checked through the Vergabe demo-company pilot
|
|
|
|
|
(`https://vergabe-teilnahme.coulomb.social/demo-company/`, customer
|
|
|
|
|
`tenant:trial:demo-company`). The welcome page is up. Starting sign-in
|
|
|
|
|
returns 302 to KeyCape `client_id=vergabe-demo-company` with PKCE, `prompt=login`,
|
|
|
|
|
and the company callback. KeyCape then sends the browser to
|
|
|
|
|
`https://auth.coulomb.social/api/oidc/authorization` with `max_age=10` and
|
|
|
|
|
client `keycape`. No credential was submitted.
|
|
|
|
|
|
|
|
|
|
The plus-address defect is still live on that same Authelia 4.38. A first-factor
|
|
|
|
|
probe for `nk-probe@example.invalid` is logged as user not found. The same probe
|
|
|
|
|
for `nk-probe+x@example.invalid` is logged as LDAP Result Code 201 Filter Compile
|
|
|
|
|
Error on `+`. Both responses to the browser are the generic authentication
|
|
|
|
|
failure, so the filter error is not visible on the sign-in page. Vendor
|
|
|
|
|
`tenant:friendly:binky` is a different tenant and is not admitted by this
|
|
|
|
|
client.
|
|
|
|
|
|
|
|
|
|
Path A is not reachable with the current network policy. Namespace `sso` is
|
|
|
|
|
default-deny. KeyCape egress allows only TCP 9091 to Authelia pods, TCP 3890
|
|
|
|
|
to LLDAP, TCP 8080 to the mfa namespace, and DNS. The pod has no egress to
|
|
|
|
|
the ingress or to port 443, so `tokenBaseURL: https://auth.coulomb.social`
|
|
|
|
|
would fail before any hairpin. The in-cluster token URL remains the only
|
|
|
|
|
allowed back-channel. key-cape is asked whether it will send
|
|
|
|
|
`X-Forwarded-Proto: https` and `X-Forwarded-Host: auth.coulomb.social` on
|
|
|
|
|
that existing call (path B). Authelia 4.39 trusting those headers from a
|
|
|
|
|
pod, rather than from Traefik, is still unverified. No live config or image
|
|
|
|
|
was changed.
|
2026-09-27 13:52:45 +02:00
|
|
|
|
|
|
|
|
2026-09-27, path B shipped by key-cape and staged on Authelia 4.38.
|
|
|
|
|
`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED` for this
|
|
|
|
|
image change only. Commit `3b0446e`, tag `main-3b0446e`, digest
|
|
|
|
|
`sha256:7c99cd6c6fdf63afaf22e47dad31e20dcb3a8b2079206f198cb425047be495b3`.
|
|
|
|
|
The token call stays in-cluster and now sets those two headers from
|
|
|
|
|
`browserBaseURL`. Authelia remains `sha256:46021dc2…` (4.38). Rollback
|
|
|
|
|
digest `sha256:82f1e5ac…`. The 4.39.28 upgrade waits for one attended
|
|
|
|
|
sign-in through the Vergabe demo-company welcome on this image.
|