net-kingdom/workplans/NK-WP-0041-onboarding-journey-usability.md
tegwick ee8f9c8a67
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
NK-WP-0041-T02: rollback confirmed; 4.39 root cause and staged fix path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 299762@bnt-lap001
Assistant-Session: d3d3cea1-869c-44f1-be2a-3d6d3550e72e
2026-09-24 01:57:20 +02:00

174 lines
7.7 KiB
Markdown

---
id: NK-WP-0041
type: workplan
title: "Fix onboarding-journey defects found in the 2026-09-23 human run"
domain: infotech
repo: net-kingdom
status: active
flavor: implementation
owner: claude-code
topic_slug: netkingdom
created: "2026-09-23"
updated: "2026-09-23"
related: [NK-WP-0036, NK-WP-0037, KEY-WP-0033]
state_hub_workstream_id: "98168f50-7a4d-5bb5-a462-1e031563b89f"
---
Found by the operator's native onboarding run (NK-WP-0036-T05) and the
blocked Vergabe sign-in (NK-WP-0037-T02). Each item names its owner, and only
items in this repository are implemented here.
## Password managers and sign-in after password setup
```task
id: NK-WP-0041-T01
status: done
priority: medium
state_hub_task_id: "82ce581a-b635-5b5b-9b08-91b61e7eb9bd"
```
The setup form had no username field, so browsers stored the new password
without a login. After setup there was no way on to the sign-in page. The
source is fixed in `identity-provisioner`:
- The setup form shows the grant's login name as a read-only
`autocomplete="username"` field. The submitted value is ignored, and the
form no longer sets `autocomplete="off"`.
- The completion page links to `PASSWORD_SETUP_SIGNIN_URL` (HTTPS only)
when there is no company return. The deployment declares
`https://users.coulomb.social/`.
- The completion page still does not show the login name, as an existing
test requires.
Four new HTTP tests cover this, and the 32 provisioner tests pass.
Released on 2026-09-24. CI published `main-6c4fcaf` as
`identity-provisioner@sha256:ffacd5d7…`, and the operator applied the
declaration on railiance01. Only the image and the new env changed; the
other resources were unchanged. The new pod checks out:
- `/healthz` returns 200 and `/readyz` returns 200 (`dependency: directory`).
- An invalid setup link returns 400.
- The served code carries the `autocomplete="username"` field.
- `PASSWORD_SETUP_SIGNIN_URL` is set.
Rollback digest: `3317a261…`. The next real setup link will show the field
to a user; note it here if the password manager still misbehaves.
## Plus-addressed email sign-in fails with an LDAP filter error
```task
id: NK-WP-0041-T02
status: wait
priority: medium
state_hub_task_id: "83633649-3e5c-57e7-8207-609380a29c25"
```
Authelia's `users_filter` accepts `uid` or `mail`. Sign-in as
`bernd.worsch+99@gmail.com` failed with "LDAP Result Code 201 Filter Compile
Error: invalid characters for escape", not a clean result. A probe with fake
addresses (`nk-probe@example.invalid` compared with `nk-probe+x@…`) showed
that only the `+` form triggers the error. Establish whether Authelia 4.38
escapes `+` DN-style inside the filter, and fix it through the reference
configuration or the version. Users who use plus-addressing cannot sign in
by email until then.
Diagnosis, 2026-09-24. This is an upstream Authelia defect.
`internal/authentication/ldap_util.go` `ldapEscape()` in v4.38.0 through
v4.38.19 (the last 4.38 release) and in v4.39.0 applies `ldap.EscapeFilter`
and then DN-escapes `, # + < > ; " =` as `\c`. Inside a filter, an escape
must be `\XX` hex, so the go-ldap filter compile fails for any username or
email containing one of those eight characters. v4.39.28 (2026-09-17) builds
the filter with `ldap.EscapeFilter(input)` only (`ldap_user_provider.go:720`).
No configuration workaround exists.
Upgrade prepared. The target is `authelia/authelia:4.39.28`, pinned as
`sha256:bd97cff4…`. The live image is the floating tag `4.38`
(`sha256:46021dc2…`). The live `authelia-config` was validated locally with
placeholder secrets under both versions: each returned exit 0 with no errors
and the same set of auto-mapped deprecation warnings. Storage is SQLite on
the PVC, and 4.39 migrates the schema on start. The rollback therefore needs
the pre-upgrade copy (`backups/db.sqlite3.pre-4.39.28`) as well as the old
digest. The daily backups continue.
## Route the portal findings to user-engine
```task
id: NK-WP-0041-T03
status: todo
priority: low
state_hub_task_id: "7350f35a-6893-5ac8-bfc5-fb2e40b3cf5d"
```
These belong to user-engine:
- The setup link is shown to the provider instead of being delivered to the
recipient (journey U04/T03: mail delivery unresolved).
- The derived login name (`bernd.worsch-99`) is not obvious to the recipient.
- There is no sign-in link from the user entry once a password has been set.
Send these to user-engine and record the reply.
### Incident 2026-09-24: 4.39.28 rollout broke KeyCape sign-in; rolled back
Timeline (UTC, 2026-09-23 on the server clock):
- **23:34.** A pre-upgrade database copy was taken
(`backups/db.sqlite3.pre-4.39.28`, 2,023,424 bytes).
- **23:35.** The operator rolled out v4.39.28. The schema migrated 15 → 29.
The first start failed the LDAP startup check (network not ready) and
restarted clean. Health, discovery and the probes passed; the `+` address
became a clean not-found. KeyCape's redirect checks passed.
- **23:45 and 23:49.** Real sign-ins passed Authelia's first factor. Then
Authelia rejected KeyCape's back-channel token request: "Error occurred
determining the effective issuer … invalid X-Forwarded-Proto header value
'http'" (`POST /api/oidc/token`). Every KeyCape sign-in was broken: the
portal, Vergabe and the OpenBao browser login. The pre-rollout checks were
redirect-only and could not see this. The 23:43 attempts had failed
separately, because of a leading space in the pasted username (4.39 does
not trim it).
- **23:53.** Rolled back with `/tmp/authelia-rollback.sh`: scale to 0,
restore the pre-upgrade database via a helper pod (the migrated copy is
kept as `backups/db.sqlite3.4.39.28-migrated`), then the 4.38 digest
`sha256:46021dc2…`. The schema is "already up to date" (15), and health
returned 200. Two restarts come from the same LDAP startup race.
Exposure: about 18 minutes in which KeyCape sign-ins failed.
Before retrying: KeyCape calls Authelia's token endpoint in-cluster over
plain HTTP, and 4.39 will not derive its issuer from an `http` forwarded
scheme. Resolve that first, either with KeyCape sending
`X-Forwarded-Proto: https` and the public host, or through an Authelia
4.39 setting for the in-cluster endpoint. Also add a real back-channel token
exchange to the upgrade acceptance, because redirect-only checks miss it.
The repository now pins the exact 4.38 digest instead of the floating tag.
Separate finding: Authelia's LDAP startup check fails on the first start
after a pod is scheduled, then passes on restart.
Operator confirmation after the rollback, 2026-09-24: sign-in as
`bernd.worsch-99` works again. `bernd.worsch+99@gmail.com` still fails on
4.38, as expected.
Root cause of the 4.39 break. KeyCape's `tokenBaseURL` is the in-cluster
`http://authelia.sso.svc.cluster.local:9091` (declared in
`sso-mfa/k8s/keycape/create-secrets.sh`). The token request sets no
forwarded headers (key-cape `adapter.go`), so Authelia sees an `http`
scheme. 4.38 accepts that; 4.39 will not derive an issuer from it.
Path forward. Stage the fix on 4.38 first, then upgrade with only one
variable changing:
- **A (preferred, config only).** Point `tokenBaseURL` at
`https://auth.coulomb.social`, so the call goes through the ingress with
`X-Forwarded-Proto: https`. First check that the KeyCape pod can reach the
public hostname in-cluster (hairpin). Apply through the guarded KeyCape
config lane with unrelated bytes preserved, then prove it with a real
login on 4.38.
- **B (code).** key-cape sends `X-Forwarded-Proto: https` and
`X-Forwarded-Host` on back-channel calls. Whether 4.39 trusts forwarded
headers from a pod is unverified.
Then retry 4.39.28. The acceptance must include a real human sign-in
through KeyCape, and the `+` email sign-in. The task waits on the choice
of A or B, with key-cape consulted.