tegwick
5a52dc8a23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 673304@bnt-lap001 Assistant-Session: 959b12c7-cf2e-440b-a60d-91229cc9e96e
56 lines
4.4 KiB
Markdown
56 lines
4.4 KiB
Markdown
# LLDAP bind split, live run 2026-10-30 (NK-WP-0054-T05)
|
|
|
|
Founder-run in the founder's terminal following `sso-mfa/k8s/lldap/BIND-SPLIT-RUNBOOK.md`, 17:39-18:53
|
|
UTC, agent (Claude Code) guiding. All receipts are value-free (`values_recorded: false`).
|
|
|
|
| Step | Result | Receipt |
|
|
| --- | --- | --- |
|
|
| 1 prepare custody | five keys changed; custody copy re-sealed to `keys/age.pub` | `01-prepare.json` |
|
|
| 2 plan | five `create`, `admin` in `lldap_admin`, no collision | `02-plan.json` |
|
|
| 3 accounts | preview, then apply; all five sign in | `03a-accounts-preview.json`, `03b-accounts.json` |
|
|
| 4 verify | `pass` | `04-verify.json` |
|
|
| 5 identity-provisioner | Secret `identity-provisioner-ldap` created; image `sha256:6d1f3632…4199` ready (readiness includes directory login) | operator output |
|
|
| 6 Authelia | `ldap_password` rotated to `svc-authelia`; configmap applied; founder sign-in with password and OTP | operator output |
|
|
| 7 KeyCape | `directory-bind-rollout.py apply`, uid unchanged, `other_secret_data_unchanged: true`; sign-in ok | operator output |
|
|
| 8 privacyIDEA resolver | `realm_bootstrap.py --switch-bind --apply`; OTP sign-in ok | operator output |
|
|
| 9 LLDAP | `LLDAP_LDAP_USER_PASS` rotated; configured super-admin `ld-admin` | operator output |
|
|
| 10 retire shared password | `admin` password changed by the founder; old shared password refused; `pass` | `10-verify.json` |
|
|
|
|
## Deviations and findings
|
|
|
|
1. **Custody copy not at the runbook path.** Step 1 failed with `FileNotFoundError` (nothing written):
|
|
only the sealed `sso-mfa/bootstrap/secrets.enc/` exists. The founder decrypted it with
|
|
`make sops-custody-shell` + `decrypt-secrets.sh`, ran step 1, re-sealed with `--no-shred`, and sealed
|
|
and shredded after step 10. Runbook setup now says so.
|
|
2. **Wrong kubeconfig for the Secret scripts.** `apply-service-secrets.py` calls plain `kubectl`, which
|
|
used the limited default identity (`ops-mason-builder`, cannot read `sso` Secrets); step 5 was
|
|
refused before any write (presence check: `NotFound`). Fixed by
|
|
`export KUBECONFIG=$HOME/.kube/config-railiance01`; the runbook setup now exports it.
|
|
3. **Stale custody value: `AUTHELIA_KEYCAPE_CLIENT_SECRET`.** Step 6 was refused before any write: a
|
|
value-free comparison showed the four other Authelia values equal to live, `ldap_password` different
|
|
(intended), and the custody client secret **not** matching the live `keycape_client_secret_hash`.
|
|
The KeyCape-Authelia client secret was changed after the March custody seal without updating custody.
|
|
Step 6 used a one-off helper that calls the same `publish()` path for `ldap_password` only (uid,
|
|
resourceVersion and full-data test, readback) and skips only the client-secret check. Reconciled the
|
|
same evening with `sso-mfa/bootstrap/reconcile-keycape-client-secret.py` (founder-run, value-free:
|
|
live pair agreed, custody updated and re-checked, re-sealed).
|
|
4. Rollouts of identity-provisioner, KeyCape and LLDAP use `Recreate`: each had a short outage of
|
|
about 30 s.
|
|
|
|
5. **privacyIDEA resolver was still bound as `admin` (found 2026-10-30 evening).** After step 10
|
|
every OTP check failed: KeyCape logged `mfa_failed` (`openbao-admin`) and `mfa_check_error`
|
|
(`user-engine-portal`); privacyIDEA logged `LDAP Bind unsuccessful: invalidCredentials (49)` from
|
|
19:00:10 UTC, last successful lookup 18:49:30 UTC, step 10 at 18:53 UTC. Founder found Bind DN
|
|
`uid=admin,…` on `lldap-coulomb` in the privacyIDEA UI and changed it to `uid=svc-privacyidea,…`;
|
|
the Bind DN persisted but the hand-entered bind password did not match. A one-off attended script
|
|
(reusing `realm_bootstrap.py`'s client and custody reader) then wrote only `BINDPW` from the custody
|
|
`LLDAP_SVC_PRIVACYIDEA_PASS`, checked the other resolver fields unchanged and resolved
|
|
`platform-root` through the resolver; the founder's OpenBao OIDC login with OTP then succeeded.
|
|
Prefer the custody-fed write over UI entry: the German UI offered only the two test buttons, and
|
|
pasting the value by hand is error-prone. No repository tool rewrites the
|
|
resolver, so step 8's apply most likely did not take effect; its OTP check passed only because the
|
|
old shared password was still valid. Runbook step 8 now requires the resolver Bind DN and test, and
|
|
step 10 a fresh OTP sign-in.
|
|
|
|
Closing condition still open: the restore drill on the first backup generation after this run
|
|
(NK-WP-0053 stages 2-5, T17/T19 under the new accounts).
|