net-kingdom/sso-mfa/k8s/lldap/BIND-SPLIT-RUNBOOK.md

# LLDAP Bind Split: Founder Runbook

NK-WP-0054-T05, founder decision `9f0f7a87` (2026-10-28). Moves Authelia, KeyCape, the privacyIDEA
resolver and identity-provisioner off the shared human `admin` bind onto their own accounts, makes
`ld-admin` LLDAP's configured super-admin, and makes the `admin` password human-only. You run every
live step in your own terminal; agents prepared and rehearsed it on synthetic directories
(`docs/evidence/2026-10-28-nk54-bind-split-rehearsal/`).

No step prints a password. Passwords come from the private custody copy or a hidden prompt. Keep the
receipts under `~/nk54/`; they are value-free and go to `docs/evidence/` afterwards.

## Fixed rules

- **Order matters.** `bind-split.py accounts` runs before the LLDAP deployment switch: LLDAP never
  creates a missing configured admin once `lldap_admin` has a member (rehearsal `--stage startup`).
- **The human `admin` password stays valid until step 10.** Until then every consumer can be put back
  on the old bind, and a backup from any point restores (the restore launchers take each identity
  from the backup itself).
- One consumer at a time. Check before the next.

## Setup (once per session)

```bash
cd ~/net-kingdom && git pull --ff-only
bridge up k3s-api-railiance01
K="kubectl --kubeconfig $HOME/.kube/config-hosteurope --server https://127.0.0.1:16444"
# The Secret scripts call plain kubectl; the default config is the limited ops-mason-builder.
export KUBECONFIG=$HOME/.kube/config-railiance01
kubectl auth can-i create secrets -n sso   # must print yes
CUSTODY=sso-mfa/bootstrap/secrets      # your private custody copy, as for make creds-apply
mkdir -p -m 700 ~/nk54
```

If only the sealed `secrets.enc/` exists, decrypt it first and seal again after step 1
(`--no-shred`, steps 5-10 still read it) and after step 10 (without `--no-shred`):
```bash
make sops-custody-shell      # then, inside: bash sso-mfa/bootstrap/decrypt-secrets.sh sso-mfa/bootstrap/secrets "$SOPS_AGE_KEY_FILE"; exit
bash sso-mfa/bootstrap/encrypt-secrets.sh sso-mfa/bootstrap/secrets "" --no-shred
```
2026-10-30: the custody `AUTHELIA_KEYCAPE_CLIENT_SECRET` no longer matched the live hash, so step 6's
`create-secrets.sh` refused; see `docs/evidence/2026-10-30-nk54-bind-split/README.md`.

## 0. Preconditions

1. **Restore drill on the current backup** (NK-WP-0053-T02/T04) has passed. It proves today's backup
   before the change. Done 2026-10-29 (`docs/evidence/2026-10-29-identity-restore-drill/`).
2. **Pin LLDAP to what runs.** Done 2026-10-29: the running digest is v0.6.2 and `deployment.yaml` is pinned. Read the running digest (no Secret is read):
   ```bash
   $K -n sso get pod -l app.kubernetes.io/name=lldap -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'; echo
   ```
   If it ends in `9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d` (v0.6.2, the rehearsed
   version), set `image:` in `sso-mfa/k8s/lldap/deployment.yaml` to
   `docker.io/lldap/lldap@sha256:9e605a66…09d` and commit. **If it is anything else, stop**: the
   rehearsals must be rerun on that digest first; never pin to an older version.
3. **Provisioner image.** Done 2026-10-29: `sha256:6d1f3632…4199`, built by the Forgejo workflow from `main` after `2ebc4f0`, is pinned. Identity provider journey acceptance is green on `4261a5a` (same provisioner code; that commit only made one test runnable without pytest). Originally: built from commit `2ebc4f0` or later through its owner pipeline (it reads
   `LLDAP_ADMIN_USERNAME`). Put its digest in `sso-mfa/k8s/identity-provisioner/deployment.yaml` and
   commit. Do not deploy it yet.

## 1. Prepare custody

```bash
python3 sso-mfa/k8s/lldap/bind-split.py prepare-custody "$CUSTODY" --receipt ~/nk54/01-prepare.json
```
Adds the four `LLDAP_SVC_*_PASS` keys and replaces `LLDAP_LDAP_USER_PASS` with a new `ld-admin`
password. It refuses a second run. Re-seal the custody copy the way you normally do after a custody
change, and put the `ld-admin` password into your password safe as the sealed LLDAP service-admin
(crisis list; it is `LLDAP_LDAP_USER_PASS` in the custody copy).

## 2. Plan (read-only)

```bash
python3 sso-mfa/k8s/lldap/bind-split.py plan --endpoint https://lldap.coulomb.social --receipt ~/nk54/02-plan.json
```
Prompts for the `admin` password. Expect every target `"action": "create"` and
`admin.in_lldap_admin: true`. Any `refuse` means an existing account collides: stop.

## 3. Create the accounts

```bash
python3 sso-mfa/k8s/lldap/bind-split.py accounts --endpoint https://lldap.coulomb.social "$CUSTODY" --receipt ~/nk54/03a-accounts-preview.json
python3 sso-mfa/k8s/lldap/bind-split.py accounts --endpoint https://lldap.coulomb.social "$CUSTODY" --apply --receipt ~/nk54/03b-accounts.json
```
Additive only: nothing uses the new accounts yet. Needs docker and the pinned image locally (it sets
passwords with LLDAP's own `lldap_set_password`). Expect `signs_in` all `true`.

## 4. Verify the accounts

```bash
python3 sso-mfa/k8s/lldap/bind-split.py verify --endpoint https://lldap.coulomb.social "$CUSTODY" --receipt ~/nk54/04-verify.json
```
`"verdict": "pass"` before any consumer moves.

## 5. identity-provisioner

```bash
bash sso-mfa/k8s/identity-provisioner/create-secrets.sh "$CUSTODY"
$K apply -f sso-mfa/k8s/identity-provisioner/deployment.yaml
$K -n sso rollout status deploy/identity-provisioner --timeout=180s
```
Its readiness includes a directory login, so `rollout status` succeeding means `svc-provisioner`
works. Rollback: `$K -n sso rollout undo deploy/identity-provisioner`.

## 6. Authelia

```bash
bash sso-mfa/k8s/authelia/create-secrets.sh "$CUSTODY" --rotate-key LLDAP_SVC_AUTHELIA_PASS
$K apply -f sso-mfa/k8s/authelia/configmap.yaml
$K -n sso rollout restart deploy/authelia && $K -n sso rollout status deploy/authelia --timeout=180s
```
Check: sign in at `https://auth.coulomb.social` as yourself (password and OTP).
Rollback (the `admin` password still works until step 10):
```bash
git show 145ca8e~1:sso-mfa/k8s/authelia/configmap.yaml | $K apply -f -
read -rsp 'admin password: ' P; echo
printf '[{"op":"replace","path":"/data/ldap_password","value":"%s"}]' "$(printf %s "$P" | base64 -w0)" \
  | $K -n sso patch secret authelia-secrets --type=json --patch-file=/dev/stdin; unset P
$K -n sso rollout restart deploy/authelia
```

## 7. KeyCape

```bash
python3 sso-mfa/k8s/keycape/directory-bind-rollout.py inspect --custody-root "$CUSTODY"
# note before.uid and before.resource_version, then:
python3 sso-mfa/k8s/keycape/directory-bind-rollout.py dry-run --custody-root "$CUSTODY" --expected-uid <uid> --expected-resource-version <rv>
python3 sso-mfa/k8s/keycape/directory-bind-rollout.py apply --custody-root "$CUSTODY" --expected-uid <uid> --expected-resource-version <rv> --attended
$K -n sso rollout restart deploy/keycape && $K -n sso rollout status deploy/keycape --timeout=180s
```
Expect `prior_bind: human`, `change_needed: true`. Check: a KeyCape sign-in (for example the users
portal or the OpenBao UI through `openbao-ui-railiance01`). Rollback: the same three commands with
`--operation revert` (prompts for the `admin` password), then restart.

## 8. privacyIDEA resolver

```bash
python3 sso-mfa/k8s/privacyidea/realm_bootstrap.py "$CUSTODY" --switch-bind               # preview
python3 sso-mfa/k8s/privacyidea/realm_bootstrap.py "$CUSTODY" --switch-bind --apply
```
Expect `actions: ["switch-bind"]`, one write; a `"result":"unverified"` line on stderr means it did not
apply. Check: as `pi-admin`, Config → Users → `lldap-coulomb` shows Bind DN
`uid=svc-privacyidea,…` and **Test LDAP Resolver** passes. An OTP sign-in alone does not prove the
switch: until step 10 retires the old password a resolver still bound as `admin` keeps working
(2026-10-30 incident, `docs/evidence/2026-10-30-nk54-bind-split/README.md`). There is no
automated rollback; the old bind is the `admin` account, which still works, so the attended repair
(`--attended`, prompts) can set it back if needed.

## 9. LLDAP

```bash
bash sso-mfa/k8s/lldap/create-secrets.sh "$CUSTODY" --rotate-key LLDAP_LDAP_USER_PASS
$K apply -f sso-mfa/k8s/lldap/deployment.yaml
$K -n sso rollout status deploy/lldap --timeout=180s
```
The configured super-admin is now `ld-admin`. Check: sign in to the LLDAP UI as `ld-admin` once, then
as `admin` (still admin through `lldap_admin`).

## 10. Make the human password human-only

Sign in to the LLDAP UI as `admin`, change your password, store it in your password safe. Then:
```bash
python3 sso-mfa/k8s/lldap/bind-split.py verify --endpoint https://lldap.coulomb.social "$CUSTODY" --old-admin-retired --receipt ~/nk54/10-verify.json
```
It prompts for the new `admin` password, then the old shared one, and requires that the old one is
refused. `"verdict": "pass"` ends the live run once a fresh OTP sign-in still succeeds afterwards
(this is the first check that no consumer still binds with the old shared password).

## 11. Afterwards

- End-to-end: a full KeyCape sign-in with MFA, and one provisioning call if one is due.
- Copy `~/nk54/*.json` to `docs/evidence/<date>-nk54-bind-split/` and commit.
- After the next daily backup, run the restore drill on that generation (NK-WP-0053 stages 2 to 5).
  It proves T17/T19 under the new accounts and closes NK-WP-0054-T05; then exception SI-X1 is retired.