NK-WP-0033 add attended resolver reconciliation receipt

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02929-244b-7391-b933-c04010e8eedb

docs/keycape-exposure-resolver-reconciliation.md

@@ -2,7 +2,7 @@
 
 Incident: `KEYCAPE-EXPOSURE-20260823-01`  
 Workplan: `NK-WP-0033`  
-NetKingdom procedure revision: `f2e578c`  
+NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (revision pinned by the approved checkout)
 Platform recovery contract: railiance-platform `453fed3`  
 Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2`
 
@@ -19,7 +19,7 @@ State Hub, Git, chat, command arguments, or ordinary logs.
 The operator must record a private approval receipt containing, at minimum:
 
 - incident `KEYCAPE-EXPOSURE-20260823-01`;
-- NetKingdom revision `f2e578c` and platform contract `453fed3`;
+- the NetKingdom revision containing `reconcile-lldap-resolver-live.sh` and platform contract `453fed3`;
 - the owner receipt `45b236c8-052f-43d3-a472-44f8e9694da2`;
 - the exact start/end window, attended driver, and independent abort operator;
 - confirmation that the replacement LLDAP credential is the provider-approved
@@ -33,35 +33,36 @@ identifier differs from the receipt.
 Run from the approved operator workstation, with the cluster context and
 provider endpoint already authorized. Do not render any Secret data.
 
-1. Verify the checked-out revision is exactly `f2e578c` and the helper has mode
-   `0755`; inspect its source, not live credential material.
+1. Verify the approved checkout contains `reconcile-lldap-resolver-live.sh`
+   with mode `0755`; inspect its source, not live credential material.
 2. Confirm LLDAP, privacyIDEA, KeyCape, Authelia, and
    identity-provisioner are Ready using deployment/pod status fields only.
-3. Confirm privacyIDEA and KeyCape health endpoints return an HTTP success
-   status, discarding response bodies. Do not use a command that prints a
-   bearer token or configuration response.
+3. Confirm privacyIDEA availability via the documented unauthenticated
+   `/token/` probe (401/403 is expected) and KeyCape discovery health,
+   discarding response bodies. Do not use a command that prints a bearer token
+   or configuration response.
 4. Confirm the approved window, driver, abort operator, provider custody, and
    cleanup workspace are ready. Stop on any drift or missing owner.
 
 ## Apply (one attended operation)
 
-1. Create one private mode-`0700` workspace with a cleanup trap. Keep the
-   pi-admin password and replacement LLDAP bind password in separate
-   mode-`0600` files or supply them only through the helper's hidden prompts.
-2. From the pinned checkout, run exactly:
+1. From the approved checkout, run exactly:
 
    ```text
-   bash sso-mfa/k8s/privacyidea/update-lldap-resolver-live.sh --apply
+   bash sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh --apply
    ```
 
-   The helper prompts for both passwords, authenticates to privacyIDEA, and
-   performs one `POST /resolver/lldap-coulomb`. It passes only protected file
-   paths to its child process, never prints values, and prints only a boolean
-   result. It must not be combined with `repair-realm-live.sh`,
-   `bootstrap-realm.sh`, `creds-rotate.sh`, or any full-bundle generator.
+   The helper securely prompts for pi-admin, replacement and predecessor
+   LLDAP passwords, and one approved MFA code; authenticates to privacyIDEA;
+   performs exactly one `POST /resolver/lldap-coulomb`; proves replacement
+   resolver lookup, privacyIDEA MFA, replacement LLDAP authentication, and
+   predecessor denial; rechecks readiness/health; securely cleans up; and
+   emits one sanitized PASS/FAIL receipt. It must not be combined with
+   `repair-realm-live.sh`, `bootstrap-realm.sh`, `creds-rotate.sh`, or any
+   full-bundle generator.
 3. Stop immediately on any non-success response, timeout, unexpected endpoint,
-   or output that is not the documented boolean result. Do not restore the
-   exposed bundle or predecessor credential.
+   or failed cleanup. Do not restore the exposed bundle or predecessor
+   credential.
 
 ## Postflight and predecessor denial
 

sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh

@@ -0,0 +1,198 @@
+#!/usr/bin/env bash
+# reconcile-lldap-resolver-live.sh — one-command attended resolver cutover.
+#
+# Performs the narrow privacyIDEA lldap-coulomb update and its safe proof set:
+# replacement LLDAP authentication, resolver lookup, one privacyIDEA MFA check,
+# predecessor LLDAP denial, health/readiness, and cleanup. It never reads a
+# Kubernetes Secret and never prints a credential, token, response body, or
+# manifest.
+#
+# Usage:
+#   ./reconcile-lldap-resolver-live.sh --apply
+
+set -euo pipefail
+
+if [[ "${1:-}" != "--apply" || "${2:-}" != "" ]]; then
+    echo "Usage: $0 --apply" >&2
+    exit 2
+fi
+if [[ ! -t 0 ]]; then
+    echo "ERROR: --apply requires an interactive terminal." >&2
+    exit 2
+fi
+
+PI_URL="${PI_URL:-https://pink.coulomb.social}"
+LLDAP_AUTH_URL="${LLDAP_AUTH_URL:-https://lldap.coulomb.social/auth/simple/login}"
+LLDAP_URL="${LLDAP_URL:-ldap://lldap.sso.svc.cluster.local:3890}"
+LLDAP_BASE_DN="${LLDAP_BASE_DN:-dc=netkingdom,dc=local}"
+LLDAP_BIND_DN="${LLDAP_BIND_DN:-uid=admin,ou=people,dc=netkingdom,dc=local}"
+RESOLVER_NAME="${RESOLVER_NAME:-lldap-coulomb}"
+MFA_USER="${MFA_USER:-platform-root}"
+MFA_REALM="${MFA_REALM:-coulomb}"
+KEYCAPE_DISCOVERY_URL="${KEYCAPE_DISCOVERY_URL:-https://kc.coulomb.social/.well-known/openid-configuration}"
+
+tmp="$(mktemp -d)"
+chmod 700 "$tmp"
+cleanup() {
+    for file in pi-admin lldap-new lldap-old otp; do
+        if [[ -f "$tmp/$file" ]]; then
+            shred -u "$tmp/$file" 2>/dev/null || rm -f "$tmp/$file"
+        fi
+    done
+    rmdir "$tmp" 2>/dev/null || true
+}
+cleanup_ok=0
+trap cleanup EXIT INT TERM
+
+prompt_secret() {
+    local label="$1" target="$2"
+    printf '%s: ' "$label" >&2
+    IFS= read -r -s value
+    printf '\n' >&2
+    if [[ -z "$value" ]]; then
+        echo "ERROR: empty protected input." >&2
+        exit 2
+    fi
+    printf '%s' "$value" > "$target"
+    unset value
+    chmod 600 "$target"
+}
+
+prompt_secret "privacyIDEA pi-admin password" "$tmp/pi-admin"
+prompt_secret "replacement LLDAP bind/admin password" "$tmp/lldap-new"
+prompt_secret "exposed predecessor LLDAP password (denial proof only)" "$tmp/lldap-old"
+prompt_secret "one-time MFA code for $MFA_USER@$MFA_REALM" "$tmp/otp"
+
+if python3 - "$tmp/pi-admin" "$tmp/lldap-new" "$tmp/lldap-old" "$tmp/otp" \
+    "$PI_URL" "$LLDAP_AUTH_URL" "$LLDAP_URL" "$LLDAP_BASE_DN" "$LLDAP_BIND_DN" \
+    "$RESOLVER_NAME" "$MFA_USER" "$MFA_REALM" "$KEYCAPE_DISCOVERY_URL" <<'PY'
+import json
+import subprocess
+import sys
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+(
+    pi_path, new_path, old_path, otp_path, pi_url, lldap_auth_url, lldap_url,
+    base_dn, bind_dn, resolver_name, mfa_user, mfa_realm, discovery_url,
+) = sys.argv[1:]
+
+def secret(path: str) -> str:
+    value = Path(path).read_text(encoding="utf-8")
+    if not value:
+        raise RuntimeError("empty protected input")
+    return value
+
+def request(url: str, payload: dict | None = None, token: str | None = None) -> tuple[int, dict | None]:
+    headers = {"Content-Type": "application/json"}
+    if token:
+        headers["Authorization"] = token
+    data = json.dumps(payload).encode("utf-8") if payload is not None else None
+    req = urllib.request.Request(url, data=data, headers=headers, method="POST" if payload is not None else "GET")
+    try:
+        with urllib.request.urlopen(req, timeout=20) as response:
+            status = response.status
+            body = response.read()
+        if not body:
+            return status, None
+        try:
+            return status, json.loads(body)
+        except json.JSONDecodeError:
+            return status, None
+    except urllib.error.HTTPError as exc:
+        return exc.code, None
+    except (urllib.error.URLError, TimeoutError):
+        return 0, None
+
+def check_k8s_ready() -> None:
+    workloads = (("sso", "lldap"), ("mfa", "privacyidea"), ("sso", "keycape"),
+                 ("sso", "authelia"), ("sso", "identity-provisioner"))
+    for namespace, name in workloads:
+        proc = subprocess.run(
+            ["kubectl", "get", "deployment", name, "-n", namespace,
+             "-o", "jsonpath={.status.readyReplicas}/{.status.replicas}"],
+            capture_output=True, text=True, timeout=20,
+        )
+        if proc.returncode != 0 or proc.stdout.strip() != "1/1":
+            raise RuntimeError(f"readiness failed: {namespace}/{name}")
+
+def check_health() -> None:
+    # privacyIDEA intentionally exposes no unauthenticated HTTP health route;
+    # /token/ is documented by the deployment as a safe availability probe.
+    for label, url, accepted in (
+        ("privacyidea", pi_url + "/token/", {200, 401, 403}),
+        ("keycape", discovery_url, set(range(200, 400))),
+    ):
+        status, _ = request(url)
+        if status not in accepted:
+            raise RuntimeError(f"health failed: {label}")
+
+def lldap_login(password: str) -> tuple[int, bool]:
+    status, body = request(lldap_auth_url, {"username": "admin", "password": password})
+    return status, status == 200 and isinstance(body, dict) and bool(body.get("token"))
+
+try:
+    check_k8s_ready()
+    check_health()
+
+    status, auth = request(pi_url + "/auth", {"username": "pi-admin", "password": secret(pi_path)})
+    pi_token = str((auth or {}).get("result", {}).get("value", {}).get("token", ""))
+    if status != 200 or not pi_token:
+        raise RuntimeError("privacyIDEA authentication failed")
+
+    resolver_body = {
+        "type": "ldapresolver", "LDAPURI": lldap_url, "BINDDN": bind_dn,
+        "BINDPW": secret(new_path), "LDAPBASE": base_dn,
+        "LOGINNAMEATTRIBUTE": "uid", "LDAPSEARCHFILTER": "(objectClass=inetOrgPerson)",
+        "LDAPFILTER": "(&(objectClass=inetOrgPerson)(uid=%s))",
+        "USERINFO": json.dumps({"username": "uid", "phone": "telephoneNumber", "mobile": "mobile", "email": "mail", "surname": "sn", "givenname": "givenName"}),
+        "UIDTYPE": "uid", "NOREFERRALS": True, "NOSCHEMAS": True,
+    }
+    status, result = request(pi_url + "/resolver/" + resolver_name, resolver_body, pi_token)
+    if status != 200 or (result or {}).get("result", {}).get("status") not in (True, "true", "True"):
+        raise RuntimeError("resolver update failed")
+
+    status, result = request(pi_url + f"/user/?realm={mfa_realm}&pagesize=1", token=pi_token)
+    users = (result or {}).get("result", {}).get("value", {}).get("users", [])
+    if status != 200 or not users:
+        raise RuntimeError("replacement resolver lookup failed")
+
+    status, result = request(
+        pi_url + "/validate/check",
+        {"user": mfa_user, "realm": mfa_realm, "pass": secret(otp_path)},
+        pi_token,
+    )
+    if status != 200 or (result or {}).get("result", {}).get("value") is not True:
+        raise RuntimeError("replacement MFA validation failed")
+
+    new_status, new_authenticated = lldap_login(secret(new_path))
+    if new_status != 200 or not new_authenticated:
+        raise RuntimeError("replacement LLDAP authentication failed")
+    old_status, old_authenticated = lldap_login(secret(old_path))
+    if old_status not in (401, 403) or old_authenticated:
+        raise RuntimeError("predecessor LLDAP authentication was not denied")
+
+    check_k8s_ready()
+    check_health()
+except (OSError, RuntimeError, subprocess.SubprocessError) as exc:
+    print("NK-WP-0033 receipt FAIL: proof checks did not pass", file=sys.stderr)
+    raise SystemExit(1)
+PY
+then
+    rc=0
+else
+    rc=$?
+fi
+
+cleanup
+trap - EXIT INT TERM
+if [[ "$rc" -ne 0 ]]; then
+    echo "NK-WP-0033 receipt FAIL: proof checks did not pass; cleanup=PASS" >&2
+    exit "$rc"
+fi
+if [[ -d "$tmp" ]]; then
+    echo "NK-WP-0033 receipt FAIL: cleanup=FAIL" >&2
+    exit 1
+fi
+echo "NK-WP-0033 receipt PASS: resolver lookup, privacyIDEA MFA, predecessor denial, readiness, health, cleanup=PASS"

workplans/NK-WP-0033-keycape-secret-exposure-rotation.md

@@ -117,11 +117,13 @@ clients are a list, so the verifier now uses Authelia's supported template
 filter over the Secret-mounted file. The revision was applied live and
 Authelia returned Ready with startup complete.
 
-NetKingdom also added the unattended-safe shape of the remaining provider
-operation in `sso-mfa/k8s/privacyidea/update-lldap-resolver-live.sh`. It is
-explicitly gated by `--apply`, requires an interactive terminal, uses protected
-0600 files, updates only `lldap-coulomb`, and emits no credential values. It
-has not been run; the exact attended runbook is pinned in
+NetKingdom now provides the supported one-command attended operation in
+`sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh`. It is explicitly
+gated by `--apply`, requires an interactive terminal, uses protected 0600
+files, updates only `lldap-coulomb`, proves replacement resolver lookup and
+privacyIDEA MFA, proves predecessor LLDAP denial with explicit 401/403 status,
+rechecks readiness/health, and emits one sanitized PASS/FAIL receipt after
+cleanup. It has not been run; the exact attended runbook is pinned in
 `docs/keycape-exposure-resolver-reconciliation.md`.
 
 Do not use `sso-mfa/bootstrap/creds-rotate.sh` through an agent as currently