docs(nk-wp-0033): record the 2026-08-27 attended session findings

platform-root login is restored and the resolver holds the current bind
credential, so T04's outstanding reconciliation is done. No green receipt
yet — the run of record is a FAIL at resolver-lookup, and T05 stays open.

Records four things the session established:

- the predecessor is dead, observed twice by hand, but not by receipt;
- the value is unrecoverable, because no KeePassXC database has ever
  existed despite platform-root-custody.md naming a safe entry, and the
  only copy lived in a Firefox entry overwritten during the session.
  T04 replaced the credential with no step to update operator custody —
  the root cause of the whole session;
- the reconciliation script had never completed a run (4a38511);
- verify-t06.sh reported success at bootstrap against a resolver that
  was misconfigured and a reconciliation path that could not execute.
  That blind spot is open and is the finding worth acting on.

Also corrects the record: the stale bind credential was real but did not
cause the lookup failure. The HTTP 400 was our own request builder.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 3377672@bnt-lap001
Assistant-Session: 15463ccf-238f-4e13-b163-93aa25c6d166

docs/keycape-exposure-resolver-reconciliation.md

@@ -2,7 +2,7 @@
 
 Incident: `KEYCAPE-EXPOSURE-20260823-01`  
 Workplan: `NK-WP-0033`  
-NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (revision pinned by the approved checkout)
+NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (repaired 2026-08-27 in `4a38511`; the previously pinned revision could not complete a run — see NK-WP-0033 T05)
 Platform recovery contract: railiance-platform `453fed3`  
 Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2`
 
@@ -99,3 +99,27 @@ T03 may move to done only after the helper run and cleanup receipt are recorded
 by the attended operator. T05 may move to done only after the resolver’s
 replacement success, predecessor denial, owner cutover receipt, and all
 residual limitations are recorded as sanitized evidence.
+
+## Revision note — 2026-08-27
+
+The pinned revision of `reconcile-lldap-resolver-live.sh` had never completed a
+run. Four defects were found by running it and are fixed in `4a38511`; see
+`NK-WP-0033` T05 for the full findings. Two change how this procedure is
+invoked:
+
+- `--predecessor-unavailable` — use when the exposed predecessor cannot be
+  produced. The denial bind is not attempted and the receipt records
+  `predecessor denial=NOT-PROVEN`. Do **not** type a placeholder at the
+  predecessor prompt instead: a wrong value also fails the bind, and the run
+  records it as a *passing* denial proof — a receipt asserting a test that never
+  ran.
+- `--note TEXT` — operator context carried verbatim in the receipt line, so a
+  claim and its caveat travel together. One line, 200 characters, no
+  credentials.
+
+`TIMEOUT`, `CACHE_TIMEOUT` and `SIZELIMIT` are now sent with the resolver body
+(default 5 / 120 / 500, overridable via the matching `LDAP_*` environment
+variables). Before this fix, `--apply` dropped them, because a resolver write
+replaces the whole object. A resolver with them unset still resolves users, but
+the WebUI refuses to save or test it — so a hand repair was silently reverted by
+the next run.

sso-mfa/k8s/privacyidea/bootstrap-realm.sh

@@ -35,6 +35,13 @@ REALM_NAME="coulomb"
 LLDAP_URL="ldap://lldap.sso.svc.cluster.local:3890"
 LLDAP_BASE_DN="dc=netkingdom,dc=local"
 LLDAP_BIND_DN="uid=admin,ou=people,dc=netkingdom,dc=local"
+# Numeric resolver parameters. A resolver with these unset still resolves
+# users, but the WebUI refuses to save or test it until they are filled in by
+# hand, and a write that omits them drops whatever was there (NK-WP-0033,
+# 2026-08-27).
+LDAP_TIMEOUT="${LDAP_TIMEOUT:-5}"
+LDAP_CACHE_TIMEOUT="${LDAP_CACHE_TIMEOUT:-120}"
+LDAP_SIZELIMIT="${LDAP_SIZELIMIT:-500}"
 
 PASS_COUNT=0
 FAIL_COUNT=0
@@ -152,7 +159,10 @@ body = {
     'USERINFO': json.dumps({\"username\": \"uid\", \"phone\": \"telephoneNumber\", \"mobile\": \"mobile\", \"email\": \"mail\", \"surname\": \"sn\", \"givenname\": \"givenName\"}),
     'UIDTYPE': 'uid',
     'NOREFERRALS': True,
-    'NOSCHEMAS': True
+    'NOSCHEMAS': True,
+    'TIMEOUT': int('$LDAP_TIMEOUT'),
+    'CACHE_TIMEOUT': int('$LDAP_CACHE_TIMEOUT'),
+    'SIZELIMIT': int('$LDAP_SIZELIMIT')
 }
 print(json.dumps(body))
 ")

sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh

@@ -10,12 +10,50 @@
 # Usage:
 #   ./reconcile-lldap-resolver-live.sh --check   # read-only proof
 #   ./reconcile-lldap-resolver-live.sh --apply   # resolver update + proof
+#
+# Options:
+#   --note TEXT                 operator context recorded verbatim in the receipt
+#   --predecessor-unavailable   the exposed predecessor cannot be produced; the
+#                               denial bind is not attempted and the receipt
+#                               records it as NOT-PROVEN
 
 set -euo pipefail
 
-MODE="${1:-}"
-if [[ "$MODE" != "--apply" && "$MODE" != "--check" || "${2:-}" != "" ]]; then
-    echo "Usage: $0 --check|--apply" >&2
+MODE=""
+NOTE=""
+PREDECESSOR="required"
+usage() {
+    echo "Usage: $0 --check|--apply [--note TEXT] [--predecessor-unavailable]" >&2
+    exit 2
+}
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --check|--apply)
+            if [[ -n "$MODE" ]]; then usage; fi
+            MODE="$1"; shift ;;
+        --note)
+            # Never put a credential here: the receipt goes to a terminal and
+            # is copied into the incident record.
+            if [[ -z "${2:-}" ]]; then usage; fi
+            NOTE="$2"; shift 2 ;;
+        --predecessor-unavailable)
+            # Typing a placeholder at the predecessor prompt also fails the
+            # bind, and would be recorded as a passing denial proof. An absent
+            # proof is honest; a fabricated one asserts a test that never ran.
+            PREDECESSOR="unavailable"; shift ;;
+        *)
+            usage ;;
+    esac
+done
+if [[ "$MODE" != "--apply" && "$MODE" != "--check" ]]; then
+    usage
+fi
+# One bounded line: a receipt that breaks its own format is not evidence
+# anyone can read back.
+NOTE="${NOTE//$'\n'/ }"
+NOTE="${NOTE//$'\r'/ }"
+if (( ${#NOTE} > 200 )); then
+    echo "ERROR: --note must be 200 characters or fewer." >&2
     exit 2
 fi
 if [[ ! -t 0 ]]; then
@@ -32,6 +70,20 @@ RESOLVER_NAME="${RESOLVER_NAME:-lldap-coulomb}"
 MFA_USER="${MFA_USER:-platform-root}"
 MFA_REALM="${MFA_REALM:-coulomb}"
 KEYCAPE_DISCOVERY_URL="${KEYCAPE_DISCOVERY_URL:-https://kc.coulomb.social/.well-known/openid-configuration}"
+# A resolver write replaces the whole object, so every field the body omits is
+# dropped. A resolver with these unset still resolves users, but the WebUI
+# refuses to save or test it until they are filled in — so omitting them here
+# silently un-repairs a resolver an operator fixed by hand.
+LDAP_TIMEOUT="${LDAP_TIMEOUT:-5}"
+LDAP_CACHE_TIMEOUT="${LDAP_CACHE_TIMEOUT:-120}"
+LDAP_SIZELIMIT="${LDAP_SIZELIMIT:-500}"
+for _v in LDAP_TIMEOUT LDAP_CACHE_TIMEOUT LDAP_SIZELIMIT; do
+    if [[ ! "${!_v}" =~ ^[0-9]+$ ]]; then
+        echo "ERROR: $_v must be a non-negative integer, got '${!_v}'." >&2
+        exit 2
+    fi
+done
+unset _v
 
 tmp="$(mktemp -d)"
 chmod 700 "$tmp"
@@ -62,12 +114,17 @@ prompt_secret() {
 
 prompt_secret "privacyIDEA pi-admin password" "$tmp/pi-admin"
 prompt_secret "replacement LLDAP bind/admin password" "$tmp/lldap-new"
-prompt_secret "exposed predecessor LLDAP password (denial proof only)" "$tmp/lldap-old"
+if [[ "$PREDECESSOR" == "required" ]]; then
+    prompt_secret "exposed predecessor LLDAP password (denial proof only)" "$tmp/lldap-old"
+else
+    echo "  [INFO] predecessor unavailable: denial proof recorded as NOT-PROVEN." >&2
+fi
 prompt_secret "one-time MFA code for $MFA_USER@$MFA_REALM" "$tmp/otp"
 
 if python3 - "$tmp/pi-admin" "$tmp/lldap-new" "$tmp/lldap-old" "$tmp/otp" \
     "$PI_URL" "$LLDAP_AUTH_URL" "$LLDAP_URL" "$LLDAP_BASE_DN" "$LLDAP_BIND_DN" \
-    "$RESOLVER_NAME" "$MFA_USER" "$MFA_REALM" "$KEYCAPE_DISCOVERY_URL" "$MODE" "$tmp/phase" <<'PY'
+    "$RESOLVER_NAME" "$MFA_USER" "$MFA_REALM" "$KEYCAPE_DISCOVERY_URL" "$MODE" "$tmp/phase" "$PREDECESSOR" \
+    "$LDAP_TIMEOUT" "$LDAP_CACHE_TIMEOUT" "$LDAP_SIZELIMIT" <<'PY'
 import json
 import subprocess
 import sys
@@ -79,6 +136,7 @@ from pathlib import Path
 (
     pi_path, new_path, old_path, otp_path, pi_url, lldap_auth_url, lldap_url,
     base_dn, bind_dn, resolver_name, mfa_user, mfa_realm, discovery_url, mode, phase_path,
+    predecessor, ldap_timeout, ldap_cache_timeout, ldap_sizelimit,
 ) = sys.argv[1:]
 
 def phase(name: str) -> None:
@@ -91,7 +149,14 @@ def secret(path: str) -> str:
     return value
 
 def request(url: str, payload: dict | None = None, token: str | None = None) -> tuple[int, dict | None]:
-    headers = {"Content-Type": "application/json"}
+    # Content-Type is only set on requests with a body — Werkzeug 3.x raises
+    # BadRequest if Content-Type: application/json is sent on a bodyless GET,
+    # and the rejection happens in front of privacyIDEA, so the reply is an HTML
+    # error page rather than a JSON result. Same fix as bootstrap-realm.sh's
+    # pi_api helper.
+    headers = {}
+    if payload is not None:
+        headers["Content-Type"] = "application/json"
     if token:
         headers["Authorization"] = token
     data = json.dumps(payload).encode("utf-8") if payload is not None else None
@@ -155,10 +220,15 @@ try:
     new_status, new_authenticated = lldap_login(secret(new_path))
     if new_status != 200 or not new_authenticated:
         raise RuntimeError("replacement LLDAP authentication failed")
-    phase("predecessor-denial")
-    old_status, old_authenticated = lldap_login(secret(old_path))
-    if old_status not in (401, 403) or old_authenticated:
-        raise RuntimeError("predecessor LLDAP authentication was not denied")
+    if predecessor == "unavailable":
+        # Not a pass. The bind is not attempted, so this run claims nothing
+        # about the predecessor's disposition.
+        phase("predecessor-denial-not-proven")
+    else:
+        phase("predecessor-denial")
+        old_status, old_authenticated = lldap_login(secret(old_path))
+        if old_status not in (401, 403) or old_authenticated:
+            raise RuntimeError("predecessor LLDAP authentication was not denied")
 
     resolver_body = {
         "type": "ldapresolver", "LDAPURI": lldap_url, "BINDDN": bind_dn,
@@ -167,6 +237,8 @@ try:
         "LDAPFILTER": "(&(objectClass=inetOrgPerson)(uid=%s))",
         "USERINFO": json.dumps({"username": "uid", "phone": "telephoneNumber", "mobile": "mobile", "email": "mail", "surname": "sn", "givenname": "givenName"}),
         "UIDTYPE": "uid", "NOREFERRALS": True, "NOSCHEMAS": True,
+        "TIMEOUT": int(ldap_timeout), "CACHE_TIMEOUT": int(ldap_cache_timeout),
+        "SIZELIMIT": int(ldap_sizelimit),
     }
     if mode == "--apply":
         phase("resolver-update")
@@ -181,7 +253,13 @@ try:
         pi_url + f"/user/?realm={mfa_realm}&username={urllib.parse.quote(mfa_user)}",
         token=pi_token,
     )
-    users = (result or {}).get("result", {}).get("value", {}).get("users", [])
+    # privacyIDEA returns result.value as a list of user objects for /user/.
+    # Accept the dict-with-"users" shape too: other endpoints use it, and a
+    # lookup that guesses wrong raises AttributeError past the except clause.
+    _value = (result or {}).get("result", {}).get("value", [])
+    if isinstance(_value, dict):
+        _value = _value.get("users", [])
+    users = _value if isinstance(_value, list) else []
     if status != 200:
         phase(f"resolver-lookup-http-{status}")
         raise RuntimeError("replacement resolver lookup failed")
@@ -201,7 +279,10 @@ try:
     phase("postflight")
     check_k8s_ready()
     check_health()
-except (OSError, RuntimeError, subprocess.SubprocessError) as exc:
+except (OSError, RuntimeError, subprocess.SubprocessError,
+        AttributeError, TypeError, KeyError, ValueError) as exc:
+    # A malformed reply must still produce a receipt naming the phase it died
+    # in. A traceback tells the operator nothing about what was proven.
     raise SystemExit(1)
 PY
 then
@@ -211,18 +292,27 @@ else
 fi
 
 phase_result="$(cat "$tmp/phase" 2>/dev/null || echo unknown)"
+if [[ "$PREDECESSOR" == "required" ]]; then
+    pred_text="predecessor denial"
+else
+    pred_text="predecessor denial=NOT-PROVEN"
+fi
+note_text=""
+if [[ -n "$NOTE" ]]; then
+    note_text="; note=$NOTE"
+fi
 cleanup
 trap - EXIT INT TERM
 if [[ "$rc" -ne 0 ]]; then
-    echo "NK-WP-0033 receipt FAIL: phase=$phase_result; cleanup=PASS" >&2
+    echo "NK-WP-0033 receipt FAIL: phase=$phase_result; $pred_text; cleanup=PASS$note_text" >&2
     exit "$rc"
 fi
 if [[ -d "$tmp" ]]; then
-    echo "NK-WP-0033 receipt FAIL: cleanup=FAIL" >&2
+    echo "NK-WP-0033 receipt FAIL: cleanup=FAIL$note_text" >&2
     exit 1
 fi
 if [[ "$MODE" == "--check" ]]; then
-    echo "NK-WP-0033 receipt PASS: read-only resolver lookup, privacyIDEA MFA, predecessor denial, readiness, health, cleanup=PASS"
+    echo "NK-WP-0033 receipt PASS: read-only resolver lookup, privacyIDEA MFA, $pred_text, readiness, health, cleanup=PASS$note_text"
 else
-    echo "NK-WP-0033 receipt PASS: resolver lookup, privacyIDEA MFA, predecessor denial, readiness, health, cleanup=PASS"
+    echo "NK-WP-0033 receipt PASS: resolver lookup, privacyIDEA MFA, $pred_text, readiness, health, cleanup=PASS$note_text"
 fi

workplans/NK-WP-0033-keycape-secret-exposure-rotation.md

@@ -168,3 +168,53 @@ rollout status, positive authentication/MFA outcomes, predecessor rejection or
 expiry, cleanup receipts, and residual limitations. Close the incident only
 after all four credential classes have an authoritative disposition and no
 temporary plaintext or stale bundle remains.
+
+### Attended session 2026-08-27 — findings
+
+The resolver reconciliation was attempted and the operator's `platform-root`
+login is restored. `bao login -method=oidc` succeeds, and
+`GET /user/?realm=coulomb&username=platform-root` returns the record. **No
+green receipt has been produced yet**; the run below is the state of record.
+
+**LLDAP bind credential — reconciled.** The `--apply` run wrote the current
+`LLDAP_LDAP_USER_PASS` into resolver `lldap-coulomb`, and the replacement
+authenticated against LLDAP in the same run. The resolver had been holding the
+pre-cutover value since 2026-08-23, which is what T04 left outstanding.
+
+**Predecessor disposition — NOT PROVEN by receipt; observed twice by hand.**
+The exposed value is dead: the operator was locked out of the LLDAP WebUI on
+2026-08-27 after logging out of a persisted session, and the `--check` run the
+same day passed `predecessor-denial` with the real value before it was lost.
+Neither observation is carried by an emitted receipt. The value is
+unrecoverable — see the custody gap below.
+
+**Custody gap — the actual root cause of this session.** T04 replaced the
+credential without a step that updates operator custody, and there was no
+custody store to update it into. `platform-root-custody.md` names a password
+safe entry `net-kingdom/LLDAP/admin`; **no KeePassXC database exists** and never
+did. The value lived only in a Firefox password-manager entry dated 2026-06-28,
+which was overwritten with the replacement on 2026-08-27, destroying the
+predecessor. Any rotation runbook that does not name where the outgoing value is
+retained will lose it the same way.
+
+**The reconciliation script had never completed a run.** Four defects, fixed in
+`4a38511`: `Content-Type` on bodyless GETs (HTTP 400 on every GET), `/user/`
+parsed as a dict when it returns a list (traceback past the except clause),
+resolver writes dropping `TIMEOUT`/`CACHE_TIMEOUT`/`SIZELIMIT` (silently
+un-repairing a hand-fixed resolver), and no way to declare an unavailable
+predecessor (forcing a placeholder that records a *passing* denial proof).
+
+**Verification blind spot — open.** `verify-t06.sh` reported success at
+bootstrap against a resolver created without the three tuning parameters, by a
+reconciliation path that could not execute. A verification that passes while its
+subject cannot run is not verification. This is the finding worth acting on
+beyond the four fixes, and it is not yet addressed.
+
+**Diagnosis correction.** The "stale bind credential" reading in T04 was real but
+was not what blocked the resolver lookup; the HTTP 400 was our own request
+builder throughout. The two faults were independent and looked like one, which
+is why each partial fix appeared to change nothing.
+
+Remaining before T05 can close: a green receipt from the repaired script, and a
+ruling on whether a predecessor disposition observed but not receipted is an
+acceptable close for this incident.