diff --git a/docs/keycape-exposure-resolver-reconciliation.md b/docs/keycape-exposure-resolver-reconciliation.md index 94c7142..136f7ec 100644 --- a/docs/keycape-exposure-resolver-reconciliation.md +++ b/docs/keycape-exposure-resolver-reconciliation.md @@ -2,7 +2,7 @@ Incident: `KEYCAPE-EXPOSURE-20260823-01` Workplan: `NK-WP-0033` -NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (revision pinned by the approved checkout) +NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (repaired 2026-08-27 in `4a38511`; the previously pinned revision could not complete a run — see NK-WP-0033 T05) Platform recovery contract: railiance-platform `453fed3` Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2` @@ -99,3 +99,27 @@ T03 may move to done only after the helper run and cleanup receipt are recorded by the attended operator. T05 may move to done only after the resolver’s replacement success, predecessor denial, owner cutover receipt, and all residual limitations are recorded as sanitized evidence. + +## Revision note — 2026-08-27 + +The pinned revision of `reconcile-lldap-resolver-live.sh` had never completed a +run. Four defects were found by running it and are fixed in `4a38511`; see +`NK-WP-0033` T05 for the full findings. Two change how this procedure is +invoked: + +- `--predecessor-unavailable` — use when the exposed predecessor cannot be + produced. The denial bind is not attempted and the receipt records + `predecessor denial=NOT-PROVEN`. Do **not** type a placeholder at the + predecessor prompt instead: a wrong value also fails the bind, and the run + records it as a *passing* denial proof — a receipt asserting a test that never + ran. +- `--note TEXT` — operator context carried verbatim in the receipt line, so a + claim and its caveat travel together. One line, 200 characters, no + credentials. + +`TIMEOUT`, `CACHE_TIMEOUT` and `SIZELIMIT` are now sent with the resolver body +(default 5 / 120 / 500, overridable via the matching `LDAP_*` environment +variables). Before this fix, `--apply` dropped them, because a resolver write +replaces the whole object. A resolver with them unset still resolves users, but +the WebUI refuses to save or test it — so a hand repair was silently reverted by +the next run. diff --git a/sso-mfa/k8s/privacyidea/bootstrap-realm.sh b/sso-mfa/k8s/privacyidea/bootstrap-realm.sh index 738ec14..347f529 100755 --- a/sso-mfa/k8s/privacyidea/bootstrap-realm.sh +++ b/sso-mfa/k8s/privacyidea/bootstrap-realm.sh @@ -35,6 +35,13 @@ REALM_NAME="coulomb" LLDAP_URL="ldap://lldap.sso.svc.cluster.local:3890" LLDAP_BASE_DN="dc=netkingdom,dc=local" LLDAP_BIND_DN="uid=admin,ou=people,dc=netkingdom,dc=local" +# Numeric resolver parameters. A resolver with these unset still resolves +# users, but the WebUI refuses to save or test it until they are filled in by +# hand, and a write that omits them drops whatever was there (NK-WP-0033, +# 2026-08-27). +LDAP_TIMEOUT="${LDAP_TIMEOUT:-5}" +LDAP_CACHE_TIMEOUT="${LDAP_CACHE_TIMEOUT:-120}" +LDAP_SIZELIMIT="${LDAP_SIZELIMIT:-500}" PASS_COUNT=0 FAIL_COUNT=0 @@ -152,7 +159,10 @@ body = { 'USERINFO': json.dumps({\"username\": \"uid\", \"phone\": \"telephoneNumber\", \"mobile\": \"mobile\", \"email\": \"mail\", \"surname\": \"sn\", \"givenname\": \"givenName\"}), 'UIDTYPE': 'uid', 'NOREFERRALS': True, - 'NOSCHEMAS': True + 'NOSCHEMAS': True, + 'TIMEOUT': int('$LDAP_TIMEOUT'), + 'CACHE_TIMEOUT': int('$LDAP_CACHE_TIMEOUT'), + 'SIZELIMIT': int('$LDAP_SIZELIMIT') } print(json.dumps(body)) ") diff --git a/sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh b/sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh index b62ccc5..541d40c 100755 --- a/sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh +++ b/sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh @@ -10,12 +10,50 @@ # Usage: # ./reconcile-lldap-resolver-live.sh --check # read-only proof # ./reconcile-lldap-resolver-live.sh --apply # resolver update + proof +# +# Options: +# --note TEXT operator context recorded verbatim in the receipt +# --predecessor-unavailable the exposed predecessor cannot be produced; the +# denial bind is not attempted and the receipt +# records it as NOT-PROVEN set -euo pipefail -MODE="${1:-}" -if [[ "$MODE" != "--apply" && "$MODE" != "--check" || "${2:-}" != "" ]]; then - echo "Usage: $0 --check|--apply" >&2 +MODE="" +NOTE="" +PREDECESSOR="required" +usage() { + echo "Usage: $0 --check|--apply [--note TEXT] [--predecessor-unavailable]" >&2 + exit 2 +} +while [[ $# -gt 0 ]]; do + case "$1" in + --check|--apply) + if [[ -n "$MODE" ]]; then usage; fi + MODE="$1"; shift ;; + --note) + # Never put a credential here: the receipt goes to a terminal and + # is copied into the incident record. + if [[ -z "${2:-}" ]]; then usage; fi + NOTE="$2"; shift 2 ;; + --predecessor-unavailable) + # Typing a placeholder at the predecessor prompt also fails the + # bind, and would be recorded as a passing denial proof. An absent + # proof is honest; a fabricated one asserts a test that never ran. + PREDECESSOR="unavailable"; shift ;; + *) + usage ;; + esac +done +if [[ "$MODE" != "--apply" && "$MODE" != "--check" ]]; then + usage +fi +# One bounded line: a receipt that breaks its own format is not evidence +# anyone can read back. +NOTE="${NOTE//$'\n'/ }" +NOTE="${NOTE//$'\r'/ }" +if (( ${#NOTE} > 200 )); then + echo "ERROR: --note must be 200 characters or fewer." >&2 exit 2 fi if [[ ! -t 0 ]]; then @@ -32,6 +70,20 @@ RESOLVER_NAME="${RESOLVER_NAME:-lldap-coulomb}" MFA_USER="${MFA_USER:-platform-root}" MFA_REALM="${MFA_REALM:-coulomb}" KEYCAPE_DISCOVERY_URL="${KEYCAPE_DISCOVERY_URL:-https://kc.coulomb.social/.well-known/openid-configuration}" +# A resolver write replaces the whole object, so every field the body omits is +# dropped. A resolver with these unset still resolves users, but the WebUI +# refuses to save or test it until they are filled in — so omitting them here +# silently un-repairs a resolver an operator fixed by hand. +LDAP_TIMEOUT="${LDAP_TIMEOUT:-5}" +LDAP_CACHE_TIMEOUT="${LDAP_CACHE_TIMEOUT:-120}" +LDAP_SIZELIMIT="${LDAP_SIZELIMIT:-500}" +for _v in LDAP_TIMEOUT LDAP_CACHE_TIMEOUT LDAP_SIZELIMIT; do + if [[ ! "${!_v}" =~ ^[0-9]+$ ]]; then + echo "ERROR: $_v must be a non-negative integer, got '${!_v}'." >&2 + exit 2 + fi +done +unset _v tmp="$(mktemp -d)" chmod 700 "$tmp" @@ -62,12 +114,17 @@ prompt_secret() { prompt_secret "privacyIDEA pi-admin password" "$tmp/pi-admin" prompt_secret "replacement LLDAP bind/admin password" "$tmp/lldap-new" -prompt_secret "exposed predecessor LLDAP password (denial proof only)" "$tmp/lldap-old" +if [[ "$PREDECESSOR" == "required" ]]; then + prompt_secret "exposed predecessor LLDAP password (denial proof only)" "$tmp/lldap-old" +else + echo " [INFO] predecessor unavailable: denial proof recorded as NOT-PROVEN." >&2 +fi prompt_secret "one-time MFA code for $MFA_USER@$MFA_REALM" "$tmp/otp" if python3 - "$tmp/pi-admin" "$tmp/lldap-new" "$tmp/lldap-old" "$tmp/otp" \ "$PI_URL" "$LLDAP_AUTH_URL" "$LLDAP_URL" "$LLDAP_BASE_DN" "$LLDAP_BIND_DN" \ - "$RESOLVER_NAME" "$MFA_USER" "$MFA_REALM" "$KEYCAPE_DISCOVERY_URL" "$MODE" "$tmp/phase" <<'PY' + "$RESOLVER_NAME" "$MFA_USER" "$MFA_REALM" "$KEYCAPE_DISCOVERY_URL" "$MODE" "$tmp/phase" "$PREDECESSOR" \ + "$LDAP_TIMEOUT" "$LDAP_CACHE_TIMEOUT" "$LDAP_SIZELIMIT" <<'PY' import json import subprocess import sys @@ -79,6 +136,7 @@ from pathlib import Path ( pi_path, new_path, old_path, otp_path, pi_url, lldap_auth_url, lldap_url, base_dn, bind_dn, resolver_name, mfa_user, mfa_realm, discovery_url, mode, phase_path, + predecessor, ldap_timeout, ldap_cache_timeout, ldap_sizelimit, ) = sys.argv[1:] def phase(name: str) -> None: @@ -91,7 +149,14 @@ def secret(path: str) -> str: return value def request(url: str, payload: dict | None = None, token: str | None = None) -> tuple[int, dict | None]: - headers = {"Content-Type": "application/json"} + # Content-Type is only set on requests with a body — Werkzeug 3.x raises + # BadRequest if Content-Type: application/json is sent on a bodyless GET, + # and the rejection happens in front of privacyIDEA, so the reply is an HTML + # error page rather than a JSON result. Same fix as bootstrap-realm.sh's + # pi_api helper. + headers = {} + if payload is not None: + headers["Content-Type"] = "application/json" if token: headers["Authorization"] = token data = json.dumps(payload).encode("utf-8") if payload is not None else None @@ -155,10 +220,15 @@ try: new_status, new_authenticated = lldap_login(secret(new_path)) if new_status != 200 or not new_authenticated: raise RuntimeError("replacement LLDAP authentication failed") - phase("predecessor-denial") - old_status, old_authenticated = lldap_login(secret(old_path)) - if old_status not in (401, 403) or old_authenticated: - raise RuntimeError("predecessor LLDAP authentication was not denied") + if predecessor == "unavailable": + # Not a pass. The bind is not attempted, so this run claims nothing + # about the predecessor's disposition. + phase("predecessor-denial-not-proven") + else: + phase("predecessor-denial") + old_status, old_authenticated = lldap_login(secret(old_path)) + if old_status not in (401, 403) or old_authenticated: + raise RuntimeError("predecessor LLDAP authentication was not denied") resolver_body = { "type": "ldapresolver", "LDAPURI": lldap_url, "BINDDN": bind_dn, @@ -167,6 +237,8 @@ try: "LDAPFILTER": "(&(objectClass=inetOrgPerson)(uid=%s))", "USERINFO": json.dumps({"username": "uid", "phone": "telephoneNumber", "mobile": "mobile", "email": "mail", "surname": "sn", "givenname": "givenName"}), "UIDTYPE": "uid", "NOREFERRALS": True, "NOSCHEMAS": True, + "TIMEOUT": int(ldap_timeout), "CACHE_TIMEOUT": int(ldap_cache_timeout), + "SIZELIMIT": int(ldap_sizelimit), } if mode == "--apply": phase("resolver-update") @@ -181,7 +253,13 @@ try: pi_url + f"/user/?realm={mfa_realm}&username={urllib.parse.quote(mfa_user)}", token=pi_token, ) - users = (result or {}).get("result", {}).get("value", {}).get("users", []) + # privacyIDEA returns result.value as a list of user objects for /user/. + # Accept the dict-with-"users" shape too: other endpoints use it, and a + # lookup that guesses wrong raises AttributeError past the except clause. + _value = (result or {}).get("result", {}).get("value", []) + if isinstance(_value, dict): + _value = _value.get("users", []) + users = _value if isinstance(_value, list) else [] if status != 200: phase(f"resolver-lookup-http-{status}") raise RuntimeError("replacement resolver lookup failed") @@ -201,7 +279,10 @@ try: phase("postflight") check_k8s_ready() check_health() -except (OSError, RuntimeError, subprocess.SubprocessError) as exc: +except (OSError, RuntimeError, subprocess.SubprocessError, + AttributeError, TypeError, KeyError, ValueError) as exc: + # A malformed reply must still produce a receipt naming the phase it died + # in. A traceback tells the operator nothing about what was proven. raise SystemExit(1) PY then @@ -211,18 +292,27 @@ else fi phase_result="$(cat "$tmp/phase" 2>/dev/null || echo unknown)" +if [[ "$PREDECESSOR" == "required" ]]; then + pred_text="predecessor denial" +else + pred_text="predecessor denial=NOT-PROVEN" +fi +note_text="" +if [[ -n "$NOTE" ]]; then + note_text="; note=$NOTE" +fi cleanup trap - EXIT INT TERM if [[ "$rc" -ne 0 ]]; then - echo "NK-WP-0033 receipt FAIL: phase=$phase_result; cleanup=PASS" >&2 + echo "NK-WP-0033 receipt FAIL: phase=$phase_result; $pred_text; cleanup=PASS$note_text" >&2 exit "$rc" fi if [[ -d "$tmp" ]]; then - echo "NK-WP-0033 receipt FAIL: cleanup=FAIL" >&2 + echo "NK-WP-0033 receipt FAIL: cleanup=FAIL$note_text" >&2 exit 1 fi if [[ "$MODE" == "--check" ]]; then - echo "NK-WP-0033 receipt PASS: read-only resolver lookup, privacyIDEA MFA, predecessor denial, readiness, health, cleanup=PASS" + echo "NK-WP-0033 receipt PASS: read-only resolver lookup, privacyIDEA MFA, $pred_text, readiness, health, cleanup=PASS$note_text" else - echo "NK-WP-0033 receipt PASS: resolver lookup, privacyIDEA MFA, predecessor denial, readiness, health, cleanup=PASS" + echo "NK-WP-0033 receipt PASS: resolver lookup, privacyIDEA MFA, $pred_text, readiness, health, cleanup=PASS$note_text" fi diff --git a/workplans/NK-WP-0033-keycape-secret-exposure-rotation.md b/workplans/NK-WP-0033-keycape-secret-exposure-rotation.md index df61c5e..11bfa72 100644 --- a/workplans/NK-WP-0033-keycape-secret-exposure-rotation.md +++ b/workplans/NK-WP-0033-keycape-secret-exposure-rotation.md @@ -168,3 +168,53 @@ rollout status, positive authentication/MFA outcomes, predecessor rejection or expiry, cleanup receipts, and residual limitations. Close the incident only after all four credential classes have an authoritative disposition and no temporary plaintext or stale bundle remains. + +### Attended session 2026-08-27 — findings + +The resolver reconciliation was attempted and the operator's `platform-root` +login is restored. `bao login -method=oidc` succeeds, and +`GET /user/?realm=coulomb&username=platform-root` returns the record. **No +green receipt has been produced yet**; the run below is the state of record. + +**LLDAP bind credential — reconciled.** The `--apply` run wrote the current +`LLDAP_LDAP_USER_PASS` into resolver `lldap-coulomb`, and the replacement +authenticated against LLDAP in the same run. The resolver had been holding the +pre-cutover value since 2026-08-23, which is what T04 left outstanding. + +**Predecessor disposition — NOT PROVEN by receipt; observed twice by hand.** +The exposed value is dead: the operator was locked out of the LLDAP WebUI on +2026-08-27 after logging out of a persisted session, and the `--check` run the +same day passed `predecessor-denial` with the real value before it was lost. +Neither observation is carried by an emitted receipt. The value is +unrecoverable — see the custody gap below. + +**Custody gap — the actual root cause of this session.** T04 replaced the +credential without a step that updates operator custody, and there was no +custody store to update it into. `platform-root-custody.md` names a password +safe entry `net-kingdom/LLDAP/admin`; **no KeePassXC database exists** and never +did. The value lived only in a Firefox password-manager entry dated 2026-06-28, +which was overwritten with the replacement on 2026-08-27, destroying the +predecessor. Any rotation runbook that does not name where the outgoing value is +retained will lose it the same way. + +**The reconciliation script had never completed a run.** Four defects, fixed in +`4a38511`: `Content-Type` on bodyless GETs (HTTP 400 on every GET), `/user/` +parsed as a dict when it returns a list (traceback past the except clause), +resolver writes dropping `TIMEOUT`/`CACHE_TIMEOUT`/`SIZELIMIT` (silently +un-repairing a hand-fixed resolver), and no way to declare an unavailable +predecessor (forcing a placeholder that records a *passing* denial proof). + +**Verification blind spot — open.** `verify-t06.sh` reported success at +bootstrap against a resolver created without the three tuning parameters, by a +reconciliation path that could not execute. A verification that passes while its +subject cannot run is not verification. This is the finding worth acting on +beyond the four fixes, and it is not yet addressed. + +**Diagnosis correction.** The "stale bind credential" reading in T04 was real but +was not what blocked the resolver lookup; the HTTP 400 was our own request +builder throughout. The two faults were independent and looked like one, which +is why each partial fix appeared to change nothing. + +Remaining before T05 can close: a green receipt from the repaired script, and a +ruling on whether a predecessor disposition observed but not receipted is an +acceptable close for this incident.