net-kingdom/docs/keycape-exposure-resolver-reconciliation.md

# KeyCape exposure: privacyIDEA resolver reconciliation

Incident: `KEYCAPE-EXPOSURE-20260823-01`
Historical workplan: `NK-WP-0033`; source maintenance: `NK-WP-0049-T02`
Exercise status: core helper exercised 2026-10-14 by custodian-codex against
local disposable privacyIDEA 3.12.2 and LLDAP, including actual LDAP bind,
password update, preserved configuration, exact user lookup and HOTP. Terminal
handling and failure paths have automated tests. The complete new attended
procedure has not run against the production cluster or KeyCape.
Evidence: Custodian `docs/evidence/2026-10-14-netkingdom-security/resolver-cutover.json`.

Both `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` and
`update-lldap-resolver-live.sh` delegate to `resolver_cutover.py`. The update
helper changes only an existing `lldap-coulomb` resolver's bind password; the
reconcile helper additionally checks cluster readiness, discovery, exact user
resolution, token-backed MFA and the supplied predecessor's HTTP denial.
Neither helper bootstraps a missing resolver or realm, repairs missing tuning,
or changes policies, tokens, directory credentials or Kubernetes resources.

## Authority and target pins

The historical platform recovery contract is railiance-platform `453fed3`;
owner cutover receipt is State Hub message
`45b236c8-052f-43d3-a472-44f8e9694da2`. These identify historical evidence,
not a fresh live grant. Founder-deferred exposure rotation remains deferred.

Before any live run, the existing owner/supervisor path must bind a private
approval record to the current NetKingdom source revision and dependencies,
incident, provider-approved replacement, owner receipt, exact start/end window,
attended driver, independent abort operator and recovery plan. Confirm predecessor
provenance separately. Do not restore an exposed predecessor or bundle.
A preview digest and TTY are technical preconditions, not authorization.

Reconcile requires an explicit absolute kubeconfig, context, expected API server
and `kube-system` namespace UID from the reviewed owner packet. It checks the
server and UID before workload reads. It never reads Kubernetes Secrets. The
five existing deployments must each have the current generation observed and
exactly one desired, actual, updated, ready and available replica, without a
deletion timestamp. No ambient default context is accepted.

Default service origins are `https://pink.coulomb.social`,
`https://lldap.coulomb.social` and `https://kc.coulomb.social`. Explicit literal
loopback URLs with ports are supported for approved forwards or fixtures.
Environment URL/tuning overrides and the old `--note` option are removed.
KeyCape discovery must identify the canonical `https://kc.coulomb.social`
issuer and its own authorization, token and JWKS endpoints, even through a forward.
The shared transport disables proxies and redirects and bounds strict JSON
responses; each extra service profile permits only its one required route and
never receives the privacyIDEA authorization token.

## Preview, apply and check

Use an attended terminal from the reviewed checkout. These are command templates;
replace public placeholders with the approved target values. No secret belongs
in argv, environment, Git, State Hub, chat or ordinary logs.

```bash
bash sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh --preview \
  --kubeconfig /absolute/approved/kubeconfig --context APPROVED_CONTEXT \
  --server https://APPROVED_SERVER:6443 --cluster-uid APPROVED_NAMESPACE_UID
```

Preview prompts only for pi-admin, authenticates with an admin-role check,
validates the existing resolver/core/tuning/realm, and returns a public-state
`snapshot_sha256`. It performs no configuration write. It creates an auth session
and can produce provider audit events. The digest excludes the bind password and
binds the target settings and observed non-secret configuration. It is not a
credential fingerprint.

Run the same command with `--apply --expected-digest REVIEWED_DIGEST` in place
of `--preview`, retaining all target and user options. The helper prompts silently
for pi-admin, the approved replacement and the actual predecessor. Inputs stay
in Python memory; no temporary secret files are created. Inherited shell tracing
is disabled before input handling. A fresh OTP/PIN is requested only after
configuration and exact user lookup succeed. `--user` defaults to `platform-root`.

Before mutation the helper authenticates the replacement to LLDAP and uses
privacyIDEA `/resolver/test` with the candidate password to prove an actual LDAP
bind. It tests supplied predecessor denial at LLDAP HTTP login and rechecks the
observed configuration. Apply sends exactly one resolver update containing only
`type` and `BINDPW`. Native privacyIDEA 3.12.2 preserves omitted fields. Readback
must preserve all other resolver data and every realm; a native resolver test
then proves the stored bind works. The API has no compare-and-swap: an exclusive
owner window remains required, and ambiguous writes are never retried.

`--check` replaces `--preview` to perform the proofs without a configuration
write; it needs no digest. Check proves the stored bind works, not that its
censored value equals the supplied replacement. Authentication, audit and OTP
counter effects still occur. It is not an effect-free operation.

The smaller `update-lldap-resolver-live.sh` supports the same preview/apply/check
sequence without cluster flags. It proves candidate and stored binds plus exact
configuration preservation, but does not claim MFA, readiness or predecessor
acceptance. It must not substitute for full incident acceptance.

## Receipts and recovery

Receipts contain fixed phase names, booleans, write counts and a public-state
digest; no raw responses, exceptions or free-form notes. `configured` means the
update helper completed its apply; `checked` means its no-write proof completed.
Reconcile returns `verified` only when all helper proofs pass. Passing a wrong
arbitrary password also produces denial: `supplied_predecessor_denied` is not
proof of predecessor provenance. The owner must match the input to custody.

Use `--predecessor-unavailable` honestly when it cannot be produced. Reconcile
then returns `incomplete`, exit 2, even if the other proofs pass; never type a
placeholder to obtain a passing receipt. Failures and handled interruptions
return `unverified`, exit 1, with `write_attempted` showing whether a partial
change may exist. Exit 0 still leaves `incident_acceptance: unverified`: the
separate owner receipt and actual KeyCape MFA path are outside this helper.

Stop on drift, wrong target, failed health/readiness or any unverified result.
After an attempted write, reconcile observed state through the approved owner
path before another attempt. Recover forward with an approved replacement; no
automatic rollback to an exposed predecessor. Record only sanitized evidence,
source revision, target references and operator outcome. NK-WP-0049 T03 remains
the owner-native recovery/acceptance gate; this source repair does not close it.

## Historical receipts and corrections

The old `--check --predecessor-unavailable` printed PASS on 2026-09-23 when run by
Bernd Worsch from railiance01 checkout `6096c395` (receipt in NK-WP-0033).
That remains historical check evidence, with predecessor denial unproven; it
cannot establish the new revision's acceptance. Historical apply last ran on
2026-08-27 before fixes in `4a38511`.

The August account attributed missing tuning to whole-object replacement.
Native privacyIDEA 3.12.2 and source inspection corrected that interpretation on
2026-10-13: omitted fields are preserved, while sending explicit defaults can
overwrite custom tuning. The October helper therefore sends only the bind
password and type. Historic claims, default-overwriting commands, temporary-file
handling and raw `--note` receipts are superseded by this procedure.