--- id: NK-WP-0034 type: workplan title: "Make the SSO/MFA verification actually verify" domain: infotech repo: net-kingdom status: finished flavor: implementation owner: codex topic_slug: infotech created: "2026-08-28" updated: "2026-09-23" related: - NK-WP-0033 - CUST-ADR-012 state_hub_workstream_id: "c87e142c-4eda-5c1b-84a9-ee5a848f3f63" --- # Make the SSO/MFA verification actually verify ## Goal Close the blind spot `NK-WP-0033` exposed: verification that passes while the thing it verifies cannot work. ## Why this exists `reconcile-lldap-resolver-live.sh` was approved, documented, revision-pinned and referenced by an incident workplan as the supported attended operation. Its first execution on 2026-08-27 found four defects, two of which meant it could never have completed a run under any circumstances. `verify-t06.sh` reported success throughout. It checked that a resolver and a realm existed. It did not check that `GET /user/?realm=coulomb&username=` returns a user — which is the entire purpose of a resolver, and would have caught two of the four on the day they landed. The resolver was also created without `TIMEOUT`, `CACHE_TIMEOUT` and `SIZELIMIT`, so the WebUI had refused to save or test it for as long as it had existed, and nothing noticed. A verification that asserts its objects exist rather than that its property holds is not a weaker check. It is a check that reports green while the system is broken, which is worse than no check, because it is believed. ## Assert the property, not the objects ```task id: NK-WP-0034-T01 status: done priority: high state_hub_task_id: "9b5d8034-0ef1-53f9-ad4d-37de536bdc62" ``` Rewrite `verify-t06.sh` so it proves what the realm and resolver are *for*: that a known user resolves through the realm, and that a privacyIDEA MFA validation against that user succeeds. Existence checks may stay as diagnostics; they may not stand in for the outcome. Include the resolver's numeric parameters in what is asserted — a resolver the WebUI cannot save or test is misconfigured whether or not it currently answers. Acceptance: with the tuning fields cleared by hand, `verify-t06.sh` fails. With them restored, it passes. Demonstrated in both directions, not argued. ## Audit the other verify scripts for the same shape ```task id: NK-WP-0034-T02 status: done priority: medium state_hub_task_id: "0bd09a40-8b4d-5f9d-8d4c-81c55aa0c565" ``` `verify-t02.sh` through `verify-t08.sh` were written the same way by the same hand. Read each and record, per script, the property it is supposed to establish and whether it establishes it. The output is a list, not a rewrite. Fixing them all is a larger decision than this task should make on its own. Acceptance: one table — script, intended property, actual assertion, verdict. ## Label every attended procedure with its exercise status ```task id: NK-WP-0034-T03 status: done priority: medium state_hub_task_id: "becb4dce-4971-5755-888f-5276c4a56fe7" ``` `docs/attended-procedure-standard.md` requires an `Exercise status:` line in each runbook. There are 32 shell scripts under `sso-mfa/k8s/`; the reconciliation runbook is the only one carrying the line. Work through them and record, honestly, which have ever been run. "Unknown" is a permitted answer and a more useful one than a guess — the point is that an operator learns before running whether they are the first. Acceptance: every attended procedure's runbook carries the line; scripts with no runbook are listed as a separate finding. ## One helper per API ```task id: NK-WP-0034-T04 status: done priority: low state_hub_task_id: "d43ae8e8-eda6-5c1d-9e1b-01966b0e92da" ``` The `Content-Type`-on-bodyless-GET defect was found and fixed in `bootstrap-realm.sh`'s `pi_api`, with a comment explaining it, and then reintroduced verbatim in a script written later in the same directory. Extract one privacyIDEA request helper and have both scripts use it. A fix applied to a copy is a fix that expires the next time someone writes a script. Acceptance: one implementation of the privacyIDEA request path in `sso-mfa/k8s/privacyidea/`; both callers use it; the GET behaviour has a test that fails if the header returns. ## Review and implementation — 2026-09-05 Reviewed the proposal against current code and implemented all repository-local changes. T01 waits only for the attended scratch/provider exercise required by its acceptance. No live credentials were requested and no production resolver was changed. - T01: replaced warning-only T06 success with an attended known-user and TOTP/HOTP verifier, requiring realm binding, numeric tuning, exact user and resolver, and token-backed success. Credentials are prompted in process; missing attendance fails. Updated realm-repair and DR callers. Eighteen tests exercise the actual shared HTTP transport, including clearing and restoring every tuning field and rejecting passthrough/static passwords. See `docs/verify-t06.md` for the remaining attended acceptance and limits. - T02: completed the per-script property/assertion/verdict table in `docs/verification-audit-2026-09-05.md`. Other verify scripts retain their implementations; their gaps are explicitly recorded for follow-up. - T03: inventoried every shell script under `sso-mfa/k8s/`, added exercise headers to component runbooks, recorded unknown history and scripts without runbooks in `docs/attended-procedure-inventory.md`. Corrected the reconciliation runbook's attempted-run claim: no successful completion receipt exists. - T04: `privacyidea/pi_api.py` now supplies the request transport to both realm bootstrap and resolver reconciliation (and T06). Authentication also uses the shared transport in bootstrap. Its shell adapter keeps the token/body off argv. Fixed bootstrap's first-success `set -e` counter exit found during the review. The HTTP fixture rejects JSON Content-Type on GET and verifies the real Python and shell adapter requests. Remaining gate: an attended operator runs the scratch target failure/restoration procedure and records receipts. An automated fixture is not reported as an exercised provider run. NK-WP-0033 still needs its own incident receipt and predecessor-disposition ruling. Validation: `python3 -m pytest tests tools -q` passed 106 tests, including 18 privacyIDEA and 15 cadence cases. Ruff lint/format, shell syntax, embedded Python compilation and `git diff --check` passed. State Hub reconciliation was attempted with both the installed CLI and current checkout. Full reconciliation remains pending because API queries/writes timed out or returned connection-refused errors. Generated index/intake metadata was reviewed; the source files remain authoritative. ## Attended acceptance — 2026-09-23 (T01 done) T01's acceptance was demonstrated in both directions on production privacyIDEA. The fields were **not** cleared on purpose: production had been missing them since 2026-08-27. The operator was Bernd Worsch, using railiance01 checkout `6096c395`. 1. **FAIL, fields absent.** `verify-t06.sh --user platform-root` failed twice (runs at 18:41:54 and 18:42:22 server time). The privacyIDEA audit shows pi-admin auth, `GET /realm/` and `GET /resolver/lldap-coulomb` succeeding, and no user lookup after them. That places the failure in the `resolver-tuning` phase. The live `lldap-coulomb` config had 11 keys and no `TIMEOUT`, `CACHE_TIMEOUT` or `SIZELIMIT`. The unrepaired `reconcile-lldap-resolver-live.sh --apply` of 2026-08-27 (NK-WP-0033 defect 3) had dropped them, and the old T06 passed regardless. 2. **Restore.** With operator approval, the repaired `reconcile-lldap-resolver-live.sh --apply --predecessor-unavailable` ran at 19:14:45 and made one resolver write. Before the write, every non-secret value was confirmed identical to the declared body, and the bind value came from `operators/lldap/admin` version 1. Receipt: `PASS: resolver lookup, privacyIDEA MFA, predecessor denial=NOT-PROVEN, readiness, health, cleanup=PASS`. A database readback shows `TIMEOUT=5`, `CACHE_TIMEOUT=120` and `SIZELIMIT=500`, 14 keys in total. 3. **PASS, fields restored.** `verify-t06.sh --user platform-root` returned `{"phase": "complete", "proofs": ["realm-binding", "resolver-tuning", "known-user-lookup", "token-backed-mfa"], "result": "PASS"}`. The scratch-target exercise planned earlier in the session is no longer needed. The failure direction was observed on the real target, attributed through the audit log, and reversed by the declared remediation. Two runs in the same session failed at `authentication` because the wrong pi-admin value was entered (audit: "Wrong credentials"). Neither was a defect in the verifier.