net-kingdom/sso-mfa/k8s/privacyidea
tegwick 4a38511d11 fix(privacyidea): repair the resolver reconciliation script (NK-WP-0033)
reconcile-lldap-resolver-live.sh had never completed a run. Four defects,
found by running it on 2026-08-27:

1. request() set Content-Type: application/json on every call, including
   bodyless GETs. Werkzeug 3.x rejects those in front of privacyIDEA, so
   every GET returned an HTML 400 while POSTs succeeded — the resolver
   write landed and the lookup immediately after it did not.
   bootstrap-realm.sh already fixed this in pi_api and said why; this
   script was written later and did not inherit it.

2. GET /user/ returns result.value as a list of user objects, not a dict
   carrying "users". With the 400 fixed, the lookup finally reached the
   parse and raised AttributeError past the except clause, so the run
   died as a traceback instead of a receipt. Both shapes now accepted,
   and the except clause catches parse errors so a failed run still
   names the phase it died in.

3. A resolver write replaces the whole object, so TIMEOUT,
   CACHE_TIMEOUT and SIZELIMIT were dropped by every --apply. A resolver
   with them unset still resolves users, but the WebUI refuses to save
   or test it — so the script silently un-repaired a resolver an
   operator had fixed by hand. Now sent, defaulting to the verified
   5/120/500 and overridable per run. Same omission fixed in
   bootstrap-realm.sh, which created the resolver that way originally.

4. The predecessor prompt could not be left empty, so an operator who
   had lost the exposed credential had to type a placeholder — which
   also fails the bind and was recorded as a PASSING denial proof.
   --predecessor-unavailable skips the bind and records NOT-PROVEN.
   --note carries operator context into the receipt line itself, so the
   claim and its caveat travel together.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 3377672@bnt-lap001
Assistant-Session: 15463ccf-238f-4e13-b163-93aa25c6d166
2026-08-27 22:20:03 +02:00
..
bootstrap-admin.sh fix(privacyidea): create pi-admin-all-rights policy in bootstrap-admin.sh 2026-03-25 11:49:26 +00:00
bootstrap-realm.sh fix(privacyidea): repair the resolver reconciliation script (NK-WP-0033) 2026-08-27 22:20:03 +02:00
check-user-mfa-state.sh chore: make T06 verify scripts executable (chmod +x for check-mfa and keycape-verify used in dry-run evidence) 2026-06-03 02:03:03 +02:00
configmap.yaml fix(privacyidea): disable response signing + raise rate limit to unblock login 2026-03-25 11:49:26 +00:00
create-secrets.sh feat(sso-mfa): T04 privacyIDEA manifests (NK-WP-0001-T04) 2026-03-19 01:22:41 +00:00
deployment.yaml fix(privacyidea): add PI_ADDRESS/PI_PORT; switch readiness probe to tcpSocket 2026-03-21 10:41:13 +00:00
enckey-bootstrap.sh feat(sso-mfa): T04 privacyIDEA manifests (NK-WP-0001-T04) 2026-03-19 01:22:41 +00:00
ingress.yaml fix(privacyidea): correct image to ghcr.io/gpappsoft, port 5001→8080 2026-03-21 09:37:38 +00:00
middleware.yaml fix(privacyidea): disable response signing + raise rate limit to unblock login 2026-03-25 11:49:26 +00:00
pvc.yaml feat(sso-mfa): T04 privacyIDEA manifests (NK-WP-0001-T04) 2026-03-19 01:22:41 +00:00
README.md docs(sso-mfa): fix stale Keycloak refs and add T04 apply section to WORKPLAN 2026-03-20 07:33:47 +00:00
reconcile-lldap-resolver-live.sh fix(privacyidea): repair the resolver reconciliation script (NK-WP-0033) 2026-08-27 22:20:03 +02:00
repair-realm-live.sh Close OpenBao OIDC admin bootstrap path 2026-06-01 21:20:53 +02:00
update-lldap-resolver-live.sh feat(privacyidea): add guarded resolver reconciliation helper 2026-08-23 14:43:39 +02:00

T04 — Phase 3: Deploy privacyIDEA

Phase 3 of NK-WP-0001: deploys the MFA core (privacyIDEA) in the mfa namespace.

Hostnames (config points CP-NK-002 / CP-NK-003):

  • pink.coulomb.social — main portal and API
  • pink-account.coulomb.social — self-service token portal

Prerequisites:

  • T02 complete: mfa namespace and NetworkPolicies applied, cert-manager running.
  • T03 complete: PostgreSQL cluster net-kingdom-pg in databases namespace is Ready.
  • T01 Phase 0a complete: gen-secrets.sh run, all secrets in KeePassXC.

Apply order

Step 1 — Create the config Secret

cd sso-mfa/k8s/privacyidea
chmod +x create-secrets.sh enckey-bootstrap.sh bootstrap-admin.sh
./create-secrets.sh

Creates privacyidea-config in the mfa namespace (PI_SECRET_KEY, PI_PEPPER, PI_SQLALCHEMY_DATABASE_URI).


Step 2 — Apply manifests

# From sso-mfa/k8s/privacyidea/
kubectl apply -f pvc.yaml
kubectl apply -f configmap.yaml
kubectl apply -f middleware.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml

Wait for the pod to reach Running state (DB migrations run on first start — allow up to 3 minutes):

kubectl get pods -n mfa -w
# Expected: privacyidea-<hash>   1/1   Running

If the pod is stuck in Init or CrashLoopBackOff, check logs:

kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous

Common causes:

  • privacyidea-config Secret missing → run create-secrets.sh
  • PostgreSQL not reachable → verify T03, check NetworkPolicies
  • Wrong DB password → re-run create-secrets.sh with corrected secrets

Step 3 — Extract key material and create DR Secrets

Run once after the pod reaches Running:

./enckey-bootstrap.sh

This generates RSA audit keys (if not already created), extracts the encryption key and audit keys from the pod, and creates two K8s Secrets as disaster-recovery copies:

  • privacyidea-enckey
  • privacyidea-auditkeys

Follow the printed instructions to store the key files in KeePassXC, then shred the local copies.


Step 4 — Bootstrap admin accounts

./bootstrap-admin.sh

Creates pi-admin (full admin) and trigger-admin (triggerchallenge only). Also creates the privacyidea-trigger-admin K8s Secret used by Keycloak in T05.

Immediately after the script completes:

  1. Log in to https://pink.coulomb.social as pi-admin.
  2. Navigate to Users → pi-admin → Enroll token and enroll a TOTP or hardware token.
  3. Log out, log back in — the MFA challenge must appear.
  4. Verify the trigger-admin-rights policy at Config → Policies.

Step 5 — Verify

cd sso-mfa/k8s
chmod +x verify-t04.sh
./verify-t04.sh

Container port note

The deployment uses containerPort: 8080. The official privacyidea/privacyidea image uses nginx internally; the default nginx port may be 80 depending on the image version.

If the pod starts but requests return "connection refused":

# Check what port the container actually listens on:
kubectl exec -n mfa <pod> -- ss -tlnp | grep LISTEN

If the container uses port 80, update:

  1. deployment.yaml: containerPort: 80, Service targetPort: 80
  2. sso-mfa/k8s/network-policies/netpol-mfa.yaml: port: 80 in privacyIDEA rules
  3. Reapply both files.

NetworkPolicy design

privacyIDEA sits entirely behind the NetworkPolicies applied in T02 (netpol-mfa.yaml):

Source Destination Port Purpose
Traefik (kube-system) privacyIDEA (mfa) 8080 User-facing portal
Keycloak (sso) privacyIDEA (mfa) 8080 Provider API (triggerchallenge)
privacyIDEA (mfa) PostgreSQL (databases) 5432 Database

Outbound to anything other than PostgreSQL and kube-dns is denied.


Post-deploy steps (after verify-t04.sh passes)

Rate limiting adjustment

The default rate limit (20 req/min, burst 5) is conservative. If Keycloak's triggerchallenge calls trigger false positives (HTTP 429), raise the average in middleware.yaml and reapply. Alternatively, the Keycloak-to-PI path is cluster-internal and not subject to the Ingress middleware.

Admin WebUI IP restriction

Update middleware.yaml privacyidea-admin-allowlist.spec.ipWhiteList.sourceRange to your actual VPN/office CIDRs and reapply:

kubectl apply -f middleware.yaml

Self-service portal

Enable the self-service portal policy in privacyIDEA: Config → Policies → New policy:

  • Scope: user
  • Action: enrollTOTP, enrollHOTP, or token types you want users to manage
  • URL for self-service: https://pink-account.coulomb.social

Disaster recovery

If the privacyidea-data PVC is lost:

  1. Create a new PVC with pvc.yaml.
  2. Restore enckey and audit keys from KeePassXC:
    # Copy pi.enc and private/public.pem from KeePassXC into a temporary pod
    kubectl run restore-helper --image=busybox -n mfa --restart=Never -- sleep 3600
    kubectl cp ./pi.enc        mfa/restore-helper:/etc/privacyidea/enckey
    kubectl cp ./private.pem   mfa/restore-helper:/etc/privacyidea/private.pem
    kubectl cp ./public.pem    mfa/restore-helper:/etc/privacyidea/public.pem
    kubectl delete pod -n mfa restore-helper
    
    Or, if the K8s Secrets survived (created by enckey-bootstrap.sh):
    kubectl get secret privacyidea-enckey -n mfa -o jsonpath='{.data.enckey}' | base64 -d > pi.enc
    
  3. Restart the privacyIDEA deployment — it will run DB migrations and use the restored key material.