net-kingdom/sso-mfa/k8s/privacyidea/README.md

T04 — Phase 3: Deploy privacyIDEA

Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See procedure inventory.

Phase 3 of NK-WP-0001: deploys the MFA core (privacyIDEA) in the mfa namespace.

Hostnames (config points CP-NK-002 / CP-NK-003):

• pink.coulomb.social — main portal and API
• pink-account.coulomb.social — self-service token portal

Prerequisites:

• T02 complete: mfa namespace and NetworkPolicies applied, cert-manager running.
• T03 complete: PostgreSQL cluster net-kingdom-pg in databases namespace is Ready.
• T01 Phase 0a complete: gen-secrets.sh run, all secrets in KeePassXC.

---

Apply order

Step 1 — Create the config Secret

cd sso-mfa/k8s/privacyidea
chmod +x create-secrets.sh enckey-bootstrap.sh bootstrap-admin.sh
./create-secrets.sh

Creates privacyidea-config in the mfa namespace (PI_SECRET_KEY, PI_PEPPER, PI_SQLALCHEMY_DATABASE_URI).

---

Step 2 — Apply manifests

# From sso-mfa/k8s/privacyidea/
kubectl apply -f pvc.yaml
kubectl apply -f configmap.yaml
kubectl apply -f middleware.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml

Wait for the pod to reach Running state (DB migrations run on first start — allow up to 3 minutes):

kubectl get pods -n mfa -w
# Expected: privacyidea-<hash>   1/1   Running

If the pod is stuck in Init or CrashLoopBackOff, check logs:

kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous

Common causes:

• privacyidea-config Secret missing → run create-secrets.sh
• PostgreSQL not reachable → verify T03, check NetworkPolicies
• Wrong DB password → reconcile the database, CNPG application Secret and privacyIDEA URI through the reviewed named PI_DB_PASSWORD rotation; ordinary reapplication refuses differing values. See service Secret custody.

---

Step 3 — Extract key material and create DR Secrets

Export the existing key set from the verified Ready pod into an owned private custody root containing a private privacyidea subdirectory:

bash enckey-bootstrap.sh /absolute/private/custody

This preserves existing local copies and creates or verifies matching DR Secrets: privacyidea-enckey (enckey) and privacyidea-auditkeys (private.pem, public.pem). Missing keys are refused. Only an explicitly reviewed first bootstrap can request --initialize-audit-keys; it never generates an encryption key. Follow key custody and PVC recovery for private export, independent backup, failure handling and recovery acceptance.

---

Step 4 — Bootstrap admin accounts

./bootstrap-admin.sh

Creates or verifies pi-admin for the current KeyCape profile. It does not provision the legacy Keycloak trigger-admin account, policy or Kubernetes Secret. Existing legacy objects are not deleted: inventory their actual consumers and approve revocation separately before removing them.

Run this only as an authorized, exclusive first-bootstrap or verification operation. Password resets and policy migration are separate procedures. The private secrets directory and privacyidea subdirectory must be owned by the operator without group/other access. Its single-link regular secrets.env file must also be private; the script parses exactly one generated alphanumeric PI_ADMIN_PASSWORD assignment without executing shell content. Values travel over kubectl stdin, not argv, environment assignments, REST bearer output or terminal text. No public-URL fallback is used. The obsolete optional pi-url argument is refused.

The in-pod adapter is scoped to privacyIDEA 3.12.2. It verifies an existing password without changing it. New account and policy rows use insert-only DB writes; conflicts fail instead of updating existing records. The bootstrap pi-admin-all-rights policy is restricted to adminuser=pi-admin. A conflicting legacy policy, unknown version, credential mismatch, ambiguous pod or command failure stops the script. Introducing a first admin policy into an existing multi-admin/policy installation requires a separate owner-reviewed migration. Configured SUPERUSER_REALM entries are refused because this local-admin profile cannot safely grant username-based rights across administrative realms. An error after a DB commit can leave created state; rerun only after reviewing metadata, using the same approved credential. A successful retry never rotates it.

The secret remains in the selected private custody file. Transfer it through the approved encrypted handoff and reconcile successful recovery before cleanup. The helper does not prove runtime agent isolation or perform MFA enrollment.

Attended acceptance remains required:

1. Review effective admin policies and current identity mappings.
2. Complete the approved MFA enrollment procedure for the admin identity.
3. Perform a fresh login and confirm the required MFA challenge and denial cases.
4. Record the result; script success and pi_admin_created are not MFA acceptance.

No live bootstrap or account/policy change was performed for this source repair. The source audit found that the upstream 3.12.2 admin add path can update an existing password, so the old retry behavior was not reliably create-only.

---

Step 5 — Verify

cd sso-mfa/k8s
chmod +x verify-t04.sh
./verify-t04.sh

---

Container port note

The deployment uses containerPort: 8080. The official privacyidea/privacyidea image uses nginx internally; the default nginx port may be 80 depending on the image version.

If the pod starts but requests return "connection refused":

# Check what port the container actually listens on:
kubectl exec -n mfa <pod> -- ss -tlnp | grep LISTEN

If the container uses port 80, update:

1. deployment.yaml: containerPort: 80, Service targetPort: 80
2. sso-mfa/k8s/network-policies/netpol-mfa.yaml: port: 80 in privacyIDEA rules
3. Reapply both files.

---

NetworkPolicy design

privacyIDEA sits entirely behind the NetworkPolicies applied in T02 (netpol-mfa.yaml):

Source	Destination	Port	Purpose
Traefik (kube-system)	privacyIDEA (mfa)	8080	User-facing portal
KeyCape (sso)	privacyIDEA (mfa)	8080	MFA API
privacyIDEA (mfa)	PostgreSQL (databases)	5432	Database

Outbound to anything other than PostgreSQL and kube-dns is denied.

---

Post-deploy steps (after verify-t04.sh passes)

Rate limiting adjustment

The default rate limit (20 req/min, burst 5) is conservative. If KeyCape's MFA calls trigger false positives (HTTP 429), raise the average in middleware.yaml and reapply. Alternatively, the KeyCape-to-PI path is cluster-internal and not subject to the Ingress middleware.

Admin WebUI IP restriction

Update middleware.yaml privacyidea-admin-allowlist.spec.ipWhiteList.sourceRange to your actual VPN/office CIDRs and reapply:

kubectl apply -f middleware.yaml

Self-service portal

Enable the self-service portal policy in privacyIDEA: Config → Policies → New policy:

• Scope: user
• Action: enrollTOTP, enrollHOTP, or token types you want users to manage
• URL for self-service: https://pink-account.coulomb.social

---

Disaster recovery

Follow key custody and PVC recovery. Restore the original matching encryption/audit files into the actual privacyidea-data PVC while the application is stopped, using the reviewed mounted recovery pod. Creating a Secret and restarting cannot restore these files; the deployment does not mount the DR Secrets. The former unmounted helper-pod example is withdrawn. Native recovery acceptance requires the matching database backup, existing-token checks and audit-signature verification before closure.

Scoped factor-state diagnostic

check-user-mfa-state.sh reports only factor counts in coulomb. It no longer reads the retired keycape-pi-token Secret, queries administrative realm/user catalogues, or guesses that the historic netkingdom realm is accessible.

bash sso-mfa/k8s/privacyidea/check-user-mfa-state.sh platform-root \
  --token-file /absolute/private/diagnostic.jwt

The file must contain an owner-admitted, unexpired provider admin-role JWT with tokenlist visibility in coulomb, optionally followed by one newline. It and its parent must be owned/private (normally 0600/0700), unlinked, and outside Git. The helper does not retrieve, mint, renew, export or cache credentials. Missing input stops before any credential or network access; raw values are never argv.

RPF-WP-0040's renewable factor-reader lane is delivered to KeyCape, not this workstation helper. That contract is not permission to extract its Secret or share its identity. An attended diagnostic needs its own explicitly admitted owner delivery/execution arrangement; this source change neither creates nor claims one. Use owner credential routing before requesting that arrangement. The historical resolver/admin-token lanes remain separately blocked.

The helper uses the canonical HTTPS provider with system CA verification, or an explicit literal loopback URL for a reviewed local forward/fixture. It refuses other origins, redirects and ambient proxies. HTTP loopback is only as trustworthy as the reviewed local process/forward. There is no insecure-TLS flag.

The existing disabled, unassigned KCFACTORSCOPE01 token proves realm visibility before and after the lookup. Provider denial, missing/mismatched proof, malformed responses, unknown enrollment states or incomplete pages return unverified, never a successful zero. The diagnostic requests at most 1000 factors and refuses larger/incomplete results rather than publishing partial counts. The provider API is paginated; see the upstream token API.

JSON output contains total, active-confirmed, active-pending and inactive counts. It deliberately records user_existence: not_checked and mfa_login: not_tested: a scoped zero-factor response does not prove that a user exists, and counts do not prove an MFA login. No token serial, metadata, provider error body or credential is emitted. Separate HTTP observations are not an atomic provider snapshot. Actual login remains verify-t06.sh/the supported user journey; pending enrollment must be confirmed before it counts as an active confirmed factor.

Configuration-preserving realm repair

The attended entrypoint remains:

bash sso-mfa/k8s/privacyidea/repair-realm-live.sh

It prompts for pi-admin in the terminal, reads current configuration, and asks for an LLDAP bind password only when the named resolver is missing. Passwords stay in process memory: no shell evaluation, environment-variable transport, password arguments or temporary credential files. MFA_USER selects the user for the final known-user lookup and fresh token-backed MFA proof. A failed proof returns nonzero even if configuration writes already succeeded.

For an owner-reviewed bootstrap using existing private generated files:

bash sso-mfa/k8s/privacyidea/bootstrap-realm.sh /absolute/private/root
bash sso-mfa/k8s/privacyidea/bootstrap-realm.sh /absolute/private/root --apply

The first command authenticates and previews configuration actions; the second executes the reviewed repair. privacyidea/secrets.env and lldap/secrets.env are parsed as literal NAME=value records, never shell code. Root and component directories must be owned/private (0700); files must be owned/private regular unlinked files (0600). Shell-escaped %q files are not the input format. The old third positional bind-password argument is rejected. An optional second positional URL, or PI_URL, selects only the canonical verified HTTPS provider or an explicit literal loopback fixture/approved forward. Proxy and redirect credential forwarding are refused by shared pi_api.py.

The supported repair creates an absent lldap-coulomb LDAP resolver and absent coulomb realm. On an existing resolver it fills only missing TIMEOUT, CACHE_TIMEOUT and SIZELIMIT, preserving valid custom values, the bind credential and other fields. It refuses a different resolver type/backend or an existing realm missing the expected binding; those need owner-reviewed migration. Existing realm bindings/priorities and defaults survive. It sets coulomb as the initial default only when the initial realm inventory is empty.

No enrollment, deletion, passthrough or other policy is written. The deployed onboarding guard, possession confirmation, pending-only cancellation and audited recovery remain with their established KeyCape/NetKingdom owner procedures. Replaying the old bootstrap's enrollTOTP, delete, disable or passthrough policy would undo that hardening. A fresh instance still needs those owner-managed onboarding components; a configuration receipt is not full IAM readiness.

The provider has no compare-and-swap operation for these writes. Use an exclusive owner maintenance window. The helper checks for drift before each write and reads back afterward; that does not make the sequence atomic. On failure it reports unverified/possible partial changes and stops, with no automatic retry, policy rewrite, deletion or rollback. Reconcile the retained state before retry.

Native disposable privacyIDEA 3.12.2 verifies that resolver POST preserves omitted fields, including the bind password. This corrects the older runbook's general replacement claim; it does not reclassify historical deployment evidence. Reference: provider resolver API. Provider/package checks do not prove live LDAP reachability, current custody, installed hooks, user enrollment or a complete KeyCape browser sign-in.

Attended resolver password reconciliation

The update and reconcile helpers share resolver_cutover.py. Follow the current procedure for preview, digest-bound minimal apply, explicit cluster pins and truthful partial receipts. Credentials stay in attended process memory. Native fixture proof is not production acceptance or authority to perform exposure rotation.

Entry-point index

Each helper in this directory, with its intended caller, from the NK-WP-0049-T02 caller ledger. "Greenfield only" helpers never update existing credentials or policy. On the current estate, re-running one needs its own owner decision. All helpers run in an attended operator session for namespace mfa. None of them prints a credential.

Helper	Purpose	Status / caller
bootstrap-admin.py	Pass one private bootstrap credential to the admitted pod over stdin	Greenfield only
bootstrap-admin-remote.py	In-pod, insert-only admin bootstrap	Greenfield only; called by bootstrap-admin.py
realm_bootstrap.py	Repair missing realm/resolver configuration while preserving policy	Library for bootstrap-realm.sh and the realm repair above
activate-onboarding-policy.py	Provider-local enrollment policy activation; never changes tokens	Current; NK-WP-0042 scoped enrollment
deploy-onboarding-guard.py	Deploy the reviewed enrollment hook, preserving runtime secrets	Current; NK-WP-0042
enable-factor-confirmation.py	Require possession proof for newly enrolled TOTP; existing factors untouched	Current; NK-WP-0042
deploy-factor-recovery.py	Deploy the private P04 factor-recovery service with provider custody and CAS guards	Current; recovery journey acceptance open under NK-WP-0042
key-custody-remote.py	In-pod key transport	Internal; invoke only through key-custody.py
keycape-scope-probe.py	Idempotent, inert realm-visibility probe; emits no key	Diagnostic
mfa_state.py	Read-only scoped factor diagnostic	Library for check-user-mfa-state.sh
verify_mfa.py	Attended functional privacyIDEA proof	Called by sso-mfa/k8s/verify-t06.sh