net-kingdom/sso-mfa/k8s/privacyidea/README.md

# T04 — Phase 3: Deploy privacyIDEA

Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See [procedure inventory](../../../docs/attended-procedure-inventory.md).

Phase 3 of NK-WP-0001: deploys the MFA core (privacyIDEA) in the `mfa` namespace.

**Hostnames (config points CP-NK-002 / CP-NK-003):**
- `pink.coulomb.social` — main portal and API
- `pink-account.coulomb.social` — self-service token portal

**Prerequisites:**
- T02 complete: `mfa` namespace and NetworkPolicies applied, cert-manager running.
- T03 complete: PostgreSQL cluster `net-kingdom-pg` in `databases` namespace is Ready.
- T01 Phase 0a complete: `gen-secrets.sh` run, all secrets in KeePassXC.

---

## Apply order

### Step 1 — Create the config Secret

```bash
cd sso-mfa/k8s/privacyidea
chmod +x create-secrets.sh enckey-bootstrap.sh bootstrap-admin.sh
./create-secrets.sh
```

Creates `privacyidea-config` in the `mfa` namespace (PI_SECRET_KEY, PI_PEPPER,
PI_SQLALCHEMY_DATABASE_URI).

---

### Step 2 — Apply manifests

```bash
# From sso-mfa/k8s/privacyidea/
kubectl apply -f pvc.yaml
kubectl apply -f configmap.yaml
kubectl apply -f middleware.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml
```

**Wait for the pod to reach Running state** (DB migrations run on first start —
allow up to 3 minutes):

```bash
kubectl get pods -n mfa -w
# Expected: privacyidea-<hash>   1/1   Running
```

If the pod is stuck in `Init` or `CrashLoopBackOff`, check logs:
```bash
kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous
```

Common causes:
- `privacyidea-config` Secret missing → run `create-secrets.sh`
- PostgreSQL not reachable → verify T03, check NetworkPolicies
- Wrong DB password → reconcile the database, CNPG application Secret and privacyIDEA URI through the reviewed named `PI_DB_PASSWORD` rotation; ordinary reapplication refuses differing values. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation).

---

### Step 3 — Extract key material and create DR Secrets

Export the existing key set from the verified Ready pod into an owned private
custody root containing a private `privacyidea` subdirectory:

```bash
bash enckey-bootstrap.sh /absolute/private/custody
```

This preserves existing local copies and creates or verifies matching DR Secrets:
`privacyidea-enckey` (`enckey`) and `privacyidea-auditkeys` (`private.pem`,
`public.pem`). Missing keys are refused. Only an explicitly reviewed first
bootstrap can request `--initialize-audit-keys`; it never generates an encryption
key. Follow [key custody and PVC recovery](KEY-RECOVERY.md) for private export,
independent backup, failure handling and recovery acceptance.

---

### Step 4 — Bootstrap admin accounts

```bash
./bootstrap-admin.sh
```

Creates or verifies `pi-admin` for the current KeyCape profile. It does not
provision the legacy Keycloak `trigger-admin` account, policy or Kubernetes Secret.
Existing legacy objects are not deleted: inventory their actual consumers and
approve revocation separately before removing them.

Run this only as an authorized, exclusive first-bootstrap or verification
operation. Password resets and policy migration are separate procedures. The private secrets directory and privacyidea
subdirectory must be owned by the operator without group/other access. Its
single-link regular `secrets.env` file must also be private; the script parses
exactly one generated alphanumeric `PI_ADMIN_PASSWORD` assignment without
executing shell content. Values travel over kubectl stdin, not argv, environment
assignments, REST bearer output or terminal text. No public-URL fallback is used.
The obsolete optional `pi-url` argument is refused.

The in-pod adapter is scoped to privacyIDEA 3.12.2. It verifies an existing
password without changing it. New account and policy rows use insert-only DB
writes; conflicts fail instead of updating existing records. The bootstrap
`pi-admin-all-rights` policy is restricted to `adminuser=pi-admin`. A conflicting
legacy policy, unknown version, credential mismatch, ambiguous pod or command
failure stops the script. Introducing a first admin policy into an existing
multi-admin/policy installation requires a separate owner-reviewed migration.
Configured `SUPERUSER_REALM` entries are refused because this local-admin profile
cannot safely grant username-based rights across administrative realms.
An error after a DB commit can leave created state; rerun only after reviewing
metadata, using the same approved credential. A successful retry never rotates it.

The secret remains in the selected private custody file. Transfer it through the
approved encrypted handoff and reconcile successful recovery before cleanup.
The helper does not prove runtime agent isolation or perform MFA enrollment.

**Attended acceptance remains required:**
1. Review effective admin policies and current identity mappings.
2. Complete the approved MFA enrollment procedure for the admin identity.
3. Perform a fresh login and confirm the required MFA challenge and denial cases.
4. Record the result; script success and `pi_admin_created` are not MFA acceptance.

No live bootstrap or account/policy change was performed for this source repair.
The source audit found that the upstream 3.12.2 `admin add` path can update an
existing password, so the old retry behavior was not reliably create-only.

---

### Step 5 — Verify

```bash
cd sso-mfa/k8s
chmod +x verify-t04.sh
./verify-t04.sh
```

---

## Container port note

The deployment uses `containerPort: 8080`. The official `privacyidea/privacyidea`
image uses nginx internally; the default nginx port may be 80 depending on the
image version.

**If the pod starts but requests return "connection refused":**
```bash
# Check what port the container actually listens on:
kubectl exec -n mfa <pod> -- ss -tlnp | grep LISTEN
```

If the container uses port 80, update:
1. `deployment.yaml`: `containerPort: 80`, Service `targetPort: 80`
2. `sso-mfa/k8s/network-policies/netpol-mfa.yaml`: `port: 80` in privacyIDEA rules
3. Reapply both files.

---

## NetworkPolicy design

privacyIDEA sits entirely behind the NetworkPolicies applied in T02 (netpol-mfa.yaml):

| Source | Destination | Port | Purpose |
|--------|-------------|------|---------|
| Traefik (kube-system) | privacyIDEA (mfa) | 8080 | User-facing portal |
| KeyCape (sso) | privacyIDEA (mfa) | 8080 | MFA API |
| privacyIDEA (mfa) | PostgreSQL (databases) | 5432 | Database |

Outbound to anything other than PostgreSQL and kube-dns is denied.

---

## Post-deploy steps (after verify-t04.sh passes)

### Rate limiting adjustment

The default rate limit (20 req/min, burst 5) is conservative. If KeyCape's
MFA calls trigger false positives (HTTP 429), raise the average
in `middleware.yaml` and reapply. Alternatively, the KeyCape-to-PI path is
cluster-internal and not subject to the Ingress middleware.

### Admin WebUI IP restriction

Update `middleware.yaml` `privacyidea-admin-allowlist.spec.ipWhiteList.sourceRange`
to your actual VPN/office CIDRs and reapply:

```bash
kubectl apply -f middleware.yaml
```

### Self-service portal

Enable the self-service portal policy in privacyIDEA:
Config → Policies → New policy:
- Scope: `user`
- Action: `enrollTOTP`, `enrollHOTP`, or token types you want users to manage
- URL for self-service: `https://pink-account.coulomb.social`

---

## Disaster recovery

Follow [key custody and PVC recovery](KEY-RECOVERY.md). Restore the original
matching encryption/audit files into the actual `privacyidea-data` PVC while the
application is stopped, using the reviewed mounted recovery pod. Creating a
Secret and restarting cannot restore these files; the deployment does not mount
the DR Secrets. The former unmounted helper-pod example is withdrawn. Native
recovery acceptance requires the matching database backup, existing-token checks
and audit-signature verification before closure.

## Scoped factor-state diagnostic

`check-user-mfa-state.sh` reports only factor counts in `coulomb`. It no longer
reads the retired `keycape-pi-token` Secret, queries administrative realm/user
catalogues, or guesses that the historic `netkingdom` realm is accessible.

```bash
bash sso-mfa/k8s/privacyidea/check-user-mfa-state.sh platform-root \
  --token-file /absolute/private/diagnostic.jwt
```

The file must contain an owner-admitted, unexpired provider admin-role JWT with
`tokenlist` visibility in `coulomb`, optionally followed by one newline. It and
its parent must be owned/private (normally 0600/0700), unlinked, and outside Git.
The helper does not retrieve, mint, renew, export or cache credentials. Missing
input stops before any credential or network access; raw values are never argv.

RPF-WP-0040's renewable factor-reader lane is delivered to **KeyCape**, not this
workstation helper. That contract is not permission to extract its Secret or
share its identity. An attended diagnostic needs its own explicitly admitted
owner delivery/execution arrangement; this source change neither creates nor
claims one. Use owner credential routing before requesting that arrangement.
The historical resolver/admin-token lanes remain separately blocked.

The helper uses the canonical HTTPS provider with system CA verification, or an
explicit literal loopback URL for a reviewed local forward/fixture. It refuses
other origins, redirects and ambient proxies. HTTP loopback is only as trustworthy
as the reviewed local process/forward. There is no insecure-TLS flag.

The existing disabled, unassigned `KCFACTORSCOPE01` token proves realm visibility
before and after the lookup. Provider denial, missing/mismatched proof, malformed
responses, unknown enrollment states or incomplete pages return `unverified`,
never a successful zero. The diagnostic requests at most 1000 factors and refuses
larger/incomplete results rather than publishing partial counts. The provider
API is paginated; see the [upstream token API](https://privacyidea.readthedocs.io/en/v3.12.1/modules/api/token.html).

JSON output contains total, active-confirmed, active-pending and inactive counts.
It deliberately records `user_existence: not_checked` and `mfa_login: not_tested`:
a scoped zero-factor response does not prove that a user exists, and counts do
not prove an MFA login. No token serial, metadata, provider error body or credential
is emitted. Separate HTTP observations are not an atomic provider snapshot.
Actual login remains `verify-t06.sh`/the supported user journey; pending enrollment
must be confirmed before it counts as an active confirmed factor.

## Configuration-preserving realm repair

The attended entrypoint remains:

```bash
bash sso-mfa/k8s/privacyidea/repair-realm-live.sh
```

It prompts for pi-admin in the terminal, reads current configuration, and asks
for an LLDAP bind password only when the named resolver is missing. Passwords
stay in process memory: no shell evaluation, environment-variable transport,
password arguments or temporary credential files. `MFA_USER` selects the user
for the final known-user lookup and fresh token-backed MFA proof. A failed proof
returns nonzero even if configuration writes already succeeded.

For an owner-reviewed bootstrap using existing private generated files:

```bash
bash sso-mfa/k8s/privacyidea/bootstrap-realm.sh /absolute/private/root
bash sso-mfa/k8s/privacyidea/bootstrap-realm.sh /absolute/private/root --apply
```

The first command authenticates and previews configuration actions; the second
executes the reviewed repair. `privacyidea/secrets.env` and `lldap/secrets.env`
are parsed as literal `NAME=value` records, never shell code. Root and component
directories must be owned/private (0700); files must be owned/private regular
unlinked files (0600). Shell-escaped `%q` files are not the input format. The old
third positional bind-password argument is rejected. An optional second
positional URL, or `PI_URL`, selects only the canonical verified HTTPS provider
or an explicit literal loopback fixture/approved forward. Proxy and redirect
credential forwarding are refused by shared `pi_api.py`.

The supported repair creates an absent `lldap-coulomb` LDAP resolver and absent
`coulomb` realm. On an existing resolver it fills only missing TIMEOUT,
CACHE_TIMEOUT and SIZELIMIT, preserving valid custom values, the bind credential
and other fields. It refuses a different resolver type/backend or an existing
realm missing the expected binding; those need owner-reviewed migration. Existing
realm bindings/priorities and defaults survive. It sets `coulomb` as the initial
default only when the initial realm inventory is empty.

No enrollment, deletion, passthrough or other policy is written. The deployed
onboarding guard, possession confirmation, pending-only cancellation and audited
recovery remain with their established KeyCape/NetKingdom owner procedures.
Replaying the old bootstrap's `enrollTOTP, delete, disable` or passthrough policy
would undo that hardening. A fresh instance still needs those owner-managed
onboarding components; a configuration receipt is not full IAM readiness.

The provider has no compare-and-swap operation for these writes. Use an exclusive
owner maintenance window. The helper checks for drift before each write and
reads back afterward; that does not make the sequence atomic. On failure it
reports unverified/possible partial changes and stops, with no automatic retry,
policy rewrite, deletion or rollback. Reconcile the retained state before retry.

Native disposable privacyIDEA 3.12.2 verifies that resolver POST preserves omitted
fields, including the bind password. This corrects the older runbook's general
replacement claim; it does not reclassify historical deployment evidence.
Reference: [provider resolver API](https://privacyidea.readthedocs.io/en/v3.12.2/modules/api/resolver.html).
Provider/package checks do not prove live LDAP reachability, current custody,
installed hooks, user enrollment or a complete KeyCape browser sign-in.

## Attended resolver password reconciliation

The update and reconcile helpers share `resolver_cutover.py`. Follow the
[current procedure](../../../docs/keycape-exposure-resolver-reconciliation.md)
for preview, digest-bound minimal apply, explicit cluster pins and truthful
partial receipts. Credentials stay in attended process memory. Native fixture
proof is not production acceptance or authority to perform exposure rotation.

## Entry-point index

Each helper in this directory, with its intended caller, from the NK-WP-0049-T02
[caller ledger](../../../docs/evidence/2026-10-23-nk49-t02-caller-ledger/caller-ledger.json).
"Greenfield only" helpers never update existing credentials or policy. On the
current estate, re-running one needs its own owner decision. All helpers run in
an attended operator session for namespace `mfa`. None of them prints a
credential.

| Helper | Purpose | Status / caller |
|---|---|---|
| `bootstrap-admin.py` | Pass one private bootstrap credential to the admitted pod over stdin | Greenfield only |
| `bootstrap-admin-remote.py` | In-pod, insert-only admin bootstrap | Greenfield only; called by `bootstrap-admin.py` |
| `realm_bootstrap.py` | Repair missing realm/resolver configuration while preserving policy | Library for `bootstrap-realm.sh` and the realm repair above |
| `activate-onboarding-policy.py` | Provider-local enrollment policy activation; never changes tokens | Current; NK-WP-0042 scoped enrollment |
| `deploy-onboarding-guard.py` | Deploy the reviewed enrollment hook, preserving runtime secrets | Current; NK-WP-0042 |
| `enable-factor-confirmation.py` | Require possession proof for newly enrolled TOTP; existing factors untouched | Current; NK-WP-0042 |
| `deploy-factor-recovery.py` | Deploy the private P04 factor-recovery service with provider custody and CAS guards | Current; recovery journey acceptance open under NK-WP-0042 |
| `key-custody-remote.py` | In-pod key transport | Internal; invoke only through `key-custody.py` |
| `keycape-scope-probe.py` | Idempotent, inert realm-visibility probe; emits no key | Diagnostic |
| `mfa_state.py` | Read-only scoped factor diagnostic | Library for `check-user-mfa-state.sh` |
| `verify_mfa.py` | Attended functional privacyIDEA proof | Called by `sso-mfa/k8s/verify-t06.sh` |