net-kingdom/sso-mfa/k8s/postgresql/README.md

# T03 — PostgreSQL (CloudNativePG)

Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See [procedure inventory](../../../docs/attended-procedure-inventory.md).

Phase 2 of NK-WP-0001: CloudNativePG cluster with `keycloak_db` and `privacyidea_db`.

## Prerequisites

- T02 complete: `databases` namespace and NetworkPolicies applied
- `kubectl` configured with cluster access
- `gen-secrets.sh` run and output stored in KeePassXC

## Apply order

### 1. Install CloudNativePG operator

```bash
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm install cnpg cnpg/cloudnative-pg \
  --namespace cnpg-system \
  --create-namespace \
  --wait
```

Verify:
```bash
kubectl get pods -n cnpg-system
kubectl get crd clusters.postgresql.cnpg.io
```

### 2. Create K8s Secrets

```bash
# From the postgresql/ directory:
chmod +x create-secrets.sh
./create-secrets.sh ../../bootstrap/secrets
```

Alternatively, if you've already shredded the generated files, reconstruct from KeePassXC:

```bash
kubectl create secret generic net-kingdom-pg-keycloak-app \
  --namespace=databases \
  --from-literal=username=keycloak \
  --from-literal=password='<PG_KEYCLOAK_PASSWORD from KeePassXC>'

kubectl create secret generic net-kingdom-pg-privacyidea-app \
  --namespace=databases \
  --from-literal=username=privacyidea \
  --from-literal=password='<PI_DB_PASSWORD from KeePassXC>'
```

### 3. Deploy the cluster

```bash
kubectl apply -f cluster.yaml
```

Wait for cluster to become ready (this provisions PVCs and runs initdb — allow 2–3 minutes):
```bash
kubectl wait --for=condition=Ready cluster/net-kingdom-pg \
  -n databases --timeout=300s
```

Check status:
```bash
kubectl get cluster -n databases
kubectl describe cluster net-kingdom-pg -n databases
kubectl get pods -n databases
```

### 4. Verify databases and users

```bash
# Connect as superuser to verify setup
kubectl exec -it -n databases \
  $(kubectl get pod -n databases -l cnpg.io/cluster=net-kingdom-pg,role=primary -o name) \
  -- psql -U postgres

# In psql:
\l                          -- list databases
\du                         -- list roles
\q
```

Expected output: `keycloak_db`, `privacyidea_db`, roles `keycloak` and `privacyidea`.

### 5. Configure backup (when object storage is available)

Uncomment the `backup:` section in `cluster.yaml` and fill in the object store endpoint.
Create the S3 credentials secret:

```bash
kubectl create secret generic net-kingdom-pg-backup-s3 \
  --namespace=databases \
  --from-literal=ACCESS_KEY_ID='<access key>' \
  --from-literal=SECRET_ACCESS_KEY='<secret key>'
```

Apply the updated cluster.yaml, then:
```bash
kubectl apply -f scheduled-backup.yaml
```

### 6. Run the restore drill

**Mandatory before marking T03 done.**

```bash
# Trigger a manual backup first
kubectl cnpg backup net-kingdom-pg -n databases

# Wait for backup to complete
kubectl get backup -n databases --watch

# Restore to a new cluster to verify
# (See CloudNativePG docs: kubectl cnpg restore or Cluster bootstrap.recovery)
```

### 7. Run the full verification script

```bash
chmod +x ../verify-t03.sh
../verify-t03.sh
```

## Secrets reference

| Secret name | Keys | Purpose |
|---|---|---|
| `net-kingdom-pg-keycloak-app` | `username`, `password` | Keycloak DB user (also bootstrap owner) |
| `net-kingdom-pg-privacyidea-app` | `username`, `password` | privacyIDEA DB user |
| `net-kingdom-pg-backup-s3` | `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY` | Object store backup (optional until backup enabled) |
| `net-kingdom-pg-superuser` | auto-created by CNPG | PostgreSQL superuser (operator-managed) |
| `net-kingdom-pg-app` | auto-created by CNPG | Initial app user (unused — we use named secrets) |

## Notes

- `cnpg.io/cluster: net-kingdom-pg` label on pods is what the NetworkPolicies in T02 target.
  Do not rename the cluster without also updating netpol-databases.yaml.
- `instances: 1` is intentional for dev/staging. Change to 3 before ThreePhoenix HA production
  deployment (requires at least 3 schedulable nodes).
- Password rotation: update the K8s Secret values and CNPG's managed.roles reconciler will
  apply the change at the next reconciliation cycle (within seconds).