tegwick
b01a30bd49
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a1346b-72a4-7d40-b8bd-8b8ca346cfa2
147 lines
4.4 KiB
Markdown
147 lines
4.4 KiB
Markdown
# T03 — PostgreSQL (CloudNativePG)
|
||
|
||
Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See [procedure inventory](../../../docs/attended-procedure-inventory.md).
|
||
|
||
Phase 2 of NK-WP-0001: CloudNativePG cluster with `keycloak_db` and `privacyidea_db`.
|
||
|
||
## Prerequisites
|
||
|
||
- T02 complete: `databases` namespace and NetworkPolicies applied
|
||
- `kubectl` configured with cluster access
|
||
- `gen-secrets.sh` run and output stored in KeePassXC
|
||
|
||
## Apply order
|
||
|
||
### 1. Install CloudNativePG operator
|
||
|
||
```bash
|
||
helm repo add cnpg https://cloudnative-pg.github.io/charts
|
||
helm repo update
|
||
helm install cnpg cnpg/cloudnative-pg \
|
||
--namespace cnpg-system \
|
||
--create-namespace \
|
||
--wait
|
||
```
|
||
|
||
Verify:
|
||
```bash
|
||
kubectl get pods -n cnpg-system
|
||
kubectl get crd clusters.postgresql.cnpg.io
|
||
```
|
||
|
||
### 2. Create K8s Secrets
|
||
|
||
```bash
|
||
# From the postgresql/ directory:
|
||
chmod +x create-secrets.sh
|
||
./create-secrets.sh ../../bootstrap/secrets
|
||
```
|
||
|
||
Alternatively, if you've already shredded the generated files, reconstruct from KeePassXC:
|
||
|
||
```bash
|
||
kubectl create secret generic net-kingdom-pg-keycloak-app \
|
||
--namespace=databases \
|
||
--from-literal=username=keycloak \
|
||
--from-literal=password='<PG_KEYCLOAK_PASSWORD from KeePassXC>'
|
||
|
||
kubectl create secret generic net-kingdom-pg-privacyidea-app \
|
||
--namespace=databases \
|
||
--from-literal=username=privacyidea \
|
||
--from-literal=password='<PI_DB_PASSWORD from KeePassXC>'
|
||
```
|
||
|
||
### 3. Deploy the cluster
|
||
|
||
```bash
|
||
kubectl apply -f cluster.yaml
|
||
```
|
||
|
||
Wait for cluster to become ready (this provisions PVCs and runs initdb — allow 2–3 minutes):
|
||
```bash
|
||
kubectl wait --for=condition=Ready cluster/net-kingdom-pg \
|
||
-n databases --timeout=300s
|
||
```
|
||
|
||
Check status:
|
||
```bash
|
||
kubectl get cluster -n databases
|
||
kubectl describe cluster net-kingdom-pg -n databases
|
||
kubectl get pods -n databases
|
||
```
|
||
|
||
### 4. Verify databases and users
|
||
|
||
```bash
|
||
# Connect as superuser to verify setup
|
||
kubectl exec -it -n databases \
|
||
$(kubectl get pod -n databases -l cnpg.io/cluster=net-kingdom-pg,role=primary -o name) \
|
||
-- psql -U postgres
|
||
|
||
# In psql:
|
||
\l -- list databases
|
||
\du -- list roles
|
||
\q
|
||
```
|
||
|
||
Expected output: `keycloak_db`, `privacyidea_db`, roles `keycloak` and `privacyidea`.
|
||
|
||
### 5. Configure backup (when object storage is available)
|
||
|
||
Uncomment the `backup:` section in `cluster.yaml` and fill in the object store endpoint.
|
||
Create the S3 credentials secret:
|
||
|
||
```bash
|
||
kubectl create secret generic net-kingdom-pg-backup-s3 \
|
||
--namespace=databases \
|
||
--from-literal=ACCESS_KEY_ID='<access key>' \
|
||
--from-literal=SECRET_ACCESS_KEY='<secret key>'
|
||
```
|
||
|
||
Apply the updated cluster.yaml, then:
|
||
```bash
|
||
kubectl apply -f scheduled-backup.yaml
|
||
```
|
||
|
||
### 6. Run the restore drill
|
||
|
||
**Mandatory before marking T03 done.**
|
||
|
||
```bash
|
||
# Trigger a manual backup first
|
||
kubectl cnpg backup net-kingdom-pg -n databases
|
||
|
||
# Wait for backup to complete
|
||
kubectl get backup -n databases --watch
|
||
|
||
# Restore to a new cluster to verify
|
||
# (See CloudNativePG docs: kubectl cnpg restore or Cluster bootstrap.recovery)
|
||
```
|
||
|
||
### 7. Run the full verification script
|
||
|
||
```bash
|
||
chmod +x ../verify-t03.sh
|
||
../verify-t03.sh
|
||
```
|
||
|
||
## Secrets reference
|
||
|
||
| Secret name | Keys | Purpose |
|
||
|---|---|---|
|
||
| `net-kingdom-pg-keycloak-app` | `username`, `password` | Keycloak DB user (also bootstrap owner) |
|
||
| `net-kingdom-pg-privacyidea-app` | `username`, `password` | privacyIDEA DB user |
|
||
| `net-kingdom-pg-backup-s3` | `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY` | Object store backup (optional until backup enabled) |
|
||
| `net-kingdom-pg-superuser` | auto-created by CNPG | PostgreSQL superuser (operator-managed) |
|
||
| `net-kingdom-pg-app` | auto-created by CNPG | Initial app user (unused — we use named secrets) |
|
||
|
||
## Notes
|
||
|
||
- `cnpg.io/cluster: net-kingdom-pg` label on pods is what the NetworkPolicies in T02 target.
|
||
Do not rename the cluster without also updating netpol-databases.yaml.
|
||
- `instances: 1` is intentional for dev/staging. Change to 3 before ThreePhoenix HA production
|
||
deployment (requires at least 3 schedulable nodes).
|
||
- Password rotation uses the explicit `PI_DB_PASSWORD` selector under the reviewed
|
||
procedure. CNPG's managed role and the privacyIDEA consumer URI must converge
|
||
on the same candidate; verify actual reconciliation and authentication before
|
||
calling it complete. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation).
|