2026-02-25 23:59:28 +01:00
# Decision Log
_Auto-generated by the Custodian State Hub._
## D1 — Ingress controller: Traefik (K3s default) vs Nginx for ThreePhoenix
**Date:** 2026-02-25
**Decided by:** Tegwick
I want to go with C and separate concerns. Nginx for external SSL will need security and functional updates on a completly different schedule to Traefik canary and production workload splitting. The second area of implementation is more complicated, volatile and will need time to settle.
---
## D2 — Durable offsite backup destination for single-server safety net
2026-03-10 13:03:36 +00:00
**Date:** 2026-02-25
**Decided by:** Tegwick
2026-02-25 23:59:28 +01:00
We will use cloud storage the backup should be encypted to be safe regardless of the location and provider and for starters I will provide a nextcloud upload space as a backend.
---
2026-03-10 13:03:36 +00:00
## D3 — HA and failover scenarios must be tested before a workplan is considered done
**Date:** 2026-03-10
**Decided by:** Tegwick
On 2026-03-10 a PostgreSQL HA failover exposed a bug (missing `pgpool-password`
secret key) that had been present since initial deployment on 2025-08-31 but was
never discovered because no pod restart had occurred in 20 days. The immediate
symptom was Gitea logins hanging silently for hours.
This incident showed that deploying an HA component and declaring it "done"
without ever triggering a failover gives false confidence. Infrastructure that
has never failed over is not HA — it is just redundant hardware.
**Policy:**
Any workplan that deploys or configures a High Availability component
(database cluster, replicated storage, redundant ingress, etc.) is **not
complete** until a failover test passes. Specifically:
1. A test script in `tests/` must exist that deliberately kills the primary
component and asserts the service remains available within an acceptable
recovery window.
2. The test must be run against a live cluster and exit 0 before the workplan
status is set to `completed` .
3. Smoke tests (`tests/smoke_kube.sh` or equivalent) must include a health
check for each HA component's connection pooler, proxy, or load balancer —
not just the backing nodes.
4. Any Helm chart values required to make HA work correctly (secrets,
passwords, topology settings) must be present in the versioned values file
before the workplan is closed, so that a `helm upgrade` cannot silently
regress the fix.
**Rationale:** A failure that only surfaces on the first real event (restart,
failover, node loss) is a deployment bug, not an operational surprise. Railiance
aims for calm ops — and calm ops requires that every failure mode we know about
has been tested before it matters.
See: `workplans/RAIL-BS-WP-0003-pgpool-ha-failover-fix.md`
---
