railiance-cluster/workplans/RCLUSTER-WP-0007-threephoenix-ha-cluster.md

id	type	title	domain	repo	status	owner	topic_slug	repo_goal_id	state_hub_workstream_id	created	updated	related
RCLUSTER-WP-0007	workplan	ThreePhoenix - HA Cluster Implementation	financials	railiance-cluster	backlog	codex	railiance	6ea441f7-7fe3-4598-922b-38baf20c0580	9e208376-23f1-40c7-9813-fac1f7d6ad3b	2026-02-25	2026-08-22	REEF-RAILIANCE-WP-0003 RAIL-BS-WP-0014 RMASTER-WP-0023 ADR-0006 ADR-0008

ThreePhoenix - HA Cluster Implementation

Goal

Implement the ThreePhoenix architecture: a self-healing three-node Kubernetes cluster substrate for Railiance production systems.

The cluster target includes:

• k3s HA with embedded etcd.
• Cluster-level storage operator hooks (not the database clusters).
• Ingress and certificate controllers (live path is Traefik, not nginx).
• Node rotation and recovery drills.
• A machine-readable one-server-loss S2 evidence pack.

This is the S2 substrate work that can clear the open failure-domain gate for critical workloads on reef-railiance, beginning with REEF-RAILIANCE-WP-0003. Cluster readiness is ADR-0006 reef evidence for verified. It must not write production-approved on a binding. Current reality is one Ready node (railiance01), local storage, Traefik. Do not report ThreePhoenix targets as live capacity (RAIL-BS-WP-0014).

Any listener this plan adds is ADR-0008 private unless a named grant exists. 6443 is not grantable as public. An Ingress object is not a grant.

The bootstrap copy of this file is retired. Do not re-open RCLUSTER-WP-0007 in railiance-bootstrap.

Implementation gate

This workplan remains backlog; its tasks must not move to progress until the preflight in docs/threephoenix-implementation-gate.md passes. In particular, three independent source-backed reef members must be approved, provisioned, and S1-converged, with their shared power, storage, host, and network dependencies recorded. The current Railiance01 node plus guests on one physical host do not satisfy that gate.

Passing the gate also requires an approved private inter-node network design, join-token custody/rotation through the platform custody path, and explicit operator approval for destructive node actions. This repository must not buy provider capacity, open public listeners, install Longhorn, or mutate the live single-node cluster merely to make the target appear ready.

Why This Belongs Before Forgejo

Forgejo will be the source forge, package base, and Actions surface for the Railiance stack. Moving it before the production cluster lifecycle is clear would make Forgejo both the migration target and the infrastructure experiment.

ThreePhoenix should come first, or at least its lifecycle gates should be designed first, so Forgejo is deployed onto a substrate whose failure and promotion behavior is already understood.

Boundary

This workplan is S2 cluster runtime work.

In scope for railiance-cluster:

• k3s HA topology and runtime configuration.
• Cluster-level storage/operator installation hooks.
• Ingress and certificate controllers.
• Cluster health, rotation, and acceptance checks.

Out of scope:

• Database cluster definitions and credentials: railiance-platform.
• Forgejo/Gitea application Helm values: railiance-apps.
• Developer workflows and Actions templates: railiance-enablement.
• OS provisioning and host hardening: railiance-infra.
• Reef membership and grouped substrate identity: reef-railiance.
• Workload-specific production approval: the owning rapp-* repo and reef-railiance binding (ADR-0006).
• PostgreSQL clusters and credentials: railiance-platform (CNPG operator may already be installed here; clusters are not).
• Forgejo/Gitea HA as a product: rapp-forgejo / railiance-apps / railiance-forge.
• Monitoring stack: railiance-telemetry (signal contract) and a future rapp-* (deployable). This file keeps only the S2 drill.

Failure-domain contract

For this workplan, three Kubernetes nodes are not sufficient merely because three node objects exist. Each node must map to a source-backed reef member, and the acceptance record must identify shared physical host, power, storage, and network dependencies. Nodes implemented as guests on one physical host do not constitute independent failure domains.

The minimum S2 claim is survival of any one Kubernetes server loss while:

• embedded-etcd quorum and the Kubernetes API remain available;
• scheduling, cluster DNS, networking, and the private service path continue;
• cluster-level secret and policy controllers remain functional; and
• the removed server can rejoin without manual reconstruction of cluster state.

Tasks

T01 - K3s HA cluster setup

id: RCLUSTER-WP-0007-T01
status: todo
priority: high
state_hub_task_id: "1f8a8668-31eb-4d79-bbcd-50f6430a8d66"

Implement the three-node k3s HA cluster setup using embedded etcd.

Minimum scope:

• Define node roles and join sequence.
• Automate first server and additional server joins.
• Validate etcd quorum.
• Document failure behavior for one missing node.
• Publish the source-backed node-to-reef-member and failure-domain map.
• Record shared dependencies; do not count co-located guests as independent domains.

Done when: three nodes can form a healthy k3s HA cluster from documented commands and the cluster retains etcd quorum, API availability, DNS, networking, and scheduling with each server removed in turn.

---

T02 - Longhorn distributed storage

id: RCLUSTER-WP-0007-T02
status: todo
priority: high
state_hub_task_id: "b1d4e0fa-da41-4b13-a7d6-34dd040cb605"

Install and validate distributed storage for stateful workloads.

Minimum scope:

• Storage prerequisites and node labeling.
• Longhorn installation or approved alternative.
• Default storage class decision.
• Volume replica and recovery behavior.
• Backup target handoff to railiance-platform where appropriate.

Done when: a test PVC survives a node disruption according to the ThreePhoenix acceptance criteria.

---

T03 - PostgreSQL HA pattern

id: RCLUSTER-WP-0007-T03
status: cancel
priority: high
state_hub_task_id: "11283b4c-7e4d-490d-91b3-0d06a593bdf0"

Cancelled 2026-08-15. Database clusters are railiance-platform (CNPG). S2 already has the operator installed on railiance01. Do not define repmgr/Pgpool or a cluster here.

---

T04 - Reference stateful application HA

id: RCLUSTER-WP-0007-T04
status: cancel
priority: high
state_hub_task_id: "4a20e593-a89d-43da-abcc-5a39a4c8b3c0"

Cancelled 2026-08-15. A source-forge HA product belongs to railiance-forge / rapp-forgejo, not S2. T01/T02/T06 already cover PVC and node disruption at the cluster layer.

---

T05 - Traefik ingress and cert-manager SSL

id: RCLUSTER-WP-0007-T05
status: todo
priority: medium
state_hub_task_id: "68315a40-dd5b-4032-a9e7-1152e38f9807"

Validate the production ingress and certificate path on the HA topology. Live railiance01 already uses Traefik (k3s default) and cert-manager. Do not introduce nginx.

Minimum scope:

• Ingress controller topology on the three-node target.
• TLS issuance and renewal.
• ADR-0008: no public Ingress class unless a grant exists; ClusterIP is the paved Service; 6443 stays operator-only. Packet policy for rapps is RAIL-K8S-WP-0003 / RMASTER-WP-0023.
• Health checks for ingress and certificate validity.

Done when: a granted representative hostname can be served with valid certificates, and an ungranted Service stays ClusterIP.

---

T06 - Phoenix CronJob automation

id: RCLUSTER-WP-0007-T06
status: todo
priority: medium
state_hub_task_id: "f658aa6a-1c48-4660-88fa-35eaa0137e12"

Implement weekly node rotation or equivalent Phoenix recovery automation.

Minimum scope:

• Define what "rotation" means for the current host reality.
• Automate safe cordon, drain, rebuild/rejoin, and validation steps where feasible.
• Include explicit human gates for destructive host actions.
• Log rotation results to State Hub.

Done when: the cluster recovery rhythm is scripted, documented, and tested without risking production data.

---

T07 - Monitoring stack and acceptance audit checklist

id: RCLUSTER-WP-0007-T07
status: todo
priority: medium
state_hub_task_id: "70f6c8ab-a700-4fb2-893e-cf5a40615044"

Add the monitoring stack and final acceptance audit checklist.

Minimum scope:

• Cluster health signals (what S2 can see). Do not stand up the telemetry stack here (railiance-telemetry).
• Storage health.
• Ingress and certificate health.
• A machine-readable one-server-loss drill: pre-failure state, failure injection, quorum and service observations, recovery time, rejoin, post-recovery state.
• A handoff that lets reef-railiance cite this as ADR-0006 reef evidence without treating it as production-approved.

Done when: the checklist and one-server-loss drill pass. Bindings stay at their last proven ADR-0006 state.

Dependencies

This workplan should precede the Forgejo production cutover. It should also shape the Stage 2 and Stage 3 gates in RAIL-BS-WP-0006 so canaries and promotions operate against the real HA substrate.

The first executable dependency is the machine-checkable/manual preflight in docs/threephoenix-implementation-gate.md; until it passes, this plan is design and dependency preparation only.

It is the substrate dependency for:

• reef-railiance/REEF-RAILIANCE-WP-0003 T04. After the S2 drill passes, the Qonto owner must rerun its own fresh live gate while Railiance01 is absent before the binding can become production-approved.
• state-hub/CUST-WP-0038 T01. State Hub consumes the published cluster readiness record and then performs its own database, API, backup, and access drills.