railiance-cluster/workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md

id	type	title	domain	repo	status	owner	topic_slug	repo_goal_id	state_hub_workstream_id	created	updated
RAIL-BS-WP-0007	workplan	ThreePhoenix - HA Cluster Implementation	financials	railiance-cluster	backlog	railiance	railiance	6ea441f7-7fe3-4598-922b-38baf20c0580	9e208376-23f1-40c7-9813-fac1f7d6ad3b	2026-02-25	2026-08-08

ThreePhoenix - HA Cluster Implementation

Goal

Implement the ThreePhoenix architecture: a self-healing three-node Kubernetes cluster substrate for Railiance production systems.

The cluster target includes:

• k3s HA with embedded etcd.
• Distributed storage.
• High-availability database patterns.
• Ingress and certificate automation.
• Node rotation and recovery drills.
• Monitoring and acceptance audits.

This is also the S2 substrate work that can clear the open failure-domain gate for critical workloads on reef-railiance, beginning with REEF-RAILIANCE-WP-0003. Cluster readiness alone does not promote a workload: each consumer retains its own identity, network, dependency, rollback, and application-level failover gates.

Why This Belongs Before Forgejo

Forgejo will be the source forge, package base, and Actions surface for the Railiance stack. Moving it before the production cluster lifecycle is clear would make Forgejo both the migration target and the infrastructure experiment.

ThreePhoenix should come first, or at least its lifecycle gates should be designed first, so Forgejo is deployed onto a substrate whose failure and promotion behavior is already understood.

Boundary

This workplan is S2 cluster runtime work.

In scope for railiance-cluster:

• k3s HA topology and runtime configuration.
• Cluster-level storage/operator installation hooks.
• Ingress and certificate controllers.
• Cluster health, rotation, and acceptance checks.

Out of scope:

• Database cluster definitions and credentials: railiance-platform.
• Forgejo/Gitea application Helm values: railiance-apps.
• Developer workflows and Actions templates: railiance-enablement.
• OS provisioning and host hardening: railiance-infra.
• Reef membership and grouped substrate identity: reef-railiance.
• Workload-specific production approval: the owning rapp-* repo and reef-railiance binding.

Failure-domain contract

For this workplan, three Kubernetes nodes are not sufficient merely because three node objects exist. Each node must map to a source-backed reef member, and the acceptance record must identify shared physical host, power, storage, and network dependencies. Nodes implemented as guests on one physical host do not constitute independent failure domains.

The minimum S2 claim is survival of any one Kubernetes server loss while:

• embedded-etcd quorum and the Kubernetes API remain available;
• scheduling, cluster DNS, networking, and the private service path continue;
• cluster-level secret and policy controllers remain functional; and
• the removed server can rejoin without manual reconstruction of cluster state.

Tasks

T01 - K3s HA cluster setup

id: RAIL-BS-WP-0007-T01
status: todo
priority: high
state_hub_task_id: "1f8a8668-31eb-4d79-bbcd-50f6430a8d66"

Implement the three-node k3s HA cluster setup using embedded etcd.

Minimum scope:

• Define node roles and join sequence.
• Automate first server and additional server joins.
• Validate etcd quorum.
• Document failure behavior for one missing node.
• Publish the source-backed node-to-reef-member and failure-domain map.
• Record shared dependencies; do not count co-located guests as independent domains.

Done when: three nodes can form a healthy k3s HA cluster from documented commands and the cluster retains etcd quorum, API availability, DNS, networking, and scheduling with each server removed in turn.

---

T02 - Longhorn distributed storage

id: RAIL-BS-WP-0007-T02
status: todo
priority: high
state_hub_task_id: "b1d4e0fa-da41-4b13-a7d6-34dd040cb605"

Install and validate distributed storage for stateful workloads.

Minimum scope:

• Storage prerequisites and node labeling.
• Longhorn installation or approved alternative.
• Default storage class decision.
• Volume replica and recovery behavior.
• Backup target handoff to railiance-platform where appropriate.

Done when: a test PVC survives a node disruption according to the ThreePhoenix acceptance criteria.

---

T03 - PostgreSQL HA pattern

id: RAIL-BS-WP-0007-T03
status: todo
priority: high
state_hub_task_id: "11283b4c-7e4d-490d-91b3-0d06a593bdf0"

Define the PostgreSQL HA runtime pattern and handoff to S3.

The original State Hub task names repmgr and Pgpool-II. Before implementation, reconcile that with the current Railiance production baseline using CloudNative PG.

Done when: the chosen HA database pattern is documented, tested, and owned by the correct layer without conflicting with railiance-platform.

---

T04 - Reference stateful application HA

id: RAIL-BS-WP-0007-T04
status: todo
priority: high
state_hub_task_id: "4a20e593-a89d-43da-abcc-5a39a4c8b3c0"

Validate a representative stateful source-forge workload on the HA cluster.

The historical task names Gitea. In the current roadmap this should become Forgejo unless a temporary Gitea reference drill is still useful.

Minimum checks:

• Repository storage survives pod reschedule and node disruption.
• Database failover behavior is understood.
• Package registry storage is included in backup/restore thinking.
• Application-level rollback is compatible with the staged promotion lifecycle.

Done when: Railiance has a proven stateful source-forge deployment pattern that can be reused for the Forgejo migration.

---

T05 - Nginx ingress and cert-manager SSL

id: RAIL-BS-WP-0007-T05
status: todo
priority: medium
state_hub_task_id: "68315a40-dd5b-4032-a9e7-1152e38f9807"

Implement and validate the production ingress and certificate path.

Minimum scope:

• Ingress controller topology.
• TLS certificate issuance and renewal.
• Private/public exposure rules.
• Health checks for ingress and certificate validity.

Done when: representative services can be exposed through the intended ingress path with valid certificates.

---

T06 - Phoenix CronJob automation

id: RAIL-BS-WP-0007-T06
status: todo
priority: medium
state_hub_task_id: "f658aa6a-1c48-4660-88fa-35eaa0137e12"

Implement weekly node rotation or equivalent Phoenix recovery automation.

Minimum scope:

• Define what "rotation" means for the current host reality.
• Automate safe cordon, drain, rebuild/rejoin, and validation steps where feasible.
• Include explicit human gates for destructive host actions.
• Log rotation results to State Hub.

Done when: the cluster recovery rhythm is scripted, documented, and tested without risking production data.

---

T07 - Monitoring stack and acceptance audit checklist

id: RAIL-BS-WP-0007-T07
status: todo
priority: medium
state_hub_task_id: "70f6c8ab-a700-4fb2-893e-cf5a40615044"

Add the monitoring stack and final acceptance audit checklist.

Minimum scope:

• Cluster health signals.
• Storage health.
• Database/operator health handoff.
• Ingress and certificate health.
• Backup/restore freshness.
• Promotion lifecycle readiness.
• A machine-readable one-server-loss drill, including pre-failure state, failure injection, quorum and service observations, recovery time, rejoin, and post-recovery state.
• A downstream handoff that lets reef-railiance and individual workload owners cite the S2 evidence without treating it as workload approval.

Done when: ThreePhoenix can be declared ready for critical workloads only after the checklist and one-server-loss drill pass.

Dependencies

This workplan should precede the Forgejo production cutover. It should also shape the Stage 2 and Stage 3 gates in RAIL-BS-WP-0006 so canaries and promotions operate against the real HA substrate.

It is the substrate dependency for:

• reef-railiance/REEF-RAILIANCE-WP-0003 T04. After the S2 drill passes, the Qonto owner must rerun its own fresh live gate while Railiance01 is absent before the binding can become production-approved.
• state-hub/CUST-WP-0038 T01. State Hub consumes the published cluster readiness record and then performs its own database, API, backup, and access drills.