---
id: RCLUSTER-WP-0007
type: workplan
title: "ThreePhoenix - HA Cluster Implementation"
domain: financials
repo: railiance-cluster
status: blocked
owner: codex
topic_slug: railiance
repo_goal_id: "6ea441f7-7fe3-4598-922b-38baf20c0580"
state_hub_workstream_id: "16da36fe-5a10-522b-b6d6-02dbb6de6c14"
created: "2026-02-25"
updated: "2026-08-22"
related:
  - REEF-RAILIANCE-WP-0003
  - RAIL-BS-WP-0014
  - RMASTER-WP-0023
  - ADR-0006
  - ADR-0008
---

# ThreePhoenix - HA Cluster Implementation

## Goal

Implement the ThreePhoenix architecture: a self-healing three-node Kubernetes
cluster substrate for Railiance production systems.

The cluster target includes:

- k3s HA with embedded etcd.
- Cluster-level storage operator hooks (not the database clusters).
- Ingress and certificate controllers (live path is Traefik, not nginx).
- Node rotation and recovery drills.
- A machine-readable one-server-loss S2 evidence pack.

This is the S2 substrate work that can clear the open failure-domain gate
for critical workloads on `reef-railiance`, beginning with
`REEF-RAILIANCE-WP-0003`. Cluster readiness is ADR-0006 **reef evidence**
for `verified`. It must not write `production-approved` on a binding.
Current reality is one Ready node (`railiance01`), local storage, Traefik.
Do not report ThreePhoenix targets as live capacity (`RAIL-BS-WP-0014`).

Any listener this plan adds is ADR-0008 `private` unless a named grant
exists. `6443` is not grantable as `public`. An Ingress object is not a
grant.

The bootstrap copy of this file is retired. Do not re-open
`RCLUSTER-WP-0007` in `railiance-bootstrap`.

## Implementation gate

This workplan is `blocked`; its tasks must not move to `progress` until
the preflight in `docs/threephoenix-implementation-gate.md` passes. In
particular, three independent source-backed reef members must be approved,
provisioned, and S1-converged, with their shared power, storage, host, and
network dependencies recorded. The current Railiance01 node plus guests on one
physical host do not satisfy that gate.

Passing the gate also requires an approved private inter-node network design,
join-token custody/rotation through the platform custody path, and explicit
operator approval for destructive node actions. This repository must not buy
provider capacity, open public listeners, install Longhorn, or mutate the live
single-node cluster merely to make the target appear ready.

## Why This Belongs Before Forgejo

Forgejo will be the source forge, package base, and Actions surface for the
Railiance stack. Moving it before the production cluster lifecycle is clear
would make Forgejo both the migration target and the infrastructure experiment.

ThreePhoenix should come first, or at least its lifecycle gates should be
designed first, so Forgejo is deployed onto a substrate whose failure and
promotion behavior is already understood.

## Boundary

This workplan is S2 cluster runtime work.

In scope for `railiance-cluster`:

- k3s HA topology and runtime configuration.
- Cluster-level storage/operator installation hooks.
- Ingress and certificate controllers.
- Cluster health, rotation, and acceptance checks.

Out of scope:

- Database cluster definitions and credentials: `railiance-platform`.
- Forgejo/Gitea application Helm values: `railiance-apps`.
- Developer workflows and Actions templates: `railiance-enablement`.
- OS provisioning and host hardening: `railiance-infra`.
- Reef membership and grouped substrate identity: `reef-railiance`.
- Workload-specific production approval: the owning `rapp-*` repo and
  `reef-railiance` binding (ADR-0006).
- PostgreSQL clusters and credentials: `railiance-platform` (CNPG
  operator may already be installed here; clusters are not).
- Forgejo/Gitea HA as a product: `rapp-forgejo` / `railiance-apps` /
  `railiance-forge`.
- Monitoring stack: `railiance-telemetry` (signal contract) and a
  future `rapp-*` (deployable). This file keeps only the S2 drill.

## Failure-domain contract

For this workplan, three Kubernetes nodes are not sufficient merely because
three node objects exist. Each node must map to a source-backed reef member,
and the acceptance record must identify shared physical host, power, storage,
and network dependencies. Nodes implemented as guests on one physical host do
not constitute independent failure domains.

The minimum S2 claim is survival of any one Kubernetes server loss while:

- embedded-etcd quorum and the Kubernetes API remain available;
- scheduling, cluster DNS, networking, and the private service path continue;
- cluster-level secret and policy controllers remain functional; and
- the removed server can rejoin without manual reconstruction of cluster
  state.

## Tasks

### T01 - K3s HA cluster setup

```task
id: RCLUSTER-WP-0007-T01
status: todo
priority: high
state_hub_task_id: "9d1f463f-e4d1-5e75-97fd-e57b833bc2b9"
```

Implement the three-node k3s HA cluster setup using embedded etcd.

Minimum scope:

- Define node roles and join sequence.
- Automate first server and additional server joins.
- Validate etcd quorum.
- Document failure behavior for one missing node.
- Publish the source-backed node-to-reef-member and failure-domain map.
- Record shared dependencies; do not count co-located guests as independent
  domains.

**Done when:** three nodes can form a healthy k3s HA cluster from documented
commands and the cluster retains etcd quorum, API availability, DNS, networking,
and scheduling with each server removed in turn.

---

### T02 - Longhorn distributed storage

```task
id: RCLUSTER-WP-0007-T02
status: todo
priority: high
state_hub_task_id: "a1f4fa2b-69a7-5de4-8c44-15e8f9baf0b2"
```

Install and validate distributed storage for stateful workloads.

Minimum scope:

- Storage prerequisites and node labeling.
- Longhorn installation or approved alternative.
- Default storage class decision.
- Volume replica and recovery behavior.
- Backup target handoff to `railiance-platform` where appropriate.

**Done when:** a test PVC survives a node disruption according to the
ThreePhoenix acceptance criteria.

---

### T03 - PostgreSQL HA pattern

```task
id: RCLUSTER-WP-0007-T03
status: cancel
priority: high
state_hub_task_id: "11283b4c-7e4d-490d-91b3-0d06a593bdf0"
```

Cancelled 2026-08-15. Database clusters are `railiance-platform` (CNPG).
S2 already has the operator installed on `railiance01`. Do not define
repmgr/Pgpool or a cluster here.

---

### T04 - Reference stateful application HA

```task
id: RCLUSTER-WP-0007-T04
status: cancel
priority: high
state_hub_task_id: "4a20e593-a89d-43da-abcc-5a39a4c8b3c0"
```

Cancelled 2026-08-15. A source-forge HA product belongs to
`railiance-forge` / `rapp-forgejo`, not S2. T01/T02/T06 already cover
PVC and node disruption at the cluster layer.

---

### T05 - Traefik ingress and cert-manager SSL

```task
id: RCLUSTER-WP-0007-T05
status: todo
priority: medium
state_hub_task_id: "02cc3adb-9d42-5540-9f32-4e5f1198b79e"
```

Validate the production ingress and certificate path on the HA topology.
Live `railiance01` already uses Traefik (k3s default) and cert-manager.
Do not introduce nginx.

Minimum scope:

- Ingress controller topology on the three-node target.
- TLS issuance and renewal.
- ADR-0008: no public Ingress class unless a grant exists; ClusterIP is
  the paved Service; `6443` stays operator-only. Packet policy for
  rapps is `RAIL-K8S-WP-0003` / `RMASTER-WP-0023`.
- Health checks for ingress and certificate validity.

**Done when:** a *granted* representative hostname can be served with
valid certificates, and an ungranted Service stays ClusterIP.

---

### T06 - Phoenix CronJob automation

```task
id: RCLUSTER-WP-0007-T06
status: todo
priority: medium
state_hub_task_id: "4f033381-24aa-5773-bca6-337ccacc0e7a"
```

Implement weekly node rotation or equivalent Phoenix recovery automation.

Minimum scope:

- Define what "rotation" means for the current host reality.
- Automate safe cordon, drain, rebuild/rejoin, and validation steps where
  feasible.
- Include explicit human gates for destructive host actions.
- Log rotation results to State Hub.

**Done when:** the cluster recovery rhythm is scripted, documented, and tested
without risking production data.

---

### T07 - Monitoring stack and acceptance audit checklist

```task
id: RCLUSTER-WP-0007-T07
status: todo
priority: medium
state_hub_task_id: "b45b3086-3beb-56d4-a028-6bf651834992"
```

Add the monitoring stack and final acceptance audit checklist.

Minimum scope:

- Cluster health signals (what S2 can see). Do not stand up the
  telemetry stack here (`railiance-telemetry`).
- Storage health.
- Ingress and certificate health.
- A machine-readable one-server-loss drill: pre-failure state, failure
  injection, quorum and service observations, recovery time, rejoin,
  post-recovery state.
- A handoff that lets `reef-railiance` cite this as ADR-0006 reef
  evidence without treating it as `production-approved`.

**Done when:** the checklist and one-server-loss drill pass. Bindings
stay at their last proven ADR-0006 state.

## Dependencies

This workplan should precede the Forgejo production cutover. It should also
shape the Stage 2 and Stage 3 gates in `RAIL-BS-WP-0006` so canaries and
promotions operate against the real HA substrate.

The first executable dependency is the machine-checkable/manual preflight in
`docs/threephoenix-implementation-gate.md`; until it passes, this plan is design
and dependency preparation only.

It is the substrate dependency for:

- `reef-railiance/REEF-RAILIANCE-WP-0003` T04. After the S2 drill passes, the
  Qonto owner must rerun its own fresh live gate while Railiance01 is absent
  before the binding can become `production-approved`.
- `state-hub/CUST-WP-0038` T01. State Hub consumes the published cluster
  readiness record and then performs its own database, API, backup, and access
  drills.