ea0452b9a9
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
296 lines
9.6 KiB
Markdown
296 lines
9.6 KiB
Markdown
---
|
|
id: RCLUSTER-WP-0007
|
|
type: workplan
|
|
title: "ThreePhoenix - HA Cluster Implementation"
|
|
domain: financials
|
|
repo: railiance-cluster
|
|
status: backlog
|
|
owner: codex
|
|
topic_slug: railiance
|
|
repo_goal_id: "6ea441f7-7fe3-4598-922b-38baf20c0580"
|
|
state_hub_workstream_id: "9e208376-23f1-40c7-9813-fac1f7d6ad3b"
|
|
created: "2026-02-25"
|
|
updated: "2026-08-22"
|
|
related:
|
|
- REEF-RAILIANCE-WP-0003
|
|
- RAIL-BS-WP-0014
|
|
- RMASTER-WP-0023
|
|
- ADR-0006
|
|
- ADR-0008
|
|
---
|
|
|
|
# ThreePhoenix - HA Cluster Implementation
|
|
|
|
## Goal
|
|
|
|
Implement the ThreePhoenix architecture: a self-healing three-node Kubernetes
|
|
cluster substrate for Railiance production systems.
|
|
|
|
The cluster target includes:
|
|
|
|
- k3s HA with embedded etcd.
|
|
- Cluster-level storage operator hooks (not the database clusters).
|
|
- Ingress and certificate controllers (live path is Traefik, not nginx).
|
|
- Node rotation and recovery drills.
|
|
- A machine-readable one-server-loss S2 evidence pack.
|
|
|
|
This is the S2 substrate work that can clear the open failure-domain gate
|
|
for critical workloads on `reef-railiance`, beginning with
|
|
`REEF-RAILIANCE-WP-0003`. Cluster readiness is ADR-0006 **reef evidence**
|
|
for `verified`. It must not write `production-approved` on a binding.
|
|
Current reality is one Ready node (`railiance01`), local storage, Traefik.
|
|
Do not report ThreePhoenix targets as live capacity (`RAIL-BS-WP-0014`).
|
|
|
|
Any listener this plan adds is ADR-0008 `private` unless a named grant
|
|
exists. `6443` is not grantable as `public`. An Ingress object is not a
|
|
grant.
|
|
|
|
The bootstrap copy of this file is retired. Do not re-open
|
|
`RCLUSTER-WP-0007` in `railiance-bootstrap`.
|
|
|
|
## Implementation gate
|
|
|
|
This workplan remains `backlog`; its tasks must not move to `progress` until
|
|
the preflight in `docs/threephoenix-implementation-gate.md` passes. In
|
|
particular, three independent source-backed reef members must be approved,
|
|
provisioned, and S1-converged, with their shared power, storage, host, and
|
|
network dependencies recorded. The current Railiance01 node plus guests on one
|
|
physical host do not satisfy that gate.
|
|
|
|
Passing the gate also requires an approved private inter-node network design,
|
|
join-token custody/rotation through the platform custody path, and explicit
|
|
operator approval for destructive node actions. This repository must not buy
|
|
provider capacity, open public listeners, install Longhorn, or mutate the live
|
|
single-node cluster merely to make the target appear ready.
|
|
|
|
## Why This Belongs Before Forgejo
|
|
|
|
Forgejo will be the source forge, package base, and Actions surface for the
|
|
Railiance stack. Moving it before the production cluster lifecycle is clear
|
|
would make Forgejo both the migration target and the infrastructure experiment.
|
|
|
|
ThreePhoenix should come first, or at least its lifecycle gates should be
|
|
designed first, so Forgejo is deployed onto a substrate whose failure and
|
|
promotion behavior is already understood.
|
|
|
|
## Boundary
|
|
|
|
This workplan is S2 cluster runtime work.
|
|
|
|
In scope for `railiance-cluster`:
|
|
|
|
- k3s HA topology and runtime configuration.
|
|
- Cluster-level storage/operator installation hooks.
|
|
- Ingress and certificate controllers.
|
|
- Cluster health, rotation, and acceptance checks.
|
|
|
|
Out of scope:
|
|
|
|
- Database cluster definitions and credentials: `railiance-platform`.
|
|
- Forgejo/Gitea application Helm values: `railiance-apps`.
|
|
- Developer workflows and Actions templates: `railiance-enablement`.
|
|
- OS provisioning and host hardening: `railiance-infra`.
|
|
- Reef membership and grouped substrate identity: `reef-railiance`.
|
|
- Workload-specific production approval: the owning `rapp-*` repo and
|
|
`reef-railiance` binding (ADR-0006).
|
|
- PostgreSQL clusters and credentials: `railiance-platform` (CNPG
|
|
operator may already be installed here; clusters are not).
|
|
- Forgejo/Gitea HA as a product: `rapp-forgejo` / `railiance-apps` /
|
|
`railiance-forge`.
|
|
- Monitoring stack: `railiance-telemetry` (signal contract) and a
|
|
future `rapp-*` (deployable). This file keeps only the S2 drill.
|
|
|
|
## Failure-domain contract
|
|
|
|
For this workplan, three Kubernetes nodes are not sufficient merely because
|
|
three node objects exist. Each node must map to a source-backed reef member,
|
|
and the acceptance record must identify shared physical host, power, storage,
|
|
and network dependencies. Nodes implemented as guests on one physical host do
|
|
not constitute independent failure domains.
|
|
|
|
The minimum S2 claim is survival of any one Kubernetes server loss while:
|
|
|
|
- embedded-etcd quorum and the Kubernetes API remain available;
|
|
- scheduling, cluster DNS, networking, and the private service path continue;
|
|
- cluster-level secret and policy controllers remain functional; and
|
|
- the removed server can rejoin without manual reconstruction of cluster
|
|
state.
|
|
|
|
## Tasks
|
|
|
|
### T01 - K3s HA cluster setup
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T01
|
|
status: todo
|
|
priority: high
|
|
state_hub_task_id: "1f8a8668-31eb-4d79-bbcd-50f6430a8d66"
|
|
```
|
|
|
|
Implement the three-node k3s HA cluster setup using embedded etcd.
|
|
|
|
Minimum scope:
|
|
|
|
- Define node roles and join sequence.
|
|
- Automate first server and additional server joins.
|
|
- Validate etcd quorum.
|
|
- Document failure behavior for one missing node.
|
|
- Publish the source-backed node-to-reef-member and failure-domain map.
|
|
- Record shared dependencies; do not count co-located guests as independent
|
|
domains.
|
|
|
|
**Done when:** three nodes can form a healthy k3s HA cluster from documented
|
|
commands and the cluster retains etcd quorum, API availability, DNS, networking,
|
|
and scheduling with each server removed in turn.
|
|
|
|
---
|
|
|
|
### T02 - Longhorn distributed storage
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T02
|
|
status: todo
|
|
priority: high
|
|
state_hub_task_id: "b1d4e0fa-da41-4b13-a7d6-34dd040cb605"
|
|
```
|
|
|
|
Install and validate distributed storage for stateful workloads.
|
|
|
|
Minimum scope:
|
|
|
|
- Storage prerequisites and node labeling.
|
|
- Longhorn installation or approved alternative.
|
|
- Default storage class decision.
|
|
- Volume replica and recovery behavior.
|
|
- Backup target handoff to `railiance-platform` where appropriate.
|
|
|
|
**Done when:** a test PVC survives a node disruption according to the
|
|
ThreePhoenix acceptance criteria.
|
|
|
|
---
|
|
|
|
### T03 - PostgreSQL HA pattern
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T03
|
|
status: cancel
|
|
priority: high
|
|
state_hub_task_id: "11283b4c-7e4d-490d-91b3-0d06a593bdf0"
|
|
```
|
|
|
|
Cancelled 2026-08-15. Database clusters are `railiance-platform` (CNPG).
|
|
S2 already has the operator installed on `railiance01`. Do not define
|
|
repmgr/Pgpool or a cluster here.
|
|
|
|
---
|
|
|
|
### T04 - Reference stateful application HA
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T04
|
|
status: cancel
|
|
priority: high
|
|
state_hub_task_id: "4a20e593-a89d-43da-abcc-5a39a4c8b3c0"
|
|
```
|
|
|
|
Cancelled 2026-08-15. A source-forge HA product belongs to
|
|
`railiance-forge` / `rapp-forgejo`, not S2. T01/T02/T06 already cover
|
|
PVC and node disruption at the cluster layer.
|
|
|
|
---
|
|
|
|
### T05 - Traefik ingress and cert-manager SSL
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T05
|
|
status: todo
|
|
priority: medium
|
|
state_hub_task_id: "68315a40-dd5b-4032-a9e7-1152e38f9807"
|
|
```
|
|
|
|
Validate the production ingress and certificate path on the HA topology.
|
|
Live `railiance01` already uses Traefik (k3s default) and cert-manager.
|
|
Do not introduce nginx.
|
|
|
|
Minimum scope:
|
|
|
|
- Ingress controller topology on the three-node target.
|
|
- TLS issuance and renewal.
|
|
- ADR-0008: no public Ingress class unless a grant exists; ClusterIP is
|
|
the paved Service; `6443` stays operator-only. Packet policy for
|
|
rapps is `RAIL-K8S-WP-0003` / `RMASTER-WP-0023`.
|
|
- Health checks for ingress and certificate validity.
|
|
|
|
**Done when:** a *granted* representative hostname can be served with
|
|
valid certificates, and an ungranted Service stays ClusterIP.
|
|
|
|
---
|
|
|
|
### T06 - Phoenix CronJob automation
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T06
|
|
status: todo
|
|
priority: medium
|
|
state_hub_task_id: "f658aa6a-1c48-4660-88fa-35eaa0137e12"
|
|
```
|
|
|
|
Implement weekly node rotation or equivalent Phoenix recovery automation.
|
|
|
|
Minimum scope:
|
|
|
|
- Define what "rotation" means for the current host reality.
|
|
- Automate safe cordon, drain, rebuild/rejoin, and validation steps where
|
|
feasible.
|
|
- Include explicit human gates for destructive host actions.
|
|
- Log rotation results to State Hub.
|
|
|
|
**Done when:** the cluster recovery rhythm is scripted, documented, and tested
|
|
without risking production data.
|
|
|
|
---
|
|
|
|
### T07 - Monitoring stack and acceptance audit checklist
|
|
|
|
```task
|
|
id: RCLUSTER-WP-0007-T07
|
|
status: todo
|
|
priority: medium
|
|
state_hub_task_id: "70f6c8ab-a700-4fb2-893e-cf5a40615044"
|
|
```
|
|
|
|
Add the monitoring stack and final acceptance audit checklist.
|
|
|
|
Minimum scope:
|
|
|
|
- Cluster health signals (what S2 can see). Do not stand up the
|
|
telemetry stack here (`railiance-telemetry`).
|
|
- Storage health.
|
|
- Ingress and certificate health.
|
|
- A machine-readable one-server-loss drill: pre-failure state, failure
|
|
injection, quorum and service observations, recovery time, rejoin,
|
|
post-recovery state.
|
|
- A handoff that lets `reef-railiance` cite this as ADR-0006 reef
|
|
evidence without treating it as `production-approved`.
|
|
|
|
**Done when:** the checklist and one-server-loss drill pass. Bindings
|
|
stay at their last proven ADR-0006 state.
|
|
|
|
## Dependencies
|
|
|
|
This workplan should precede the Forgejo production cutover. It should also
|
|
shape the Stage 2 and Stage 3 gates in `RAIL-BS-WP-0006` so canaries and
|
|
promotions operate against the real HA substrate.
|
|
|
|
The first executable dependency is the machine-checkable/manual preflight in
|
|
`docs/threephoenix-implementation-gate.md`; until it passes, this plan is design
|
|
and dependency preparation only.
|
|
|
|
It is the substrate dependency for:
|
|
|
|
- `reef-railiance/REEF-RAILIANCE-WP-0003` T04. After the S2 drill passes, the
|
|
Qonto owner must rerun its own fresh live gate while Railiance01 is absent
|
|
before the binding can become `production-approved`.
|
|
- `state-hub/CUST-WP-0038` T01. State Hub consumes the published cluster
|
|
readiness record and then performs its own database, API, backup, and access
|
|
drills.
|