From e2374eb454f7a913d24726a6f8a3ba4a7c4db972 Mon Sep 17 00:00:00 2001 From: codex Date: Sat, 8 Aug 2026 22:44:09 +0200 Subject: [PATCH] Define ThreePhoenix failure-domain acceptance --- ...RAIL-BS-WP-0007-threephoenix-ha-cluster.md | 49 +++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md b/workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md index a8df990..5f93e55 100644 --- a/workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md +++ b/workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md @@ -10,7 +10,7 @@ topic_slug: railiance repo_goal_id: "6ea441f7-7fe3-4598-922b-38baf20c0580" state_hub_workstream_id: "9e208376-23f1-40c7-9813-fac1f7d6ad3b" created: "2026-02-25" -updated: "2026-07-07" +updated: "2026-08-08" --- # ThreePhoenix - HA Cluster Implementation @@ -29,6 +29,12 @@ The cluster target includes: - Node rotation and recovery drills. - Monitoring and acceptance audits. +This is also the S2 substrate work that can clear the open failure-domain gate +for critical workloads on `reef-railiance`, beginning with +`REEF-RAILIANCE-WP-0003`. Cluster readiness alone does not promote a workload: +each consumer retains its own identity, network, dependency, rollback, and +application-level failover gates. + ## Why This Belongs Before Forgejo Forgejo will be the source forge, package base, and Actions surface for the @@ -56,6 +62,25 @@ Out of scope: - Forgejo/Gitea application Helm values: `railiance-apps`. - Developer workflows and Actions templates: `railiance-enablement`. - OS provisioning and host hardening: `railiance-infra`. +- Reef membership and grouped substrate identity: `reef-railiance`. +- Workload-specific production approval: the owning `rapp-*` repo and + `reef-railiance` binding. + +## Failure-domain contract + +For this workplan, three Kubernetes nodes are not sufficient merely because +three node objects exist. Each node must map to a source-backed reef member, +and the acceptance record must identify shared physical host, power, storage, +and network dependencies. Nodes implemented as guests on one physical host do +not constitute independent failure domains. + +The minimum S2 claim is survival of any one Kubernetes server loss while: + +- embedded-etcd quorum and the Kubernetes API remain available; +- scheduling, cluster DNS, networking, and the private service path continue; +- cluster-level secret and policy controllers remain functional; and +- the removed server can rejoin without manual reconstruction of cluster + state. ## Tasks @@ -76,9 +101,13 @@ Minimum scope: - Automate first server and additional server joins. - Validate etcd quorum. - Document failure behavior for one missing node. +- Publish the source-backed node-to-reef-member and failure-domain map. +- Record shared dependencies; do not count co-located guests as independent + domains. **Done when:** three nodes can form a healthy k3s HA cluster from documented -commands. +commands and the cluster retains etcd quorum, API availability, DNS, networking, +and scheduling with each server removed in turn. --- @@ -218,12 +247,26 @@ Minimum scope: - Ingress and certificate health. - Backup/restore freshness. - Promotion lifecycle readiness. +- A machine-readable one-server-loss drill, including pre-failure state, + failure injection, quorum and service observations, recovery time, rejoin, + and post-recovery state. +- A downstream handoff that lets `reef-railiance` and individual workload + owners cite the S2 evidence without treating it as workload approval. **Done when:** ThreePhoenix can be declared ready for critical workloads only -after the checklist passes. +after the checklist and one-server-loss drill pass. ## Dependencies This workplan should precede the Forgejo production cutover. It should also shape the Stage 2 and Stage 3 gates in `RAIL-BS-WP-0006` so canaries and promotions operate against the real HA substrate. + +It is the substrate dependency for: + +- `reef-railiance/REEF-RAILIANCE-WP-0003` T04. After the S2 drill passes, the + Qonto owner must rerun its own fresh live gate while Railiance01 is absent + before the binding can become `production-approved`. +- `state-hub/CUST-WP-0038` T01. State Hub consumes the published cluster + readiness record and then performs its own database, API, backup, and access + drills.