Define ThreePhoenix failure-domain acceptance

workplans/RAIL-BS-WP-0007-threephoenix-ha-cluster.md

@@ -10,7 +10,7 @@ topic_slug: railiance
 repo_goal_id: "6ea441f7-7fe3-4598-922b-38baf20c0580"
 state_hub_workstream_id: "9e208376-23f1-40c7-9813-fac1f7d6ad3b"
 created: "2026-02-25"
-updated: "2026-07-07"
+updated: "2026-08-08"
 ---
 
 # ThreePhoenix - HA Cluster Implementation
@@ -29,6 +29,12 @@ The cluster target includes:
 - Node rotation and recovery drills.
 - Monitoring and acceptance audits.
 
+This is also the S2 substrate work that can clear the open failure-domain gate
+for critical workloads on `reef-railiance`, beginning with
+`REEF-RAILIANCE-WP-0003`. Cluster readiness alone does not promote a workload:
+each consumer retains its own identity, network, dependency, rollback, and
+application-level failover gates.
+
 ## Why This Belongs Before Forgejo
 
 Forgejo will be the source forge, package base, and Actions surface for the
@@ -56,6 +62,25 @@ Out of scope:
 - Forgejo/Gitea application Helm values: `railiance-apps`.
 - Developer workflows and Actions templates: `railiance-enablement`.
 - OS provisioning and host hardening: `railiance-infra`.
+- Reef membership and grouped substrate identity: `reef-railiance`.
+- Workload-specific production approval: the owning `rapp-*` repo and
+  `reef-railiance` binding.
+
+## Failure-domain contract
+
+For this workplan, three Kubernetes nodes are not sufficient merely because
+three node objects exist. Each node must map to a source-backed reef member,
+and the acceptance record must identify shared physical host, power, storage,
+and network dependencies. Nodes implemented as guests on one physical host do
+not constitute independent failure domains.
+
+The minimum S2 claim is survival of any one Kubernetes server loss while:
+
+- embedded-etcd quorum and the Kubernetes API remain available;
+- scheduling, cluster DNS, networking, and the private service path continue;
+- cluster-level secret and policy controllers remain functional; and
+- the removed server can rejoin without manual reconstruction of cluster
+  state.
 
 ## Tasks
 
@@ -76,9 +101,13 @@ Minimum scope:
 - Automate first server and additional server joins.
 - Validate etcd quorum.
 - Document failure behavior for one missing node.
+- Publish the source-backed node-to-reef-member and failure-domain map.
+- Record shared dependencies; do not count co-located guests as independent
+  domains.
 
 **Done when:** three nodes can form a healthy k3s HA cluster from documented
-commands.
+commands and the cluster retains etcd quorum, API availability, DNS, networking,
+and scheduling with each server removed in turn.
 
 ---
 
@@ -218,12 +247,26 @@ Minimum scope:
 - Ingress and certificate health.
 - Backup/restore freshness.
 - Promotion lifecycle readiness.
+- A machine-readable one-server-loss drill, including pre-failure state,
+  failure injection, quorum and service observations, recovery time, rejoin,
+  and post-recovery state.
+- A downstream handoff that lets `reef-railiance` and individual workload
+  owners cite the S2 evidence without treating it as workload approval.
 
 **Done when:** ThreePhoenix can be declared ready for critical workloads only
-after the checklist passes.
+after the checklist and one-server-loss drill pass.
 
 ## Dependencies
 
 This workplan should precede the Forgejo production cutover. It should also
 shape the Stage 2 and Stage 3 gates in `RAIL-BS-WP-0006` so canaries and
 promotions operate against the real HA substrate.
+
+It is the substrate dependency for:
+
+- `reef-railiance/REEF-RAILIANCE-WP-0003` T04. After the S2 drill passes, the
+  Qonto owner must rerun its own fresh live gate while Railiance01 is absent
+  before the binding can become `production-approved`.
+- `state-hub/CUST-WP-0038` T01. State Hub consumes the published cluster
+  readiness record and then performs its own database, API, backup, and access
+  drills.