Publish cluster resource evidence and gate ThreePhoenix
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02669-87ee-7a31-b111-edc95a16e0fa
This commit is contained in:
codex 2026-08-22 12:34:25 +02:00
parent bfc991258d
commit ea0452b9a9
11 changed files with 2768 additions and 23 deletions

View file

@ -21,6 +21,14 @@ k3s-install: ## Install k3s and Helm on all inventory hosts
smoke: ## Run Kubernetes smoke tests
bash tests/smoke_kube.sh
cluster-observe: ## Capture non-secret cluster capacity and allocation evidence
python3 tools/observe_cluster_resources.py \
$(if $(REMOTE),--remote $(REMOTE),) \
--output-dir docs/evidence/resource-reef-railiance-k3s/observations
test-unit: ## Run non-live unit tests
python3 -m unittest discover -s tests -p 'test_*.py' -v
test-ha-failover: ## Run HA failover test (D3) — kills primary PG pod, asserts recovery
bash tests/test_ha_failover.sh $(if $(GITEA_URL),$(GITEA_URL),)
@ -43,4 +51,4 @@ help: ## Show this help
/^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 } \
/^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST)
.PHONY: backup restore preflight k3s-install smoke test-ha-failover verify-activity-core reconcile-activity-core-llm-connect deploy-activity-core-triage-robustness admin-sync-smoke help
.PHONY: backup restore preflight k3s-install smoke cluster-observe test-unit test-ha-failover verify-activity-core reconcile-activity-core-llm-connect deploy-activity-core-triage-robustness admin-sync-smoke help

View file

@ -90,8 +90,8 @@ future `rail-kubernetes` repo and must not be treated as long-term S2 scope.
`platform-pg`, `state-hub-db`, `target-revenue-pg`, `user-engine-pg`
- No monitoring stack is deployed on either cluster — observability is Q2 and
homed in `railiance-telemetry`, which is seeded but not implemented
- k3s API access is source-restricted by UFW; see `railiance-infra`
`RAIL-HO-WP-0009`
- k3s API access is tunnel-only on Railiance01; see `railiance-infra`
`RAIL-HO-WP-0009` and ADR-005. It is not a public/source-allowlisted surface.
---
@ -130,15 +130,15 @@ future `rail-kubernetes` repo and must not be treated as long-term S2 scope.
```capability
type: infrastructure
title: Kubernetes cluster provisioning (k3s)
description: Install and configure a production k3s cluster including Helm, ingress controller, CNI plugin, and kubeconfig management on Railiance servers.
description: Install and configure a production k3s cluster including Helm, Traefik ingress, networking, and kubeconfig management on Railiance servers.
keywords: [kubernetes, k3s, cluster, helm, ingress, cni, k8s, provisioning]
```
```capability
type: infrastructure
title: Cluster operators and addon management
description: Deploy and manage cluster-wide operators and addons (cert-manager, CloudNative PG operator, ArgoCD, nginx ingress) on the running Railiance Kubernetes cluster.
keywords: [operator, addon, cert-manager, cnpg, argocd, admission, kubernetes, cluster]
description: Deploy and manage cluster-wide operators and addons, including cert-manager, CloudNative PG, Traefik, Knative Serving, External Secrets, and OpenBao integration, on the running Kubernetes cluster.
keywords: [operator, addon, cert-manager, cnpg, traefik, knative, admission, kubernetes, cluster]
```
```capability
@ -152,10 +152,10 @@ keywords: [backup, restore, age, encryption, k3s, state, helm, kubeconfig, disas
## Notes
Runs on COULOMBCORE (92.205.130.254). State Hub access via ops-bridge reverse
tunnel - `bridge up state-hub-coulombcore` from the workstation (see ADR-004).
Gitea Helm values were migrated to S5 (railiance-apps) in RAIL-HO-WP-0004-T06 -
boundary violation resolved.
The live primary is Railiance01 (`92.205.62.239`). CoulombCore
(`92.205.130.254`) is the older frozen cluster and remains a legacy recovery
surface, not the default deployment target. State Hub and Kubernetes API access
use the corresponding ops-bridge tunnels (see ADR-004/ADR-005).
`bin/railiance` lifecycle subcommands, `railiance/app.toml`, and overlay/wrapper
scaffolding stay in this repo only as a migration bridge. They are not the

View file

@ -16,7 +16,7 @@
| workplan | RAIL-BS-WP-0012 | finished | — | workplans/RAIL-BS-WP-0012-rail-kubernetes-extraction.md |
| workplan | RAIL-BS-WP-0013 | finished | — | workplans/RAIL-BS-WP-0013-knative-substrate.md |
| workplan | RCLUSTER-WP-0007 | backlog | — | workplans/RCLUSTER-WP-0007-threephoenix-ha-cluster.md |
| workplan | RCLUSTER-WP-0014 | ready | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| workplan | RCLUSTER-WP-0014 | finished | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RAIL-BS-ADHOC-2026-07-27-T01 | done | — | workplans/ADHOC-2026-07-27.md |
| task | RAIL-BS-WP-0008-T01 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md |
| task | RAIL-BS-WP-0008-T02 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md |
@ -48,6 +48,6 @@
| task | RCLUSTER-WP-0007-T05 | todo | — | workplans/RCLUSTER-WP-0007-threephoenix-ha-cluster.md |
| task | RCLUSTER-WP-0007-T06 | todo | — | workplans/RCLUSTER-WP-0007-threephoenix-ha-cluster.md |
| task | RCLUSTER-WP-0007-T07 | todo | — | workplans/RCLUSTER-WP-0007-threephoenix-ha-cluster.md |
| task | RCLUSTER-WP-0014-T01 | todo | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RCLUSTER-WP-0014-T02 | todo | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RCLUSTER-WP-0014-T03 | todo | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RCLUSTER-WP-0014-T01 | done | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RCLUSTER-WP-0014-T02 | done | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |
| task | RCLUSTER-WP-0014-T03 | done | — | workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md |

View file

@ -0,0 +1,14 @@
# Cluster resource evidence
`make cluster-observe` captures a non-secret, timestamped observation for
`resource:railiance:reef-railiance:k3s`. It separates allocatable capacity,
requests, limits, optional observed usage, PVC demand, storage-class behavior,
and measurement gaps. It never reads Secret objects or pod environments.
The allocation section publishes raw numerators, denominators, residual
capacity, and `cluster-raw-drivers-v1`. It does not select a financial
allocation formula or write booked cost; those decisions remain with
resource-control.
Current failure-domain evidence remains explicit: one Ready node on
`railiance01` with local storage. ThreePhoenix is a target, not live capacity.

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,37 @@
# ThreePhoenix implementation gate
`RCLUSTER-WP-0007` stays in backlog until every gate below is backed by
reviewable source or evidence. This prevents a target architecture from being
reported as live capacity and prevents destructive work on the current
single-node production cluster.
## Required preflight
- Three reef members have stable source-backed identities and map to three
independently provisioned, S1-converged hosts.
- The failure-domain record covers physical host, provider/site, power,
storage, and network dependencies. Co-located guests count as one domain.
- Private inter-node addresses, routes, and firewall rules are approved. The
Kubernetes API remains operator-only and no public 6443 grant is possible.
- k3s server roles, embedded-etcd join order, version pin, rollback path, and
quorum checks are reviewed.
- Join-token creation, delivery, rotation, and revocation use the approved
platform custody path; no token value is committed or copied into evidence.
- Storage prerequisites and the Longhorn-or-alternative decision are approved
against the actual host/storage failure domains.
- Backups and restore points exist before any migration or destructive test.
- An operator explicitly approves each drain, rebuild, detach, or failure
injection against a named host and maintenance window.
## Evidence required to open implementation
Record a dated preflight artifact that identifies the three reef-member and
host records, links the network and custody decisions, records current backup
checks, and names the approving operator. After that evidence is reviewed, the
workplan may move from `backlog` to `ready`; individual tasks still follow
their own `todo``progress``done` lifecycle.
Until then, allowed work is limited to source-only design, validation tooling,
and dependency preparation. Provider purchase, live cluster joins, firewall
mutation, Longhorn installation, and failure injection remain outside the
authorized stage.

View file

@ -0,0 +1,38 @@
from __future__ import annotations
import importlib.util
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location("cluster_resources", ROOT / "tools" / "observe_cluster_resources.py")
assert SPEC and SPEC.loader
module = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(module)
class ClusterResourceTests(unittest.TestCase):
def test_quantity_conversion(self) -> None:
self.assertEqual(250, module.cpu_millicores("250m"))
self.assertEqual(2000, module.cpu_millicores("2"))
self.assertEqual(1024**3, module.bytes_value("1Gi"))
def test_aggregation_and_missing_metrics(self) -> None:
snapshots = {
"nodes": {"items": [{"metadata": {"name": "node-1", "labels": {"kubernetes.io/hostname": "node-1"}}, "spec": {}, "status": {"allocatable": {"cpu": "2", "memory": "4Gi", "ephemeral-storage": "20Gi", "pods": "110"}, "conditions": [{"type": "Ready", "status": "True"}]}}]},
"pods": {"items": [{"metadata": {"name": "api-abc123", "namespace": "apps", "labels": {"app.kubernetes.io/name": "api"}}, "spec": {"containers": [{"resources": {"requests": {"cpu": "250m", "memory": "128Mi"}, "limits": {"cpu": "1", "memory": "512Mi"}}}]}}]},
"pvcs": {"items": [{"metadata": {"name": "data", "namespace": "apps"}, "spec": {"storageClassName": "local-path", "resources": {"requests": {"storage": "1Gi"}}}, "status": {"phase": "Bound"}}]},
"pvs": {"items": [{"spec": {"capacity": {"storage": "2Gi"}}}]},
"storageclasses": {"items": [{"metadata": {"name": "local-path", "annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}, "provisioner": "local-path", "reclaimPolicy": "Delete", "volumeBindingMode": "WaitForFirstConsumer"}]},
"node_top": None,
"pod_top": None,
}
result = module.build_observation(snapshots, "2026-08-22T00:00:00Z")
self.assertEqual(250, result["allocation"]["denominators"]["cpu_requested_millicores"])
self.assertEqual(1750, result["allocation"]["residual_capacity"]["cpu_millicores"])
self.assertIn("pod metrics unavailable", result["measurement_gaps"])
self.assertFalse(result["failure_domain"]["threephoenix_target_is_live"])
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Capture non-secret cluster capacity, utilization, and allocation evidence."""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
CPU_FACTORS = {"n": 0.000001, "u": 0.001, "m": 1.0, "": 1000.0}
BYTE_FACTORS = {
"": 1,
"Ki": 1024,
"Mi": 1024**2,
"Gi": 1024**3,
"Ti": 1024**4,
"K": 1000,
"M": 1000**2,
"G": 1000**3,
"T": 1000**4,
}
def cpu_millicores(value: str | None) -> int:
match = re.fullmatch(r"([0-9]+(?:\.[0-9]+)?)(n|u|m)?", str(value or "0"))
if not match:
raise ValueError(f"unsupported CPU quantity: {value}")
return round(float(match.group(1)) * CPU_FACTORS[match.group(2) or ""])
def bytes_value(value: str | None) -> int:
match = re.fullmatch(r"([0-9]+(?:\.[0-9]+)?)(Ki|Mi|Gi|Ti|K|M|G|T)?", str(value or "0"))
if not match:
raise ValueError(f"unsupported byte quantity: {value}")
return round(float(match.group(1)) * BYTE_FACTORS[match.group(2) or ""])
def workload_name(pod: dict[str, Any]) -> str:
metadata = pod.get("metadata") or {}
labels = metadata.get("labels") or {}
for key in ("app.kubernetes.io/part-of", "app.kubernetes.io/name", "app"):
if labels.get(key):
return str(labels[key])
owners = metadata.get("ownerReferences") or []
if owners and owners[0].get("name"):
return str(owners[0]["name"])
return re.sub(r"-[a-f0-9]{6,}(?:-[a-z0-9]{5})?$", "", str(metadata.get("name", "unknown")))
def parse_top(lines: str | None, *, pods: bool) -> dict[tuple[str, str] | str, dict[str, int]]:
result: dict[tuple[str, str] | str, dict[str, int]] = {}
for line in (lines or "").splitlines():
fields = line.split()
if pods and len(fields) >= 4:
result[(fields[0], fields[1])] = {
"cpu_millicores": cpu_millicores(fields[2]),
"memory_bytes": bytes_value(fields[3]),
}
elif not pods and len(fields) >= 4:
result[fields[0]] = {
"cpu_millicores": cpu_millicores(fields[1]),
# kubectl top nodes: NAME CPU CPU% MEMORY MEMORY%
"memory_bytes": bytes_value(fields[3]),
}
return result
def build_observation(snapshots: dict[str, Any], captured_at: str) -> dict[str, Any]:
gaps: list[str] = []
node_top = parse_top(snapshots.get("node_top", ""), pods=False)
pod_top = parse_top(snapshots.get("pod_top", ""), pods=True)
if snapshots.get("node_top") is None:
gaps.append("node metrics unavailable")
if snapshots.get("pod_top") is None:
gaps.append("pod metrics unavailable")
nodes = []
alloc_cpu = alloc_memory = alloc_pods = 0
for node in snapshots["nodes"].get("items") or []:
meta = node.get("metadata") or {}
status = node.get("status") or {}
alloc = status.get("allocatable") or {}
labels = meta.get("labels") or {}
name = str(meta.get("name"))
topology = {
"region": labels.get("topology.kubernetes.io/region"),
"zone": labels.get("topology.kubernetes.io/zone"),
"hostname": labels.get("kubernetes.io/hostname"),
"provider_id": (node.get("spec") or {}).get("providerID"),
}
if not topology["region"] or not topology["zone"]:
gaps.append(f"node {name} lacks independent region/zone labels")
row = {
"name": name,
"ready": any(c.get("type") == "Ready" and c.get("status") == "True" for c in status.get("conditions") or []),
"allocatable": {
"cpu_millicores": cpu_millicores(alloc.get("cpu")),
"memory_bytes": bytes_value(alloc.get("memory")),
"ephemeral_storage_bytes": bytes_value(alloc.get("ephemeral-storage")),
"pods": int(alloc.get("pods") or 0),
},
"topology": topology,
"observed": node_top.get(name),
}
nodes.append(row)
alloc_cpu += row["allocatable"]["cpu_millicores"]
alloc_memory += row["allocatable"]["memory_bytes"]
alloc_pods += row["allocatable"]["pods"]
workloads: dict[tuple[str, str], dict[str, Any]] = {}
for pod in snapshots["pods"].get("items") or []:
meta = pod.get("metadata") or {}
namespace = str(meta.get("namespace", "default"))
name = workload_name(pod)
key = (namespace, name)
row = workloads.setdefault(key, {
"namespace": namespace,
"workload": name,
"pods": 0,
"requests": {"cpu_millicores": 0, "memory_bytes": 0},
"limits": {"cpu_millicores": 0, "memory_bytes": 0},
"observed": {"cpu_millicores": 0, "memory_bytes": 0} if snapshots.get("pod_top") is not None else None,
})
row["pods"] += 1
for container in (pod.get("spec") or {}).get("containers") or []:
resources = container.get("resources") or {}
requests = resources.get("requests") or {}
limits = resources.get("limits") or {}
row["requests"]["cpu_millicores"] += cpu_millicores(requests.get("cpu"))
row["requests"]["memory_bytes"] += bytes_value(requests.get("memory"))
row["limits"]["cpu_millicores"] += cpu_millicores(limits.get("cpu"))
row["limits"]["memory_bytes"] += bytes_value(limits.get("memory"))
metric = pod_top.get((namespace, str(meta.get("name"))))
if metric and row["observed"] is not None:
row["observed"]["cpu_millicores"] += metric["cpu_millicores"]
row["observed"]["memory_bytes"] += metric["memory_bytes"]
storage_classes = []
for item in snapshots["storageclasses"].get("items") or []:
storage_classes.append({
"name": (item.get("metadata") or {}).get("name"),
"provisioner": item.get("provisioner"),
"reclaim_policy": item.get("reclaimPolicy"),
"volume_binding_mode": item.get("volumeBindingMode"),
"default": (item.get("metadata") or {}).get("annotations", {}).get("storageclass.kubernetes.io/is-default-class") == "true",
})
pv_capacity = sum(bytes_value(((item.get("spec") or {}).get("capacity") or {}).get("storage")) for item in snapshots["pvs"].get("items") or [])
pvc_rows = []
pvc_requested = 0
for item in snapshots["pvcs"].get("items") or []:
meta = item.get("metadata") or {}
spec = item.get("spec") or {}
requested = bytes_value(((spec.get("resources") or {}).get("requests") or {}).get("storage"))
pvc_requested += requested
pvc_rows.append({
"namespace": meta.get("namespace"),
"name": meta.get("name"),
"storage_class": spec.get("storageClassName"),
"requested_bytes": requested,
"phase": (item.get("status") or {}).get("phase"),
})
workload_rows = sorted(workloads.values(), key=lambda row: (row["namespace"], row["workload"]))
requested_cpu = sum(row["requests"]["cpu_millicores"] for row in workload_rows)
requested_memory = sum(row["requests"]["memory_bytes"] for row in workload_rows)
return {
"schema_version": "railiance.cluster-resource-observation.v1",
"record_type": "usage_observation",
"resource_id": "resource:railiance:reef-railiance:k3s",
"source": "railiance-cluster",
"workplan_id": "RCLUSTER-WP-0014",
"reef": "reef-railiance",
"captured_at": captured_at,
"capacity": {
"nodes": nodes,
"totals": {"cpu_millicores": alloc_cpu, "memory_bytes": alloc_memory, "pods": alloc_pods, "pv_bytes": pv_capacity},
},
"utilization": {"workloads": workload_rows},
"storage": {"classes": storage_classes, "claims": pvc_rows},
"failure_domain": {
"current_contract": "single Ready node on railiance01; local storage is correlated with node loss",
"independent_domains_claimed": 1 if nodes else 0,
"threephoenix_target_is_live": False,
},
"allocation": {
"method_version": "cluster-raw-drivers-v1",
"drivers": ["cpu_requests", "memory_requests", "pvc_requested_bytes", "observed_usage"],
"denominators": {"cpu_requested_millicores": requested_cpu, "memory_requested_bytes": requested_memory, "pvc_requested_bytes": pvc_requested},
"residual_capacity": {"cpu_millicores": max(alloc_cpu - requested_cpu, 0), "memory_bytes": max(alloc_memory - requested_memory, 0), "pv_bytes": max(pv_capacity - pvc_requested, 0)},
"selection_owner": "resource-control",
},
"measurement_gaps": sorted(set(gaps)),
"provenance": {
"commands": [
"kubectl get nodes -o json", "kubectl get pods -A -o json",
"kubectl get pvc -A -o json", "kubectl get pv -o json",
"kubectl get storageclass -o json", "kubectl top nodes --no-headers",
"kubectl top pods -A --no-headers",
],
"secret_surfaces_read": False,
},
}
def run_json(command: list[str]) -> dict[str, Any]:
return json.loads(subprocess.check_output(command, text=True))
def run_optional(command: list[str]) -> str | None:
completed = subprocess.run(command, text=True, capture_output=True, check=False)
return completed.stdout if completed.returncode == 0 else None
def collect(kubectl: str, remote: str | None = None) -> dict[str, Any]:
base = ["ssh", "-o", "BatchMode=yes", remote, kubectl] if remote else [kubectl]
snapshots = {
"nodes": run_json(base + ["get", "nodes", "-o", "json"]),
"pods": run_json(base + ["get", "pods", "-A", "-o", "json"]),
"pvcs": run_json(base + ["get", "pvc", "-A", "-o", "json"]),
"pvs": run_json(base + ["get", "pv", "-o", "json"]),
"storageclasses": run_json(base + ["get", "storageclass", "-o", "json"]),
"node_top": run_optional(base + ["top", "nodes", "--no-headers"]),
"pod_top": run_optional(base + ["top", "pods", "-A", "--no-headers"]),
}
captured_at = datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
return build_observation(snapshots, captured_at)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--kubectl", default="kubectl")
parser.add_argument("--remote", help="Optional SSH host on which kubectl runs")
parser.add_argument("--output-dir", type=Path)
args = parser.parse_args()
try:
observation = collect(args.kubectl, args.remote)
except (OSError, subprocess.CalledProcessError, json.JSONDecodeError, ValueError) as exc:
print(f"cluster observation failed: {exc}", file=sys.stderr)
return 1
rendered = json.dumps(observation, indent=2) + "\n"
if args.output_dir:
args.output_dir.mkdir(parents=True, exist_ok=True)
stamp = observation["captured_at"].replace(":", "").replace("-", "")
destination = args.output_dir / f"{stamp}.json"
destination.write_text(rendered, encoding="utf-8")
(args.output_dir / "latest.json").write_text(rendered, encoding="utf-8")
print(destination)
else:
sys.stdout.write(rendered)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -10,7 +10,7 @@ topic_slug: railiance
repo_goal_id: "6ea441f7-7fe3-4598-922b-38baf20c0580"
state_hub_workstream_id: "9e208376-23f1-40c7-9813-fac1f7d6ad3b"
created: "2026-02-25"
updated: "2026-08-15"
updated: "2026-08-22"
related:
- REEF-RAILIANCE-WP-0003
- RAIL-BS-WP-0014
@ -48,6 +48,21 @@ grant.
The bootstrap copy of this file is retired. Do not re-open
`RCLUSTER-WP-0007` in `railiance-bootstrap`.
## Implementation gate
This workplan remains `backlog`; its tasks must not move to `progress` until
the preflight in `docs/threephoenix-implementation-gate.md` passes. In
particular, three independent source-backed reef members must be approved,
provisioned, and S1-converged, with their shared power, storage, host, and
network dependencies recorded. The current Railiance01 node plus guests on one
physical host do not satisfy that gate.
Passing the gate also requires an approved private inter-node network design,
join-token custody/rotation through the platform custody path, and explicit
operator approval for destructive node actions. This repository must not buy
provider capacity, open public listeners, install Longhorn, or mutate the live
single-node cluster merely to make the target appear ready.
## Why This Belongs Before Forgejo
Forgejo will be the source forge, package base, and Actions surface for the
@ -267,6 +282,10 @@ This workplan should precede the Forgejo production cutover. It should also
shape the Stage 2 and Stage 3 gates in `RAIL-BS-WP-0006` so canaries and
promotions operate against the real HA substrate.
The first executable dependency is the machine-checkable/manual preflight in
`docs/threephoenix-implementation-gate.md`; until it passes, this plan is design
and dependency preparation only.
It is the substrate dependency for:
- `reef-railiance/REEF-RAILIANCE-WP-0003` T04. After the S2 drill passes, the

View file

@ -4,11 +4,11 @@ type: workplan
title: "Publish cluster capacity, utilization, and allocation evidence"
domain: financials
repo: railiance-cluster
status: ready
status: finished
owner: codex
topic_slug: railiance
created: "2026-08-11"
updated: "2026-08-15"
updated: "2026-08-22"
related:
- RAIL-BS-WP-0007
- RESOURCE-WP-0003
@ -42,7 +42,7 @@ fin-hub allocate. No booked EUR in this repo.
```task
id: RCLUSTER-WP-0014-T01
status: todo
status: done
priority: high
state_hub_task_id: "a144bf1d-0402-4526-9bda-17846390270c"
```
@ -56,11 +56,17 @@ metrics, and source commands.
Done when repeated observations use a stable machine-readable interface and
resource-control can distinguish headroom, reservations, and actual usage.
**Outcome (2026-08-22):** `make cluster-observe REMOTE=railiance01` now emits
versioned, timestamped JSON with node allocatable/observed resources,
namespace/workload requests and limits, PVC/PV/storage-class state, stable
units, source commands, and explicit gaps without reading Secrets or pod
environments.
## T02 — Publish the failure-domain and storage contract
```task
id: RCLUSTER-WP-0014-T02
status: todo
status: done
priority: high
state_hub_task_id: "6460d9aa-3ded-420a-bf01-f0ccdeee0a02"
```
@ -73,11 +79,16 @@ current capacity.
Done when consumers can calculate correlated-failure risk and recognize which
state remains local to `railiance01`.
**Outcome (2026-08-22):** the evidence contract reports one Ready node,
`local-path` storage, `threephoenix_ready: false`, and the missing independent
region/zone labels. Target HA and Longhorn capacity are not represented as
current facts.
## T03 — Define technical allocation-driver evidence
```task
id: RCLUSTER-WP-0014-T03
status: todo
status: done
priority: medium
state_hub_task_id: "e5ff9b6c-e883-4356-9eca-b31757661d2f"
```
@ -90,8 +101,22 @@ do not convert the result into booked spend.
Done when resource-control can create an explainable allocation and fin-hub can
reconcile it to referenced financial facts.
**Outcome (2026-08-22):** `cluster-raw-drivers-v1` publishes CPU requests,
memory requests, PVC bytes, observed usage, totals, residual capacity, and
uncertainty, while leaving formula selection and booked cost to
`resource-control`.
## Acceptance
- [ ] Capacity evidence separates allocatable, requested, limited, and observed.
- [ ] Current single-host/local-storage risk is machine-readable.
- [ ] Allocation drivers reconcile to explicit totals and residuals.
- [x] Capacity evidence separates allocatable, requested, limited, and observed.
- [x] Current single-host/local-storage risk is machine-readable.
- [x] Allocation drivers reconcile to explicit totals and residuals.
## Completion evidence
- `python3 -m unittest discover -s tests -p 'test_*.py' -v`: 2 tests passed.
- Live read-only capture:
`docs/evidence/resource-reef-railiance-k3s/observations/20260822T100349Z.json`.
- Observed totals: 1 node, 53 workloads, 25 claims; 4,000m allocatable CPU,
3,830m requested CPU, 16,770,076,672 allocatable memory bytes, and
137,438,953,472 requested PVC bytes.