feat: complete cluster self-sufficiency hardening
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Multi-Context Image / build-and-push (push) Successful in 45s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a06d83-1cbc-71f2-b0dc-e0f48cedae43
This commit is contained in:
tegwick 2026-09-04 22:23:42 +02:00
parent 87047faa8e
commit 0b44cf12f5
14 changed files with 232 additions and 40 deletions

View file

@ -1,24 +1,43 @@
# State Hub Cluster Operating Model
This document describes how State Hub runs after the pragmatic cluster migration
(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
known pragmatic limitations.
This document describes the primary State Hub runtime on railiance01 and the
repository boundary established by `STATE-ADR-002`. It is the operator runbook
for day-to-day use, rollback, and known pragmatic limitations.
## Runtime Summary
| Component | Location | Notes |
|-----------|----------|-------|
| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
| API workload | railiance01 k3s, namespace `state-hub` | Single-replica Deployment |
| MCP workload | railiance01 k3s, namespace `state-hub` | ClusterIP only; opt-in for clients |
| Database | railiance01 CNPG cluster `state-hub-db`, namespace `databases` | One primary instance |
| Image registry | `forgejo.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
| In-cluster access | `http://10.43.68.154:8000` | `state-hub` ClusterIP |
| Workstation access | `http://127.0.0.1:8000` | Host listener/tunnel to the primary |
| WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |
State Hub is **not** publicly exposed. Access stays on the private tunnel /
ops-bridge path.
Deployment handoff assets live under `deploy/railiance/` and were promoted to
the coulombcore cluster during cutover (2026-07-03).
Deployment handoff assets live under `deploy/railiance/`; the running release is
on railiance01.
## Repository Boundary
The invariant is: **workstation coding agents push to Forgejo; cluster
infrastructure reads exact Forge commits; neither reads the other's disk.**
- File-backed work records remain authoritative in their repository.
- `rmgr sync --push` pushes the local commit, verifies Forgejo, and requests
central reconciliation of that exact commit.
- Central reconciliation clones into ephemeral storage. It never uses a
workstation `local_path` or mounts an operator home directory.
- `host_paths` describes where a checkout is visible to a particular host; it
is not shared storage.
- Cluster Forge reads use the `state-hub` service account and an OpenBao-backed
workload credential. Personal SSH keys are not mounted.
See [`ADR-002`](adr/ADR-002-forge-repository-boundary.md).
## How Agents Reach State Hub
@ -39,13 +58,14 @@ make register-mcp
make mcp-http # SSE on :8001
```
### Remote machines (Railiance01, CoulombCore, Haskelseed, …)
### Remote and in-cluster callers
Bring up the managed tunnel mesh, then register MCP against the remote API port:
On railiance01, use the service address directly. Elsewhere, use the managed
edge relay or tunnel selected by the operator:
```bash
make bridges
make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
curl -fsS http://10.43.68.154:8000/state/health
statehub --api-base http://127.0.0.1:18080 outbox status
```
Restart the agent runtime after MCP registration.
@ -66,7 +86,7 @@ equivalent State Hub MCP helpers.
### Cluster database (CNPG)
The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
The `state-hub-db` cluster is managed by CloudNativePG on railiance01 k3s.
Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
current backup path until `CUST-WP-0038` or a disaster-control workplan adds
automated retention.
@ -162,7 +182,7 @@ Re-enablement is tracked outside this workplan (service-inventory gap).
Ops-run claims, leases, completion, and failure remain on activity-core in
Railiance. State Hub only exposes the cached read projection described in
[`ops-run-read-projection.md`](ops-run-read-projection.md). Configure the
coulombcore State Hub deployment with a private `ACTIVITY_CORE_URL` and an
railiance01 State Hub deployment with a private `ACTIVITY_CORE_URL` and an
OpenBao/ESO-injected `ACTIVITY_CORE_WORKER_TOKEN`.
If the Railiance edge or tunnel is unavailable, State Hub serves the last
@ -174,7 +194,7 @@ to State Hub as a failover mechanism.
This deployment is intentionally **not** highly available:
- One API replica on one k3s node.
- One API replica on the single railiance01 k3s node.
- One CNPG instance (no synchronous replica).
- No public ingress; tunnel dependency for all remote access.
- Cluster and tunnel outages require the WSL2 fallback or the offline write