feat: complete cluster self-sufficiency hardening
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a06d83-1cbc-71f2-b0dc-e0f48cedae43
This commit is contained in:
parent
87047faa8e
commit
0b44cf12f5
14 changed files with 232 additions and 40 deletions
59
docs/adr/ADR-002-forge-repository-boundary.md
Normal file
59
docs/adr/ADR-002-forge-repository-boundary.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
id: STATE-ADR-002
|
||||
type: architecture-decision-record
|
||||
title: "Use Forge commits, never another machine's checkout, as the cluster repository boundary"
|
||||
status: accepted
|
||||
owner: state-hub
|
||||
date: "2026-09-04"
|
||||
deciders:
|
||||
- operator
|
||||
- state-hub
|
||||
related:
|
||||
- STATE-WP-0081
|
||||
- STATE-WP-0083
|
||||
- STATE-WP-0084
|
||||
- STATE-WP-0086
|
||||
---
|
||||
|
||||
# Use Forge commits as the cluster repository boundary
|
||||
|
||||
## Context
|
||||
|
||||
The former consistency sweep mounted an operator home directory into the
|
||||
production State Hub pod. The cluster therefore saw stale checkouts, wrote
|
||||
root-owned files into human workspaces, inherited a personal SSH identity, and
|
||||
could not distinguish an absent work record from an out-of-date disk.
|
||||
|
||||
Repository files remain authoritative, but a local checkout is only one
|
||||
machine's observation of those files. It is not a cross-machine transport.
|
||||
|
||||
## Decision
|
||||
|
||||
Workstation agents commit and push repository authority to Forgejo. Cluster
|
||||
infrastructure reads a named, verified Forge commit into ephemeral storage.
|
||||
Neither side reads or writes the other side's checkout.
|
||||
|
||||
`local_path` and `host_paths` remain compatibility/observation fields for tools
|
||||
running on the named host. They must not select central reconciliation input.
|
||||
Central reconciliation records both the expected and derived commit and refuses
|
||||
the projection if they differ.
|
||||
|
||||
Cluster Forge access uses a workload service identity whose credential is
|
||||
obtained through Kubernetes authentication to OpenBao. Personal SSH directories
|
||||
and home-directory hostPaths are prohibited.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A workstation change is invisible centrally until it is committed and pushed.
|
||||
- A cluster result is attributable to an exact Forge commit.
|
||||
- Temporary clones and credentials may be discarded after reconciliation.
|
||||
- Offline writes use the explicit edge outbox; shared disks are not a fallback.
|
||||
- Scripts needing repository contents locally resolve the current host's
|
||||
`host_paths` entry and must fail clearly when no checkout is present.
|
||||
|
||||
## Verification
|
||||
|
||||
- The live State Hub pod has no operator-home or SSH hostPath.
|
||||
- `rmgr sync --push` verifies `primary/railliance01` and exact commit equality.
|
||||
- `docs/evidence/repository-registry-audit-20260904.json` records no active
|
||||
repository without a remote and no stale Gitea remote.
|
||||
|
|
@ -1,24 +1,43 @@
|
|||
# State Hub Cluster Operating Model
|
||||
|
||||
This document describes how State Hub runs after the pragmatic cluster migration
|
||||
(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
|
||||
known pragmatic limitations.
|
||||
This document describes the primary State Hub runtime on railiance01 and the
|
||||
repository boundary established by `STATE-ADR-002`. It is the operator runbook
|
||||
for day-to-day use, rollback, and known pragmatic limitations.
|
||||
|
||||
## Runtime Summary
|
||||
|
||||
| Component | Location | Notes |
|
||||
|-----------|----------|-------|
|
||||
| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
|
||||
| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
|
||||
| API workload | railiance01 k3s, namespace `state-hub` | Single-replica Deployment |
|
||||
| MCP workload | railiance01 k3s, namespace `state-hub` | ClusterIP only; opt-in for clients |
|
||||
| Database | railiance01 CNPG cluster `state-hub-db`, namespace `databases` | One primary instance |
|
||||
| Image registry | `forgejo.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
|
||||
| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
|
||||
| In-cluster access | `http://10.43.68.154:8000` | `state-hub` ClusterIP |
|
||||
| Workstation access | `http://127.0.0.1:8000` | Host listener/tunnel to the primary |
|
||||
| WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |
|
||||
|
||||
State Hub is **not** publicly exposed. Access stays on the private tunnel /
|
||||
ops-bridge path.
|
||||
|
||||
Deployment handoff assets live under `deploy/railiance/` and were promoted to
|
||||
the coulombcore cluster during cutover (2026-07-03).
|
||||
Deployment handoff assets live under `deploy/railiance/`; the running release is
|
||||
on railiance01.
|
||||
|
||||
## Repository Boundary
|
||||
|
||||
The invariant is: **workstation coding agents push to Forgejo; cluster
|
||||
infrastructure reads exact Forge commits; neither reads the other's disk.**
|
||||
|
||||
- File-backed work records remain authoritative in their repository.
|
||||
- `rmgr sync --push` pushes the local commit, verifies Forgejo, and requests
|
||||
central reconciliation of that exact commit.
|
||||
- Central reconciliation clones into ephemeral storage. It never uses a
|
||||
workstation `local_path` or mounts an operator home directory.
|
||||
- `host_paths` describes where a checkout is visible to a particular host; it
|
||||
is not shared storage.
|
||||
- Cluster Forge reads use the `state-hub` service account and an OpenBao-backed
|
||||
workload credential. Personal SSH keys are not mounted.
|
||||
|
||||
See [`ADR-002`](adr/ADR-002-forge-repository-boundary.md).
|
||||
|
||||
## How Agents Reach State Hub
|
||||
|
||||
|
|
@ -39,13 +58,14 @@ make register-mcp
|
|||
make mcp-http # SSE on :8001
|
||||
```
|
||||
|
||||
### Remote machines (Railiance01, CoulombCore, Haskelseed, …)
|
||||
### Remote and in-cluster callers
|
||||
|
||||
Bring up the managed tunnel mesh, then register MCP against the remote API port:
|
||||
On railiance01, use the service address directly. Elsewhere, use the managed
|
||||
edge relay or tunnel selected by the operator:
|
||||
|
||||
```bash
|
||||
make bridges
|
||||
make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
|
||||
curl -fsS http://10.43.68.154:8000/state/health
|
||||
statehub --api-base http://127.0.0.1:18080 outbox status
|
||||
```
|
||||
|
||||
Restart the agent runtime after MCP registration.
|
||||
|
|
@ -66,7 +86,7 @@ equivalent State Hub MCP helpers.
|
|||
|
||||
### Cluster database (CNPG)
|
||||
|
||||
The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
|
||||
The `state-hub-db` cluster is managed by CloudNativePG on railiance01 k3s.
|
||||
Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
|
||||
current backup path until `CUST-WP-0038` or a disaster-control workplan adds
|
||||
automated retention.
|
||||
|
|
@ -162,7 +182,7 @@ Re-enablement is tracked outside this workplan (service-inventory gap).
|
|||
Ops-run claims, leases, completion, and failure remain on activity-core in
|
||||
Railiance. State Hub only exposes the cached read projection described in
|
||||
[`ops-run-read-projection.md`](ops-run-read-projection.md). Configure the
|
||||
coulombcore State Hub deployment with a private `ACTIVITY_CORE_URL` and an
|
||||
railiance01 State Hub deployment with a private `ACTIVITY_CORE_URL` and an
|
||||
OpenBao/ESO-injected `ACTIVITY_CORE_WORKER_TOKEN`.
|
||||
|
||||
If the Railiance edge or tunnel is unavailable, State Hub serves the last
|
||||
|
|
@ -174,7 +194,7 @@ to State Hub as a failover mechanism.
|
|||
|
||||
This deployment is intentionally **not** highly available:
|
||||
|
||||
- One API replica on one k3s node.
|
||||
- One API replica on the single railiance01 k3s node.
|
||||
- One CNPG instance (no synchronous replica).
|
||||
- No public ingress; tunnel dependency for all remote access.
|
||||
- Cluster and tunnel outages require the WSL2 fallback or the offline write
|
||||
|
|
|
|||
36
docs/evidence/repository-registry-audit-20260904.json
Normal file
36
docs/evidence/repository-registry-audit-20260904.json
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
{
|
||||
"schema": "state-hub.repository-registry-audit.v1",
|
||||
"captured_at": "2026-09-04T19:50:00Z",
|
||||
"source": "primary/railliance01 managed_repos database projection",
|
||||
"workplan_task": "STATE-WP-0081-T05",
|
||||
"counts": {
|
||||
"repositories": 131,
|
||||
"active": 126,
|
||||
"active_missing_remote": 0,
|
||||
"stale_gitea_remote": 0,
|
||||
"workstation_local_path": 127
|
||||
},
|
||||
"repairs": [
|
||||
{
|
||||
"repo": "maturity-engine",
|
||||
"field": "remote_url",
|
||||
"value": "forgejo-remote:coulomb/maturity-engine.git",
|
||||
"evidence": "verified origin of /home/worsch/maturity-engine"
|
||||
}
|
||||
],
|
||||
"archived_missing_remote": [
|
||||
"markitect-project",
|
||||
"railiance-bootstrap",
|
||||
"railiance-hosts",
|
||||
"vergabe_teilnahme"
|
||||
],
|
||||
"interpretation": {
|
||||
"local_path": "compatibility projection for workstation tools; not consumed by central Forge reconciliation",
|
||||
"host_paths": "per-host checkout observations; not a cluster storage contract",
|
||||
"central_source": "exact Forge commit fetched ephemerally by primary reconciliation"
|
||||
},
|
||||
"observations": [
|
||||
"GET /repos/ did not return within 180 seconds; direct read-only database projection was used for this bounded audit",
|
||||
"Archived records without a remote are retained as history and are not active reconciliation inputs"
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue