state-hub/docs/cluster-operating-model.md

# State Hub Cluster Operating Model

This document describes the primary State Hub runtime on railiance01 and the
repository boundary established by `STATE-ADR-002`. It is the operator runbook
for day-to-day use, rollback, and known pragmatic limitations.

## Runtime Summary

| Component | Location | Notes |
|-----------|----------|-------|
| API workload | railiance01 k3s, namespace `state-hub` | Single-replica Deployment |
| MCP workload | railiance01 k3s, namespace `state-hub` | ClusterIP only; opt-in for clients |
| Database | railiance01 CNPG cluster `state-hub-db`, namespace `databases` | One primary instance |
| Image registry | `forgejo.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
| In-cluster access | `http://10.43.68.154:8000` | `state-hub` ClusterIP |
| Workstation access | `http://127.0.0.1:8000` | Host listener/tunnel to the primary |
| WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |

State Hub is **not** publicly exposed. Access stays on the private tunnel /
ops-bridge path.

Deployment handoff assets live under `deploy/railiance/`; the running release is
on railiance01.

## Repository Boundary

The invariant is: **workstation coding agents push to Forgejo; cluster
infrastructure reads exact Forge commits; neither reads the other's disk.**

- File-backed work records remain authoritative in their repository.
- `rmgr sync --push` pushes the local commit, verifies Forgejo, and requests
  central reconciliation of that exact commit.
- Central reconciliation clones into ephemeral storage. It never uses a
  workstation `local_path` or mounts an operator home directory.
- `host_paths` describes where a checkout is visible to a particular host; it
  is not shared storage.
- Cluster Forge reads use the `state-hub` service account and an OpenBao-backed
  workload credential. Personal SSH keys are not mounted.

See [`ADR-002`](adr/ADR-002-forge-repository-boundary.md).

## How Agents Reach State Hub

### Primary operator workstation (WSL2)

The cluster API is the production writer. Port `8000` on the workstation is
forwarded to the cluster service through ops-bridge:

```bash
bridge status    # state-hub-primary should be connected
curl -fsS http://127.0.0.1:8000/state/health
```

Local MCP registration (default):

```bash
make register-mcp
make mcp-http    # SSE on :8001
```

### Remote and in-cluster callers

On railiance01, use the service address directly. Elsewhere, use the managed
edge relay or tunnel selected by the operator:

```bash
curl -fsS http://10.43.68.154:8000/state/health
statehub --api-base http://127.0.0.1:18080 outbox status
```

Restart the agent runtime after MCP registration.

Onboarding details: [`docs/onboarding.md`](onboarding.md).

### Claude Code / Codex session start

```bash
cat .custodian-brief.md
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
```

When MCP tools are available, prefer `get_domain_summary("infotech")` or
equivalent State Hub MCP helpers.

## Backups and Restores

### Cluster database (CNPG)

The `state-hub-db` cluster is managed by CloudNativePG on railiance01 k3s.
Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
current backup path until `CUST-WP-0038` or a disaster-control workplan adds
automated retention.

Manual cluster dump (operator):

```bash
# Port-forward the rw service, then pg_dump from an operator shell
KUBECONFIG=~/.kube/config kubectl port-forward -n databases svc/state-hub-db-rw 15432:5432
pg_dump -h 127.0.0.1 -p 15432 -U state_hub -Fc state_hub > state-hub-$(date +%Y%m%d).dump
```

Restore into an isolated test database before any production restore attempt.
The T01 drill (2026-05-02) proved the WSL2 dump/restore path; repeat that
discipline before any live restore.

### WSL2 fallback database

The legacy Docker Postgres (`infra-postgres-1`) remains available for rollback.
It is **not** receiving normal writes after cutover.

To take a WSL2 snapshot while fallback is stopped:

```bash
docker exec infra-postgres-1 pg_dump -U custodian -Fc custodian > wsl2-state-hub.dump
```

## Roll Back to WSL2

Use this when the cluster deployment is unhealthy and operators need the last
known-good local writer.

1. Stop forwarding the primary tunnel:

   ```bash
   bridge down state-hub-primary
   ```

2. Start the local stack:

   ```bash
   cd ~/state-hub
   make api
   ```

3. Verify local health:

   ```bash
   curl -fsS http://127.0.0.1:8000/state/health
   ```

4. Re-register MCP if needed (`make register-mcp` without tunnel overrides).

5. Record a progress event documenting the rollback and the triggering incident.

Returning to cluster-primary:

```bash
bridge up state-hub-primary
# stop local uvicorn if it would conflict on :8000
fuser -k 8000/tcp 2>/dev/null || true
curl -fsS http://127.0.0.1:8000/state/health
```

Cutover sequence reference: `CUST-WP-0011-T07` (2026-07-03).

## Consistency Sync

File-backed workplans remain authoritative (ADR-001). After commits:

```bash
uv run --project ~/repo-manager rmgr sync --path /path/to/repo --push
```

The command verifies `primary/railliance01`, then central derives and applies
the exact pushed Forgejo commit. `statehub fix-consistency` remains the broad
deep-audit tool; it is not the interactive registration/update path.

```bash
curl -s -X POST http://127.0.0.1:8000/consistency/sweep/remote-all \
  -H "Content-Type: application/json" \
  -d '{"max_seconds": 300}' | python3 -m json.tool
```

Runbook: [`docs/consistency-sweep-runbook.md`](consistency-sweep-runbook.md).

**Known gap:** scheduled activity-core sweeps paused after the 2026-07-03
cutover while the bridge target chain was rewired. Manual sweeps succeed.
Re-enablement is tracked outside this workplan (service-inventory gap).

## Activity-core ops run visibility

Ops-run claims, leases, completion, and failure remain on activity-core in
Railiance. State Hub only exposes the cached read projection described in
[`ops-run-read-projection.md`](ops-run-read-projection.md). Configure the
railiance01 State Hub deployment with a private `ACTIVITY_CORE_URL` and an
OpenBao/ESO-injected `ACTIVITY_CORE_WORKER_TOKEN`.

If the Railiance edge or tunnel is unavailable, State Hub serves the last
successful projection with `stale: true`; before any successful pull it returns
`available: false`. Alert on those flags. Do not add claim or completion writes
to State Hub as a failover mechanism.

## Pragmatic Limitations (Single-Node)

This deployment is intentionally **not** highly available:

- One API replica on the single railiance01 k3s node.
- One CNPG instance (no synchronous replica).
- No public ingress; tunnel dependency for all remote access.
- Cluster and tunnel outages require the WSL2 fallback or the offline write
  buffer (`docs/offline-write-buffer.md`).

Long-term HA, replicated storage, tested failover, and WSL2 retirement belong
to **`CUST-WP-0038`**.

## WSL2 Retirement

Do **not** retire the WSL2 State Hub instance in normal operations. It remains
the disaster-recovery fallback until `CUST-WP-0038` (or a separate human
decision) explicitly approves retirement.

## Operator Checklist

Daily or after infra changes:

```bash
bridge check
curl -fsS http://127.0.0.1:8000/state/health
KUBECONFIG=~/.kube/config kubectl get pods -n state-hub
KUBECONFIG=~/.kube/config kubectl get cluster -n databases state-hub-db
```

After image or chart changes, see `deploy/railiance/README.md` and
`docs/container-image.md`.

## References

- `workplans/CUST-WP-0011-state-hub-threephoenix-migration.md` — migration plan
- `workplans/CUST-WP-0038-state-hub-threephoenix-ha.md` — future HA target
- `deploy/railiance/README.md` — Helm/CNPG handoff
- `the-custodian/ops/service-inventory.yml` — live endpoint inventory