state-hub/workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md

---
id: STATE-WP-0081
type: workplan
title: "Cluster self-sufficiency: remove workstation coupling and fix the registrar"
domain: infotech
repo: state-hub
status: finished
owner: codex
topic_slug: infotech
created: "2026-08-21"
updated: "2026-09-04"
parent_project: prj-state-hub-retirement
parent_workplan: SHR-WP-0001
related:
  - STATE-WP-0079
  - RMGR-WP-0005
  - RMGR-WP-0008
  - STATE-WP-0083
  - STATE-WP-0084
  - STATE-WP-0086
  - 01a06e07-658e-73fa-9164-0b522504c0ee
  - 01a06e0c-3feb-7920-a6f2-961b03dc170d
  - ADR-007
  - ADR-010
state_hub_workstream_id: "bb2798fd-0680-5027-8479-3af3a5b048de"
---

# Cluster self-sufficiency: remove workstation coupling and fix the registrar

## Goal

Make the railiance01-hosted State Hub stand on its own: its own clones, its own
git identity, no hostPath into an operator's home directory, and no dependence
on workstation paths, processes, or checkouts. Fix the identifier registrar as
part of that, because the registrar is broken *by* this coupling rather than
beside it.

End state: **workstation coding agents push to forgejo; cluster infrastructure
reads from forgejo. Neither reads the other's disk.**

## Re-baseline — 2026-09-04

The original plan assumed the cluster sweep would regain write access to an
operator checkout and then migrate that checkout onto a PVC. That implementation
shape was superseded by the Forge-derived projection work in `STATE-WP-0083`,
`STATE-WP-0084`, and `STATE-WP-0086`: the central service now clones exact Forge
commits ephemerally and reconciles them without writing identifiers into a
shared checkout.

The live State Hub pod no longer mounts `/home/tegwick` or `/root/.ssh`. It uses
the `state-hub` service account and an OpenBao-projected Forge read credential.
The registrar guard has also been retired. Remaining work is limited to runtime
hardening, completing the repository-model audit, deciding the dashboard
disposition, and publishing the operating boundary.

## Admissibility under the retirement freeze

`policies/retirement-freeze.md` allows changes that fix operational risk or
reduce scope. This is both: the sweep is currently broken in production, and the
work removes a coupling rather than adding capability. It establishes no new
permanent ownership here — the deployment chart already lives in this repo at
`deploy/railiance/apps/charts/state-hub/`.

## The coupling, as measured 2026-08-21

| # | Coupling | Evidence |
| --- | --- | --- |
| 1 | Pod mounts the operator home as a hostPath | `sweep.hostPath: /home/tegwick`, mounted rw at the same path |
| 2 | Pod runs as root | `securityContext: {}`, no `runAsUser`; produced **999 root-owned files** under `~/state-hub` |
| 3 | Pod uses the operator's personal SSH key as its service identity | `sweep.sshHostPath: /home/tegwick/.ssh` → `/root/.ssh` |
| 4 | Hub repo registry points at workstation paths | **72 of 75** records have `local_path: /home/worsch/...`, a path railiance01 will never have |
| 5 | Hub repo registry records the retired git host | `remote_url: gitea-remote:...` after the 2026-08-21 forgejo transition |
| 6 | Dashboard is a workstation process | Observable dev server on `127.0.0.1:3000`, not served from the cluster |

**And the fault that blocks everything:** inside the pod `/home/tegwick` is
mounted **read-only** — `ro,relatime,discard,errors=remount-ro` — although the
chart sets no `readOnly` on that volumeMount and `securityContext` is empty. The
host has the same device mounted `rw` with a healthy disk and no dmesg errors, so
this is imposed by the runtime or an admission path, not by hardware.

The sweep therefore dies with
`OSError: [Errno 30] Read-only file system: '/home/tegwick/info-tech-canon/.custodian-brief.md'`
before it can mint anything. That is the whole registrar problem: **the registrar
cannot write identifiers into files it cannot write.** It explains the 12 queued
sync requests and why the newest `custodian-sync` commits are from July.

## Restore the write path

```task
id: STATE-WP-0081-T01
status: cancel
priority: high
state_hub_task_id: "7dca680b-d34d-5bfe-838e-b463a4487a9c"
```

Find why the hostPath mounts read-only against the chart's own spec, and restore
writes. Candidates in order: an admission controller or PodSecurity policy
forcing hostPath read-only; a k3s/containerd default for hostPath volumes; a
missing explicit `readOnly: false` on the volumeMount.

This is the minimum to make the registrar function, and the only task here that
unblocks anything today. Do it first even though T02 later deletes the mount —
a working baseline makes every later change verifiable.

Verification is end-to-end, not a green pod: run the sweep and confirm
`EBIND-WP-0002` gets a `state_hub_workstream_id` written back into its file.

**Superseded 2026-09-04.** The shared write path was deliberately abandoned.
Forge-derived reconciliation now assigns deterministic identifiers without a
cluster writer touching the operator checkout; `EBIND-WP-0002` is registered and
finished. See `STATE-WP-0083`, `STATE-WP-0084`, and `STATE-WP-0086`.

## Give the pod its own clones

```task
id: STATE-WP-0081-T02
status: cancel
priority: high
state_hub_task_id: "3fd4d23c-6b67-5b92-b312-25a7295773fd"
```

Replace the `sweep-repos` hostPath with storage the pod owns — a PVC the sweep
clones into and maintains, seeded from forgejo.

This is the change that actually severs coupling #1 and #2. Mounting a human's
home directory into a production workload is what created 999 root-owned files,
what made `git pull` fail on the host, and what put the operator's checkouts one
`reset --hard` away from a scheduled job.

Sizing input: the current tree is ~78 repos; `markitect_project` alone is 24 MB.

Keep the sweep's repo list driven by the hub's repo registry, not by whatever
happens to be on a disk.

**Superseded 2026-09-04.** The service uses short-lived Forge clones rather than
a long-lived PVC clone tree. The required boundary is achieved: the live pod has
no hostPath into an operator home directory.

## Replace the operator SSH key with a service identity

```task
id: STATE-WP-0081-T03
status: done
priority: high
state_hub_task_id: "8e5a5eb8-f29c-5cab-a19d-3447b6e0c26a"
```

Issue a dedicated forgejo deploy key or service account for the sweep and mount
it from a Secret. Remove `sweep.sshHostPath` and the `/root/.ssh` mount.

The current arrangement gives a root-running production workload the operator's
personal private key. It is read-only, so this is a blast-radius problem rather
than a live compromise — but the key that can push to every repository in the
fleet should not be the same key a human uses interactively.

Credential custody routes through OpenBao, not this repo — see
`.claude/rules/credential-routing.md`. Do not put key material in the chart.

**Completed 2026-09-04.** The live workload uses its Kubernetes service account
and an OpenBao-projected Forge read token. Neither the operator SSH directory nor
a personal private key is mounted into the pod.

## Run as a non-root user

```task
id: STATE-WP-0081-T04
status: done
priority: medium
state_hub_task_id: "57ab0ced-ef3f-5260-b7d6-e2a78ff4e023"
```

Set an explicit non-root `runAsUser`/`runAsGroup`, disallow privilege escalation,
drop unnecessary capabilities, and use a read-only root filesystem if the
application permits it. The PVC dependency no longer applies because T02 was
superseded by ephemeral Forge clones.

Closes the recurrence: today's ownership fix will be undone by the next sweep
while the pod still runs as root.

**Completed 2026-09-04.** Helm release revision 58 runs the API and MCP pods as
UID/GID 10001 with `runAsNonRoot`, `RuntimeDefault` seccomp, all capabilities
dropped, privilege escalation disabled, and a read-only root filesystem. A
writable `emptyDir` is mounted only at `/tmp`; the migration hook uses the same
security context. Both rollouts and `/state/health` passed with zero restarts.

## Correct the hub repo registry

```task
id: STATE-WP-0081-T05
status: done
priority: high
state_hub_task_id: "530fbd27-e463-51d2-b09c-ba9827d057de"
```

72 of 75 repo records carry `local_path: /home/worsch/...` and stale
`remote_url: gitea-remote:...`. Both are wrong for a cluster that reads from
forgejo into its own clone tree.

Decide first whether `local_path` should be **per-instance rather than global** —
one column cannot describe a workstation checkout and a cluster clone at once,
and its current single value is precisely how the workstation leaked into
cluster configuration. `RMGR-WP-0008` is building the repository-representation
surface that inherits this; settle the shape with it rather than patching values
that will move.

`remote_url` correction is unambiguous and can proceed immediately.

**Progress 2026-09-04.** Repository representation now carries per-host
`host_paths`, a compatibility `local_path`, and Forge remotes. Cluster
reconciliation consumes Forge commits rather than workstation paths. Complete a
fleet-wide audit before closing this task, including confirmation that no
cluster consumer still interprets `local_path` as an authoritative checkout.

**Completed 2026-09-04.** The primary database audit recorded 131 repositories,
126 active, no active repository without a remote, and no stale Gitea remote.
The one active missing value (`maturity-engine`) was repaired from its verified
Forgejo origin. Four remote-less records are archived history. Evidence:
`docs/evidence/repository-registry-audit-20260904.json`. The unbounded collection
endpoint performance finding is routed as intake
`01a06e0c-3feb-7920-a6f2-961b03dc170d`.

## Serve the dashboard from the cluster

```task
id: STATE-WP-0081-T06
status: done
priority: medium
state_hub_task_id: "1d74e5e2-fb0f-5342-afbb-6ce2986823f4"
```

The dashboard runs as an Observable dev server on the workstation at
`127.0.0.1:3000`. Serve it from the cluster behind the same ingress as the API so
it survives the workstation being off, and so what the operator sees is what the
cluster holds.

Check first whether this is worth building here at all: `hub-projection-ui` is
dispositioned `replace` → `hub-core` (14 items, slice B5 in
`docs/retirement-cutover-slice-plan.md`). If B5 lands first this task is a
redirect, not a build. Confirm with `HUB-WP-0004` before writing any chart.

**Completed by handoff 2026-09-04.** Building a new permanent State Hub
deployment surface would conflict with the retirement direction. The existing
dashboard remains an operator/development surface while its replacement is
routed to Hub Core as intake `01a06e07-658e-73fa-9164-0b522504c0ee`.

## State and enforce the boundary

```task
id: STATE-WP-0081-T07
status: done
priority: medium
state_hub_task_id: "12a2ce9f-7cdf-501b-9018-4c960d45e855"
```

Write the rule down so it stops being re-derived: **workstation coding agents
push to forgejo; cluster infrastructure reads from forgejo; neither reads the
other's disk.**

Record it where agents will meet it — `docs/`, the repo `AGENTS.md` templates, and
as an ADR if it constrains other repos, which it does.

Include the failure this prevents. On 2026-08-21 railiance01's 70 checkouts were
found still pointed at the retired gitea host, six weeks stale, and the
`evidence-binder` workplan the registrar was asked to index **did not exist** in
the copy the cluster could see. A shared-disk assumption made a stale reader look
like a queue.

**Completed 2026-09-04.** `STATE-ADR-002`, the cluster operating model, this
repository's agent instructions, and the generated Codex template now carry the
Forge-commit boundary and explain the failure it prevents.

## Close out the registrar

```task
id: STATE-WP-0081-T08
status: done
priority: high
state_hub_task_id: "782c502a-5ce2-5ec3-8f07-b550c9705013"
```

With the write path restored, confirm the registrar end-to-end and drain the
backlog: 12 queued requests from `evidence-binder`, `kaizen-agentic`,
`glas-harness`, and `agentic-resources`, plus this fleet's own unregistered
`RMGR-WP-0008` and `RMGR-WP-0009`.

Then remove the interim: `RMGR-WP-0005-T03` (UUIDv5 derivation, keyed on
`(namespace, identifier)` and deriving for live records only per the 2026-08-21
`ADR-007` amendment) makes writeback idempotent and retires the single-writer
rule entirely. Coordinate rather than duplicate — the derivation belongs to
`repo-manager`.

Reply to the queued agents when it is done; several have been waiting since
2026-08-20.

**Completed 2026-09-04.** `EBIND-WP-0002`, `RMGR-WP-0008`, and `RMGR-WP-0009`
are finished; exact-commit reconciliation is operational; and the interim
single-writer registrar guard was retired by the Forge-derived projection path.

## Acceptance

- [x] Shared-checkout writeback superseded; `EBIND-WP-0002` registered
- [x] Pod uses ephemeral Forge clones; no hostPath into any home directory
- [x] Pod authenticates with a dedicated service identity
- [x] Pod runs as non-root with an explicit restrictive security context
- [x] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
- [x] Dashboard reachable without the workstation, or formally handed to `hub-core`
- [x] Boundary rule written and discoverable by agents
- [x] Registrar queue drained; interim single-writer rule retired