state-hub/workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	parent_project	parent_workplan	related	state_hub_workstream_id
STATE-WP-0081	workplan	Cluster self-sufficiency: remove workstation coupling and fix the registrar	infotech	state-hub	finished	codex	infotech	2026-08-21	2026-09-04	prj-state-hub-retirement	SHR-WP-0001	STATE-WP-0079 RMGR-WP-0005 RMGR-WP-0008 STATE-WP-0083 STATE-WP-0084 STATE-WP-0086 01a06e07-658e-73fa-9164-0b522504c0ee 01a06e0c-3feb-7920-a6f2-961b03dc170d ADR-007 ADR-010	bb2798fd-0680-5027-8479-3af3a5b048de

Cluster self-sufficiency: remove workstation coupling and fix the registrar

Goal

Make the railiance01-hosted State Hub stand on its own: its own clones, its own git identity, no hostPath into an operator's home directory, and no dependence on workstation paths, processes, or checkouts. Fix the identifier registrar as part of that, because the registrar is broken by this coupling rather than beside it.

End state: workstation coding agents push to forgejo; cluster infrastructure reads from forgejo. Neither reads the other's disk.

Re-baseline — 2026-09-04

The original plan assumed the cluster sweep would regain write access to an operator checkout and then migrate that checkout onto a PVC. That implementation shape was superseded by the Forge-derived projection work in STATE-WP-0083, STATE-WP-0084, and STATE-WP-0086: the central service now clones exact Forge commits ephemerally and reconciles them without writing identifiers into a shared checkout.

The live State Hub pod no longer mounts /home/tegwick or /root/.ssh. It uses the state-hub service account and an OpenBao-projected Forge read credential. The registrar guard has also been retired. Remaining work is limited to runtime hardening, completing the repository-model audit, deciding the dashboard disposition, and publishing the operating boundary.

Admissibility under the retirement freeze

policies/retirement-freeze.md allows changes that fix operational risk or reduce scope. This is both: the sweep is currently broken in production, and the work removes a coupling rather than adding capability. It establishes no new permanent ownership here — the deployment chart already lives in this repo at deploy/railiance/apps/charts/state-hub/.

The coupling, as measured 2026-08-21

#	Coupling	Evidence
1	Pod mounts the operator home as a hostPath	sweep.hostPath: /home/tegwick, mounted rw at the same path
2	Pod runs as root	securityContext: {}, no runAsUser; produced 999 root-owned files under ~/state-hub
3	Pod uses the operator's personal SSH key as its service identity	sweep.sshHostPath: /home/tegwick/.ssh → /root/.ssh
4	Hub repo registry points at workstation paths	72 of 75 records have local_path: /home/worsch/..., a path railiance01 will never have
5	Hub repo registry records the retired git host	remote_url: gitea-remote:... after the 2026-08-21 forgejo transition
6	Dashboard is a workstation process	Observable dev server on 127.0.0.1:3000, not served from the cluster

And the fault that blocks everything: inside the pod /home/tegwick is mounted read-only — ro,relatime,discard,errors=remount-ro — although the chart sets no readOnly on that volumeMount and securityContext is empty. The host has the same device mounted rw with a healthy disk and no dmesg errors, so this is imposed by the runtime or an admission path, not by hardware.

The sweep therefore dies with OSError: [Errno 30] Read-only file system: '/home/tegwick/info-tech-canon/.custodian-brief.md' before it can mint anything. That is the whole registrar problem: the registrar cannot write identifiers into files it cannot write. It explains the 12 queued sync requests and why the newest custodian-sync commits are from July.

Restore the write path

id: STATE-WP-0081-T01
status: cancel
priority: high
state_hub_task_id: "7dca680b-d34d-5bfe-838e-b463a4487a9c"

Find why the hostPath mounts read-only against the chart's own spec, and restore writes. Candidates in order: an admission controller or PodSecurity policy forcing hostPath read-only; a k3s/containerd default for hostPath volumes; a missing explicit readOnly: false on the volumeMount.

This is the minimum to make the registrar function, and the only task here that unblocks anything today. Do it first even though T02 later deletes the mount — a working baseline makes every later change verifiable.

Verification is end-to-end, not a green pod: run the sweep and confirm EBIND-WP-0002 gets a state_hub_workstream_id written back into its file.

Superseded 2026-09-04. The shared write path was deliberately abandoned. Forge-derived reconciliation now assigns deterministic identifiers without a cluster writer touching the operator checkout; EBIND-WP-0002 is registered and finished. See STATE-WP-0083, STATE-WP-0084, and STATE-WP-0086.

Give the pod its own clones

id: STATE-WP-0081-T02
status: cancel
priority: high
state_hub_task_id: "3fd4d23c-6b67-5b92-b312-25a7295773fd"

Replace the sweep-repos hostPath with storage the pod owns — a PVC the sweep clones into and maintains, seeded from forgejo.

This is the change that actually severs coupling #1 and #2. Mounting a human's home directory into a production workload is what created 999 root-owned files, what made git pull fail on the host, and what put the operator's checkouts one reset --hard away from a scheduled job.

Sizing input: the current tree is ~78 repos; markitect_project alone is 24 MB.

Keep the sweep's repo list driven by the hub's repo registry, not by whatever happens to be on a disk.

Superseded 2026-09-04. The service uses short-lived Forge clones rather than a long-lived PVC clone tree. The required boundary is achieved: the live pod has no hostPath into an operator home directory.

Replace the operator SSH key with a service identity

id: STATE-WP-0081-T03
status: done
priority: high
state_hub_task_id: "8e5a5eb8-f29c-5cab-a19d-3447b6e0c26a"

Issue a dedicated forgejo deploy key or service account for the sweep and mount it from a Secret. Remove sweep.sshHostPath and the /root/.ssh mount.

The current arrangement gives a root-running production workload the operator's personal private key. It is read-only, so this is a blast-radius problem rather than a live compromise — but the key that can push to every repository in the fleet should not be the same key a human uses interactively.

Credential custody routes through OpenBao, not this repo — see .claude/rules/credential-routing.md. Do not put key material in the chart.

Completed 2026-09-04. The live workload uses its Kubernetes service account and an OpenBao-projected Forge read token. Neither the operator SSH directory nor a personal private key is mounted into the pod.

Run as a non-root user

id: STATE-WP-0081-T04
status: done
priority: medium
state_hub_task_id: "57ab0ced-ef3f-5260-b7d6-e2a78ff4e023"

Set an explicit non-root runAsUser/runAsGroup, disallow privilege escalation, drop unnecessary capabilities, and use a read-only root filesystem if the application permits it. The PVC dependency no longer applies because T02 was superseded by ephemeral Forge clones.

Closes the recurrence: today's ownership fix will be undone by the next sweep while the pod still runs as root.

Completed 2026-09-04. Helm release revision 58 runs the API and MCP pods as UID/GID 10001 with runAsNonRoot, RuntimeDefault seccomp, all capabilities dropped, privilege escalation disabled, and a read-only root filesystem. A writable emptyDir is mounted only at /tmp; the migration hook uses the same security context. Both rollouts and /state/health passed with zero restarts.

Correct the hub repo registry

id: STATE-WP-0081-T05
status: done
priority: high
state_hub_task_id: "530fbd27-e463-51d2-b09c-ba9827d057de"

72 of 75 repo records carry local_path: /home/worsch/... and stale remote_url: gitea-remote:.... Both are wrong for a cluster that reads from forgejo into its own clone tree.

Decide first whether local_path should be per-instance rather than global — one column cannot describe a workstation checkout and a cluster clone at once, and its current single value is precisely how the workstation leaked into cluster configuration. RMGR-WP-0008 is building the repository-representation surface that inherits this; settle the shape with it rather than patching values that will move.

remote_url correction is unambiguous and can proceed immediately.

Progress 2026-09-04. Repository representation now carries per-host host_paths, a compatibility local_path, and Forge remotes. Cluster reconciliation consumes Forge commits rather than workstation paths. Complete a fleet-wide audit before closing this task, including confirmation that no cluster consumer still interprets local_path as an authoritative checkout.

Completed 2026-09-04. The primary database audit recorded 131 repositories, 126 active, no active repository without a remote, and no stale Gitea remote. The one active missing value (maturity-engine) was repaired from its verified Forgejo origin. Four remote-less records are archived history. Evidence: docs/evidence/repository-registry-audit-20260904.json. The unbounded collection endpoint performance finding is routed as intake 01a06e0c-3feb-7920-a6f2-961b03dc170d.

Serve the dashboard from the cluster

id: STATE-WP-0081-T06
status: done
priority: medium
state_hub_task_id: "1d74e5e2-fb0f-5342-afbb-6ce2986823f4"

The dashboard runs as an Observable dev server on the workstation at 127.0.0.1:3000. Serve it from the cluster behind the same ingress as the API so it survives the workstation being off, and so what the operator sees is what the cluster holds.

Check first whether this is worth building here at all: hub-projection-ui is dispositioned replace → hub-core (14 items, slice B5 in docs/retirement-cutover-slice-plan.md). If B5 lands first this task is a redirect, not a build. Confirm with HUB-WP-0004 before writing any chart.

Completed by handoff 2026-09-04. Building a new permanent State Hub deployment surface would conflict with the retirement direction. The existing dashboard remains an operator/development surface while its replacement is routed to Hub Core as intake 01a06e07-658e-73fa-9164-0b522504c0ee.

State and enforce the boundary

id: STATE-WP-0081-T07
status: done
priority: medium
state_hub_task_id: "12a2ce9f-7cdf-501b-9018-4c960d45e855"

Write the rule down so it stops being re-derived: workstation coding agents push to forgejo; cluster infrastructure reads from forgejo; neither reads the other's disk.

Record it where agents will meet it — docs/, the repo AGENTS.md templates, and as an ADR if it constrains other repos, which it does.

Include the failure this prevents. On 2026-08-21 railiance01's 70 checkouts were found still pointed at the retired gitea host, six weeks stale, and the evidence-binder workplan the registrar was asked to index did not exist in the copy the cluster could see. A shared-disk assumption made a stale reader look like a queue.

Completed 2026-09-04. STATE-ADR-002, the cluster operating model, this repository's agent instructions, and the generated Codex template now carry the Forge-commit boundary and explain the failure it prevents.

Close out the registrar

id: STATE-WP-0081-T08
status: done
priority: high
state_hub_task_id: "782c502a-5ce2-5ec3-8f07-b550c9705013"

With the write path restored, confirm the registrar end-to-end and drain the backlog: 12 queued requests from evidence-binder, kaizen-agentic, glas-harness, and agentic-resources, plus this fleet's own unregistered RMGR-WP-0008 and RMGR-WP-0009.

Then remove the interim: RMGR-WP-0005-T03 (UUIDv5 derivation, keyed on (namespace, identifier) and deriving for live records only per the 2026-08-21 ADR-007 amendment) makes writeback idempotent and retires the single-writer rule entirely. Coordinate rather than duplicate — the derivation belongs to repo-manager.

Reply to the queued agents when it is done; several have been waiting since 2026-08-20.

Completed 2026-09-04. EBIND-WP-0002, RMGR-WP-0008, and RMGR-WP-0009 are finished; exact-commit reconciliation is operational; and the interim single-writer registrar guard was retired by the Forge-derived projection path.

Acceptance

• Shared-checkout writeback superseded; EBIND-WP-0002 registered
• Pod uses ephemeral Forge clones; no hostPath into any home directory
• Pod authenticates with a dedicated service identity
• Pod runs as non-root with an explicit restrictive security context
• No /home/worsch path in any cluster-consumed record; remote_url values current
• Dashboard reachable without the workstation, or formally handed to hub-core
• Boundary rule written and discoverable by agents
• Registrar queue drained; interim single-writer rule retired