state-hub/workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md

---
id: STATE-WP-0081
type: workplan
title: "Cluster self-sufficiency: remove workstation coupling and fix the registrar"
domain: infotech
repo: state-hub
status: proposed
owner: codex
topic_slug: infotech
created: "2026-08-21"
updated: "2026-08-21"
parent_project: prj-state-hub-retirement
parent_workplan: SHR-WP-0001
related:
  - STATE-WP-0079
  - RMGR-WP-0005
  - RMGR-WP-0008
  - ADR-007
  - ADR-010
state_hub_workstream_id: "bb2798fd-0680-5027-8479-3af3a5b048de"
---

# Cluster self-sufficiency: remove workstation coupling and fix the registrar

## Goal

Make the railiance01-hosted State Hub stand on its own: its own clones, its own
git identity, no hostPath into an operator's home directory, and no dependence
on workstation paths, processes, or checkouts. Fix the identifier registrar as
part of that, because the registrar is broken *by* this coupling rather than
beside it.

End state: **workstation coding agents push to forgejo; cluster infrastructure
reads from forgejo. Neither reads the other's disk.**

## Admissibility under the retirement freeze

`policies/retirement-freeze.md` allows changes that fix operational risk or
reduce scope. This is both: the sweep is currently broken in production, and the
work removes a coupling rather than adding capability. It establishes no new
permanent ownership here — the deployment chart already lives in this repo at
`deploy/railiance/apps/charts/state-hub/`.

## The coupling, as measured 2026-08-21

| # | Coupling | Evidence |
| --- | --- | --- |
| 1 | Pod mounts the operator home as a hostPath | `sweep.hostPath: /home/tegwick`, mounted rw at the same path |
| 2 | Pod runs as root | `securityContext: {}`, no `runAsUser`; produced **999 root-owned files** under `~/state-hub` |
| 3 | Pod uses the operator's personal SSH key as its service identity | `sweep.sshHostPath: /home/tegwick/.ssh` → `/root/.ssh` |
| 4 | Hub repo registry points at workstation paths | **72 of 75** records have `local_path: /home/worsch/...`, a path railiance01 will never have |
| 5 | Hub repo registry records the retired git host | `remote_url: gitea-remote:...` after the 2026-08-21 forgejo transition |
| 6 | Dashboard is a workstation process | Observable dev server on `127.0.0.1:3000`, not served from the cluster |

**And the fault that blocks everything:** inside the pod `/home/tegwick` is
mounted **read-only** — `ro,relatime,discard,errors=remount-ro` — although the
chart sets no `readOnly` on that volumeMount and `securityContext` is empty. The
host has the same device mounted `rw` with a healthy disk and no dmesg errors, so
this is imposed by the runtime or an admission path, not by hardware.

The sweep therefore dies with
`OSError: [Errno 30] Read-only file system: '/home/tegwick/info-tech-canon/.custodian-brief.md'`
before it can mint anything. That is the whole registrar problem: **the registrar
cannot write identifiers into files it cannot write.** It explains the 12 queued
sync requests and why the newest `custodian-sync` commits are from July.

## Restore the write path

```task
id: STATE-WP-0081-T01
status: todo
priority: high
state_hub_task_id: "7dca680b-d34d-5bfe-838e-b463a4487a9c"
```

Find why the hostPath mounts read-only against the chart's own spec, and restore
writes. Candidates in order: an admission controller or PodSecurity policy
forcing hostPath read-only; a k3s/containerd default for hostPath volumes; a
missing explicit `readOnly: false` on the volumeMount.

This is the minimum to make the registrar function, and the only task here that
unblocks anything today. Do it first even though T02 later deletes the mount —
a working baseline makes every later change verifiable.

Verification is end-to-end, not a green pod: run the sweep and confirm
`EBIND-WP-0002` gets a `state_hub_workstream_id` written back into its file.

## Give the pod its own clones

```task
id: STATE-WP-0081-T02
status: todo
priority: high
state_hub_task_id: "3fd4d23c-6b67-5b92-b312-25a7295773fd"
```

Replace the `sweep-repos` hostPath with storage the pod owns — a PVC the sweep
clones into and maintains, seeded from forgejo.

This is the change that actually severs coupling #1 and #2. Mounting a human's
home directory into a production workload is what created 999 root-owned files,
what made `git pull` fail on the host, and what put the operator's checkouts one
`reset --hard` away from a scheduled job.

Sizing input: the current tree is ~78 repos; `markitect_project` alone is 24 MB.

Keep the sweep's repo list driven by the hub's repo registry, not by whatever
happens to be on a disk.

## Replace the operator SSH key with a service identity

```task
id: STATE-WP-0081-T03
status: todo
priority: high
state_hub_task_id: "8e5a5eb8-f29c-5cab-a19d-3447b6e0c26a"
```

Issue a dedicated forgejo deploy key or service account for the sweep and mount
it from a Secret. Remove `sweep.sshHostPath` and the `/root/.ssh` mount.

The current arrangement gives a root-running production workload the operator's
personal private key. It is read-only, so this is a blast-radius problem rather
than a live compromise — but the key that can push to every repository in the
fleet should not be the same key a human uses interactively.

Credential custody routes through OpenBao, not this repo — see
`.claude/rules/credential-routing.md`. Do not put key material in the chart.

## Run as a non-root user

```task
id: STATE-WP-0081-T04
status: todo
priority: medium
state_hub_task_id: "57ab0ced-ef3f-5260-b7d6-e2a78ff4e023"
```

Set `runAsUser`/`runAsGroup` and a `fsGroup` matching the PVC. Depends on T02:
once the pod owns its storage there is no reason for it to be root.

Closes the recurrence: today's ownership fix will be undone by the next sweep
while the pod still runs as root.

## Correct the hub repo registry

```task
id: STATE-WP-0081-T05
status: todo
priority: high
state_hub_task_id: "530fbd27-e463-51d2-b09c-ba9827d057de"
```

72 of 75 repo records carry `local_path: /home/worsch/...` and stale
`remote_url: gitea-remote:...`. Both are wrong for a cluster that reads from
forgejo into its own clone tree.

Decide first whether `local_path` should be **per-instance rather than global** —
one column cannot describe a workstation checkout and a cluster clone at once,
and its current single value is precisely how the workstation leaked into
cluster configuration. `RMGR-WP-0008` is building the repository-representation
surface that inherits this; settle the shape with it rather than patching values
that will move.

`remote_url` correction is unambiguous and can proceed immediately.

## Serve the dashboard from the cluster

```task
id: STATE-WP-0081-T06
status: todo
priority: medium
state_hub_task_id: "1d74e5e2-fb0f-5342-afbb-6ce2986823f4"
```

The dashboard runs as an Observable dev server on the workstation at
`127.0.0.1:3000`. Serve it from the cluster behind the same ingress as the API so
it survives the workstation being off, and so what the operator sees is what the
cluster holds.

Check first whether this is worth building here at all: `hub-projection-ui` is
dispositioned `replace` → `hub-core` (14 items, slice B5 in
`docs/retirement-cutover-slice-plan.md`). If B5 lands first this task is a
redirect, not a build. Confirm with `HUB-WP-0004` before writing any chart.

## State and enforce the boundary

```task
id: STATE-WP-0081-T07
status: todo
priority: medium
state_hub_task_id: "12a2ce9f-7cdf-501b-9018-4c960d45e855"
```

Write the rule down so it stops being re-derived: **workstation coding agents
push to forgejo; cluster infrastructure reads from forgejo; neither reads the
other's disk.**

Record it where agents will meet it — `docs/`, the repo `AGENTS.md` templates, and
as an ADR if it constrains other repos, which it does.

Include the failure this prevents. On 2026-08-21 railiance01's 70 checkouts were
found still pointed at the retired gitea host, six weeks stale, and the
`evidence-binder` workplan the registrar was asked to index **did not exist** in
the copy the cluster could see. A shared-disk assumption made a stale reader look
like a queue.

## Close out the registrar

```task
id: STATE-WP-0081-T08
status: todo
priority: high
state_hub_task_id: "782c502a-5ce2-5ec3-8f07-b550c9705013"
```

With the write path restored, confirm the registrar end-to-end and drain the
backlog: 12 queued requests from `evidence-binder`, `kaizen-agentic`,
`glas-harness`, and `agentic-resources`, plus this fleet's own unregistered
`RMGR-WP-0008` and `RMGR-WP-0009`.

Then remove the interim: `RMGR-WP-0005-T03` (UUIDv5 derivation, keyed on
`(namespace, identifier)` and deriving for live records only per the 2026-08-21
`ADR-007` amendment) makes writeback idempotent and retires the single-writer
rule entirely. Coordinate rather than duplicate — the derivation belongs to
`repo-manager`.

Reply to the queued agents when it is done; several have been waiting since
2026-08-20.

## Acceptance

- [ ] Sweep writes successfully from the pod; `EBIND-WP-0002` registered
- [ ] Pod uses its own clone volume; no hostPath into any home directory
- [ ] Pod authenticates with a dedicated key, runs as non-root
- [ ] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
- [ ] Dashboard reachable without the workstation, or formally handed to `hub-core`
- [ ] Boundary rule written and discoverable by agents
- [ ] Registrar queue drained; interim single-writer rule retired or explicitly deferred