feat: complete cluster self-sufficiency hardening

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a06d83-1cbc-71f2-b0dc-e0f48cedae43

AGENTS.md

@@ -261,3 +261,10 @@ To create a new workplan:
 1. Write the file following the format above
 2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`.
 3. Use `statehub fix-consistency` only for a separate deep consistency audit.
+
+## Repository / cluster boundary
+
+Workstation agents commit and push to Forgejo. Cluster infrastructure reads a
+verified Forge commit into ephemeral storage. Neither side reads or writes the
+other side's checkout. `local_path` and `host_paths` are host observations, not
+central storage configuration. See `docs/adr/ADR-002-forge-repository-boundary.md`.

Makefile

@@ -9,7 +9,7 @@ RAILIANCE_STATE_HUB_RELEASE ?= state-hub
 RAILIANCE_STATE_HUB_NAMESPACE ?= state-hub
 RAILIANCE_STATE_HUB_CHART ?= deploy/railiance/apps/charts/state-hub
 RAILIANCE_STATE_HUB_VALUES ?= deploy/railiance/apps/helm/state-hub-values.yaml
-RAILIANCE_STATE_HUB_IMAGE_TAG ?= f2e042a
+RAILIANCE_STATE_HUB_IMAGE_TAG ?= main-cdff3b7
 RAILIANCE_STATE_HUB_PLATFORM_DIR ?= deploy/railiance/platform
 RAILIANCE_STATE_HUB_APP_MANIFESTS ?= deploy/railiance/apps/manifests
 # Codex/WSL non-login shells may not source ~/.profile; keep uv discoverable.

SCOPE.md

@@ -2,9 +2,10 @@
 
 ## One-Liner
 
-State Hub is the local-first coordination service for repo-anchored workplans,
+State Hub is the transitional coordination service for repo-anchored workplans,
 tasks, decisions, progress events, repo classification and metadata, MCP
-tooling, and dashboard telemetry.
+tooling, and dashboard telemetry. Its primary runtime is on railiance01; local
+repository files remain authoritative for repository-owned work records.
 
 ## In Scope
 
@@ -23,13 +24,16 @@ tooling, and dashboard telemetry.
 
 | Repo | Role |
 | --- | --- |
-| `hub-core` | Shared library — State Hub mounts router factories and MCP from `hub_core` |
+| `hub-core` | Shared library and selected target runtime for capabilities retained after State Hub retirement |
-| `state-hub` | This repo — dev coordination host, workplan file authority |
+| `state-hub` | This repo — transitional coordination runtime and read projection; repository files retain work-record authority |
-| `core-hub` | Production framework — `/api/v2` at `hub.coulomb.social`; does not replace State Hub dev MCP |
+| `core-hub` | Production `/api/v2` compatibility runtime on railiance01 during consolidation into `hub-core` |
 
 Canon: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md`
 
 State Hub does **not** implement Inter-Hub `/api/v2` routes; those belong to core-hub.
+New permanent cross-domain capability should land in its disposition owner, not
+expand State Hub. The retirement program is governed by
+`prj-state-hub-retirement`.
 
 ## Out Of Scope
 

deploy/railiance/README.md

@@ -21,7 +21,7 @@ boundaries used for the actual cluster rollout.
 The current image is pinned to:
 
 ```text
-forgejo.coulomb.social/coulomb/state-hub:f2e042a
+forgejo.coulomb.social/coulomb/state-hub:main-cdff3b7
 ```
 
 railiance01 has already pulled this tag with `crictl`, and the image serves
@@ -87,4 +87,4 @@ DATABASE_URL=postgresql+asyncpg://state_hub:<url-encoded-password>@state-hub-db-
 Optional runtime settings such as `CORS_ORIGINS` can live in the chart
 ConfigMap. The default chart keeps public ingress disabled; access should use
 the existing private tunnel/ops-bridge path until a separate exposure decision
-is recorded.
+is recorded.

deploy/railiance/apps/charts/state-hub/templates/deployment.yaml

@@ -25,8 +25,9 @@ spec:
       {{- with .Values.imagePullSecrets }}
       imagePullSecrets: {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if or .Values.sweep.enabled .Values.classificationAllowed.enabled .Values.forgeRead.enabled }}
       volumes:
+        - name: runtime-tmp
+          emptyDir: {}
         {{- if .Values.sweep.enabled }}
         - name: sweep-repos
           hostPath:
@@ -57,7 +58,6 @@ spec:
                   {{- end }}
                   expirationSeconds: {{ .Values.forgeRead.openbao.expirationSeconds }}
         {{- end }}
-      {{- end }}
       containers:
         - name: state-hub
           image: {{ include "statehub.image" . | quote }}
@@ -76,8 +76,9 @@ spec:
                   - -c
                   - git config --global --add safe.directory '*'
           {{- end }}
-          {{- if or .Values.sweep.enabled .Values.classificationAllowed.enabled .Values.forgeRead.enabled }}
           volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
             {{- if .Values.sweep.enabled }}
             - name: sweep-repos
               mountPath: {{ .Values.sweep.hostPath | quote }}
@@ -96,6 +97,8 @@ spec:
               readOnly: true
             {{- end }}
           env:
+            - name: HOME
+              value: /tmp
             {{- if .Values.sweep.enabled }}
             - name: STATE_HUB_SWEEP_HOSTNAME
               value: {{ .Values.sweep.hostname | quote }}
@@ -122,7 +125,6 @@ spec:
             - name: FORGE_READ_SECRET_KEY
               value: {{ .Values.forgeRead.openbao.secretKey | quote }}
             {{- end }}
-          {{- end }}
           envFrom:
             {{- if .Values.config.enabled }}
             - configMapRef:

deploy/railiance/apps/charts/state-hub/templates/mcp-deployment.yaml

@@ -21,6 +21,9 @@ spec:
       {{- with .Values.imagePullSecrets }}
       imagePullSecrets: {{- toYaml . | nindent 8 }}
       {{- end }}
+      volumes:
+        - name: runtime-tmp
+          emptyDir: {}
       containers:
         - name: state-hub-mcp
           image: {{ include "statehub.image" . | quote }}
@@ -30,11 +33,16 @@ spec:
           # Use -m so the WORKDIR (/app) lands on sys.path — invoking the file
           # directly would put /app/mcp_server there and break its own imports.
           command: ["/app/.venv/bin/python", "-m", "mcp_server.server"]
+          volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
           ports:
             - name: sse
               containerPort: {{ .Values.mcp.service.targetPort }}
               protocol: TCP
           env:
+            - name: HOME
+              value: /tmp
             - name: MCP_TRANSPORT
               value: {{ .Values.mcp.transport | quote }}
             - name: MCP_PORT

deploy/railiance/apps/charts/state-hub/templates/migrate-job.yaml

@@ -21,14 +21,25 @@ spec:
       labels: {{- include "statehub.labels" . | nindent 8 }}
     spec:
       restartPolicy: Never
+      securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }}
       {{- with .Values.imagePullSecrets }}
       imagePullSecrets: {{- toYaml . | nindent 8 }}
       {{- end }}
+      volumes:
+        - name: runtime-tmp
+          emptyDir: {}
       containers:
         - name: migrate
           image: {{ include "statehub.image" . | quote }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
+          securityContext: {{- toYaml .Values.securityContext | nindent 12 }}
           command: ["/app/.venv/bin/python", "-m", "alembic", "upgrade", "head"]
+          volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
+          env:
+            - name: HOME
+              value: /tmp
           envFrom:
             {{- if .Values.config.enabled }}
             - configMapRef:

deploy/railiance/apps/charts/state-hub/values.yaml

@@ -162,14 +162,29 @@ probes:
     timeoutSeconds: 5
     failureThreshold: 3
 
-podSecurityContext: {}
+# STATE-WP-0081-T04. The image and all writable runtime state are independent
-securityContext: {}
+# of root. Keep these defaults restrictive so API, MCP, and migration pods have
+# the same security posture in every installation.
+podSecurityContext:
+  runAsNonRoot: true
+  runAsUser: 10001
+  runAsGroup: 10001
+  fsGroup: 10001
+  seccompProfile:
+    type: RuntimeDefault
+securityContext:
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop:
+      - ALL
+  readOnlyRootFilesystem: true
 
 nodeSelector: {}
 tolerations: []
 affinity: {}
 
-# Consistency sweep: mount railiance01 clone tree and match host_paths hostname.
+# Legacy consistency sweep settings. Kept only for rollback compatibility;
+# production uses Forge-derived reconciliation and must leave this disabled.
 sweep:
   enabled: false
   hostname: ""

deploy/railiance/apps/helm/state-hub-values.yaml

@@ -6,7 +6,7 @@ namespace:
 
 image:
   repository: forgejo.coulomb.social/coulomb/state-hub
-  tag: "main-5dd04dc"
+  tag: "main-cdff3b7"
 
 ingress:
   enabled: false

docs/adr/ADR-002-forge-repository-boundary.md

@@ -0,0 +1,59 @@
+---
+id: STATE-ADR-002
+type: architecture-decision-record
+title: "Use Forge commits, never another machine's checkout, as the cluster repository boundary"
+status: accepted
+owner: state-hub
+date: "2026-09-04"
+deciders:
+  - operator
+  - state-hub
+related:
+  - STATE-WP-0081
+  - STATE-WP-0083
+  - STATE-WP-0084
+  - STATE-WP-0086
+---
+
+# Use Forge commits as the cluster repository boundary
+
+## Context
+
+The former consistency sweep mounted an operator home directory into the
+production State Hub pod. The cluster therefore saw stale checkouts, wrote
+root-owned files into human workspaces, inherited a personal SSH identity, and
+could not distinguish an absent work record from an out-of-date disk.
+
+Repository files remain authoritative, but a local checkout is only one
+machine's observation of those files. It is not a cross-machine transport.
+
+## Decision
+
+Workstation agents commit and push repository authority to Forgejo. Cluster
+infrastructure reads a named, verified Forge commit into ephemeral storage.
+Neither side reads or writes the other side's checkout.
+
+`local_path` and `host_paths` remain compatibility/observation fields for tools
+running on the named host. They must not select central reconciliation input.
+Central reconciliation records both the expected and derived commit and refuses
+the projection if they differ.
+
+Cluster Forge access uses a workload service identity whose credential is
+obtained through Kubernetes authentication to OpenBao. Personal SSH directories
+and home-directory hostPaths are prohibited.
+
+## Consequences
+
+- A workstation change is invisible centrally until it is committed and pushed.
+- A cluster result is attributable to an exact Forge commit.
+- Temporary clones and credentials may be discarded after reconciliation.
+- Offline writes use the explicit edge outbox; shared disks are not a fallback.
+- Scripts needing repository contents locally resolve the current host's
+  `host_paths` entry and must fail clearly when no checkout is present.
+
+## Verification
+
+- The live State Hub pod has no operator-home or SSH hostPath.
+- `rmgr sync --push` verifies `primary/railliance01` and exact commit equality.
+- `docs/evidence/repository-registry-audit-20260904.json` records no active
+  repository without a remote and no stale Gitea remote.

docs/cluster-operating-model.md

@@ -1,24 +1,43 @@
 # State Hub Cluster Operating Model
 
-This document describes how State Hub runs after the pragmatic cluster migration
+This document describes the primary State Hub runtime on railiance01 and the
-(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
+repository boundary established by `STATE-ADR-002`. It is the operator runbook
-known pragmatic limitations.
+for day-to-day use, rollback, and known pragmatic limitations.
 
 ## Runtime Summary
 
 | Component | Location | Notes |
 |-----------|----------|-------|
-| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
+| API workload | railiance01 k3s, namespace `state-hub` | Single-replica Deployment |
-| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
+| MCP workload | railiance01 k3s, namespace `state-hub` | ClusterIP only; opt-in for clients |
+| Database | railiance01 CNPG cluster `state-hub-db`, namespace `databases` | One primary instance |
 | Image registry | `forgejo.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
-| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
+| In-cluster access | `http://10.43.68.154:8000` | `state-hub` ClusterIP |
+| Workstation access | `http://127.0.0.1:8000` | Host listener/tunnel to the primary |
 | WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |
 
 State Hub is **not** publicly exposed. Access stays on the private tunnel /
 ops-bridge path.
 
-Deployment handoff assets live under `deploy/railiance/` and were promoted to
+Deployment handoff assets live under `deploy/railiance/`; the running release is
-the coulombcore cluster during cutover (2026-07-03).
+on railiance01.
+
+## Repository Boundary
+
+The invariant is: **workstation coding agents push to Forgejo; cluster
+infrastructure reads exact Forge commits; neither reads the other's disk.**
+
+- File-backed work records remain authoritative in their repository.
+- `rmgr sync --push` pushes the local commit, verifies Forgejo, and requests
+  central reconciliation of that exact commit.
+- Central reconciliation clones into ephemeral storage. It never uses a
+  workstation `local_path` or mounts an operator home directory.
+- `host_paths` describes where a checkout is visible to a particular host; it
+  is not shared storage.
+- Cluster Forge reads use the `state-hub` service account and an OpenBao-backed
+  workload credential. Personal SSH keys are not mounted.
+
+See [`ADR-002`](adr/ADR-002-forge-repository-boundary.md).
 
 ## How Agents Reach State Hub
 
@@ -39,13 +58,14 @@ make register-mcp
 make mcp-http    # SSE on :8001
 ```
 
-### Remote machines (Railiance01, CoulombCore, Haskelseed, …)
+### Remote and in-cluster callers
 
-Bring up the managed tunnel mesh, then register MCP against the remote API port:
+On railiance01, use the service address directly. Elsewhere, use the managed
+edge relay or tunnel selected by the operator:
 
 ```bash
-make bridges
+curl -fsS http://10.43.68.154:8000/state/health
-make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
+statehub --api-base http://127.0.0.1:18080 outbox status
 ```
 
 Restart the agent runtime after MCP registration.
@@ -66,7 +86,7 @@ equivalent State Hub MCP helpers.
 
 ### Cluster database (CNPG)
 
-The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
+The `state-hub-db` cluster is managed by CloudNativePG on railiance01 k3s.
 Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
 current backup path until `CUST-WP-0038` or a disaster-control workplan adds
 automated retention.
@@ -162,7 +182,7 @@ Re-enablement is tracked outside this workplan (service-inventory gap).
 Ops-run claims, leases, completion, and failure remain on activity-core in
 Railiance. State Hub only exposes the cached read projection described in
 [`ops-run-read-projection.md`](ops-run-read-projection.md). Configure the
-coulombcore State Hub deployment with a private `ACTIVITY_CORE_URL` and an
+railiance01 State Hub deployment with a private `ACTIVITY_CORE_URL` and an
 OpenBao/ESO-injected `ACTIVITY_CORE_WORKER_TOKEN`.
 
 If the Railiance edge or tunnel is unavailable, State Hub serves the last
@@ -174,7 +194,7 @@ to State Hub as a failover mechanism.
 
 This deployment is intentionally **not** highly available:
 
-- One API replica on one k3s node.
+- One API replica on the single railiance01 k3s node.
 - One CNPG instance (no synchronous replica).
 - No public ingress; tunnel dependency for all remote access.
 - Cluster and tunnel outages require the WSL2 fallback or the offline write

docs/evidence/repository-registry-audit-20260904.json

@@ -0,0 +1,36 @@
+{
+  "schema": "state-hub.repository-registry-audit.v1",
+  "captured_at": "2026-09-04T19:50:00Z",
+  "source": "primary/railliance01 managed_repos database projection",
+  "workplan_task": "STATE-WP-0081-T05",
+  "counts": {
+    "repositories": 131,
+    "active": 126,
+    "active_missing_remote": 0,
+    "stale_gitea_remote": 0,
+    "workstation_local_path": 127
+  },
+  "repairs": [
+    {
+      "repo": "maturity-engine",
+      "field": "remote_url",
+      "value": "forgejo-remote:coulomb/maturity-engine.git",
+      "evidence": "verified origin of /home/worsch/maturity-engine"
+    }
+  ],
+  "archived_missing_remote": [
+    "markitect-project",
+    "railiance-bootstrap",
+    "railiance-hosts",
+    "vergabe_teilnahme"
+  ],
+  "interpretation": {
+    "local_path": "compatibility projection for workstation tools; not consumed by central Forge reconciliation",
+    "host_paths": "per-host checkout observations; not a cluster storage contract",
+    "central_source": "exact Forge commit fetched ephemerally by primary reconciliation"
+  },
+  "observations": [
+    "GET /repos/ did not return within 180 seconds; direct read-only database projection was used for this bounded audit",
+    "Archived records without a remote are retained as history and are not active reconciliation inputs"
+  ]
+}

scripts/project_rules/agents-codex.template

@@ -37,6 +37,11 @@ before declaring State Hub unavailable; a managed Codex permission profile may
 still enforce isolated networking. Experimental MCP can be enabled explicitly
 with `make -C ~/state-hub configure-codex WITH_MCP=1`.
 
+Repository files cross machine boundaries through committed Forge revisions:
+workstation agents push to Forgejo and cluster infrastructure reads a verified
+commit into ephemeral storage. Neither side reads or writes the other's
+checkout. `local_path` and `host_paths` describe host-local observations only.
+
 ### Orient at session start
 
 ```bash

workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md

@@ -4,7 +4,7 @@ type: workplan
 title: "Cluster self-sufficiency: remove workstation coupling and fix the registrar"
 domain: infotech
 repo: state-hub
-status: active
+status: finished
 owner: codex
 topic_slug: infotech
 created: "2026-08-21"
@@ -18,6 +18,8 @@ related:
   - STATE-WP-0083
   - STATE-WP-0084
   - STATE-WP-0086
+  - 01a06e07-658e-73fa-9164-0b522504c0ee
+  - 01a06e0c-3feb-7920-a6f2-961b03dc170d
   - ADR-007
   - ADR-010
 state_hub_workstream_id: "bb2798fd-0680-5027-8479-3af3a5b048de"
@@ -162,7 +164,7 @@ a personal private key is mounted into the pod.
 
 ```task
 id: STATE-WP-0081-T04
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "57ab0ced-ef3f-5260-b7d6-e2a78ff4e023"
 ```
@@ -175,11 +177,17 @@ superseded by ephemeral Forge clones.
 Closes the recurrence: today's ownership fix will be undone by the next sweep
 while the pod still runs as root.
 
+**Completed 2026-09-04.** Helm release revision 58 runs the API and MCP pods as
+UID/GID 10001 with `runAsNonRoot`, `RuntimeDefault` seccomp, all capabilities
+dropped, privilege escalation disabled, and a read-only root filesystem. A
+writable `emptyDir` is mounted only at `/tmp`; the migration hook uses the same
+security context. Both rollouts and `/state/health` passed with zero restarts.
+
 ## Correct the hub repo registry
 
 ```task
 id: STATE-WP-0081-T05
-status: progress
+status: done
 priority: high
 state_hub_task_id: "530fbd27-e463-51d2-b09c-ba9827d057de"
 ```
@@ -203,11 +211,19 @@ reconciliation consumes Forge commits rather than workstation paths. Complete a
 fleet-wide audit before closing this task, including confirmation that no
 cluster consumer still interprets `local_path` as an authoritative checkout.
 
+**Completed 2026-09-04.** The primary database audit recorded 131 repositories,
+126 active, no active repository without a remote, and no stale Gitea remote.
+The one active missing value (`maturity-engine`) was repaired from its verified
+Forgejo origin. Four remote-less records are archived history. Evidence:
+`docs/evidence/repository-registry-audit-20260904.json`. The unbounded collection
+endpoint performance finding is routed as intake
+`01a06e0c-3feb-7920-a6f2-961b03dc170d`.
+
 ## Serve the dashboard from the cluster
 
 ```task
 id: STATE-WP-0081-T06
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "1d74e5e2-fb0f-5342-afbb-6ce2986823f4"
 ```
@@ -222,11 +238,16 @@ dispositioned `replace` → `hub-core` (14 items, slice B5 in
 `docs/retirement-cutover-slice-plan.md`). If B5 lands first this task is a
 redirect, not a build. Confirm with `HUB-WP-0004` before writing any chart.
 
+**Completed by handoff 2026-09-04.** Building a new permanent State Hub
+deployment surface would conflict with the retirement direction. The existing
+dashboard remains an operator/development surface while its replacement is
+routed to Hub Core as intake `01a06e07-658e-73fa-9164-0b522504c0ee`.
+
 ## State and enforce the boundary
 
 ```task
 id: STATE-WP-0081-T07
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "12a2ce9f-7cdf-501b-9018-4c960d45e855"
 ```
@@ -244,6 +265,10 @@ found still pointed at the retired gitea host, six weeks stale, and the
 the copy the cluster could see. A shared-disk assumption made a stale reader look
 like a queue.
 
+**Completed 2026-09-04.** `STATE-ADR-002`, the cluster operating model, this
+repository's agent instructions, and the generated Codex template now carry the
+Forge-commit boundary and explain the failure it prevents.
+
 ## Close out the registrar
 
 ```task
@@ -276,8 +301,8 @@ single-writer registrar guard was retired by the Forge-derived projection path.
 - [x] Shared-checkout writeback superseded; `EBIND-WP-0002` registered
 - [x] Pod uses ephemeral Forge clones; no hostPath into any home directory
 - [x] Pod authenticates with a dedicated service identity
-- [ ] Pod runs as non-root with an explicit restrictive security context
+- [x] Pod runs as non-root with an explicit restrictive security context
-- [ ] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
+- [x] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
-- [ ] Dashboard reachable without the workstation, or formally handed to `hub-core`
+- [x] Dashboard reachable without the workstation, or formally handed to `hub-core`
-- [ ] Boundary rule written and discoverable by agents
+- [x] Boundary rule written and discoverable by agents
 - [x] Registrar queue drained; interim single-writer rule retired