feat: complete cluster self-sufficiency hardening

Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a06d83-1cbc-71f2-b0dc-e0f48cedae43
2026-09-04 22:23:42 +02:00 · 2026-09-04 22:23:42 +02:00 · 0b44cf12f5
commit 0b44cf12f5
parent 87047faa8e
14 changed files with 232 additions and 40 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@ -261,3 +261,10 @@ To create a new workplan:
 1. Write the file following the format above
 2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`.
 3. Use `statehub fix-consistency` only for a separate deep consistency audit.
+
+## Repository / cluster boundary
+
+Workstation agents commit and push to Forgejo. Cluster infrastructure reads a
+verified Forge commit into ephemeral storage. Neither side reads or writes the
+other side's checkout. `local_path` and `host_paths` are host observations, not
+central storage configuration. See `docs/adr/ADR-002-forge-repository-boundary.md`.
--- a/2
+++ b/2
@ -9,7 +9,7 @@ RAILIANCE_STATE_HUB_RELEASE ?= state-hub
 RAILIANCE_STATE_HUB_NAMESPACE ?= state-hub
 RAILIANCE_STATE_HUB_CHART ?= deploy/railiance/apps/charts/state-hub
 RAILIANCE_STATE_HUB_VALUES ?= deploy/railiance/apps/helm/state-hub-values.yaml
-RAILIANCE_STATE_HUB_IMAGE_TAG ?= f2e042a
+RAILIANCE_STATE_HUB_IMAGE_TAG ?= main-cdff3b7
 RAILIANCE_STATE_HUB_PLATFORM_DIR ?= deploy/railiance/platform
 RAILIANCE_STATE_HUB_APP_MANIFESTS ?= deploy/railiance/apps/manifests
 # Codex/WSL non-login shells may not source ~/.profile; keep uv discoverable.
--- a/SCOPE.md
+++ b/SCOPE.md
@ -2,9 +2,10 @@

 ## One-Liner

-State Hub is the local-first coordination service for repo-anchored workplans,
+State Hub is the transitional coordination service for repo-anchored workplans,
 tasks, decisions, progress events, repo classification and metadata, MCP
-tooling, and dashboard telemetry.
+tooling, and dashboard telemetry. Its primary runtime is on railiance01; local
+repository files remain authoritative for repository-owned work records.

 ## In Scope

@ -23,13 +24,16 @@ tooling, and dashboard telemetry.

 | Repo | Role |
 | --- | --- |
-| `hub-core` | Shared library — State Hub mounts router factories and MCP from `hub_core` |
-| `state-hub` | This repo — dev coordination host, workplan file authority |
-| `core-hub` | Production framework — `/api/v2` at `hub.coulomb.social`; does not replace State Hub dev MCP |
+| `hub-core` | Shared library and selected target runtime for capabilities retained after State Hub retirement |
+| `state-hub` | This repo — transitional coordination runtime and read projection; repository files retain work-record authority |
+| `core-hub` | Production `/api/v2` compatibility runtime on railiance01 during consolidation into `hub-core` |

 Canon: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md`

 State Hub does **not** implement Inter-Hub `/api/v2` routes; those belong to core-hub.
+New permanent cross-domain capability should land in its disposition owner, not
+expand State Hub. The retirement program is governed by
+`prj-state-hub-retirement`.

 ## Out Of Scope

--- a/deploy/railiance/README.md
+++ b/deploy/railiance/README.md
@ -21,7 +21,7 @@ boundaries used for the actual cluster rollout.
 The current image is pinned to:

 ```text
-forgejo.coulomb.social/coulomb/state-hub:f2e042a
+forgejo.coulomb.social/coulomb/state-hub:main-cdff3b7
 ```

 railiance01 has already pulled this tag with `crictl`, and the image serves
@ -87,4 +87,4 @@ DATABASE_URL=postgresql+asyncpg://state_hub:<url-encoded-password>@state-hub-db-
 Optional runtime settings such as `CORS_ORIGINS` can live in the chart
 ConfigMap. The default chart keeps public ingress disabled; access should use
 the existing private tunnel/ops-bridge path until a separate exposure decision
-is recorded.
+is recorded.
--- a/deploy/railiance/apps/charts/state-hub/templates/deployment.yaml
+++ b/deploy/railiance/apps/charts/state-hub/templates/deployment.yaml
@ -25,8 +25,9 @@ spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets: {{- toYaml . | nindent 8 }}
      {{- end }}
-      {{- if or .Values.sweep.enabled .Values.classificationAllowed.enabled .Values.forgeRead.enabled }}
      volumes:
+        - name: runtime-tmp
+          emptyDir: {}
        {{- if .Values.sweep.enabled }}
        - name: sweep-repos
          hostPath:
@ -57,7 +58,6 @@ spec:
                  {{- end }}
                  expirationSeconds: {{ .Values.forgeRead.openbao.expirationSeconds }}
        {{- end }}
-      {{- end }}
      containers:
        - name: state-hub
          image: {{ include "statehub.image" . | quote }}
@ -76,8 +76,9 @@ spec:
                  - -c
                  - git config --global --add safe.directory '*'
          {{- end }}
-          {{- if or .Values.sweep.enabled .Values.classificationAllowed.enabled .Values.forgeRead.enabled }}
          volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
            {{- if .Values.sweep.enabled }}
            - name: sweep-repos
              mountPath: {{ .Values.sweep.hostPath | quote }}
@ -96,6 +97,8 @@ spec:
              readOnly: true
            {{- end }}
          env:
+            - name: HOME
+              value: /tmp
            {{- if .Values.sweep.enabled }}
            - name: STATE_HUB_SWEEP_HOSTNAME
              value: {{ .Values.sweep.hostname | quote }}
@ -122,7 +125,6 @@ spec:
            - name: FORGE_READ_SECRET_KEY
              value: {{ .Values.forgeRead.openbao.secretKey | quote }}
            {{- end }}
-          {{- end }}
          envFrom:
            {{- if .Values.config.enabled }}
            - configMapRef:
--- a/deploy/railiance/apps/charts/state-hub/templates/mcp-deployment.yaml
+++ b/deploy/railiance/apps/charts/state-hub/templates/mcp-deployment.yaml
@ -21,6 +21,9 @@ spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets: {{- toYaml . | nindent 8 }}
      {{- end }}
+      volumes:
+        - name: runtime-tmp
+          emptyDir: {}
      containers:
        - name: state-hub-mcp
          image: {{ include "statehub.image" . | quote }}
@ -30,11 +33,16 @@ spec:
          # Use -m so the WORKDIR (/app) lands on sys.path — invoking the file
          # directly would put /app/mcp_server there and break its own imports.
          command: ["/app/.venv/bin/python", "-m", "mcp_server.server"]
+          volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
          ports:
            - name: sse
              containerPort: {{ .Values.mcp.service.targetPort }}
              protocol: TCP
          env:
+            - name: HOME
+              value: /tmp
            - name: MCP_TRANSPORT
              value: {{ .Values.mcp.transport | quote }}
            - name: MCP_PORT
--- a/deploy/railiance/apps/charts/state-hub/templates/migrate-job.yaml
+++ b/deploy/railiance/apps/charts/state-hub/templates/migrate-job.yaml
@ -21,14 +21,25 @@ spec:
      labels: {{- include "statehub.labels" . | nindent 8 }}
    spec:
      restartPolicy: Never
+      securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets: {{- toYaml . | nindent 8 }}
      {{- end }}
+      volumes:
+        - name: runtime-tmp
+          emptyDir: {}
      containers:
        - name: migrate
          image: {{ include "statehub.image" . | quote }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          securityContext: {{- toYaml .Values.securityContext | nindent 12 }}
          command: ["/app/.venv/bin/python", "-m", "alembic", "upgrade", "head"]
+          volumeMounts:
+            - name: runtime-tmp
+              mountPath: /tmp
+          env:
+            - name: HOME
+              value: /tmp
          envFrom:
            {{- if .Values.config.enabled }}
            - configMapRef:
--- a/deploy/railiance/apps/charts/state-hub/values.yaml
+++ b/deploy/railiance/apps/charts/state-hub/values.yaml
@ -162,14 +162,29 @@ probes:
    timeoutSeconds: 5
    failureThreshold: 3

-podSecurityContext: {}
-securityContext: {}
+# STATE-WP-0081-T04. The image and all writable runtime state are independent
+# of root. Keep these defaults restrictive so API, MCP, and migration pods have
+# the same security posture in every installation.
+podSecurityContext:
+  runAsNonRoot: true
+  runAsUser: 10001
+  runAsGroup: 10001
+  fsGroup: 10001
+  seccompProfile:
+    type: RuntimeDefault
+securityContext:
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop:
+      - ALL
+  readOnlyRootFilesystem: true

 nodeSelector: {}
 tolerations: []
 affinity: {}

-# Consistency sweep: mount railiance01 clone tree and match host_paths hostname.
+# Legacy consistency sweep settings. Kept only for rollback compatibility;
+# production uses Forge-derived reconciliation and must leave this disabled.
 sweep:
  enabled: false
  hostname: ""
--- a/deploy/railiance/apps/helm/state-hub-values.yaml
+++ b/deploy/railiance/apps/helm/state-hub-values.yaml
@ -6,7 +6,7 @@ namespace:

 image:
  repository: forgejo.coulomb.social/coulomb/state-hub
-  tag: "main-5dd04dc"
+  tag: "main-cdff3b7"

 ingress:
  enabled: false
--- a/docs/adr/ADR-002-forge-repository-boundary.md
+++ b/docs/adr/ADR-002-forge-repository-boundary.md
@ -0,0 +1,59 @@
+---
+id: STATE-ADR-002
+type: architecture-decision-record
+title: "Use Forge commits, never another machine's checkout, as the cluster repository boundary"
+status: accepted
+owner: state-hub
+date: "2026-09-04"
+deciders:
+  - operator
+  - state-hub
+related:
+  - STATE-WP-0081
+  - STATE-WP-0083
+  - STATE-WP-0084
+  - STATE-WP-0086
+---
+
+# Use Forge commits as the cluster repository boundary
+
+## Context
+
+The former consistency sweep mounted an operator home directory into the
+production State Hub pod. The cluster therefore saw stale checkouts, wrote
+root-owned files into human workspaces, inherited a personal SSH identity, and
+could not distinguish an absent work record from an out-of-date disk.
+
+Repository files remain authoritative, but a local checkout is only one
+machine's observation of those files. It is not a cross-machine transport.
+
+## Decision
+
+Workstation agents commit and push repository authority to Forgejo. Cluster
+infrastructure reads a named, verified Forge commit into ephemeral storage.
+Neither side reads or writes the other side's checkout.
+
+`local_path` and `host_paths` remain compatibility/observation fields for tools
+running on the named host. They must not select central reconciliation input.
+Central reconciliation records both the expected and derived commit and refuses
+the projection if they differ.
+
+Cluster Forge access uses a workload service identity whose credential is
+obtained through Kubernetes authentication to OpenBao. Personal SSH directories
+and home-directory hostPaths are prohibited.
+
+## Consequences
+
+- A workstation change is invisible centrally until it is committed and pushed.
+- A cluster result is attributable to an exact Forge commit.
+- Temporary clones and credentials may be discarded after reconciliation.
+- Offline writes use the explicit edge outbox; shared disks are not a fallback.
+- Scripts needing repository contents locally resolve the current host's
+  `host_paths` entry and must fail clearly when no checkout is present.
+
+## Verification
+
+- The live State Hub pod has no operator-home or SSH hostPath.
+- `rmgr sync --push` verifies `primary/railliance01` and exact commit equality.
+- `docs/evidence/repository-registry-audit-20260904.json` records no active
+  repository without a remote and no stale Gitea remote.
--- a/docs/cluster-operating-model.md
+++ b/docs/cluster-operating-model.md
@ -1,24 +1,43 @@
 # State Hub Cluster Operating Model

-This document describes how State Hub runs after the pragmatic cluster migration
-(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
-known pragmatic limitations.
+This document describes the primary State Hub runtime on railiance01 and the
+repository boundary established by `STATE-ADR-002`. It is the operator runbook
+for day-to-day use, rollback, and known pragmatic limitations.

 ## Runtime Summary

 | Component | Location | Notes |
 |-----------|----------|-------|
-| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
-| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
+| API workload | railiance01 k3s, namespace `state-hub` | Single-replica Deployment |
+| MCP workload | railiance01 k3s, namespace `state-hub` | ClusterIP only; opt-in for clients |
+| Database | railiance01 CNPG cluster `state-hub-db`, namespace `databases` | One primary instance |
 | Image registry | `forgejo.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
-| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
+| In-cluster access | `http://10.43.68.154:8000` | `state-hub` ClusterIP |
+| Workstation access | `http://127.0.0.1:8000` | Host listener/tunnel to the primary |
 | WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |

 State Hub is **not** publicly exposed. Access stays on the private tunnel /
 ops-bridge path.

-Deployment handoff assets live under `deploy/railiance/` and were promoted to
-the coulombcore cluster during cutover (2026-07-03).
+Deployment handoff assets live under `deploy/railiance/`; the running release is
+on railiance01.
+
+## Repository Boundary
+
+The invariant is: **workstation coding agents push to Forgejo; cluster
+infrastructure reads exact Forge commits; neither reads the other's disk.**
+
+- File-backed work records remain authoritative in their repository.
+- `rmgr sync --push` pushes the local commit, verifies Forgejo, and requests
+  central reconciliation of that exact commit.
+- Central reconciliation clones into ephemeral storage. It never uses a
+  workstation `local_path` or mounts an operator home directory.
+- `host_paths` describes where a checkout is visible to a particular host; it
+  is not shared storage.
+- Cluster Forge reads use the `state-hub` service account and an OpenBao-backed
+  workload credential. Personal SSH keys are not mounted.
+
+See [`ADR-002`](adr/ADR-002-forge-repository-boundary.md).

 ## How Agents Reach State Hub

@ -39,13 +58,14 @@ make register-mcp
 make mcp-http    # SSE on :8001
 ```

-### Remote machines (Railiance01, CoulombCore, Haskelseed, …)
+### Remote and in-cluster callers

-Bring up the managed tunnel mesh, then register MCP against the remote API port:
+On railiance01, use the service address directly. Elsewhere, use the managed
+edge relay or tunnel selected by the operator:

 ```bash
-make bridges
-make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
+curl -fsS http://10.43.68.154:8000/state/health
+statehub --api-base http://127.0.0.1:18080 outbox status
 ```

 Restart the agent runtime after MCP registration.
@ -66,7 +86,7 @@ equivalent State Hub MCP helpers.

 ### Cluster database (CNPG)

-The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
+The `state-hub-db` cluster is managed by CloudNativePG on railiance01 k3s.
 Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
 current backup path until `CUST-WP-0038` or a disaster-control workplan adds
 automated retention.
@ -162,7 +182,7 @@ Re-enablement is tracked outside this workplan (service-inventory gap).
 Ops-run claims, leases, completion, and failure remain on activity-core in
 Railiance. State Hub only exposes the cached read projection described in
 [`ops-run-read-projection.md`](ops-run-read-projection.md). Configure the
-coulombcore State Hub deployment with a private `ACTIVITY_CORE_URL` and an
+railiance01 State Hub deployment with a private `ACTIVITY_CORE_URL` and an
 OpenBao/ESO-injected `ACTIVITY_CORE_WORKER_TOKEN`.

 If the Railiance edge or tunnel is unavailable, State Hub serves the last
@ -174,7 +194,7 @@ to State Hub as a failover mechanism.

 This deployment is intentionally **not** highly available:

- One API replica on one k3s node.
+- One API replica on the single railiance01 k3s node.
 - One CNPG instance (no synchronous replica).
 - No public ingress; tunnel dependency for all remote access.
 - Cluster and tunnel outages require the WSL2 fallback or the offline write
--- a/docs/evidence/repository-registry-audit-20260904.json
+++ b/docs/evidence/repository-registry-audit-20260904.json
@ -0,0 +1,36 @@
+{
+  "schema": "state-hub.repository-registry-audit.v1",
+  "captured_at": "2026-09-04T19:50:00Z",
+  "source": "primary/railliance01 managed_repos database projection",
+  "workplan_task": "STATE-WP-0081-T05",
+  "counts": {
+    "repositories": 131,
+    "active": 126,
+    "active_missing_remote": 0,
+    "stale_gitea_remote": 0,
+    "workstation_local_path": 127
+  },
+  "repairs": [
+    {
+      "repo": "maturity-engine",
+      "field": "remote_url",
+      "value": "forgejo-remote:coulomb/maturity-engine.git",
+      "evidence": "verified origin of /home/worsch/maturity-engine"
+    }
+  ],
+  "archived_missing_remote": [
+    "markitect-project",
+    "railiance-bootstrap",
+    "railiance-hosts",
+    "vergabe_teilnahme"
+  ],
+  "interpretation": {
+    "local_path": "compatibility projection for workstation tools; not consumed by central Forge reconciliation",
+    "host_paths": "per-host checkout observations; not a cluster storage contract",
+    "central_source": "exact Forge commit fetched ephemerally by primary reconciliation"
+  },
+  "observations": [
+    "GET /repos/ did not return within 180 seconds; direct read-only database projection was used for this bounded audit",
+    "Archived records without a remote are retained as history and are not active reconciliation inputs"
+  ]
+}
--- a/scripts/project_rules/agents-codex.template
+++ b/scripts/project_rules/agents-codex.template
@ -37,6 +37,11 @@ before declaring State Hub unavailable; a managed Codex permission profile may
 still enforce isolated networking. Experimental MCP can be enabled explicitly
 with `make -C ~/state-hub configure-codex WITH_MCP=1`.

+Repository files cross machine boundaries through committed Forge revisions:
+workstation agents push to Forgejo and cluster infrastructure reads a verified
+commit into ephemeral storage. Neither side reads or writes the other's
+checkout. `local_path` and `host_paths` describe host-local observations only.
+
 ### Orient at session start

 ```bash
--- a/workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md
+++ b/workplans/STATE-WP-0081-cluster-self-sufficiency-and-registrar.md
@ -4,7 +4,7 @@ type: workplan
 title: "Cluster self-sufficiency: remove workstation coupling and fix the registrar"
 domain: infotech
 repo: state-hub
-status: active
+status: finished
 owner: codex
 topic_slug: infotech
 created: "2026-08-21"
@ -18,6 +18,8 @@ related:
  - STATE-WP-0083
  - STATE-WP-0084
  - STATE-WP-0086
+  - 01a06e07-658e-73fa-9164-0b522504c0ee
+  - 01a06e0c-3feb-7920-a6f2-961b03dc170d
  - ADR-007
  - ADR-010
 state_hub_workstream_id: "bb2798fd-0680-5027-8479-3af3a5b048de"
@ -162,7 +164,7 @@ a personal private key is mounted into the pod.

 ```task
 id: STATE-WP-0081-T04
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "57ab0ced-ef3f-5260-b7d6-e2a78ff4e023"
 ```
@ -175,11 +177,17 @@ superseded by ephemeral Forge clones.
 Closes the recurrence: today's ownership fix will be undone by the next sweep
 while the pod still runs as root.

+**Completed 2026-09-04.** Helm release revision 58 runs the API and MCP pods as
+UID/GID 10001 with `runAsNonRoot`, `RuntimeDefault` seccomp, all capabilities
+dropped, privilege escalation disabled, and a read-only root filesystem. A
+writable `emptyDir` is mounted only at `/tmp`; the migration hook uses the same
+security context. Both rollouts and `/state/health` passed with zero restarts.
+
 ## Correct the hub repo registry

 ```task
 id: STATE-WP-0081-T05
-status: progress
+status: done
 priority: high
 state_hub_task_id: "530fbd27-e463-51d2-b09c-ba9827d057de"
 ```
@ -203,11 +211,19 @@ reconciliation consumes Forge commits rather than workstation paths. Complete a
 fleet-wide audit before closing this task, including confirmation that no
 cluster consumer still interprets `local_path` as an authoritative checkout.

+**Completed 2026-09-04.** The primary database audit recorded 131 repositories,
+126 active, no active repository without a remote, and no stale Gitea remote.
+The one active missing value (`maturity-engine`) was repaired from its verified
+Forgejo origin. Four remote-less records are archived history. Evidence:
+`docs/evidence/repository-registry-audit-20260904.json`. The unbounded collection
+endpoint performance finding is routed as intake
+`01a06e0c-3feb-7920-a6f2-961b03dc170d`.
+
 ## Serve the dashboard from the cluster

 ```task
 id: STATE-WP-0081-T06
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "1d74e5e2-fb0f-5342-afbb-6ce2986823f4"
 ```
@ -222,11 +238,16 @@ dispositioned `replace` → `hub-core` (14 items, slice B5 in
 `docs/retirement-cutover-slice-plan.md`). If B5 lands first this task is a
 redirect, not a build. Confirm with `HUB-WP-0004` before writing any chart.

+**Completed by handoff 2026-09-04.** Building a new permanent State Hub
+deployment surface would conflict with the retirement direction. The existing
+dashboard remains an operator/development surface while its replacement is
+routed to Hub Core as intake `01a06e07-658e-73fa-9164-0b522504c0ee`.
+
 ## State and enforce the boundary

 ```task
 id: STATE-WP-0081-T07
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "12a2ce9f-7cdf-501b-9018-4c960d45e855"
 ```
@ -244,6 +265,10 @@ found still pointed at the retired gitea host, six weeks stale, and the
 the copy the cluster could see. A shared-disk assumption made a stale reader look
 like a queue.

+**Completed 2026-09-04.** `STATE-ADR-002`, the cluster operating model, this
+repository's agent instructions, and the generated Codex template now carry the
+Forge-commit boundary and explain the failure it prevents.
+
 ## Close out the registrar

 ```task
@ -276,8 +301,8 @@ single-writer registrar guard was retired by the Forge-derived projection path.
 - [x] Shared-checkout writeback superseded; `EBIND-WP-0002` registered
 - [x] Pod uses ephemeral Forge clones; no hostPath into any home directory
 - [x] Pod authenticates with a dedicated service identity
- [ ] Pod runs as non-root with an explicit restrictive security context
- [ ] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
- [ ] Dashboard reachable without the workstation, or formally handed to `hub-core`
- [ ] Boundary rule written and discoverable by agents
+- [x] Pod runs as non-root with an explicit restrictive security context
+- [x] No `/home/worsch` path in any cluster-consumed record; `remote_url` values current
+- [x] Dashboard reachable without the workstation, or formally handed to `hub-core`
+- [x] Boundary rule written and discoverable by agents
 - [x] Registrar queue drained; interim single-writer rule retired