the-custodian/workplans/CUST-WP-0067-hub-authority-target-resolution.md

id	type	title	domain	repo	status	owner	created	updated	quality_dor	quality_dor_at	quality_dor_by	quality_dor_note	state_hub_workstream_id
CUST-WP-0067	workplan	Resolve hub target ambiguity and restore repo onboarding to the authoritative hub	infotech	the-custodian	active	codex	2026-08-24	2026-08-24	DoR-Ok	2026-08-24	codex	Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml.	16249302-2767-55df-aec0-d92c2751c225

Resolve hub target ambiguity and restore repo onboarding to the authoritative hub

Goal

Make "the authoritative hub" a resolvable address rather than a policy claim, so that repository onboarding lands on central instead of a local cache that believes it is primary. Then recover the seven weeks of workstation-originated repo registrations that never reached central, and correct the ADR-010 framing that treats this class of gap as a refresh problem.

Context

ADR-010 documented two State Hub instances sharing port 8000, separated only by IP family, and named the consequence: every tool defaulting to 127.0.0.1 reaches the local instance while believing it is the primary. That condition was never remediated. As of 2026-08-24 it is still live:

LISTEN 127.0.0.1:8000   uvicorn (pid 3258)             local cache
LISTEN     [::1]:8000   ssh -L 8000:10.43.68.154:8000  central

The collision is declared in ~/.config/bridge/tunnels.yaml, where the tunnel state-hub-primary takes local_port: 8000 — the port the local uvicorn already holds. ssh -L binds both address families, the IPv4 bind loses to uvicorn, and the IPv6 bind succeeds, so ExitOnForwardFailure=yes does not fire. _reject_duplicate_local_ports in ops-bridge guards only bridge tunnel against bridge tunnel; it cannot see a non-bridge listener.

Measured repository divergence, 2026-08-24:

cache (127.0.0.1:8000)     122
central ([::1]:8000)        78
central-only                 0
cache-only                  44

Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh of the cache would delete the 44 rather than reconcile them, and central holds no path to re-derive records for repositories it has never been told exist.

Onboarding by registration date shows a clean break:

Registered	central	cache-only
2026-02 – 2026-06	74	0
2026-07	1	24
2026-08	3	20

Central is otherwise live — last_state_synced_at runs to 2026-08-22, and the three August rows (repo-manager, rail-kubernetes, fin-hub) were registered from the railiance side. Only workstation-originated onboarding is lost.

Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have an origin remote and are pushed (soul-frame and rein-openweights are ahead of their remote), and 35 carry .repo-classification.yaml. agent-harness has no working copy and needs separate disposition.

hub-record-authority.yaml assigns managed_repos to repo-manager as file-derived. Repo Manager owns the record type but exposes no command that onboards a repository into a hub, and registrar-reconcile --confirm-primary defaults its "primary" to 127.0.0.1:8000 — confirming projections against the cache and reporting success while central holds nothing.

Repair the crashing status printer

id: CUST-WP-0067-T01
status: done
priority: low
state_hub_task_id: "f3608db4-20a5-58fb-a965-885eb14858af"

statehub status raised KeyError: 'in_progress' at custodian_cli.py:565. The task vocabulary is wait|todo|progress|done|cancel; in_progress and blocked are not task statuses in this schema and the totals block never carried them. Fix the key names, tolerate the workplans/workstreams rename, and print the resolved API base so the operator can see which instance answered.

Done (2026-08-24): custodian_cli.py cmd_status now reads progress/todo/wait, accepts either totals key for workplans, uses .get() defaults throughout, and prints API base: as its second line. Verified against the live API.

Retire the local hub instance and the reverse-tunnel relay

id: CUST-WP-0067-T02
status: progress
priority: high
state_hub_task_id: "4093e928-d752-5a91-96c3-2e80f0e1dac5"

Decision, 2026-08-24: rather than making two hub instances coexist safely, retire the second one. The local postgres:16-alpine + uvicorn instance is what impersonates central, and it is redundant — ADR-010 decision 3 already states that local work requires no hub at all, and Repo Manager already maintains a file-derived local index (index_store.py, rmgr cache status / cache rebuild). Two caching layers exist and one of them is a database pretending to be the primary.

With the local instance gone, state-hub-primary binds 127.0.0.1:8000 unchanged and every existing http://127.0.0.1:8000 default becomes correct without editing a single call site. The port collision cannot recur because only one process binds the port.

Retire the reverse tunnels too. state-hub-railiance01 forwards a remote box's :18000 back to this workstation's :8000, so a remote agent following the documented port map reaches the workstation rather than the primary — which on railiance01 is its own machine. That topology assumed the workstation was the hub; it has not been since the primary moved.

Sequence matters: export the cache-only records first, stop serving second, discard the cache data only after T05 proves central holds everything.

Acceptance: no local hub process listening; 127.0.0.1:8000 answers from central; MCP dev-hub resolves to central; reverse state-hub-* tunnels removed or repointed; the cache-only recovery export is committed.

Progress (2026-08-24): docs/recovery/cache-only-repos-2026-08-24.json captures all 44 cache-only repository records with working-copy presence, classification file presence, and HEAD sha — the recovery source for T05. ops-bridge now pins local forwards to 127.0.0.1 (commit 2213847), so a contested port fails loudly instead of silently landing on [::1].

The local hub instance is retired: make api (uvicorn on 127.0.0.1:8000) is stopped, state-hub-primary restarted onto the freed IPv4 address, and the orphaned [::1] forward removed. Exactly one process now binds 8000 and it is the tunnel to central — /repos/ returns 78 there, not the cache's 122, and statehub status reports central's 10 active workplans rather than 27. The MCP server on :8001 needed no change: it targets http://127.0.0.1:8000 and now proxies central. No call site was edited; retiring the impersonator made the existing defaults correct.

The Postgres container is deliberately left running with its data intact. It is the recovery source of last resort until T05 proves central holds all 44, per the sequencing above.

Remaining: the reverse relay tunnels state-hub-railiance01 and state-hub-mcp-railiance01 still route remote agents back to this workstation. Retiring them requires editing ~/.config/bridge/tunnels.yaml, which is outside any repo and was blocked in-session; the change and its sequencing constraint are recorded in docs/recovery/tunnels-yaml-proposed-changes-CUST-WP-0067.md. Remote agents and the documented port map must be repointed before removal.

Make the hub target explicit and unspoofable

id: CUST-WP-0067-T03
status: todo
priority: high
state_hub_task_id: "29977448-1a74-5a4d-b729-974c15b6bbde"

Retiring the local instance removes today's impersonator but not the ability for a future one to appear. Give the hub an instance identity it can assert — role served from the health or summary endpoint — and make registrar-reconcile --confirm-primary refuse anything that does not assert primary.

_check_primary currently asserts only status == ok and db == connected. Both instances passed it. It is a liveness check wearing an authority check's name, and it is what allowed a cache to certify itself as the registrar.

Acceptance: --confirm-primary fails against a non-primary instance and names what it reached; a hub reports its role; statehub status shows role alongside the API base.

Give Repo Manager a real onboarding write path

id: CUST-WP-0067-T04
status: done
priority: high
state_hub_task_id: "ffa141d5-331d-543d-8b87-516f27973a22"

Repo Manager owns managed_repos as file-derived in hub-record-authority.yaml and exposes no command for it. Add onboarding that follows ADR-010 decision 5: write .repo-classification.yaml in the target repository, commit, push, and have central derive the record. Central must not accept a push of derived state, so the command makes the source file correct and reachable, then triggers and verifies derivation.

Resolve the bootstrap gap explicitly — central derives from repositories it already knows about, so a never-registered repository is never scanned. The onboarding path must be able to introduce a repository central has not seen.

Acceptance: onboarding a fresh repository from the workstation produces a central record with no manual step; re-running is idempotent.

Done (2026-08-24): rmgr repo-onboard (repo-manager 1be6d85). Validates the classification file, then refuses to proceed if it has uncommitted changes, no upstream, or unpushed commits — a hub record whose backing file is only local cannot be re-derived by anyone. --api-base has no default on purpose; a silent localhost default is the original defect, not a convenience. Six guard tests; full suite 122 passed.

This uncovered why onboarding only ever worked from a workstation, which is deeper than the tunnel: api/classification.py resolved its allowed-values file from three hardcoded developer checkout paths and nothing else, so inside the container every classification write returned 500. Fixed with REPO_CLASSIFICATION_ALLOWED_PATH (state-hub 05dda8d); delivering the canon file into the pod is T09.

Onboard the 44 cache-only repositories to central

id: CUST-WP-0067-T05
status: done
priority: medium
state_hub_task_id: "078159e6-5ecd-5f9d-b3ec-1fabf60955f7"

Runs after T04. With the cache retired this is no longer a convergence of two hubs — it is onboarding 44 repositories to central from their files, which is T04 applied to the recovery export.

Ten of the 44 lack .repo-classification.yaml and need one authored with the owner rather than guessed; classification is not mechanical, per CUST-WP-0065-T01. Push soul-frame and rein-openweights first — both are ahead of their remote, and central derives from what it can fetch. agent-harness has no working copy: decide restore-from-remote or drop and record it. Do not preserve it as a hub-only record.

Acceptance: every record in the recovery export exists on central or carries a written disposition; only then may the local cache database be discarded.

Done (2026-08-24): 33 of 44 onboarded to central, which went from 78 to 111 repositories. The remaining 11 each carry a written disposition in docs/recovery/onboarding-disposition-2026-08-24.md: nine need a classification file authored with their owner, soul-frame has unpushed research commits that are the owner's to publish, and agent-harness has no working copy and must be restored or explicitly dropped.

None were forced. Classification is not mechanical (CUST-WP-0065-T01), and a confident wrong domain is harder to correct than a missing one.

All 33 landed with their domain but no classification fields, pending T09. They are re-derivable: the classification files are committed and pushed, so re-running onboarding projects them once the hub can validate.

The local cache database may not be discarded yet — 11 records still exist only there.

Correct the ADR-010 framing and record the retirement

id: CUST-WP-0067-T06
status: done
priority: medium
state_hub_task_id: "007bfcf1-3b17-5d4a-b16a-b80ebf273934"

ADR-010 decision 2 says a divergent database is a merge problem and a stale cache is a refresh problem. For managed_repos neither held: the gap was a strict subset in the cache's favour, and refreshing would have destroyed rather than reconciled. Record that third shape — cache-only records whose authoritative source exists but was never introduced to central — with re-derivation from source as its remedy.

Record the mechanical cause, which the ADR observed but did not diagnose: an unbound ssh -L binds every loopback family, so the IPv4 bind losing to a local listener still leaves a working [::1] forward and ExitOnForwardFailure never fires. The ADR treated the shared port as the hazard; the missing bind address was what made it silent.

Note also that ADR-010 reads as though its remediation landed. It did not — the condition it measured was still live seven weeks later. Supersede decision 2 for this record class and record the local-instance retirement as the outcome.

Acceptance: ADR-010 revised with a dated superseding note linked to this workplan; ops-bridge and the port map documented as the structural fix.

Done (2026-08-24): three dated corrections added to ADR-010, following the in-file blockquote convention ADR-007 already uses.

• Decision 2 — the merge/refresh dichotomy is not exhaustive. Records the third shape: cache-only records whose authoritative source exists and is reachable but was never introduced to central, remedied by re-derivation rather than refresh or merge. Refreshing would have destroyed the 44.
• Context — the ADR recorded the shared port but not why it was silent, which is the part that let it persist for seven weeks. An unbound ssh -L binds every loopback family and ExitOnForwardFailure fires only when all binds fail, so the tunnel reported success while serving an address nothing resolved to by default.
• Outcome section — states plainly that the ADR's own remediation never landed, and that the condition it measured was still live seven weeks later. An ADR describing a fix in the present tense reads as a fix that happened.

Also records the deeper cause as a second instance of the ADR's own theme: the hub resolved its classification vocabulary from developer-workstation paths, so authority depended on where a process happened to run.

Repoint the fleet's copied port map

id: CUST-WP-0067-T07
status: todo
priority: high
state_hub_task_id: "dbc672d5-805a-567a-b548-4e8870e4fb2f"

The remote port map is not configuration. It is prose copied into roughly 120 AGENTS.md files across both machines, with no template or generator behind it — a grep for the block finds only the files themselves and past agent session logs. That is how one wrong topology propagated fleet-wide and survived the primary moving off the workstation.

Every copy still tells a remote agent to reach the hub at 127.0.0.1:18000, which forwards back to the workstation. The global agent instructions are already repointed to the in-cluster address (2026-08-24); these copies are not, and they outrank nothing but are read first in their own repos.

Two things are needed, and the second matters more than the first:

1. A scripted, reviewable repoint of the block across all affected repos on both machines — mechanical, but 120 repos times commit and push, so it needs a dry run and a diff sample reviewed before it runs.
2. A single source for the block, so the next topology change is one edit rather than 120. Repo Manager already owns repository representation and scaffold; this belongs there. Without it this task recurs verbatim.

Retiring state-hub-railiance01 is blocked on this: the tunnel is what those 120 files currently depend on.

Acceptance: no AGENTS.md instructs a remote agent to use 127.0.0.1:18000; the block has one authoritative source; a spot check on both machines confirms the rendered guidance matches the live topology.

Deploy an MCP server on central

id: CUST-WP-0067-T08
status: done
priority: medium
state_hub_task_id: "b17505c0-ea13-5ee2-87ca-8c06607f3b5b"

The Helm chart deploys a single container serving HTTP on 8000. No MCP server runs on central, so dev-hub for remote agents exists only as a reverse tunnel to the workstation's MCP — which now proxies central, making the path remote → workstation → central for a service on the remote's own machine.

The work is small: the same image already contains mcp_server/server.py, run with MCP_TRANSPORT=sse MCP_PORT=8001 and API_BASE pointed at the in-cluster service. It needs a Deployment, a Service, and chart values.

Until this lands, remote agents should prefer the REST surface; the session protocol documents a curl form for every operation, so this is a degraded convenience rather than a blocker. Retiring state-hub-mcp-railiance01 is blocked on this task.

Acceptance: an MCP endpoint reachable from the node without traversing the workstation; remote dev-hub registration points at it; the reverse MCP tunnel removed.

Progress (2026-08-24): chart support is written and validated — a gated mcp Deployment and ClusterIP Service running the same image with -m mcp_server.server, API_BASE defaulted to the in-cluster API Service, and tcpSocket probes. helm lint and helm template are clean, and the templates render nothing when mcp.enabled is false (the default).

Two things had to be fixed before the YAML would have worked:

• mcp_server/server.py hardcoded host="127.0.0.1". A Service routes to the pod IP, so a loopback bind is unreachable from anywhere but the pod itself. Now MCP_HOST, still defaulting to loopback — the MCP layer proxies an unauthenticated API and must not land on every interface by accident.
• The container command uses -m mcp_server.server rather than the file path, so /app lands on sys.path instead of /app/mcp_server.

Deliberately no Ingress for this Service, for the same reason the API's ingress is disabled: neither has authentication.

Blocked on a release, not on more code. The running image is main-76e6eda, which predates the MCP_HOST change, so enabling mcp today would deploy a pod that binds loopback and never becomes reachable. The order is: merge, let CI build main-<sha>, then bump the tag and flip mcp.enabled: true in deploy/railiance/apps/helm/state-hub-values.yaml in one commit, then helm upgrade. The values file is deliberately left at enabled: false so an unrelated helm upgrade cannot ship a broken MCP.

Release built (2026-08-24). Pushing revealed why the image would not have rebuilt: the CI path filter matched api/** while the Dockerfile copies nine more paths, so changes to mcp_server/ — and to migrations/, which is the serious one, since a schema migration would not have shipped — merged to main without producing an image. Filter corrected to match the Dockerfile (8221216); run #284 succeeded and main-8221216 pulls on the node.

The production values file also drifted: it pinned main-697dd49 while the cluster ran main-76e6eda, and it omitted sweep.hostname, which the live release carries — so deploying from that file alone would have silently dropped it. Both corrected; helm template from the file now renders byte-identical output to the intended release.

Remaining, both blocked on operator permission:

1. Commit and push deploy/railiance/apps/helm/state-hub-values.yaml (tag bump plus mcp.enabled: true). The edit is on disk, uncommitted — the permission classifier declined the commit twice.
2. helm upgrade — also declined. Dry run is clean and adds only the state-hub-mcp Deployment and ClusterIP Service:

cd ~/state-hub
helm upgrade state-hub ./deploy/railiance/apps/charts/state-hub \
  -n state-hub -f deploy/railiance/apps/helm/state-hub-values.yaml \
  --wait --timeout 4m

Node headroom was checked first: CPU requests sit at 92% of ~4000m, leaving ~320m. The MCP pod requests 50m, which keeps enough for the API's maxSurge=1 roll.

After the upgrade, verify from the node rather than through a tunnel:

ssh tegwick@92.205.62.239 'curl -sI http://<mcp-clusterip>:8001/sse'
kubectl -n state-hub get pods -l app.kubernetes.io/component=mcp

Done (2026-08-24): deployed and verified. state-hub-mcp is Running, serving SSE on ClusterIP 10.43.110.80:8001 — confirmed from the node itself rather than through a tunnel: HTTP 200, text/event-stream, and a real MCP endpoint handshake, not merely an open port. End-to-end the pod reaches the in-cluster API (API_BASE=http://state-hub:8000) and reads central's data, 79 repositories rather than the retired cache's 122.

The global agent instructions now point remote dev-hub at that ClusterIP, and the remote port map lists no tunnel for either service.

state-hub-mcp-railiance01 is now unblocked for removal — the last thing depending on it is the documented registration, which has been repointed.

Deliver the classification allowed-values file to central

id: CUST-WP-0067-T09
status: progress
priority: high
state_hub_task_id: "45a2ba22-9d50-5ba8-9fa7-6e345ddce395"

api/classification.py resolved its allowed-values file from three hardcoded developer workstation checkouts, so in the container none exist and every classification write returned 500. This is why repository classification could only ever be written from a workstation, and why all 33 repositories onboarded under T05 carry a domain but no classification fields.

REPO_CLASSIFICATION_ALLOWED_PATH now overrides the search (state-hub 05dda8d), but the file still has to reach the pod. Canon lives in the-custodian, and the hub must not grow a second copy that silently drifts from it — mount it from a ConfigMap generated from the canon file, and make the generation step part of the release rather than a manual copy.

Once it serves, re-run onboarding across the 33 to project their classification.

Acceptance: a classification PATCH succeeds against central; the 33 records carry their categories; the allowed-values source in the cluster is traceable to the-custodian/canon/standards/repo-classification.allowed.yaml.

Progress (2026-08-24): delivery is built and validated. The chart carries a generated copy of the canon file, mounted as a ConfigMap at /etc/state-hub/classification, with REPO_CLASSIFICATION_ALLOWED_PATH set to it.

The copy is the risk, so it is owned rather than trusted: scripts/sync_classification_allowed.py regenerates it from canon, and --check fails on drift. make check-classification-allowed and make railiance-state-hub-render both exit non-zero when the copy diverges, so a release cannot ship a stale vocabulary and silently validate against yesterday's rules — verified by injecting drift.

Rendered ConfigMap parses and matches canon (version 1.0, 6 categories, 14 domains). The volumeMounts/env blocks were merged rather than appended: adding a second pair would have produced duplicate YAML keys the moment sweep was re-enabled. Both sweep states render correctly; helm lint clean.

Remaining: one command. Merged, CI green (ac21acc), image main-ac21acc built, and the production values file bumped and pushed. The helm upgrade itself was declined by the permission classifier, as in T08.

Live today: release 19 on main-8221216, with state-hub and state-hub-mcp running and no REPO_CLASSIFICATION_ALLOWED_PATH in the pod — so classification writes still 500.

cd ~/state-hub
helm upgrade state-hub ./deploy/railiance/apps/charts/state-hub \
  -n state-hub -f deploy/railiance/apps/helm/state-hub-values.yaml \
  --wait --timeout 4m

Then confirm the vocabulary is actually mounted, not merely that the pod is up:

kubectl -n state-hub exec deploy/state-hub -- \
  head -3 /etc/state-hub/classification/repo-classification.allowed.yaml

Then project classification across the 33 already-onboarded repositories by re-running onboarding; it is idempotent:

while read -r slug path; do
  uv run --project ~/repo-manager rmgr repo-onboard \
    --path "$path" --api-base http://127.0.0.1:8000
done < <list of onboarded repos>

After that the local cache database holds nothing unique except the 11 dispositioned records.