the-custodian/workplans/CUST-WP-0067-hub-authority-target-resolution.md

---
id: CUST-WP-0067
type: workplan
title: "Resolve hub target ambiguity and restore repo onboarding to the authoritative hub"
domain: infotech
repo: the-custodian
status: active
owner: codex
created: "2026-08-24"
updated: "2026-08-24"
quality_dor: DoR-Ok
quality_dor_at: "2026-08-24"
quality_dor_by: codex
quality_dor_note: "Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml."
state_hub_workstream_id: "16249302-2767-55df-aec0-d92c2751c225"
---

# Resolve hub target ambiguity and restore repo onboarding to the authoritative hub

## Goal

Make "the authoritative hub" a resolvable address rather than a policy claim, so
that repository onboarding lands on central instead of a local cache that
believes it is primary. Then recover the seven weeks of workstation-originated
repo registrations that never reached central, and correct the ADR-010 framing
that treats this class of gap as a refresh problem.

## Context

`ADR-010` documented two State Hub instances sharing port 8000, separated only
by IP family, and named the consequence: every tool defaulting to `127.0.0.1`
reaches the local instance while believing it is the primary. That condition was
never remediated. As of 2026-08-24 it is still live:

```text
LISTEN 127.0.0.1:8000   uvicorn (pid 3258)             local cache
LISTEN     [::1]:8000   ssh -L 8000:10.43.68.154:8000  central
```

The collision is declared in `~/.config/bridge/tunnels.yaml`, where the tunnel
`state-hub-primary` takes `local_port: 8000` — the port the local uvicorn
already holds. `ssh -L` binds both address families, the IPv4 bind loses to
uvicorn, and the IPv6 bind succeeds, so `ExitOnForwardFailure=yes` does not
fire. `_reject_duplicate_local_ports` in `ops-bridge` guards only bridge tunnel
against bridge tunnel; it cannot see a non-bridge listener.

Measured repository divergence, 2026-08-24:

```text
cache (127.0.0.1:8000)     122
central ([::1]:8000)        78
central-only                 0
cache-only                  44
```

Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh
of the cache would delete the 44 rather than reconcile them, and central holds
no path to re-derive records for repositories it has never been told exist.

Onboarding by registration date shows a clean break:

| Registered | central | cache-only |
|---|---|---|
| 2026-02 – 2026-06 | 74 | 0 |
| 2026-07 | 1 | 24 |
| 2026-08 | 3 | 20 |

Central is otherwise live — `last_state_synced_at` runs to 2026-08-22, and the
three August rows (`repo-manager`, `rail-kubernetes`, `fin-hub`) were registered
from the railiance side. Only workstation-originated onboarding is lost.

Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have
an `origin` remote and are pushed (`soul-frame` and `rein-openweights` are ahead
of their remote), and 35 carry `.repo-classification.yaml`. `agent-harness` has
no working copy and needs separate disposition.

`hub-record-authority.yaml` assigns `managed_repos` to `repo-manager` as
`file-derived`. Repo Manager owns the record type but exposes no command that
onboards a repository into a hub, and `registrar-reconcile --confirm-primary`
defaults its "primary" to `127.0.0.1:8000` — confirming projections against the
cache and reporting success while central holds nothing.

## Repair the crashing status printer

```task
id: CUST-WP-0067-T01
status: done
priority: low
state_hub_task_id: "f3608db4-20a5-58fb-a965-885eb14858af"
```

`statehub status` raised `KeyError: 'in_progress'` at `custodian_cli.py:565`.
The task vocabulary is `wait|todo|progress|done|cancel`; `in_progress` and
`blocked` are not task statuses in this schema and the totals block never
carried them. Fix the key names, tolerate the `workplans`/`workstreams` rename,
and print the resolved API base so the operator can see which instance answered.

**Done (2026-08-24):** `custodian_cli.py` `cmd_status` now reads
`progress`/`todo`/`wait`, accepts either totals key for workplans, uses `.get()`
defaults throughout, and prints `API base:` as its second line. Verified against
the live API.

## Retire the local hub instance and the reverse-tunnel relay

```task
id: CUST-WP-0067-T02
status: progress
priority: high
state_hub_task_id: "4093e928-d752-5a91-96c3-2e80f0e1dac5"
```

Decision, 2026-08-24: rather than making two hub instances coexist safely,
retire the second one. The local `postgres:16-alpine` + uvicorn instance is what
impersonates central, and it is redundant — `ADR-010` decision 3 already states
that local work requires no hub at all, and Repo Manager already maintains a
file-derived local index (`index_store.py`, `rmgr cache status` / `cache
rebuild`). Two caching layers exist and one of them is a database pretending to
be the primary.

With the local instance gone, `state-hub-primary` binds `127.0.0.1:8000`
unchanged and every existing `http://127.0.0.1:8000` default becomes correct
without editing a single call site. The port collision cannot recur because only
one process binds the port.

Retire the reverse tunnels too. `state-hub-railiance01` forwards a remote box's
`:18000` back to this workstation's `:8000`, so a remote agent following the
documented port map reaches the workstation rather than the primary — which on
railiance01 is its own machine. That topology assumed the workstation was the
hub; it has not been since the primary moved.

Sequence matters: export the cache-only records first, stop serving second,
discard the cache data only after T05 proves central holds everything.

Acceptance: no local hub process listening; `127.0.0.1:8000` answers from
central; MCP `dev-hub` resolves to central; reverse `state-hub-*` tunnels removed
or repointed; the cache-only recovery export is committed.

**Progress (2026-08-24):** `docs/recovery/cache-only-repos-2026-08-24.json`
captures all 44 cache-only repository records with working-copy presence,
classification file presence, and HEAD sha — the recovery source for T05.
`ops-bridge` now pins local forwards to `127.0.0.1` (commit `2213847`), so a
contested port fails loudly instead of silently landing on `[::1]`.

The local hub instance is retired: `make api` (uvicorn on `127.0.0.1:8000`) is
stopped, `state-hub-primary` restarted onto the freed IPv4 address, and the
orphaned `[::1]` forward removed. Exactly one process now binds 8000 and it is
the tunnel to central — `/repos/` returns 78 there, not the cache's 122, and
`statehub status` reports central's 10 active workplans rather than 27. The MCP
server on `:8001` needed no change: it targets `http://127.0.0.1:8000` and now
proxies central. No call site was edited; retiring the impersonator made the
existing defaults correct.

The Postgres container is deliberately left running with its data intact. It is
the recovery source of last resort until T05 proves central holds all 44, per
the sequencing above.

**Remaining:** the reverse relay tunnels `state-hub-railiance01` and
`state-hub-mcp-railiance01` still route remote agents back to this workstation.
Retiring them requires editing `~/.config/bridge/tunnels.yaml`, which is outside
any repo and was blocked in-session; the change and its sequencing constraint
are recorded in `docs/recovery/tunnels-yaml-proposed-changes-CUST-WP-0067.md`.
Remote agents and the documented port map must be repointed *before* removal.

## Make the hub target explicit and unspoofable

```task
id: CUST-WP-0067-T03
status: todo
priority: high
state_hub_task_id: "29977448-1a74-5a4d-b729-974c15b6bbde"
```

Retiring the local instance removes today's impersonator but not the ability for
a future one to appear. Give the hub an instance identity it can assert — role
served from the health or summary endpoint — and make
`registrar-reconcile --confirm-primary` refuse anything that does not assert
`primary`.

`_check_primary` currently asserts only `status == ok` and `db == connected`.
Both instances passed it. It is a liveness check wearing an authority check's
name, and it is what allowed a cache to certify itself as the registrar.

Acceptance: `--confirm-primary` fails against a non-primary instance and names
what it reached; a hub reports its role; `statehub status` shows role alongside
the API base.

## Give Repo Manager a real onboarding write path

```task
id: CUST-WP-0067-T04
status: todo
priority: high
state_hub_task_id: "ffa141d5-331d-543d-8b87-516f27973a22"
```

Repo Manager owns `managed_repos` as `file-derived` in
`hub-record-authority.yaml` and exposes no command for it. Add onboarding that
follows `ADR-010` decision 5: write `.repo-classification.yaml` in the target
repository, commit, push, and have central derive the record. Central must not
accept a push of derived state, so the command makes the source file correct and
reachable, then triggers and verifies derivation.

Resolve the bootstrap gap explicitly — central derives from repositories it
already knows about, so a never-registered repository is never scanned. The
onboarding path must be able to introduce a repository central has not seen.

Acceptance: onboarding a fresh repository from the workstation produces a central
record with no manual step; re-running is idempotent.

## Onboard the 44 cache-only repositories to central

```task
id: CUST-WP-0067-T05
status: todo
priority: medium
state_hub_task_id: "078159e6-5ecd-5f9d-b3ec-1fabf60955f7"
```

Runs after T04. With the cache retired this is no longer a convergence of two
hubs — it is onboarding 44 repositories to central from their files, which is
T04 applied to the recovery export.

Ten of the 44 lack `.repo-classification.yaml` and need one authored with the
owner rather than guessed; classification is not mechanical, per
`CUST-WP-0065-T01`. Push `soul-frame` and `rein-openweights` first — both are
ahead of their remote, and central derives from what it can fetch.
`agent-harness` has no working copy: decide restore-from-remote or drop and
record it. Do not preserve it as a hub-only record.

Acceptance: every record in the recovery export exists on central or carries a
written disposition; only then may the local cache database be discarded.

## Correct the ADR-010 framing and record the retirement

```task
id: CUST-WP-0067-T06
status: todo
priority: medium
state_hub_task_id: "007bfcf1-3b17-5d4a-b16a-b80ebf273934"
```

`ADR-010` decision 2 says a divergent database is a merge problem and a stale
cache is a refresh problem. For `managed_repos` neither held: the gap was a
strict subset in the cache's favour, and refreshing would have destroyed rather
than reconciled. Record that third shape — cache-only records whose
authoritative source exists but was never introduced to central — with
re-derivation from source as its remedy.

Record the mechanical cause, which the ADR observed but did not diagnose: an
unbound `ssh -L` binds every loopback family, so the IPv4 bind losing to a local
listener still leaves a working `[::1]` forward and `ExitOnForwardFailure` never
fires. The ADR treated the shared port as the hazard; the missing bind address
was what made it silent.

Note also that ADR-010 reads as though its remediation landed. It did not — the
condition it measured was still live seven weeks later. Supersede decision 2 for
this record class and record the local-instance retirement as the outcome.

Acceptance: ADR-010 revised with a dated superseding note linked to this
workplan; `ops-bridge` and the port map documented as the structural fix.

## Repoint the fleet's copied port map

```task
id: CUST-WP-0067-T07
status: todo
priority: high
```

The remote port map is not configuration. It is prose copied into roughly 120
`AGENTS.md` files across both machines, with no template or generator behind it
— a grep for the block finds only the files themselves and past agent session
logs. That is how one wrong topology propagated fleet-wide and survived the
primary moving off the workstation.

Every copy still tells a remote agent to reach the hub at `127.0.0.1:18000`,
which forwards back to the workstation. The global agent instructions are
already repointed to the in-cluster address (2026-08-24); these copies are not,
and they outrank nothing but are read first in their own repos.

Two things are needed, and the second matters more than the first:

1. A scripted, reviewable repoint of the block across all affected repos on both
   machines — mechanical, but 120 repos times commit and push, so it needs a dry
   run and a diff sample reviewed before it runs.
2. A single source for the block, so the next topology change is one edit rather
   than 120. Repo Manager already owns repository representation and `scaffold`;
   this belongs there. Without it this task recurs verbatim.

Retiring `state-hub-railiance01` is blocked on this: the tunnel is what those
120 files currently depend on.

Acceptance: no `AGENTS.md` instructs a remote agent to use `127.0.0.1:18000`;
the block has one authoritative source; a spot check on both machines confirms
the rendered guidance matches the live topology.

## Deploy an MCP server on central

```task
id: CUST-WP-0067-T08
status: todo
priority: medium
```

The Helm chart deploys a single container serving HTTP on 8000. No MCP server
runs on central, so `dev-hub` for remote agents exists only as a reverse tunnel
to the workstation's MCP — which now proxies central, making the path
remote → workstation → central for a service on the remote's own machine.

The work is small: the same image already contains `mcp_server/server.py`, run
with `MCP_TRANSPORT=sse MCP_PORT=8001` and `API_BASE` pointed at the in-cluster
service. It needs a Deployment, a Service, and chart values.

Until this lands, remote agents should prefer the REST surface; the session
protocol documents a `curl` form for every operation, so this is a degraded
convenience rather than a blocker. Retiring `state-hub-mcp-railiance01` is
blocked on this task.

Acceptance: an MCP endpoint reachable from the node without traversing the
workstation; remote `dev-hub` registration points at it; the reverse MCP tunnel
removed.