the-custodian/workplans/CUST-WP-0067-hub-authority-target-resolution.md

---
id: CUST-WP-0067
type: workplan
title: "Resolve hub target ambiguity and restore repo onboarding to the authoritative hub"
domain: infotech
repo: the-custodian
status: active
owner: codex
created: "2026-08-24"
updated: "2026-08-24"
quality_dor: DoR-Ok
quality_dor_at: "2026-08-24"
quality_dor_by: codex
quality_dor_note: "Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml."
state_hub_workstream_id: "16249302-2767-55df-aec0-d92c2751c225"
---

# Resolve hub target ambiguity and restore repo onboarding to the authoritative hub

## Goal

Make "the authoritative hub" a resolvable address rather than a policy claim, so
that repository onboarding lands on central instead of a local cache that
believes it is primary. Then recover the seven weeks of workstation-originated
repo registrations that never reached central, and correct the ADR-010 framing
that treats this class of gap as a refresh problem.

## Context

`ADR-010` documented two State Hub instances sharing port 8000, separated only
by IP family, and named the consequence: every tool defaulting to `127.0.0.1`
reaches the local instance while believing it is the primary. That condition was
never remediated. As of 2026-08-24 it is still live:

```text
LISTEN 127.0.0.1:8000   uvicorn (pid 3258)             local cache
LISTEN     [::1]:8000   ssh -L 8000:10.43.68.154:8000  central
```

The collision is declared in `~/.config/bridge/tunnels.yaml`, where the tunnel
`state-hub-primary` takes `local_port: 8000` — the port the local uvicorn
already holds. `ssh -L` binds both address families, the IPv4 bind loses to
uvicorn, and the IPv6 bind succeeds, so `ExitOnForwardFailure=yes` does not
fire. `_reject_duplicate_local_ports` in `ops-bridge` guards only bridge tunnel
against bridge tunnel; it cannot see a non-bridge listener.

Measured repository divergence, 2026-08-24:

```text
cache (127.0.0.1:8000)     122
central ([::1]:8000)        78
central-only                 0
cache-only                  44
```

Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh
of the cache would delete the 44 rather than reconcile them, and central holds
no path to re-derive records for repositories it has never been told exist.

Onboarding by registration date shows a clean break:

| Registered | central | cache-only |
|---|---|---|
| 2026-02 – 2026-06 | 74 | 0 |
| 2026-07 | 1 | 24 |
| 2026-08 | 3 | 20 |

Central is otherwise live — `last_state_synced_at` runs to 2026-08-22, and the
three August rows (`repo-manager`, `rail-kubernetes`, `fin-hub`) were registered
from the railiance side. Only workstation-originated onboarding is lost.

Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have
an `origin` remote and are pushed (`soul-frame` and `rein-openweights` are ahead
of their remote), and 35 carry `.repo-classification.yaml`. `agent-harness` has
no working copy and needs separate disposition.

`hub-record-authority.yaml` assigns `managed_repos` to `repo-manager` as
`file-derived`. Repo Manager owns the record type but exposes no command that
onboards a repository into a hub, and `registrar-reconcile --confirm-primary`
defaults its "primary" to `127.0.0.1:8000` — confirming projections against the
cache and reporting success while central holds nothing.

## Repair the crashing status printer

```task
id: CUST-WP-0067-T01
status: done
priority: low
state_hub_task_id: "f3608db4-20a5-58fb-a965-885eb14858af"
```

`statehub status` raised `KeyError: 'in_progress'` at `custodian_cli.py:565`.
The task vocabulary is `wait|todo|progress|done|cancel`; `in_progress` and
`blocked` are not task statuses in this schema and the totals block never
carried them. Fix the key names, tolerate the `workplans`/`workstreams` rename,
and print the resolved API base so the operator can see which instance answered.

**Done (2026-08-24):** `custodian_cli.py` `cmd_status` now reads
`progress`/`todo`/`wait`, accepts either totals key for workplans, uses `.get()`
defaults throughout, and prints `API base:` as its second line. Verified against
the live API.

## Retire the local hub instance and the reverse-tunnel relay

```task
id: CUST-WP-0067-T02
status: progress
priority: high
state_hub_task_id: "4093e928-d752-5a91-96c3-2e80f0e1dac5"
```

Decision, 2026-08-24: rather than making two hub instances coexist safely,
retire the second one. The local `postgres:16-alpine` + uvicorn instance is what
impersonates central, and it is redundant — `ADR-010` decision 3 already states
that local work requires no hub at all, and Repo Manager already maintains a
file-derived local index (`index_store.py`, `rmgr cache status` / `cache
rebuild`). Two caching layers exist and one of them is a database pretending to
be the primary.

With the local instance gone, `state-hub-primary` binds `127.0.0.1:8000`
unchanged and every existing `http://127.0.0.1:8000` default becomes correct
without editing a single call site. The port collision cannot recur because only
one process binds the port.

Retire the reverse tunnels too. `state-hub-railiance01` forwards a remote box's
`:18000` back to this workstation's `:8000`, so a remote agent following the
documented port map reaches the workstation rather than the primary — which on
railiance01 is its own machine. That topology assumed the workstation was the
hub; it has not been since the primary moved.

Sequence matters: export the cache-only records first, stop serving second,
discard the cache data only after T05 proves central holds everything.

Acceptance: no local hub process listening; `127.0.0.1:8000` answers from
central; MCP `dev-hub` resolves to central; reverse `state-hub-*` tunnels removed
or repointed; the cache-only recovery export is committed.

**Done (2026-08-24):** `docs/recovery/cache-only-repos-2026-08-24.json` captures
all 44 cache-only repository records with working-copy presence, classification
file presence, and HEAD sha — the recovery source for T05.
`ops-bridge` now pins local forwards to `127.0.0.1` (commit `2213847`), so a
contested port fails loudly instead of silently landing on `[::1]`.

## Make the hub target explicit and unspoofable

```task
id: CUST-WP-0067-T03
status: todo
priority: high
state_hub_task_id: "29977448-1a74-5a4d-b729-974c15b6bbde"
```

Retiring the local instance removes today's impersonator but not the ability for
a future one to appear. Give the hub an instance identity it can assert — role
served from the health or summary endpoint — and make
`registrar-reconcile --confirm-primary` refuse anything that does not assert
`primary`.

`_check_primary` currently asserts only `status == ok` and `db == connected`.
Both instances passed it. It is a liveness check wearing an authority check's
name, and it is what allowed a cache to certify itself as the registrar.

Acceptance: `--confirm-primary` fails against a non-primary instance and names
what it reached; a hub reports its role; `statehub status` shows role alongside
the API base.

## Give Repo Manager a real onboarding write path

```task
id: CUST-WP-0067-T04
status: todo
priority: high
state_hub_task_id: "ffa141d5-331d-543d-8b87-516f27973a22"
```

Repo Manager owns `managed_repos` as `file-derived` in
`hub-record-authority.yaml` and exposes no command for it. Add onboarding that
follows `ADR-010` decision 5: write `.repo-classification.yaml` in the target
repository, commit, push, and have central derive the record. Central must not
accept a push of derived state, so the command makes the source file correct and
reachable, then triggers and verifies derivation.

Resolve the bootstrap gap explicitly — central derives from repositories it
already knows about, so a never-registered repository is never scanned. The
onboarding path must be able to introduce a repository central has not seen.

Acceptance: onboarding a fresh repository from the workstation produces a central
record with no manual step; re-running is idempotent.

## Onboard the 44 cache-only repositories to central

```task
id: CUST-WP-0067-T05
status: todo
priority: medium
state_hub_task_id: "078159e6-5ecd-5f9d-b3ec-1fabf60955f7"
```

Runs after T04. With the cache retired this is no longer a convergence of two
hubs — it is onboarding 44 repositories to central from their files, which is
T04 applied to the recovery export.

Ten of the 44 lack `.repo-classification.yaml` and need one authored with the
owner rather than guessed; classification is not mechanical, per
`CUST-WP-0065-T01`. Push `soul-frame` and `rein-openweights` first — both are
ahead of their remote, and central derives from what it can fetch.
`agent-harness` has no working copy: decide restore-from-remote or drop and
record it. Do not preserve it as a hub-only record.

Acceptance: every record in the recovery export exists on central or carries a
written disposition; only then may the local cache database be discarded.

## Correct the ADR-010 framing and record the retirement

```task
id: CUST-WP-0067-T06
status: todo
priority: medium
state_hub_task_id: "007bfcf1-3b17-5d4a-b16a-b80ebf273934"
```

`ADR-010` decision 2 says a divergent database is a merge problem and a stale
cache is a refresh problem. For `managed_repos` neither held: the gap was a
strict subset in the cache's favour, and refreshing would have destroyed rather
than reconciled. Record that third shape — cache-only records whose
authoritative source exists but was never introduced to central — with
re-derivation from source as its remedy.

Record the mechanical cause, which the ADR observed but did not diagnose: an
unbound `ssh -L` binds every loopback family, so the IPv4 bind losing to a local
listener still leaves a working `[::1]` forward and `ExitOnForwardFailure` never
fires. The ADR treated the shared port as the hazard; the missing bind address
was what made it silent.

Note also that ADR-010 reads as though its remediation landed. It did not — the
condition it measured was still live seven weeks later. Supersede decision 2 for
this record class and record the local-instance retirement as the outcome.

Acceptance: ADR-010 revised with a dated superseding note linked to this
workplan; `ops-bridge` and the port map documented as the structural fix.