the-custodian/workplans/CUST-WP-0067-hub-authority-target-resolution.md

id	type	title	domain	repo	status	owner	created	updated	quality_dor	quality_dor_at	quality_dor_by	quality_dor_note
CUST-WP-0067	workplan	Resolve hub target ambiguity and restore repo onboarding to the authoritative hub	infotech	the-custodian	active	codex	2026-08-24	2026-08-24	DoR-Ok	2026-08-24	codex	Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml.

Resolve hub target ambiguity and restore repo onboarding to the authoritative hub

Goal

Make "the authoritative hub" a resolvable address rather than a policy claim, so that repository onboarding lands on central instead of a local cache that believes it is primary. Then recover the seven weeks of workstation-originated repo registrations that never reached central, and correct the ADR-010 framing that treats this class of gap as a refresh problem.

Context

ADR-010 documented two State Hub instances sharing port 8000, separated only by IP family, and named the consequence: every tool defaulting to 127.0.0.1 reaches the local instance while believing it is the primary. That condition was never remediated. As of 2026-08-24 it is still live:

LISTEN 127.0.0.1:8000   uvicorn (pid 3258)             local cache
LISTEN     [::1]:8000   ssh -L 8000:10.43.68.154:8000  central

The collision is declared in ~/.config/bridge/tunnels.yaml, where the tunnel state-hub-primary takes local_port: 8000 — the port the local uvicorn already holds. ssh -L binds both address families, the IPv4 bind loses to uvicorn, and the IPv6 bind succeeds, so ExitOnForwardFailure=yes does not fire. _reject_duplicate_local_ports in ops-bridge guards only bridge tunnel against bridge tunnel; it cannot see a non-bridge listener.

Measured repository divergence, 2026-08-24:

cache (127.0.0.1:8000)     122
central ([::1]:8000)        78
central-only                 0
cache-only                  44

Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh of the cache would delete the 44 rather than reconcile them, and central holds no path to re-derive records for repositories it has never been told exist.

Onboarding by registration date shows a clean break:

Registered	central	cache-only
2026-02 – 2026-06	74	0
2026-07	1	24
2026-08	3	20

Central is otherwise live — last_state_synced_at runs to 2026-08-22, and the three August rows (repo-manager, rail-kubernetes, fin-hub) were registered from the railiance side. Only workstation-originated onboarding is lost.

Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have an origin remote and are pushed (soul-frame and rein-openweights are ahead of their remote), and 35 carry .repo-classification.yaml. agent-harness has no working copy and needs separate disposition.

hub-record-authority.yaml assigns managed_repos to repo-manager as file-derived. Repo Manager owns the record type but exposes no command that onboards a repository into a hub, and registrar-reconcile --confirm-primary defaults its "primary" to 127.0.0.1:8000 — confirming projections against the cache and reporting success while central holds nothing.

Repair the crashing status printer

id: CUST-WP-0067-T01
status: done
priority: low

statehub status raised KeyError: 'in_progress' at custodian_cli.py:565. The task vocabulary is wait|todo|progress|done|cancel; in_progress and blocked are not task statuses in this schema and the totals block never carried them. Fix the key names, tolerate the workplans/workstreams rename, and print the resolved API base so the operator can see which instance answered.

Done (2026-08-24): custodian_cli.py cmd_status now reads progress/todo/wait, accepts either totals key for workplans, uses .get() defaults throughout, and prints API base: as its second line. Verified against the live API.

Eliminate the port collision

id: CUST-WP-0067-T02
status: todo
priority: high

Move state-hub-primary off local_port: 8000 to 18000, matching the port map already reserved for the primary in the global agent instructions. Leave the local cache on 8000 so nothing that currently resolves to it changes behaviour — this step is deliberately non-breaking and must stay that way.

Extend the ops-bridge duplicate-port guard so it rejects a direction: local tunnel whose local_port is already bound by any listener, not only by another bridge tunnel. The current guard could not have caught this.

Acceptance: state-hub-primary binds 127.0.0.1:18000; [::1]:8000 no longer serves a hub; curl 127.0.0.1:18000/state/health reports the central instance and curl 127.0.0.1:8000/state/health reports the cache; the two return different repo counts; the guard fails a synthetic config that reintroduces the collision.

Make the hub target explicit and unspoofable

id: CUST-WP-0067-T03
status: todo
priority: high

Remove the http://127.0.0.1:8000 default from every call site that claims to reach the primary: custodian_cli.py:28, statehub_register.py:22, repo_manager/cli.py:56,405, repo_manager/commands/registrar_reconcile.py:410.

A default that silently resolves to a cache is worse than a missing one. Give the hub an identity it can assert — instance role served from the health or summary endpoint — and make registrar-reconcile --confirm-primary refuse to run against anything that does not assert primary. Confirming against a cache is a false green and is the specific failure that let this run for seven weeks.

Acceptance: --confirm-primary against the cache exits non-zero with a message naming the instance it reached; against central it succeeds; no code path reaches a hub without an explicitly resolved target.

Give Repo Manager a real onboarding write path

id: CUST-WP-0067-T04
status: todo
priority: high

Repo Manager owns managed_repos as file-derived and has no command for it. Add onboarding that follows ADR-010 decision 5: write .repo-classification.yaml in the target repository, commit, push, and have central derive the record. Central must not accept a push of derived state, so the command's job is to make the source file correct and reachable, then trigger and verify derivation.

Resolve the bootstrap gap explicitly — central derives from repositories it already knows about, so a never-registered repository is never scanned. The onboarding path must be able to introduce a repository central has not seen.

Acceptance: onboarding a fresh repository from the workstation produces a central record with no manual step; re-running is idempotent; the cache is not written directly.

Backfill the 44 cache-only registrations

id: CUST-WP-0067-T05
status: todo
priority: medium

Runs only after T02–T04; backfilling before the target is unambiguous refills the cache. Drive the 43 on-disk repositories through the T04 path. Nine lack .repo-classification.yaml (binky-control, clay-borg, direkt-vermittlung-de, polycode-sim, railiance-telemetry, ralph-workplan, rein-openweights, testdrive-jsui, timeline-svg) and need one authored with the owner rather than guessed — classification is not mechanical, per CUST-WP-0065-T01. Push soul-frame and rein-openweights first.

agent-harness has no working copy: decide restore-from-remote or drop, and record the decision. Do not preserve it as a hub-only record — that is the ADR-001 violation ADR-010 calls out.

Acceptance: central and cache repo counts converge; the cache-only set is empty or every remainder has a written disposition.

Correct the ADR-010 repo-record framing

id: CUST-WP-0067-T06
status: todo
priority: medium

ADR-010 decision 2 says a divergent database is a merge problem and a stale cache is a refresh problem. For managed_repos neither holds: the gap is a strict subset in the cache's favour, and refreshing destroys rather than reconciles. Record the third shape — cache-only records whose authoritative source exists but was never introduced to central — and state that its remedy is re-derivation from source, not refresh and not merge.

Also correct the implicit assumption that the ADR's own remediation happened. The measurement stands; the fix did not land, and the ADR reads as though it did.

Acceptance: ADR-010 revised with a superseding note dated and linked to this workplan; the port-collision remediation recorded as an outcome rather than an observation.