--- id: CUST-WP-0067 type: workplan title: "Resolve hub target ambiguity and restore repo onboarding to the authoritative hub" domain: infotech repo: the-custodian status: active owner: codex created: "2026-08-24" updated: "2026-08-24" quality_dor: DoR-Ok quality_dor_at: "2026-08-24" quality_dor_by: codex quality_dor_note: "Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml." --- # Resolve hub target ambiguity and restore repo onboarding to the authoritative hub ## Goal Make "the authoritative hub" a resolvable address rather than a policy claim, so that repository onboarding lands on central instead of a local cache that believes it is primary. Then recover the seven weeks of workstation-originated repo registrations that never reached central, and correct the ADR-010 framing that treats this class of gap as a refresh problem. ## Context `ADR-010` documented two State Hub instances sharing port 8000, separated only by IP family, and named the consequence: every tool defaulting to `127.0.0.1` reaches the local instance while believing it is the primary. That condition was never remediated. As of 2026-08-24 it is still live: ```text LISTEN 127.0.0.1:8000 uvicorn (pid 3258) local cache LISTEN [::1]:8000 ssh -L 8000:10.43.68.154:8000 central ``` The collision is declared in `~/.config/bridge/tunnels.yaml`, where the tunnel `state-hub-primary` takes `local_port: 8000` — the port the local uvicorn already holds. `ssh -L` binds both address families, the IPv4 bind loses to uvicorn, and the IPv6 bind succeeds, so `ExitOnForwardFailure=yes` does not fire. `_reject_duplicate_local_ports` in `ops-bridge` guards only bridge tunnel against bridge tunnel; it cannot see a non-bridge listener. Measured repository divergence, 2026-08-24: ```text cache (127.0.0.1:8000) 122 central ([::1]:8000) 78 central-only 0 cache-only 44 ``` Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh of the cache would delete the 44 rather than reconcile them, and central holds no path to re-derive records for repositories it has never been told exist. Onboarding by registration date shows a clean break: | Registered | central | cache-only | |---|---|---| | 2026-02 – 2026-06 | 74 | 0 | | 2026-07 | 1 | 24 | | 2026-08 | 3 | 20 | Central is otherwise live — `last_state_synced_at` runs to 2026-08-22, and the three August rows (`repo-manager`, `rail-kubernetes`, `fin-hub`) were registered from the railiance side. Only workstation-originated onboarding is lost. Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have an `origin` remote and are pushed (`soul-frame` and `rein-openweights` are ahead of their remote), and 35 carry `.repo-classification.yaml`. `agent-harness` has no working copy and needs separate disposition. `hub-record-authority.yaml` assigns `managed_repos` to `repo-manager` as `file-derived`. Repo Manager owns the record type but exposes no command that onboards a repository into a hub, and `registrar-reconcile --confirm-primary` defaults its "primary" to `127.0.0.1:8000` — confirming projections against the cache and reporting success while central holds nothing. ## Repair the crashing status printer ```task id: CUST-WP-0067-T01 status: done priority: low ``` `statehub status` raised `KeyError: 'in_progress'` at `custodian_cli.py:565`. The task vocabulary is `wait|todo|progress|done|cancel`; `in_progress` and `blocked` are not task statuses in this schema and the totals block never carried them. Fix the key names, tolerate the `workplans`/`workstreams` rename, and print the resolved API base so the operator can see which instance answered. **Done (2026-08-24):** `custodian_cli.py` `cmd_status` now reads `progress`/`todo`/`wait`, accepts either totals key for workplans, uses `.get()` defaults throughout, and prints `API base:` as its second line. Verified against the live API. ## Eliminate the port collision ```task id: CUST-WP-0067-T02 status: todo priority: high ``` Move `state-hub-primary` off `local_port: 8000` to `18000`, matching the port map already reserved for the primary in the global agent instructions. Leave the local cache on 8000 so nothing that currently resolves to it changes behaviour — this step is deliberately non-breaking and must stay that way. Extend the `ops-bridge` duplicate-port guard so it rejects a `direction: local` tunnel whose `local_port` is already bound by any listener, not only by another bridge tunnel. The current guard could not have caught this. Acceptance: `state-hub-primary` binds `127.0.0.1:18000`; `[::1]:8000` no longer serves a hub; `curl 127.0.0.1:18000/state/health` reports the central instance and `curl 127.0.0.1:8000/state/health` reports the cache; the two return different repo counts; the guard fails a synthetic config that reintroduces the collision. ## Make the hub target explicit and unspoofable ```task id: CUST-WP-0067-T03 status: todo priority: high ``` Remove the `http://127.0.0.1:8000` default from every call site that claims to reach the primary: `custodian_cli.py:28`, `statehub_register.py:22`, `repo_manager/cli.py:56,405`, `repo_manager/commands/registrar_reconcile.py:410`. A default that silently resolves to a cache is worse than a missing one. Give the hub an identity it can assert — instance role served from the health or summary endpoint — and make `registrar-reconcile --confirm-primary` refuse to run against anything that does not assert `primary`. Confirming against a cache is a false green and is the specific failure that let this run for seven weeks. Acceptance: `--confirm-primary` against the cache exits non-zero with a message naming the instance it reached; against central it succeeds; no code path reaches a hub without an explicitly resolved target. ## Give Repo Manager a real onboarding write path ```task id: CUST-WP-0067-T04 status: todo priority: high ``` Repo Manager owns `managed_repos` as `file-derived` and has no command for it. Add onboarding that follows ADR-010 decision 5: write `.repo-classification.yaml` in the target repository, commit, push, and have central derive the record. Central must not accept a push of derived state, so the command's job is to make the source file correct and reachable, then trigger and verify derivation. Resolve the bootstrap gap explicitly — central derives from repositories it already knows about, so a never-registered repository is never scanned. The onboarding path must be able to introduce a repository central has not seen. Acceptance: onboarding a fresh repository from the workstation produces a central record with no manual step; re-running is idempotent; the cache is not written directly. ## Backfill the 44 cache-only registrations ```task id: CUST-WP-0067-T05 status: todo priority: medium ``` Runs only after T02–T04; backfilling before the target is unambiguous refills the cache. Drive the 43 on-disk repositories through the T04 path. Nine lack `.repo-classification.yaml` (`binky-control`, `clay-borg`, `direkt-vermittlung-de`, `polycode-sim`, `railiance-telemetry`, `ralph-workplan`, `rein-openweights`, `testdrive-jsui`, `timeline-svg`) and need one authored with the owner rather than guessed — classification is not mechanical, per `CUST-WP-0065-T01`. Push `soul-frame` and `rein-openweights` first. `agent-harness` has no working copy: decide restore-from-remote or drop, and record the decision. Do not preserve it as a hub-only record — that is the ADR-001 violation ADR-010 calls out. Acceptance: central and cache repo counts converge; the cache-only set is empty or every remainder has a written disposition. ## Correct the ADR-010 repo-record framing ```task id: CUST-WP-0067-T06 status: todo priority: medium ``` ADR-010 decision 2 says a divergent database is a merge problem and a stale cache is a refresh problem. For `managed_repos` neither holds: the gap is a strict subset in the cache's favour, and refreshing destroys rather than reconciles. Record the third shape — cache-only records whose authoritative source exists but was never introduced to central — and state that its remedy is re-derivation from source, not refresh and not merge. Also correct the implicit assumption that the ADR's own remediation happened. The measurement stands; the fix did not land, and the ADR reads as though it did. Acceptance: ADR-010 revised with a superseding note dated and linked to this workplan; the port-collision remediation recorded as an outcome rather than an observation.