All 44 cache-only repository records resolved: 43 onboarded and classified, agent-harness dropped by owner decision. Central holds 121 repositories. Corrects an earlier claim in this workplan that the cache would hold nothing unique once the repository records were resolved. That generalised a repository-scoped result: the cache still holds ~400 workplans and ~1,950 tasks that exist nowhere else, which is ADR-010's original divergence. The cache database still must not be discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
679 lines
32 KiB
Markdown
679 lines
32 KiB
Markdown
---
|
||
id: CUST-WP-0067
|
||
type: workplan
|
||
title: "Resolve hub target ambiguity and restore repo onboarding to the authoritative hub"
|
||
domain: infotech
|
||
repo: the-custodian
|
||
status: active
|
||
owner: codex
|
||
created: "2026-08-24"
|
||
updated: "2026-08-24"
|
||
quality_dor: DoR-Ok
|
||
quality_dor_at: "2026-08-24"
|
||
quality_dor_by: codex
|
||
quality_dor_note: "Live divergence measured (122 cache / 78 central, strict subset, 0 central-only); root cause located in configuration (port collision plus 127.0.0.1 defaults in four call sites); recovery source confirmed on disk and pushed for 43 of 44 records; owner boundaries follow ADR-010 and hub-record-authority.yaml."
|
||
state_hub_workstream_id: "16249302-2767-55df-aec0-d92c2751c225"
|
||
---
|
||
|
||
# Resolve hub target ambiguity and restore repo onboarding to the authoritative hub
|
||
|
||
## Goal
|
||
|
||
Make "the authoritative hub" a resolvable address rather than a policy claim, so
|
||
that repository onboarding lands on central instead of a local cache that
|
||
believes it is primary. Then recover the seven weeks of workstation-originated
|
||
repo registrations that never reached central, and correct the ADR-010 framing
|
||
that treats this class of gap as a refresh problem.
|
||
|
||
## Context
|
||
|
||
`ADR-010` documented two State Hub instances sharing port 8000, separated only
|
||
by IP family, and named the consequence: every tool defaulting to `127.0.0.1`
|
||
reaches the local instance while believing it is the primary. That condition was
|
||
never remediated. As of 2026-08-24 it is still live:
|
||
|
||
```text
|
||
LISTEN 127.0.0.1:8000 uvicorn (pid 3258) local cache
|
||
LISTEN [::1]:8000 ssh -L 8000:10.43.68.154:8000 central
|
||
```
|
||
|
||
The collision is declared in `~/.config/bridge/tunnels.yaml`, where the tunnel
|
||
`state-hub-primary` takes `local_port: 8000` — the port the local uvicorn
|
||
already holds. `ssh -L` binds both address families, the IPv4 bind loses to
|
||
uvicorn, and the IPv6 bind succeeds, so `ExitOnForwardFailure=yes` does not
|
||
fire. `_reject_duplicate_local_ports` in `ops-bridge` guards only bridge tunnel
|
||
against bridge tunnel; it cannot see a non-bridge listener.
|
||
|
||
Measured repository divergence, 2026-08-24:
|
||
|
||
```text
|
||
cache (127.0.0.1:8000) 122
|
||
central ([::1]:8000) 78
|
||
central-only 0
|
||
cache-only 44
|
||
```
|
||
|
||
Central is a strict subset. That is not the shape ADR-010 anticipated. A refresh
|
||
of the cache would delete the 44 rather than reconcile them, and central holds
|
||
no path to re-derive records for repositories it has never been told exist.
|
||
|
||
Onboarding by registration date shows a clean break:
|
||
|
||
| Registered | central | cache-only |
|
||
|---|---|---|
|
||
| 2026-02 – 2026-06 | 74 | 0 |
|
||
| 2026-07 | 1 | 24 |
|
||
| 2026-08 | 3 | 20 |
|
||
|
||
Central is otherwise live — `last_state_synced_at` runs to 2026-08-22, and the
|
||
three August rows (`repo-manager`, `rail-kubernetes`, `fin-hub`) were registered
|
||
from the railiance side. Only workstation-originated onboarding is lost.
|
||
|
||
Nothing is unrecoverable: 43 of the 44 have a working copy on disk, all 43 have
|
||
an `origin` remote and are pushed (`soul-frame` and `rein-openweights` are ahead
|
||
of their remote), and 35 carry `.repo-classification.yaml`. `agent-harness` has
|
||
no working copy and needs separate disposition.
|
||
|
||
`hub-record-authority.yaml` assigns `managed_repos` to `repo-manager` as
|
||
`file-derived`. Repo Manager owns the record type but exposes no command that
|
||
onboards a repository into a hub, and `registrar-reconcile --confirm-primary`
|
||
defaults its "primary" to `127.0.0.1:8000` — confirming projections against the
|
||
cache and reporting success while central holds nothing.
|
||
|
||
## Repair the crashing status printer
|
||
|
||
```task
|
||
id: CUST-WP-0067-T01
|
||
status: done
|
||
priority: low
|
||
state_hub_task_id: "f3608db4-20a5-58fb-a965-885eb14858af"
|
||
```
|
||
|
||
`statehub status` raised `KeyError: 'in_progress'` at `custodian_cli.py:565`.
|
||
The task vocabulary is `wait|todo|progress|done|cancel`; `in_progress` and
|
||
`blocked` are not task statuses in this schema and the totals block never
|
||
carried them. Fix the key names, tolerate the `workplans`/`workstreams` rename,
|
||
and print the resolved API base so the operator can see which instance answered.
|
||
|
||
**Done (2026-08-24):** `custodian_cli.py` `cmd_status` now reads
|
||
`progress`/`todo`/`wait`, accepts either totals key for workplans, uses `.get()`
|
||
defaults throughout, and prints `API base:` as its second line. Verified against
|
||
the live API.
|
||
|
||
## Retire the local hub instance and the reverse-tunnel relay
|
||
|
||
```task
|
||
id: CUST-WP-0067-T02
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "4093e928-d752-5a91-96c3-2e80f0e1dac5"
|
||
```
|
||
|
||
Decision, 2026-08-24: rather than making two hub instances coexist safely,
|
||
retire the second one. The local `postgres:16-alpine` + uvicorn instance is what
|
||
impersonates central, and it is redundant — `ADR-010` decision 3 already states
|
||
that local work requires no hub at all, and Repo Manager already maintains a
|
||
file-derived local index (`index_store.py`, `rmgr cache status` / `cache
|
||
rebuild`). Two caching layers exist and one of them is a database pretending to
|
||
be the primary.
|
||
|
||
With the local instance gone, `state-hub-primary` binds `127.0.0.1:8000`
|
||
unchanged and every existing `http://127.0.0.1:8000` default becomes correct
|
||
without editing a single call site. The port collision cannot recur because only
|
||
one process binds the port.
|
||
|
||
Retire the reverse tunnels too. `state-hub-railiance01` forwards a remote box's
|
||
`:18000` back to this workstation's `:8000`, so a remote agent following the
|
||
documented port map reaches the workstation rather than the primary — which on
|
||
railiance01 is its own machine. That topology assumed the workstation was the
|
||
hub; it has not been since the primary moved.
|
||
|
||
Sequence matters: export the cache-only records first, stop serving second,
|
||
discard the cache data only after T05 proves central holds everything.
|
||
|
||
Acceptance: no local hub process listening; `127.0.0.1:8000` answers from
|
||
central; MCP `dev-hub` resolves to central; reverse `state-hub-*` tunnels removed
|
||
or repointed; the cache-only recovery export is committed.
|
||
|
||
**Progress (2026-08-24):** `docs/recovery/cache-only-repos-2026-08-24.json`
|
||
captures all 44 cache-only repository records with working-copy presence,
|
||
classification file presence, and HEAD sha — the recovery source for T05.
|
||
`ops-bridge` now pins local forwards to `127.0.0.1` (commit `2213847`), so a
|
||
contested port fails loudly instead of silently landing on `[::1]`.
|
||
|
||
The local hub instance is retired: `make api` (uvicorn on `127.0.0.1:8000`) is
|
||
stopped, `state-hub-primary` restarted onto the freed IPv4 address, and the
|
||
orphaned `[::1]` forward removed. Exactly one process now binds 8000 and it is
|
||
the tunnel to central — `/repos/` returns 78 there, not the cache's 122, and
|
||
`statehub status` reports central's 10 active workplans rather than 27. The MCP
|
||
server on `:8001` needed no change: it targets `http://127.0.0.1:8000` and now
|
||
proxies central. No call site was edited; retiring the impersonator made the
|
||
existing defaults correct.
|
||
|
||
The Postgres container is deliberately left running with its data intact. It is
|
||
the recovery source of last resort until T05 proves central holds all 44, per
|
||
the sequencing above.
|
||
|
||
**Remaining:** the reverse relay tunnels `state-hub-railiance01` and
|
||
`state-hub-mcp-railiance01` still route remote agents back to this workstation.
|
||
Retiring them requires editing `~/.config/bridge/tunnels.yaml`, which is outside
|
||
any repo and was blocked in-session; the change and its sequencing constraint
|
||
are recorded in `docs/recovery/tunnels-yaml-proposed-changes-CUST-WP-0067.md`.
|
||
Remote agents and the documented port map must be repointed *before* removal.
|
||
|
||
## Make the hub target explicit and unspoofable
|
||
|
||
```task
|
||
id: CUST-WP-0067-T03
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "29977448-1a74-5a4d-b729-974c15b6bbde"
|
||
```
|
||
|
||
Retiring the local instance removes today's impersonator but not the ability for
|
||
a future one to appear. Give the hub an instance identity it can assert — role
|
||
served from the health or summary endpoint — and make
|
||
`registrar-reconcile --confirm-primary` refuse anything that does not assert
|
||
`primary`.
|
||
|
||
`_check_primary` currently asserts only `status == ok` and `db == connected`.
|
||
Both instances passed it. It is a liveness check wearing an authority check's
|
||
name, and it is what allowed a cache to certify itself as the registrar.
|
||
|
||
Acceptance: `--confirm-primary` fails against a non-primary instance and names
|
||
what it reached; a hub reports its role; `statehub status` shows role alongside
|
||
the API base.
|
||
|
||
**Built (2026-08-25).** The hub can now declare what it is: `instance_role` and
|
||
`instance_label` in settings, surfaced on `/state/health`, defaulting to
|
||
`unknown` — an instance that has not declared itself is not the primary.
|
||
Production values declare `primary` / `railiance01`; the chart default stays
|
||
`unknown` because a chart can be installed anywhere.
|
||
|
||
`_check_primary` now requires `instance_role=primary`. An instance declaring
|
||
nothing is refused with a message naming what to set, and proceeding anyway
|
||
requires an explicit `--allow-unverified-primary` rather than a silent default —
|
||
the same principle as `--api-base` having no default.
|
||
|
||
Verified against the live hub, not only in tests: it refuses central today
|
||
(which declares nothing yet) and accepts with the explicit flag. Four tests
|
||
cover `_check_primary` directly; the existing suite monkeypatched it and never
|
||
exercised it, which is why it never caught the original defect. repo-manager
|
||
126 passed, state-hub 656 passed.
|
||
|
||
Worth recording: `registrar-reconcile` returns `noop` before reaching the
|
||
authority check when there is nothing to write. That is correct — no write, no
|
||
authority needed — but it means a live test on a clean repo proves nothing
|
||
about the guard.
|
||
|
||
**Defect found in this task's own work (2026-08-25).** The first deploy landed
|
||
and central still reported `unknown` while its ConfigMap correctly held
|
||
`primary`. pydantic-settings derives the env var from the *field name*, so a
|
||
field named `instance_role` binds `INSTANCE_ROLE` and silently ignored the
|
||
chart's `STATE_HUB_INSTANCE_ROLE`. The value reached the pod and was discarded.
|
||
|
||
That is the same failure this workplan exists to close — configuration declared
|
||
but never reaching what it configures — reintroduced while building the guard
|
||
against it. The verification was too weak in a way this session had already
|
||
learned twice: `helm template` rendering the key was treated as evidence it
|
||
bound, exactly as `Running` was nearly treated as serving and an open port as an
|
||
MCP handshake.
|
||
|
||
Fixed by renaming the fields to `state_hub_instance_role` /
|
||
`state_hub_instance_label`, matching the existing `state_hub_report_dir`
|
||
precedent so the env var the chart already sets is the one that binds.
|
||
`tests/test_instance_identity.py` now asserts the **env var name itself** binds,
|
||
which is the check that would have caught this before deploy, plus the `unknown`
|
||
default and rejection of invalid roles.
|
||
|
||
**Done (2026-08-25).** Deployed and verified end to end. Central reports
|
||
`instance_role=primary`, `instance_label=railiance01`; `statehub status` prints
|
||
`Instance: primary (railiance01)`; and `--confirm-primary` now accepts it with
|
||
no escape hatch, because authority is proven rather than assumed.
|
||
|
||
The guard is what closes this workplan's failure mode rather than merely
|
||
cleaning up after it: a future second instance cannot pass itself off as the
|
||
primary by being healthy.
|
||
|
||
## Give Repo Manager a real onboarding write path
|
||
|
||
```task
|
||
id: CUST-WP-0067-T04
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "ffa141d5-331d-543d-8b87-516f27973a22"
|
||
```
|
||
|
||
Repo Manager owns `managed_repos` as `file-derived` in
|
||
`hub-record-authority.yaml` and exposes no command for it. Add onboarding that
|
||
follows `ADR-010` decision 5: write `.repo-classification.yaml` in the target
|
||
repository, commit, push, and have central derive the record. Central must not
|
||
accept a push of derived state, so the command makes the source file correct and
|
||
reachable, then triggers and verifies derivation.
|
||
|
||
Resolve the bootstrap gap explicitly — central derives from repositories it
|
||
already knows about, so a never-registered repository is never scanned. The
|
||
onboarding path must be able to introduce a repository central has not seen.
|
||
|
||
Acceptance: onboarding a fresh repository from the workstation produces a central
|
||
record with no manual step; re-running is idempotent.
|
||
|
||
**Done (2026-08-24):** `rmgr repo-onboard` (repo-manager `1be6d85`). Validates
|
||
the classification file, then refuses to proceed if it has uncommitted changes,
|
||
no upstream, or unpushed commits — a hub record whose backing file is only local
|
||
cannot be re-derived by anyone. `--api-base` has no default on purpose; a silent
|
||
localhost default is the original defect, not a convenience. Six guard tests;
|
||
full suite 122 passed.
|
||
|
||
This uncovered why onboarding only ever worked from a workstation, which is
|
||
deeper than the tunnel: `api/classification.py` resolved its allowed-values file
|
||
from three hardcoded developer checkout paths and nothing else, so inside the
|
||
container every classification write returned 500. Fixed with
|
||
`REPO_CLASSIFICATION_ALLOWED_PATH` (state-hub `05dda8d`); delivering the canon
|
||
file into the pod is `T09`.
|
||
|
||
## Onboard the 44 cache-only repositories to central
|
||
|
||
```task
|
||
id: CUST-WP-0067-T05
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "078159e6-5ecd-5f9d-b3ec-1fabf60955f7"
|
||
```
|
||
|
||
Runs after T04. With the cache retired this is no longer a convergence of two
|
||
hubs — it is onboarding 44 repositories to central from their files, which is
|
||
T04 applied to the recovery export.
|
||
|
||
Ten of the 44 lack `.repo-classification.yaml` and need one authored with the
|
||
owner rather than guessed; classification is not mechanical, per
|
||
`CUST-WP-0065-T01`. Push `soul-frame` and `rein-openweights` first — both are
|
||
ahead of their remote, and central derives from what it can fetch.
|
||
`agent-harness` has no working copy: decide restore-from-remote or drop and
|
||
record it. Do not preserve it as a hub-only record.
|
||
|
||
Acceptance: every record in the recovery export exists on central or carries a
|
||
written disposition; only then may the local cache database be discarded.
|
||
|
||
**Done (2026-08-24):** 33 of 44 onboarded to central, which went from 78 to 111
|
||
repositories. The remaining 11 each carry a written disposition in
|
||
`docs/recovery/onboarding-disposition-2026-08-24.md`: nine need a classification
|
||
file authored with their owner, `soul-frame` has unpushed research commits that
|
||
are the owner's to publish, and `agent-harness` was **dropped on 2026-08-25** by owner decision — superseded by
|
||
`glas-harness` and the `rein` concept, both verified present and classified on
|
||
central first. Ten records remain outstanding.
|
||
|
||
None were forced. Classification is not mechanical (`CUST-WP-0065-T01`), and a
|
||
confident wrong domain is harder to correct than a missing one.
|
||
|
||
All 33 landed with their domain but no classification fields, pending `T09`.
|
||
They are re-derivable: the classification files are committed and pushed, so
|
||
re-running onboarding projects them once the hub can validate.
|
||
|
||
**Closed 2026-08-25.** All 44 resolved: 43 onboarded to central and classified,
|
||
`agent-harness` dropped by owner decision. `soul-frame`'s research commits were
|
||
pushed by the owner; the nine repositories lacking a classification file had
|
||
their sector and category decided one at a time with the owner, validated
|
||
against canon, committed, pushed, and onboarded. Central holds 121 repositories.
|
||
|
||
**The cache database still must not be discarded, for a different reason.**
|
||
Repository records are fully recovered, but they were only this task's scope.
|
||
Measured the same day: 1075 workplans and 6073 tasks in the cache against 669
|
||
and 4123 on central — roughly 400 workplans and 1,950 tasks that exist only
|
||
there. That is `ADR-010`'s original divergence, tracked as `T10`. An earlier
|
||
note in this workplan said the cache would hold nothing unique once the 11 were
|
||
resolved; that was wrong, because it generalised a repository-scoped result to
|
||
every record type.
|
||
|
||
## Correct the ADR-010 framing and record the retirement
|
||
|
||
```task
|
||
id: CUST-WP-0067-T06
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "007bfcf1-3b17-5d4a-b16a-b80ebf273934"
|
||
```
|
||
|
||
`ADR-010` decision 2 says a divergent database is a merge problem and a stale
|
||
cache is a refresh problem. For `managed_repos` neither held: the gap was a
|
||
strict subset in the cache's favour, and refreshing would have destroyed rather
|
||
than reconciled. Record that third shape — cache-only records whose
|
||
authoritative source exists but was never introduced to central — with
|
||
re-derivation from source as its remedy.
|
||
|
||
Record the mechanical cause, which the ADR observed but did not diagnose: an
|
||
unbound `ssh -L` binds every loopback family, so the IPv4 bind losing to a local
|
||
listener still leaves a working `[::1]` forward and `ExitOnForwardFailure` never
|
||
fires. The ADR treated the shared port as the hazard; the missing bind address
|
||
was what made it silent.
|
||
|
||
Note also that ADR-010 reads as though its remediation landed. It did not — the
|
||
condition it measured was still live seven weeks later. Supersede decision 2 for
|
||
this record class and record the local-instance retirement as the outcome.
|
||
|
||
Acceptance: ADR-010 revised with a dated superseding note linked to this
|
||
workplan; `ops-bridge` and the port map documented as the structural fix.
|
||
|
||
**Done (2026-08-24):** three dated corrections added to `ADR-010`, following the
|
||
in-file blockquote convention `ADR-007` already uses.
|
||
|
||
- **Decision 2** — the merge/refresh dichotomy is not exhaustive. Records the
|
||
third shape: cache-only records whose authoritative source exists and is
|
||
reachable but was never introduced to central, remedied by re-derivation
|
||
rather than refresh or merge. Refreshing would have destroyed the 44.
|
||
- **Context** — the ADR recorded the shared port but not why it was silent,
|
||
which is the part that let it persist for seven weeks. An unbound `ssh -L`
|
||
binds every loopback family and `ExitOnForwardFailure` fires only when all
|
||
binds fail, so the tunnel reported success while serving an address nothing
|
||
resolved to by default.
|
||
- **Outcome section** — states plainly that the ADR's own remediation never
|
||
landed, and that the condition it measured was still live seven weeks later.
|
||
An ADR describing a fix in the present tense reads as a fix that happened.
|
||
|
||
Also records the deeper cause as a second instance of the ADR's own theme:
|
||
the hub resolved its classification vocabulary from developer-workstation paths,
|
||
so authority depended on where a process happened to run.
|
||
|
||
## Repoint the fleet's copied port map
|
||
|
||
```task
|
||
id: CUST-WP-0067-T07
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "dbc672d5-805a-567a-b548-4e8870e4fb2f"
|
||
```
|
||
|
||
The remote port map is not configuration. It is prose copied into roughly 120
|
||
`AGENTS.md` files across both machines, with no template or generator behind it
|
||
— a grep for the block finds only the files themselves and past agent session
|
||
logs. That is how one wrong topology propagated fleet-wide and survived the
|
||
primary moving off the workstation.
|
||
|
||
Every copy still tells a remote agent to reach the hub at `127.0.0.1:18000`,
|
||
which forwards back to the workstation. The global agent instructions are
|
||
already repointed to the in-cluster address (2026-08-24); these copies are not,
|
||
and they outrank nothing but are read first in their own repos.
|
||
|
||
Two things are needed, and the second matters more than the first:
|
||
|
||
1. A scripted, reviewable repoint of the block across all affected repos on both
|
||
machines — mechanical, but 120 repos times commit and push, so it needs a dry
|
||
run and a diff sample reviewed before it runs.
|
||
2. A single source for the block, so the next topology change is one edit rather
|
||
than 120. Repo Manager already owns repository representation and `scaffold`;
|
||
this belongs there. Without it this task recurs verbatim.
|
||
|
||
Retiring `state-hub-railiance01` is blocked on this: the tunnel is what those
|
||
120 files currently depend on.
|
||
|
||
Acceptance: no `AGENTS.md` instructs a remote agent to use `127.0.0.1:18000`;
|
||
the block has one authoritative source; a spot check on both machines confirms
|
||
the rendered guidance matches the live topology.
|
||
|
||
**Files repointed (2026-08-25).** Both machines now report zero stale rows.
|
||
178 files, one row each:
|
||
|
||
- **Workstation, 100 files** — all committed and pushed, `AGENTS.md` staged
|
||
alone so unrelated working-tree state was never swept into a commit.
|
||
- **Node, 78 files** — 74 picked the change up by `git pull` since they are
|
||
clones of the same origins. Four needed individual handling.
|
||
|
||
The block was near-uniform: 99 exact matches and one variant in
|
||
`whitehat-security` ("Remote tunnel", different table rule), which the pattern
|
||
covered. No file matched `18000` without being rewritten, so there was no
|
||
silent-skip tail. The `Local workstation` row was left alone — since `T02` it
|
||
is correct, because `127.0.0.1:8000` now *is* central.
|
||
|
||
Individual cases on the node:
|
||
|
||
| Repository | Condition | Action |
|
||
|---|---|---|
|
||
| `the-custodian` | untracked files only | pulled |
|
||
| `inter-hub` | origin never had the fix | edited, committed, pushed |
|
||
| `activity-core` | 39 behind, unrelated tracked file dirty | took `AGENTS.md` from origin only; pre-existing condition left for its owner |
|
||
| `binky-control` | 6 pre-existing unpushed commits | committed, **not pushed** — publishing someone else's work is not a side effect of a docs repoint |
|
||
|
||
**Remaining — the half that stops this recurring.** The block still has no single
|
||
source; it was copy-propagated by past sessions, and a grep for it finds only
|
||
the files themselves and old session logs. Until Repo Manager renders it (it
|
||
already owns `scaffold`), the next topology change is another 178-file pass.
|
||
|
||
**Also remaining, deliberately separate:** 68 non-`AGENTS.md` files still
|
||
reference `127.0.0.1:18000`, and they must not be blanket-rewritten —
|
||
21 scripts and code paths that actually configure things (`register-mcp.sh`,
|
||
`dev_hub_up.sh`), 21 docs and runbooks, and 26 workplans, reports and snapshots
|
||
that are historical record. Rewriting a finished workplan or a dated report
|
||
would falsify what was true at the time.
|
||
|
||
`state-hub-railiance01` is now unblocked for removal: nothing documented still
|
||
points at it.
|
||
|
||
## Deploy an MCP server on central
|
||
|
||
```task
|
||
id: CUST-WP-0067-T08
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "b17505c0-ea13-5ee2-87ca-8c06607f3b5b"
|
||
```
|
||
|
||
The Helm chart deploys a single container serving HTTP on 8000. No MCP server
|
||
runs on central, so `dev-hub` for remote agents exists only as a reverse tunnel
|
||
to the workstation's MCP — which now proxies central, making the path
|
||
remote → workstation → central for a service on the remote's own machine.
|
||
|
||
The work is small: the same image already contains `mcp_server/server.py`, run
|
||
with `MCP_TRANSPORT=sse MCP_PORT=8001` and `API_BASE` pointed at the in-cluster
|
||
service. It needs a Deployment, a Service, and chart values.
|
||
|
||
Until this lands, remote agents should prefer the REST surface; the session
|
||
protocol documents a `curl` form for every operation, so this is a degraded
|
||
convenience rather than a blocker. Retiring `state-hub-mcp-railiance01` is
|
||
blocked on this task.
|
||
|
||
Acceptance: an MCP endpoint reachable from the node without traversing the
|
||
workstation; remote `dev-hub` registration points at it; the reverse MCP tunnel
|
||
removed.
|
||
|
||
**Progress (2026-08-24):** chart support is written and validated — a gated
|
||
`mcp` Deployment and ClusterIP Service running the same image with
|
||
`-m mcp_server.server`, `API_BASE` defaulted to the in-cluster API Service, and
|
||
tcpSocket probes. `helm lint` and `helm template` are clean, and the templates
|
||
render nothing when `mcp.enabled` is false (the default).
|
||
|
||
Two things had to be fixed before the YAML would have worked:
|
||
|
||
- `mcp_server/server.py` hardcoded `host="127.0.0.1"`. A Service routes to the
|
||
pod IP, so a loopback bind is unreachable from anywhere but the pod itself.
|
||
Now `MCP_HOST`, still defaulting to loopback — the MCP layer proxies an
|
||
unauthenticated API and must not land on every interface by accident.
|
||
- The container command uses `-m mcp_server.server` rather than the file path,
|
||
so `/app` lands on `sys.path` instead of `/app/mcp_server`.
|
||
|
||
Deliberately **no Ingress** for this Service, for the same reason the API's
|
||
ingress is disabled: neither has authentication.
|
||
|
||
**Blocked on a release, not on more code.** The running image is
|
||
`main-76e6eda`, which predates the `MCP_HOST` change, so enabling `mcp` today
|
||
would deploy a pod that binds loopback and never becomes reachable. The order
|
||
is: merge, let CI build `main-<sha>`, then bump the tag *and* flip
|
||
`mcp.enabled: true` in `deploy/railiance/apps/helm/state-hub-values.yaml` in one
|
||
commit, then `helm upgrade`. The values file is deliberately left at
|
||
`enabled: false` so an unrelated `helm upgrade` cannot ship a broken MCP.
|
||
|
||
**Release built (2026-08-24).** Pushing revealed why the image would not have
|
||
rebuilt: the CI path filter matched `api/**` while the Dockerfile copies nine
|
||
more paths, so changes to `mcp_server/` — and to `migrations/`, which is the
|
||
serious one, since a schema migration would not have shipped — merged to main
|
||
without producing an image. Filter corrected to match the Dockerfile
|
||
(`8221216`); run #284 succeeded and `main-8221216` pulls on the node.
|
||
|
||
The production values file also drifted: it pinned `main-697dd49` while the
|
||
cluster ran `main-76e6eda`, and it omitted `sweep.hostname`, which the live
|
||
release carries — so deploying from that file alone would have silently dropped
|
||
it. Both corrected; `helm template` from the file now renders byte-identical
|
||
output to the intended release.
|
||
|
||
**Remaining, both blocked on operator permission:**
|
||
|
||
1. Commit and push `deploy/railiance/apps/helm/state-hub-values.yaml` (tag bump
|
||
plus `mcp.enabled: true`). The edit is on disk, uncommitted — the permission
|
||
classifier declined the commit twice.
|
||
2. `helm upgrade` — also declined. Dry run is clean and adds only the
|
||
`state-hub-mcp` Deployment and ClusterIP Service:
|
||
|
||
```bash
|
||
cd ~/state-hub
|
||
helm upgrade state-hub ./deploy/railiance/apps/charts/state-hub \
|
||
-n state-hub -f deploy/railiance/apps/helm/state-hub-values.yaml \
|
||
--wait --timeout 4m
|
||
```
|
||
|
||
Node headroom was checked first: CPU requests sit at 92% of ~4000m, leaving
|
||
~320m. The MCP pod requests 50m, which keeps enough for the API's `maxSurge=1`
|
||
roll.
|
||
|
||
After the upgrade, verify from the node rather than through a tunnel:
|
||
|
||
```bash
|
||
ssh tegwick@92.205.62.239 'curl -sI http://<mcp-clusterip>:8001/sse'
|
||
kubectl -n state-hub get pods -l app.kubernetes.io/component=mcp
|
||
```
|
||
|
||
**Done (2026-08-24):** deployed and verified. `state-hub-mcp` is Running,
|
||
serving SSE on ClusterIP `10.43.110.80:8001` — confirmed from the node itself
|
||
rather than through a tunnel: HTTP 200, `text/event-stream`, and a real MCP
|
||
endpoint handshake, not merely an open port. End-to-end the pod reaches the
|
||
in-cluster API (`API_BASE=http://state-hub:8000`) and reads central's data, 79
|
||
repositories rather than the retired cache's 122.
|
||
|
||
The global agent instructions now point remote `dev-hub` at that ClusterIP, and
|
||
the remote port map lists no tunnel for either service.
|
||
|
||
`state-hub-mcp-railiance01` is now unblocked for removal — the last thing
|
||
depending on it is the documented registration, which has been repointed.
|
||
|
||
|
||
## Deliver the classification allowed-values file to central
|
||
|
||
```task
|
||
id: CUST-WP-0067-T09
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "45a2ba22-9d50-5ba8-9fa7-6e345ddce395"
|
||
```
|
||
|
||
`api/classification.py` resolved its allowed-values file from three hardcoded
|
||
developer workstation checkouts, so in the container none exist and every
|
||
classification write returned 500. This is why repository classification could
|
||
only ever be written from a workstation, and why all 33 repositories onboarded
|
||
under `T05` carry a domain but no classification fields.
|
||
|
||
`REPO_CLASSIFICATION_ALLOWED_PATH` now overrides the search (state-hub
|
||
`05dda8d`), but the file still has to reach the pod. Canon lives in
|
||
`the-custodian`, and the hub must not grow a second copy that silently drifts
|
||
from it — mount it from a ConfigMap generated from the canon file, and make the
|
||
generation step part of the release rather than a manual copy.
|
||
|
||
Once it serves, re-run onboarding across the 33 to project their classification.
|
||
|
||
Acceptance: a classification PATCH succeeds against central; the 33 records
|
||
carry their categories; the allowed-values source in the cluster is traceable
|
||
to `the-custodian/canon/standards/repo-classification.allowed.yaml`.
|
||
|
||
**Progress (2026-08-24):** delivery is built and validated. The chart carries a
|
||
generated copy of the canon file, mounted as a ConfigMap at
|
||
`/etc/state-hub/classification`, with `REPO_CLASSIFICATION_ALLOWED_PATH` set to
|
||
it.
|
||
|
||
The copy is the risk, so it is owned rather than trusted:
|
||
`scripts/sync_classification_allowed.py` regenerates it from canon, and
|
||
`--check` fails on drift. `make check-classification-allowed` and
|
||
`make railiance-state-hub-render` both exit non-zero when the copy diverges, so
|
||
a release cannot ship a stale vocabulary and silently validate against
|
||
yesterday's rules — verified by injecting drift.
|
||
|
||
Rendered ConfigMap parses and matches canon (version 1.0, 6 categories, 14
|
||
domains). The `volumeMounts`/`env` blocks were merged rather than appended:
|
||
adding a second pair would have produced duplicate YAML keys the moment `sweep`
|
||
was re-enabled. Both `sweep` states render correctly; `helm lint` clean.
|
||
|
||
**Remaining: one command.** Merged, CI green (`ac21acc`), image
|
||
`main-ac21acc` built, and the production values file bumped and pushed. The
|
||
`helm upgrade` itself was declined by the permission classifier, as in T08.
|
||
|
||
Live today: release 19 on `main-8221216`, with `state-hub` and `state-hub-mcp`
|
||
running and no `REPO_CLASSIFICATION_ALLOWED_PATH` in the pod — so classification
|
||
writes still 500.
|
||
|
||
```bash
|
||
cd ~/state-hub
|
||
helm upgrade state-hub ./deploy/railiance/apps/charts/state-hub \
|
||
-n state-hub -f deploy/railiance/apps/helm/state-hub-values.yaml \
|
||
--wait --timeout 4m
|
||
```
|
||
|
||
Then confirm the vocabulary is actually mounted, not merely that the pod is up:
|
||
|
||
```bash
|
||
kubectl -n state-hub exec deploy/state-hub -- \
|
||
head -3 /etc/state-hub/classification/repo-classification.allowed.yaml
|
||
```
|
||
|
||
Then project classification across the 33 already-onboarded repositories by
|
||
re-running onboarding; it is idempotent:
|
||
|
||
```bash
|
||
while read -r slug path; do
|
||
uv run --project ~/repo-manager rmgr repo-onboard \
|
||
--path "$path" --api-base http://127.0.0.1:8000
|
||
done < <list of onboarded repos>
|
||
```
|
||
|
||
After that the local cache database holds nothing unique except the 11
|
||
dispositioned records.
|
||
|
||
**Done (2026-08-25).** Operator ran the upgrade; release 20 on `main-ac21acc`
|
||
with the vocabulary mounted and `REPO_CLASSIFICATION_ALLOWED_PATH` set.
|
||
|
||
Classification writes against central now succeed — the first time they have
|
||
been possible from anywhere but a workstation. All 33 onboarded repositories
|
||
were re-run through `rmgr repo-onboard` and projected their classification;
|
||
every one returned `updated` with no warnings, where before each returned
|
||
`created-unclassified`. Central holds 111 repositories, 100 of them classified.
|
||
|
||
The idempotence mattered here: the same command that onboarded them projected
|
||
their classification later, with no separate backfill path to write or trust.
|
||
|
||
|
||
## Re-derive the workplan and task divergence
|
||
|
||
```task
|
||
id: CUST-WP-0067-T10
|
||
status: todo
|
||
priority: high
|
||
```
|
||
|
||
`ADR-010`'s original finding was about workplans, and it is still open. The
|
||
local cache holds 1075 workplans and 6073 tasks against central's 669 and 4123
|
||
— roughly 400 workplans and 1,950 tasks that exist nowhere else. Until these
|
||
are re-derived or dispositioned, the cache database cannot be discarded, which
|
||
means the retired instance's data is still load-bearing.
|
||
|
||
`ADR-010` assessed most of the equivalent divergence as file-backed and
|
||
therefore rebuildable, with a smaller orphan set needing case-by-case handling.
|
||
That assessment is from 2026-08-17 and predates this workplan's changes; it
|
||
should be re-measured rather than assumed.
|
||
|
||
The pattern from `T04`/`T05` applies: identify the authoritative source file,
|
||
confirm it is committed and pushed, re-derive onto central, and disposition in
|
||
writing whatever has no source. `fix-consistency` C-06 and the registrar already
|
||
do this for workplans, so the work is likely driving and verifying them rather
|
||
than writing something new.
|
||
|
||
This may warrant promotion to its own workplan — it is larger than the
|
||
repository-record subset and touches every repo with work records.
|
||
|
||
Acceptance: every cache-only workplan and task either exists on central or
|
||
carries a written disposition; the cache database can then be discarded, and
|
||
`ADR-010`'s divergence is closed rather than merely described.
|