state-hub/README.md

# State Hub

## Hub stack glossary

| Name | Role |
| --- | --- |
| **hub-core** | Shared library — router factories and MCP primitives |
| **state-hub** | This service — dev coordination, workplans, agent MCP |
| **core-hub** | Production framework — `/api/v2` at `hub.coulomb.social` |

Ecosystem architecture: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md`

Cross-repo regression: `/home/worsch/the-custodian/scripts/hub-ecosystem-regression.sh`

---

State Hub is the live coordination service for the Custodian ecosystem:
PostgreSQL persistence, FastAPI API, FastMCP server, Observable dashboard,
consistency tooling, and repo/workplan synchronization.

This repository is the standalone home for the service. It was extracted from
the former embedded implementation at:

```text
/home/worsch/the-custodian/state-hub
```

## Extraction State

The extraction workplan `CUST-WP-0043 - State Hub Repo Extraction` is complete.

Current state:

- The implementation has been imported here with subtree history.
- `CUST-WP-0042` has been re-homed into this repository.
- The old embedded tree in `the-custodian` remains only as a pointer.
- This repository is authoritative for State Hub code, docs, tests, dashboard,
  migrations, scripts, policies, and State Hub-local workplans.

## Workplans

New State Hub-local workplans should use the prefix:

```text
STATE-WP-0001
```

Legacy Custodian-hosted State Hub plans, such as `CUST-WP-0042`, may retain
their existing IDs when that preserves State Hub workstream/task continuity.
Do not create duplicate workstreams manually; write the workplan file first,
then run consistency sync.

---

## Stack

| Layer | Technology | Port |
|-------|-----------|------|
| Database | PostgreSQL 16-alpine (Docker) | `127.0.0.1:5432` |
| API | FastAPI + SQLAlchemy 2.0 async + asyncpg | `127.0.0.1:8000` |
| MCP server | FastMCP SSE | `127.0.0.1:8001` |
| Dashboard | Observable Framework | `127.0.0.1:3000` |
| CLI | `custodian` (Python, uv entry point) | — |

All services bind to `127.0.0.1` only — nothing exposed to the network.

**Production:** the primary State Hub API runs in the railiance01-hosted k3s
cluster. Workstation port `8000` reaches it through the ops-bridge
`state-hub-primary` tunnel. See
[`docs/cluster-operating-model.md`](docs/cluster-operating-model.md) for access,
rollback, backups, and pragmatic limitations.

Repository rename operations use the phased, UUID-preserving workflow in
[`docs/repository-rename-operations.md`](docs/repository-rename-operations.md).

---

## Setup

### Prerequisites

- Docker Engine
- Python 3.12+ with `uv` (`pip install uv`)
- Node.js 18+ (dashboard only)

### First-time local development setup

The local database and API are retained for development and disaster fallback.
They are not the normal production access path and conflict with the production
tunnel on port `8000`.

```bash
cd /home/worsch/state-hub

cp .env.example .env          # edit POSTGRES_PASSWORD
make install                  # uv sync
make db                       # docker compose up postgres
make migrate                  # alembic upgrade head
make seed                     # insert 6 canonical topics
make api                      # LOCAL fallback: db + migrate + uvicorn :8000
```

For a dashboard against that deliberate local backend, use
`make dashboard-local`.

### Production dashboard (normal operation)

The dashboard remains a workstation process, but its API is the primary State
Hub on railiance01. Start the tunnel first, then start the dashboard:

```bash
make bridges      # connect workstation :8000 to primary/railiance01
make dashboard    # verify the primary identity, then serve http://127.0.0.1:3000
```

`make dashboard` refuses to start if port `8000` is unreachable or identifies
itself as anything other than `primary/railiance01`. This prevents the empty
local fallback database from being mistaken for production.

The MCP adapter is optional and is not required by the dashboard:

```bash
make mcp-http     # optional local SSE adapter on :8001
```

### Fast work-record synchronization

Workplan files remain authoritative. Repo Manager assigns missing UUIDv5
identifiers locally, pushes the file commit, and asks the primary State Hub to
derive that exact commit from Forgejo in one transactional request:

```bash
make bridges
make sync
```

The sync refuses an uncommitted or behind branch and will not write to a local
empty database: the API must identify itself as `primary/railliance01`. If the
primary is unavailable, Repo Manager writes an explicit pending receipt under
the checkout's local `.git/repo-manager/` state; rerunning `make sync` safely
replays the current pushed state.

Use `statehub fix-consistency` only for the broader consistency/quality audit.
It is no longer the normal registration and task-update path.

### CLI

```bash
make install-cli              # symlink .venv/bin/custodian → ~/.local/bin
custodian status              # API health + summary totals
custodian register-project    # register cwd as a Custodian project
```

---

## Makefile Targets

| Target | What it does |
|--------|-------------|
| `make install` | `uv sync` — install Python deps + entry points |
| `make install-cli` | Symlink `custodian` to `~/.local/bin` |
| `make start` | Print the production UI and local fallback startup paths |
| `make bridges` | Connect managed tunnels, including workstation `:8000` to the railiance01 primary |
| `make db` | Start the local fallback/development Postgres container |
| `make db-tools` | Start postgres + pgadmin (http://127.0.0.1:5050) |
| `make migrate` | `alembic upgrade head` |
| `make seed` | Insert 6 canonical topics (legacy bootstrap) |
| `make register-from-classification REPO=slug` | Upsert repo from `.repo-classification.yaml` |
| `make register-from-classification-all` | Bulk reclassify all repos with classification files |
| `make api` | Start the local fallback API; replaces the production tunnel on `:8000` |
| `make dashboard-install` | Install dashboard npm deps from `dashboard/package-lock.json` |
| `make dashboard-check` | Build the Observable dashboard as a smoke/regression check |
| `make dashboard` | Verify `primary/railiance01`, then start the Observable dashboard |
| `make dashboard-local` | Start the dashboard against an intentional local/fallback API |
| `make sync` | Push missing deterministic IDs and reconcile this repo from the exact Forgejo commit |
| `make check` | Require a healthy `primary/railiance01` on workstation `:8000` |
| `make check-local` | Print `/state/health` without enforcing production identity |
| `make test` | Python test suite plus `make dashboard-check` |
| `make register-project DOMAIN=x PROJECT_PATH=y` | Register a project |
| `make clean` | `docker compose down -v` (destroys DB volume) |

---

## Database Schema

Repo-anchored coordination spine (STATE-WP-0065):

```
domains (14 market domains: infotech, financials, communication, …)
managed_repos (classification: category, domain, capability_tags, business_stake, …)
└── workplans (repo_id required; topic_id optional legacy tag)
    └── tasks
        └── progress_events
topics (optional cross-repo tag; domain_id → market domain)
decisions (FK: topic_id and/or workplan_id)
```

Each registered repo carries a committed `.repo-classification.yaml` (canon
standard v1.0). Registration and reclassification use
`make register-from-classification`.

### Key enums / vocabularies

| Field | Values |
|------|--------|
| `workplan_status` | `proposed` · `ready` · `active` · `blocked` · `backlog` · `finished` · `archived` |
| `task_status` | `wait` · `todo` · `progress` · `done` · `cancel` |
| `repo category` | `experimental` · `research` · `project` · `tooling` · `product` · `business` |
| `market domain` | 14 fixed slugs — see `the-custodian/canon/standards/repo-classification.allowed.yaml` |

### Governance constraints encoded in schema

- No hard DELETE endpoints — only soft: `archived`, `cancel`, `superseded`
- `progress_events` has no `updated_at` and no DELETE endpoint (append-only per constitution §5)
- `decisions` with financial/legal keywords + `pending` type → auto-set `escalation_note` (§4)

---

## API

Interactive docs at http://127.0.0.1:8000/docs once the API is running.

### Key endpoint: `/state/summary`

Returns a full snapshot in one call — used by both the MCP server and dashboard:

```json
{
  "generated_at": "...",
  "totals": {
    "topics":      { "active": 6, "paused": 0, "archived": 0, "total": 6 },
    "workstreams": { "ready": 1, "active": 1, "blocked": 0, "finished": 1, "total": 3 },
    "tasks":       { "wait": 0, "todo": 9, "progress": 0, "done": 11, "cancel": 0, "total": 20 },
    "decisions":   { "open": 1, "resolved": 0, "escalated": 0, "total": 1 }
  },
  "topics": [...],             // topics with nested workstream stubs
  "blocking_decisions": [...], // pending decisions only
  "waiting_tasks": [...],
  "recent_progress": [...],    // last 20 events
  "open_workplans": [...],
  "open_workstreams": [...]   // legacy alias of open_workplans until STATE-WP-0069 T06 retires it
}
```

**Caching:** responses are revision-gated — the API compares cheap per-table
`MAX(updated_at)` / `MAX(created_at)` watermarks before rebuilding. Unchanged
data returns the cached snapshot (`X-StateHub-Cache: hit-revision`). When core
data changes, the last good snapshot may be served immediately while a
background refresh runs (`X-StateHub-Cache: stale`). Force a synchronous rebuild
with `?refresh=true` or `Cache-Control: no-cache`. Infrastructure probes should
use `/state/health`, not `/state/summary`.

### Router summary

| Prefix | Operations |
|--------|-----------|
| `/topics` | CRUD (soft-delete: `archived`) |
| `/workplans` | Preferred CRUD surface for repo-backed workplans (soft-delete: `archived`) |
| `/workstreams` | Legacy compatibility CRUD surface; usage is recorded by legacy-meter |
| `/tasks` | CRUD (soft-delete: `cancel`); `PATCH` updates status |
| `/decisions` | CRUD (soft-delete: `superseded`); auto-escalation |
| `/progress` | `GET` list + `POST` append — no DELETE |
| `/legacy-meter` | Register, meter, and review legacy interface usage |
| `/state/summary` | Full snapshot |
| `/state/health` | DB connectivity check |

See `docs/workplan-terminology-transition.md` for the workstream-to-workplan
compatibility policy and retirement criteria.

---

## MCP Server

Runs as a persistent SSE service on `:8001`, independent of the Claude Code session.
Restart it anytime without restarting Claude Code.

```bash
make mcp-http   # start (or restart) the MCP SSE server on :8001
```

Registered at user scope in `~/.claude.json`:
```json
{ "type": "sse", "url": "http://127.0.0.1:8001/sse" }
```

To re-register from scratch:
```bash
claude mcp remove state-hub -s user 2>/dev/null || true
claude mcp add-json -s user state-hub '{"type":"sse","url":"http://127.0.0.1:8001/sse"}'
```

See `mcp_server/TOOLS.md` for the full tool reference card (30 lines, faster than reading `server.py`).

### Tools at a glance

**Query** (read-only): `get_state_summary` · `get_topic` · `list_blocked_tasks` · `list_pending_decisions` · `get_recent_progress`

**Mutate** (each auto-emits a progress event): `create_task` · `update_task_status` · `record_decision` · `resolve_decision` · `add_progress_event` · `create_workplan` · `update_workplan_status` · `register_repo_from_classification`

**Resources**: `state://summary` · `state://topics` · `state://workplans/{topic_slug}` · `state://decisions/blocking` · `state://tasks/blocked`

Legacy `workstream_*` tool names remain as aliases — see `mcp_server/TOOLS.md`.

---

## `custodian` CLI

Installed into `.venv/bin/custodian` by `uv sync`; symlinked to `~/.local/bin` by `make install-cli`.

```
custodian register-project [--domain DOMAIN] [--path PATH]
```

- `--path` defaults to current working directory
- `--domain` is auto-detected from `project_charter_v*.md` frontmatter if omitted

```
custodian status
```

Prints API health, totals, and any blocking decisions.

### What `register-project` does

1. Verifies the API is reachable
2. Looks up the topic ID for the domain via `/topics/?status=active`
3. Checks that `state-hub` is in `~/.claude.json`
4. Writes `$PROJECT_PATH/CLAUDE.md` from `scripts/project_claude_md.template`
5. Posts a `milestone` progress event recording the registration

---

## Project Registration Scripts

| Script | Purpose |
|--------|---------|
| `scripts/register_project.sh` | Shell version of `custodian register-project` |
| `scripts/patch_mcp_cwd.py` | Legacy: patched `cwd` for the old stdio registration (no longer needed) |
| `scripts/project_claude_md.template` | CLAUDE.md template with `{PROJECT_NAME}`, `{DOMAIN}`, `{TOPIC_ID}` |
| `scripts/seed.py` | Insert the 6 canonical topics into a fresh database |
| `scripts/pull_image.py` | WSL2 workaround: pull Docker images via Python urllib with Range-request chunking |

---

## Dashboard

Four pages at http://127.0.0.1:3000 (dev) or built with `npm run build`:

| Page | Content |
|------|---------|
| **Overview** | Status cards, task-by-status chart, recent activity feed, decisions due within 7 days |
| **Workstreams** | Filterable table by domain/status/owner; selected workstream task list; progress timeline |
| **Decisions** | Pending tab (with escalation highlights) and Made tab; resolution velocity chart |
| **Progress** | Append-only event feed with author badges; 30-day event volume chart |

Data loaders (`src/data/*.json.py`) are Python scripts that call the local API. They run at dev-server start and on `npm run build`. Clear the cache if data appears stale:

```bash
rm -rf dashboard/src/.observablehq/cache/
```

---

## Known Issues / WSL2 Notes

- **TLS bad record MAC on large downloads**: WSL2 corrupts packets on big TCP transfers. Use `scripts/pull_image.py` instead of `docker pull` for future image pulls.
- **MCP server is now SSE, not stdio**: Re-registration is `claude mcp add-json -s user state-hub '{"type":"sse","url":"http://127.0.0.1:8001/sse"}'`. The `patch_mcp_cwd.py` script and `.mcp.json` config are legacy artifacts from the old stdio setup.
- **AsyncSession concurrency**: SQLAlchemy 2.0 async sessions don't support concurrent operations. All queries in `/state/summary` run sequentially on a single session.