reuse-surface/docs/RegistryFederation.md

# Registry Federation

**Repository:** `reuse-surface`  
**Audience:** Architects and agents composing multi-repo capability indexes

---

## Purpose

helix_forge capabilities may be registered in multiple repositories. Federation
composes capability indexes from configured sources into a single discovery
surface without silently merging duplicate IDs.

Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
endpoints, published index artifacts, etc.). Remote indexes are cached under
`registry/federation/cache/` for offline reuse and faster compose.

## Manifest

`registry/federation/sources.yaml` lists index sources:

```yaml
version: 1
domain: helix_forge
collision_policy: warn
sources:
  - repo: reuse-surface
    index: registry/indexes/capabilities.yaml
    enabled: true
    required: true

  - repo: sibling-repo
    url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
    enabled: false
    required: false
    cache_ttl_seconds: 86400
    auth_env: FEDERATION_TOKEN
    auth_header: Authorization
```

Schema: `schemas/federation.schema.yaml`

### Source fields

| Field | Meaning |
|---|---|
| `repo` | Source repository slug |
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
| `enabled` | Include this source in compose |
| `required` | Fail compose if index missing or remote fetch fails with no cache |
| `domain` | Optional domain label |
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
| `auth_env` | Environment variable holding token or full header value for `url` sources |
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |

Each source must specify **either** `index` **or** `url`, not both.

### Local workstation roster (WP-0014)

`registry/federation/local-repo-roster.yaml` tracks every git repo at
`~/<slug>/` on the custodian workstation:

| Field | Meaning |
|---|---|
| `status` | `established` or `pending` |
| `batch` | Rollout batch (`B01`–`B06`) or `null` for pre-rollout repos |
| `publish_check` | `pass`, `fail`, or `pending` (Gitea raw URL probe) |
| `hub_registered` | Registered on `https://reuse.coulomb.social` |
| `seed_capability_ids` | Entries copied from reuse-surface by `owner` |

**Scope:** one directory level under `$HOME` with a `.git` directory; excludes
dot-directories and non-git folders. Rollout milestone:
`history/2026-06-16-local-repo-registry-rollout-complete.md`.

```bash
reuse-surface stats --roster registry/federation/local-repo-roster.yaml --federation-ready
```

## Index publish contract (domain repos)

Before a sibling repo can register on the hosted hub, it must publish
`registry/indexes/capabilities.yaml` at a **stable raw HTTP(S) URL** that
returns **200** with valid YAML (not a redirect to login or HTML).

### Required index fields

| Field | Requirement |
|---|---|
| `version` | Integer manifest version |
| `domain` | Domain slug (e.g. `helix_forge`) |
| `capabilities[]` | Non-empty or explicitly empty list |
| Per row: `id`, `name`, `summary`, `vector`, `path` | Match entry front matter |

Entry bodies remain in the source repo; the index is the federation surface.

### Raw URL shape (Gitea or Forgejo)

The organization is mid-transition from Gitea to Forgejo (REUSE-WP-0019);
sibling repos migrate their git remote independently, so both forms are
valid depending on which host a given repo currently lives on. Prefer the
branch-qualified form — both forges serve it directly with no redirect,
unlike the shorter `/raw/<branch>/...` form which 303-redirects:

```text
https://gitea.coulomb.social/coulomb/<repo>/raw/branch/<branch>/registry/indexes/capabilities.yaml
https://forgejo.coulomb.social/coulomb/<repo>/raw/branch/<branch>/registry/indexes/capabilities.yaml
```

Use `main` (or the repo's default branch). `reuse-surface federation
migrate-host` rewrites a repo's registered URL from one host to the other
once its remote has actually migrated (verifies reachability before
writing — never blind-rewrites a repo that hasn't moved yet). Verify before
registration:

```bash
curl -fsSI "<raw-url>" | head -n1   # expect HTTP/2 200 or HTTP/1.1 200
curl -fsS "<raw-url>" | head
```

### Auth expectations

- **Public indexes:** no auth; hub fetches without credentials.
- **Private indexes:** set `auth_env` on the hub registration (or local `url`
  source) to an environment variable holding a Bearer token or full header value.
  The hub stores `auth_env` / `auth_header` names only — never secret values.

### Local repo rollout tracking

Workstation-wide registry establishment is tracked in
`registry/federation/local-repo-roster.yaml` (workplan **REUSE-WP-0014**).

**Scope:** every git repository at `~/<slug>/` (one level under `$HOME`).
Update roster `status`, `hub_registered`, and `publish_check` after each repo
completes the establishment checklist below.

## Sibling onboarding (CLI)

```bash
cd ../state-hub
reuse-surface establish --scaffold --domain helix_forge
# optional: LLM_CONNECT_URL=... reuse-surface establish --discover --dry-run
reuse-surface validate
git push origin main
reuse-surface establish --publish-check \
  --raw-url https://forgejo.coulomb.social/coulomb/state-hub/raw/branch/main/registry/indexes/capabilities.yaml
```

### Ongoing maintenance (from sibling repo)

```bash
export LLM_CONNECT_URL=http://127.0.0.1:8080   # optional
reuse-surface maintain --all --from-git-since origin/main
reuse-surface maintain --all --auto --no-llm    # CI / pre-commit
reuse-surface maintain --publish \
  --raw-url https://forgejo.coulomb.social/coulomb/state-hub/raw/branch/main/registry/indexes/capabilities.yaml \
  --all --auto --no-llm
```

Copy `templates/Makefile.registry.fragment` for `make registry-maintain` /
`make registry-check`. Optional pre-commit hook:
`reuse-surface establish --scaffold --hook`.

### Registration checklist

1. Merge capability index to the default branch.
2. Confirm raw URL returns 200 YAML.
3. `reuse-surface hub register --repo <slug> --url <raw-url> --domain helix_forge`
4. `curl -fsS "$REUSE_SURFACE_URL/v1/federated" | jq '.capabilities | length'`
5. Optionally `reuse-surface hub sync --merge` to refresh local `sources.yaml`.

**Current blocks (2026-06-16):** `state-hub`, `feature-control`,
`identity-canon`, `shard-wiki`, and `config-atlas` raw URLs return **303** (not
published). See `history/2026-06-16-hub-registration-blocks.md` for probe evidence
and owner follow-ups.

## Compose workflow

```bash
reuse-surface federation compose
reuse-surface federation compose --refresh   # bypass remote cache
```

Writes `registry/indexes/federated.yaml` with:

- Merged `capabilities` from all enabled sources
- `source_repo` and `source_index` on every row
- `source_url` when the row came from a remote source
- `collision_policy` and per-source counts

### Remote cache

Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
`.gitkeep` is tracked.

When a refetch fails, compose reuses a stale cache and emits a warning. Required
remote sources without cache fail compose with a clear error.

### Collision policy

`warn` (default): duplicate IDs across sources are kept but reported as
warnings. Consumers must inspect `source_repo` before choosing an entry.

### Id namespace ownership

Federation aggregates `capability.*` ids. A few repos own **distinct id families**
that are *not* capability indexes and therefore never collide with `capability.*`:

| Namespace | Owner repo | Index (not federated here) |
|---|---|---|
| `surface.*` (configuration surfaces) | `config-atlas` | `registry/indexes/surfaces.yaml` |

config-atlas federates only its `capabilities.yaml` (the atlas capability itself);
its `surface.*` registry is a typed sibling. The `surface.*` namespace is reserved
for config-atlas so the two id families stay orthogonal under `collision_policy`.

### Owner migration and deduplication

After REUSE-WP-0014, many capabilities remain in both `reuse-surface` and their
`owner` repo index (seeded from reuse-surface during establishment). Federation
compose warns on these duplicates; it does **not** merge or prefer one source.

| Rule | Behavior |
|---|---|
| Canonical owner | The repo named in the entry `owner` field |
| Federation winner | Consumers pick the row where `source_repo` matches `owner` |
| reuse-surface copies | Planning stubs until owner index is published and fetchable |
| Removal | Separate commit per owner in `reuse-surface` — see `history/2026-06-16-federation-deduplication-plan.md` |
| Blocked owners | Keep reuse-surface row while owner `publish_check` fails (Gitea 404) |

Workstation rollout status: `registry/federation/local-repo-roster.yaml` (60 local
repos, publish pass/fail per slug).

Post-rollout compose (2026-06-16): **60** hub-synced URL sources, **37** federated
capability rows, **16** duplicate-ID warnings (mostly owner-migrated entries still
listed in reuse-surface).

## Hosted federation hub

Production hub: **`https://reuse.coulomb.social`** (Railiance `railiance01`,
companion deploy **RAILIANCE-WP-0007**).

The hub stores **repo registrations** (index URLs and metadata) and serves a
composed federated index at `GET /v1/federated`. It does not host capability
entry bodies — only coordinates which published indexes participate.

### Register and discover via CLI

```bash
export REUSE_SURFACE_URL=https://reuse.coulomb.social
export REUSE_SURFACE_TOKEN=<write-token>   # cluster secret reuse-surface-env

reuse-surface hub status
reuse-surface hub list
reuse-surface hub register --repo reuse-surface \
  --url https://gitea.coulomb.social/coulomb/reuse-surface/raw/main/registry/indexes/capabilities.yaml \
  --domain helix_forge
curl -fsS "$REUSE_SURFACE_URL/v1/federated" | jq '.capabilities | length'
```

Read endpoints are public; writes require `REUSE_SURFACE_TOKEN` (Bearer). API
spec: `specs/FederationHubAPI.md`.

### Automatic recompose and freshness (REUSE-WP-0019-T02/T03/T06)

The hub recomposes automatically rather than waiting for a manual
`reuse-surface federation compose --refresh`:

- **Forgejo org webhook** (primary): a single org-level webhook on `coulomb`
  fires on every push, HMAC-signed. `POST /v1/webhooks/forgejo` verifies the
  signature, checks whether the push touched `registry/indexes/`, and — only
  if so — recomposes. It never parses pushed file content, only paths.
- **Scheduled fallback**: `.forgejo/workflows/recompose-fallback.yaml` in
  this repo calls `POST /v1/federated/compose` (token-auth) every 6 hours,
  in case a webhook delivery is ever missed.
- **Freshness visibility**: `GET /v1/federated` carries `composed_at`
  (timestamp of the last *forced* recompose) and `stale` (set by the
  webhook, cleared on the next successful recompose). `reuse-surface stats`
  surfaces both plus an `age_days`/`stale_warning` computed against a
  threshold (`REUSE_SURFACE_FRESHNESS_DAYS`, default 7 days) — this repo's
  own CI runs an informational (non-failing) freshness check on every push.

Setting up the org webhook or rotating its secret is an operator action on
the Forgejo instance and the cluster Secret, not something `reuse-surface`
itself automates — see `railiance-apps/docs/reuse-surface-on-railiance01.md`
(the authoritative operator runbook) and `docs/deploy/reuse-kubernetes.md`
for the current secret custody chain (OpenBao → `ExternalSecret` →
`reuse-surface-env`).

### Hub vs local `sources.yaml`

| Workflow | When to use |
|---|---|
| **Hub** | Shared membership across agents and repos; no per-machine `sources.yaml` edits |
| **Local compose** | Offline development, CI with checked-in sources, or hub unavailable |

Local `registry/federation/sources.yaml` remains valid for `reuse-surface
federation compose`. Use `reuse-surface hub sync` to materialize `sources.yaml`
from hub `GET /v1/repos` state.

### hub sync

```bash
export REUSE_SURFACE_URL=https://reuse.coulomb.social
reuse-surface hub sync --dry-run          # preview manifest
reuse-surface hub sync --merge            # hub URL sources + local index sources
reuse-surface hub sync                    # replace with hub-enabled registrations
```

| Flag | Behavior |
|---|---|
| `--merge` | Keep local `index` sources whose `repo` slug is not on the hub |
| `--replace` (default) | Write only hub-enabled registrations as `url` sources |
| `--output` | Override manifest path (default `registry/federation/sources.yaml`) |
| `--dry-run` | Print YAML without writing |

After sync, run `reuse-surface federation compose` to verify offline compose.

## Agent query pattern

1. **Hub path:** `GET /v1/federated` or `reuse-surface hub list` for registered
   repos; fetch composed capabilities from the hub.
2. **Local path:** Run `reuse-surface federation compose` after manifest or
   sibling index changes; read `registry/indexes/federated.yaml`.
3. Open `path` in the source repo for full entry detail when local; follow
   `source_url` / `source_index` when remote.
4. Run `reuse-surface graph --check` before relying on relation navigation.

### Cross-repo discovery without local checkout

Register published raw index URLs on the hub, or enable a `url` source in
`sources.yaml` pointing at Gitea/GitHub/static hosts. Set `auth_env` when the
endpoint requires a token. Agents without sibling repo clones can discover
capabilities from the hub or from HTTP sources plus the local index.

### Query before building (`plan-check`)

Before drafting a new workplan, run `reuse-surface plan-check` (REUSE-WP-0018)
against a draft workplan file or a free-text intent. It matches your intent
against `registry/indexes/federated.yaml` and returns a `reuse | extend | new`
verdict — advisory only, never a gate:

```bash
reuse-surface plan-check workplans/XXX-WP-NNNN-something.md
reuse-surface plan-check --intent "parse invoices and file evidence"
```

On a `new` verdict, `--file-request` bridges the gap to a State Hub
capability request so other domains can see the unmet need; `report gaps
--check-capability-requests` surfaces open requests with no matching
federated capability. See `specs/PlanCheck.md` for the full design.

## Relation graphs

```bash
reuse-surface graph
reuse-surface graph --check
reuse-surface graph --stdout
```

Generates `docs/graph/capability-graph.mmd` from local entry `relations`.
`--check` reports `depends_on` cycles and broken relation targets against the
federated ID set.

## CI integration

Gitea CI runs:

```bash
reuse-surface validate --relations --fail-on-warnings
reuse-surface federation compose
reuse-surface catalog
reuse-surface graph --check --fail-on-warnings
pytest -q
```

CI uses local sources only (remote examples are disabled). Warnings on missing
optional sibling indexes do not fail CI; schema validation errors do.