reuse-surface/SCOPE.md

316 lines
16 KiB
Markdown
Raw Permalink Normal View History

2026-06-15 00:04:09 +02:00
# SCOPE
## One-liner
Capability registry for planning and implementation reuse based on discovery and delivery maturity.
## Core Idea
2026-06-15 01:04:30 +02:00
`reuse-surface` provides a registry-centric reuse layer for capabilities. It
makes capabilities visible, comparable, assessable, and reusable for planning,
implementation, and operation. A capability that is not registered is invisible
for reuse within this product boundary.
2026-06-15 00:04:09 +02:00
## In Scope
2026-06-15 01:04:30 +02:00
- Maintain the capability maturity model, standards, schemas, registry formats,
sample entries, indexes, validation guidance, CLI tooling, hub service, and
agent instructions.
2026-06-15 01:04:30 +02:00
- Keep `INTENT.md`, `specs/`, registry artifacts, and State Hub workplans
aligned on the registry-first reuse boundary.
- Support humans and agents as registry consumers through Markdown-first
authoring and machine-readable metadata.
- Record decisions, progress, and workplan status through State Hub.
- Verify changes with `reuse-surface validate`, `git diff --check`, and ADR-001
2026-06-15 01:04:30 +02:00
consistency checks.
2026-06-15 00:04:09 +02:00
## Out of Scope
- Host or operate the registered capabilities themselves (except the federation
hub coordinator, which stores repo metadata and index URLs only).
2026-06-15 01:04:30 +02:00
- Replace package registries, service catalogs, issue trackers, or project
management systems.
- Judge internal code quality as capability maturity.
- Own unrelated adjacent systems or make irreversible operational decisions
without human approval.
2026-06-15 00:04:09 +02:00
## Relevant When
- Deciding whether a capability already exists before planning or building one
(`reuse-surface plan-check`).
- Registering a new capability so it becomes visible for reuse.
- Promoting a capability along the D/A/C/R maturity axes with evidence.
- Comparing candidate capabilities by maturity, scope, relations, and consumer
guidance.
- Detecting overlap or duplication between capabilities across repos.
- Adding a sibling repo to the federation, or debugging why its index does not
appear in the federated view.
- Recording or interpreting reuse telemetry as R-axis evidence.
---
## Not Relevant When
- Hosting, running, or operating the capabilities themselves — the registry
describes capabilities, it does not execute them.
- Looking for a package registry, service catalog, issue tracker, or project
management system. Those are different tools with different guarantees.
- Judging internal code quality. Maturity here is about discovery and delivery,
not about how the implementation is written.
- Deploying the hub. The Kubernetes release lives in `railiance-apps`
(`charts/reuse-surface/`); this repo owns the image and the deploy guide only.
- Routing credentials. See `.claude/rules/credential-routing.md` — ops-warden
issues SSH certificates, OpenBao holds secrets.
---
## How It Fits
`reuse-surface` sits between planning and implementation. A workplan or intent
is checked against the federated index before work starts; the verdict is
reuse, extend, or new. Capabilities registered by sibling repos flow in through
federation, and observed reuse flows back as maturity evidence.
- **Upstream:** each sibling repo publishes `registry/indexes/capabilities.yaml`
from its own checkout. The repo owns its entries; this registry never edits
them.
- **Here:** the federation composer merges member indexes into
`registry/indexes/federated.yaml`, and the hosted hub serves the same view at
`GET /v1/federated`.
- **Downstream:** humans and agents query the index, the CLI, or the hub before
building. `plan-check --record-outcome` returns telemetry that feeds the
R axis.
State Hub is the coordination layer, not the registry: workplans and decisions
live there, capability descriptions live here.
---
## Terminology
| Term | Meaning |
|---|---|
| **Capability** | A reusable unit of function, described by a registry entry. Not a package, not a service — either can implement one. |
| **Maturity vector** | `D / A / C / R` — Discovery, Availability, Consumability, Reliability. See `specs/CapabilityMaturityStandard.md`. |
| **Registry entry** | Markdown with YAML front matter under `registry/capabilities/`, validated against `schemas/capability.schema.yaml`. |
| **Index** | `registry/indexes/capabilities.yaml` — one row per entry, the discovery surface for this repo. |
| **Federated index** | The composed view across all member repos. |
| **Member / source** | A repo registered in `registry/federation/sources.yaml` or on the hub. |
| **Hub** | The hosted service at `https://reuse.coulomb.social` that composes and serves the federated index. |
| **Promotion** | Raising a maturity axis, backed by evidence and recorded in `promotion_history`. |
**Note on two similar formats.** The fenced `capability` blocks in a repo's
`SCOPE.md` (`type` / `title` / `description` / `keywords`) are *not* the same
shape as index rows in `registry/indexes/capabilities.yaml`
(`id` / `name` / `summary` / `vector` / `domain` / `status` / `owner` / `path` /
`tags` / `consumption_modes`). SCOPE blocks are a prose-level advertisement;
index rows are validated registry data and require an `id`. Copying one shape
into the other is a real and observed failure mode — it silently breaks
federation for that member.
---
## Related / Overlapping
| Repo / system | Relationship |
|---|---|
| **State Hub** (`~/state-hub`) | Coordination read model: workplans, tasks, decisions, progress. Complementary — it tracks *work*, this tracks *capabilities*. |
| **railiance-apps** | Owns the Kubernetes release for the hub (`charts/reuse-surface/`, RAILIANCE-WP-0007). Deploys what this repo builds. |
| **ops-warden** | Credential and access routing. Explicitly out of scope here. |
| **Sibling domain repos** | Federation members. Each owns its own entries and index; this repo owns only composition and the standard. |
| **Package registries** (PyPI, npm, OCI) | Distribute artifacts. This registry describes capabilities and may reference artifacts, but does not host them. |
---
## Provided Capabilities
```capability
type: registry
title: Capability registration and maturity assessment
description: Register capabilities as validated Markdown entries with D/A/C/R maturity vectors, promotion history, and evidence, so they become discoverable and comparable for planning and implementation reuse.
keywords: [registry, capability, maturity, discovery, promotion, reuse, governance]
```
```capability
type: service
title: Capability index federation
description: Compose capability indexes published by many repos into one federated view, served locally by CLI and in production by a hosted hub with webhook-driven refresh and staleness visibility.
keywords: [federation, index, compose, hub, webhook, capabilities, cross-repo]
```
```capability
type: tooling
title: Pre-build reuse check
description: Match a draft workplan or free-text intent against the federated capability index and return a reuse, extend, or new verdict, bridging a new verdict to a State Hub capability request and recording the outcome as reuse telemetry.
keywords: [plan-check, reuse, verdict, planning, telemetry, capability-request]
```
---
## What Is Possible Now
The MVP registry foundation, CLI tooling (REUSE-WP-0003), federation stack
(WP-0005/0010), and hosted hub (WP-0011) are in place. Humans and agents can:
- **Discover capabilities** via `registry/indexes/capabilities.yaml`,
`reuse-surface query`, or `GET https://reuse.coulomb.social/v1/federated`
- **Add a new capability** at D0/A0/C0/R0 using
`templates/capability-entry.template.md`
- **Promote capabilities** with evidence, `promotion_history`, and index vector
updates
- **Compare candidates** using maturity vectors, scope, relations, and consumer
guidance
- **Record expectations** through `external_evidence.completeness` and
`external_evidence.reliability`
- **Validate entries automatically** with `reuse-surface validate`
- **Export a machine-readable bundle** with `reuse-surface export`
- **Detect overlap candidates** with `reuse-surface overlaps`
- **Generate a human-readable catalog** with `reuse-surface catalog`
- **Browse a searchable catalog** at `docs/catalog/search.html`
- **Compose federated indexes** with `reuse-surface federation compose`
(local paths and remote HTTP URLs with cache)
- **Register federation sources on the hosted hub** with `reuse-surface hub`
against `https://reuse.coulomb.social`
- **Sync local federation manifest from hub** with `reuse-surface hub sync`
- **Export planning cohorts** with `reuse-surface report cohorts`
- **Report registry gaps** with `reuse-surface report gaps` (roster blockers,
empty scaffolds, dedup stubs)
- **Bootstrap a sibling registry** with `reuse-surface establish --scaffold`
- **Verify index publish readiness** with `reuse-surface establish --publish-check`
- **View registry stats** with `reuse-surface stats` (per-repo or
`--roster registry/federation/local-repo-roster.yaml --federation-ready`)
- **Draft or refresh entries** with `reuse-surface establish --discover` and
`reuse-surface update` (optional llm-connect backend)
- **Maintain registry interactively or automatically** with `reuse-surface maintain`
(TTY prompts, `--auto`, optional llm-connect, `--publish` chain)
- **Run the hub locally or in a container** with `reuse-surface serve`
- **Generate relation graphs** with `reuse-surface graph`
- **Explore relations interactively** at `docs/graph/index.html`
- **Avoid duplicates** by querying the index and checking overlaps before adding
entries
REUSE-WP-0018 T01/T02/T04/T06: plan-check deterministic matching + State Hub bridge T01: specs/PlanCheck.md design doc, plan-check-result.schema.json and reuse-event.schema.json (the latter shared with WP-0019's reuse telemetry). T02: reuse_surface/plan_check.py + 'reuse-surface plan-check' CLI command. Deterministic token-Jaccard matching against registry/indexes/federated.yaml (reuses overlaps.py's TOKEN_RE rather than a second scoring method), with reuse/extend/new verdicts, markdown and --format json output, staleness warning, and --record-outcome JSONL telemetry. T04: reuse_surface/statehub_bridge.py bridges plan-check 'new' verdicts to State Hub capability requests (--file-request) and surfaces open requests with no matching capability (report gaps --check-capability-requests, opt-in to stay offline-safe). Verified against the live local State Hub API; status field (not catalog_entry_id presence) is the correct open/closed signal, and the list endpoint needs a longer timeout than the health check (~7s observed with 5 rows). T06: docs (tools/README.md, RegistryFederation.md, SCOPE.md, IntentScopeGapAnalysis.md priority 29) and an informational CI smoke step. T03 (LLM rerank) not started -- llm-connect isn't running on this workstation. T05 (ecosystem rollout) remains blocked: WP-0017 has drafted entries for all 61 repos but they're still local-only pending its own T05 push/publish pass, so the federated index isn't yet worth rolling out plan-check as ecosystem convention. 16 new tests, all mocked -- no network calls in the default test run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 00:57:18 +02:00
- **Check before building** with `reuse-surface plan-check` (REUSE-WP-0018) —
match a draft workplan or free-text intent against the federated index and
get a reuse/extend/new verdict; `--file-request` bridges a `new` verdict to
a State Hub capability request; `report gaps --check-capability-requests`
surfaces open requests with no matching capability
- **Get automatic hub refresh** (REUSE-WP-0019-T02/T03) — a Forgejo push
webhook (`POST /v1/webhooks/forgejo`, HMAC-signed) recomposes the hub's
federated index when a registered repo's `registry/indexes/` changes,
with `composed_at`/`stale` visibility on `GET /v1/federated` and a
scheduled fallback recompose if a webhook delivery is ever missed
- **Record reuse telemetry** (REUSE-WP-0019-T04) — `plan-check
--record-outcome` and `reuse-surface record-reuse` post facts to
`POST /v1/reuse-events` when the hub is reachable, falling back to
`registry/telemetry/plan-check-events.jsonl` otherwise (same schema
either way); `GET /v1/reuse-events?capability_id=` aggregates them
REUSE-WP-0019-T05: reuse telemetry aggregation into R-axis evidence reuse_surface/reports.py: collect_reuse_events() merges the hub's GET /v1/reuse-events (if reachable) with this repo's local JSONL fallback, deduped. collect_reuse_report() aggregates per-capability consumer counts, outcome breakdown, and last-used. collect_reused_by_suggestions() proposes evidence-gated relation_add patches -- only for capabilities this repo owns, only for consumer repos not already listed -- reusing the existing patches.py:apply_patches mechanism (relation_add already isn't in SAFE_DETERMINISTIC_KINDS, so it was already never auto-applied by maintain --auto). New CLI: reuse-surface report reuse [--capability-id] [--format] [--suggest-relations] [--apply]. --apply requires --suggest-relations and is the only thing that writes -- nothing happens automatically from telemetry alone. schemas/capability.schema.yaml: added relations.reused_by as a new repoSlugList type, distinct from the existing capability-id relations, since reused-by targets are consumer repo slugs. specs/CapabilityMaturityStandard.md Sec8.9: what observed-reuse evidence counts toward R2->R3 (single corroborating consumer) vs R3->R4+ (multiple independent consumers) and what it never substitutes for. 19 new pytest cases, 162 total pass. Live-verified with synthetic local events against a real capability entry: --suggest-relations --apply correctly wrote relations.reused_by via the real apply_patches path (reverted after, since it was a smoke test). Deliberately deferred: surfacing consumer counts in the catalog/graph -- graph.py's relation model is capability-to-capability edges, a different namespace than repo-slug reused_by targets; catalog.py doesn't currently parse full front matter per entry. Left for a follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 22:47:51 +02:00
- **Aggregate reuse telemetry into R-axis evidence** (REUSE-WP-0019-T05) —
`reuse-surface report reuse` shows per-capability consumer counts,
outcomes, and last-used; `--suggest-relations` proposes evidence-gated
`relations.reused_by` patches (via the same `apply_patches` mechanism
`maintain` uses), applied only with an explicit `--apply` — never
automatic. `specs/CapabilityMaturityStandard.md` §8.9 documents what
observed-reuse evidence does (and doesn't) count toward R2/R3+
Registry **tooling** availability is **A4** (CLI plus hosted hub HTTP API).
Registry **authoring** remains Markdown-first; consumption combines entries, the
index, CLI automation, and the production hub.
## What Is Not Possible Yet
- **Multi-domain federation** — the federated index is still composed under a
single `helix_forge` manifest domain, though member rows have begun carrying
their own (`evidence-binder` publishes `domain: infotech`)
- **Planning analytics breadth** — `report gaps` shipped (REUSE-WP-0015-T03);
no roadmap views or standardization tracker beyond `overlaps` and compose
collision warnings
- **Managed platform posture** — hub runs as a container (A5 artifact) without
implemented SLO, multi-replica, or Postgres backing (criteria documented)
- **Formal consumer feedback loop** for registry workflows (reliability evidence
is mostly structural: CI/tests, not production telemetry)
See `tools/README.md` for command reference.
2026-06-15 00:04:09 +02:00
## Current State
- **Status:** active registry with CLI, federation, production hub, and
workstation-wide coverage campaign complete (REUSE-WP-0017 finished 2026-07-07).
- **Capabilities (reuse-surface):** 2 helix_forge meta-registry entries in
`registry/capabilities/`.
- **Workstation roster:** 62 local git repos at `~/<slug>/` tracked in
`registry/federation/local-repo-roster.yaml` — all **established**, **62/62**
hub-registered (inter-hub registration disabled on hub), **62/62**
publish-check pass, **62/62** capability coverage (`has` or explicit `none`).
- **Federation:** `registry/federation/sources.yaml`**61** enabled sources;
`registry/indexes/federated.yaml`**64** composed capability rows
(inter-hub excluded; core-hub included; 0 duplicate-ID warnings at last compose).
- **CLI / service:** `reuse_surface/` — validate, query, export, overlaps,
catalog, federation, graph, hub client, establish/update/stats, `serve`
(FastAPI hub).
- **Production hub:** `https://reuse.coulomb.social`**61** enabled repo
registrations; `GET /v1/federated` serves **64** capabilities from published
raw URLs (compose refreshed 2026-08-21). Runs image
`forgejo.coulomb.social/coulomb/reuse-surface:main-b035664` (Helm revision 8);
`/health`, `/v1/repos`, `/v1/federated`, and `/v1/reuse-events` all answer.
- **Specs:** `specs/FederationHubAPI.md`, `schemas/hub-registration.schema.yaml`.
- **Docs:** `docs/CapabilityRegistryConcept.md`, `docs/RegistryFederation.md`,
`docs/IntentScopeGapAnalysis.md`, deploy guide `docs/deploy/reuse-kubernetes.md`.
- **CI:** `.forgejo/workflows/ci.yml` — validate, federation compose, catalog,
graph, pytest, informational `report cohorts`, `stats --roster`, `report gaps`
(migrated from Gitea 2026-07-07, REUSE-WP-0019-T03). Also
`.forgejo/workflows/image.yaml` (container build/push) and
`recompose-fallback.yaml` (scheduled hub recompose, webhook backstop).
- **Relation graph:** `docs/graph/capability-graph.mmd`, `docs/graph/index.html`.
- **Searchable catalog:** `docs/catalog/search.html`.
- **Workplans:** REUSE-WP-0001 through REUSE-WP-0015 finished (archived);
**REUSE-WP-0017** capability coverage campaign finished (2026-07-07);
REUSE-WP-0019-T06: hub freshness monitoring, docs, close workplan reuse_surface/stats.py: _hub_summary() now reports composed_at, stale, age_days, freshness_threshold_days (REUSE_SURFACE_FRESHNESS_DAYS env, default 7), and a computed stale_warning. New hub_client.hub_federated() backs it. format_stats_markdown surfaces a STALE marker when triggered. .forgejo/workflows/ci.yml: new informational (non-failing) hub freshness check against the live production hub on every push -- prints a ::warning:: annotation when stale, never fails the build. docs/RegistryFederation.md: new section tying together the webhook (T02), scheduled fallback (T03), and freshness visibility (T06) into one explanation. docs/deploy/reuse-kubernetes.md: updated for the T03 Forgejo migration and the now-automated image.yaml build; image promotion checklist updated for the known /health ingress bug (verify via /v1/repos or /v1/federated instead). 14 new pytest cases, 173 total pass. Live-verified against production: reuse-surface stats correctly showed composed_at/age_days for the real federated index. Separately discovered and confirmed (via a live signed webhook test) that reuse-surface-env moving to ExternalSecret/OpenBao custody (railiance-apps commit 706f6c7, found while updating these docs) did not break the T02/T03 webhook -- the synced value still matches what the hub actually uses. REUSE-WP-0019 is now fully complete (T01-T06). SCOPE.md and docs/IntentScopeGapAnalysis.md updated to reflect closure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 00:09:58 +02:00
**REUSE-WP-0018** plan-check consumption loop finished (2026-07-07);
**REUSE-WP-0019** Forgejo automation and reuse telemetry finished
(2026-07-08, all six tasks T01T06).
- **Assessment history:** `history/` — intent/scope assessments, rollout
milestone, dedup plan, per-repo follow-up.
- **Self-assessed vector:** `D5 / A4 / C5 / R3` (see `docs/IntentScopeGapAnalysis.md`).
## Repository Layout
```text
reuse-surface/
├── INTENT.md
├── SCOPE.md
├── AGENTS.md
├── pyproject.toml
├── Dockerfile
├── reuse_surface/ # CLI, hub service, federation, graph, catalog
├── specs/
├── schemas/
├── templates/
├── registry/
│ ├── capabilities/ # per-entry Markdown
│ ├── indexes/ # capabilities.yaml, federated.yaml
│ └── federation/ # sources.yaml, local-repo-roster.yaml, cache/
├── docs/
├── tools/
└── workplans/
└── archived/
```
2026-06-15 00:04:09 +02:00
## Getting Oriented
- Start with: INTENT.md
- Registry concept: docs/CapabilityRegistryConcept.md
- Intent vs scope gaps: docs/IntentScopeGapAnalysis.md
- Assessment snapshots: history/
2026-06-15 01:04:30 +02:00
- Product requirements: specs/ProductRequirementsDocument.md
- Use cases: specs/UseCaseCatalog.md
- Maturity standard: specs/CapabilityMaturityStandard.md
- Hub API: specs/FederationHubAPI.md
- Registry index: registry/indexes/capabilities.yaml
- Registry guidance: registry/README.md
- Federation guide: docs/RegistryFederation.md
- Hub deployment: docs/deploy/reuse-kubernetes.md
- Generated catalog: docs/CapabilityCatalog.md
- Searchable catalog: docs/catalog/search.html
- Relation graph: docs/graph/capability-graph.mmd
- Graph explorer: docs/graph/index.html
- CLI reference: tools/README.md
2026-06-15 00:04:09 +02:00
- Agent instructions: AGENTS.md
2026-06-15 01:04:30 +02:00
- Workplans: workplans/