reuse-surface/docs/IntentScopeGapAnalysis.md

244 lines
13 KiB
Markdown
Raw Permalink Normal View History

# INTENT ↔ SCOPE Gap Analysis
**Repository:** `reuse-surface`
**Artifact:** `docs/IntentScopeGapAnalysis.md`
**Status:** Living analysis
**Updated:** 2026-06-16
**Purpose:** Record alignment, drift, and open gaps between declared intent and
current delivered scope so future workplans can close them deliberately.
**Canon:** Product truth in `INTENT.md`; delivery truth in `SCOPE.md`.
---
## 1. Summary
`INTENT.md` defines the long-term capability registry product: a reuse surface
that makes capabilities visible, assessable, and consumable for planning and
implementation across repos and organizations.
`SCOPE.md` (refreshed postWP-0014) describes current delivery:
| Dimension | Delivered today |
|---|---|
| Registry product | 20 helix_forge entries in reuse-surface; maturity model, schema, validation |
| Tooling | CLI **A3+**; hosted hub HTTP API **A4** at `https://reuse.coulomb.social` |
| Federation | 60 hub registrations; 60 URL sources; 37 composed federated rows |
| Workstation rollout | 60/60 local repos established (`local-repo-roster.yaml`) |
| Authoring aids | `establish`, `update`, `stats`, optional llm-connect discover |
| Observability | `report cohorts`, catalog/graph UIs, pytest + CI gates |
REUSE-WP-0001 through REUSE-WP-0014 closed the MVP, federation hub, intent
alignment, registry bootstrap, and **local workstation registry rollout**. The
documents remain **directionally aligned** on registry-first reuse, four maturity
dimensions, and human/agent consumers.
**Open gaps** (operator or follow-on workplans):
1. **Planning analytics breadth** — gap report shipped; no roadmap views or
standardization tracker.
3. **Hub automation** — no polling/webhooks for federated refresh.
4. **Managed platform posture** — A6/Postgres/multi-replica not implemented.
5. **Consumer reliability evidence** — catalog mostly R0R2; thin telemetry loop.
**Current reuse-surface product vector (self-assessment):** `D5 / A4 / C5 / R3`
---
## 2. Alignment Matrix
| Topic | INTENT.md | SCOPE.md | Status |
|---|---|---|---|
| Registry-first boundary | Unregistered capabilities invisible | Same | **Aligned** |
| Four maturity dimensions | D, A, C, R; internal vs external evidence | Same in entries and schema | **Aligned** |
| Human and agent consumers | Registry formats for both | Markdown + YAML + CLI + hub API | **Aligned** |
| Reuse over inventory | Explicit principle | Query, overlaps, federation, dedup plan | **Aligned** |
| Planning vs implementation reuse | Distinct dimensions | Query, vectors, `consumption_modes` | **Aligned** |
| Technical foundation | Eventually technical | CLI A3, hub A4, container A5 artifact | **Aligned** |
| Cross-repo capability visibility | Reuse across repos/products/orgs | 60 repos scaffolded; 20 federated caps | **Partial** |
| Generalized primitives (D7) | Beyond single repo/domain | helix_forge only; no D7 pipeline view | **Partial** |
| Success criteria (8 outcomes) | See §5 | Most met; analytics + normalization partial | **Partial** |
| Repository layout | Tree in INTENT §Initial Repository Role | Matches SCOPE; roster artifact added | **Aligned** |
| State Hub / workplans | Not in INTENT | ADR-001 sync; delivery process | SCOPE-only (OK) |
| Hosting capabilities | Out of scope | Hub stores metadata/URLs only | **Aligned** |
---
## 3. INTENT → SCOPE Gaps (Open)
What `INTENT.md` still expects beyond current `SCOPE.md` delivery.
### 3.1 Cross-repo federation fetch and namespace (Medium)
| INTENT claim | SCOPE reality | Gap |
|---|---|---|
| Capabilities reusable across repos, products, orgs | 60 repos on hub; 20 federated rows | Single domain (`helix_forge`) |
| Find capabilities before rebuilding at network scale | Hub + compose + roster | — |
| Identify duplicates (success criterion) | `overlaps` + compose (0 duplicate warnings) | — |
**Follow-up:** `history/2026-06-16-wp0014-remaining-work-by-repo.md` (Gitea);
`history/2026-06-16-federation-deduplication-plan.md` (reuse-surface dedup).
### 3.2 Planning support breadth (LowMedium)
| INTENT claim | SCOPE reality | Gap |
|---|---|---|
| Plan prototype/MVP/enhancement/platform work | `report cohorts`, `report gaps`, query | No roadmap views |
| Identify gaps, overlaps, standardization candidates | `overlaps`, `report gaps` | No standardization workflow |
| Track progress to generalized capabilities (D7) | Per-entry `promotion_history` | No cross-entry D7 pipeline |
### 3.3 Hub operations (LowMedium)
| INTENT claim | SCOPE reality | Gap |
|---|---|---|
| Operational federation | Production hub; `hub sync` | On-demand compose; no webhooks |
| Dependable platform consumption (R4+) | Container deploy doc | SQLite single-replica; no SLO enforcement |
### 3.4 Consumer reliability evidence (LowMedium)
| INTENT claim | SCOPE reality | Gap |
|---|---|---|
| Reliability from consumer signals | Schema + README checklist | Most entries R0R2 |
| Trust for important workflows (R4+) | CI + hub smoke | No formal consumer_feedback loop |
### 3.5 Owner-repo capability depth (Low — ongoing)
| INTENT claim | SCOPE reality | Gap |
|---|---|---|
| Rich registry across organization | 60 scaffolds; 8 seeded | 52 repos at capability_count 0 |
| Discover mature capabilities network-wide | 20 federated rows | Growth is per-repo, optional |
---
## 4. Closed Gaps (Historical)
| Workplan / milestone | Gap closed | Evidence |
|---|---|---|
| WP-0003 WP-0011 | MVP CLI, validation, catalog, federation, hub | Archived workplans; pytest |
| WP-0012 | `hub sync`, cohort reports, INTENT layout, hub hardening doc | `hub_sync.py`, `docs/deploy/` |
| WP-0013 | Registry bootstrap in sibling repos | `establish`, `update`, `stats` |
| WP-0014 | Workstation registry rollout; hub 60 registrations | `local-repo-roster.yaml`; milestone history |
| WP-0015-T02 | Federated ID deduplication | 2 local entries; 0 compose duplicate warnings |
| WP-0015-T03 | Planning gap report | `reuse-surface report gaps` |
| WP-0015 | Gitea publish + federation polish | 60/60 publish pass; org transfer to `coulomb/` |
| Priority 18 | Sibling / workstation hub membership | 60/60 `hub list` |
| Priority 1924 | See §8 | WP-0012/0013 deliveries |
---
## 5. INTENT Success Criteria vs SCOPE Delivery
From `INTENT.md` §Success Criteria:
| # | Criterion | Met? | SCOPE evidence | Remaining gap |
|---|---|---|---|---|
| 1 | Find reusable capabilities before rebuilding | **Yes** | Index, query, catalog, hub `/v1/federated`, federation compose | — |
| 2 | Compare maturity consistently | **Yes** | Vectors, schema, graph relations | — |
| 3 | Distinguish conceptual readiness from delivery | **Yes** | D vs A in model and index | — |
| 4 | Distinguish internal assessment from external evidence | **Yes** | `maturity` vs `external_evidence` | — |
| 5 | Plan prototype/MVP/enhancement/platform work | **Partial** | `report cohorts`, `report gaps`, query | No roadmap views |
| 6 | Identify gaps, duplicates, overlaps, standardization | **Partial** | `overlaps`, `report gaps`, deduped index | Standardization tracker |
| 7 | Track progress to generalized capabilities (D7) | **Partial** | `promotion_history` per entry | No org-wide D7 view |
| 8 | Make reuse normal in architecture work | **Yes** | AGENTS.md, 60-repo rollout, 60/60 publish pass | — |
**Score:** 4 fully met, 4 partial — consistent with **C5** product completeness for
known registry expectations, not yet saturated (**C6**).
---
## 6. Completeness Assessment (SCOPE vs INTENT)
| Area | INTENT expectation | Current delivery | Level |
|---|---|---|---|
| Registry model and principles | Full | INTENT + specs | C5 |
| Maturity standard | Full | `CapabilityMaturityStandard.md` | C5 |
| Entry authoring | Full for MVP | Template + schema + README | C5 |
| Seed registry (reuse-surface) | Examples + meta | 3 entries (meta + 1 stub) | C4 |
| Discovery surface | Machine-readable | Index, query, export, hub API, federated | C5 |
| Validation | Tooling | `validate` + CI | C5 |
| Search / filter | Supported | query, catalog HTML | C4 |
| Federation membership | Cross-repo | 60 hub members; compose + sync | C5 |
| Federation fetch reliability | Published indexes | 60/60 publish pass | C5 |
| Workstation bootstrap | Org-wide visibility | Roster + establish tooling | C4 |
| Agent instructions | Expected | AGENTS.md, tools README | C4 |
| Technical consumption | A3+ tools | CLI A3, hub A4 | C4 |
| Planning analytics | Success criteria | `report cohorts`, `report gaps` | C4 |
| Documentation canon | Concept + living delta | Concept doc + this file | C4 |
**Overall completeness vs INTENT:** **C5C6** for reuse-surface product scope;
**C6** for fetchable federation (60/60 publish pass).
---
## 7. Reliability Assessment (SCOPE vs INTENT)
| Signal | State |
|---|---|
| Automated tests | pytest (registry, federation, hub, hub_sync, stats, reports) |
| CI gates | validate, federation compose, catalog, graph, pytest |
| Production hub | `reuse.coulomb.social` — TLS, 60 registrations |
| Workstation rollout | Deterministic establish/validate; roster tracked |
| Consumer feedback | Informal; 3 entries with structured `consumer_feedback` |
| Known friction | hub SQLite single-replica |
**Overall reliability vs INTENT consumer-evidence framing:** **R3 (Usable)**
CI and production paths support normal agent/operator workflows with documented
limitations. **R4 (Dependable)** blocked on SLO, backup automation, and
consumer telemetry.
---
## 8. Priority Gap Closure Order
### Closed (priorities 124)
See §4 and archived workplans `workplans/archived/`.
### Proposed next (priorities 2528)
| Priority | Gap | Suggested outcome | Status |
|---|---|---|---|
| 25 | Gitea publish visibility | Raw URL HTTP 200 for all roster rows | **Closed** (WP-0015-T01) |
| 26 | Federated ID deduplication | Per-owner removal from reuse-surface index | **Closed** (WP-0015-T02) |
| 27 | Planning analytics + standardization | Gap report or standardization tracker | **Partial** — gap report shipped (T03); tracker deferred |
| 28 | Registry maintenance automation | Interactive `maintain` + `--auto` with llm-connect | **Closed** (WP-0016) |
REUSE-WP-0018-T03: LLM semantic rerank for plan-check llm-connect came up locally (mock provider, 127.0.0.1:8080), unblocking this task. reuse_surface/plan_check.py: build_rerank_prompt/request_rerank/apply_rerank, reusing llm_bridge.execute_prompt/extract_json_object (same pattern as maintain_llm.py's request_maintain_patches). New schema schemas/plan-check-rerank.schema.json rejects malformed responses (missing fields, non-JSON, invented candidate ids outside the input set) rather than guessing. Per design principle 3 (deterministic matches always rank first), apply_rerank appends LLM-scored entries after the deterministic list (kind: 'llm') instead of reordering it -- the trusted base result is identical with or without the rerank pass. Graceful skip via a 'notes' field when LLM_CONNECT_URL is unset or the response is malformed, mirroring maintain.py's no_llm/skip pattern. New --llm-url/--no-llm CLI flags. Also extended plan-check-result.schema.json for 'notes' and the (pre-existing, previously unschema'd) 'filed_capability_request' field. 9 new pytest cases; 89 total pass. Live-verified against the running llm-connect instance: it correctly rejected the mock provider's non-JSON response and surfaced the skip note, with the deterministic verdict and match order completely unaffected. REUSE-WP-0018 is now fully done (T01-T06). Updated docs/IntentScopeGapAnalysis.md priority 29 to Closed and the workplan's own frontmatter status to finished. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:27:39 +02:00
| 29 | Consumption loop (query-before-build) | `plan-check` command + State Hub capability-request bridge | **Closed** — all six tasks (T01T06) shipped 2026-07-07; propagation of the ecosystem convention into the shared rules template is now the-custodian's own follow-through, not a reuse-surface blocker |
**Workplan:** `workplans/REUSE-WP-0016-interactive-registry-maintain.md` (priority 28);
REUSE-WP-0018 T01/T02/T04/T06: plan-check deterministic matching + State Hub bridge T01: specs/PlanCheck.md design doc, plan-check-result.schema.json and reuse-event.schema.json (the latter shared with WP-0019's reuse telemetry). T02: reuse_surface/plan_check.py + 'reuse-surface plan-check' CLI command. Deterministic token-Jaccard matching against registry/indexes/federated.yaml (reuses overlaps.py's TOKEN_RE rather than a second scoring method), with reuse/extend/new verdicts, markdown and --format json output, staleness warning, and --record-outcome JSONL telemetry. T04: reuse_surface/statehub_bridge.py bridges plan-check 'new' verdicts to State Hub capability requests (--file-request) and surfaces open requests with no matching capability (report gaps --check-capability-requests, opt-in to stay offline-safe). Verified against the live local State Hub API; status field (not catalog_entry_id presence) is the correct open/closed signal, and the list endpoint needs a longer timeout than the health check (~7s observed with 5 rows). T06: docs (tools/README.md, RegistryFederation.md, SCOPE.md, IntentScopeGapAnalysis.md priority 29) and an informational CI smoke step. T03 (LLM rerank) not started -- llm-connect isn't running on this workstation. T05 (ecosystem rollout) remains blocked: WP-0017 has drafted entries for all 61 repos but they're still local-only pending its own T05 push/publish pass, so the federated index isn't yet worth rolling out plan-check as ecosystem convention. 16 new tests, all mocked -- no network calls in the default test run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 00:57:18 +02:00
`workplans/REUSE-WP-0015-federation-polish-and-planning-analytics.md` (2527);
`workplans/REUSE-WP-0018-plan-check-consumption-loop.md` (priority 29)
**Assessment:** `history/2026-06-16-intent-scope-assessment.md`
**Follow-up docs:**
- `history/2026-06-16-wp0014-remaining-work-by-repo.md`
- `history/2026-06-16-federation-deduplication-plan.md`
- `registry/federation/local-repo-roster.yaml`
---
## 9. Document Maintenance Rules
- Re-run this analysis when `INTENT.md` or `SCOPE.md` changes materially.
- When a gap closes, update §3§8 and adjust the self-assessment vector in `SCOPE.md`.
- New workplans should reference priority rows from §8.
- Prefer `SCOPE.md` for delivery truth; `INTENT.md` for product truth.
---
## 10. Change Log
| Date | Change |
|---|---|
| 2026-06-15 | Initial analysis after REUSE-WP-0002 |
| 2026-06-15 | WP-00030011 progress; vector milestones |
| 2026-06-16 | WP-0012 closed priorities 1923 |
| 2026-06-17 | WP-0013 closed priority 24 |
| 2026-06-16 | WP-0014 closed priority 18; 60 workstation repos |
| 2026-06-16 | **SCOPE refresh + full INTENT success-criteria mapping**; priorities 2527 proposed |
| 2026-06-16 | Assessment persisted; **REUSE-WP-0015** created for priorities 2527 |
REUSE-WP-0018 T01/T02/T04/T06: plan-check deterministic matching + State Hub bridge T01: specs/PlanCheck.md design doc, plan-check-result.schema.json and reuse-event.schema.json (the latter shared with WP-0019's reuse telemetry). T02: reuse_surface/plan_check.py + 'reuse-surface plan-check' CLI command. Deterministic token-Jaccard matching against registry/indexes/federated.yaml (reuses overlaps.py's TOKEN_RE rather than a second scoring method), with reuse/extend/new verdicts, markdown and --format json output, staleness warning, and --record-outcome JSONL telemetry. T04: reuse_surface/statehub_bridge.py bridges plan-check 'new' verdicts to State Hub capability requests (--file-request) and surfaces open requests with no matching capability (report gaps --check-capability-requests, opt-in to stay offline-safe). Verified against the live local State Hub API; status field (not catalog_entry_id presence) is the correct open/closed signal, and the list endpoint needs a longer timeout than the health check (~7s observed with 5 rows). T06: docs (tools/README.md, RegistryFederation.md, SCOPE.md, IntentScopeGapAnalysis.md priority 29) and an informational CI smoke step. T03 (LLM rerank) not started -- llm-connect isn't running on this workstation. T05 (ecosystem rollout) remains blocked: WP-0017 has drafted entries for all 61 repos but they're still local-only pending its own T05 push/publish pass, so the federated index isn't yet worth rolling out plan-check as ecosystem convention. 16 new tests, all mocked -- no network calls in the default test run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 00:57:18 +02:00
| 2026-06-16 | **REUSE-WP-0016** closed priority 28 (interactive `maintain`, `--auto`, templates) |
| 2026-07-07 | **REUSE-WP-0018** partially closed priority 29 (`plan-check` deterministic matching + State Hub capability-request bridge; LLM rerank and ecosystem rollout remain open) |
REUSE-WP-0018-T03: LLM semantic rerank for plan-check llm-connect came up locally (mock provider, 127.0.0.1:8080), unblocking this task. reuse_surface/plan_check.py: build_rerank_prompt/request_rerank/apply_rerank, reusing llm_bridge.execute_prompt/extract_json_object (same pattern as maintain_llm.py's request_maintain_patches). New schema schemas/plan-check-rerank.schema.json rejects malformed responses (missing fields, non-JSON, invented candidate ids outside the input set) rather than guessing. Per design principle 3 (deterministic matches always rank first), apply_rerank appends LLM-scored entries after the deterministic list (kind: 'llm') instead of reordering it -- the trusted base result is identical with or without the rerank pass. Graceful skip via a 'notes' field when LLM_CONNECT_URL is unset or the response is malformed, mirroring maintain.py's no_llm/skip pattern. New --llm-url/--no-llm CLI flags. Also extended plan-check-result.schema.json for 'notes' and the (pre-existing, previously unschema'd) 'filed_capability_request' field. 9 new pytest cases; 89 total pass. Live-verified against the running llm-connect instance: it correctly rejected the mock provider's non-JSON response and surfaced the skip note, with the deterministic verdict and match order completely unaffected. REUSE-WP-0018 is now fully done (T01-T06). Updated docs/IntentScopeGapAnalysis.md priority 29 to Closed and the workplan's own frontmatter status to finished. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:27:39 +02:00
| 2026-07-07 | **REUSE-WP-0018-T05** closed following REUSE-WP-0017 completion (61 published capabilities); only T03 (LLM rerank) remains open under priority 29 |
REUSE-WP-0019-T06: hub freshness monitoring, docs, close workplan reuse_surface/stats.py: _hub_summary() now reports composed_at, stale, age_days, freshness_threshold_days (REUSE_SURFACE_FRESHNESS_DAYS env, default 7), and a computed stale_warning. New hub_client.hub_federated() backs it. format_stats_markdown surfaces a STALE marker when triggered. .forgejo/workflows/ci.yml: new informational (non-failing) hub freshness check against the live production hub on every push -- prints a ::warning:: annotation when stale, never fails the build. docs/RegistryFederation.md: new section tying together the webhook (T02), scheduled fallback (T03), and freshness visibility (T06) into one explanation. docs/deploy/reuse-kubernetes.md: updated for the T03 Forgejo migration and the now-automated image.yaml build; image promotion checklist updated for the known /health ingress bug (verify via /v1/repos or /v1/federated instead). 14 new pytest cases, 173 total pass. Live-verified against production: reuse-surface stats correctly showed composed_at/age_days for the real federated index. Separately discovered and confirmed (via a live signed webhook test) that reuse-surface-env moving to ExternalSecret/OpenBao custody (railiance-apps commit 706f6c7, found while updating these docs) did not break the T02/T03 webhook -- the synced value still matches what the hub actually uses. REUSE-WP-0019 is now fully complete (T01-T06). SCOPE.md and docs/IntentScopeGapAnalysis.md updated to reflect closure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 00:09:58 +02:00
| 2026-07-07 | **REUSE-WP-0018-T03** closed once `llm-connect` came up locally; priority 29 fully closed (all six T01T06 tasks shipped) |
| 2026-07-08 | **REUSE-WP-0019** finished (all six T01T06 tasks shipped): forge host abstraction + `migrate-host`, hub webhook/staleness tracking, this repo's own Forgejo migration + CI, reuse telemetry store/recording, telemetry aggregation into R-axis evidence, and freshness monitoring in `stats`/CI. Closes SCOPE "not possible yet" item *automatic hub refresh* |