markitect-main/roadmap/infospace-s3-closeout/PLAN.md

# Infospace Tooling — Stage 3 Close-out

## Context

Stages 1 and 2 of the infospace tooling roadmap are complete. Stage 3 used the
Wealth of Nations / VSM example to validate the tooling end-to-end. Most of S3
is done; this workstream finishes the remaining tasks, addresses deferred cleanup,
and formally closes the roadmap.

**Parent roadmap:** `roadmap/infospace-tooling/PLAN.md`
**Example location:** `examples/infospace-with-history/`

**Status: CLOSED (2026-04-22).** All acceptance criteria except the cosmetic
per-chapter history (C.7) are met. Final metrics: 988 entities, 988 evaluations,
6/6 viability thresholds PASS (`per_entity_mean = 3.957`). Tooling work that
came out of this close-out landed as commits `c0615c2d` (gemini retry,
unified skip-existing, non-destructive metrics I/O) and `d44a4cd3`
(`infospace entity` lookup, `evaluate --model-fallback`, `llm-check`
stale-key advisory, `build_state` type guard).

### State at workstream open (2026-02-26)

| Item | Status |
|------|--------|
| S3.1 Migrate example to infospace config | ✅ Done |
| S3.3 Per-entity eval batch | ✅ 985/988 complete; metrics.yaml updated |
| S3.4 Tutorial rewrite | ✅ Done |
| S3.5 Supply-chain-vsm composition demo | ✅ Done |
| S3.2 Clean per-chapter git history | ⏳ Deferred — included here |
| 3 missing evaluations | ⏳ Outstanding |
| 4 follow-up items (commit b055c8d7) | ⏳ Outstanding |

### State at workstream close (2026-04-22)

| Task | Status |
|------|--------|
| C.1 Complete 3 missing entity evaluations | ✅ Done (commit f325f89d) |
| C.2 Run eval-summary and verify viability  | ✅ Done — 6/6 PASS |
| C.3 Refresh metrics report (988 entities)  | ✅ Done — snapshot `090bb961` |
| C.4 Document advanced usage patterns        | ✅ Done — `examples/infospace-with-history/docs/advanced-usage.md` |
| C.5 Composition-examples documentation      | ✅ Done — `docs/composition-guide.md` |
| C.6 Performance benchmarking note           | ✅ Done — `examples/infospace-with-history/docs/performance-notes.md` |
| C.7 Clean per-chapter git history           | ⏭️ Deferred indefinitely — see note below |
| C.8 Formally close S3 roadmap               | ✅ This commit |

**C.7 disposition.** The task assumed a pre-existing `clean-example-history`
branch with chapters 1–8 already committed; that branch no longer exists in
the repo. The task is explicitly cosmetic ("does not change output files"),
and the output files themselves are canonical. Reconstructing a 35-commit
per-chapter history from scratch would be archaeological rather than useful.
Closing as "won't do" unless a specific archival need surfaces. If revisited,
entities can be grouped by their `## Source Chapter` markdown section to
reconstruct chapter membership.

---

## Tasks

### C.1 — Complete the 3 missing entity evaluations

985 of 988 entities have evaluation files. Identify and evaluate the remaining 3.

```bash
cd examples/infospace-with-history
# Identify missing slugs
comm -23 \
  <(ls output/entities/*.md | xargs -I{} basename {} .md | sort) \
  <(ls output/evaluations/*.md | xargs -I{} basename {} .md | sort)
# Evaluate each missing entity individually
markitect infospace evaluate --entity <slug> --provider openrouter
```

**Acceptance:** `ls output/evaluations/*.md | wc -l` returns 988.

---

### C.2 — Run eval-summary and verify viability

Run the aggregation command to update per_entity_mean from all 988 evaluations,
then check all 6 viability gates pass.

```bash
cd examples/infospace-with-history
unset OPENROUTER_API_KEY  # stale env var guard
markitect infospace eval-summary --update-metrics
markitect infospace viability
```

Current sample reading (985 entities): `per_entity_mean = 3.956` against threshold 3.5.
Expected: all 6 metrics pass.

**Acceptance:** `markitect infospace viability` exits 0 and shows 6/6 PASS.

---

### C.3 — Refresh the metrics report

The metrics report was generated from chapters 1–4 only. Regenerate it from
the full 988-entity set.

```bash
cd examples/infospace-with-history
markitect infospace check --provider openrouter   # or reuse existing check outputs
markitect infospace history                        # confirm snapshot recorded
```

**Acceptance:** `output/metrics/metrics.yaml` reflects all 988 entities; a dated
snapshot exists in the metrics history.

---

### C.4 — Document advanced usage patterns

Write `examples/infospace-with-history/docs/advanced-usage.md` covering:

- Incremental evaluation (adding entities after initial run, skip-if-exists behaviour)
- Re-evaluating after guideline changes (`--force` flag)
- Interpreting per-entity score distributions and identifying outliers
- Using `markitect infospace entities --sort-by score` to triage low scorers
- Reading and acting on collection check outputs (redundancy pairs, coverage gaps)

**Acceptance:** File exists with ≥ 4 documented patterns, each with a worked command example.

---

### C.5 — Add composition examples to documentation

Document how the supply-chain-vsm example (`examples/supply-chain-vsm/`) demonstrates
composition. Add a `docs/composition-guide.md` covering:

- What composition means (discipline binding)
- How supply-chain-vsm binds WoN as a discipline
- How to create a new infospace that uses an existing one as a discipline
- Viability requirement: the discipline must pass its own thresholds before binding

Reference `examples/supply-chain-vsm/` throughout.

**Acceptance:** `docs/composition-guide.md` exists and links to supply-chain-vsm.

---

### C.6 — Performance benchmarking note

Rather than a full benchmarking guide (out of scope for a 988-entity example),
record observed timings in a `docs/performance-notes.md`:

- Eval batch duration (~4 hrs for 988 entities via OpenRouter)
- Tokens per entity (rough estimate from usage logs)
- Embedding cache hit rate after first run
- Recommendation: provider choice (OpenRouter vs Gemini) for different dataset sizes

**Acceptance:** File exists with at least 4 concrete measurements or estimates.

---

### C.7 — S3.2: Clean per-chapter git history (deferred cleanup)

Create a clean branch where each of the 35 processed chapters has its own commit.
Chapters 1–8 are already done on branch `clean-example-history`; 27 remain.

This is a cosmetic/archival task — it does not change output files.

```bash
git checkout clean-example-history
# For each remaining chapter (9–35):
#   cherry-pick or re-commit the chapter output files with a per-chapter message
git log --oneline clean-example-history  # verify 35 chapter commits
```

**Acceptance:** Branch `clean-example-history` has exactly 35 chapter commits
(one per chapter), rebased onto current main.

**Note:** This task can be done independently of C.1–C.6. Low urgency — do last.

---

### C.8 — Formally close the S3 roadmap

Update `roadmap/infospace-tooling/PLAN.md` to mark all S3 tasks as complete.
Add a close-out summary at the top of the file with final metrics and date.
Commit with a `docs(roadmap)` message.

**Acceptance:** PLAN.md header shows all stages complete; committed to main.

---

## Task order

```
C.1 → C.2 → C.3
              ↓
         C.4, C.5, C.6 (parallel)
              ↓
             C.8
C.7 (independent, do last)
```

## Out of scope

- Adding new entities or chapters (the WoN example is complete at 988 entities)
- Re-running collection checks from scratch (existing results are valid)
- Publishing the example as a standalone dataset