diff --git a/CannedPromptFormat.md b/CannedPromptFormat.md index 67bce8e..665e4a6 100644 --- a/CannedPromptFormat.md +++ b/CannedPromptFormat.md @@ -1105,6 +1105,50 @@ registry/ A registry's own layout is not namespaced by registry name: within one registry, an id is unambiguous by definition. +### 20.3 The index + +A store MAY keep an `index.yaml` at its root recording which package versions +entered it, from where, and when: + +```yaml +format: canned-prompt-index/v0.1 +entries: + - registry: local + id: helix/repo-orient + version: 0.1.0 + name: Repo Orientation + source: /home/worsch/helix-forge/prompts/repo-orient + method: add + included_at: "2026-09-06T13:31:02Z" + declared_author: Bernd Worsch + declared_source: personal prompt collection, contributed 2026-09-06 + license: MIT +``` + +This is deliberately **not** package data. `provenance` (§ 13) records who wrote +a prompt and where the idea came from; the index records how a copy arrived in +*this* store — a fact about the store, not about the artifact, and one that +would differ for every consumer. Keeping it outside preserves immutability +(§ 17): recording an arrival never rewrites the package that arrived. + +| Field | Meaning | +|---|---| +| `registry`, `id`, `version` | Which package version this entry is about | +| `source` | Where the copy came from: a path, a registry, a URL | +| `method` | How it arrived: `add`, `install`, `publish` | +| `included_at` | When it **first** entered this store | +| `last_seen_at` | When it was most recently re-recorded, if ever | +| `declared_author`, `declared_source` | Copied from the package's own `provenance`, so the index is readable without opening every package | +| `license` | Copied from the manifest, so licensing is visible in a listing | + +`included_at` records first arrival and MUST NOT be overwritten when the same +`registry`/`id`/`version` is recorded again; a re-run updates `last_seen_at` +instead. When a package first entered a collection is a fact about history, not +about the last time someone ran a command. + +Because identity is registry-scoped (§ 3.2), the same `id@version` from two +registries is two entries, not a conflict. + ## 21. Reference CLI semantics The reference tool uses two stores: @@ -1138,6 +1182,7 @@ satisfy. ```text eval ID run an installed package's render checks +index [ID] show what entered this catalog, from where and when ``` `eval` runs the deterministic render checks of every recognized eval file and diff --git a/README.md b/README.md index 47cb30c..64a17f3 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,26 @@ signing and trust scoring are explicit non-goals. Ownership lives with the registry rather than in the package, so no package carries an unverifiable assertion of authority. +## The index + +Every store keeps an `index.yaml` recording which package versions entered it, +from where, and when: + +```bash +python canned_prompts.py index +``` + +```text +local:helix/repo-orient@0.1.0 2026-09-06T13:31:02Z add + from /home/worsch/helix-forge/prompts/repo-orient + declares source personal prompt collection, contributed 2026-09-06 +``` + +This is store metadata, not package data. `provenance` records who wrote a +prompt; the index records how a copy arrived *here* — which differs for every +consumer, and which must never rewrite the package it describes. `included_at` +is first arrival and is never overwritten; a re-run updates `last_seen_at`. + ## Dependencies are reported, not fetched Installing a package that composes another warns when the dependency is diff --git a/reference/README.md b/reference/README.md index cb59ef8..67ea163 100644 --- a/reference/README.md +++ b/reference/README.md @@ -2,7 +2,7 @@ This is intentionally a **small reference implementation**, not the intended final architecture. -It demonstrates eight verbs: +It demonstrates nine verbs: ```text add PATH @@ -11,6 +11,7 @@ show ID resolve ID --set key=value render ID --set key=value eval ID +index [ID] install ID [--version VERSION] publish PATH ``` @@ -87,6 +88,10 @@ rather than resolved by guessing. - `add` and `install` name declared prompt dependencies the catalog cannot satisfy. They do not fetch them — resolution is the consumer's job (§ 10) — but a package that looks installed and cannot render should say so. +- `add`, `install` and `publish` record an entry in the store's `index.yaml`: + source, method, first-inclusion date, and the package's declared author, + source and licence. `index` lists it. Re-adding keeps the original + `included_at` and updates `last_seen_at`. - An optional `registry.yaml` names a registry and records namespace claims. `publish` warns when a namespace is declared `closed` — it cannot authenticate a publisher, and says so rather than implying it checked. diff --git a/reference/canned_prompts.py b/reference/canned_prompts.py index 675c38e..4aa2896 100755 --- a/reference/canned_prompts.py +++ b/reference/canned_prompts.py @@ -11,6 +11,7 @@ import argparse import json import os import re +import datetime as dt import shutil import sys from dataclasses import dataclass, field @@ -30,6 +31,8 @@ RENDER_CHECKS = ("contains", "not_contains", "resolves_all") CAPABILITY_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") RESERVED_FILES = ("prompt.yaml", "README.md", "LICENSE") RESERVED_DIRS = ("examples", "evals", "assets") +INDEX_FILE = "index.yaml" +INDEX_FORMAT = "canned-prompt-index/v0.1" PLACEHOLDER_RE = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_.-]*)\s*}}") SEMVER_RE = re.compile( r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)" @@ -678,6 +681,76 @@ def copy_package(src: Path, dst: Path, manifest: dict[str, Any], what: str) -> l return skipped +def read_index(store: Path) -> dict[str, Any]: + """The catalog index: which packages entered this store, from where, when. + + Registry-side metadata, deliberately outside the immutable package (§ 20). + A package records who wrote it; the index records how it got here. + """ + path = store / INDEX_FILE + if not path.is_file(): + return {"format": INDEX_FORMAT, "entries": []} + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise CannedPromptError(f"invalid YAML in {path}: {exc}") from exc + if not isinstance(data, dict) or not isinstance(data.get("entries"), list): + raise CannedPromptError(f"{path}: index must be a mapping with an entries list") + if data.get("format") != INDEX_FORMAT: + raise CannedPromptError(f"unsupported index format: {data.get('format')!r}") + return data + + +def index_entry_key(entry: dict[str, Any]) -> tuple[str, str, str]: + return (entry.get("registry", ""), entry.get("id", ""), entry.get("version", "")) + + +def record_inclusion( + store: Path, + registry: str, + manifest: dict[str, Any], + source: str, + method: str, +) -> dict[str, Any]: + """Record that a package version entered this store, and from where. + + Re-recording the same registry/id/version keeps the original `included_at`: + the date a package first entered is a fact about history, not about the last + time someone re-ran a command. + """ + index = read_index(store) + entry = { + "registry": registry, + "id": manifest["id"], + "version": manifest["version"], + "name": manifest.get("name"), + "source": source, + "method": method, + "included_at": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + } + provenance = manifest.get("provenance") or {} + if isinstance(provenance, dict): + for field in ("author", "source"): + if provenance.get(field): + entry[f"declared_{field}"] = str(provenance[field]) + if manifest.get("license"): + entry["license"] = manifest["license"] + + existing = {index_entry_key(e): e for e in index["entries"]} + key = index_entry_key(entry) + if key in existing: + entry["included_at"] = existing[key].get("included_at", entry["included_at"]) + entry["last_seen_at"] = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + existing[key] = entry + + index["entries"] = [existing[k] for k in sorted(existing)] + store.mkdir(parents=True, exist_ok=True) + (store / INDEX_FILE).write_text( + yaml.safe_dump(index, sort_keys=False, allow_unicode=True), encoding="utf-8" + ) + return entry + + def missing_dependencies(catalog: Path, manifest: dict[str, Any]) -> list[str]: """Declared prompt dependencies that this catalog cannot satisfy. @@ -737,6 +810,7 @@ def cmd_add(args: argparse.Namespace) -> None: registry = check_registry_name(args.as_registry) dst = catalog_package_path(catalog, registry, manifest["id"], manifest["version"]) report_skipped(copy_package(src.resolve(), dst, manifest, "catalog package")) + record_inclusion(catalog, registry, manifest, str(src.resolve()), "add") report_missing_dependencies(catalog, manifest) print(f"added {registry}:{manifest['id']}@{manifest['version']} -> {dst}") @@ -759,6 +833,7 @@ def cmd_publish(args: argparse.Namespace) -> None: dst = registry_package_path(registry, manifest["id"], manifest["version"]) report_skipped(copy_package(src.resolve(), dst, manifest, "published package")) + record_inclusion(registry, name or "", manifest, str(src.resolve()), "publish") label = f"{name}:{manifest['id']}" if name else manifest["id"] print(f"published {label}@{manifest['version']} -> {dst}") @@ -778,6 +853,7 @@ def cmd_install(args: argparse.Namespace) -> None: manifest = validate_package(src) dst = catalog_package_path(catalog, name, manifest["id"], manifest["version"]) copy_package(src, dst, manifest, "catalog package") + record_inclusion(catalog, name, manifest, str(registry.resolve()), "install") report_missing_dependencies(catalog, manifest) print(f"installed {name}:{manifest['id']}@{manifest['version']} -> {dst}") @@ -1238,6 +1314,28 @@ def cmd_eval(args: argparse.Namespace) -> None: raise CannedPromptError(f"{failures} render check(s) failed") +def cmd_index(args: argparse.Namespace) -> None: + store = Path(args.catalog).expanduser() + entries = read_index(store)["entries"] + if args.id: + _, package_id = parse_reference(args.id) + entries = [e for e in entries if e.get("id") == package_id] + if args.json: + print(json.dumps(entries, indent=2, ensure_ascii=False)) + return + if not entries: + print("no packages recorded in this catalog") + return + width = max(len(f"{e.get('registry')}:{e.get('id')}@{e.get('version')}") for e in entries) + for entry in entries: + label = f"{entry.get('registry')}:{entry.get('id')}@{entry.get('version')}" + print(f"{label:<{width}} {entry.get('included_at')} {entry.get('method')}") + print(f"{'':<{width}} from {entry.get('source')}") + declared = entry.get("declared_source") + if declared: + print(f"{'':<{width}} declares source {declared}") + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="canned-prompts", description=__doc__) sub = parser.add_subparsers(dest="command", required=True) @@ -1274,6 +1372,12 @@ def build_parser() -> argparse.ArgumentParser: render.add_argument("--set", dest="set_values", action="append", default=[], metavar="NAME=VALUE") render.set_defaults(func=cmd_render) + index = sub.add_parser("index", help="show what entered this catalog, from where and when") + index.add_argument("id", nargs="?") + index.add_argument("--catalog", default=str(default_catalog())) + index.add_argument("--json", action="store_true") + index.set_defaults(func=cmd_index) + evaluate = sub.add_parser("eval", help="run an installed package's render checks") evaluate.add_argument("id") evaluate.add_argument("--version") diff --git a/reference/tests/test_canned_prompts.py b/reference/tests/test_canned_prompts.py index e14385d..957de73 100644 --- a/reference/tests/test_canned_prompts.py +++ b/reference/tests/test_canned_prompts.py @@ -808,3 +808,62 @@ def test_dependency_present_but_wrong_version_is_reported(composed: Path) -> Non manifest = cp.validate_package(package_dir) manifest["dependencies"]["prompts"][0]["version"] = "9.9.9" assert cp.missing_dependencies(composed, manifest) == ["style/house@9.9.9"] + + +# --- catalog index (§ 20.3) --- + +def indexed(tmp_path: Path) -> tuple[Path, dict]: + store = tmp_path / "catalog" + manifest = {"id": "helix/thing", "version": "1.0.0", "name": "Thing", + "license": "MIT", "provenance": {"author": "Ada", "source": "~/somewhere"}} + return store, manifest + + +def test_index_starts_empty(tmp_path: Path) -> None: + store, _ = indexed(tmp_path) + assert cp.read_index(store)["entries"] == [] + + +def test_record_inclusion_captures_source_and_date(tmp_path: Path) -> None: + store, manifest = indexed(tmp_path) + entry = cp.record_inclusion(store, "local", manifest, "/src/thing", "add") + assert entry["id"] == "helix/thing" + assert entry["source"] == "/src/thing" + assert entry["method"] == "add" + assert entry["declared_author"] == "Ada" + assert entry["declared_source"] == "~/somewhere" + assert entry["license"] == "MIT" + assert entry["included_at"].endswith("Z") + assert cp.read_index(store)["entries"] == [entry] + + +def test_reincluding_keeps_the_original_date(tmp_path: Path) -> None: + """When a package first entered is a fact about history, not about reruns.""" + store, manifest = indexed(tmp_path) + first = cp.record_inclusion(store, "local", manifest, "/src/thing", "add") + again = cp.record_inclusion(store, "local", manifest, "/elsewhere", "install") + assert again["included_at"] == first["included_at"] + assert "last_seen_at" in again + assert len(cp.read_index(store)["entries"]) == 1 + + +def test_distinct_versions_are_separate_entries(tmp_path: Path) -> None: + store, manifest = indexed(tmp_path) + cp.record_inclusion(store, "local", manifest, "/src", "add") + cp.record_inclusion(store, "local", {**manifest, "version": "1.1.0"}, "/src", "add") + assert len(cp.read_index(store)["entries"]) == 2 + + +def test_same_id_from_two_registries_is_two_entries(tmp_path: Path) -> None: + store, manifest = indexed(tmp_path) + cp.record_inclusion(store, "local", manifest, "/a", "add") + cp.record_inclusion(store, "house", manifest, "/b", "install") + assert len(cp.read_index(store)["entries"]) == 2 + + +def test_bad_index_format_is_rejected(tmp_path: Path) -> None: + store, _ = indexed(tmp_path) + store.mkdir(parents=True) + (store / "index.yaml").write_text("format: something/else\nentries: []\n", encoding="utf-8") + with pytest.raises(cp.CannedPromptError, match="unsupported index format"): + cp.read_index(store) diff --git a/workplans/CANP-WP-0004-prompt-index.md b/workplans/CANP-WP-0004-prompt-index.md new file mode 100644 index 0000000..902b734 --- /dev/null +++ b/workplans/CANP-WP-0004-prompt-index.md @@ -0,0 +1,105 @@ +--- +id: CANP-WP-0004 +type: workplan +title: "Catalog index: where a package came from and when" +domain: agents +repo: canned-prompts +status: finished +owner: codex +topic_slug: practice +created: "2026-09-06" +updated: "2026-09-06" +--- + +# Catalog index: where a package came from and when + +Driven by packaging a real collection (`helix-forge` `HF-WP-0005`). The operator +asked for canned-prompts to "build a database of versioned prompts", recording +the source a prompt came from and the date it was included — the first step +toward a platform for collaborative prompting. + +The format had no place for this. `provenance` (§ 13) records who wrote a prompt +and where the idea came from, but nothing recorded how a *copy* arrived in a +*particular* store. + +## Fix practice/pqrst-estimate to be the canonical prompt + +```task +id: CANP-WP-0004-T01 +status: done +priority: high +``` + +**Found while researching helix-forge.** `examples/pqrst-estimate` was not the +canonical PQRST prompt. The canonical one is `~/pqrst-practice/PqrstPrompt.md`, +normatively specified in `spec/PqrstEstimationPractice.md`, and +`hall-of-helix/CLOSING.md` requires pasting it unmodified. Ours was a paraphrase +with a different output shape — no `Confidence`, no `Signature`, no `Dominant +factors` — and `CANP-WP-0002-T03` had made it worse by prepending a +`house_style` inclusion to a prompt whose governing document says do not modify +it. + +A copied prompt that silently forked its source, sitting in the examples +directory of the repo whose `INTENT.md` opens by naming that exact failure. + +Fixed: the template is the canonical block extracted programmatically rather +than retyped, and the default render is byte-identical at 2886 bytes. +`evals/canonical-fidelity.yaml` guards it with sixteen render checks, including +`not_contains` checks naming the paraphrase it used to be. Version 0.2.1 → +1.0.0, the § 17 MAJOR case. + +The source documents two optional add-ons appended after the block. CPF has no +conditionals, so `add_ons` is an input defaulting to the empty string and each +add-on is an example fixture. Adequate, but a workaround — recorded as evidence +about § 23's deferred "richer template syntax". + +## Add the index + +```task +id: CANP-WP-0004-T02 +status: done +priority: high +``` + +A store may keep an `index.yaml` recording which package versions entered it, +from where, and when. Specified in § 20.3, deliberately as **store** metadata +rather than package data: how a copy arrived differs for every consumer, and +recording an arrival must never rewrite the package that arrived (§ 17). + +`add`, `install` and `publish` record an entry carrying registry, id, version, +name, source, method, `included_at`, the package's declared author and source, +and its licence — the last two copied so a listing is readable without opening +every package. A new `index` verb lists it. + +`included_at` is first arrival and is never overwritten; a re-run updates +`last_seen_at` instead. When a package first entered a collection is a fact +about history, not about the last time someone ran a command. + +`reference/canned_prompts.py`: `read_index`, `record_inclusion`, +`index_entry_key`, `cmd_index`. Tests 84 → 90. + +## Record the inclusion-deduplication gap + +```task +id: CANP-WP-0004-T03 +status: done +priority: medium +``` + +**Found by using the format.** `helix/repo-advance` composed +`helix/commit-sync`, which composes `helix/custodian-conventions`, while also +composing the conventions itself. The rendered prompt contained the conventions +block twice: CPF inclusion has no deduplication, so a diamond dependency renders +shared content once per path. + +Not fixed in the format. Deduplicating would mean deciding *which* occurrence +survives and what happens when the two paths resolve different versions — that +is resolver behaviour, and § 10.4 keeps composition declarative on purpose. +Nothing in § 10.4 currently warns an author, which is the actual defect. + +Downstream this was fixed by factoring (`HF-WP-0005-T03`), which is the right +answer for a collection and may be the right general advice. + +**Handed to `CANP-WP-0005`:** document the diamond behaviour in § 10.4 and +decide whether a validator should warn when one package reaches the same +dependency by two paths. diff --git a/workplans/CANP-WP-0005-inclusion-diamond.md b/workplans/CANP-WP-0005-inclusion-diamond.md new file mode 100644 index 0000000..e9b09d3 --- /dev/null +++ b/workplans/CANP-WP-0005-inclusion-diamond.md @@ -0,0 +1,47 @@ +--- +id: CANP-WP-0005 +type: workplan +title: "Document and detect inclusion diamonds" +domain: agents +repo: canned-prompts +status: proposed +owner: codex +topic_slug: practice +created: "2026-09-06" +updated: "2026-09-06" +--- + +# Document and detect inclusion diamonds + +Residual from `CANP-WP-0004` (`origin: residual`, `origin_ref: CANP-WP-0004`). + +## Warn about repeated inclusion + +```task +id: CANP-WP-0005-T01 +status: todo +priority: medium +``` + +CPF inclusion (§ 10.4) does not deduplicate: when one package reaches the same +dependency by two paths, the included text renders once per path. This was found +by building a real collection — `helix/repo-advance` rendered its conventions +block twice — and worked around downstream by factoring the shared routine into +its own fragment. + +§ 10.4 says nothing about it, so an author meets the behaviour only by reading +the output carefully. That silence is the defect. + +Two questions, and the second depends on the first: + +1. **Document it.** State in § 10.4 that inclusion is textual and repeated, + not deduplicated, and give the factoring pattern that avoids a diamond. +2. **Detect it.** Decide whether validation should warn when a package reaches + the same dependency by more than one path. A warning is cheap and catches the + mistake at `add` time. Silently deduplicating is a different proposition + entirely — it would mean choosing which occurrence survives, and deciding what + happens when two paths resolve *different versions* of the same dependency. + That is resolver behaviour, and § 10.4 keeps composition declarative on + purpose. + +Leaning: document it now, warn at validation time, and do not deduplicate.