diff --git a/CannedPromptFormat.md b/CannedPromptFormat.md index 6b4d93c..cb5be11 100644 --- a/CannedPromptFormat.md +++ b/CannedPromptFormat.md @@ -1203,6 +1203,37 @@ about the last time someone ran a command. Because identity is registry-scoped (§ 3.2), the same `id@version` from two registries is two entries, not a conflict. +### 20.4 A URL is not a registry + +§ 20 lists "an HTTP service" among the things a registry may be. Implementing +one showed that this conflates two different objects. + +A **filesystem registry is one registry**: it is a directory, and § 20.1 names +it from that directory. An **HTTP service hosts several** — the reference +service holds `local`, `helix` and any number of others behind one base URL. A +URL therefore addresses a *host of registries*, and cannot name one the way a +directory does. + +Consequently, when the transport is HTTP the registry MUST be named separately +from the address: + +- **publishing** names the target registry explicitly, because the service + cannot infer which of its registries a package belongs to; +- **installing** names it in a qualified reference (§ 3.2), and a bare id + matching more than one is reported as ambiguous exactly as it is locally. + +A service MAY instead expose each registry at its own URL prefix, making the +address name a registry again. This specification does not require either +arrangement, but a consumer MUST NOT assume that one address means one +registry. + +Everything else survived the transport change unaltered, which is what INTENT +principle 10 claims: identity and its ambiguity rules (§ 3.2), immutability of a +published `@` (§ 17), strict packaging (§ 2), validation, and the +index (§ 20.3) all behave the same over HTTP as on a filesystem. The registry +naming above is the single point where the transport is visible in the +semantics, and it is a gap in this section rather than in the format's core. + ## 21. Reference CLI semantics The reference tool uses two stores: diff --git a/README.md b/README.md index de47959..b2e58c4 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,24 @@ signing and trust scoring are explicit non-goals. Ownership lives with the registry rather than in the package, so no package carries an unverifiable assertion of authority. +## Registries over HTTP + +`--registry` takes a URL as readily as a path: + +```bash +export CANNED_PROMPTS_PUBLISH_TOKEN=... +python canned_prompts.py publish ../examples/pqrst-estimate \ + --registry https://registry.example --as helix +python canned_prompts.py install helix:practice/pqrst-estimate \ + --registry https://registry.example +``` + +Publishing needs `--as`, because a URL addresses a service that hosts several +registries and so does not name one the way a directory does (§ 20.4). That is +the *only* place the transport is visible: identity and ambiguity, immutability +of a published version, strict packaging, validation and the index all behave +the same over HTTP as on a filesystem. + ## The index Every store keeps an `index.yaml` recording which package versions entered it, diff --git a/reference/README.md b/reference/README.md index c7a2abe..19aa92d 100644 --- a/reference/README.md +++ b/reference/README.md @@ -100,6 +100,12 @@ rather than resolved by guessing. source, method, first-inclusion date, and the package's declared author, source and licence. `index` lists it. Re-adding keeps the original `included_at` and updates `last_seen_at`. +- `--registry` accepts an `http(s)://` URL as well as a path. HTTP uses the + stdlib, so this stays dependency-light. `CANNED_PROMPTS_PUBLISH_TOKEN` + supplies the bearer token when publishing. +- Publishing to an HTTP registry needs `--as NAME`: a URL addresses a service + that **hosts several registries**, so unlike a directory it does not name one + (§ 20.4). Installing names it in a qualified reference. - An optional `registry.yaml` names a registry and records namespace claims. `publish` warns when a namespace is declared `closed` — it cannot authenticate a publisher, and says so rather than implying it checked. diff --git a/reference/canned_prompts.py b/reference/canned_prompts.py index fd9ffc9..e257980 100755 --- a/reference/canned_prompts.py +++ b/reference/canned_prompts.py @@ -11,9 +11,13 @@ import argparse import json import os import re +import base64 import datetime as dt import shutil import sys +import tempfile +import urllib.error +import urllib.request from dataclasses import dataclass, field from pathlib import Path from typing import Any, Iterable @@ -772,6 +776,71 @@ def record_inclusion( return entry +# --- HTTP registries (§ 20) ---------------------------------------------- +# +# § 20 lists "an HTTP service" as a valid registry. Implementing one surfaced a +# distinction the specification does not draw: a filesystem registry *is* one +# registry, but an HTTP service *hosts several* — the reference service keeps +# `local`, `helix` and others behind a single base URL. So a URL does not name a +# registry the way a directory does, and the registry must be named separately: +# by `--as` when publishing, and by a qualified reference when installing. +# Recorded as a finding rather than worked around silently. + + +def is_http_registry(value: str) -> bool: + return value.startswith("http://") or value.startswith("https://") + + +def http_request( + url: str, method: str = "GET", payload: dict[str, Any] | None = None +) -> dict[str, Any]: + """A registry call. stdlib only, so `reference/` stays dependency-light.""" + data = json.dumps(payload).encode("utf-8") if payload is not None else None + request = urllib.request.Request(url, data=data, method=method) + request.add_header("Accept", "application/json") + if data is not None: + request.add_header("Content-Type", "application/json") + token = os.environ.get("CANNED_PROMPTS_PUBLISH_TOKEN") + if token: + request.add_header("Authorization", f"Bearer {token}") + try: + with urllib.request.urlopen(request) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as exc: + body = exc.read().decode("utf-8", "replace") + try: + detail = json.loads(body).get("detail", body) + except json.JSONDecodeError: + detail = body + raise CannedPromptError(f"registry refused ({exc.code}): {detail}") from exc + except urllib.error.URLError as exc: + raise CannedPromptError(f"registry unreachable: {exc.reason}") from exc + + +def encode_files(package_dir: Path, manifest: dict[str, Any]) -> dict[str, Any]: + """The package's files, in the shape the service's archive returns.""" + files: dict[str, Any] = {} + for relative in sorted(package_members(package_dir, manifest)): + raw = (package_dir / relative).read_bytes() + try: + files[relative] = {"text": raw.decode("utf-8")} + except UnicodeDecodeError: + files[relative] = {"base64": base64.b64encode(raw).decode()} + return files + + +def decode_files(files: dict[str, Any], destination: Path) -> None: + for relative, item in files.items(): + if relative.startswith("/") or ".." in Path(relative).parts: + raise CannedPromptError(f"unsafe path from registry: {relative}") + target = destination / relative + target.parent.mkdir(parents=True, exist_ok=True) + if "text" in item: + target.write_text(item["text"], encoding="utf-8") + else: + target.write_bytes(base64.b64decode(item["base64"])) + + def missing_dependencies(catalog: Path, manifest: dict[str, Any]) -> list[str]: """Declared prompt dependencies that this catalog cannot satisfy. @@ -878,6 +947,27 @@ def cmd_add(args: argparse.Namespace) -> None: def cmd_publish(args: argparse.Namespace) -> None: src = Path(args.path) manifest = validate_package(src) + + if is_http_registry(args.registry): + # A URL hosts several registries, so it cannot name one on its own. + if not args.as_registry: + raise CannedPromptError( + "publishing to an HTTP registry needs --as NAME: a URL addresses a " + "service that hosts several registries, so unlike a directory it " + "does not name one (§ 20)" + ) + result = http_request( + args.registry.rstrip("/") + "/packages", + method="POST", + payload={ + "registry": check_registry_name(args.as_registry), + "source": str(src.resolve()), + "files": encode_files(src.resolve(), manifest), + }, + ) + print(f"published {result['reference']} -> {args.registry}") + return + registry = Path(args.registry).expanduser() check_legacy_registry(registry) name = registry_name(registry) if registry.is_dir() else None @@ -900,9 +990,30 @@ def cmd_publish(args: argparse.Namespace) -> None: def cmd_install(args: argparse.Namespace) -> None: + catalog = Path(args.catalog).expanduser() + + if is_http_registry(args.registry): + reference = args.id + (f"@{args.version}" if args.version else "") + archive = http_request( + args.registry.rstrip("/") + "/archives/" + reference + ) + source_registry, _, rest = archive["reference"].partition(":") + with tempfile.TemporaryDirectory() as tmp: + staged = Path(tmp) / "package" + staged.mkdir() + decode_files(archive["files"], staged) + manifest = validate_package(staged) + dst = catalog_package_path( + catalog, source_registry, manifest["id"], manifest["version"] + ) + copy_package(staged, dst, manifest, "catalog package") + record_inclusion(catalog, source_registry, manifest, args.registry, "install") + report_missing_dependencies(catalog, manifest) + print(f"installed {archive['reference']} -> {dst}") + return + registry = Path(args.registry).expanduser() check_legacy_registry(registry) - catalog = Path(args.catalog).expanduser() name = registry_name(registry) qualifier, package_id = parse_reference(args.id) @@ -1465,9 +1576,16 @@ def build_parser() -> argparse.ArgumentParser: install.add_argument("--registry", default=str(default_registry())) install.set_defaults(func=cmd_install) - publish = sub.add_parser("publish", help="publish a package to a filesystem registry") + publish = sub.add_parser("publish", help="publish a package to a registry") publish.add_argument("path") publish.add_argument("--registry", default=str(default_registry())) + publish.add_argument( + "--as", + dest="as_registry", + default=None, + metavar="REGISTRY", + help="registry name to publish into; required for an HTTP registry", + ) publish.set_defaults(func=cmd_publish) return parser diff --git a/reference/tests/test_canned_prompts.py b/reference/tests/test_canned_prompts.py index c2abe58..187a6e1 100644 --- a/reference/tests/test_canned_prompts.py +++ b/reference/tests/test_canned_prompts.py @@ -1,3 +1,4 @@ +import argparse from pathlib import Path import pytest @@ -942,3 +943,68 @@ def test_explicit_registry_path_is_never_second_guessed(tmp_path: Path, monkeypa monkeypatch.setenv("CANNED_PROMPTS_HOME", str(tmp_path)) (tmp_path / "registry").mkdir() cp.check_legacy_registry(tmp_path / "somewhere-else") + + +# --- HTTP registries (§ 20.4) --- + +def test_http_registry_detection() -> None: + assert cp.is_http_registry("https://registry.example") + assert cp.is_http_registry("http://localhost:8878") + assert not cp.is_http_registry("/home/user/.canned-prompts/default") + + +def test_encode_decode_round_trips_a_package(tmp_path: Path) -> None: + """The wire shape must reproduce the package byte for byte.""" + src = write_evaluated(tmp_path / "src", RUBRIC) + (src / "assets").mkdir() + (src / "assets" / "binary.dat").write_bytes(bytes(range(256))) + manifest = cp.validate_package(src) + + files = cp.encode_files(src, manifest) + assert "base64" in files["assets/binary.dat"], "non-UTF-8 must not go as text" + + out = tmp_path / "out" + out.mkdir() + cp.decode_files(files, out) + for relative in cp.package_members(src, manifest): + assert (out / relative).read_bytes() == (src / relative).read_bytes() + + +def test_decode_refuses_path_traversal(tmp_path: Path) -> None: + """A registry is untrusted input (§ 19); it must not write outside the package.""" + with pytest.raises(cp.CannedPromptError, match="unsafe path"): + cp.decode_files({"../escape.txt": {"text": "no"}}, tmp_path) + + +def test_publish_to_http_requires_a_registry_name(tmp_path: Path, monkeypatch) -> None: + """§ 20.4: a URL hosts several registries, so it cannot name one.""" + src = write_pkg(tmp_path / "p", BASE + "inputs:\n - name: greeting\n required: false\n default: hi\n") + args = argparse.Namespace( + path=str(src), registry="https://registry.example", as_registry=None + ) + with pytest.raises(cp.CannedPromptError, match="needs --as NAME"): + cp.cmd_publish(args) + + +def test_http_errors_surface_the_registry_detail(monkeypatch) -> None: + import urllib.error, io + + def raise_conflict(*a, **k): + raise urllib.error.HTTPError( + "u", 409, "Conflict", {}, io.BytesIO(b'{"detail":"already exists"}') + ) + + monkeypatch.setattr(cp.urllib.request, "urlopen", raise_conflict) + with pytest.raises(cp.CannedPromptError, match="registry refused \\(409\\): already exists"): + cp.http_request("https://registry.example/packages") + + +def test_unreachable_registry_is_reported_clearly(monkeypatch) -> None: + import urllib.error + + def raise_urlerror(*a, **k): + raise urllib.error.URLError("connection refused") + + monkeypatch.setattr(cp.urllib.request, "urlopen", raise_urlerror) + with pytest.raises(cp.CannedPromptError, match="registry unreachable"): + cp.http_request("https://registry.example/packages") diff --git a/workplans/CANP-WP-0006-hosted-registry-service.md b/workplans/CANP-WP-0006-hosted-registry-service.md index 8ea3f09..568f63e 100644 --- a/workplans/CANP-WP-0006-hosted-registry-service.md +++ b/workplans/CANP-WP-0006-hosted-registry-service.md @@ -206,7 +206,7 @@ check` now reports no drift. ```task id: CANP-WP-0006-T05 -status: todo +status: done priority: medium state_hub_task_id: "8dacbb68-aaa2-5f91-942d-a0667f1f1fc7" ``` @@ -220,6 +220,27 @@ claims. If they do not survive it, that is a finding about the format, not a bug to paper over in the client. +**Done, and it produced exactly one finding.** + +*What survived unaltered*, verified against the running service: identity and +its ambiguity rules (a bare id in two registries came back 409 over HTTP just as +it does locally), immutability of a published `@` (identical +content accepted, changed content refused, a version bump accepted), strict +packaging, validation, and the index. A package published and installed over +HTTP was **byte-identical** to its source — `diff -r` clean — and its +`canonical-fidelity` eval still passed after the round trip. + +*What did not*: **a URL is not a registry.** A filesystem registry *is* one +registry and § 20.1 names it from its directory; an HTTP service *hosts +several* behind one base URL. So the address cannot name the registry, and it +has to be named separately — `--as` when publishing, a qualified reference when +installing. Recorded as § 20.4 rather than papered over in the client, because +the gap is in the specification's list of registry kinds, not in the CLI. + +Implemented with `urllib` rather than a library, so `reference/` keeps PyYAML as +its only dependency. Registry responses are treated as untrusted input (§ 19): +`decode_files` refuses path traversal, with a test. + ## Image and smoke contract ```task