From c076d8395ecf62295001c825fc2b5c45c7817384 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 6 Sep 2026 20:39:04 +0200 Subject: [PATCH] CANP-WP-0006 T05: HTTP registries, and one finding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --registry now accepts an http(s):// URL as well as a path, implemented with urllib so reference/ keeps PyYAML as its only dependency. This was the first real test of INTENT principle 10's claim that a hosted registry layers on without changing package semantics. Verified against the running service, almost everything survived the transport unaltered: identity and its ambiguity rules (a bare id in two registries returns 409 over HTTP just as it does locally), immutability of a published id@version (identical content accepted, changed content refused, version bump accepted), strict packaging, validation, and the index. A package published and then installed over HTTP was byte-identical to its source — diff -r clean — and its canonical-fidelity eval still passed after the round trip. One thing did not survive: a URL is not a registry. A filesystem registry IS one registry and section 20.1 names it from its directory; an HTTP service HOSTS SEVERAL behind one base URL. The address therefore cannot name the registry, so it must be named separately — --as when publishing, a qualified reference when installing. Recorded as section 20.4 rather than worked around silently in the client, because the gap is in the specification's list of registry kinds, not in the CLI. Publishing to an HTTP registry without --as fails with that explanation rather than guessing a registry name. Registry responses are treated as untrusted input (section 19): decode_files refuses path traversal, with a test. The wire shape round-trips binary content through base64, also tested. Reference tests 99 -> 105. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM Assistant: claude-code Assistant-Model: opus Assistant-Process: 388925@bnt-lap001 Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502 --- CannedPromptFormat.md | 31 +++++ README.md | 18 +++ reference/README.md | 6 + reference/canned_prompts.py | 122 +++++++++++++++++- reference/tests/test_canned_prompts.py | 66 ++++++++++ .../CANP-WP-0006-hosted-registry-service.md | 23 +++- 6 files changed, 263 insertions(+), 3 deletions(-) diff --git a/CannedPromptFormat.md b/CannedPromptFormat.md index 6b4d93c..cb5be11 100644 --- a/CannedPromptFormat.md +++ b/CannedPromptFormat.md @@ -1203,6 +1203,37 @@ about the last time someone ran a command. Because identity is registry-scoped (§ 3.2), the same `id@version` from two registries is two entries, not a conflict. +### 20.4 A URL is not a registry + +§ 20 lists "an HTTP service" among the things a registry may be. Implementing +one showed that this conflates two different objects. + +A **filesystem registry is one registry**: it is a directory, and § 20.1 names +it from that directory. An **HTTP service hosts several** — the reference +service holds `local`, `helix` and any number of others behind one base URL. A +URL therefore addresses a *host of registries*, and cannot name one the way a +directory does. + +Consequently, when the transport is HTTP the registry MUST be named separately +from the address: + +- **publishing** names the target registry explicitly, because the service + cannot infer which of its registries a package belongs to; +- **installing** names it in a qualified reference (§ 3.2), and a bare id + matching more than one is reported as ambiguous exactly as it is locally. + +A service MAY instead expose each registry at its own URL prefix, making the +address name a registry again. This specification does not require either +arrangement, but a consumer MUST NOT assume that one address means one +registry. + +Everything else survived the transport change unaltered, which is what INTENT +principle 10 claims: identity and its ambiguity rules (§ 3.2), immutability of a +published `@` (§ 17), strict packaging (§ 2), validation, and the +index (§ 20.3) all behave the same over HTTP as on a filesystem. The registry +naming above is the single point where the transport is visible in the +semantics, and it is a gap in this section rather than in the format's core. + ## 21. Reference CLI semantics The reference tool uses two stores: diff --git a/README.md b/README.md index de47959..b2e58c4 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,24 @@ signing and trust scoring are explicit non-goals. Ownership lives with the registry rather than in the package, so no package carries an unverifiable assertion of authority. +## Registries over HTTP + +`--registry` takes a URL as readily as a path: + +```bash +export CANNED_PROMPTS_PUBLISH_TOKEN=... +python canned_prompts.py publish ../examples/pqrst-estimate \ + --registry https://registry.example --as helix +python canned_prompts.py install helix:practice/pqrst-estimate \ + --registry https://registry.example +``` + +Publishing needs `--as`, because a URL addresses a service that hosts several +registries and so does not name one the way a directory does (§ 20.4). That is +the *only* place the transport is visible: identity and ambiguity, immutability +of a published version, strict packaging, validation and the index all behave +the same over HTTP as on a filesystem. + ## The index Every store keeps an `index.yaml` recording which package versions entered it, diff --git a/reference/README.md b/reference/README.md index c7a2abe..19aa92d 100644 --- a/reference/README.md +++ b/reference/README.md @@ -100,6 +100,12 @@ rather than resolved by guessing. source, method, first-inclusion date, and the package's declared author, source and licence. `index` lists it. Re-adding keeps the original `included_at` and updates `last_seen_at`. +- `--registry` accepts an `http(s)://` URL as well as a path. HTTP uses the + stdlib, so this stays dependency-light. `CANNED_PROMPTS_PUBLISH_TOKEN` + supplies the bearer token when publishing. +- Publishing to an HTTP registry needs `--as NAME`: a URL addresses a service + that **hosts several registries**, so unlike a directory it does not name one + (§ 20.4). Installing names it in a qualified reference. - An optional `registry.yaml` names a registry and records namespace claims. `publish` warns when a namespace is declared `closed` — it cannot authenticate a publisher, and says so rather than implying it checked. diff --git a/reference/canned_prompts.py b/reference/canned_prompts.py index fd9ffc9..e257980 100755 --- a/reference/canned_prompts.py +++ b/reference/canned_prompts.py @@ -11,9 +11,13 @@ import argparse import json import os import re +import base64 import datetime as dt import shutil import sys +import tempfile +import urllib.error +import urllib.request from dataclasses import dataclass, field from pathlib import Path from typing import Any, Iterable @@ -772,6 +776,71 @@ def record_inclusion( return entry +# --- HTTP registries (§ 20) ---------------------------------------------- +# +# § 20 lists "an HTTP service" as a valid registry. Implementing one surfaced a +# distinction the specification does not draw: a filesystem registry *is* one +# registry, but an HTTP service *hosts several* — the reference service keeps +# `local`, `helix` and others behind a single base URL. So a URL does not name a +# registry the way a directory does, and the registry must be named separately: +# by `--as` when publishing, and by a qualified reference when installing. +# Recorded as a finding rather than worked around silently. + + +def is_http_registry(value: str) -> bool: + return value.startswith("http://") or value.startswith("https://") + + +def http_request( + url: str, method: str = "GET", payload: dict[str, Any] | None = None +) -> dict[str, Any]: + """A registry call. stdlib only, so `reference/` stays dependency-light.""" + data = json.dumps(payload).encode("utf-8") if payload is not None else None + request = urllib.request.Request(url, data=data, method=method) + request.add_header("Accept", "application/json") + if data is not None: + request.add_header("Content-Type", "application/json") + token = os.environ.get("CANNED_PROMPTS_PUBLISH_TOKEN") + if token: + request.add_header("Authorization", f"Bearer {token}") + try: + with urllib.request.urlopen(request) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as exc: + body = exc.read().decode("utf-8", "replace") + try: + detail = json.loads(body).get("detail", body) + except json.JSONDecodeError: + detail = body + raise CannedPromptError(f"registry refused ({exc.code}): {detail}") from exc + except urllib.error.URLError as exc: + raise CannedPromptError(f"registry unreachable: {exc.reason}") from exc + + +def encode_files(package_dir: Path, manifest: dict[str, Any]) -> dict[str, Any]: + """The package's files, in the shape the service's archive returns.""" + files: dict[str, Any] = {} + for relative in sorted(package_members(package_dir, manifest)): + raw = (package_dir / relative).read_bytes() + try: + files[relative] = {"text": raw.decode("utf-8")} + except UnicodeDecodeError: + files[relative] = {"base64": base64.b64encode(raw).decode()} + return files + + +def decode_files(files: dict[str, Any], destination: Path) -> None: + for relative, item in files.items(): + if relative.startswith("/") or ".." in Path(relative).parts: + raise CannedPromptError(f"unsafe path from registry: {relative}") + target = destination / relative + target.parent.mkdir(parents=True, exist_ok=True) + if "text" in item: + target.write_text(item["text"], encoding="utf-8") + else: + target.write_bytes(base64.b64decode(item["base64"])) + + def missing_dependencies(catalog: Path, manifest: dict[str, Any]) -> list[str]: """Declared prompt dependencies that this catalog cannot satisfy. @@ -878,6 +947,27 @@ def cmd_add(args: argparse.Namespace) -> None: def cmd_publish(args: argparse.Namespace) -> None: src = Path(args.path) manifest = validate_package(src) + + if is_http_registry(args.registry): + # A URL hosts several registries, so it cannot name one on its own. + if not args.as_registry: + raise CannedPromptError( + "publishing to an HTTP registry needs --as NAME: a URL addresses a " + "service that hosts several registries, so unlike a directory it " + "does not name one (§ 20)" + ) + result = http_request( + args.registry.rstrip("/") + "/packages", + method="POST", + payload={ + "registry": check_registry_name(args.as_registry), + "source": str(src.resolve()), + "files": encode_files(src.resolve(), manifest), + }, + ) + print(f"published {result['reference']} -> {args.registry}") + return + registry = Path(args.registry).expanduser() check_legacy_registry(registry) name = registry_name(registry) if registry.is_dir() else None @@ -900,9 +990,30 @@ def cmd_publish(args: argparse.Namespace) -> None: def cmd_install(args: argparse.Namespace) -> None: + catalog = Path(args.catalog).expanduser() + + if is_http_registry(args.registry): + reference = args.id + (f"@{args.version}" if args.version else "") + archive = http_request( + args.registry.rstrip("/") + "/archives/" + reference + ) + source_registry, _, rest = archive["reference"].partition(":") + with tempfile.TemporaryDirectory() as tmp: + staged = Path(tmp) / "package" + staged.mkdir() + decode_files(archive["files"], staged) + manifest = validate_package(staged) + dst = catalog_package_path( + catalog, source_registry, manifest["id"], manifest["version"] + ) + copy_package(staged, dst, manifest, "catalog package") + record_inclusion(catalog, source_registry, manifest, args.registry, "install") + report_missing_dependencies(catalog, manifest) + print(f"installed {archive['reference']} -> {dst}") + return + registry = Path(args.registry).expanduser() check_legacy_registry(registry) - catalog = Path(args.catalog).expanduser() name = registry_name(registry) qualifier, package_id = parse_reference(args.id) @@ -1465,9 +1576,16 @@ def build_parser() -> argparse.ArgumentParser: install.add_argument("--registry", default=str(default_registry())) install.set_defaults(func=cmd_install) - publish = sub.add_parser("publish", help="publish a package to a filesystem registry") + publish = sub.add_parser("publish", help="publish a package to a registry") publish.add_argument("path") publish.add_argument("--registry", default=str(default_registry())) + publish.add_argument( + "--as", + dest="as_registry", + default=None, + metavar="REGISTRY", + help="registry name to publish into; required for an HTTP registry", + ) publish.set_defaults(func=cmd_publish) return parser diff --git a/reference/tests/test_canned_prompts.py b/reference/tests/test_canned_prompts.py index c2abe58..187a6e1 100644 --- a/reference/tests/test_canned_prompts.py +++ b/reference/tests/test_canned_prompts.py @@ -1,3 +1,4 @@ +import argparse from pathlib import Path import pytest @@ -942,3 +943,68 @@ def test_explicit_registry_path_is_never_second_guessed(tmp_path: Path, monkeypa monkeypatch.setenv("CANNED_PROMPTS_HOME", str(tmp_path)) (tmp_path / "registry").mkdir() cp.check_legacy_registry(tmp_path / "somewhere-else") + + +# --- HTTP registries (§ 20.4) --- + +def test_http_registry_detection() -> None: + assert cp.is_http_registry("https://registry.example") + assert cp.is_http_registry("http://localhost:8878") + assert not cp.is_http_registry("/home/user/.canned-prompts/default") + + +def test_encode_decode_round_trips_a_package(tmp_path: Path) -> None: + """The wire shape must reproduce the package byte for byte.""" + src = write_evaluated(tmp_path / "src", RUBRIC) + (src / "assets").mkdir() + (src / "assets" / "binary.dat").write_bytes(bytes(range(256))) + manifest = cp.validate_package(src) + + files = cp.encode_files(src, manifest) + assert "base64" in files["assets/binary.dat"], "non-UTF-8 must not go as text" + + out = tmp_path / "out" + out.mkdir() + cp.decode_files(files, out) + for relative in cp.package_members(src, manifest): + assert (out / relative).read_bytes() == (src / relative).read_bytes() + + +def test_decode_refuses_path_traversal(tmp_path: Path) -> None: + """A registry is untrusted input (§ 19); it must not write outside the package.""" + with pytest.raises(cp.CannedPromptError, match="unsafe path"): + cp.decode_files({"../escape.txt": {"text": "no"}}, tmp_path) + + +def test_publish_to_http_requires_a_registry_name(tmp_path: Path, monkeypatch) -> None: + """§ 20.4: a URL hosts several registries, so it cannot name one.""" + src = write_pkg(tmp_path / "p", BASE + "inputs:\n - name: greeting\n required: false\n default: hi\n") + args = argparse.Namespace( + path=str(src), registry="https://registry.example", as_registry=None + ) + with pytest.raises(cp.CannedPromptError, match="needs --as NAME"): + cp.cmd_publish(args) + + +def test_http_errors_surface_the_registry_detail(monkeypatch) -> None: + import urllib.error, io + + def raise_conflict(*a, **k): + raise urllib.error.HTTPError( + "u", 409, "Conflict", {}, io.BytesIO(b'{"detail":"already exists"}') + ) + + monkeypatch.setattr(cp.urllib.request, "urlopen", raise_conflict) + with pytest.raises(cp.CannedPromptError, match="registry refused \\(409\\): already exists"): + cp.http_request("https://registry.example/packages") + + +def test_unreachable_registry_is_reported_clearly(monkeypatch) -> None: + import urllib.error + + def raise_urlerror(*a, **k): + raise urllib.error.URLError("connection refused") + + monkeypatch.setattr(cp.urllib.request, "urlopen", raise_urlerror) + with pytest.raises(cp.CannedPromptError, match="registry unreachable"): + cp.http_request("https://registry.example/packages") diff --git a/workplans/CANP-WP-0006-hosted-registry-service.md b/workplans/CANP-WP-0006-hosted-registry-service.md index 8ea3f09..568f63e 100644 --- a/workplans/CANP-WP-0006-hosted-registry-service.md +++ b/workplans/CANP-WP-0006-hosted-registry-service.md @@ -206,7 +206,7 @@ check` now reports no drift. ```task id: CANP-WP-0006-T05 -status: todo +status: done priority: medium state_hub_task_id: "8dacbb68-aaa2-5f91-942d-a0667f1f1fc7" ``` @@ -220,6 +220,27 @@ claims. If they do not survive it, that is a finding about the format, not a bug to paper over in the client. +**Done, and it produced exactly one finding.** + +*What survived unaltered*, verified against the running service: identity and +its ambiguity rules (a bare id in two registries came back 409 over HTTP just as +it does locally), immutability of a published `@` (identical +content accepted, changed content refused, a version bump accepted), strict +packaging, validation, and the index. A package published and installed over +HTTP was **byte-identical** to its source — `diff -r` clean — and its +`canonical-fidelity` eval still passed after the round trip. + +*What did not*: **a URL is not a registry.** A filesystem registry *is* one +registry and § 20.1 names it from its directory; an HTTP service *hosts +several* behind one base URL. So the address cannot name the registry, and it +has to be named separately — `--as` when publishing, a qualified reference when +installing. Recorded as § 20.4 rather than papered over in the client, because +the gap is in the specification's list of registry kinds, not in the CLI. + +Implemented with `urllib` rather than a library, so `reference/` keeps PyYAML as +its only dependency. Registry responses are treated as untrusted input (§ 19): +`decode_files` refuses path traversal, with a test. + ## Image and smoke contract ```task