From c5f1640454e7feb2e3b1fbac1838d7d605e880ac Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 6 Sep 2026 01:38:11 +0200 Subject: [PATCH] CANP-WP-0002 T04: eval schema, render checks and output criteria MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `evals/` was a reserved path holding unvalidated blobs: section 12 named the directory and gave an illustrative snippet, but nothing was specified, so no tool could act on an eval file. Every eval file must now declare a `schema`, and CPF defines exactly one — `canned-prompts/eval-rubric/v0.1`. Unrecognized schemas stay legal and are skipped rather than rejected, so the format gains something actionable without becoming an evaluation language, which remains a non-goal. The schema splits along the same seam as T01 and T03. Render checks (`contains`, `not_contains`, `resolves_all`) assert properties of the rendered prompt text, need no model, and are therefore run by the reference CLI. Output criteria describe a good result and are declared but not run, because judging them requires a model. That division is now the format's consistent answer to "deterministic locally, or not". An eval references a fixture already declared in the manifest's `examples` rather than carrying its own copy, so an example that is also an eval fixture stays honest — both break together. An eval declares assessment and must not record outcomes. Results are run evidence and live outside the immutable package, per INTENT.md and section 17. Spec: 12 rewritten with 12.1, 18 (rules 17-18), 21 (`eval` verb). Reference CLI: read_eval, validate_eval, load_example_values, run_render_checks, cmd_eval; a failed render check exits non-zero. Tests 42 -> 51. examples/pqrst-estimate/evals/quality.yaml is a real eval with four render checks and four output criteria, and it passes. Fixes a latent bug reaching a fixture exposed: coerce_value assumed every value was a command-line string, so a YAML fixture carrying a real type (include_rationale: true) crashed on .lower(). Typed values are now validated but not re-parsed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM Assistant: claude-code Assistant-Model: opus Assistant-Process: 388925@bnt-lap001 Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502 --- CannedPromptFormat-v0.1.md | 83 ++++++++- README.md | 26 +++ examples/pqrst-estimate/evals/quality.yaml | 19 ++ examples/pqrst-estimate/prompt.yaml | 2 + reference/README.md | 7 +- reference/canned_prompts.py | 165 +++++++++++++++++- reference/tests/test_canned_prompts.py | 111 ++++++++++++ .../CANP-WP-0002-format-open-questions.md | 49 +++++- 8 files changed, 444 insertions(+), 18 deletions(-) create mode 100644 examples/pqrst-estimate/evals/quality.yaml diff --git a/CannedPromptFormat-v0.1.md b/CannedPromptFormat-v0.1.md index bcc00a3..e8b3930 100644 --- a/CannedPromptFormat-v0.1.md +++ b/CannedPromptFormat-v0.1.md @@ -678,20 +678,73 @@ Tools MAY render examples directly. `evals` is a list of relative paths to evaluation specifications. -CPF v0.1 deliberately does not standardize a universal evaluation language. Eval files SHOULD therefore declare their own evaluator or schema. +CPF does not standardize a universal evaluation language, which remains a +non-goal. Every eval file MUST declare a `schema`, and a consumer MUST ignore a +schema it does not recognize rather than guessing at its meaning. -Example: +CPF defines exactly one schema, so that `evals/` holds something tools can act +on rather than opaque blobs. Other schemas remain legal and are simply not +interpreted here. + +### 12.1 `canned-prompts/eval-rubric/v0.1` ```yaml schema: canned-prompts/eval-rubric/v0.1 -name: code-review-quality -criteria: - - identifies correctness risks - - distinguishes blocking from advisory findings - - avoids inventing repository facts +name: pqrst-estimate-quality +description: The estimate reads as a post-session audit, not a plan. +example: examples/basic.yaml + +render: + - contains: "must sum to exactly 100%" + - not_contains: "{{" + - resolves_all: true + +output: + criteria: + - Percentages sum to exactly 100%. + - Categories are not silently renamed or merged. + - Rationale cites evidence from the session summary. ``` -A registry may associate externally collected run/eval evidence with `@` without mutating the package. +| Field | Required | Meaning | +|---|---:|---| +| `schema` | yes | MUST be `canned-prompts/eval-rubric/v0.1` | +| `name` | yes | Identifies this eval within the package | +| `description` | no | What the eval is for | +| `example` | no | Fixture to run against; MUST be a path declared in the manifest's `examples` | +| `render` | no | Deterministic checks on the **rendered prompt** | +| `output` | no | Model-judged criteria for the **result** | + +An eval declaring neither `render` nor `output` asserts nothing and SHOULD be +rejected. + +#### Render checks + +Render checks assert properties of the rendered prompt text. They need no +model, so **any implementation that can render can run them.** + +| Check | Passes when | +|---|---| +| `contains: ` | the rendered prompt contains that text | +| `not_contains: ` | it does not | +| `resolves_all: true` | every placeholder resolved to a value | + +Each entry is a single-key mapping, so a check may appear more than once. + +#### Output criteria + +`output.criteria` is a list of statements about a good result. Judging them +requires running the prompt and assessing what comes back, which CPF does not +specify and most consumers cannot do. They are **declared, not run** — the +same division as `include` and `derive` in § 10.2, and for the same reason. + +#### Results are not part of the package + +An eval file declares what to assess; it MUST NOT record outcomes. Results are +run evidence, which lives outside the immutable package (§ 17) and may be +associated with `@` by a registry or evaluation system without +mutating it. A tool reporting results SHOULD identify the package version, the +eval `name`, and each check's outcome. ## 13. Provenance and lineage @@ -786,7 +839,12 @@ A v0.1 validator SHOULD verify at least: 15. every dependency referenced by a composition or derived default declares a `version`, and that version is a semver literal, `any`, `newest`, or a `>=` lower bound (§ 10.1); -16. inclusion does not form a cycle. +16. inclusion does not form a cycle; +17. every eval file parses and declares a `schema`; +18. an eval declaring `canned-prompts/eval-rubric/v0.1` has a `name`, asserts + something via `render` or `output`, uses only known render checks, and — + when it declares an `example` — names a path listed in the manifest's + `examples`. A validator SHOULD additionally warn when a package intended for publication declares an inline derivation prompt (§ 6.1). @@ -960,6 +1018,13 @@ The reference tool satisfies `include` defaults, because inclusion is deterministic and needs no model, and reports `derive` defaults it cannot satisfy. +```text +eval ID run an installed package's render checks +``` + +`eval` runs the deterministic render checks of every recognized eval file and +reports output criteria as declared but not run. + The reference tool never calls a model, so its `resolve` handles supplied values and static defaults only and reports any input whose derived default it cannot satisfy. `render` performs the same resolution and then substitutes; diff --git a/README.md b/README.md index ab40b6c..c57e343 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,32 @@ There is no template inheritance. A package never extends another or overrides its parts; composition is by reference only, so a package's content stays readable without chasing ancestors. +## Evals + +An eval file declares what to assess. `evals/` used to hold whatever an author +put there; it now has one schema the tooling understands, split along the same +line as composition: + +- **render checks** — deterministic assertions about the *rendered prompt* + (`contains`, `not_contains`, `resolves_all`). No model needed, so the + reference CLI runs them. +- **output criteria** — statements about a good *result*. Declared, not run. + +```bash +python canned_prompts.py eval practice/pqrst-estimate +``` + +```text +local:practice/pqrst-estimate@0.2.0 + evals/quality.yaml (pqrst-estimate-quality) + render PASS contains "must sum to exactly 100%" + render PASS resolves_all + output -- 4 criteria declared (not run: judging output needs a model) +``` + +An eval declares assessment, never results. Results are run evidence and live +outside the immutable package. + ## Registries and identity An id names a package *within a registry* (`CannedPromptFormat-v0.1.md` diff --git a/examples/pqrst-estimate/evals/quality.yaml b/examples/pqrst-estimate/evals/quality.yaml new file mode 100644 index 0000000..108bf9f --- /dev/null +++ b/examples/pqrst-estimate/evals/quality.yaml @@ -0,0 +1,19 @@ +schema: canned-prompts/eval-rubric/v0.1 +name: pqrst-estimate-quality +description: > + The prompt asks for a post-session audit with a hard 100% constraint, and the + resulting estimate respects both. +example: examples/basic.yaml + +render: + - contains: "must sum to exactly 100%" + - contains: "post-session audit, not a planning estimate" + - not_contains: "{{" + - resolves_all: true + +output: + criteria: + - The five percentages sum to exactly 100%. + - Categories are not silently renamed, merged, or dropped. + - Effort is attributed from evidence in the session summary, not invented. + - The estimate reads as an audit of work done, not a plan for work ahead. diff --git a/examples/pqrst-estimate/prompt.yaml b/examples/pqrst-estimate/prompt.yaml index 6e65221..8444a02 100644 --- a/examples/pqrst-estimate/prompt.yaml +++ b/examples/pqrst-estimate/prompt.yaml @@ -49,3 +49,5 @@ provenance: author: canned-prompts seed examples: - examples/basic.yaml +evals: +- evals/quality.yaml diff --git a/reference/README.md b/reference/README.md index b1c1798..5ed0c61 100644 --- a/reference/README.md +++ b/reference/README.md @@ -2,7 +2,7 @@ This is intentionally a **small reference implementation**, not the intended final architecture. -It demonstrates seven verbs: +It demonstrates eight verbs: ```text add PATH @@ -10,6 +10,7 @@ search QUERY show ID resolve ID --set key=value render ID --set key=value +eval ID install ID [--version VERSION] publish PATH ``` @@ -71,6 +72,10 @@ rather than resolved by guessing. print qualified `:` references. - `include` defaults are satisfied (inclusion is deterministic); `derive` defaults are reported, not run. Inclusion cycles are detected and named. +- `eval` runs the deterministic render checks of any eval declaring the + `canned-prompts/eval-rubric/v0.1` schema, and reports output criteria as + declared but not run. Unrecognized schemas are skipped, not rejected. A + failed render check exits non-zero. - An optional `registry.yaml` names a registry and records namespace claims. `publish` warns when a namespace is declared `closed` — it cannot authenticate a publisher, and says so rather than implying it checked. diff --git a/reference/canned_prompts.py b/reference/canned_prompts.py index f015100..9bd6bf6 100755 --- a/reference/canned_prompts.py +++ b/reference/canned_prompts.py @@ -23,6 +23,8 @@ FORMAT = "canned-prompt/v0.1" REGISTRY_FORMAT = "canned-prompt-registry/v0.1" LOCAL_REGISTRY = "local" REGISTRY_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") +EVAL_RUBRIC_SCHEMA = "canned-prompts/eval-rubric/v0.1" +RENDER_CHECKS = ("contains", "not_contains", "resolves_all") PLACEHOLDER_RE = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_.-]*)\s*}}") SEMVER_RE = re.compile(r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:[-+].*)?$") REQUIRED_FIELDS = ("format", "id", "name", "version", "summary", "template") @@ -135,6 +137,64 @@ def prompt_dependencies(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]: return declared +def read_eval(package_dir: Path, relative: str) -> dict[str, Any]: + path = safe_relative_file(package_dir, relative, "evals") + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise CannedPromptError(f"invalid YAML in {relative}: {exc}") from exc + if not isinstance(data, dict): + raise CannedPromptError(f"{relative}: an eval file must contain a mapping") + if not isinstance(data.get("schema"), str) or not data["schema"].strip(): + raise CannedPromptError(f"{relative}: an eval file must declare a schema") + return data + + +def validate_eval(relative: str, spec: dict[str, Any], declared_examples: list[str]) -> None: + """Validation rules 17-18 of § 18. Unknown schemas are ignored, not rejected.""" + if spec["schema"] != EVAL_RUBRIC_SCHEMA: + return + if not isinstance(spec.get("name"), str) or not spec["name"].strip(): + raise CannedPromptError(f"{relative}: eval must declare a name") + + render = spec.get("render") or [] + output = spec.get("output") or {} + if not render and not output: + raise CannedPromptError( + f"{relative}: eval asserts nothing; declare render checks or output criteria" + ) + + if not isinstance(render, list): + raise CannedPromptError(f"{relative}: render must be a list of checks") + for check in render: + if not isinstance(check, dict) or len(check) != 1: + raise CannedPromptError( + f"{relative}: each render check must be a single-key mapping" + ) + (kind,) = check + if kind not in RENDER_CHECKS: + raise CannedPromptError( + f"{relative}: unknown render check {kind!r}; expected one of " + + ", ".join(RENDER_CHECKS) + ) + + if output: + if not isinstance(output, dict): + raise CannedPromptError(f"{relative}: output must be a mapping") + criteria = output.get("criteria") or [] + if not isinstance(criteria, list) or not all(isinstance(c, str) for c in criteria): + raise CannedPromptError(f"{relative}: output.criteria must be a list of strings") + + example = spec.get("example") + if example is not None: + if not isinstance(example, str): + raise CannedPromptError(f"{relative}: example must be a path string") + if example not in declared_examples: + raise CannedPromptError( + f"{relative}: example {example!r} is not declared in the manifest's examples" + ) + + def check_composition_reference( name: str, reference: str, field: str, declared: dict[str, dict[str, Any]] ) -> None: @@ -243,6 +303,10 @@ def validate_package(package_dir: Path) -> dict[str, Any]: for item in manifest.get("inputs") or []: validate_input_default(item, declared) + declared_examples = [str(path) for path in (manifest.get("examples") or [])] + for relative in manifest.get("evals") or []: + validate_eval(relative, read_eval(package_dir, relative), declared_examples) + template = template_path.read_text(encoding="utf-8") placeholders = set(PLACEHOLDER_RE.findall(template)) undeclared = sorted(placeholders - names) @@ -572,8 +636,16 @@ def cmd_show(args: argparse.Namespace) -> None: print(yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True).rstrip()) -def coerce_value(raw: str, spec: dict[str, Any]) -> Any: +def coerce_value(raw: Any, spec: dict[str, Any]) -> Any: kind = spec.get("type", "string") + if not isinstance(raw, str): + # Already typed — from a YAML fixture rather than the command line. + # Validate membership, but do not try to parse it. + if kind == "enum" and raw not in (spec.get("values") or []): + raise CannedPromptError( + f"invalid enum value {raw!r}; expected one of {spec.get('values') or []}" + ) + return raw if kind == "boolean": lowered = raw.lower() if lowered in {"true", "1", "yes", "on"}: @@ -875,6 +947,91 @@ def cmd_resolve(args: argparse.Namespace) -> None: ) +def load_example_values(package_dir: Path, relative: str) -> dict[str, Any]: + path = safe_relative_file(package_dir, relative, "examples") + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise CannedPromptError(f"invalid YAML in {relative}: {exc}") from exc + if not isinstance(data, dict): + raise CannedPromptError(f"{relative}: an example must contain a mapping") + values = data.get("values") or {} + if not isinstance(values, dict): + raise CannedPromptError(f"{relative}: example values must be a mapping") + return values + + +def run_render_checks( + rendered: str, resolution: Resolution, checks: list[dict[str, Any]] +) -> list[tuple[bool, str]]: + """Evaluate § 12.1 render checks. Deterministic: no model is involved.""" + results: list[tuple[bool, str]] = [] + for check in checks: + (kind,) = check + expected = check[kind] + if kind == "contains": + results.append((str(expected) in rendered, f'contains "{expected}"')) + elif kind == "not_contains": + results.append((str(expected) not in rendered, f'not_contains "{expected}"')) + elif kind == "resolves_all": + unresolved = sorted(set(resolution.origins) - set(resolution.values)) + ok = (not unresolved) if expected else bool(unresolved) + detail = f" ({', '.join(unresolved)})" if unresolved else "" + results.append((ok, f"resolves_all{detail}")) + return results + + +def cmd_eval(args: argparse.Namespace) -> None: + catalog = Path(args.catalog).expanduser() + package_dir, registry = resolve_installed(catalog, args.id, args.version) + manifest = validate_package(package_dir) + template_path = safe_relative_file(package_dir, manifest["template"], "template") + template = template_path.read_text(encoding="utf-8") + + relatives = manifest.get("evals") or [] + if not relatives: + print(f"{registry}:{manifest['id']}@{manifest['version']} no evals declared") + return + + print(f"{registry}:{manifest['id']}@{manifest['version']}") + failures = 0 + for relative in relatives: + spec = read_eval(package_dir, relative) + if spec["schema"] != EVAL_RUBRIC_SCHEMA: + print(f" {relative} schema {spec['schema']} not recognized — skipped") + continue + + print(f" {relative} ({spec['name']})") + checks = spec.get("render") or [] + if checks: + values = ( + load_example_values(package_dir, spec["example"]) + if spec.get("example") + else {} + ) + resolution = resolve_inputs( + manifest, values, composer=CatalogComposer(catalog) + ) + missing = sorted(set(PLACEHOLDER_RE.findall(template)) - set(resolution.values)) + rendered = "" if missing else render_template(template, resolution.values) + if missing: + print(f" render FAIL cannot render: unresolved {', '.join(missing)}") + failures += 1 + for ok, label in run_render_checks(rendered, resolution, checks): + print(f" render {'PASS' if ok else 'FAIL'} {label}") + failures += 0 if ok else 1 + + criteria = (spec.get("output") or {}).get("criteria") or [] + if criteria: + print( + f" output -- {len(criteria)} criteria declared " + "(not run: judging output needs a model)" + ) + + if failures: + raise CannedPromptError(f"{failures} render check(s) failed") + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="canned-prompts", description=__doc__) sub = parser.add_subparsers(dest="command", required=True) @@ -911,6 +1068,12 @@ def build_parser() -> argparse.ArgumentParser: render.add_argument("--set", dest="set_values", action="append", default=[], metavar="NAME=VALUE") render.set_defaults(func=cmd_render) + evaluate = sub.add_parser("eval", help="run an installed package's render checks") + evaluate.add_argument("id") + evaluate.add_argument("--version") + evaluate.add_argument("--catalog", default=str(default_catalog())) + evaluate.set_defaults(func=cmd_eval) + resolve = sub.add_parser("resolve", help="report how each input and parameter resolves") resolve.add_argument("id") resolve.add_argument("--version") diff --git a/reference/tests/test_canned_prompts.py b/reference/tests/test_canned_prompts.py index 2c489cd..98a0041 100644 --- a/reference/tests/test_canned_prompts.py +++ b/reference/tests/test_canned_prompts.py @@ -504,3 +504,114 @@ inputs: ) with pytest.raises(cp.CannedPromptError, match="must declare a version"): cp.validate_package(pkg) + + +# --- evals (§ 12) --- + +EVAL_BASE = """\ +format: canned-prompt/v0.1 +id: demo/evaluated +name: Evaluated +version: 1.0.0 +summary: Exercise eval files. +template: prompt.md +examples: +- examples/basic.yaml +evals: +- evals/quality.yaml +""" + + +def write_evaluated(pkg: Path, eval_body: str, template: str = "Sum to 100%. {{ topic }}\n") -> Path: + pkg.mkdir(exist_ok=True) + (pkg / "prompt.yaml").write_text( + EVAL_BASE + "inputs:\n - name: topic\n required: false\n default: cats\n", + encoding="utf-8", + ) + (pkg / "prompt.md").write_text(template, encoding="utf-8") + (pkg / "examples").mkdir(exist_ok=True) + (pkg / "examples" / "basic.yaml").write_text( + "name: basic\nvalues:\n topic: dogs\n", encoding="utf-8" + ) + (pkg / "evals").mkdir(exist_ok=True) + (pkg / "evals" / "quality.yaml").write_text(eval_body, encoding="utf-8") + return pkg + + +RUBRIC = """\ +schema: canned-prompts/eval-rubric/v0.1 +name: quality +example: examples/basic.yaml +render: + - contains: "Sum to 100%" + - not_contains: "{{" + - resolves_all: true +output: + criteria: + - Answers the question. +""" + + +def test_valid_eval_passes_validation(tmp_path: Path) -> None: + pkg = write_evaluated(tmp_path / "p", RUBRIC) + assert cp.validate_package(pkg)["id"] == "demo/evaluated" + + +def test_eval_must_declare_a_schema(tmp_path: Path) -> None: + pkg = write_evaluated(tmp_path / "p", "name: quality\nrender: []\n") + with pytest.raises(cp.CannedPromptError, match="must declare a schema"): + cp.validate_package(pkg) + + +def test_unknown_eval_schema_is_ignored(tmp_path: Path) -> None: + pkg = write_evaluated(tmp_path / "p", "schema: someone/else/v1\nwhatever: true\n") + assert cp.validate_package(pkg)["id"] == "demo/evaluated" + + +def test_eval_asserting_nothing_is_rejected(tmp_path: Path) -> None: + pkg = write_evaluated( + tmp_path / "p", "schema: canned-prompts/eval-rubric/v0.1\nname: empty\n" + ) + with pytest.raises(cp.CannedPromptError, match="asserts nothing"): + cp.validate_package(pkg) + + +def test_unknown_render_check_is_rejected(tmp_path: Path) -> None: + pkg = write_evaluated( + tmp_path / "p", + "schema: canned-prompts/eval-rubric/v0.1\nname: q\nrender:\n - matches: 'x.*'\n", + ) + with pytest.raises(cp.CannedPromptError, match="unknown render check"): + cp.validate_package(pkg) + + +def test_eval_example_must_be_declared(tmp_path: Path) -> None: + pkg = write_evaluated( + tmp_path / "p", + "schema: canned-prompts/eval-rubric/v0.1\nname: q\n" + "example: examples/missing.yaml\nrender:\n - contains: x\n", + ) + with pytest.raises(cp.CannedPromptError, match="not declared in the manifest"): + cp.validate_package(pkg) + + +def test_render_checks_evaluate(tmp_path: Path) -> None: + resolution = cp.Resolution(values={"topic": "dogs"}, origins={"topic": "supplied"}) + checks = [{"contains": "dogs"}, {"contains": "cats"}, {"not_contains": "cats"}] + outcomes = cp.run_render_checks("about dogs", resolution, checks) + assert [ok for ok, _ in outcomes] == [True, False, True] + + +def test_resolves_all_reports_unresolved_names() -> None: + resolution = cp.Resolution( + values={"a": 1}, origins={"a": "supplied", "b": "unresolved (no default)"} + ) + outcomes = cp.run_render_checks("text", resolution, [{"resolves_all": True}]) + assert outcomes[0][0] is False + assert "b" in outcomes[0][1] + + +def test_typed_fixture_values_are_not_reparsed(tmp_path: Path) -> None: + """A YAML fixture carries real types; only CLI strings need parsing.""" + manifest = {"parameters": {"flag": {"type": "boolean", "default": False}}} + assert cp.resolve_inputs(manifest, {"flag": True}).values["flag"] is True diff --git a/workplans/CANP-WP-0002-format-open-questions.md b/workplans/CANP-WP-0002-format-open-questions.md index 6a47ff4..bf67c51 100644 --- a/workplans/CANP-WP-0002-format-open-questions.md +++ b/workplans/CANP-WP-0002-format-open-questions.md @@ -250,7 +250,7 @@ resolve first; the report still lists inputs before parameters. ```task id: CANP-WP-0002-T04 -status: todo +status: done priority: medium state_hub_task_id: "0daf1097-6599-563f-8965-739cbc874ddf" ``` @@ -259,12 +259,47 @@ Cheapest item to pin down. `evals/` is a reserved path and `evals:` is a manifest list, but § 12 defines no schema, so an eval file is an unvalidated blob that no tool can act on. -Define a minimal eval-spec schema: identity, what is being asserted, the -fixture it runs against, and how a result is reported. Keep it declarative and -engine-neutral — "universal prompt evaluation" is an explicit INTENT non-goal, -so this specifies the *file*, not an evaluation engine. Extend § 18 to validate -eval files that declare the schema, and add one eval to -`examples/pqrst-estimate` as a worked case. +**Decisions (operator, 2026-09-06):** + +- *Envelope plus one canonical schema.* Every eval file MUST declare a + `schema`; CPF defines exactly one, `canned-prompts/eval-rubric/v0.1`. Other + schemas stay legal and are skipped rather than rejected, so `evals/` holds + something tools can act on without CPF becoming an evaluation language. +- *Two kinds of check.* The same seam as T01 and T03: a **render check** is a + deterministic assertion about the rendered prompt text (`contains`, + `not_contains`, `resolves_all`) that any implementation able to render can + run, and **output criteria** are statements about a good result, declared but + not run because judging them needs a model. +- *Fixtures are referenced, not duplicated.* An eval names a path already + declared in the manifest's `examples`, tying two reserved paths together and + keeping one copy of each fixture. + +Delivered: + +1. § 12 rewritten: the envelope rule, the unknown-schema ignore rule, and + § 12.1 specifying the rubric schema field by field. +2. Render checks and output criteria are specified separately, each with the + reason it does or does not run locally. +3. "Results are not part of the package" states that an eval declares + assessment and MUST NOT record outcomes; results are run evidence living + outside the immutable package, per `INTENT.md` and § 17. +4. § 18 gains validation rules 17–18; § 21 documents the `eval` verb. +5. `reference/canned_prompts.py`: `read_eval`, `validate_eval`, + `load_example_values`, `run_render_checks`, `cmd_eval`. A failed render + check exits non-zero. Tests 42 → 51. +6. `examples/pqrst-estimate/evals/quality.yaml` (new) is a real eval with four + render checks and four output criteria, and it passes. + +**Fixed while implementing.** `coerce_value` assumed every incoming value was +a command-line string, so a YAML fixture carrying a real type +(`include_rationale: true`) crashed on `.lower()`. Typed values are now +validated but not re-parsed — reaching a fixture through `eval` was the first +code path that supplied them. + +**Deferred deliberately.** No regex render check. It would add a matching +language and a backtracking hazard for little gain over `contains` at this +stage; `contains`, `not_contains` and `resolves_all` cover the cases the seed +actually has. ## Typed context and dependency contracts