CANP-WP-0002 T04: eval schema, render checks and output criteria
`evals/` was a reserved path holding unvalidated blobs: section 12 named the directory and gave an illustrative snippet, but nothing was specified, so no tool could act on an eval file. Every eval file must now declare a `schema`, and CPF defines exactly one — `canned-prompts/eval-rubric/v0.1`. Unrecognized schemas stay legal and are skipped rather than rejected, so the format gains something actionable without becoming an evaluation language, which remains a non-goal. The schema splits along the same seam as T01 and T03. Render checks (`contains`, `not_contains`, `resolves_all`) assert properties of the rendered prompt text, need no model, and are therefore run by the reference CLI. Output criteria describe a good result and are declared but not run, because judging them requires a model. That division is now the format's consistent answer to "deterministic locally, or not". An eval references a fixture already declared in the manifest's `examples` rather than carrying its own copy, so an example that is also an eval fixture stays honest — both break together. An eval declares assessment and must not record outcomes. Results are run evidence and live outside the immutable package, per INTENT.md and section 17. Spec: 12 rewritten with 12.1, 18 (rules 17-18), 21 (`eval` verb). Reference CLI: read_eval, validate_eval, load_example_values, run_render_checks, cmd_eval; a failed render check exits non-zero. Tests 42 -> 51. examples/pqrst-estimate/evals/quality.yaml is a real eval with four render checks and four output criteria, and it passes. Fixes a latent bug reaching a fixture exposed: coerce_value assumed every value was a command-line string, so a YAML fixture carrying a real type (include_rationale: true) crashed on .lower(). Typed values are now validated but not re-parsed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM Assistant: claude-code Assistant-Model: opus Assistant-Process: 388925@bnt-lap001 Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
This commit is contained in:
parent
cac6bc135f
commit
c5f1640454
8 changed files with 444 additions and 18 deletions
|
|
@ -678,20 +678,73 @@ Tools MAY render examples directly.
|
|||
|
||||
`evals` is a list of relative paths to evaluation specifications.
|
||||
|
||||
CPF v0.1 deliberately does not standardize a universal evaluation language. Eval files SHOULD therefore declare their own evaluator or schema.
|
||||
CPF does not standardize a universal evaluation language, which remains a
|
||||
non-goal. Every eval file MUST declare a `schema`, and a consumer MUST ignore a
|
||||
schema it does not recognize rather than guessing at its meaning.
|
||||
|
||||
Example:
|
||||
CPF defines exactly one schema, so that `evals/` holds something tools can act
|
||||
on rather than opaque blobs. Other schemas remain legal and are simply not
|
||||
interpreted here.
|
||||
|
||||
### 12.1 `canned-prompts/eval-rubric/v0.1`
|
||||
|
||||
```yaml
|
||||
schema: canned-prompts/eval-rubric/v0.1
|
||||
name: code-review-quality
|
||||
criteria:
|
||||
- identifies correctness risks
|
||||
- distinguishes blocking from advisory findings
|
||||
- avoids inventing repository facts
|
||||
name: pqrst-estimate-quality
|
||||
description: The estimate reads as a post-session audit, not a plan.
|
||||
example: examples/basic.yaml
|
||||
|
||||
render:
|
||||
- contains: "must sum to exactly 100%"
|
||||
- not_contains: "{{"
|
||||
- resolves_all: true
|
||||
|
||||
output:
|
||||
criteria:
|
||||
- Percentages sum to exactly 100%.
|
||||
- Categories are not silently renamed or merged.
|
||||
- Rationale cites evidence from the session summary.
|
||||
```
|
||||
|
||||
A registry may associate externally collected run/eval evidence with `<id>@<version>` without mutating the package.
|
||||
| Field | Required | Meaning |
|
||||
|---|---:|---|
|
||||
| `schema` | yes | MUST be `canned-prompts/eval-rubric/v0.1` |
|
||||
| `name` | yes | Identifies this eval within the package |
|
||||
| `description` | no | What the eval is for |
|
||||
| `example` | no | Fixture to run against; MUST be a path declared in the manifest's `examples` |
|
||||
| `render` | no | Deterministic checks on the **rendered prompt** |
|
||||
| `output` | no | Model-judged criteria for the **result** |
|
||||
|
||||
An eval declaring neither `render` nor `output` asserts nothing and SHOULD be
|
||||
rejected.
|
||||
|
||||
#### Render checks
|
||||
|
||||
Render checks assert properties of the rendered prompt text. They need no
|
||||
model, so **any implementation that can render can run them.**
|
||||
|
||||
| Check | Passes when |
|
||||
|---|---|
|
||||
| `contains: <text>` | the rendered prompt contains that text |
|
||||
| `not_contains: <text>` | it does not |
|
||||
| `resolves_all: true` | every placeholder resolved to a value |
|
||||
|
||||
Each entry is a single-key mapping, so a check may appear more than once.
|
||||
|
||||
#### Output criteria
|
||||
|
||||
`output.criteria` is a list of statements about a good result. Judging them
|
||||
requires running the prompt and assessing what comes back, which CPF does not
|
||||
specify and most consumers cannot do. They are **declared, not run** — the
|
||||
same division as `include` and `derive` in § 10.2, and for the same reason.
|
||||
|
||||
#### Results are not part of the package
|
||||
|
||||
An eval file declares what to assess; it MUST NOT record outcomes. Results are
|
||||
run evidence, which lives outside the immutable package (§ 17) and may be
|
||||
associated with `<id>@<version>` by a registry or evaluation system without
|
||||
mutating it. A tool reporting results SHOULD identify the package version, the
|
||||
eval `name`, and each check's outcome.
|
||||
|
||||
## 13. Provenance and lineage
|
||||
|
||||
|
|
@ -786,7 +839,12 @@ A v0.1 validator SHOULD verify at least:
|
|||
15. every dependency referenced by a composition or derived default declares a
|
||||
`version`, and that version is a semver literal, `any`, `newest`, or a
|
||||
`>=` lower bound (§ 10.1);
|
||||
16. inclusion does not form a cycle.
|
||||
16. inclusion does not form a cycle;
|
||||
17. every eval file parses and declares a `schema`;
|
||||
18. an eval declaring `canned-prompts/eval-rubric/v0.1` has a `name`, asserts
|
||||
something via `render` or `output`, uses only known render checks, and —
|
||||
when it declares an `example` — names a path listed in the manifest's
|
||||
`examples`.
|
||||
|
||||
A validator SHOULD additionally warn when a package intended for publication
|
||||
declares an inline derivation prompt (§ 6.1).
|
||||
|
|
@ -960,6 +1018,13 @@ The reference tool satisfies `include` defaults, because inclusion is
|
|||
deterministic and needs no model, and reports `derive` defaults it cannot
|
||||
satisfy.
|
||||
|
||||
```text
|
||||
eval ID run an installed package's render checks
|
||||
```
|
||||
|
||||
`eval` runs the deterministic render checks of every recognized eval file and
|
||||
reports output criteria as declared but not run.
|
||||
|
||||
The reference tool never calls a model, so its `resolve` handles supplied
|
||||
values and static defaults only and reports any input whose derived default it
|
||||
cannot satisfy. `render` performs the same resolution and then substitutes;
|
||||
|
|
|
|||
26
README.md
26
README.md
|
|
@ -114,6 +114,32 @@ There is no template inheritance. A package never extends another or overrides
|
|||
its parts; composition is by reference only, so a package's content stays
|
||||
readable without chasing ancestors.
|
||||
|
||||
## Evals
|
||||
|
||||
An eval file declares what to assess. `evals/` used to hold whatever an author
|
||||
put there; it now has one schema the tooling understands, split along the same
|
||||
line as composition:
|
||||
|
||||
- **render checks** — deterministic assertions about the *rendered prompt*
|
||||
(`contains`, `not_contains`, `resolves_all`). No model needed, so the
|
||||
reference CLI runs them.
|
||||
- **output criteria** — statements about a good *result*. Declared, not run.
|
||||
|
||||
```bash
|
||||
python canned_prompts.py eval practice/pqrst-estimate
|
||||
```
|
||||
|
||||
```text
|
||||
local:practice/pqrst-estimate@0.2.0
|
||||
evals/quality.yaml (pqrst-estimate-quality)
|
||||
render PASS contains "must sum to exactly 100%"
|
||||
render PASS resolves_all
|
||||
output -- 4 criteria declared (not run: judging output needs a model)
|
||||
```
|
||||
|
||||
An eval declares assessment, never results. Results are run evidence and live
|
||||
outside the immutable package.
|
||||
|
||||
## Registries and identity
|
||||
|
||||
An id names a package *within a registry* (`CannedPromptFormat-v0.1.md`
|
||||
|
|
|
|||
19
examples/pqrst-estimate/evals/quality.yaml
Normal file
19
examples/pqrst-estimate/evals/quality.yaml
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
schema: canned-prompts/eval-rubric/v0.1
|
||||
name: pqrst-estimate-quality
|
||||
description: >
|
||||
The prompt asks for a post-session audit with a hard 100% constraint, and the
|
||||
resulting estimate respects both.
|
||||
example: examples/basic.yaml
|
||||
|
||||
render:
|
||||
- contains: "must sum to exactly 100%"
|
||||
- contains: "post-session audit, not a planning estimate"
|
||||
- not_contains: "{{"
|
||||
- resolves_all: true
|
||||
|
||||
output:
|
||||
criteria:
|
||||
- The five percentages sum to exactly 100%.
|
||||
- Categories are not silently renamed, merged, or dropped.
|
||||
- Effort is attributed from evidence in the session summary, not invented.
|
||||
- The estimate reads as an audit of work done, not a plan for work ahead.
|
||||
|
|
@ -49,3 +49,5 @@ provenance:
|
|||
author: canned-prompts seed
|
||||
examples:
|
||||
- examples/basic.yaml
|
||||
evals:
|
||||
- evals/quality.yaml
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
This is intentionally a **small reference implementation**, not the intended final architecture.
|
||||
|
||||
It demonstrates seven verbs:
|
||||
It demonstrates eight verbs:
|
||||
|
||||
```text
|
||||
add PATH
|
||||
|
|
@ -10,6 +10,7 @@ search QUERY
|
|||
show ID
|
||||
resolve ID --set key=value
|
||||
render ID --set key=value
|
||||
eval ID
|
||||
install ID [--version VERSION]
|
||||
publish PATH
|
||||
```
|
||||
|
|
@ -71,6 +72,10 @@ rather than resolved by guessing.
|
|||
print qualified `<registry>:<id>` references.
|
||||
- `include` defaults are satisfied (inclusion is deterministic); `derive`
|
||||
defaults are reported, not run. Inclusion cycles are detected and named.
|
||||
- `eval` runs the deterministic render checks of any eval declaring the
|
||||
`canned-prompts/eval-rubric/v0.1` schema, and reports output criteria as
|
||||
declared but not run. Unrecognized schemas are skipped, not rejected. A
|
||||
failed render check exits non-zero.
|
||||
- An optional `registry.yaml` names a registry and records namespace claims.
|
||||
`publish` warns when a namespace is declared `closed` — it cannot
|
||||
authenticate a publisher, and says so rather than implying it checked.
|
||||
|
|
|
|||
|
|
@ -23,6 +23,8 @@ FORMAT = "canned-prompt/v0.1"
|
|||
REGISTRY_FORMAT = "canned-prompt-registry/v0.1"
|
||||
LOCAL_REGISTRY = "local"
|
||||
REGISTRY_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
|
||||
EVAL_RUBRIC_SCHEMA = "canned-prompts/eval-rubric/v0.1"
|
||||
RENDER_CHECKS = ("contains", "not_contains", "resolves_all")
|
||||
PLACEHOLDER_RE = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_.-]*)\s*}}")
|
||||
SEMVER_RE = re.compile(r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:[-+].*)?$")
|
||||
REQUIRED_FIELDS = ("format", "id", "name", "version", "summary", "template")
|
||||
|
|
@ -135,6 +137,64 @@ def prompt_dependencies(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
|||
return declared
|
||||
|
||||
|
||||
def read_eval(package_dir: Path, relative: str) -> dict[str, Any]:
|
||||
path = safe_relative_file(package_dir, relative, "evals")
|
||||
try:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as exc:
|
||||
raise CannedPromptError(f"invalid YAML in {relative}: {exc}") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise CannedPromptError(f"{relative}: an eval file must contain a mapping")
|
||||
if not isinstance(data.get("schema"), str) or not data["schema"].strip():
|
||||
raise CannedPromptError(f"{relative}: an eval file must declare a schema")
|
||||
return data
|
||||
|
||||
|
||||
def validate_eval(relative: str, spec: dict[str, Any], declared_examples: list[str]) -> None:
|
||||
"""Validation rules 17-18 of § 18. Unknown schemas are ignored, not rejected."""
|
||||
if spec["schema"] != EVAL_RUBRIC_SCHEMA:
|
||||
return
|
||||
if not isinstance(spec.get("name"), str) or not spec["name"].strip():
|
||||
raise CannedPromptError(f"{relative}: eval must declare a name")
|
||||
|
||||
render = spec.get("render") or []
|
||||
output = spec.get("output") or {}
|
||||
if not render and not output:
|
||||
raise CannedPromptError(
|
||||
f"{relative}: eval asserts nothing; declare render checks or output criteria"
|
||||
)
|
||||
|
||||
if not isinstance(render, list):
|
||||
raise CannedPromptError(f"{relative}: render must be a list of checks")
|
||||
for check in render:
|
||||
if not isinstance(check, dict) or len(check) != 1:
|
||||
raise CannedPromptError(
|
||||
f"{relative}: each render check must be a single-key mapping"
|
||||
)
|
||||
(kind,) = check
|
||||
if kind not in RENDER_CHECKS:
|
||||
raise CannedPromptError(
|
||||
f"{relative}: unknown render check {kind!r}; expected one of "
|
||||
+ ", ".join(RENDER_CHECKS)
|
||||
)
|
||||
|
||||
if output:
|
||||
if not isinstance(output, dict):
|
||||
raise CannedPromptError(f"{relative}: output must be a mapping")
|
||||
criteria = output.get("criteria") or []
|
||||
if not isinstance(criteria, list) or not all(isinstance(c, str) for c in criteria):
|
||||
raise CannedPromptError(f"{relative}: output.criteria must be a list of strings")
|
||||
|
||||
example = spec.get("example")
|
||||
if example is not None:
|
||||
if not isinstance(example, str):
|
||||
raise CannedPromptError(f"{relative}: example must be a path string")
|
||||
if example not in declared_examples:
|
||||
raise CannedPromptError(
|
||||
f"{relative}: example {example!r} is not declared in the manifest's examples"
|
||||
)
|
||||
|
||||
|
||||
def check_composition_reference(
|
||||
name: str, reference: str, field: str, declared: dict[str, dict[str, Any]]
|
||||
) -> None:
|
||||
|
|
@ -243,6 +303,10 @@ def validate_package(package_dir: Path) -> dict[str, Any]:
|
|||
for item in manifest.get("inputs") or []:
|
||||
validate_input_default(item, declared)
|
||||
|
||||
declared_examples = [str(path) for path in (manifest.get("examples") or [])]
|
||||
for relative in manifest.get("evals") or []:
|
||||
validate_eval(relative, read_eval(package_dir, relative), declared_examples)
|
||||
|
||||
template = template_path.read_text(encoding="utf-8")
|
||||
placeholders = set(PLACEHOLDER_RE.findall(template))
|
||||
undeclared = sorted(placeholders - names)
|
||||
|
|
@ -572,8 +636,16 @@ def cmd_show(args: argparse.Namespace) -> None:
|
|||
print(yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True).rstrip())
|
||||
|
||||
|
||||
def coerce_value(raw: str, spec: dict[str, Any]) -> Any:
|
||||
def coerce_value(raw: Any, spec: dict[str, Any]) -> Any:
|
||||
kind = spec.get("type", "string")
|
||||
if not isinstance(raw, str):
|
||||
# Already typed — from a YAML fixture rather than the command line.
|
||||
# Validate membership, but do not try to parse it.
|
||||
if kind == "enum" and raw not in (spec.get("values") or []):
|
||||
raise CannedPromptError(
|
||||
f"invalid enum value {raw!r}; expected one of {spec.get('values') or []}"
|
||||
)
|
||||
return raw
|
||||
if kind == "boolean":
|
||||
lowered = raw.lower()
|
||||
if lowered in {"true", "1", "yes", "on"}:
|
||||
|
|
@ -875,6 +947,91 @@ def cmd_resolve(args: argparse.Namespace) -> None:
|
|||
)
|
||||
|
||||
|
||||
def load_example_values(package_dir: Path, relative: str) -> dict[str, Any]:
|
||||
path = safe_relative_file(package_dir, relative, "examples")
|
||||
try:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as exc:
|
||||
raise CannedPromptError(f"invalid YAML in {relative}: {exc}") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise CannedPromptError(f"{relative}: an example must contain a mapping")
|
||||
values = data.get("values") or {}
|
||||
if not isinstance(values, dict):
|
||||
raise CannedPromptError(f"{relative}: example values must be a mapping")
|
||||
return values
|
||||
|
||||
|
||||
def run_render_checks(
|
||||
rendered: str, resolution: Resolution, checks: list[dict[str, Any]]
|
||||
) -> list[tuple[bool, str]]:
|
||||
"""Evaluate § 12.1 render checks. Deterministic: no model is involved."""
|
||||
results: list[tuple[bool, str]] = []
|
||||
for check in checks:
|
||||
(kind,) = check
|
||||
expected = check[kind]
|
||||
if kind == "contains":
|
||||
results.append((str(expected) in rendered, f'contains "{expected}"'))
|
||||
elif kind == "not_contains":
|
||||
results.append((str(expected) not in rendered, f'not_contains "{expected}"'))
|
||||
elif kind == "resolves_all":
|
||||
unresolved = sorted(set(resolution.origins) - set(resolution.values))
|
||||
ok = (not unresolved) if expected else bool(unresolved)
|
||||
detail = f" ({', '.join(unresolved)})" if unresolved else ""
|
||||
results.append((ok, f"resolves_all{detail}"))
|
||||
return results
|
||||
|
||||
|
||||
def cmd_eval(args: argparse.Namespace) -> None:
|
||||
catalog = Path(args.catalog).expanduser()
|
||||
package_dir, registry = resolve_installed(catalog, args.id, args.version)
|
||||
manifest = validate_package(package_dir)
|
||||
template_path = safe_relative_file(package_dir, manifest["template"], "template")
|
||||
template = template_path.read_text(encoding="utf-8")
|
||||
|
||||
relatives = manifest.get("evals") or []
|
||||
if not relatives:
|
||||
print(f"{registry}:{manifest['id']}@{manifest['version']} no evals declared")
|
||||
return
|
||||
|
||||
print(f"{registry}:{manifest['id']}@{manifest['version']}")
|
||||
failures = 0
|
||||
for relative in relatives:
|
||||
spec = read_eval(package_dir, relative)
|
||||
if spec["schema"] != EVAL_RUBRIC_SCHEMA:
|
||||
print(f" {relative} schema {spec['schema']} not recognized — skipped")
|
||||
continue
|
||||
|
||||
print(f" {relative} ({spec['name']})")
|
||||
checks = spec.get("render") or []
|
||||
if checks:
|
||||
values = (
|
||||
load_example_values(package_dir, spec["example"])
|
||||
if spec.get("example")
|
||||
else {}
|
||||
)
|
||||
resolution = resolve_inputs(
|
||||
manifest, values, composer=CatalogComposer(catalog)
|
||||
)
|
||||
missing = sorted(set(PLACEHOLDER_RE.findall(template)) - set(resolution.values))
|
||||
rendered = "" if missing else render_template(template, resolution.values)
|
||||
if missing:
|
||||
print(f" render FAIL cannot render: unresolved {', '.join(missing)}")
|
||||
failures += 1
|
||||
for ok, label in run_render_checks(rendered, resolution, checks):
|
||||
print(f" render {'PASS' if ok else 'FAIL'} {label}")
|
||||
failures += 0 if ok else 1
|
||||
|
||||
criteria = (spec.get("output") or {}).get("criteria") or []
|
||||
if criteria:
|
||||
print(
|
||||
f" output -- {len(criteria)} criteria declared "
|
||||
"(not run: judging output needs a model)"
|
||||
)
|
||||
|
||||
if failures:
|
||||
raise CannedPromptError(f"{failures} render check(s) failed")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(prog="canned-prompts", description=__doc__)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
|
@ -911,6 +1068,12 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
render.add_argument("--set", dest="set_values", action="append", default=[], metavar="NAME=VALUE")
|
||||
render.set_defaults(func=cmd_render)
|
||||
|
||||
evaluate = sub.add_parser("eval", help="run an installed package's render checks")
|
||||
evaluate.add_argument("id")
|
||||
evaluate.add_argument("--version")
|
||||
evaluate.add_argument("--catalog", default=str(default_catalog()))
|
||||
evaluate.set_defaults(func=cmd_eval)
|
||||
|
||||
resolve = sub.add_parser("resolve", help="report how each input and parameter resolves")
|
||||
resolve.add_argument("id")
|
||||
resolve.add_argument("--version")
|
||||
|
|
|
|||
|
|
@ -504,3 +504,114 @@ inputs:
|
|||
)
|
||||
with pytest.raises(cp.CannedPromptError, match="must declare a version"):
|
||||
cp.validate_package(pkg)
|
||||
|
||||
|
||||
# --- evals (§ 12) ---
|
||||
|
||||
EVAL_BASE = """\
|
||||
format: canned-prompt/v0.1
|
||||
id: demo/evaluated
|
||||
name: Evaluated
|
||||
version: 1.0.0
|
||||
summary: Exercise eval files.
|
||||
template: prompt.md
|
||||
examples:
|
||||
- examples/basic.yaml
|
||||
evals:
|
||||
- evals/quality.yaml
|
||||
"""
|
||||
|
||||
|
||||
def write_evaluated(pkg: Path, eval_body: str, template: str = "Sum to 100%. {{ topic }}\n") -> Path:
|
||||
pkg.mkdir(exist_ok=True)
|
||||
(pkg / "prompt.yaml").write_text(
|
||||
EVAL_BASE + "inputs:\n - name: topic\n required: false\n default: cats\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(pkg / "prompt.md").write_text(template, encoding="utf-8")
|
||||
(pkg / "examples").mkdir(exist_ok=True)
|
||||
(pkg / "examples" / "basic.yaml").write_text(
|
||||
"name: basic\nvalues:\n topic: dogs\n", encoding="utf-8"
|
||||
)
|
||||
(pkg / "evals").mkdir(exist_ok=True)
|
||||
(pkg / "evals" / "quality.yaml").write_text(eval_body, encoding="utf-8")
|
||||
return pkg
|
||||
|
||||
|
||||
RUBRIC = """\
|
||||
schema: canned-prompts/eval-rubric/v0.1
|
||||
name: quality
|
||||
example: examples/basic.yaml
|
||||
render:
|
||||
- contains: "Sum to 100%"
|
||||
- not_contains: "{{"
|
||||
- resolves_all: true
|
||||
output:
|
||||
criteria:
|
||||
- Answers the question.
|
||||
"""
|
||||
|
||||
|
||||
def test_valid_eval_passes_validation(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(tmp_path / "p", RUBRIC)
|
||||
assert cp.validate_package(pkg)["id"] == "demo/evaluated"
|
||||
|
||||
|
||||
def test_eval_must_declare_a_schema(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(tmp_path / "p", "name: quality\nrender: []\n")
|
||||
with pytest.raises(cp.CannedPromptError, match="must declare a schema"):
|
||||
cp.validate_package(pkg)
|
||||
|
||||
|
||||
def test_unknown_eval_schema_is_ignored(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(tmp_path / "p", "schema: someone/else/v1\nwhatever: true\n")
|
||||
assert cp.validate_package(pkg)["id"] == "demo/evaluated"
|
||||
|
||||
|
||||
def test_eval_asserting_nothing_is_rejected(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(
|
||||
tmp_path / "p", "schema: canned-prompts/eval-rubric/v0.1\nname: empty\n"
|
||||
)
|
||||
with pytest.raises(cp.CannedPromptError, match="asserts nothing"):
|
||||
cp.validate_package(pkg)
|
||||
|
||||
|
||||
def test_unknown_render_check_is_rejected(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(
|
||||
tmp_path / "p",
|
||||
"schema: canned-prompts/eval-rubric/v0.1\nname: q\nrender:\n - matches: 'x.*'\n",
|
||||
)
|
||||
with pytest.raises(cp.CannedPromptError, match="unknown render check"):
|
||||
cp.validate_package(pkg)
|
||||
|
||||
|
||||
def test_eval_example_must_be_declared(tmp_path: Path) -> None:
|
||||
pkg = write_evaluated(
|
||||
tmp_path / "p",
|
||||
"schema: canned-prompts/eval-rubric/v0.1\nname: q\n"
|
||||
"example: examples/missing.yaml\nrender:\n - contains: x\n",
|
||||
)
|
||||
with pytest.raises(cp.CannedPromptError, match="not declared in the manifest"):
|
||||
cp.validate_package(pkg)
|
||||
|
||||
|
||||
def test_render_checks_evaluate(tmp_path: Path) -> None:
|
||||
resolution = cp.Resolution(values={"topic": "dogs"}, origins={"topic": "supplied"})
|
||||
checks = [{"contains": "dogs"}, {"contains": "cats"}, {"not_contains": "cats"}]
|
||||
outcomes = cp.run_render_checks("about dogs", resolution, checks)
|
||||
assert [ok for ok, _ in outcomes] == [True, False, True]
|
||||
|
||||
|
||||
def test_resolves_all_reports_unresolved_names() -> None:
|
||||
resolution = cp.Resolution(
|
||||
values={"a": 1}, origins={"a": "supplied", "b": "unresolved (no default)"}
|
||||
)
|
||||
outcomes = cp.run_render_checks("text", resolution, [{"resolves_all": True}])
|
||||
assert outcomes[0][0] is False
|
||||
assert "b" in outcomes[0][1]
|
||||
|
||||
|
||||
def test_typed_fixture_values_are_not_reparsed(tmp_path: Path) -> None:
|
||||
"""A YAML fixture carries real types; only CLI strings need parsing."""
|
||||
manifest = {"parameters": {"flag": {"type": "boolean", "default": False}}}
|
||||
assert cp.resolve_inputs(manifest, {"flag": True}).values["flag"] is True
|
||||
|
|
|
|||
|
|
@ -250,7 +250,7 @@ resolve first; the report still lists inputs before parameters.
|
|||
|
||||
```task
|
||||
id: CANP-WP-0002-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "0daf1097-6599-563f-8965-739cbc874ddf"
|
||||
```
|
||||
|
|
@ -259,12 +259,47 @@ Cheapest item to pin down. `evals/` is a reserved path and `evals:` is a
|
|||
manifest list, but § 12 defines no schema, so an eval file is an unvalidated
|
||||
blob that no tool can act on.
|
||||
|
||||
Define a minimal eval-spec schema: identity, what is being asserted, the
|
||||
fixture it runs against, and how a result is reported. Keep it declarative and
|
||||
engine-neutral — "universal prompt evaluation" is an explicit INTENT non-goal,
|
||||
so this specifies the *file*, not an evaluation engine. Extend § 18 to validate
|
||||
eval files that declare the schema, and add one eval to
|
||||
`examples/pqrst-estimate` as a worked case.
|
||||
**Decisions (operator, 2026-09-06):**
|
||||
|
||||
- *Envelope plus one canonical schema.* Every eval file MUST declare a
|
||||
`schema`; CPF defines exactly one, `canned-prompts/eval-rubric/v0.1`. Other
|
||||
schemas stay legal and are skipped rather than rejected, so `evals/` holds
|
||||
something tools can act on without CPF becoming an evaluation language.
|
||||
- *Two kinds of check.* The same seam as T01 and T03: a **render check** is a
|
||||
deterministic assertion about the rendered prompt text (`contains`,
|
||||
`not_contains`, `resolves_all`) that any implementation able to render can
|
||||
run, and **output criteria** are statements about a good result, declared but
|
||||
not run because judging them needs a model.
|
||||
- *Fixtures are referenced, not duplicated.* An eval names a path already
|
||||
declared in the manifest's `examples`, tying two reserved paths together and
|
||||
keeping one copy of each fixture.
|
||||
|
||||
Delivered:
|
||||
|
||||
1. § 12 rewritten: the envelope rule, the unknown-schema ignore rule, and
|
||||
§ 12.1 specifying the rubric schema field by field.
|
||||
2. Render checks and output criteria are specified separately, each with the
|
||||
reason it does or does not run locally.
|
||||
3. "Results are not part of the package" states that an eval declares
|
||||
assessment and MUST NOT record outcomes; results are run evidence living
|
||||
outside the immutable package, per `INTENT.md` and § 17.
|
||||
4. § 18 gains validation rules 17–18; § 21 documents the `eval` verb.
|
||||
5. `reference/canned_prompts.py`: `read_eval`, `validate_eval`,
|
||||
`load_example_values`, `run_render_checks`, `cmd_eval`. A failed render
|
||||
check exits non-zero. Tests 42 → 51.
|
||||
6. `examples/pqrst-estimate/evals/quality.yaml` (new) is a real eval with four
|
||||
render checks and four output criteria, and it passes.
|
||||
|
||||
**Fixed while implementing.** `coerce_value` assumed every incoming value was
|
||||
a command-line string, so a YAML fixture carrying a real type
|
||||
(`include_rationale: true`) crashed on `.lower()`. Typed values are now
|
||||
validated but not re-parsed — reaching a fixture through `eval` was the first
|
||||
code path that supplied them.
|
||||
|
||||
**Deferred deliberately.** No regex render check. It would add a matching
|
||||
language and a backtracking hazard for little gain over `contains` at this
|
||||
stage; `contains`, `not_contains` and `resolves_all` cover the cases the seed
|
||||
actually has.
|
||||
|
||||
## Typed context and dependency contracts
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue