coulomb/canned-prompts
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CANP-WP-0002 T04: eval schema, render checks and output criteria

Browse source 
`evals/` was a reserved path holding unvalidated blobs: section 12 named the
directory and gave an illustrative snippet, but nothing was specified, so no
tool could act on an eval file.

Every eval file must now declare a `schema`, and CPF defines exactly one —
`canned-prompts/eval-rubric/v0.1`. Unrecognized schemas stay legal and are
skipped rather than rejected, so the format gains something actionable without
becoming an evaluation language, which remains a non-goal.

The schema splits along the same seam as T01 and T03. Render checks
(`contains`, `not_contains`, `resolves_all`) assert properties of the rendered
prompt text, need no model, and are therefore run by the reference CLI. Output
criteria describe a good result and are declared but not run, because judging
them requires a model. That division is now the format's consistent answer to
"deterministic locally, or not".

An eval references a fixture already declared in the manifest's `examples`
rather than carrying its own copy, so an example that is also an eval fixture
stays honest — both break together.

An eval declares assessment and must not record outcomes. Results are run
evidence and live outside the immutable package, per INTENT.md and section 17.

Spec: 12 rewritten with 12.1, 18 (rules 17-18), 21 (`eval` verb).

Reference CLI: read_eval, validate_eval, load_example_values,
run_render_checks, cmd_eval; a failed render check exits non-zero.
Tests 42 -> 51.

examples/pqrst-estimate/evals/quality.yaml is a real eval with four render
checks and four output criteria, and it passes.

Fixes a latent bug reaching a fixture exposed: coerce_value assumed every
value was a command-line string, so a YAML fixture carrying a real type
(include_rationale: true) crashed on .lower(). Typed values are now validated
but not re-parsed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
This commit is contained in:
tegwick 2026-09-06 01:38:11 +02:00
parent cac6bc135f
commit c5f1640454
8 changed files with 444 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

7
reference/README.md
View file

@ -2,7 +2,7 @@
This is intentionally a **small reference implementation**, not the intended final architecture.
It demonstrates seven verbs:
It demonstrates eight verbs:
```text
add PATH
@ -10,6 +10,7 @@ search QUERY
show ID
resolve ID --set key=value
render ID --set key=value
eval ID
install ID [--version VERSION]
publish PATH
```
@ -71,6 +72,10 @@ rather than resolved by guessing.
print qualified `<registry>:<id>` references.
- `include` defaults are satisfied (inclusion is deterministic); `derive`
defaults are reported, not run. Inclusion cycles are detected and named.
- `eval` runs the deterministic render checks of any eval declaring the
`canned-prompts/eval-rubric/v0.1` schema, and reports output criteria as
declared but not run. Unrecognized schemas are skipped, not rejected. A
failed render check exits non-zero.
- An optional `registry.yaml` names a registry and records namespace claims.
`publish` warns when a namespace is declared `closed` — it cannot
authenticate a publisher, and says so rather than implying it checked.