39 lines
1.6 KiB
YAML
39 lines
1.6 KiB
YAML
|
schema: canned-prompts/eval-rubric/v0.1
|
||
|
|
name: canonical-fidelity
|
||
|
|
description: >
|
||
|
|
The default render must be the canonical prompt from
|
||
|
|
~/pqrst-practice/PqrstPrompt.md, verbatim. hall-of-helix CLOSING.md requires
|
||
|
|
pasting that block unmodified, so any drift in this package is a defect —
|
||
|
|
these checks are what stop it recurring.
|
||
|
|
example: examples/basic.yaml
|
||
|
|
|
||
|
|
render:
|
||
|
|
- resolves_all: true
|
||
|
|
- not_contains: "{{"
|
||
|
|
# The five dimensions, by their canonical names.
|
||
|
|
- contains: "P — Main Problem:"
|
||
|
|
- contains: "Q — Quality and Tests:"
|
||
|
|
- contains: "R — Research and Context Clarification:"
|
||
|
|
- contains: "S — Security and Credentials:"
|
||
|
|
- contains: "T — Task Organization:"
|
||
|
|
# The rules that make the record auditable rather than decorative.
|
||
|
|
- contains: "must sum to exactly 100"
|
||
|
|
- contains: "retrospective audit, not a planning target"
|
||
|
|
- contains: "S may legitimately be 0%"
|
||
|
|
- contains: '"Dominant factors" must name concrete session facts'
|
||
|
|
# The exact output shape the hall files.
|
||
|
|
- contains: "PQRST-Estimate"
|
||
|
|
- contains: "Signature: P<int> Q<int> R<int> S<int> T<int>"
|
||
|
|
- contains: "Confidence: <low|medium|high>"
|
||
|
|
# Guard against the paraphrase this package used to be.
|
||
|
|
- not_contains: "Include rationale:"
|
||
|
|
- not_contains: "Percentages must sum to 100%."
|
||
|
|
|
||
|
|
output:
|
||
|
|
criteria:
|
||
|
|
- The five values are integers in 0..100 summing to exactly 100.
|
||
|
|
- Signature agrees with the individual values.
|
||
|
|
- Dominant factors names concrete session facts rather than restating percentages.
|
||
|
|
- S is 0 when no security-specific work occurred, and is not inflated.
|
||
|
|
- No sixth dimension is introduced inside the 5-tuple.
|