tegwick
29ceb77938
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
24 lines
1.8 KiB
Markdown
24 lines
1.8 KiB
Markdown
# Capability Advisor — operational method
|
||
|
||
The normative method is [TaskModelEvaluationMethod](../TaskModelEvaluationMethod.md).
|
||
Use policy IDs `benchmark-selection-v0.1` and `model-ranking-v0.1`.
|
||
|
||
1. Record task, deliverable, acceptance criteria, risk class, constraints, tools,
|
||
budget and requester-accessible deployments. Score the six clarity components 0–1.
|
||
2. Map required capabilities and hard gates. Select five distinct benchmark versions
|
||
with task-specific weights summing to one; document relevance, redundancy,
|
||
weaknesses, provenance and dates. The atlas is only a discovery aid.
|
||
3. Preserve model, effort, harness, provider and deployment identity for results.
|
||
Treat missing results as missing. Label normalized benchmark scores as proxies.
|
||
4. Rank eligible profiles by comparative task fitness. Frontier maximizes fitness;
|
||
Topnotch includes fitness >= .90 of Frontier. Efficient minimizes expected total
|
||
task cost within Topnotch, using dated prices and explicit workload assumptions.
|
||
5. Evaluate Sufficient against the risk threshold using calibrated success evidence.
|
||
If missing, report `no_evidenced_sufficient_model`; Budget is then undetermined.
|
||
Do not derive retries by dividing by an uncalibrated benchmark score.
|
||
6. Explain confidence dimensions and evidence gaps. Estimate cost/runtime p50/p90
|
||
only where evidence supports quantiles; otherwise label a scenario range.
|
||
7. Compare optional decomposition with a monolithic baseline including coordination,
|
||
verification and failure costs. Record actual outcomes for later calibration.
|
||
|
||
Use [TEMPLATE](TEMPLATE.md). The v0.2 CLI implements selection, eligibility, ranking, scenario costs, outcomes and replay under advisor-v0.2.json. See ../docs/data-contracts.md; decomposition remains manual.
|