25 lines
1.6 KiB
Markdown
25 lines
1.6 KiB
Markdown
|
# Capability Advisor — operational method
|
|||
|
|
|
|||
|
|
The normative method is [TaskModelEvaluationMethod](../TaskModelEvaluationMethod.md).
|
|||
|
|
Use policy IDs `benchmark-selection-v0.1` and `model-ranking-v0.1`.
|
|||
|
|
|
|||
|
|
1. Record task, deliverable, acceptance criteria, risk class, constraints, tools,
|
|||
|
|
budget and requester-accessible deployments. Score the six clarity components 0–1.
|
|||
|
|
2. Map required capabilities and hard gates. Select five distinct benchmark versions
|
|||
|
|
with task-specific weights summing to one; document relevance, redundancy,
|
|||
|
|
weaknesses, provenance and dates. The atlas is only a discovery aid.
|
|||
|
|
3. Preserve model, effort, harness, provider and deployment identity for results.
|
|||
|
|
Treat missing results as missing. Label normalized benchmark scores as proxies.
|
|||
|
|
4. Rank eligible profiles by comparative task fitness. Frontier maximizes fitness;
|
|||
|
|
Topnotch includes fitness >= .90 of Frontier. Efficient minimizes expected total
|
|||
|
|
task cost within Topnotch, using dated prices and explicit workload assumptions.
|
|||
|
|
5. Evaluate Sufficient against the risk threshold using calibrated success evidence.
|
|||
|
|
If missing, report `no_evidenced_sufficient_model`; Budget is then undetermined.
|
|||
|
|
Do not derive retries by dividing by an uncalibrated benchmark score.
|
|||
|
|
6. Explain confidence dimensions and evidence gaps. Estimate cost/runtime p50/p90
|
|||
|
|
only where evidence supports quantiles; otherwise label a scenario range.
|
|||
|
|
7. Compare optional decomposition with a monolithic baseline including coordination,
|
|||
|
|
verification and failure costs. Record actual outcomes for later calibration.
|
|||
|
|
|
|||
|
|
Use [TEMPLATE](TEMPLATE.md). This is a manual workflow in v0.1, not an API promise.
|