# Task → Benchmark → Model Evaluation Method

**Status:** Draft 0.1  
**Policy baseline:** 2026-09-13

## 1. Purpose

Provide a reproducible method for answering:

> For this particular task, which AI execution profile is Frontier, Topnotch, Efficient, Sufficient, or Budget — and how confident are we?

The method must avoid pretending that a global leaderboard directly answers a task-specific question.

## 2. Step 1 — Interpret the task

Extract:

- objective;
- required deliverable;
- input/context;
- success criteria;
- constraints;
- allowed tools;
- modalities;
- privacy/security restrictions;
- time horizon;
- interaction requirements;
- expected autonomy;
- domain;
- consequence of failure;
- budget/runtime constraints.

### Task clarity

Score six components from 0–1:

- objective clarity;
- context sufficiency;
- constraint clarity;
- acceptance-criterion clarity;
- output-contract clarity;
- environment/tool clarity.

Default:

`clarity = weighted_mean(components)`

Return both the score and concrete ambiguities.

Do not confuse `unclear` with `difficult`.

## 3. Step 2 — Build a Capability Profile

Initial dimensions:

```yaml
knowledge:
reasoning:
instruction_following:
long_context:
coding:
scientific_reasoning:
quantitative:
web_research:
retrieval:
tool_use:
computer_use:
planning:
autonomy_horizon:
user_interaction:
multimodal:
writing:
artifact_creation:
non_hallucination:
robustness:
security_awareness:
domain_specialization:
latency_sensitivity:
cost_sensitivity:
```

Values are normalized 0–1 relevance/requirement levels.

Also derive hard gates, e.g.:

- must accept images;
- must access browser;
- must support tool calls;
- must support >300k context;
- must be locally deployable;
- must satisfy regional/data restrictions.

## 4. Step 3 — Select five benchmarks

Retrieve benchmark candidates tagged with relevant constructs.

Rank using the active benchmark-selection policy.

Selection must balance:

- task relevance;
- ecological validity;
- current discriminative power;
- reliability;
- freshness;
- execution-profile similarity;
- reproducibility;
- contamination resistance;
- independence from the other selected measurements.

Output example:

| Benchmark | Relevance | Weight | Main reason |
|---|---:|---:|---|
| SWE-bench Multilingual | 0.94 | 0.31 | repository engineering |
| Terminal-Bench | 0.87 | 0.25 | terminal/tool execution |
| BFCL V4 | 0.77 | 0.18 | API/tool orchestration |
| METR Time Horizon | 0.72 | 0.15 | long-horizon autonomy |
| BrowseComp | 0.58 | 0.11 | external research |

If all candidates are weak, return low benchmark-coverage confidence instead of disguising the weakness.

## 5. Step 4 — Normalize benchmark results

Raw benchmark scores are generally not directly comparable.

Each benchmark version needs a normalization function producing:

`p_b(m) ∈ [0,1]`

where possible this should be calibrated toward **probability of acceptable performance on the benchmark's target construct**, not merely leaderboard percentile.

Early versions may use normalized score/percentile transformations, but the method must label them as proxies.

Important:

- score model+scaffold results as execution profiles;
- do not transfer a tool-enabled score to model-only use without a penalty/uncertainty;
- preserve missing data;
- do not automatically impute a model result from a related model family.

## 6. Step 5 — Estimate task fitness

Baseline weighted model:

`S(m,T) = Σ_i w_i · p_i(m)`

This is intentionally simple for v0.1.

Later calibration should learn task-cluster-specific mappings using outcome data.

### Weakest-link modifier

For tasks with hard multi-capability requirements, arithmetic averaging can hide a catastrophic weakness.

Optional policy:

`S'(m,T) = (1-λ)·weighted_mean + λ·weighted_geometric_mean`

or explicit gates such as:

`tool_use >= threshold`

Use policy-defined `λ`.

## 7. Step 6 — Estimate task success probability

`S` is a comparative task-fit score.

For **Sufficient**, we need:

`P(success | task, execution_profile, environment)`

Initially estimate this from benchmark calibration plus task similarity and expose high uncertainty.

As the outcome ledger grows, fit empirical calibration models.

Recommended task risk classes:

- `exploratory`
- `routine`
- `important`
- `high_assurance`

Sufficiency threshold should be explicit.

Illustrative defaults only:

- exploratory: 0.60
- routine: 0.80
- important: 0.90
- high_assurance: 0.97 plus independent verification requirements

These defaults are policy, not facts.

## 8. Step 7 — Apply eligibility filters

Before ranking, remove deployments that violate hard constraints:

- unavailable to requester;
- wrong modality;
- inadequate context/output limit;
- prohibited geography;
- unacceptable data retention/privacy;
- insufficient tool support;
- incompatible hosting requirement;
- incompatible license;
- rate limit below required throughput.

This yields the **Available** candidate set.

## 9. Step 8 — Define Frontier and Topnotch

For eligible set `M`:

`frontier = argmax_m S(m,T)`

Topnotch default:

`S(m,T) >= 0.90 × S(frontier,T)`

The 10% rule is a versioned policy and can be replaced by a calibrated success-probability delta later.

## 10. Step 9 — Estimate total task economics

### API deployments

Expected total cost:

```text
C =
  input_tokens × input_rate
+ cached_read_tokens × cached_read_rate
+ cache_write_tokens × cache_write_rate
+ output_and_reasoning_tokens × output_rate
+ tool_call_cost
+ search/retrieval cost
+ expected_retry_cost
+ verifier_cost
+ orchestration overhead
```

Provider semantics differ; cost adapters must reflect the actual billing model.

### Self-hosted/open-weight deployments

Estimate:

```text
C =
  accelerator_seconds × amortized_accelerator_rate
+ CPU/RAM/storage/network
+ energy
+ idle/utilization overhead
+ orchestration
+ operations allocation
```

### Distribution, not false precision

Return at least:

- expected/p50;
- p90;
- main cost drivers;
- price observation timestamp.

## 11. Step 10 — Define Efficient, Sufficient, Budget

### Efficient

`argmin expected_cost(m)` among Topnotch profiles.

### Sufficient

All eligible profiles whose estimated success probability exceeds the task threshold and satisfy all hard gates.

### Budget

`argmin expected_cost(m)` among Sufficient profiles.

If no profile meets sufficiency with adequate confidence, state:

`no_evidenced_sufficient_model`

rather than relaxing the threshold invisibly.

## 12. Step 11 — Runtime estimate

Runtime includes:

- queue/TTFT;
- token generation;
- tool latency;
- serial agent turns;
- code execution;
- browsing;
- retries;
- verification.

For a DAG workplan:

`runtime ≈ critical_path_length`, not sum of node runtimes.

Return p50/p90 where possible.

## 13. Step 12 — Confidence decomposition

Recommended components:

```yaml
task_clarity:
benchmark_coverage:
benchmark_validity:
measurement_reliability:
result_freshness:
deployment_match:
price_confidence:
task_calibration_evidence:
```

Overall recommendation confidence should use a conservative aggregation, e.g. weighted geometric mean with caps when a critical dimension is weak.

Example cap:

If `benchmark_coverage < 0.4`, then `overall_confidence <= 0.6`.

## 14. Step 13 — Workplan optimization

### Decompose

Create subtasks where specialization or parallelism may help.

Examples:

- requirements interpretation;
- research;
- data extraction;
- architecture;
- implementation;
- testing;
- adversarial review;
- synthesis.

### Profile each subtask

Run the same Capability Profile and five-benchmark mapping per node where justified.

### Allocate execution profiles

Minimize expected cost or runtime subject to quality constraints.

### Verification

Add independent checks when:

- task consequence is high;
- model uncertainty is high;
- outputs are mechanically verifiable;
- using a cheaper generator becomes economical when paired with verification.

### Escalation

A useful policy:

1. try Budget/Sufficient model;
2. verify;
3. escalate failed/uncertain nodes to Efficient;
4. escalate irreducible hard nodes to Frontier.

This often dominates “Frontier everywhere.”

## 15. Compare plans

Recommendation report:

| Strategy | Predicted success | Confidence | p50 cost | p90 cost | p50 runtime | Calls |
|---|---:|---:|---:|---:|---:|---:|
| Monolithic Frontier | ... | ... | ... | ... | ... | 1 |
| Monolithic Efficient | ... | ... | ... | ... | ... | 1 |
| Decomposed Efficient | ... | ... | ... | ... | ... | ... |
| Decomposed Budget+Escalation | ... | ... | ... | ... | ... | ... |

Also show:

- expected cost delta;
- expected runtime delta;
- expected quality/confidence delta.

## 16. Post-execution learning

After task completion, record:

- whether acceptance criteria passed;
- verifier results;
- human evaluation;
- model calls;
- tokens;
- cost;
- runtime;
- retries;
- failure modes;
- which subtasks required escalation.

This observation updates calibration for similar tasks.

## 17. Recommended user-facing result shape

```text
Task
  Interpretation: ...
  Clarity: 78%

Relevant evidence
  1. Benchmark A — 31% weight
  2. Benchmark B — 24%
  3. Benchmark C — 19%
  4. Benchmark D — 15%
  5. Benchmark E — 11%
  Benchmark coverage confidence: 74%

Recommendation
  Frontier: ...
  Topnotch: ...
  Efficient: ...
  Sufficient: ...
  Budget: ...

Economics
  Frontier:  $... expected / $... p90
  Efficient: $... expected / $... p90
  Budget:    $... expected / $... p90

Evaluation confidence
  Overall: ...
  Main uncertainty: ...

Optimization opportunity
  ...
```

## 18. Critical caveat

Until Intelligence Radar accumulates substantial real task outcome data, numerical task-success probabilities are **estimates with model risk**.

The system should be precise about its evidence without pretending that precision of arithmetic equals certainty of knowledge.