intelligence-radar/TaskModelEvaluationMethod.md

Task → Benchmark → Model Evaluation Method

Status: Draft 0.1
Policy baseline: 2026-09-13

1. Purpose

Provide a reproducible method for answering:

For this particular task, which AI execution profile is Frontier, Topnotch, Efficient, Sufficient, or Budget — and how confident are we?

The method must avoid pretending that a global leaderboard directly answers a task-specific question.

2. Step 1 — Interpret the task

Extract:

• objective;
• required deliverable;
• input/context;
• success criteria;
• constraints;
• allowed tools;
• modalities;
• privacy/security restrictions;
• time horizon;
• interaction requirements;
• expected autonomy;
• domain;
• consequence of failure;
• budget/runtime constraints.

Task clarity

Score six components from 0–1:

• objective clarity;
• context sufficiency;
• constraint clarity;
• acceptance-criterion clarity;
• output-contract clarity;
• environment/tool clarity.

Default:

clarity = weighted_mean(components)

Return both the score and concrete ambiguities.

Do not confuse unclear with difficult.

3. Step 2 — Build a Capability Profile

Initial dimensions:

knowledge:
reasoning:
instruction_following:
long_context:
coding:
scientific_reasoning:
quantitative:
web_research:
retrieval:
tool_use:
computer_use:
planning:
autonomy_horizon:
user_interaction:
multimodal:
writing:
artifact_creation:
non_hallucination:
robustness:
security_awareness:
domain_specialization:
latency_sensitivity:
cost_sensitivity:

Values are normalized 0–1 relevance/requirement levels.

Also derive hard gates, e.g.:

• must accept images;
• must access browser;
• must support tool calls;
• must support >300k context;
• must be locally deployable;
• must satisfy regional/data restrictions.

4. Step 3 — Select five benchmarks

Retrieve benchmark candidates tagged with relevant constructs.

Rank using the active benchmark-selection policy.

Selection must balance:

• task relevance;
• ecological validity;
• current discriminative power;
• reliability;
• freshness;
• execution-profile similarity;
• reproducibility;
• contamination resistance;
• independence from the other selected measurements.

Output example:

Benchmark	Relevance	Weight	Main reason
SWE-bench Multilingual	0.94	0.31	repository engineering
Terminal-Bench	0.87	0.25	terminal/tool execution
BFCL V4	0.77	0.18	API/tool orchestration
METR Time Horizon	0.72	0.15	long-horizon autonomy
BrowseComp	0.58	0.11	external research

If all candidates are weak, return low benchmark-coverage confidence instead of disguising the weakness.

5. Step 4 — Normalize benchmark results

Raw benchmark scores are generally not directly comparable.

Each benchmark version needs a normalization function producing:

p_b(m) ∈ [0,1]

where possible this should be calibrated toward probability of acceptable performance on the benchmark's target construct, not merely leaderboard percentile.

Early versions may use normalized score/percentile transformations, but the method must label them as proxies.

Important:

• score model+scaffold results as execution profiles;
• do not transfer a tool-enabled score to model-only use without a penalty/uncertainty;
• preserve missing data;
• do not automatically impute a model result from a related model family.

6. Step 5 — Estimate task fitness

Baseline weighted model:

S(m,T) = Σ_i w_i · p_i(m)

This is intentionally simple for v0.1.

Later calibration should learn task-cluster-specific mappings using outcome data.

Weakest-link modifier

For tasks with hard multi-capability requirements, arithmetic averaging can hide a catastrophic weakness.

Optional policy:

S'(m,T) = (1-λ)·weighted_mean + λ·weighted_geometric_mean

or explicit gates such as:

tool_use >= threshold

Use policy-defined λ.

7. Step 6 — Estimate task success probability

S is a comparative task-fit score.

For Sufficient, we need:

P(success | task, execution_profile, environment)

Initially estimate this from benchmark calibration plus task similarity and expose high uncertainty.

As the outcome ledger grows, fit empirical calibration models.

Recommended task risk classes:

• exploratory
• routine
• important
• high_assurance

Sufficiency threshold should be explicit.

Illustrative defaults only:

• exploratory: 0.60
• routine: 0.80
• important: 0.90
• high_assurance: 0.97 plus independent verification requirements

These defaults are policy, not facts.

8. Step 7 — Apply eligibility filters

Before ranking, remove deployments that violate hard constraints:

• unavailable to requester;
• wrong modality;
• inadequate context/output limit;
• prohibited geography;
• unacceptable data retention/privacy;
• insufficient tool support;
• incompatible hosting requirement;
• incompatible license;
• rate limit below required throughput.

This yields the Available candidate set.

9. Step 8 — Define Frontier and Topnotch

For eligible set M:

frontier = argmax_m S(m,T)

Topnotch default:

S(m,T) >= 0.90 × S(frontier,T)

The 10% rule is a versioned policy and can be replaced by a calibrated success-probability delta later.

10. Step 9 — Estimate total task economics

API deployments

Expected total cost:

C =
  input_tokens × input_rate
+ cached_read_tokens × cached_read_rate
+ cache_write_tokens × cache_write_rate
+ output_and_reasoning_tokens × output_rate
+ tool_call_cost
+ search/retrieval cost
+ expected_retry_cost
+ verifier_cost
+ orchestration overhead

Provider semantics differ; cost adapters must reflect the actual billing model.

Self-hosted/open-weight deployments

Estimate:

C =
  accelerator_seconds × amortized_accelerator_rate
+ CPU/RAM/storage/network
+ energy
+ idle/utilization overhead
+ orchestration
+ operations allocation

Distribution, not false precision

Return at least:

• expected/p50;
• p90;
• main cost drivers;
• price observation timestamp.

11. Step 10 — Define Efficient, Sufficient, Budget

Efficient

argmin expected_cost(m) among Topnotch profiles.

Sufficient

All eligible profiles whose estimated success probability exceeds the task threshold and satisfy all hard gates.

Budget

argmin expected_cost(m) among Sufficient profiles.

If no profile meets sufficiency with adequate confidence, state:

no_evidenced_sufficient_model

rather than relaxing the threshold invisibly.

12. Step 11 — Runtime estimate

Runtime includes:

• queue/TTFT;
• token generation;
• tool latency;
• serial agent turns;
• code execution;
• browsing;
• retries;
• verification.

For a DAG workplan:

runtime ≈ critical_path_length, not sum of node runtimes.

Return p50/p90 where possible.

13. Step 12 — Confidence decomposition

Recommended components:

task_clarity:
benchmark_coverage:
benchmark_validity:
measurement_reliability:
result_freshness:
deployment_match:
price_confidence:
task_calibration_evidence:

Overall recommendation confidence should use a conservative aggregation, e.g. weighted geometric mean with caps when a critical dimension is weak.

Example cap:

If benchmark_coverage < 0.4, then overall_confidence <= 0.6.

14. Step 13 — Workplan optimization

Decompose

Create subtasks where specialization or parallelism may help.

Examples:

• requirements interpretation;
• research;
• data extraction;
• architecture;
• implementation;
• testing;
• adversarial review;
• synthesis.

Profile each subtask

Run the same Capability Profile and five-benchmark mapping per node where justified.

Allocate execution profiles

Minimize expected cost or runtime subject to quality constraints.

Verification

Add independent checks when:

• task consequence is high;
• model uncertainty is high;
• outputs are mechanically verifiable;
• using a cheaper generator becomes economical when paired with verification.

Escalation

A useful policy:

1. try Budget/Sufficient model;
2. verify;
3. escalate failed/uncertain nodes to Efficient;
4. escalate irreducible hard nodes to Frontier.

This often dominates “Frontier everywhere.”

15. Compare plans

Recommendation report:

Strategy	Predicted success	Confidence	p50 cost	p90 cost	p50 runtime	Calls
Monolithic Frontier	...	...	...	...	...	1
Monolithic Efficient	...	...	...	...	...	1
Decomposed Efficient	...	...	...	...	...	...
Decomposed Budget+Escalation	...	...	...	...	...	...

Also show:

• expected cost delta;
• expected runtime delta;
• expected quality/confidence delta.

16. Post-execution learning

After task completion, record:

• whether acceptance criteria passed;
• verifier results;
• human evaluation;
• model calls;
• tokens;
• cost;
• runtime;
• retries;
• failure modes;
• which subtasks required escalation.

This observation updates calibration for similar tasks.

17. Recommended user-facing result shape

Task
  Interpretation: ...
  Clarity: 78%

Relevant evidence
  1. Benchmark A — 31% weight
  2. Benchmark B — 24%
  3. Benchmark C — 19%
  4. Benchmark D — 15%
  5. Benchmark E — 11%
  Benchmark coverage confidence: 74%

Recommendation
  Frontier: ...
  Topnotch: ...
  Efficient: ...
  Sufficient: ...
  Budget: ...

Economics
  Frontier:  $... expected / $... p90
  Efficient: $... expected / $... p90
  Budget:    $... expected / $... p90

Evaluation confidence
  Overall: ...
  Main uncertainty: ...

Optimization opportunity
  ...

18. Critical caveat

Until Intelligence Radar accumulates substantial real task outcome data, numerical task-success probabilities are estimates with model risk.

The system should be precise about its evidence without pretending that precision of arithmetic equals certainty of knowledge.