Integrate founding explorations and bootstrap operational radar
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
775f93b920
commit
d0e6438609
57 changed files with 4723 additions and 2 deletions
446
TaskModelEvaluationMethod.md
Normal file
446
TaskModelEvaluationMethod.md
Normal file
|
|
@ -0,0 +1,446 @@
|
|||
# Task → Benchmark → Model Evaluation Method
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Policy baseline:** 2026-09-13
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Provide a reproducible method for answering:
|
||||
|
||||
> For this particular task, which AI execution profile is Frontier, Topnotch, Efficient, Sufficient, or Budget — and how confident are we?
|
||||
|
||||
The method must avoid pretending that a global leaderboard directly answers a task-specific question.
|
||||
|
||||
## 2. Step 1 — Interpret the task
|
||||
|
||||
Extract:
|
||||
|
||||
- objective;
|
||||
- required deliverable;
|
||||
- input/context;
|
||||
- success criteria;
|
||||
- constraints;
|
||||
- allowed tools;
|
||||
- modalities;
|
||||
- privacy/security restrictions;
|
||||
- time horizon;
|
||||
- interaction requirements;
|
||||
- expected autonomy;
|
||||
- domain;
|
||||
- consequence of failure;
|
||||
- budget/runtime constraints.
|
||||
|
||||
### Task clarity
|
||||
|
||||
Score six components from 0–1:
|
||||
|
||||
- objective clarity;
|
||||
- context sufficiency;
|
||||
- constraint clarity;
|
||||
- acceptance-criterion clarity;
|
||||
- output-contract clarity;
|
||||
- environment/tool clarity.
|
||||
|
||||
Default:
|
||||
|
||||
`clarity = weighted_mean(components)`
|
||||
|
||||
Return both the score and concrete ambiguities.
|
||||
|
||||
Do not confuse `unclear` with `difficult`.
|
||||
|
||||
## 3. Step 2 — Build a Capability Profile
|
||||
|
||||
Initial dimensions:
|
||||
|
||||
```yaml
|
||||
knowledge:
|
||||
reasoning:
|
||||
instruction_following:
|
||||
long_context:
|
||||
coding:
|
||||
scientific_reasoning:
|
||||
quantitative:
|
||||
web_research:
|
||||
retrieval:
|
||||
tool_use:
|
||||
computer_use:
|
||||
planning:
|
||||
autonomy_horizon:
|
||||
user_interaction:
|
||||
multimodal:
|
||||
writing:
|
||||
artifact_creation:
|
||||
non_hallucination:
|
||||
robustness:
|
||||
security_awareness:
|
||||
domain_specialization:
|
||||
latency_sensitivity:
|
||||
cost_sensitivity:
|
||||
```
|
||||
|
||||
Values are normalized 0–1 relevance/requirement levels.
|
||||
|
||||
Also derive hard gates, e.g.:
|
||||
|
||||
- must accept images;
|
||||
- must access browser;
|
||||
- must support tool calls;
|
||||
- must support >300k context;
|
||||
- must be locally deployable;
|
||||
- must satisfy regional/data restrictions.
|
||||
|
||||
## 4. Step 3 — Select five benchmarks
|
||||
|
||||
Retrieve benchmark candidates tagged with relevant constructs.
|
||||
|
||||
Rank using the active benchmark-selection policy.
|
||||
|
||||
Selection must balance:
|
||||
|
||||
- task relevance;
|
||||
- ecological validity;
|
||||
- current discriminative power;
|
||||
- reliability;
|
||||
- freshness;
|
||||
- execution-profile similarity;
|
||||
- reproducibility;
|
||||
- contamination resistance;
|
||||
- independence from the other selected measurements.
|
||||
|
||||
Output example:
|
||||
|
||||
| Benchmark | Relevance | Weight | Main reason |
|
||||
|---|---:|---:|---|
|
||||
| SWE-bench Multilingual | 0.94 | 0.31 | repository engineering |
|
||||
| Terminal-Bench | 0.87 | 0.25 | terminal/tool execution |
|
||||
| BFCL V4 | 0.77 | 0.18 | API/tool orchestration |
|
||||
| METR Time Horizon | 0.72 | 0.15 | long-horizon autonomy |
|
||||
| BrowseComp | 0.58 | 0.11 | external research |
|
||||
|
||||
If all candidates are weak, return low benchmark-coverage confidence instead of disguising the weakness.
|
||||
|
||||
## 5. Step 4 — Normalize benchmark results
|
||||
|
||||
Raw benchmark scores are generally not directly comparable.
|
||||
|
||||
Each benchmark version needs a normalization function producing:
|
||||
|
||||
`p_b(m) ∈ [0,1]`
|
||||
|
||||
where possible this should be calibrated toward **probability of acceptable performance on the benchmark's target construct**, not merely leaderboard percentile.
|
||||
|
||||
Early versions may use normalized score/percentile transformations, but the method must label them as proxies.
|
||||
|
||||
Important:
|
||||
|
||||
- score model+scaffold results as execution profiles;
|
||||
- do not transfer a tool-enabled score to model-only use without a penalty/uncertainty;
|
||||
- preserve missing data;
|
||||
- do not automatically impute a model result from a related model family.
|
||||
|
||||
## 6. Step 5 — Estimate task fitness
|
||||
|
||||
Baseline weighted model:
|
||||
|
||||
`S(m,T) = Σ_i w_i · p_i(m)`
|
||||
|
||||
This is intentionally simple for v0.1.
|
||||
|
||||
Later calibration should learn task-cluster-specific mappings using outcome data.
|
||||
|
||||
### Weakest-link modifier
|
||||
|
||||
For tasks with hard multi-capability requirements, arithmetic averaging can hide a catastrophic weakness.
|
||||
|
||||
Optional policy:
|
||||
|
||||
`S'(m,T) = (1-λ)·weighted_mean + λ·weighted_geometric_mean`
|
||||
|
||||
or explicit gates such as:
|
||||
|
||||
`tool_use >= threshold`
|
||||
|
||||
Use policy-defined `λ`.
|
||||
|
||||
## 7. Step 6 — Estimate task success probability
|
||||
|
||||
`S` is a comparative task-fit score.
|
||||
|
||||
For **Sufficient**, we need:
|
||||
|
||||
`P(success | task, execution_profile, environment)`
|
||||
|
||||
Initially estimate this from benchmark calibration plus task similarity and expose high uncertainty.
|
||||
|
||||
As the outcome ledger grows, fit empirical calibration models.
|
||||
|
||||
Recommended task risk classes:
|
||||
|
||||
- `exploratory`
|
||||
- `routine`
|
||||
- `important`
|
||||
- `high_assurance`
|
||||
|
||||
Sufficiency threshold should be explicit.
|
||||
|
||||
Illustrative defaults only:
|
||||
|
||||
- exploratory: 0.60
|
||||
- routine: 0.80
|
||||
- important: 0.90
|
||||
- high_assurance: 0.97 plus independent verification requirements
|
||||
|
||||
These defaults are policy, not facts.
|
||||
|
||||
## 8. Step 7 — Apply eligibility filters
|
||||
|
||||
Before ranking, remove deployments that violate hard constraints:
|
||||
|
||||
- unavailable to requester;
|
||||
- wrong modality;
|
||||
- inadequate context/output limit;
|
||||
- prohibited geography;
|
||||
- unacceptable data retention/privacy;
|
||||
- insufficient tool support;
|
||||
- incompatible hosting requirement;
|
||||
- incompatible license;
|
||||
- rate limit below required throughput.
|
||||
|
||||
This yields the **Available** candidate set.
|
||||
|
||||
## 9. Step 8 — Define Frontier and Topnotch
|
||||
|
||||
For eligible set `M`:
|
||||
|
||||
`frontier = argmax_m S(m,T)`
|
||||
|
||||
Topnotch default:
|
||||
|
||||
`S(m,T) >= 0.90 × S(frontier,T)`
|
||||
|
||||
The 10% rule is a versioned policy and can be replaced by a calibrated success-probability delta later.
|
||||
|
||||
## 10. Step 9 — Estimate total task economics
|
||||
|
||||
### API deployments
|
||||
|
||||
Expected total cost:
|
||||
|
||||
```text
|
||||
C =
|
||||
input_tokens × input_rate
|
||||
+ cached_read_tokens × cached_read_rate
|
||||
+ cache_write_tokens × cache_write_rate
|
||||
+ output_and_reasoning_tokens × output_rate
|
||||
+ tool_call_cost
|
||||
+ search/retrieval cost
|
||||
+ expected_retry_cost
|
||||
+ verifier_cost
|
||||
+ orchestration overhead
|
||||
```
|
||||
|
||||
Provider semantics differ; cost adapters must reflect the actual billing model.
|
||||
|
||||
### Self-hosted/open-weight deployments
|
||||
|
||||
Estimate:
|
||||
|
||||
```text
|
||||
C =
|
||||
accelerator_seconds × amortized_accelerator_rate
|
||||
+ CPU/RAM/storage/network
|
||||
+ energy
|
||||
+ idle/utilization overhead
|
||||
+ orchestration
|
||||
+ operations allocation
|
||||
```
|
||||
|
||||
### Distribution, not false precision
|
||||
|
||||
Return at least:
|
||||
|
||||
- expected/p50;
|
||||
- p90;
|
||||
- main cost drivers;
|
||||
- price observation timestamp.
|
||||
|
||||
## 11. Step 10 — Define Efficient, Sufficient, Budget
|
||||
|
||||
### Efficient
|
||||
|
||||
`argmin expected_cost(m)` among Topnotch profiles.
|
||||
|
||||
### Sufficient
|
||||
|
||||
All eligible profiles whose estimated success probability exceeds the task threshold and satisfy all hard gates.
|
||||
|
||||
### Budget
|
||||
|
||||
`argmin expected_cost(m)` among Sufficient profiles.
|
||||
|
||||
If no profile meets sufficiency with adequate confidence, state:
|
||||
|
||||
`no_evidenced_sufficient_model`
|
||||
|
||||
rather than relaxing the threshold invisibly.
|
||||
|
||||
## 12. Step 11 — Runtime estimate
|
||||
|
||||
Runtime includes:
|
||||
|
||||
- queue/TTFT;
|
||||
- token generation;
|
||||
- tool latency;
|
||||
- serial agent turns;
|
||||
- code execution;
|
||||
- browsing;
|
||||
- retries;
|
||||
- verification.
|
||||
|
||||
For a DAG workplan:
|
||||
|
||||
`runtime ≈ critical_path_length`, not sum of node runtimes.
|
||||
|
||||
Return p50/p90 where possible.
|
||||
|
||||
## 13. Step 12 — Confidence decomposition
|
||||
|
||||
Recommended components:
|
||||
|
||||
```yaml
|
||||
task_clarity:
|
||||
benchmark_coverage:
|
||||
benchmark_validity:
|
||||
measurement_reliability:
|
||||
result_freshness:
|
||||
deployment_match:
|
||||
price_confidence:
|
||||
task_calibration_evidence:
|
||||
```
|
||||
|
||||
Overall recommendation confidence should use a conservative aggregation, e.g. weighted geometric mean with caps when a critical dimension is weak.
|
||||
|
||||
Example cap:
|
||||
|
||||
If `benchmark_coverage < 0.4`, then `overall_confidence <= 0.6`.
|
||||
|
||||
## 14. Step 13 — Workplan optimization
|
||||
|
||||
### Decompose
|
||||
|
||||
Create subtasks where specialization or parallelism may help.
|
||||
|
||||
Examples:
|
||||
|
||||
- requirements interpretation;
|
||||
- research;
|
||||
- data extraction;
|
||||
- architecture;
|
||||
- implementation;
|
||||
- testing;
|
||||
- adversarial review;
|
||||
- synthesis.
|
||||
|
||||
### Profile each subtask
|
||||
|
||||
Run the same Capability Profile and five-benchmark mapping per node where justified.
|
||||
|
||||
### Allocate execution profiles
|
||||
|
||||
Minimize expected cost or runtime subject to quality constraints.
|
||||
|
||||
### Verification
|
||||
|
||||
Add independent checks when:
|
||||
|
||||
- task consequence is high;
|
||||
- model uncertainty is high;
|
||||
- outputs are mechanically verifiable;
|
||||
- using a cheaper generator becomes economical when paired with verification.
|
||||
|
||||
### Escalation
|
||||
|
||||
A useful policy:
|
||||
|
||||
1. try Budget/Sufficient model;
|
||||
2. verify;
|
||||
3. escalate failed/uncertain nodes to Efficient;
|
||||
4. escalate irreducible hard nodes to Frontier.
|
||||
|
||||
This often dominates “Frontier everywhere.”
|
||||
|
||||
## 15. Compare plans
|
||||
|
||||
Recommendation report:
|
||||
|
||||
| Strategy | Predicted success | Confidence | p50 cost | p90 cost | p50 runtime | Calls |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Monolithic Frontier | ... | ... | ... | ... | ... | 1 |
|
||||
| Monolithic Efficient | ... | ... | ... | ... | ... | 1 |
|
||||
| Decomposed Efficient | ... | ... | ... | ... | ... | ... |
|
||||
| Decomposed Budget+Escalation | ... | ... | ... | ... | ... | ... |
|
||||
|
||||
Also show:
|
||||
|
||||
- expected cost delta;
|
||||
- expected runtime delta;
|
||||
- expected quality/confidence delta.
|
||||
|
||||
## 16. Post-execution learning
|
||||
|
||||
After task completion, record:
|
||||
|
||||
- whether acceptance criteria passed;
|
||||
- verifier results;
|
||||
- human evaluation;
|
||||
- model calls;
|
||||
- tokens;
|
||||
- cost;
|
||||
- runtime;
|
||||
- retries;
|
||||
- failure modes;
|
||||
- which subtasks required escalation.
|
||||
|
||||
This observation updates calibration for similar tasks.
|
||||
|
||||
## 17. Recommended user-facing result shape
|
||||
|
||||
```text
|
||||
Task
|
||||
Interpretation: ...
|
||||
Clarity: 78%
|
||||
|
||||
Relevant evidence
|
||||
1. Benchmark A — 31% weight
|
||||
2. Benchmark B — 24%
|
||||
3. Benchmark C — 19%
|
||||
4. Benchmark D — 15%
|
||||
5. Benchmark E — 11%
|
||||
Benchmark coverage confidence: 74%
|
||||
|
||||
Recommendation
|
||||
Frontier: ...
|
||||
Topnotch: ...
|
||||
Efficient: ...
|
||||
Sufficient: ...
|
||||
Budget: ...
|
||||
|
||||
Economics
|
||||
Frontier: $... expected / $... p90
|
||||
Efficient: $... expected / $... p90
|
||||
Budget: $... expected / $... p90
|
||||
|
||||
Evaluation confidence
|
||||
Overall: ...
|
||||
Main uncertainty: ...
|
||||
|
||||
Optimization opportunity
|
||||
...
|
||||
```
|
||||
|
||||
## 18. Critical caveat
|
||||
|
||||
Until Intelligence Radar accumulates substantial real task outcome data, numerical task-success probabilities are **estimates with model risk**.
|
||||
|
||||
The system should be precise about its evidence without pretending that precision of arithmetic equals certainty of knowledge.
|
||||
Loading…
Add table
Add a link
Reference in a new issue