29 lines
1.3 KiB
Markdown
29 lines
1.3 KiB
Markdown
|
# Capability comparison — 2026-09-13
|
||
|
|
|
||
|
|
Source-backed comparison; evidence gaps remain explicit.
|
||
|
|
|
||
|
|
Recommendation: `024a468396eb7d73840df04271cff4c42cd14ced1b249810c399b5a565dea5f1`
|
||
|
|
Policy: `advisor-v0.2`
|
||
|
|
|
||
|
|
| Profile | Eligible | Fitness proxy | Scenario USD | Outcome trials |
|
||
|
|
| --- | --- | --- | --- | --- |
|
||
|
|
| claude-sonnet-5-api-default | False | unknown | 0.0700 | 0 |
|
||
|
|
| claude-haiku-4-5-api-default | False | unknown | 0.0500 | 0 |
|
||
|
|
|
||
|
|
Frontier: undetermined
|
||
|
|
Efficient: undetermined
|
||
|
|
Budget: undetermined
|
||
|
|
|
||
|
|
## Evidence gaps
|
||
|
|
|
||
|
|
- Fewer than five relevant distinct measurements; ranking withheld.
|
||
|
|
- No eligible profile has complete benchmark evidence.
|
||
|
|
- no_evidenced_sufficient_model
|
||
|
|
- claude-sonnet-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||
|
|
- claude-sonnet-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||
|
|
- claude-haiku-4-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||
|
|
- claude-haiku-4-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||
|
|
|
||
|
|
Cost is a declared token/tool/retry/verification scenario, not a measured success-cost distribution.
|
||
|
|
Source review does not prove requester access or task success. See saved inputs for capture references.
|