tegwick
d0e6438609
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
24 lines
1.4 KiB
Markdown
24 lines
1.4 KiB
Markdown
# Worked example — evidence-limited routing
|
|
|
|
Task: produce a sourced technical diligence memo and migration plan for a coding-agent stack.
|
|
This is a manual workflow illustration, not a live model recommendation.
|
|
|
|
Missing context: current stack, task distribution, acceptable failure rate, region,
|
|
and access to candidate deployments. Acceptance: reviewed memo plus reproducible cost model.
|
|
Risk policy: important, provisional threshold .90.
|
|
|
|
Candidate instruments: SWE-bench Multilingual (repository work), Terminal-Bench
|
|
(shell work), BrowseComp (source discovery), GDPval (deliverables), BFCL (tool use).
|
|
Their exact versions, result dates, independence and requester harness match need
|
|
verification. No weights or model results are invented to fill the ticket.
|
|
|
|
Frontier / Topnotch / Efficient: undetermined pending eligible profile evidence.
|
|
Sufficient: `no_evidenced_sufficient_model`. Budget: undetermined.
|
|
Cost/runtime: unknown until prices and representative traces are observed.
|
|
|
|
Next: collect versioned results and deployment prices, run representative work samples,
|
|
then compare retrieval → analysis → independent verification against monolithic execution.
|
|
Record measured outcomes; do not assert an unsupported percentage saving.
|
|
|
|
The original Grok example, with named winners and assumed savings, is preserved in
|
|
the source ZIP and superseded by this evidence-limited example.
|