Build reproducible capability advisor and scheduled source capture
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
a1edefd7c5
commit
29ceb77938
68 changed files with 2431 additions and 85 deletions
28
docs/evidence/source-comparison-2026-09-13.md
Normal file
28
docs/evidence/source-comparison-2026-09-13.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Capability comparison — 2026-09-13
|
||||
|
||||
Source-backed comparison; evidence gaps remain explicit.
|
||||
|
||||
Recommendation: `024a468396eb7d73840df04271cff4c42cd14ced1b249810c399b5a565dea5f1`
|
||||
Policy: `advisor-v0.2`
|
||||
|
||||
| Profile | Eligible | Fitness proxy | Scenario USD | Outcome trials |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| claude-sonnet-5-api-default | False | unknown | 0.0700 | 0 |
|
||||
| claude-haiku-4-5-api-default | False | unknown | 0.0500 | 0 |
|
||||
|
||||
Frontier: undetermined
|
||||
Efficient: undetermined
|
||||
Budget: undetermined
|
||||
|
||||
## Evidence gaps
|
||||
|
||||
- Fewer than five relevant distinct measurements; ranking withheld.
|
||||
- No eligible profile has complete benchmark evidence.
|
||||
- no_evidenced_sufficient_model
|
||||
- claude-sonnet-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||||
- claude-sonnet-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||||
- claude-haiku-4-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||||
- claude-haiku-4-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||||
|
||||
Cost is a declared token/tool/retry/verification scenario, not a measured success-cost distribution.
|
||||
Source review does not prove requester access or task success. See saved inputs for capture references.
|
||||
32
docs/evidence/v0.2-validation.md
Normal file
32
docs/evidence/v0.2-validation.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# v0.2 implementation evidence — 2026-09-13
|
||||
|
||||
`make check`: 21 behavior tests cover rank/eligibility, missing or stale evidence,
|
||||
price conflicts, numeric validation, aggregate exclusion, outcome deduplication,
|
||||
empirical bounds, historical replay including JSON ordering, source integrity,
|
||||
retry/size limits, scheduler idempotence and ISO-week handling. Repository checks
|
||||
validate archive hashes, observation envelopes, retained captures and report replay.
|
||||
|
||||
`make demo-advisor`: full synthetic ranking returns the fixture Frontier and
|
||||
Efficient, withholds Budget without outcomes, and reproduces the saved report.
|
||||
The fixture remains isolated from the operational ledger.
|
||||
|
||||
Four primary pages were captured from SWE-bench and Anthropic. Original bytes,
|
||||
retrieval receipts and ingestion results are retained under data/. A source-backed
|
||||
comparison is in source-comparison-2026-09-13.md. It explicitly withholds winners
|
||||
because matching benchmark measurements and requester access are not evidenced.
|
||||
|
||||
The initial price search contained stale promotion information. Inspection of
|
||||
[Anthropic's launch-page changelog](https://www.anthropic.com/news/claude-sonnet-5)
|
||||
resolved it: the August update superseded the previously scheduled price increase.
|
||||
The reviewed price observation reflects the current captured
|
||||
[pricing page](https://claude.com/pricing), with a one-day validity policy.
|
||||
|
||||
`run-due` completed both 2026-09-13 daily and 2026-W37 weekly jobs successfully.
|
||||
Their receipts say `existing_preserved`, accurately distinguishing the imported
|
||||
historical draft from a newly generated publication. A repeat invocation performed
|
||||
no completed jobs. No external app automation or notifications were used.
|
||||
|
||||
The user timer was installed on bnt-lap001. `systemctl --user` reports timer
|
||||
ActiveState=active, SubState=waiting; an explicit service invocation returned
|
||||
Result=success and ExecMainStatus=0. The timer checks every 15 minutes while the
|
||||
workstation user manager runs; it is not a continuously available hosted worker.
|
||||
Loading…
Add table
Add a link
Reference in a new issue