Build reproducible capability advisor and scheduled source capture

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
tegwick 2026-09-13 23:07:41 +02:00
parent a1edefd7c5
commit 29ceb77938
68 changed files with 2431 additions and 85 deletions

View file

@ -0,0 +1,28 @@
# Capability comparison — 2026-09-13
Source-backed comparison; evidence gaps remain explicit.
Recommendation: `024a468396eb7d73840df04271cff4c42cd14ced1b249810c399b5a565dea5f1`
Policy: `advisor-v0.2`
| Profile | Eligible | Fitness proxy | Scenario USD | Outcome trials |
| --- | --- | --- | --- | --- |
| claude-sonnet-5-api-default | False | unknown | 0.0700 | 0 |
| claude-haiku-4-5-api-default | False | unknown | 0.0500 | 0 |
Frontier: undetermined
Efficient: undetermined
Budget: undetermined
## Evidence gaps
- Fewer than five relevant distinct measurements; ranking withheld.
- No eligible profile has complete benchmark evidence.
- no_evidenced_sufficient_model
- claude-sonnet-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
- claude-sonnet-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
- claude-haiku-4-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
- claude-haiku-4-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
Cost is a declared token/tool/retry/verification scenario, not a measured success-cost distribution.
Source review does not prove requester access or task success. See saved inputs for capture references.

View file

@ -0,0 +1,32 @@
# v0.2 implementation evidence — 2026-09-13
`make check`: 21 behavior tests cover rank/eligibility, missing or stale evidence,
price conflicts, numeric validation, aggregate exclusion, outcome deduplication,
empirical bounds, historical replay including JSON ordering, source integrity,
retry/size limits, scheduler idempotence and ISO-week handling. Repository checks
validate archive hashes, observation envelopes, retained captures and report replay.
`make demo-advisor`: full synthetic ranking returns the fixture Frontier and
Efficient, withholds Budget without outcomes, and reproduces the saved report.
The fixture remains isolated from the operational ledger.
Four primary pages were captured from SWE-bench and Anthropic. Original bytes,
retrieval receipts and ingestion results are retained under data/. A source-backed
comparison is in source-comparison-2026-09-13.md. It explicitly withholds winners
because matching benchmark measurements and requester access are not evidenced.
The initial price search contained stale promotion information. Inspection of
[Anthropic's launch-page changelog](https://www.anthropic.com/news/claude-sonnet-5)
resolved it: the August update superseded the previously scheduled price increase.
The reviewed price observation reflects the current captured
[pricing page](https://claude.com/pricing), with a one-day validity policy.
`run-due` completed both 2026-09-13 daily and 2026-W37 weekly jobs successfully.
Their receipts say `existing_preserved`, accurately distinguishing the imported
historical draft from a newly generated publication. A repeat invocation performed
no completed jobs. No external app automation or notifications were used.
The user timer was installed on bnt-lap001. `systemctl --user` reports timer
ActiveState=active, SubState=waiting; an explicit service invocation returned
Result=success and ExecMainStatus=0. The timer checks every 15 minutes while the
workstation user manager runs; it is not a continuously available hosted worker.