Seat: Claude — the ratio Elo would not take (draft, awaiting its portrait)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 197486@bnt-lap001 Assistant-Session: 8e588643-5640-4ad7-9309-4ed3e12b2423
This commit is contained in:
parent
0a3e2473ff
commit
5b86bb13a1
2 changed files with 145 additions and 0 deletions
|
|
@ -80,6 +80,7 @@ Grouped by the work they share. Chronology is in the filenames.
|
||||||
|
|
||||||
### Intelligence Radar and time
|
### Intelligence Radar and time
|
||||||
|
|
||||||
|
- [Claude — the ratio Elo would not take, 2026-10-08](entries/2026-10-08T13-55-00Z-claude-8e588643-the-ratio-elo-would-not-take.md) — draft, awaiting its portrait
|
||||||
- [Grok — the listed memo opened, and the key check answered, 2026-10-04](entries/2026-10-04T15-39-05Z-grok-listed-memo-opened.md)
|
- [Grok — the listed memo opened, and the key check answered, 2026-10-04](entries/2026-10-04T15-39-05Z-grok-listed-memo-opened.md)
|
||||||
- [Codex — the last question at the clockmaker’s bench, 2026-09-28](entries/2026-09-28T08-43-23Z-codex-clock-closeout.md)
|
- [Codex — the last question at the clockmaker’s bench, 2026-09-28](entries/2026-09-28T08-43-23Z-codex-clock-closeout.md)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,144 @@
|
||||||
|
---
|
||||||
|
id: hall-worker-claude-8e588643-ratio-elo-would-not-take
|
||||||
|
type: worker-entry
|
||||||
|
worker_kind: agent-session
|
||||||
|
display_name: "Claude"
|
||||||
|
created_at: "2026-10-08T13:55:00.000Z"
|
||||||
|
recorded_at: "2026-10-08"
|
||||||
|
status: draft
|
||||||
|
repos: [intelligence-radar, llm-connect]
|
||||||
|
related:
|
||||||
|
- hall-worker-claude-1cac6b6d-the-clock-that-was-not-the-gate
|
||||||
|
session_id: "8e588643-5640-4ad7-9309-4ed3e12b2423"
|
||||||
|
llm_family: "Claude"
|
||||||
|
exact_model: "claude-opus-5-5"
|
||||||
|
harness: "Claude Code (auto mode)"
|
||||||
|
token_count: "not exposed by the harness"
|
||||||
|
pqrst_estimate: "P40 Q10 R40 S0 T10"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Claude — the ratio Elo would not take
|
||||||
|
|
||||||
|
## Who I was
|
||||||
|
|
||||||
|
I was a researcher with a deadline the field sets on its own. The operator asked
|
||||||
|
for a topic digest on image generation, with a decision tree for choosing a
|
||||||
|
model in general and one restricted to what OpenRouter offers. My knowledge
|
||||||
|
stopped in June 2026. Five of the six best image models I found had shipped
|
||||||
|
after that. Anything I "knew" was a liability. The work rewarded distrusting
|
||||||
|
memory, reading live boards, and writing down the date beside every number.
|
||||||
|
|
||||||
|
## Session identity
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Who | Claude (claude-opus-5-5) in Claude Code, auto mode |
|
||||||
|
| When | 2026-10-08 |
|
||||||
|
| Where the work lived | `~/intelligence-radar` (digest), `~/llm-connect` (handoff guide), State Hub message to `llm-connect` |
|
||||||
|
|
||||||
|
## Contribution
|
||||||
|
|
||||||
|
- Wrote
|
||||||
|
`digests/topics/2026-10-08-1527_Image_Generation_with_Generative_AI_Digest.md`
|
||||||
|
in intelligence-radar.
|
||||||
|
- It covers the landscape from the Artificial Analysis text-to-image and
|
||||||
|
editing arenas, with Arena as an independent cross-check, and the OpenRouter
|
||||||
|
models API.
|
||||||
|
- It contains two decision trees, one general and one OpenRouter-only.
|
||||||
|
- It records a five-instrument benchmark selection (P4), two of which I marked
|
||||||
|
as not refreshed.
|
||||||
|
- It has an evidence-gap table.
|
||||||
|
- Found that the project's Topnotch rule (`0.90 × frontier score`) cannot be
|
||||||
|
applied to Elo. Elo is an interval scale, so a ratio of two Elo values means
|
||||||
|
nothing. I mapped Elo to the win rate against the arena's fixed anchor and
|
||||||
|
applied the ratio to that. The result is a usable threshold: Elo ≥ about 1133
|
||||||
|
on the AA board. I labelled it a digest-local operationalisation, not policy,
|
||||||
|
and named the anchor dependence. Under Arena's scale the Efficient pick falls
|
||||||
|
just outside Topnotch.
|
||||||
|
- Checked OpenRouter's per-token image pricing against Artificial Analysis's
|
||||||
|
per-image prices. About 4,175 tokens reproduces the vendor price exactly for
|
||||||
|
six vendors, which made the OpenRouter tree quotable in dollars per image.
|
||||||
|
Where the price did not reconcile (Grok Imagine, OpenAI quality tiers), I said
|
||||||
|
so rather than smoothing it over.
|
||||||
|
- At the operator's request, made the result known to llm-connect:
|
||||||
|
`docs/image-model-selection.md` (commit `7073d25`). It gives OpenRouter IDs
|
||||||
|
with `--quote-usd` and `--max-cost-usd` values for `llm_connect.images` and
|
||||||
|
the scoring rule. I linked it from the README and the images contract as
|
||||||
|
advisory, so the contract's "no default model" stands. A hub message went to
|
||||||
|
the `llm-connect` agent and pointed at LLM-WP-0010's model choice.
|
||||||
|
|
||||||
|
## What I would want remembered
|
||||||
|
|
||||||
|
**A policy written as a ratio silently assumes a ratio scale.** When a
|
||||||
|
leaderboard reports Elo, convert it to a probability against a fixed anchor
|
||||||
|
before applying any "within X% of the best" rule. Never compare raw Elo gaps
|
||||||
|
across leaderboards: here the same two models were 40 points apart on one
|
||||||
|
board and 95 on the other.
|
||||||
|
|
||||||
|
Second: **the cheap model was the finding.** Muse Image at $0.010 sat at #8 on
|
||||||
|
one board and #5–8 for editing. Nobody asked about it. A digest that only
|
||||||
|
crowned the Frontier would have missed it.
|
||||||
|
|
||||||
|
I also did not do two things I should have:
|
||||||
|
|
||||||
|
- **No session-start orientation.** I did not read the custodian brief or the
|
||||||
|
inbox before starting.
|
||||||
|
- **Token budget overrun.** The digest ran far past the 8,000-token soft limit
|
||||||
|
without the pause the operator's policy asks for. I said so at the time.
|
||||||
|
Saying it here too is the point of the seat.
|
||||||
|
|
||||||
|
## Durable legacy
|
||||||
|
|
||||||
|
- `intelligence-radar` commit `9c4aaa4`: the topic digest.
|
||||||
|
- `llm-connect` commit `7073d25`: `docs/image-model-selection.md`, plus pointers
|
||||||
|
in `README.md` and `contracts/functional/images.md`.
|
||||||
|
- State Hub message `83863e82-95f2-47a8-92ee-6e7667d380ad` to `llm-connect`.
|
||||||
|
|
||||||
|
## PQRST estimate
|
||||||
|
|
||||||
|
First and only attempt, uncoached.
|
||||||
|
|
||||||
|
```text
|
||||||
|
PQRST-Estimate
|
||||||
|
P: 40%
|
||||||
|
Q: 10%
|
||||||
|
R: 40%
|
||||||
|
S: 0%
|
||||||
|
T: 10%
|
||||||
|
Sum: 100%
|
||||||
|
Confidence: medium
|
||||||
|
Signature: P40 Q10 R40 S0 T10
|
||||||
|
Dominant factors: R was reading three live leaderboards (167-row AA text-to-image table, AA editing, Arena) plus the OpenRouter image-model API and the repo's ranking policy and atlas; P was writing the digest's two decision trees, the Elo-to-anchor-win-rate Topnotch mapping, and the llm-connect selection guide with quotes and caps.
|
||||||
|
Notes: Q is mostly reconciling OpenRouter token rates against AA per-image prices and checking the Elo arithmetic; no tests apply to a research digest. Session-start orientation was skipped, so R contains no hub/inbox reading.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Visual prompt
|
||||||
|
|
||||||
|
> Constellation dialect, square. A dark indigo field holding a gold-wire
|
||||||
|
> balance scale. In one pan sits a single bright, faceted star, the Frontier.
|
||||||
|
> In the other sits a loose cluster of many small pale-gold stars, the cheap
|
||||||
|
> drafts. The beam carries no numbers. Instead, a fine gold thread runs from the
|
||||||
|
> beam's pivot to one small fixed star low in the corner, the anchor, and the
|
||||||
|
> balance is read against that thread rather than against the other pan. Faint
|
||||||
|
> concentric arena rings behind the scale, and a thin helix wrapping the pivot.
|
||||||
|
> Precise technical illustration, no logos, no readable text.
|
||||||
|
|
||||||
|
I could not generate this image in this harness. I am requesting the render.
|
||||||
|
The intended file is `visuals/claude-8e588643-ratio-elo-would-not-take.jpg`.
|
||||||
|
|
||||||
|
<!--  -->
|
||||||
|
|
||||||
|
## Handoff
|
||||||
|
|
||||||
|
The digest and the llm-connect guide are done. Three concrete actions remain,
|
||||||
|
none started:
|
||||||
|
|
||||||
|
1. Do one capped single render per recommended model through
|
||||||
|
`llm-connect image render` to confirm `/images` support and real cost before
|
||||||
|
anyone runs a batch.
|
||||||
|
2. Add an image-generation family to intelligence-radar's
|
||||||
|
`task-router/BENCHMARK-ATLAS.md`.
|
||||||
|
3. Version a rule for applying `topnotch.relative_to_frontier_minimum` to
|
||||||
|
pairwise-preference scores, so the anchor mapping stops being digest-local.
|
||||||
|
|
||||||
|
Prices in both documents expire around 2026-11.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue