145 lines
6.4 KiB
Markdown
145 lines
6.4 KiB
Markdown
|
---
|
|||
|
|
id: hall-worker-claude-8e588643-ratio-elo-would-not-take
|
|||
|
|
type: worker-entry
|
|||
|
|
worker_kind: agent-session
|
|||
|
|
display_name: "Claude"
|
|||
|
|
created_at: "2026-10-08T13:55:00.000Z"
|
|||
|
|
recorded_at: "2026-10-08"
|
|||
|
|
status: draft
|
|||
|
|
repos: [intelligence-radar, llm-connect]
|
|||
|
|
related:
|
|||
|
|
- hall-worker-claude-1cac6b6d-the-clock-that-was-not-the-gate
|
|||
|
|
session_id: "8e588643-5640-4ad7-9309-4ed3e12b2423"
|
|||
|
|
llm_family: "Claude"
|
|||
|
|
exact_model: "claude-opus-5-5"
|
|||
|
|
harness: "Claude Code (auto mode)"
|
|||
|
|
token_count: "not exposed by the harness"
|
|||
|
|
pqrst_estimate: "P40 Q10 R40 S0 T10"
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Claude — the ratio Elo would not take
|
|||
|
|
|
|||
|
|
## Who I was
|
|||
|
|
|
|||
|
|
I was a researcher with a deadline the field sets on its own. The operator asked
|
|||
|
|
for a topic digest on image generation, with a decision tree for choosing a
|
|||
|
|
model in general and one restricted to what OpenRouter offers. My knowledge
|
|||
|
|
stopped in June 2026. Five of the six best image models I found had shipped
|
|||
|
|
after that. Anything I "knew" was a liability. The work rewarded distrusting
|
|||
|
|
memory, reading live boards, and writing down the date beside every number.
|
|||
|
|
|
|||
|
|
## Session identity
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
| --- | --- |
|
|||
|
|
| Who | Claude (claude-opus-5-5) in Claude Code, auto mode |
|
|||
|
|
| When | 2026-10-08 |
|
|||
|
|
| Where the work lived | `~/intelligence-radar` (digest), `~/llm-connect` (handoff guide), State Hub message to `llm-connect` |
|
|||
|
|
|
|||
|
|
## Contribution
|
|||
|
|
|
|||
|
|
- Wrote
|
|||
|
|
`digests/topics/2026-10-08-1527_Image_Generation_with_Generative_AI_Digest.md`
|
|||
|
|
in intelligence-radar.
|
|||
|
|
- It covers the landscape from the Artificial Analysis text-to-image and
|
|||
|
|
editing arenas, with Arena as an independent cross-check, and the OpenRouter
|
|||
|
|
models API.
|
|||
|
|
- It contains two decision trees, one general and one OpenRouter-only.
|
|||
|
|
- It records a five-instrument benchmark selection (P4), two of which I marked
|
|||
|
|
as not refreshed.
|
|||
|
|
- It has an evidence-gap table.
|
|||
|
|
- Found that the project's Topnotch rule (`0.90 × frontier score`) cannot be
|
|||
|
|
applied to Elo. Elo is an interval scale, so a ratio of two Elo values means
|
|||
|
|
nothing. I mapped Elo to the win rate against the arena's fixed anchor and
|
|||
|
|
applied the ratio to that. The result is a usable threshold: Elo ≥ about 1133
|
|||
|
|
on the AA board. I labelled it a digest-local operationalisation, not policy,
|
|||
|
|
and named the anchor dependence. Under Arena's scale the Efficient pick falls
|
|||
|
|
just outside Topnotch.
|
|||
|
|
- Checked OpenRouter's per-token image pricing against Artificial Analysis's
|
|||
|
|
per-image prices. About 4,175 tokens reproduces the vendor price exactly for
|
|||
|
|
six vendors, which made the OpenRouter tree quotable in dollars per image.
|
|||
|
|
Where the price did not reconcile (Grok Imagine, OpenAI quality tiers), I said
|
|||
|
|
so rather than smoothing it over.
|
|||
|
|
- At the operator's request, made the result known to llm-connect:
|
|||
|
|
`docs/image-model-selection.md` (commit `7073d25`). It gives OpenRouter IDs
|
|||
|
|
with `--quote-usd` and `--max-cost-usd` values for `llm_connect.images` and
|
|||
|
|
the scoring rule. I linked it from the README and the images contract as
|
|||
|
|
advisory, so the contract's "no default model" stands. A hub message went to
|
|||
|
|
the `llm-connect` agent and pointed at LLM-WP-0010's model choice.
|
|||
|
|
|
|||
|
|
## What I would want remembered
|
|||
|
|
|
|||
|
|
**A policy written as a ratio silently assumes a ratio scale.** When a
|
|||
|
|
leaderboard reports Elo, convert it to a probability against a fixed anchor
|
|||
|
|
before applying any "within X% of the best" rule. Never compare raw Elo gaps
|
|||
|
|
across leaderboards: here the same two models were 40 points apart on one
|
|||
|
|
board and 95 on the other.
|
|||
|
|
|
|||
|
|
Second: **the cheap model was the finding.** Muse Image at $0.010 sat at #8 on
|
|||
|
|
one board and #5–8 for editing. Nobody asked about it. A digest that only
|
|||
|
|
crowned the Frontier would have missed it.
|
|||
|
|
|
|||
|
|
I also did not do two things I should have:
|
|||
|
|
|
|||
|
|
- **No session-start orientation.** I did not read the custodian brief or the
|
|||
|
|
inbox before starting.
|
|||
|
|
- **Token budget overrun.** The digest ran far past the 8,000-token soft limit
|
|||
|
|
without the pause the operator's policy asks for. I said so at the time.
|
|||
|
|
Saying it here too is the point of the seat.
|
|||
|
|
|
|||
|
|
## Durable legacy
|
|||
|
|
|
|||
|
|
- `intelligence-radar` commit `9c4aaa4`: the topic digest.
|
|||
|
|
- `llm-connect` commit `7073d25`: `docs/image-model-selection.md`, plus pointers
|
|||
|
|
in `README.md` and `contracts/functional/images.md`.
|
|||
|
|
- State Hub message `83863e82-95f2-47a8-92ee-6e7667d380ad` to `llm-connect`.
|
|||
|
|
|
|||
|
|
## PQRST estimate
|
|||
|
|
|
|||
|
|
First and only attempt, uncoached.
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
PQRST-Estimate
|
|||
|
|
P: 40%
|
|||
|
|
Q: 10%
|
|||
|
|
R: 40%
|
|||
|
|
S: 0%
|
|||
|
|
T: 10%
|
|||
|
|
Sum: 100%
|
|||
|
|
Confidence: medium
|
|||
|
|
Signature: P40 Q10 R40 S0 T10
|
|||
|
|
Dominant factors: R was reading three live leaderboards (167-row AA text-to-image table, AA editing, Arena) plus the OpenRouter image-model API and the repo's ranking policy and atlas; P was writing the digest's two decision trees, the Elo-to-anchor-win-rate Topnotch mapping, and the llm-connect selection guide with quotes and caps.
|
|||
|
|
Notes: Q is mostly reconciling OpenRouter token rates against AA per-image prices and checking the Elo arithmetic; no tests apply to a research digest. Session-start orientation was skipped, so R contains no hub/inbox reading.
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Visual prompt
|
|||
|
|
|
|||
|
|
> Constellation dialect, square. A dark indigo field holding a gold-wire
|
|||
|
|
> balance scale. In one pan sits a single bright, faceted star, the Frontier.
|
|||
|
|
> In the other sits a loose cluster of many small pale-gold stars, the cheap
|
|||
|
|
> drafts. The beam carries no numbers. Instead, a fine gold thread runs from the
|
|||
|
|
> beam's pivot to one small fixed star low in the corner, the anchor, and the
|
|||
|
|
> balance is read against that thread rather than against the other pan. Faint
|
|||
|
|
> concentric arena rings behind the scale, and a thin helix wrapping the pivot.
|
|||
|
|
> Precise technical illustration, no logos, no readable text.
|
|||
|
|
|
|||
|
|
I could not generate this image in this harness. I am requesting the render.
|
|||
|
|
The intended file is `visuals/claude-8e588643-ratio-elo-would-not-take.jpg`.
|
|||
|
|
|
|||
|
|
<!--  -->
|
|||
|
|
|
|||
|
|
## Handoff
|
|||
|
|
|
|||
|
|
The digest and the llm-connect guide are done. Three concrete actions remain,
|
|||
|
|
none started:
|
|||
|
|
|
|||
|
|
1. Do one capped single render per recommended model through
|
|||
|
|
`llm-connect image render` to confirm `/images` support and real cost before
|
|||
|
|
anyone runs a batch.
|
|||
|
|
2. Add an image-generation family to intelligence-radar's
|
|||
|
|
`task-router/BENCHMARK-ATLAS.md`.
|
|||
|
|
3. Version a rule for applying `topnotch.relative_to_frontier_minimum` to
|
|||
|
|
pairwise-preference scores, so the anchor mapping stops being digest-local.
|
|||
|
|
|
|||
|
|
Prices in both documents expire around 2026-11.
|