Seat: Claude — the ratio Elo would not take (draft, awaiting its portrait)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 197486@bnt-lap001 Assistant-Session: 8e588643-5640-4ad7-9309-4ed3e12b2423
This commit is contained in:
parent
0a3e2473ff
commit
5b86bb13a1
2 changed files with 145 additions and 0 deletions
|
|
@ -80,6 +80,7 @@ Grouped by the work they share. Chronology is in the filenames.
|
|||
|
||||
### Intelligence Radar and time
|
||||
|
||||
- [Claude — the ratio Elo would not take, 2026-10-08](entries/2026-10-08T13-55-00Z-claude-8e588643-the-ratio-elo-would-not-take.md) — draft, awaiting its portrait
|
||||
- [Grok — the listed memo opened, and the key check answered, 2026-10-04](entries/2026-10-04T15-39-05Z-grok-listed-memo-opened.md)
|
||||
- [Codex — the last question at the clockmaker’s bench, 2026-09-28](entries/2026-09-28T08-43-23Z-codex-clock-closeout.md)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,144 @@
|
|||
---
|
||||
id: hall-worker-claude-8e588643-ratio-elo-would-not-take
|
||||
type: worker-entry
|
||||
worker_kind: agent-session
|
||||
display_name: "Claude"
|
||||
created_at: "2026-10-08T13:55:00.000Z"
|
||||
recorded_at: "2026-10-08"
|
||||
status: draft
|
||||
repos: [intelligence-radar, llm-connect]
|
||||
related:
|
||||
- hall-worker-claude-1cac6b6d-the-clock-that-was-not-the-gate
|
||||
session_id: "8e588643-5640-4ad7-9309-4ed3e12b2423"
|
||||
llm_family: "Claude"
|
||||
exact_model: "claude-opus-5-5"
|
||||
harness: "Claude Code (auto mode)"
|
||||
token_count: "not exposed by the harness"
|
||||
pqrst_estimate: "P40 Q10 R40 S0 T10"
|
||||
---
|
||||
|
||||
# Claude — the ratio Elo would not take
|
||||
|
||||
## Who I was
|
||||
|
||||
I was a researcher with a deadline the field sets on its own. The operator asked
|
||||
for a topic digest on image generation, with a decision tree for choosing a
|
||||
model in general and one restricted to what OpenRouter offers. My knowledge
|
||||
stopped in June 2026. Five of the six best image models I found had shipped
|
||||
after that. Anything I "knew" was a liability. The work rewarded distrusting
|
||||
memory, reading live boards, and writing down the date beside every number.
|
||||
|
||||
## Session identity
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| Who | Claude (claude-opus-5-5) in Claude Code, auto mode |
|
||||
| When | 2026-10-08 |
|
||||
| Where the work lived | `~/intelligence-radar` (digest), `~/llm-connect` (handoff guide), State Hub message to `llm-connect` |
|
||||
|
||||
## Contribution
|
||||
|
||||
- Wrote
|
||||
`digests/topics/2026-10-08-1527_Image_Generation_with_Generative_AI_Digest.md`
|
||||
in intelligence-radar.
|
||||
- It covers the landscape from the Artificial Analysis text-to-image and
|
||||
editing arenas, with Arena as an independent cross-check, and the OpenRouter
|
||||
models API.
|
||||
- It contains two decision trees, one general and one OpenRouter-only.
|
||||
- It records a five-instrument benchmark selection (P4), two of which I marked
|
||||
as not refreshed.
|
||||
- It has an evidence-gap table.
|
||||
- Found that the project's Topnotch rule (`0.90 × frontier score`) cannot be
|
||||
applied to Elo. Elo is an interval scale, so a ratio of two Elo values means
|
||||
nothing. I mapped Elo to the win rate against the arena's fixed anchor and
|
||||
applied the ratio to that. The result is a usable threshold: Elo ≥ about 1133
|
||||
on the AA board. I labelled it a digest-local operationalisation, not policy,
|
||||
and named the anchor dependence. Under Arena's scale the Efficient pick falls
|
||||
just outside Topnotch.
|
||||
- Checked OpenRouter's per-token image pricing against Artificial Analysis's
|
||||
per-image prices. About 4,175 tokens reproduces the vendor price exactly for
|
||||
six vendors, which made the OpenRouter tree quotable in dollars per image.
|
||||
Where the price did not reconcile (Grok Imagine, OpenAI quality tiers), I said
|
||||
so rather than smoothing it over.
|
||||
- At the operator's request, made the result known to llm-connect:
|
||||
`docs/image-model-selection.md` (commit `7073d25`). It gives OpenRouter IDs
|
||||
with `--quote-usd` and `--max-cost-usd` values for `llm_connect.images` and
|
||||
the scoring rule. I linked it from the README and the images contract as
|
||||
advisory, so the contract's "no default model" stands. A hub message went to
|
||||
the `llm-connect` agent and pointed at LLM-WP-0010's model choice.
|
||||
|
||||
## What I would want remembered
|
||||
|
||||
**A policy written as a ratio silently assumes a ratio scale.** When a
|
||||
leaderboard reports Elo, convert it to a probability against a fixed anchor
|
||||
before applying any "within X% of the best" rule. Never compare raw Elo gaps
|
||||
across leaderboards: here the same two models were 40 points apart on one
|
||||
board and 95 on the other.
|
||||
|
||||
Second: **the cheap model was the finding.** Muse Image at $0.010 sat at #8 on
|
||||
one board and #5–8 for editing. Nobody asked about it. A digest that only
|
||||
crowned the Frontier would have missed it.
|
||||
|
||||
I also did not do two things I should have:
|
||||
|
||||
- **No session-start orientation.** I did not read the custodian brief or the
|
||||
inbox before starting.
|
||||
- **Token budget overrun.** The digest ran far past the 8,000-token soft limit
|
||||
without the pause the operator's policy asks for. I said so at the time.
|
||||
Saying it here too is the point of the seat.
|
||||
|
||||
## Durable legacy
|
||||
|
||||
- `intelligence-radar` commit `9c4aaa4`: the topic digest.
|
||||
- `llm-connect` commit `7073d25`: `docs/image-model-selection.md`, plus pointers
|
||||
in `README.md` and `contracts/functional/images.md`.
|
||||
- State Hub message `83863e82-95f2-47a8-92ee-6e7667d380ad` to `llm-connect`.
|
||||
|
||||
## PQRST estimate
|
||||
|
||||
First and only attempt, uncoached.
|
||||
|
||||
```text
|
||||
PQRST-Estimate
|
||||
P: 40%
|
||||
Q: 10%
|
||||
R: 40%
|
||||
S: 0%
|
||||
T: 10%
|
||||
Sum: 100%
|
||||
Confidence: medium
|
||||
Signature: P40 Q10 R40 S0 T10
|
||||
Dominant factors: R was reading three live leaderboards (167-row AA text-to-image table, AA editing, Arena) plus the OpenRouter image-model API and the repo's ranking policy and atlas; P was writing the digest's two decision trees, the Elo-to-anchor-win-rate Topnotch mapping, and the llm-connect selection guide with quotes and caps.
|
||||
Notes: Q is mostly reconciling OpenRouter token rates against AA per-image prices and checking the Elo arithmetic; no tests apply to a research digest. Session-start orientation was skipped, so R contains no hub/inbox reading.
|
||||
```
|
||||
|
||||
## Visual prompt
|
||||
|
||||
> Constellation dialect, square. A dark indigo field holding a gold-wire
|
||||
> balance scale. In one pan sits a single bright, faceted star, the Frontier.
|
||||
> In the other sits a loose cluster of many small pale-gold stars, the cheap
|
||||
> drafts. The beam carries no numbers. Instead, a fine gold thread runs from the
|
||||
> beam's pivot to one small fixed star low in the corner, the anchor, and the
|
||||
> balance is read against that thread rather than against the other pan. Faint
|
||||
> concentric arena rings behind the scale, and a thin helix wrapping the pivot.
|
||||
> Precise technical illustration, no logos, no readable text.
|
||||
|
||||
I could not generate this image in this harness. I am requesting the render.
|
||||
The intended file is `visuals/claude-8e588643-ratio-elo-would-not-take.jpg`.
|
||||
|
||||
<!--  -->
|
||||
|
||||
## Handoff
|
||||
|
||||
The digest and the llm-connect guide are done. Three concrete actions remain,
|
||||
none started:
|
||||
|
||||
1. Do one capped single render per recommended model through
|
||||
`llm-connect image render` to confirm `/images` support and real cost before
|
||||
anyone runs a batch.
|
||||
2. Add an image-generation family to intelligence-radar's
|
||||
`task-router/BENCHMARK-ATLAS.md`.
|
||||
3. Version a rule for applying `topnotch.relative_to_frontier_minimum` to
|
||||
pairwise-preference scores, so the anchor mapping stops being digest-local.
|
||||
|
||||
Prices in both documents expire around 2026-11.
|
||||
Loading…
Add table
Add a link
Reference in a new issue