coulomb/llm-connect
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add image-model selection guide from the intelligence-radar digest
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 18s
Details

Browse source 
Maps the 2026-10-08 image-generation decision tree onto OpenRouter model IDs
with per-image quotes and caps for llm_connect.images, and documents the
anchor-win-rate scoring that makes Elo leaderboards usable for Topnotch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 197486@bnt-lap001
Assistant-Session: 8e588643-5640-4ad7-9309-4ed3e12b2423
This commit is contained in:
tegwick 2026-10-08 16:33:32 +02:00
parent 19fcb19e84
commit 7073d2543c
3 changed files with 371 additions and 281 deletions
Download patch file Download diff file Expand all files Collapse all files

561
README.md
View file

@ -1,280 +1,281 @@
# llm-connect
Pluggable LLM adapters for Python and the commandline. Supports OpenRouter, Gemini,
OpenAI, and the Claude Code CLI out of the box, with a clean abstract interface for adding
your own.
## Quick start
```python
from llm_connect import create_adapter, RunConfig
adapter = create_adapter("gemini", model="gemini-2.5-flash")
config = RunConfig(temperature=0.7, max_tokens=1000)
response = adapter.execute_prompt("Summarise the value chain concept.", config)
print(response.content)
```
## Installation
```bash
pip install -e /path/to/llm-connect # local editable install
# or, once published:
pip install llm-connect
```
**Requires:** Python 3.10+, `toml`
## Providers
| Provider key | Class | Notes |
|---|---|---|
| `"openrouter"` | `OpenRouterAdapter` | OpenAI-compatible endpoint; supports all OpenRouter models |
| `"gemini"` | `GeminiAdapter` | Google Generative Language REST API; supports free tier |
```python
from llm_connect import create_adapter
# OpenRouter (default model: moonshotai/kimi-k3)
adapter = create_adapter("openrouter")
adapter = create_adapter("openrouter", model="anthropic/claude-sonnet-4")
# Gemini (uses GEMINI_API_KEY env var or apikey-geminifree.txt)
adapter = create_adapter("gemini", model="gemini-2.5-flash")
# OpenAI (uses OPENAI_API_KEY env var)
adapter = create_adapter("openai", model="gpt-4.1-mini")
# Claude Code CLI (uses locally installed claude binary)
adapter = create_adapter("claude-code")
```
## CLI
```bash
# Run a prompt; content on stdout, tokens + €/USD cost on stderr
llm-connect run "Summarise the value chain concept."
llm-connect run "Hello" --provider mock --model moonshotai/kimi-k3 --json
# Estimate cost without calling a provider
llm-connect cost estimate --model moonshotai/kimi-k3 \
--prompt-tokens 1000 --completion-tokens 500
# Weekly spend (Mon-based; default timezone Europe/Berlin)
llm-connect spend week # current week: Mon 00:00 → now
llm-connect spend week --last # previous week: Mon → Sun
llm-connect spend week --json
# Provider prepaid / key remaining (backend-scoped; does not change defaults)
llm-connect balance # current default provider
llm-connect balance --provider openrouter # one-shot backend select
llm-connect balance --json
# Square text-to-image. The model and the quote are required when a cap is set.
# Receipt JSON on stdout; image bytes in --output. Credentials stay in the env.
llm-connect image render \
--prompt "precise technical illustration of a gear, no text" \
--model "<image-model>" \
--output visuals/gear.jpg \
--max-cost-usd 0.15 \
--quote-usd 0.08
```
Usage events append to a JSONL ledger (default
`~/.local/share/llm-connect/usage.jsonl`; override with `--ledger` or
`LLM_CONNECT_USAGE_LEDGER`). Costs are list-price estimates (USD rate table +
EUR via snapshot or `LLM_CONNECT_EUR_PER_USD`).
`balance` is **provider-scoped**: it reports the selected backend’s prepaid /
key remaining (OpenRouter today). `--provider` applies only to that
invocation and does not switch the library default provider or model.
Unsupported backends fail clearly instead of falling back to OpenRouter.
| Env | Purpose |
|---|---|
| `LLM_CONNECT_USAGE_LEDGER` | Usage JSONL path; enables library/server auto-recording when set |
| `LLM_CONNECT_EUR_PER_USD` | Euros per one USD (FX override) |
| `LLM_CONNECT_TZ` | Timezone for weekly spend windows (default `Europe/Berlin`) |
## API keys
Keys are resolved in this order (first found wins):
1. Explicit `api_key` argument to the constructor
2. Environment variable (e.g. `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `OPENAI_API_KEY`)
3. Key file in the project root (e.g. `apikey-openrouter.txt`, `apikey-geminifree.txt`)
## Core types
### `RunConfig`
Controls a single LLM call.
```python
from llm_connect import RunConfig
config = RunConfig(
model_name="gemini-2.5-flash", # overrides adapter default
temperature=0.3,
max_tokens=2000,
timeout_seconds=60,
)
```
| Field | Default | Description |
|---|---|---|
| `model_name` | `"gpt-4"` | Model identifier (adapter may override) |
| `temperature` | `0.7` | Sampling temperature |
| `max_tokens` | `2000` | Maximum output tokens |
| `model_params` | `{}` | Portable extras translated by each adapter; see `docs/adapter-model-params.md` |
| `max_depth` | `3` | Max nesting depth for recursive calls |
| `skip_if_exists` | `True` | Skip if identical input hash already processed |
| `timeout_seconds` | `300` | Request timeout |
### `LLMResponse`
Returned by every `execute_prompt` call.
```python
response = adapter.execute_prompt(prompt, config)
print(response.content) # generated text
print(response.model) # model actually used
print(response.usage) # {"prompt_tokens": …, "completion_tokens": …, "total_tokens": …}
print(response.finish_reason) # "stop", "length", etc.
```
## Server diagnostics
Serve mode can include a debug envelope without changing normal responses:
```bash
LLM_CONNECT_DEBUG=1 python -m llm_connect.server --provider openrouter
curl 'http://127.0.0.1:8080/execute?debug=1' -d '{"prompt":"hi"}'
```
Set `LLM_CONNECT_AUDIT_DIR=/path/to/audit` to write per-call replay records,
then parse one without another provider call:
```bash
python -m llm_connect.replay /path/to/audit/record.json --json
```
## Server runtime profiles
Serve mode enables named runtime profiles by default. A client can send
`config.model_name="custodian-triage-balanced"` and the server resolves it to
the configured provider/model before calling the adapter.
Useful runtime environment variables:
```bash
LLM_CONNECT_HOST=0.0.0.0
LLM_CONNECT_PORT=8080
LLM_CONNECT_PROVIDER=openrouter
LLM_CONNECT_MODEL=google/gemini-2.5-flash
LLM_CONNECT_CUSTODIAN_TRIAGE_PROVIDER=openrouter
LLM_CONNECT_CUSTODIAN_TRIAGE_MODEL=google/gemini-2.5-flash
```
For local smoke tests without provider credentials:
```bash
export LLM_CONNECT_MOCK_RESPONSE="$(python -c 'import json; print(json.dumps(json.load(open("fixtures/activity_core/daily-triage-valid-content.json"))))')"
python -m llm_connect.server --provider mock
python scripts/smoke_activity_core_endpoint.py --url http://127.0.0.1:8080
```
Disable profile dispatch with `--disable-profiles`. Set
`LLM_CONNECT_STRICT_PROFILES=1` or pass `--strict-profiles` to reject direct
model names that are not configured profiles.
## Writing your own adapter
```python
from llm_connect import LLMAdapter, RunConfig, LLMResponse
class MyAdapter(LLMAdapter):
def execute_prompt(self, prompt: str, config: RunConfig) -> LLMResponse:
# call your API here
return LLMResponse(content="...", model="my-model")
def validate_config(self, config: RunConfig) -> bool:
return True
```
## TOML configuration chain
The `resolve_llm()` function walks a 7-level priority chain to pick a
provider and model. This is used by the `llm-helper` integration but is also
available standalone:
```python
from llm_connect.toml_config import resolve_llm
resolved = resolve_llm(app_name="myapp")
print(resolved.provider, resolved.model, resolved.provider_source)
```
Priority order (highest first):
1. CLI flags (`cli_provider`, `cli_model` arguments)
2. Env var `{APP_NAME}_HELPER_MODEL` (model only)
3. User preference — `~/.config/{app_name}/config.toml` `[llm.preference]`
4. Directory preference — `.{app_name}.toml` `[llm.preference]`
5. Directory default — `.{app_name}.toml` `[llm.default]`
6. User default — `~/.config/{app_name}/config.toml` `[llm.default]`
7. Hardcoded fallback — `gemini / gemini-2.5-flash`
Example config file (`~/.config/myapp/config.toml`):
```toml
[llm.default]
provider = "gemini"
model = "gemini-2.5-flash"
[llm.preference]
provider = "openrouter"
model = "anthropic/claude-sonnet-4"
```
## Embeddings
```python
from llm_connect import create_embedding_adapter, EmbeddingCache
adapter = create_embedding_adapter("openai", model="text-embedding-3-small")
cache = EmbeddingCache(cache_dir=".embeddings")
# Get embedding (cached after first call)
vec = cache.get_or_compute("my text", lambda t: adapter.embed([t])[0])
```
## Exceptions
```python
from llm_connect.exceptions import (
LLMError, # base
LLMConfigurationError,# bad key, unknown provider
LLMAPIError, # HTTP error from provider (has .status_code)
LLMRateLimitError, # 429
LLMTimeoutError, # request timed out
LLMSubprocessError, # claude CLI failed (has .return_code, .stderr)
)
```
## Testing
```python
from llm_connect import MockLLMAdapter, RunConfig
mock = MockLLMAdapter(mock_response="Test response")
config = RunConfig()
response = mock.execute_prompt("any prompt", config)
assert response.content == "Test response"
assert mock.call_count == 1
```
## Origin
Extracted from the [markitect](https://github.com/worsch/markitect) project.
The `markitect.llm` module remains a re-export shim pointing here.
# llm-connect
Pluggable LLM adapters for Python and the commandline. Supports OpenRouter, Gemini,
OpenAI, and the Claude Code CLI out of the box, with a clean abstract interface for adding
your own.
## Quick start
```python
from llm_connect import create_adapter, RunConfig
adapter = create_adapter("gemini", model="gemini-2.5-flash")
config = RunConfig(temperature=0.7, max_tokens=1000)
response = adapter.execute_prompt("Summarise the value chain concept.", config)
print(response.content)
```
## Installation
```bash
pip install -e /path/to/llm-connect # local editable install
# or, once published:
pip install llm-connect
```
**Requires:** Python 3.10+, `toml`
## Providers
| Provider key | Class | Notes |
|---|---|---|
| `"openrouter"` | `OpenRouterAdapter` | OpenAI-compatible endpoint; supports all OpenRouter models |
| `"gemini"` | `GeminiAdapter` | Google Generative Language REST API; supports free tier |
```python
from llm_connect import create_adapter
# OpenRouter (default model: moonshotai/kimi-k3)
adapter = create_adapter("openrouter")
adapter = create_adapter("openrouter", model="anthropic/claude-sonnet-4")
# Gemini (uses GEMINI_API_KEY env var or apikey-geminifree.txt)
adapter = create_adapter("gemini", model="gemini-2.5-flash")
# OpenAI (uses OPENAI_API_KEY env var)
adapter = create_adapter("openai", model="gpt-4.1-mini")
# Claude Code CLI (uses locally installed claude binary)
adapter = create_adapter("claude-code")
```
## CLI
```bash
# Run a prompt; content on stdout, tokens + €/USD cost on stderr
llm-connect run "Summarise the value chain concept."
llm-connect run "Hello" --provider mock --model moonshotai/kimi-k3 --json
# Estimate cost without calling a provider
llm-connect cost estimate --model moonshotai/kimi-k3 \
--prompt-tokens 1000 --completion-tokens 500
# Weekly spend (Mon-based; default timezone Europe/Berlin)
llm-connect spend week # current week: Mon 00:00 → now
llm-connect spend week --last # previous week: Mon → Sun
llm-connect spend week --json
# Provider prepaid / key remaining (backend-scoped; does not change defaults)
llm-connect balance # current default provider
llm-connect balance --provider openrouter # one-shot backend select
llm-connect balance --json
# Square text-to-image. The model and the quote are required when a cap is set.
# Which model and quote: docs/image-model-selection.md
# Receipt JSON on stdout; image bytes in --output. Credentials stay in the env.
llm-connect image render \
--prompt "precise technical illustration of a gear, no text" \
--model "<image-model>" \
--output visuals/gear.jpg \
--max-cost-usd 0.15 \
--quote-usd 0.08
```
Usage events append to a JSONL ledger (default
`~/.local/share/llm-connect/usage.jsonl`; override with `--ledger` or
`LLM_CONNECT_USAGE_LEDGER`). Costs are list-price estimates (USD rate table +
EUR via snapshot or `LLM_CONNECT_EUR_PER_USD`).
`balance` is **provider-scoped**: it reports the selected backend’s prepaid /
key remaining (OpenRouter today). `--provider` applies only to that
invocation and does not switch the library default provider or model.
Unsupported backends fail clearly instead of falling back to OpenRouter.
| Env | Purpose |
|---|---|
| `LLM_CONNECT_USAGE_LEDGER` | Usage JSONL path; enables library/server auto-recording when set |
| `LLM_CONNECT_EUR_PER_USD` | Euros per one USD (FX override) |
| `LLM_CONNECT_TZ` | Timezone for weekly spend windows (default `Europe/Berlin`) |
## API keys
Keys are resolved in this order (first found wins):
1. Explicit `api_key` argument to the constructor
2. Environment variable (e.g. `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `OPENAI_API_KEY`)
3. Key file in the project root (e.g. `apikey-openrouter.txt`, `apikey-geminifree.txt`)
## Core types
### `RunConfig`
Controls a single LLM call.
```python
from llm_connect import RunConfig
config = RunConfig(
model_name="gemini-2.5-flash", # overrides adapter default
temperature=0.3,
max_tokens=2000,
timeout_seconds=60,
)
```
| Field | Default | Description |
|---|---|---|
| `model_name` | `"gpt-4"` | Model identifier (adapter may override) |
| `temperature` | `0.7` | Sampling temperature |
| `max_tokens` | `2000` | Maximum output tokens |
| `model_params` | `{}` | Portable extras translated by each adapter; see `docs/adapter-model-params.md` |
| `max_depth` | `3` | Max nesting depth for recursive calls |
| `skip_if_exists` | `True` | Skip if identical input hash already processed |
| `timeout_seconds` | `300` | Request timeout |
### `LLMResponse`
Returned by every `execute_prompt` call.
```python
response = adapter.execute_prompt(prompt, config)
print(response.content) # generated text
print(response.model) # model actually used
print(response.usage) # {"prompt_tokens": …, "completion_tokens": …, "total_tokens": …}
print(response.finish_reason) # "stop", "length", etc.
```
## Server diagnostics
Serve mode can include a debug envelope without changing normal responses:
```bash
LLM_CONNECT_DEBUG=1 python -m llm_connect.server --provider openrouter
curl 'http://127.0.0.1:8080/execute?debug=1' -d '{"prompt":"hi"}'
```
Set `LLM_CONNECT_AUDIT_DIR=/path/to/audit` to write per-call replay records,
then parse one without another provider call:
```bash
python -m llm_connect.replay /path/to/audit/record.json --json
```
## Server runtime profiles
Serve mode enables named runtime profiles by default. A client can send
`config.model_name="custodian-triage-balanced"` and the server resolves it to
the configured provider/model before calling the adapter.
Useful runtime environment variables:
```bash
LLM_CONNECT_HOST=0.0.0.0
LLM_CONNECT_PORT=8080
LLM_CONNECT_PROVIDER=openrouter
LLM_CONNECT_MODEL=google/gemini-2.5-flash
LLM_CONNECT_CUSTODIAN_TRIAGE_PROVIDER=openrouter
LLM_CONNECT_CUSTODIAN_TRIAGE_MODEL=google/gemini-2.5-flash
```
For local smoke tests without provider credentials:
```bash
export LLM_CONNECT_MOCK_RESPONSE="$(python -c 'import json; print(json.dumps(json.load(open("fixtures/activity_core/daily-triage-valid-content.json"))))')"
python -m llm_connect.server --provider mock
python scripts/smoke_activity_core_endpoint.py --url http://127.0.0.1:8080
```
Disable profile dispatch with `--disable-profiles`. Set
`LLM_CONNECT_STRICT_PROFILES=1` or pass `--strict-profiles` to reject direct
model names that are not configured profiles.
## Writing your own adapter
```python
from llm_connect import LLMAdapter, RunConfig, LLMResponse
class MyAdapter(LLMAdapter):
def execute_prompt(self, prompt: str, config: RunConfig) -> LLMResponse:
# call your API here
return LLMResponse(content="...", model="my-model")
def validate_config(self, config: RunConfig) -> bool:
return True
```
## TOML configuration chain
The `resolve_llm()` function walks a 7-level priority chain to pick a
provider and model. This is used by the `llm-helper` integration but is also
available standalone:
```python
from llm_connect.toml_config import resolve_llm
resolved = resolve_llm(app_name="myapp")
print(resolved.provider, resolved.model, resolved.provider_source)
```
Priority order (highest first):
1. CLI flags (`cli_provider`, `cli_model` arguments)
2. Env var `{APP_NAME}_HELPER_MODEL` (model only)
3. User preference — `~/.config/{app_name}/config.toml` `[llm.preference]`
4. Directory preference — `.{app_name}.toml` `[llm.preference]`
5. Directory default — `.{app_name}.toml` `[llm.default]`
6. User default — `~/.config/{app_name}/config.toml` `[llm.default]`
7. Hardcoded fallback — `gemini / gemini-2.5-flash`
Example config file (`~/.config/myapp/config.toml`):
```toml
[llm.default]
provider = "gemini"
model = "gemini-2.5-flash"
[llm.preference]
provider = "openrouter"
model = "anthropic/claude-sonnet-4"
```
## Embeddings
```python
from llm_connect import create_embedding_adapter, EmbeddingCache
adapter = create_embedding_adapter("openai", model="text-embedding-3-small")
cache = EmbeddingCache(cache_dir=".embeddings")
# Get embedding (cached after first call)
vec = cache.get_or_compute("my text", lambda t: adapter.embed([t])[0])
```
## Exceptions
```python
from llm_connect.exceptions import (
LLMError, # base
LLMConfigurationError,# bad key, unknown provider
LLMAPIError, # HTTP error from provider (has .status_code)
LLMRateLimitError, # 429
LLMTimeoutError, # request timed out
LLMSubprocessError, # claude CLI failed (has .return_code, .stderr)
)
```
## Testing
```python
from llm_connect import MockLLMAdapter, RunConfig
mock = MockLLMAdapter(mock_response="Test response")
config = RunConfig()
response = mock.execute_prompt("any prompt", config)
assert response.content == "Test response"
assert mock.call_count == 1
```
## Origin
Extracted from the [markitect](https://github.com/worsch/markitect) project.
The `markitect.llm` module remains a re-export shim pointing here.

3
contracts/functional/images.md
View file

@ -43,7 +43,8 @@ comes from the explicit argument, `OPENROUTER_API_KEY`, or
`apikey-openrouter.txt`. Only `b64_json` is accepted. A URL in the response
is not fetched.
There is no default image model.
There is no default image model. Advisory selection guidance, outside this
contract: [`docs/image-model-selection.md`](../../docs/image-model-selection.md).
## CLI

88
docs/image-model-selection.md Normal file
View file

@ -0,0 +1,88 @@
# Choosing an image model for `llm_connect.images`
`llm_connect.images` has no default image model and does not choose one (see
[`contracts/functional/images.md`](../contracts/functional/images.md)). This
guide is advisory input for the caller. It does not change that contract.
**Source:** intelligence-radar topic digest
`digests/topics/2026-10-08-1527_Image_Generation_with_Generative_AI_Digest.md`
(evidence observed 2026-10-08). Rankings and prices in this field change
monthly. Re-derive the table from the digest's sources before relying on it
after about 2026-11.
## Picking a model
All IDs below are OpenRouter model IDs. Prices are for one 1024² image. The
quote is the per-image estimate plus some margin, and the cap leaves room for
the provider-reported cost to come in above the quote.
| Need | Model | ≈ $/img | `--quote-usd` | `--max-cost-usd` |
|---|---|---|---|---|
| Best result, cost secondary (Frontier) | `openai/gpt-image-2.5-sunburst` | 0.21 | 0.25 | 0.30 |
| Same quality, faster | `openai/gpt-image-2.5-flare` | 0.21 | 0.25 | 0.30 |
| Near-frontier at scale (Efficient) | `google/gemini-nano-banana-2.1` | 0.034 | 0.04 | 0.08 |
| Efficient, better-evidenced fallback | `microsoft/mai-image-2.6` | 0.039 | 0.045 | 0.08 |
| Routine imagery (Budget) | `meta/muse-image` | 0.010 | 0.012 | 0.03 |
| Drafts, thumbnails, bulk variants | `recraft/recraft-v4.1-flash` | 0.007 | 0.008 | 0.02 |
| Text-heavy (posters, infographics, CJK) | `openai/gpt-image-2.5-sunburst` | 0.21 | 0.25 | 0.30 |
| Brand / consistent style set | `recraft/recraft-v4-styles` | 0.035 | 0.04 | 0.08 |
| Vector-style output | `recraft/recraft-v4.1-vector` | 0.08 | 0.09 | 0.15 |
Decision order:
1. Need vector output, brand style, or heavy text? Take that row.
2. Otherwise decide how much an excellent image is worth compared with a good
one:
- the image is the product → Frontier;
- many images, and quality matters → Efficient;
- illustrative or routine → Budget;
- exploration → bulk.
3. For important assets, **explore cheap, then finish expensive**. Render 10–20
Budget variants, then re-render the chosen one or two with the Frontier
model. 20 × Muse plus 2 × Sunburst costs about the same as three Frontier
attempts. This is a hypothesis to test with quality-ledger rows, not a
measured result.
Not available on OpenRouter, and therefore not reachable from this module:
self-hostable open weights (Qwen-Image-2.1, FLUX.2 [dev]), Ideogram,
Midjourney and Reve.
## Scoring rule behind the classes
Image leaderboards publish pairwise-preference **Elo**. Elo is an interval
scale, so "within 10% of the best model" cannot be computed as a ratio of Elo
values. The digest makes the scores usable as follows:
```
task_score(m) = P(m beats anchor) = 1 / (1 + 10^(-(Elo_m - Elo_anchor)/400))
```
Here the anchor is a fixed reference model. On the Artificial Analysis board
that is FLUX.2 [dev], Elo 1000.
- **Frontier:** the model with the highest `task_score`.
- **Topnotch:** `task_score >= 0.90 × task_score(frontier)`. With the
Frontier at Elo 1198, this means Elo ≥ about 1133.
- **Efficient:** the cheapest Topnotch model.
- **Sufficient / Budget:** a per-task floor. For routine imagery the digest uses
a provisional floor of Elo ≳ 1100, the level of the frontier from about ten
months earlier. Budget is the cheapest model above that floor.
Callers that already use `AdaptiveRoutingPolicy` quality floors can use the
same mapping. Convert Elo to an anchor win rate first, then apply a ratio
floor. Do not threshold raw Elo differences across leaderboards: the scales
differ. Arena's top gap is about 95 points where Artificial Analysis shows
about 40.
## Caveats
- OpenRouter bills most image models through a per-token rate. For Recraft,
Krea, Muse, Qwen, Seedream and FLUX 3, about 4,175 image tokens reproduces
the vendor's per-image price. For OpenAI and Google, cost depends on the
quality tier. Let the receipt's `cost_source=provider` correct the quotes
after the first runs.
- It is not verified here that every listed model accepts OpenRouter's
`/images` endpoint with explicit `size` and `aspect_ratio=1:1`. Do one
capped single render per model before running a batch.
- Nano Banana 2.1 had only a few thousand arena votes on 2026-10-08. If its
score drops, MAI-Image-2.6 becomes the Efficient pick.