tegwick
ed7c632155
Add pluggable balance registry, OpenRouter credits/key limit client, and llm-connect balance with one-shot --provider that does not change library defaults.
271 lines
8.7 KiB
Markdown
271 lines
8.7 KiB
Markdown
# llm-connect
|
||
|
||
Pluggable LLM adapters for Python and the commandline. Supports OpenRouter, Gemini,
|
||
OpenAI, and the Claude Code CLI out of the box, with a clean abstract interface for adding
|
||
your own.
|
||
|
||
## Quick start
|
||
|
||
```python
|
||
from llm_connect import create_adapter, RunConfig
|
||
|
||
adapter = create_adapter("gemini", model="gemini-2.5-flash")
|
||
config = RunConfig(temperature=0.7, max_tokens=1000)
|
||
response = adapter.execute_prompt("Summarise the value chain concept.", config)
|
||
print(response.content)
|
||
```
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
pip install -e /path/to/llm-connect # local editable install
|
||
# or, once published:
|
||
pip install llm-connect
|
||
```
|
||
|
||
**Requires:** Python 3.10+, `toml`
|
||
|
||
## Providers
|
||
|
||
| Provider key | Class | Notes |
|
||
|---|---|---|
|
||
| `"openrouter"` | `OpenRouterAdapter` | OpenAI-compatible endpoint; supports all OpenRouter models |
|
||
| `"gemini"` | `GeminiAdapter` | Google Generative Language REST API; supports free tier |
|
||
|
||
```python
|
||
from llm_connect import create_adapter
|
||
|
||
# OpenRouter (default model: moonshotai/kimi-k3)
|
||
adapter = create_adapter("openrouter")
|
||
adapter = create_adapter("openrouter", model="anthropic/claude-sonnet-4")
|
||
|
||
# Gemini (uses GEMINI_API_KEY env var or apikey-geminifree.txt)
|
||
adapter = create_adapter("gemini", model="gemini-2.5-flash")
|
||
|
||
# OpenAI (uses OPENAI_API_KEY env var)
|
||
adapter = create_adapter("openai", model="gpt-4.1-mini")
|
||
|
||
# Claude Code CLI (uses locally installed claude binary)
|
||
adapter = create_adapter("claude-code")
|
||
```
|
||
|
||
## CLI
|
||
|
||
```bash
|
||
# Run a prompt; content on stdout, tokens + €/USD cost on stderr
|
||
llm-connect run "Summarise the value chain concept."
|
||
llm-connect run "Hello" --provider mock --model moonshotai/kimi-k3 --json
|
||
|
||
# Estimate cost without calling a provider
|
||
llm-connect cost estimate --model moonshotai/kimi-k3 \
|
||
--prompt-tokens 1000 --completion-tokens 500
|
||
|
||
# Weekly spend (Mon-based; default timezone Europe/Berlin)
|
||
llm-connect spend week # current week: Mon 00:00 → now
|
||
llm-connect spend week --last # previous week: Mon → Sun
|
||
llm-connect spend week --json
|
||
|
||
# Provider prepaid / key remaining (backend-scoped; does not change defaults)
|
||
llm-connect balance # current default provider
|
||
llm-connect balance --provider openrouter # one-shot backend select
|
||
llm-connect balance --json
|
||
```
|
||
|
||
Usage events append to a JSONL ledger (default
|
||
`~/.local/share/llm-connect/usage.jsonl`; override with `--ledger` or
|
||
`LLM_CONNECT_USAGE_LEDGER`). Costs are list-price estimates (USD rate table +
|
||
EUR via snapshot or `LLM_CONNECT_EUR_PER_USD`).
|
||
|
||
`balance` is **provider-scoped**: it reports the selected backend’s prepaid /
|
||
key remaining (OpenRouter today). `--provider` applies only to that
|
||
invocation and does not switch the library default provider or model.
|
||
Unsupported backends fail clearly instead of falling back to OpenRouter.
|
||
|
||
| Env | Purpose |
|
||
|---|---|
|
||
| `LLM_CONNECT_USAGE_LEDGER` | Usage JSONL path; enables library/server auto-recording when set |
|
||
| `LLM_CONNECT_EUR_PER_USD` | Euros per one USD (FX override) |
|
||
| `LLM_CONNECT_TZ` | Timezone for weekly spend windows (default `Europe/Berlin`) |
|
||
|
||
## API keys
|
||
|
||
Keys are resolved in this order (first found wins):
|
||
|
||
1. Explicit `api_key` argument to the constructor
|
||
2. Environment variable (e.g. `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `OPENAI_API_KEY`)
|
||
3. Key file in the project root (e.g. `apikey-openrouter.txt`, `apikey-geminifree.txt`)
|
||
|
||
## Core types
|
||
|
||
### `RunConfig`
|
||
|
||
Controls a single LLM call.
|
||
|
||
```python
|
||
from llm_connect import RunConfig
|
||
|
||
config = RunConfig(
|
||
model_name="gemini-2.5-flash", # overrides adapter default
|
||
temperature=0.3,
|
||
max_tokens=2000,
|
||
timeout_seconds=60,
|
||
)
|
||
```
|
||
|
||
| Field | Default | Description |
|
||
|---|---|---|
|
||
| `model_name` | `"gpt-4"` | Model identifier (adapter may override) |
|
||
| `temperature` | `0.7` | Sampling temperature |
|
||
| `max_tokens` | `2000` | Maximum output tokens |
|
||
| `model_params` | `{}` | Portable extras translated by each adapter; see `docs/adapter-model-params.md` |
|
||
| `max_depth` | `3` | Max nesting depth for recursive calls |
|
||
| `skip_if_exists` | `True` | Skip if identical input hash already processed |
|
||
| `timeout_seconds` | `300` | Request timeout |
|
||
|
||
### `LLMResponse`
|
||
|
||
Returned by every `execute_prompt` call.
|
||
|
||
```python
|
||
response = adapter.execute_prompt(prompt, config)
|
||
print(response.content) # generated text
|
||
print(response.model) # model actually used
|
||
print(response.usage) # {"prompt_tokens": …, "completion_tokens": …, "total_tokens": …}
|
||
print(response.finish_reason) # "stop", "length", etc.
|
||
```
|
||
|
||
## Server diagnostics
|
||
|
||
Serve mode can include a debug envelope without changing normal responses:
|
||
|
||
```bash
|
||
LLM_CONNECT_DEBUG=1 python -m llm_connect.server --provider openrouter
|
||
curl 'http://127.0.0.1:8080/execute?debug=1' -d '{"prompt":"hi"}'
|
||
```
|
||
|
||
Set `LLM_CONNECT_AUDIT_DIR=/path/to/audit` to write per-call replay records,
|
||
then parse one without another provider call:
|
||
|
||
```bash
|
||
python -m llm_connect.replay /path/to/audit/record.json --json
|
||
```
|
||
|
||
## Server runtime profiles
|
||
|
||
Serve mode enables named runtime profiles by default. A client can send
|
||
`config.model_name="custodian-triage-balanced"` and the server resolves it to
|
||
the configured provider/model before calling the adapter.
|
||
|
||
Useful runtime environment variables:
|
||
|
||
```bash
|
||
LLM_CONNECT_HOST=0.0.0.0
|
||
LLM_CONNECT_PORT=8080
|
||
LLM_CONNECT_PROVIDER=openrouter
|
||
LLM_CONNECT_MODEL=google/gemini-2.5-flash
|
||
LLM_CONNECT_CUSTODIAN_TRIAGE_PROVIDER=openrouter
|
||
LLM_CONNECT_CUSTODIAN_TRIAGE_MODEL=google/gemini-2.5-flash
|
||
```
|
||
|
||
For local smoke tests without provider credentials:
|
||
|
||
```bash
|
||
export LLM_CONNECT_MOCK_RESPONSE="$(python -c 'import json; print(json.dumps(json.load(open("fixtures/activity_core/daily-triage-valid-content.json"))))')"
|
||
python -m llm_connect.server --provider mock
|
||
python scripts/smoke_activity_core_endpoint.py --url http://127.0.0.1:8080
|
||
```
|
||
|
||
Disable profile dispatch with `--disable-profiles`. Set
|
||
`LLM_CONNECT_STRICT_PROFILES=1` or pass `--strict-profiles` to reject direct
|
||
model names that are not configured profiles.
|
||
|
||
## Writing your own adapter
|
||
|
||
```python
|
||
from llm_connect import LLMAdapter, RunConfig, LLMResponse
|
||
|
||
class MyAdapter(LLMAdapter):
|
||
def execute_prompt(self, prompt: str, config: RunConfig) -> LLMResponse:
|
||
# call your API here
|
||
return LLMResponse(content="...", model="my-model")
|
||
|
||
def validate_config(self, config: RunConfig) -> bool:
|
||
return True
|
||
```
|
||
|
||
## TOML configuration chain
|
||
|
||
The `resolve_llm()` function walks a 7-level priority chain to pick a
|
||
provider and model. This is used by the `llm-helper` integration but is also
|
||
available standalone:
|
||
|
||
```python
|
||
from llm_connect.toml_config import resolve_llm
|
||
|
||
resolved = resolve_llm(app_name="myapp")
|
||
print(resolved.provider, resolved.model, resolved.provider_source)
|
||
```
|
||
|
||
Priority order (highest first):
|
||
|
||
1. CLI flags (`cli_provider`, `cli_model` arguments)
|
||
2. Env var `{APP_NAME}_HELPER_MODEL` (model only)
|
||
3. User preference — `~/.config/{app_name}/config.toml` `[llm.preference]`
|
||
4. Directory preference — `.{app_name}.toml` `[llm.preference]`
|
||
5. Directory default — `.{app_name}.toml` `[llm.default]`
|
||
6. User default — `~/.config/{app_name}/config.toml` `[llm.default]`
|
||
7. Hardcoded fallback — `gemini / gemini-2.5-flash`
|
||
|
||
Example config file (`~/.config/myapp/config.toml`):
|
||
|
||
```toml
|
||
[llm.default]
|
||
provider = "gemini"
|
||
model = "gemini-2.5-flash"
|
||
|
||
[llm.preference]
|
||
provider = "openrouter"
|
||
model = "anthropic/claude-sonnet-4"
|
||
```
|
||
|
||
## Embeddings
|
||
|
||
```python
|
||
from llm_connect import create_embedding_adapter, EmbeddingCache
|
||
|
||
adapter = create_embedding_adapter("openai", model="text-embedding-3-small")
|
||
cache = EmbeddingCache(cache_dir=".embeddings")
|
||
|
||
# Get embedding (cached after first call)
|
||
vec = cache.get_or_compute("my text", lambda t: adapter.embed([t])[0])
|
||
```
|
||
|
||
## Exceptions
|
||
|
||
```python
|
||
from llm_connect.exceptions import (
|
||
LLMError, # base
|
||
LLMConfigurationError,# bad key, unknown provider
|
||
LLMAPIError, # HTTP error from provider (has .status_code)
|
||
LLMRateLimitError, # 429
|
||
LLMTimeoutError, # request timed out
|
||
LLMSubprocessError, # claude CLI failed (has .return_code, .stderr)
|
||
)
|
||
```
|
||
|
||
## Testing
|
||
|
||
```python
|
||
from llm_connect import MockLLMAdapter, RunConfig
|
||
|
||
mock = MockLLMAdapter(mock_response="Test response")
|
||
config = RunConfig()
|
||
response = mock.execute_prompt("any prompt", config)
|
||
assert response.content == "Test response"
|
||
assert mock.call_count == 1
|
||
```
|
||
|
||
## Origin
|
||
|
||
Extracted from the [markitect](https://github.com/worsch/markitect) project.
|
||
The `markitect.llm` module remains a re-export shim pointing here.
|