Add --provider routing, --routing-config <yaml>, and --quality-floor
<float> to generate run, generate resume, and generate from-source.
The CLI flag wiring constructs a RoutingAssistedGenerationAdapter from
the parsed config, with the workspace handed in so any ledger_path in
the config resolves relative to it. --quality-floor overrides the
config-level default_quality_floor for a single invocation.
run_generation gains routing_config + quality_floor kwargs and
_adapter_for grew a "routing" branch. Missing --routing-config with
--provider routing fails fast with InfospaceError("missing_routing_config");
missing API key for any candidate fails fast with
InfospaceError("missing_routing_api_key").
Two small bug fixes surfaced while writing T03:
- routing._identify_adapter now also reads ``_model`` from llm-connect
adapters (their public attribute is private), so the per-stage
adapter-choice line shows the model id rather than just the class
name.
- budget.TOKEN_EVENTS_PATH corrected from /state/token-events to the
state-hub HTTP endpoint /token-events/ that actually exists; the
failure-isolation in emit_token_event already kept the prior typo
from breaking runs, but the hub never saw the events.
Five new tests cover: _adapter_for refusal on missing config,
_adapter_for happy path, run_generation end-to-end through routing
with a stubbed OpenRouterAdapter.execute_prompt (no network),
workspace-relative ledger resolution, and a CLI subprocess smoke
asserting fast-fail on missing API key.
173 tests pass, 1 skipped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
T03 — wrap_with_shadow_sampling() helper in routing.py: builds a
llm-connect ShadowingAdapter around any candidate LLMAdapter with a
caller-supplied baseline, grader, and QualityLedger. async_shadow=True
by default so production load is not doubled; on_shadow_error escape
hatch keeps caller logs informed when a baseline outage swallows the
shadow path. The returned adapter is still an LLMAdapter so it slots
into a RoutingPolicy rule without further code change.
T04 — generation report enrichment plus a small CLI helper:
- _collect_adapter_choices walks artifact provenance, groups by
(stage_id, adapter_id), and surfaces calls + prompt/completion tokens
per (stage, adapter) pair in a new ## Per-stage adapter choices
section. Runs that did not go through the bridge have no
provider_metadata.adapter_id and emit an empty list, so fixture-only
reports stay terse.
- summarise_quality_ledger() rolls a llm-connect QualityLedger up by
(task_type, adapter_id) with mean quality, mean cost, observations,
and cumulative tokens.
- infospace-bench routing ledger <path> CLI prints the rollup as JSON.
Five new tests cover shadow happy-path, shadow failure isolation,
ledger rollup, the routing CLI, and the report's adapter-choice
aggregation. Closes IB-WP-0018: T01-T05 are all done and the workplan
status flips from blocked to done now that LLM-WP-0004's primitives
have shipped.
144 tests pass, 1 skipped (the OpenRouter live smoke, gated as before).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
T01 — task-type taxonomy. docs/routing-task-types.md names the five
generation stages as the default identity-mapped task types
(summarize-source, extract-entities, extract-relations,
evaluate-entity, synthesize-report) and records the recommended quality
floors per stage. The taxonomy explicitly does not decide which adapter
ships per task type, where the ledger lives, or what a quality score
means — those stay with the caller per the LLM-WP-0004 scope guardrail.
T02 — RoutingAssistedGenerationAdapter bridge in
src/infospace_bench/routing.py. Wraps any llm-connect RoutingPolicy or
AdaptiveRoutingPolicy as an infospace-bench AssistedGenerationAdapter:
maps stage_id -> task_type (overridable), resolves an LLMAdapter,
delegates execute_prompt with a configurable RunConfig, and surfaces
the resolved adapter id, task type, model, usage, and finish_reason
back on AssistedGenerationResult.metadata. Provider tag stays
back-compatible with the strings already used in run records and the
budget rollup (openrouter / claude_code / openai / gemini / mock /
routing).
T05 — eight tests in tests/test_routing_adapter.py cover: static-policy
per-stage resolution, stage_to_task_type overrides, default-mapping
completeness, fall-through for unmapped stage ids, the adaptive path
selecting the cheaper qualifying adapter when a quality_floor is set,
adaptive policy falling back to static when no floor is set, response
metadata round-trip with provider tagging, and estimated_cost_per_1k
pass-through.
Adds llm-connect as a path dependency on pyproject.toml and to the
pytest pythonpath. Static OpenRouter and fixture paths are unchanged;
this commit only adds the option of routing.
139 tests pass, 1 skipped (the OpenRouter live smoke, gated as before).
T03 (shadow-mode integration) and T04 (CLI + per-stage chosen-adapter
in the generation report) follow next.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
