infospace-bench/tests
tegwick d4c9c56f5c IB-WP-0019-T04: plan-vs-actual variance and surfacing
After every generate run, compute variance between the executing plan
snapshot and the just-recorded usage rollup, persist it to
output/budget/summary.yaml (overwrite-on-run), and surface it both in
the generate status JSON (new budget_summary field) and as a "Plan
variance" line in reports/generation-summary.md.

Variance fields: calls / prompt_tokens / total_tokens each carry
{estimated, actual, delta, ratio}; cost_usd carries {estimated,
actual_known, actual_estimated_from_rates, actual_total, delta, ratio};
per_workflow rolls the per-bucket usage up to the same workflow_id grain
the plan reports. Runs whose snapshot_id cannot be resolved (no prior
plan, or pruned from the retention window) still record a variance row
with null comparison fields and snapshot_resolved=false, so the
consumer always sees a current summary.

Reordered run_generation so usage and variance are written before the
generation report, allowing the report to embed the variance line on
the same pass.

110 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-17 20:06:19 +02:00
..
test_agentic_memory_profile.py Agentic memory profile 2026-05-15 16:01:35 +02:00
test_archive.py archive: include contracts/, schemas/; report skipped top-level dirs 2026-05-17 12:21:19 +02:00
test_budget_registry.py IB-WP-0019-T04: plan-vs-actual variance and surfacing 2026-05-17 20:06:19 +02:00
test_cli.py IB-WP-0014: archive-list, restore, retention annotation, docs (T03-T05) 2026-05-17 11:46:23 +02:00
test_engine_boundary.py engine and lifecycle 2026-05-14 16:26:42 +02:00
test_epub3_intake.py IB-WP-0016-T02: chapter-aware chunking and stable IDs 2026-05-17 15:52:47 +02:00
test_evaluation.py Initial implementation 2026-05-14 11:32:25 +02:00
test_evaluation_history.py eval history and metrics 2026-05-14 15:35:04 +02:00
test_generic_generator.py generic source-to-infospace generator 2026-05-14 19:33:22 +02:00
test_inspection.py Initial implementation 2026-05-14 11:32:25 +02:00
test_legacy_pilot.py Kontextual Engine Integration Boundary 2026-05-14 16:43:29 +02:00
test_lifecycle.py Initial implementation 2026-05-14 11:32:25 +02:00
test_markdown_adapter.py markitect-tool integration 2026-05-14 14:53:16 +02:00
test_plan_scale.py IB-WP-0016-T03: scale-aware planning 2026-05-17 18:18:09 +02:00
test_reference_pilot.py Initial implementation 2026-05-14 11:32:25 +02:00
test_replacement_readiness.py command parity and migration guide 2026-05-14 17:16:39 +02:00
test_semantics.py entity relationship model 2026-05-14 15:06:17 +02:00
test_trading_literature_profile.py IB-WP-0016-T04: trading-literature profile 2026-05-17 18:59:45 +02:00
test_wealth_vsm_generation.py infospace pipeline for wealth of nations example 2026-05-14 18:04:38 +02:00
test_workflow.py acceptance matrix and workflow generation 2026-05-14 16:01:32 +02:00