diff --git a/Makefile b/Makefile index ce08512..02f4e61 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loop-lint self-tests loc all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -40,6 +40,10 @@ self-tests: python3 tools/rule-coverage.py --self-test python3 tools/dep-weight.py --self-test +# CB-01/CB-02: live spend since the last commit. +cost-budget: cost-test + python3 tools/cb-cost.py --budget + cost-pin: cost-test python3 tools/cb-cost.py --pin fc76445 --composition --by-task diff --git a/specs/CostAccounting.md b/specs/CostAccounting.md index c22530c..7fa8ee3 100644 --- a/specs/CostAccounting.md +++ b/specs/CostAccounting.md @@ -145,7 +145,7 @@ assertion against a fixture whose expected value is known and fails loudly; Per InnerLoop §Step 4, the acceptance table is checked against the contracts in this same spec: -- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated). +- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated). Under a main-file-only collector the target is unreachable — this is the defect the adversarial review caught, where a target of $92.21 would have been hit *only* by a broken collector. @@ -172,7 +172,63 @@ contracts in this same spec: necessarily shows a lossy projection. This is a limitation of the sink, not of the metric; `make cost` remains the authority. -## 7. Revisions to M-D2-CST +## 7. The live cost budget + +Replaces the 8k/10k per-task **token** budget, which was never referenced +or enforced, which CB-WP-0001 T08 exceeded by orders of magnitude with no +signal, and which InnerLoop v1.1 now marks dead. It implied a control that +did not exist. + +**The design constraint that shapes this.** Per-task cost needs the commit +that *closes* the task (CA-08), so it is unavoidably retrospective — a +per-task budget can only ever report a breach after the money is spent. +What is observable mid-task is **spend since the last commit**, because +the transcript is an append-live file (CA-07). That is the only number a +budget can actually fire on, so that is what the budget binds. + +> **CB-01.** The budget is the **open remainder**: USD spent since the +> last commit, computable at any instant by `make cost-budget`. + +> **CB-02.** Thresholds: **soft $10.00**, **hard $22.00**. +> +> - **soft** — state progress as a percentage and decide: continue, or +> commit what works and decompose. +> - **hard** — stop. Commit what works, or decompose. Uncommitted work is +> also *unattributable* work, so a hard breach costs twice: once in +> money, once in the 33% attribution gap. + +Calibration, measured over the 32 non-empty commit intervals of +CB-WP-0001 (`cb-cost --pin fc76445`): + +| | spend per interval | +|---|---| +| min | $0.07 | +| p50 | $1.40 | +| p90 | $9.36 | +| **max** | **$10.80** (T07, the Cargo workspace scaffold) | + +Soft is set at **$10.00**, just below the observed maximum and just above +p90: on the pass this is calibrated from it would have fired exactly once, +on the single most expensive interval. Hard is set at **$22.00**, roughly +2× the observed maximum — a value never reached in 32 intervals, so +reaching it means the session is doing something the calibration data has +no example of, which is precisely when stopping is right. + +**Why these are not retargeted-after-the-fact.** T07's concern is a target +moved to accommodate a measurement. These thresholds are set *above* every +observed value, so they bind on future work rather than ratifying present +work — no interval in the calibration set breaches hard, and one breaches +soft. If a future pass routinely breaches, that is signal, not a reason to +raise them. + +**Known limit:** the budget cannot fire *between* turns without something +running the command. It is available to an agent at any point, in CI, and +in `make all`; it is not a daemon. An agent that never runs it gets no +signal — which is exactly the dead-policy failure this replaces, one level +up. The mitigation is that it costs nothing to run and is on the one +command surface. + +## 8. Revisions to M-D2-CST `specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is redefined from "tokens × pricepoint" — which named no instrument and was diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md index 4ae75f5..2f28a68 100644 --- a/specs/InnerLoop.md +++ b/specs/InnerLoop.md @@ -317,6 +317,16 @@ The loop exists to be driven by agents. Therefore: that does not exist. Replacement in USD is CB-WP-0003 T05; until then this is documentation of a gap, not a rule. +6a. **Live cost budget** *(replaces the above)* — spend since the last + commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`. + Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90 + $9.36, max $10.80), so both thresholds bind on future work rather than + ratifying past work. Contract: [CostAccounting.md](CostAccounting.md) + §7 (CB-01, CB-02). It fires on the open remainder rather than per task + because per-task cost needs the commit that closes the task, and a + budget that can only report after the money is spent is the dead policy + this replaces. + **Enforcement status.** Rules above that a command can check are enforced by `make loop-lint`; the full classification of every InnerLoop rule as executable / checkable / decorative, with the failure class each catches, diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index 12efe0a..8ad4a54 100644 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py index f76a6b0..59ced0e 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -456,6 +456,12 @@ def self_test(): finally: os.unlink(partial) + # CB-02: thresholds must be ordered, or the budget silently never fires. + ap_defaults = {"soft": 10.00, "hard": 22.00} + check("CB-02 budget thresholds ordered and positive", + 0 < ap_defaults["soft"] < ap_defaults["hard"], + f"soft ${ap_defaults['soft']:.2f} < hard ${ap_defaults['hard']:.2f}") + # AC-6: zero responses must not report $0.00 as an answer. with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: fh.write(json.dumps({"type": "user", "message": {}}) + "\n") @@ -481,6 +487,41 @@ def self_test(): return 0 if ok else 1 +def budget(slug, soft, hard): + """Live cost budget (specs/CostAccounting.md §7). + + The per-task figure needs the commit that closes the task, so it can + only ever be retrospective. What IS observable mid-task is spend since + the LAST commit — the open remainder — because the transcript is an + append-live file. That is the number a budget can actually fire on. + """ + try: + rep = collect(slug, None) + except Abort as e: + print(f"ABORT — {e}", file=sys.stderr) + return 1 + open_spend = rep["by_task"].get(OPEN_REMAINDER, 0.0) + head = subprocess.run( + ["git", "-C", REPO, "log", "-1", "--format=%h %s"], + capture_output=True, text=True, check=True).stdout.strip() + + print("cost budget — spend since the last commit") + print(f" last commit {head}") + print(f" open spend ${open_spend:,.2f}") + print(f" soft / hard ${soft:,.2f} / ${hard:,.2f}") + if open_spend > hard: + print(f"\n HARD BREACH — ${open_spend:,.2f} > ${hard:,.2f}. Commit what " + f"works, or stop and decompose. Uncommitted work is also " + f"unattributable.", file=sys.stderr) + return 1 + if open_spend > soft: + print(f"\n soft breach — ${open_spend:,.2f} > ${soft:,.2f}. State progress " + f"as a percentage and decide: continue, or commit and decompose.") + return 0 + print("\n within budget") + return 0 + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--slug", default="-home-worsch-clay-borg") @@ -489,6 +530,10 @@ def main(): ap.add_argument("--composition", action="store_true") ap.add_argument("--session-shape", action="store_true", help="SH-1..SH-3 (always shown in the default report)") + ap.add_argument("--budget", action="store_true", + help="CB-01/CB-02: spend since the last commit, live") + ap.add_argument("--soft", type=float, default=10.00) + ap.add_argument("--hard", type=float, default=22.00) ap.add_argument("--self-test", action="store_true") ap.add_argument("--json", action="store_true") args = ap.parse_args() @@ -496,6 +541,9 @@ def main(): if args.self_test: return self_test() + if args.budget: + return budget(args.slug, args.soft, args.hard) + try: rep = collect(args.slug, args.pin) except Abort as e: diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md index 819c0dc..682a48c 100644 --- a/workplans/CB-WP-0003-loop-hardening.md +++ b/workplans/CB-WP-0003-loop-hardening.md @@ -203,7 +203,7 @@ attribution. ```task id: CB-WP-0003-T05 -status: todo +status: done priority: medium state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09" ```