T05: live cost budget replaces the dead token budget

The 8k/10k per-task token budget was never referenced or enforced and
T08 blew past it silently. Replaced with a budget that can actually
fire.

The design constraint is the interesting part: per-task cost needs the
commit that CLOSES the task, so a per-task budget is unavoidably
retrospective -- it can only report a breach after the money is spent,
which is the dead-policy failure again. What IS observable mid-task is
spend since the last commit, because the transcript is append-live. So
the budget binds on the open remainder.

  CB-01  budget = USD since the last commit, via `make cost-budget`
  CB-02  soft $10.00 (state progress, decide), hard $22.00 (stop)

Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50
$1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum
-- it would have fired exactly once on the calibration pass. Hard is ~2x
the observed max, a value never reached in 32 intervals, so reaching it
means the session is doing something the data has no example of.

Both thresholds are set ABOVE every observed value, so they bind on
future work rather than ratifying present work -- the distinction T07
is about.

Stated limit: it is a command, not a daemon. An agent that never runs it
gets no signal, which is the dead-policy failure one level up. Mitigated
only by being free to run and on the one command surface.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:21:30 +02:00
parent d382fd4555
commit 06628e83e1
6 changed files with 122 additions and 4 deletions

View file

@ -3,7 +3,7 @@
CARGO := cargo
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loop-lint self-tests loc all
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all
## fmt + clippy (deny warnings) + HashMap deny-lint
check:
@ -40,6 +40,10 @@ self-tests:
python3 tools/rule-coverage.py --self-test
python3 tools/dep-weight.py --self-test
# CB-01/CB-02: live spend since the last commit.
cost-budget: cost-test
python3 tools/cb-cost.py --budget
cost-pin: cost-test
python3 tools/cb-cost.py --pin fc76445 --composition --by-task

View file

@ -145,7 +145,7 @@ assertion against a fixture whose expected value is known and fails loudly;
Per InnerLoop §Step 4, the acceptance table is checked against the
contracts in this same spec:
- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated).
- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated).
Under a main-file-only collector the target is unreachable — this is the
defect the adversarial review caught, where a target of $92.21 would have
been hit *only* by a broken collector.
@ -172,7 +172,63 @@ contracts in this same spec:
necessarily shows a lossy projection. This is a limitation of the sink,
not of the metric; `make cost` remains the authority.
## 7. Revisions to M-D2-CST
## 7. The live cost budget
Replaces the 8k/10k per-task **token** budget, which was never referenced
or enforced, which CB-WP-0001 T08 exceeded by orders of magnitude with no
signal, and which InnerLoop v1.1 now marks dead. It implied a control that
did not exist.
**The design constraint that shapes this.** Per-task cost needs the commit
that *closes* the task (CA-08), so it is unavoidably retrospective — a
per-task budget can only ever report a breach after the money is spent.
What is observable mid-task is **spend since the last commit**, because
the transcript is an append-live file (CA-07). That is the only number a
budget can actually fire on, so that is what the budget binds.
> **CB-01.** The budget is the **open remainder**: USD spent since the
> last commit, computable at any instant by `make cost-budget`.
> **CB-02.** Thresholds: **soft $10.00**, **hard $22.00**.
>
> - **soft** — state progress as a percentage and decide: continue, or
> commit what works and decompose.
> - **hard** — stop. Commit what works, or decompose. Uncommitted work is
> also *unattributable* work, so a hard breach costs twice: once in
> money, once in the 33% attribution gap.
Calibration, measured over the 32 non-empty commit intervals of
CB-WP-0001 (`cb-cost --pin fc76445`):
| | spend per interval |
|---|---|
| min | $0.07 |
| p50 | $1.40 |
| p90 | $9.36 |
| **max** | **$10.80** (T07, the Cargo workspace scaffold) |
Soft is set at **$10.00**, just below the observed maximum and just above
p90: on the pass this is calibrated from it would have fired exactly once,
on the single most expensive interval. Hard is set at **$22.00**, roughly
2× the observed maximum — a value never reached in 32 intervals, so
reaching it means the session is doing something the calibration data has
no example of, which is precisely when stopping is right.
**Why these are not retargeted-after-the-fact.** T07's concern is a target
moved to accommodate a measurement. These thresholds are set *above* every
observed value, so they bind on future work rather than ratifying present
work — no interval in the calibration set breaches hard, and one breaches
soft. If a future pass routinely breaches, that is signal, not a reason to
raise them.
**Known limit:** the budget cannot fire *between* turns without something
running the command. It is available to an agent at any point, in CI, and
in `make all`; it is not a daemon. An agent that never runs it gets no
signal — which is exactly the dead-policy failure this replaces, one level
up. The mitigation is that it costs nothing to run and is on the one
command surface.
## 8. Revisions to M-D2-CST
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
redefined from "tokens × pricepoint" — which named no instrument and was

View file

@ -317,6 +317,16 @@ The loop exists to be driven by agents. Therefore:
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
this is documentation of a gap, not a rule.
6a. **Live cost budget** *(replaces the above)* — spend since the last
commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`.
Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90
$9.36, max $10.80), so both thresholds bind on future work rather than
ratifying past work. Contract: [CostAccounting.md](CostAccounting.md)
§7 (CB-01, CB-02). It fires on the open remainder rather than per task
because per-task cost needs the commit that closes the task, and a
budget that can only report after the money is spent is the dead policy
this replaces.
**Enforcement status.** Rules above that a command can check are enforced
by `make loop-lint`; the full classification of every InnerLoop rule as
executable / checkable / decorative, with the failure class each catches,

View file

@ -456,6 +456,12 @@ def self_test():
finally:
os.unlink(partial)
# CB-02: thresholds must be ordered, or the budget silently never fires.
ap_defaults = {"soft": 10.00, "hard": 22.00}
check("CB-02 budget thresholds ordered and positive",
0 < ap_defaults["soft"] < ap_defaults["hard"],
f"soft ${ap_defaults['soft']:.2f} < hard ${ap_defaults['hard']:.2f}")
# AC-6: zero responses must not report $0.00 as an answer.
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
fh.write(json.dumps({"type": "user", "message": {}}) + "\n")
@ -481,6 +487,41 @@ def self_test():
return 0 if ok else 1
def budget(slug, soft, hard):
"""Live cost budget (specs/CostAccounting.md §7).
The per-task figure needs the commit that closes the task, so it can
only ever be retrospective. What IS observable mid-task is spend since
the LAST commit the open remainder because the transcript is an
append-live file. That is the number a budget can actually fire on.
"""
try:
rep = collect(slug, None)
except Abort as e:
print(f"ABORT — {e}", file=sys.stderr)
return 1
open_spend = rep["by_task"].get(OPEN_REMAINDER, 0.0)
head = subprocess.run(
["git", "-C", REPO, "log", "-1", "--format=%h %s"],
capture_output=True, text=True, check=True).stdout.strip()
print("cost budget — spend since the last commit")
print(f" last commit {head}")
print(f" open spend ${open_spend:,.2f}")
print(f" soft / hard ${soft:,.2f} / ${hard:,.2f}")
if open_spend > hard:
print(f"\n HARD BREACH — ${open_spend:,.2f} > ${hard:,.2f}. Commit what "
f"works, or stop and decompose. Uncommitted work is also "
f"unattributable.", file=sys.stderr)
return 1
if open_spend > soft:
print(f"\n soft breach — ${open_spend:,.2f} > ${soft:,.2f}. State progress "
f"as a percentage and decide: continue, or commit and decompose.")
return 0
print("\n within budget")
return 0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--slug", default="-home-worsch-clay-borg")
@ -489,6 +530,10 @@ def main():
ap.add_argument("--composition", action="store_true")
ap.add_argument("--session-shape", action="store_true",
help="SH-1..SH-3 (always shown in the default report)")
ap.add_argument("--budget", action="store_true",
help="CB-01/CB-02: spend since the last commit, live")
ap.add_argument("--soft", type=float, default=10.00)
ap.add_argument("--hard", type=float, default=22.00)
ap.add_argument("--self-test", action="store_true")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
@ -496,6 +541,9 @@ def main():
if args.self_test:
return self_test()
if args.budget:
return budget(args.slug, args.soft, args.hard)
try:
rep = collect(args.slug, args.pin)
except Abort as e:

View file

@ -203,7 +203,7 @@ attribution.
```task
id: CB-WP-0003-T05
status: todo
status: done
priority: medium
state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09"
```