T05: live cost budget replaces the dead token budget
The 8k/10k per-task token budget was never referenced or enforced and T08 blew past it silently. Replaced with a budget that can actually fire. The design constraint is the interesting part: per-task cost needs the commit that CLOSES the task, so a per-task budget is unavoidably retrospective -- it can only report a breach after the money is spent, which is the dead-policy failure again. What IS observable mid-task is spend since the last commit, because the transcript is append-live. So the budget binds on the open remainder. CB-01 budget = USD since the last commit, via `make cost-budget` CB-02 soft $10.00 (state progress, decide), hard $22.00 (stop) Calibrated on the 32 non-empty commit intervals of CB-WP-0001: p50 $1.40, p90 $9.36, max $10.80. Soft sits just below the observed maximum -- it would have fired exactly once on the calibration pass. Hard is ~2x the observed max, a value never reached in 32 intervals, so reaching it means the session is doing something the data has no example of. Both thresholds are set ABOVE every observed value, so they bind on future work rather than ratifying present work -- the distinction T07 is about. Stated limit: it is a command, not a daemon. An agent that never runs it gets no signal, which is the dead-policy failure one level up. Mitigated only by being free to run and on the one command surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d382fd4555
commit
06628e83e1
6 changed files with 122 additions and 4 deletions
6
Makefile
6
Makefile
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
CARGO := cargo
|
||||
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loop-lint self-tests loc all
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all
|
||||
|
||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||
check:
|
||||
|
|
@ -40,6 +40,10 @@ self-tests:
|
|||
python3 tools/rule-coverage.py --self-test
|
||||
python3 tools/dep-weight.py --self-test
|
||||
|
||||
# CB-01/CB-02: live spend since the last commit.
|
||||
cost-budget: cost-test
|
||||
python3 tools/cb-cost.py --budget
|
||||
|
||||
cost-pin: cost-test
|
||||
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
||||
|
||||
|
|
|
|||
|
|
@ -145,7 +145,7 @@ assertion against a fixture whose expected value is known and fails loudly;
|
|||
Per InnerLoop §Step 4, the acceptance table is checked against the
|
||||
contracts in this same spec:
|
||||
|
||||
- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated).
|
||||
- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated).
|
||||
Under a main-file-only collector the target is unreachable — this is the
|
||||
defect the adversarial review caught, where a target of $92.21 would have
|
||||
been hit *only* by a broken collector.
|
||||
|
|
@ -172,7 +172,63 @@ contracts in this same spec:
|
|||
necessarily shows a lossy projection. This is a limitation of the sink,
|
||||
not of the metric; `make cost` remains the authority.
|
||||
|
||||
## 7. Revisions to M-D2-CST
|
||||
## 7. The live cost budget
|
||||
|
||||
Replaces the 8k/10k per-task **token** budget, which was never referenced
|
||||
or enforced, which CB-WP-0001 T08 exceeded by orders of magnitude with no
|
||||
signal, and which InnerLoop v1.1 now marks dead. It implied a control that
|
||||
did not exist.
|
||||
|
||||
**The design constraint that shapes this.** Per-task cost needs the commit
|
||||
that *closes* the task (CA-08), so it is unavoidably retrospective — a
|
||||
per-task budget can only ever report a breach after the money is spent.
|
||||
What is observable mid-task is **spend since the last commit**, because
|
||||
the transcript is an append-live file (CA-07). That is the only number a
|
||||
budget can actually fire on, so that is what the budget binds.
|
||||
|
||||
> **CB-01.** The budget is the **open remainder**: USD spent since the
|
||||
> last commit, computable at any instant by `make cost-budget`.
|
||||
|
||||
> **CB-02.** Thresholds: **soft $10.00**, **hard $22.00**.
|
||||
>
|
||||
> - **soft** — state progress as a percentage and decide: continue, or
|
||||
> commit what works and decompose.
|
||||
> - **hard** — stop. Commit what works, or decompose. Uncommitted work is
|
||||
> also *unattributable* work, so a hard breach costs twice: once in
|
||||
> money, once in the 33% attribution gap.
|
||||
|
||||
Calibration, measured over the 32 non-empty commit intervals of
|
||||
CB-WP-0001 (`cb-cost --pin fc76445`):
|
||||
|
||||
| | spend per interval |
|
||||
|---|---|
|
||||
| min | $0.07 |
|
||||
| p50 | $1.40 |
|
||||
| p90 | $9.36 |
|
||||
| **max** | **$10.80** (T07, the Cargo workspace scaffold) |
|
||||
|
||||
Soft is set at **$10.00**, just below the observed maximum and just above
|
||||
p90: on the pass this is calibrated from it would have fired exactly once,
|
||||
on the single most expensive interval. Hard is set at **$22.00**, roughly
|
||||
2× the observed maximum — a value never reached in 32 intervals, so
|
||||
reaching it means the session is doing something the calibration data has
|
||||
no example of, which is precisely when stopping is right.
|
||||
|
||||
**Why these are not retargeted-after-the-fact.** T07's concern is a target
|
||||
moved to accommodate a measurement. These thresholds are set *above* every
|
||||
observed value, so they bind on future work rather than ratifying present
|
||||
work — no interval in the calibration set breaches hard, and one breaches
|
||||
soft. If a future pass routinely breaches, that is signal, not a reason to
|
||||
raise them.
|
||||
|
||||
**Known limit:** the budget cannot fire *between* turns without something
|
||||
running the command. It is available to an agent at any point, in CI, and
|
||||
in `make all`; it is not a daemon. An agent that never runs it gets no
|
||||
signal — which is exactly the dead-policy failure this replaces, one level
|
||||
up. The mitigation is that it costs nothing to run and is on the one
|
||||
command surface.
|
||||
|
||||
## 8. Revisions to M-D2-CST
|
||||
|
||||
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
|
||||
redefined from "tokens × pricepoint" — which named no instrument and was
|
||||
|
|
|
|||
|
|
@ -317,6 +317,16 @@ The loop exists to be driven by agents. Therefore:
|
|||
that does not exist. Replacement in USD is CB-WP-0003 T05; until then
|
||||
this is documentation of a gap, not a rule.
|
||||
|
||||
6a. **Live cost budget** *(replaces the above)* — spend since the last
|
||||
commit, soft **$10.00**, hard **$22.00**, checked by `make cost-budget`.
|
||||
Calibrated on the 32 commit intervals of CB-WP-0001 (p50 $1.40, p90
|
||||
$9.36, max $10.80), so both thresholds bind on future work rather than
|
||||
ratifying past work. Contract: [CostAccounting.md](CostAccounting.md)
|
||||
§7 (CB-01, CB-02). It fires on the open remainder rather than per task
|
||||
because per-task cost needs the commit that closes the task, and a
|
||||
budget that can only report after the money is spent is the dead policy
|
||||
this replaces.
|
||||
|
||||
**Enforcement status.** Rules above that a command can check are enforced
|
||||
by `make loop-lint`; the full classification of every InnerLoop rule as
|
||||
executable / checkable / decorative, with the failure class each catches,
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -456,6 +456,12 @@ def self_test():
|
|||
finally:
|
||||
os.unlink(partial)
|
||||
|
||||
# CB-02: thresholds must be ordered, or the budget silently never fires.
|
||||
ap_defaults = {"soft": 10.00, "hard": 22.00}
|
||||
check("CB-02 budget thresholds ordered and positive",
|
||||
0 < ap_defaults["soft"] < ap_defaults["hard"],
|
||||
f"soft ${ap_defaults['soft']:.2f} < hard ${ap_defaults['hard']:.2f}")
|
||||
|
||||
# AC-6: zero responses must not report $0.00 as an answer.
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||
fh.write(json.dumps({"type": "user", "message": {}}) + "\n")
|
||||
|
|
@ -481,6 +487,41 @@ def self_test():
|
|||
return 0 if ok else 1
|
||||
|
||||
|
||||
def budget(slug, soft, hard):
|
||||
"""Live cost budget (specs/CostAccounting.md §7).
|
||||
|
||||
The per-task figure needs the commit that closes the task, so it can
|
||||
only ever be retrospective. What IS observable mid-task is spend since
|
||||
the LAST commit — the open remainder — because the transcript is an
|
||||
append-live file. That is the number a budget can actually fire on.
|
||||
"""
|
||||
try:
|
||||
rep = collect(slug, None)
|
||||
except Abort as e:
|
||||
print(f"ABORT — {e}", file=sys.stderr)
|
||||
return 1
|
||||
open_spend = rep["by_task"].get(OPEN_REMAINDER, 0.0)
|
||||
head = subprocess.run(
|
||||
["git", "-C", REPO, "log", "-1", "--format=%h %s"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
||||
print("cost budget — spend since the last commit")
|
||||
print(f" last commit {head}")
|
||||
print(f" open spend ${open_spend:,.2f}")
|
||||
print(f" soft / hard ${soft:,.2f} / ${hard:,.2f}")
|
||||
if open_spend > hard:
|
||||
print(f"\n HARD BREACH — ${open_spend:,.2f} > ${hard:,.2f}. Commit what "
|
||||
f"works, or stop and decompose. Uncommitted work is also "
|
||||
f"unattributable.", file=sys.stderr)
|
||||
return 1
|
||||
if open_spend > soft:
|
||||
print(f"\n soft breach — ${open_spend:,.2f} > ${soft:,.2f}. State progress "
|
||||
f"as a percentage and decide: continue, or commit and decompose.")
|
||||
return 0
|
||||
print("\n within budget")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--slug", default="-home-worsch-clay-borg")
|
||||
|
|
@ -489,6 +530,10 @@ def main():
|
|||
ap.add_argument("--composition", action="store_true")
|
||||
ap.add_argument("--session-shape", action="store_true",
|
||||
help="SH-1..SH-3 (always shown in the default report)")
|
||||
ap.add_argument("--budget", action="store_true",
|
||||
help="CB-01/CB-02: spend since the last commit, live")
|
||||
ap.add_argument("--soft", type=float, default=10.00)
|
||||
ap.add_argument("--hard", type=float, default=22.00)
|
||||
ap.add_argument("--self-test", action="store_true")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
|
@ -496,6 +541,9 @@ def main():
|
|||
if args.self_test:
|
||||
return self_test()
|
||||
|
||||
if args.budget:
|
||||
return budget(args.slug, args.soft, args.hard)
|
||||
|
||||
try:
|
||||
rep = collect(args.slug, args.pin)
|
||||
except Abort as e:
|
||||
|
|
|
|||
|
|
@ -203,7 +203,7 @@ attribution.
|
|||
|
||||
```task
|
||||
id: CB-WP-0003-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue