Compare commits
4 commits
f4dccb6f99
...
06483bc587
| Author | SHA1 | Date | |
|---|---|---|---|
| 06483bc587 | |||
| 2b6bf29e8f | |||
| cd2dc5380a | |||
| 38f237fc5c |
10 changed files with 887 additions and 97 deletions
6
Makefile
6
Makefile
|
|
@ -24,7 +24,7 @@ TOOLS := $(REPO)/tools
|
|||
# Every cargo recipe runs at the repo root; the shell does not persist cd.
|
||||
IN_REPO := cd $(REPO) &&
|
||||
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play all
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget shape-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check size-metrics runtime-metrics build-time am6 replay-test loc play gate-review all
|
||||
|
||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||
check:
|
||||
|
|
@ -36,6 +36,10 @@ check:
|
|||
play:
|
||||
$(IN_REPO) $(CARGO) run -q -p cb-play -- $(ARGS)
|
||||
|
||||
## which control gates are due for a keep-or-kill argument (ADR-0006 D3)
|
||||
gate-review:
|
||||
$(PY) $(TOOLS)/gate-review.py
|
||||
|
||||
## unit + scenario-format tests
|
||||
test:
|
||||
$(IN_REPO) $(CARGO) test --workspace
|
||||
|
|
|
|||
134
evidence/CB-EV-0008-adaptive-gates.md
Normal file
134
evidence/CB-EV-0008-adaptive-gates.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# CB-EV-0008 — did the gate changes fire?
|
||||
|
||||
CB-WP-0009 T04. Measured 2026-08-01 at `HEAD` after T03. This pass is
|
||||
`kind: meta`, opened above the budget line on recorded maintainer
|
||||
authorization (ADR-0006 D2's first use).
|
||||
|
||||
---
|
||||
|
||||
## 1. Does the windowed meta share differ from cumulative?
|
||||
|
||||
**Yes at one pass, no at three — and the prediction did not say which.**
|
||||
|
||||
| reading | value |
|
||||
|---|---|
|
||||
| CB-WP-0008 alone (product) | **0%** |
|
||||
| trailing 3 passes, before this one | 36% |
|
||||
| trailing 3 passes, now | **45%** |
|
||||
| lifetime, all 9 passes | 50% |
|
||||
|
||||
T01 predicted the windowed figure would differ sharply from cumulative,
|
||||
refuted within 20%. Against the **single-pass** window that is decisive:
|
||||
0% versus 50%. Against the **trailing-3** window it is not: 45% versus
|
||||
50% is a 10% relative gap, inside the refutation band.
|
||||
|
||||
Both are true, and the honest reading is that **the prediction was
|
||||
underspecified** — it was written before the window size was chosen, and
|
||||
T01 then chose three passes for a stated reason (one pass is a coin flip:
|
||||
a meta pass reads 100%, a product pass 0%). So:
|
||||
|
||||
- D1 is **confirmed for the failure it was written against**: the v1.5
|
||||
budget could not see a 100%-product pass, and now it can.
|
||||
- D1 is **not** confirmed as "trailing-3 always differs from lifetime."
|
||||
Right now it does not, because the recent mix happens to resemble the
|
||||
lifetime mix. That is a fact about this project's history, not evidence
|
||||
for or against the machinery.
|
||||
|
||||
**Carried as an open question**, not resolved by argument: whether three
|
||||
is the right window. It should be revisited when a pass makes the two
|
||||
diverge, which is the only cheap evidence available.
|
||||
|
||||
## 2. What does `gate-review` say on its first run?
|
||||
|
||||
```
|
||||
9 gate(s) — 0 due, 2 silent, 0 drifted
|
||||
```
|
||||
|
||||
The two silent entries are the uncomfortable ones, and they are supposed
|
||||
to be:
|
||||
|
||||
- **CHAOS** — the chaos roll has never changed a tier. Its 12-declaration
|
||||
window (4 used) exists precisely to find out whether it ever will.
|
||||
- **GATE-REVIEW** — this registry itself, which has caught nothing
|
||||
because it is one hour old, and which retires on 2026-12-31 if it has
|
||||
not by then retired, tightened or forced the re-justification of
|
||||
anything.
|
||||
|
||||
A registry where every gate looked current and productive would have been
|
||||
a registry written to look good. Two of nine are on notice.
|
||||
|
||||
The `caught` lists are the load-bearing part, and they are honest in the
|
||||
other direction too: **M-D1-MUT has six entries, LOOP-LINT two, and the
|
||||
DFD gate one that landed this week** — `gr_scenarios` stale at 21 after
|
||||
CB-WP-0008 T03 added three scenarios, caught by `make all` going red
|
||||
before the push.
|
||||
|
||||
## 3. Did D4 hold for this pass?
|
||||
|
||||
**Three of four tasks shipped a command; the fourth is this file.**
|
||||
|
||||
| task | shipped |
|
||||
|---|---|
|
||||
| T01 | `cb-cost.pass_costs`, windowed `make status` |
|
||||
| T02 | `gates.toml`, `make gate-review`, a `loop-lint` check |
|
||||
| T03 | InnerLoop v1.6 — **prose** |
|
||||
| T04 | this evidence file — **prose** |
|
||||
|
||||
D4 says a `meta` pass must ship *at least one* executable check, so the
|
||||
rule holds. But the pass makes the boundary clear: **the spec change and
|
||||
the evidence file are prose, and a rule requiring every task to cash out
|
||||
would be wrong** — the spec edit is what makes the commands normative,
|
||||
and the evidence file is what checks them.
|
||||
|
||||
So D4 stands as written (per pass, not per task), and this pass is on
|
||||
record as its first exercise.
|
||||
|
||||
## 4. Cost
|
||||
|
||||
| pass | kind | responses | cost | $/response |
|
||||
|---|---|---|---|---|
|
||||
| CB-WP-0006 | meta | 158 | $57.22 | 0.362 |
|
||||
| CB-WP-0007 | meta | 21 | $7.95 | 0.298* |
|
||||
| CB-WP-0008 | product | 134 | $17.38 | **0.123** |
|
||||
| **CB-WP-0009** | **meta** | **38** | **$6.73** | **0.177** |
|
||||
|
||||
\* the CB-WP-0007 figure is from the earlier hand-windowed measurement;
|
||||
the table's per-pass windows are now computed by `pass_costs`.
|
||||
|
||||
**This is the interesting number.** D4's claim is that loop work that
|
||||
cashes out into a command is cheap. This pass cashed out three commands
|
||||
and ran at **0.177** — cheaper than every previous meta pass (0.228,
|
||||
0.298, 0.362) and still **1.4× the product pass**. Partial support, not
|
||||
vindication, and stated as such.
|
||||
|
||||
Session shape over the same window is worse than the product pass and
|
||||
breaches both targets:
|
||||
|
||||
| metric | this pass | CB-WP-0008 | target |
|
||||
|---|---|---|---|
|
||||
| SH-1 mean context | **290,171** | 161,994 | ≤ 200,000 |
|
||||
| SH-2 p90 context | **311,243** | 215,664 | ≤ 300,000 |
|
||||
| SH-3 batching | 0.0% | 6.9% | ≥ 20% |
|
||||
|
||||
The mechanical cause is visible and boring: this pass ran on top of an
|
||||
already-long session with no compaction, where CB-WP-0008 started
|
||||
immediately after one. The shape budget reports it; nothing gates it; and
|
||||
per CB-RES-0005 §4 nothing should.
|
||||
|
||||
**Meta share after this pass: 45% of the trailing three, against a soft
|
||||
25%.** Reported under the rule this pass wrote. Nothing was displaced —
|
||||
CB-WP-0008 shipped stage 0 first, and this work was maintainer-authorized
|
||||
on top of it — but the number is over the line and the next pass should
|
||||
be product to bring it back.
|
||||
|
||||
## 5. Open
|
||||
|
||||
- **Is 3 the right trailing window?** Unanswered; revisit when a pass
|
||||
makes single-pass and trailing-3 diverge.
|
||||
- **CHAOS and GATE-REVIEW have caught nothing.** Both have dates.
|
||||
- **`mech environment setup` reads 17 turns / $2.90** this pass, which is
|
||||
the cost classifier scoring inline `python3 - <<EOF` edits as
|
||||
environment setup. The category is mislabelled rather than the work
|
||||
being wrong — a small classifier defect, recorded, not chased.
|
||||
- **The next pass should be `product`.** The budget says so, and this is
|
||||
the first time it has said so about work that had not started yet.
|
||||
134
gates.toml
Normal file
134
gates.toml
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# The gate registry (ADR-0006 D3, CB-WP-0009 T02).
|
||||
#
|
||||
# A gate without an expiry is a permanent tax justified once. Every
|
||||
# standing control mechanism gets an entry here saying what it checks,
|
||||
# what it has actually **caught**, when its keep-or-kill argument is due,
|
||||
# and what would retire it.
|
||||
#
|
||||
# `make gate-review` reports what is overdue and what has caught nothing.
|
||||
# It reports; it does not fail the build — CB-RES-0005 §4: a gate that
|
||||
# blocks the remedy when the metric breaches is a trap, not a gate.
|
||||
#
|
||||
# `caught` is the load-bearing field. An empty `caught` is not proof a
|
||||
# gate is useless — it may be preventing rather than missing — but it
|
||||
# means the argument has to be made out loud on `review_by`.
|
||||
|
||||
# Targets in `make all` that are build or acceptance steps rather than
|
||||
# *control* gates — they measure the product, not how we work. Listed
|
||||
# explicitly so a new target has to be classified rather than ignored;
|
||||
# `loop-lint` fails when a target is in neither list.
|
||||
not_control_gates = [
|
||||
"check", "test", "sim", "bench-test", "size-metrics", "runtime-metrics",
|
||||
"am6", "replay-test", "dep-weight", "self-tests", "env-test",
|
||||
]
|
||||
|
||||
[[gate]]
|
||||
id = "CB-01/CB-02"
|
||||
name = "cost budget"
|
||||
target = "cost-budget"
|
||||
checks = "spend since the last commit; soft $10, hard $22"
|
||||
added = "2026-07-30"
|
||||
review_by = "2026-11-30"
|
||||
caught = [
|
||||
"CB-WP-0005: hard breach forced the Phase C re-plan",
|
||||
"CB-WP-0006 T07: $12.06 in one task, the pass's most expensive",
|
||||
]
|
||||
retire_if = "two consecutive passes never approach the soft line, or commits get small enough that the window is always trivial"
|
||||
|
||||
[[gate]]
|
||||
id = "SH-1/SH-2/SH-3"
|
||||
name = "session-shape budget"
|
||||
target = "shape-budget"
|
||||
checks = "mean and p90 context, and batching rate, since the last commit"
|
||||
added = "2026-08-01"
|
||||
review_by = "2026-11-30"
|
||||
caught = [
|
||||
"first run fired HARD at 656,574 against a 300,000 ceiling, which is what prompted the compaction before CB-WP-0008",
|
||||
]
|
||||
retire_if = "context stops correlating with cost, or the model's context handling makes the number unactionable"
|
||||
|
||||
[[gate]]
|
||||
id = "M-D1-MUT"
|
||||
name = "mutation coverage of acceptance rows"
|
||||
target = "mutation-check"
|
||||
checks = "each acceptance row's assertion must go red for a stated reason when mutated"
|
||||
added = "2026-07-31"
|
||||
review_by = "2026-12-31"
|
||||
caught = [
|
||||
"AM-6 measuring contention, not throughput",
|
||||
"AM-5's 61% measurement error under load",
|
||||
"peak RSS over-reported 3x",
|
||||
"K10's first round trip not reproducing",
|
||||
"the bench workload existing twice",
|
||||
"AM-2's expect matching its own passing output (EXPECT-VACUOUS)",
|
||||
]
|
||||
retire_if = "a full pass adds rows without finding anything, twice running — the harness costs real money per run"
|
||||
|
||||
[[gate]]
|
||||
id = "DFD"
|
||||
name = "single source of fact"
|
||||
target = "facts-check"
|
||||
checks = "every tagged number in the docs matches the tool that measures it"
|
||||
added = "2026-07-31"
|
||||
review_by = "2026-12-31"
|
||||
caught = [
|
||||
"gr_scenarios stale at 21 after CB-WP-0008 T03 added three scenarios",
|
||||
]
|
||||
retire_if = "the untagged-literal count reaches zero and stays there, meaning the docs stopped restating measured numbers"
|
||||
|
||||
[[gate]]
|
||||
id = "AM-1b"
|
||||
name = "kernel spec->code link"
|
||||
target = "coverage"
|
||||
checks = "every numbered K-rule is named in the source; binds 2026-08-31"
|
||||
added = "2026-07-31"
|
||||
review_by = "2026-08-31"
|
||||
caught = [
|
||||
"3 unlinked K-rules at introduction (15/18); 18/18 today",
|
||||
]
|
||||
retire_if = "it stays at 100% through two passes that add kernel rules — at that point it is measuring a habit, not enforcing one"
|
||||
|
||||
[[gate]]
|
||||
id = "META-25"
|
||||
name = "meta budget"
|
||||
target = "status"
|
||||
checks = "share of the trailing 3 passes spent on the loop itself; soft 25%"
|
||||
added = "2026-08-01"
|
||||
review_by = "2026-11-30"
|
||||
caught = [
|
||||
"its own cumulative-window defect, reported in CB-EV-0007 §3 and fixed by CB-WP-0009 T01",
|
||||
]
|
||||
retire_if = "product and meta stop being separable, or the share sits under the line for four passes without anyone consulting it"
|
||||
|
||||
[[gate]]
|
||||
id = "LOOP-LINT"
|
||||
name = "executable InnerLoop rules"
|
||||
target = "loop-lint"
|
||||
checks = "loadability, unmeasured verdicts, tier and chaos declarations, review trails, self-test entry points, and this registry"
|
||||
added = "2026-07-30"
|
||||
review_by = "2026-12-31"
|
||||
caught = [
|
||||
"four loadability breaches (401, 427, 406, 409 lines), each fixed structurally rather than by raising the limit",
|
||||
"a reporting tool with no --self-test entry point (tools/repo.py)",
|
||||
]
|
||||
retire_if = "two passes run with no finding while artifacts keep growing — that would mean it is measuring the wrong properties"
|
||||
|
||||
[[gate]]
|
||||
id = "CHAOS"
|
||||
name = "the chaos roll"
|
||||
target = ""
|
||||
checks = "d4 on each tier declaration, 12-declaration calibration window"
|
||||
added = "2026-07-30"
|
||||
review_by = "2026-09-30"
|
||||
caught = []
|
||||
retire_if = "the window closes with no overridden tier producing a different outcome than the argued one — the evaluation this window exists to make possible"
|
||||
|
||||
[[gate]]
|
||||
id = "GATE-REVIEW"
|
||||
name = "this registry"
|
||||
target = "gate-review"
|
||||
checks = "gates past their review date, and gates that have caught nothing"
|
||||
added = "2026-08-01"
|
||||
review_by = "2026-12-31"
|
||||
caught = []
|
||||
retire_if = "it has retired, tightened, or forced the re-justification of nothing by its review date — then it is a ritual, and ADR-0006 D4 says rituals cash out or go"
|
||||
|
|
@ -1,29 +1,53 @@
|
|||
# The Inner Loop — Assimilate and Surpass
|
||||
|
||||
Status: **v1.5** — corrected from CB-WP-0007 (session shape) on
|
||||
2026-08-01. Change from v1.4: a **soft 25% meta budget** (below), after a
|
||||
spend review found the project 38% product / 62% loop-meta with cost per
|
||||
response degraded 2.9x from its best window.
|
||||
Status: **v1.6** — corrected from CB-WP-0008 (stage 0) and ADR-0006 on
|
||||
2026-08-01. Changes from v1.5: budgets measure the window they govern and
|
||||
a breach never blocks its own repair (D1/D2); loop work cashes out into
|
||||
something runnable in the pass that invents it (D4); the tier table
|
||||
covers changes to the loop's own operation (D5).
|
||||
|
||||
> **Meta budget — soft, 25% of spend per pass.** Work on the loop's own
|
||||
> instruments and process is capped at a quarter of a pass. Measured by
|
||||
> `make status` from each workplan's `kind:` frontmatter
|
||||
> (`product` | `meta` | `mixed`).
|
||||
> **Meta budget — soft, 25% of the trailing three passes.** Work on the
|
||||
> loop's own instruments and process is capped at a quarter of recent
|
||||
> spend. Measured by `make status` from each workplan's `kind:`
|
||||
> frontmatter (`product` | `meta` | `mixed`), over a **trailing 3-pass
|
||||
> window**; the lifetime figure is reported as history and is not the
|
||||
> metric (ADR-0006 D1).
|
||||
>
|
||||
> **Soft on purpose.** A task already started may be finished — stopping
|
||||
> mid-task to satisfy a ratio wastes the work and leaves the tree in a
|
||||
> worse state than either finishing or never starting. What the budget
|
||||
> forbids is *opening* new meta work above the line.
|
||||
> mid-task to satisfy a ratio wastes the work and leaves the tree worse
|
||||
> than either finishing or never starting. What the budget constrains is
|
||||
> *opening* new meta work above the line, and it constrains it with two
|
||||
> exemptions:
|
||||
>
|
||||
> 1. **Repairing the instrument that reports a breach is always in
|
||||
> budget.** A gate must never block its own remedy (ADR-0006 D2,
|
||||
> CB-RES-0005 §4). v1.5 had no such exemption, and the first thing it
|
||||
> did was forbid fixing the budget's own defect.
|
||||
> 2. **The maintainer may authorize above-line work**, recorded as
|
||||
> `authorized_above_budget:` in the workplan frontmatter. Undocumented
|
||||
> above-line meta work is a violation the evidence file must report.
|
||||
>
|
||||
> A pass that exceeds it **says so in its evidence file and names the
|
||||
> product work displaced**. That is the whole enforcement: this is a
|
||||
> reporting budget, not a gate, for the same reason the session-shape
|
||||
> budget is (CB-RES-0005 §4) — it constrains judgment, not artifacts.
|
||||
> product work displaced**. That is the whole enforcement: a reporting
|
||||
> budget, not a gate, for the same reason the session-shape budget is —
|
||||
> it constrains judgment, not artifacts.
|
||||
|
||||
> **Loop work cashes out (v1.6, ADR-0006 D4).** A `meta` pass must ship at
|
||||
> least one executable check, command, or gate. Findings that stay prose
|
||||
> are recorded as findings and do **not** count as delivery.
|
||||
>
|
||||
> *(v1.5, from the CB-WP-0007 spend review: CB-WP-0005 and CB-WP-0006 cost
|
||||
> ~$74, 31% of all spend, for zero measured efficiency gain. Their return
|
||||
> was correctness of claims, which is real and is not optimization. The
|
||||
> budget exists so that distinction has to be made out loud.)*
|
||||
> Measured basis: cost per response across five passes ran 0.123, 0.228,
|
||||
> 0.362, 0.298, 0.123 dollars, and the reading that survived the data is
|
||||
> that **cost tracks distance from a runnable check** — the passes that
|
||||
> shipped a command were cheap, the passes that argued about what a
|
||||
> number means were not (CB-EV-0007 §5). **Refuted if** a prose-only meta
|
||||
> pass lowers cost per response or catches a class no command could.
|
||||
|
||||
> **Every gate is an experiment (v1.6, ADR-0006 D3).** Each standing
|
||||
> control gate has an entry in `gates.toml` naming what it has caught, a
|
||||
> `review_by` date, and what would retire it. `make gate-review` reports
|
||||
> what is overdue and what has caught nothing; `loop-lint` fails when the
|
||||
> registry drifts from the Makefile in either direction.
|
||||
|
||||
v1.4 — corrected from CB-WP-0005 (assertion coverage) on
|
||||
2026-07-31. Change from v1.3: where a claim rests on numbers, the
|
||||
|
|
@ -31,28 +55,10 @@ adversarial reviewer must read the assertion behind each quoted number and
|
|||
**mutate it** — re-running the command that prints a number is not
|
||||
verification of that number (§Step 2).
|
||||
|
||||
v1.3 changed from v1.2: single source of fact is now executable
|
||||
(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift
|
||||
class its first gate.
|
||||
|
||||
v1.2 changes from v1.1: single source of fact; review targets the
|
||||
harness and states its sampling limit; correction vs retarget; the chaos
|
||||
roll's calibration window; the live cost budget. The design goal is now
|
||||
stated: **optimize for cheap correction, not for exhaustive prevention.**
|
||||
Rationale: `history/260731-loop-hardening-retrospective.md`.
|
||||
|
||||
v1.1 — corrected from CB-WP-0002 (cost accounting) on
|
||||
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||||
target; inherited numbers are re-derived before use; every reporting tool
|
||||
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||||
`history/260731-cost-accounting-retrospective.md`.
|
||||
|
||||
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||||
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||||
validity (the positive control), metric feasibility and instrument naming,
|
||||
four implementation rules the pass earned, and the requirement that
|
||||
evidence state what it does not support. Rationale and the failures behind
|
||||
each: `history/260731-inner-loop-retrospective.md`.
|
||||
v1.5 (meta budget), v1.3 (executable single source of fact), v1.2
|
||||
(design goal: cheap correction over exhaustive prevention), v1.1
|
||||
(instruments emit their own targets) and v1.0: see
|
||||
`specs/InnerLoopReference.md` §Change log.
|
||||
|
||||
Normative process for building every Clay-Borg capability. Referenced by
|
||||
all workplans.
|
||||
|
|
@ -111,7 +117,7 @@ are never skipped for code-producing work.
|
|||
| Tier | Weight of steps 1–3 | Structural trigger (forces at least this tier) |
|
||||
|---|---|---|
|
||||
| **L** | Full: separate survey, adversarial review, ADR | Creates a new capability port, or is named a high-leverage pass by the maintainer |
|
||||
| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, or adds/updates an external dependency |
|
||||
| **M** | Survey and ADR merged into one document; review optional | Touches a canonical interface, adds/updates an external dependency, **or changes whether or how the loop constrains its own operation** — budgets, gates, review requirements, or these tier rules (v1.6, ADR-0006 D5) |
|
||||
| **S** | One provenance paragraph in the commit message | Everything else (utilities, fixes, refactors inside a boundary) |
|
||||
|
||||
**The chaos roll.** After deriving the structural tier, roll **d4**
|
||||
|
|
@ -119,32 +125,13 @@ are never skipped for code-producing work.
|
|||
random (`shuf -e S M L -n 1`), overriding the structural derivation — up or
|
||||
down.
|
||||
|
||||
> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was
|
||||
> d10 and the mechanism **never fired**: two rolls across two workplans
|
||||
> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and
|
||||
> ~2 tier decisions per workplan it would take roughly twenty workplans to
|
||||
> observe four overrides, so the mechanism was set at a rate that prevented
|
||||
> its own evaluation — the one option T06 ruled out.
|
||||
>
|
||||
> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated
|
||||
> and either kept, returned to d10, or deleted. Expected ~3 firings in the
|
||||
> window, which is enough to see whether an overridden tier produces a
|
||||
> different outcome than the argued one.
|
||||
>
|
||||
> **Stated cost:** a chaos-L override on work that would have been S buys a
|
||||
> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001
|
||||
> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the
|
||||
> window is expected to cost **$20–30**. That is the price of finding out
|
||||
> whether the mechanism is worth keeping, and it is cheaper than carrying an
|
||||
> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration
|
||||
(`tier: M (structural L, chaos 10→M)`). **Record the roll every time,
|
||||
including when it changes nothing** (`tier: L (structural L, chaos 4)`),
|
||||
so a mechanism that never fires is visible rather than assumed. Purpose:
|
||||
an occasional random
|
||||
reweighting keeps the classification honest — arguing everything into S
|
||||
stops paying off when audits can compare argued tiers against the random
|
||||
sample — and occasionally forces a deep look at something "obviously
|
||||
trivial", which is where local optima hide.
|
||||
> **Calibration window, opened 2026-07-31, running to 12 tier
|
||||
> declarations** (declaration 4 of 12 as of 2026-08-01). The rate was
|
||||
> raised from d10 to d4 because at d10 the mechanism never fired and
|
||||
> so prevented its own evaluation. Record the roll every time,
|
||||
> including when it changes nothing (`tier: L (structural L, chaos 4)`).
|
||||
> Rationale, cost estimate and the two dead rolls:
|
||||
> `specs/InnerLoopReference.md` §Chaos roll — calibration.
|
||||
|
||||
Chaos limits: a rolled-down tier relaxes *process* weight only. Invariants
|
||||
(zero foreign types in canonical interfaces, determinism, passing
|
||||
|
|
|
|||
|
|
@ -139,3 +139,59 @@ A capability has completed the loop when all of the following are committed:
|
|||
([CostAccounting.md](CostAccounting.md))
|
||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||
file): what the loop itself should change
|
||||
|
||||
## Chaos roll — calibration (moved from InnerLoop v1.6)
|
||||
|
||||
> **Calibration window, opened 2026-07-31 (CB-WP-0003 T06).** The rate was
|
||||
> d10 and the mechanism **never fired**: two rolls across two workplans
|
||||
> (CB-WP-0001: 9, CB-WP-0002: 2), against ~0.2 expected firings. At d10 and
|
||||
> ~2 tier decisions per workplan it would take roughly twenty workplans to
|
||||
> observe four overrides, so the mechanism was set at a rate that prevented
|
||||
> its own evaluation — the one option T06 ruled out.
|
||||
>
|
||||
> Raised to **d4 (25%) for the next 12 tier declarations**, then evaluated
|
||||
> and either kept, returned to d10, or deleted. Expected ~3 firings in the
|
||||
> window, which is enough to see whether an overridden tier produces a
|
||||
> different outcome than the argued one.
|
||||
>
|
||||
> **Stated cost:** a chaos-L override on work that would have been S buys a
|
||||
> full survey, adversarial review, and ADR. Measured comparable: CB-WP-0001
|
||||
> T03 (a tier-L survey) cost **$9.91**. At 25% over 12 declarations the
|
||||
> window is expected to cost **$20–30**. That is the price of finding out
|
||||
> whether the mechanism is worth keeping, and it is cheaper than carrying an
|
||||
> unevaluated ritual indefinitely. Both rolls are recorded in the tier declaration
|
||||
(`tier: M (structural L, chaos 10→M)`). **Record the roll every time,
|
||||
including when it changes nothing** (`tier: L (structural L, chaos 4)`),
|
||||
so a mechanism that never fires is visible rather than assumed. Purpose:
|
||||
an occasional random
|
||||
reweighting keeps the classification honest — arguing everything into S
|
||||
stops paying off when audits can compare argued tiers against the random
|
||||
sample — and occasionally forces a deep look at something "obviously
|
||||
trivial", which is where local optima hide.
|
||||
|
||||
|
||||
## Change log — v1.0 to v1.3 (moved from InnerLoop v1.6)
|
||||
|
||||
v1.3 changed from v1.2: single source of fact is now executable
|
||||
(`make facts-check`, CB-WP-0004 T04), giving the duplicated-fact-drift
|
||||
class its first gate.
|
||||
|
||||
v1.2 changes from v1.1: single source of fact; review targets the
|
||||
harness and states its sampling limit; correction vs retarget; the chaos
|
||||
roll's calibration window; the live cost budget. The design goal is now
|
||||
stated: **optimize for cheap correction, not for exhaustive prevention.**
|
||||
Rationale: `history/260731-loop-hardening-retrospective.md`.
|
||||
|
||||
v1.1 — corrected from CB-WP-0002 (cost accounting) on
|
||||
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||||
target; inherited numbers are re-derived before use; every reporting tool
|
||||
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||||
`history/260731-cost-accounting-retrospective.md`.
|
||||
|
||||
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||||
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||||
validity (the positive control), metric feasibility and instrument naming,
|
||||
four implementation rules the pass earned, and the requirement that
|
||||
evidence state what it does not support. Rationale and the failures behind
|
||||
each: `history/260731-inner-loop-retrospective.md`.
|
||||
|
||||
|
|
|
|||
|
|
@ -524,6 +524,55 @@ def collect(slug, pin_ref=None, since_ref=None):
|
|||
}
|
||||
|
||||
|
||||
def pass_costs(slug, boundaries):
|
||||
"""Cost per pass window, from one read of the transcripts.
|
||||
|
||||
CB-WP-0009 T01 / ADR-0006 D1: a budget must measure the window it
|
||||
governs. `boundaries` is `[(label, start_iso)]` in ascending order;
|
||||
each window runs to the next label's start, the last to now.
|
||||
Responses before the first boundary land in `_before`.
|
||||
|
||||
Reads the transcripts **once**. Calling `collect()` per boundary
|
||||
would re-read every transcript per window, and `make status` is
|
||||
supposed to stay cheap enough that nobody replaces it with `ls`.
|
||||
"""
|
||||
prices = load_prices()
|
||||
stale = check_price_sheet_age(prices)
|
||||
if stale:
|
||||
raise Abort(stale)
|
||||
paths = transcript_paths(slug)
|
||||
if not paths:
|
||||
raise Abort(f"no transcripts found for {slug}")
|
||||
responses = []
|
||||
for path in paths:
|
||||
responses.extend(read_responses(path, None))
|
||||
if not responses:
|
||||
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
|
||||
|
||||
ordered = sorted(boundaries, key=lambda b: b[1])
|
||||
out = {label: {"cost": 0.0, "responses": 0} for label, _ in ordered}
|
||||
out["_before"] = {"cost": 0.0, "responses": 0}
|
||||
|
||||
for r in responses:
|
||||
cost = price_of(prices, r["model"], r["toks"], r["timestamp"]) or 0.0
|
||||
bucket = "_before"
|
||||
for label, start in ordered:
|
||||
if r["timestamp"] > start:
|
||||
bucket = label
|
||||
else:
|
||||
break
|
||||
out[bucket]["cost"] += cost
|
||||
out[bucket]["responses"] += 1
|
||||
|
||||
# Positive control: a bucketing that placed nothing in any real window
|
||||
# would report a confident 0% for every pass.
|
||||
placed = sum(v["responses"] for k, v in out.items() if k != "_before")
|
||||
if ordered and placed == 0:
|
||||
raise Abort("pass_costs bucketed every response before the first "
|
||||
"boundary — the boundaries are wrong, not the spend")
|
||||
return out
|
||||
|
||||
|
||||
def render(rep, by_task=False, composition=False):
|
||||
print(f"M-D2-CST cost report — {rep['slug']}")
|
||||
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
|
||||
|
|
@ -625,6 +674,31 @@ def self_test():
|
|||
check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong,
|
||||
f"${got:.4f} (1h would be ${wrong:.4f})")
|
||||
|
||||
# ADR-0006 D1: pass_costs buckets by window, and refuses to report a
|
||||
# confident 0% when every response landed before the first boundary.
|
||||
slug = "-home-worsch-clay-borg"
|
||||
try:
|
||||
real = pass_costs(slug, [("early", "2020-01-01T00:00:00Z")])
|
||||
check("pass_costs places responses in a real window",
|
||||
real["early"]["responses"] > 0 and real["early"]["cost"] > 0,
|
||||
f"{real['early']['responses']} response(s)")
|
||||
two = pass_costs(slug, [("a", "2020-01-01T00:00:00Z"),
|
||||
("b", "2026-07-31T00:00:00Z")])
|
||||
check("pass_costs splits at a boundary rather than pooling",
|
||||
two["a"]["responses"] > 0 and two["b"]["responses"] > 0,
|
||||
f"a={two['a']['responses']} b={two['b']['responses']}")
|
||||
check("pass_costs windows sum to the unwindowed total",
|
||||
abs((two["a"]["cost"] + two["b"]["cost"] + two["_before"]["cost"])
|
||||
- real["early"]["cost"] - real["_before"]["cost"]) < 0.01)
|
||||
except Abort as e:
|
||||
check("pass_costs places responses in a real window", False, str(e))
|
||||
try:
|
||||
pass_costs(slug, [("future", "2099-01-01T00:00:00Z")])
|
||||
check("pass_costs aborts when nothing lands in any window", False,
|
||||
"no Abort raised — every pass would read $0.00 and 0%")
|
||||
except Abort:
|
||||
check("pass_costs aborts when nothing lands in any window", True)
|
||||
|
||||
# AC-5: dedup invariant is enforced.
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||
|
|
|
|||
218
tools/gate-review.py
Normal file
218
tools/gate-review.py
Normal file
|
|
@ -0,0 +1,218 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Which control gates are due for a keep-or-kill argument? (ADR-0006 D3)
|
||||
|
||||
CB-WP-0009 T02. Six passes produced five standing control mechanisms and
|
||||
no way to retire any of them. Gates accumulate monotonically because each
|
||||
one was justified once, at the moment it was cheapest to justify.
|
||||
|
||||
This reads `gates.toml` and reports two things:
|
||||
|
||||
* gates past `review_by` — the date someone said they would argue for
|
||||
keeping it;
|
||||
* gates whose `caught` list is **empty** — which is not proof a gate is
|
||||
useless (it may be preventing rather than missing), but is the
|
||||
argument that has to be made out loud rather than never.
|
||||
|
||||
It **reports**. It does not fail the build, for CB-RES-0005 §4's reason:
|
||||
a gate that blocks the remedy when the metric breaches is a trap. The
|
||||
only failure exit here is a broken registry — a file that cannot be read
|
||||
would otherwise report "0 gates, all healthy".
|
||||
|
||||
Usage:
|
||||
python3 tools/gate-review.py
|
||||
python3 tools/gate-review.py --self-test
|
||||
"""
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
from repo import ROOT, enter_root
|
||||
|
||||
REGISTRY = os.path.join(ROOT, "gates.toml")
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError: # pragma: no cover - Python < 3.11
|
||||
import tomli as tomllib
|
||||
|
||||
|
||||
class Fail(Exception):
|
||||
pass
|
||||
|
||||
|
||||
REQUIRED = ("id", "name", "checks", "added", "review_by", "retire_if")
|
||||
|
||||
|
||||
def load(path=REGISTRY):
|
||||
"""Every gate, validated. A registry that parses but says nothing is
|
||||
the harness-does-nothing failure this project keeps finding."""
|
||||
if not os.path.exists(path):
|
||||
raise Fail(f"{os.path.relpath(path, ROOT)} is missing")
|
||||
with open(path, "rb") as fh:
|
||||
data = tomllib.load(fh)
|
||||
gates = data.get("gate") or []
|
||||
if not gates:
|
||||
raise Fail("registry parsed but contains no gates")
|
||||
for g in gates:
|
||||
missing = [k for k in REQUIRED if not g.get(k)]
|
||||
if missing:
|
||||
raise Fail(f"gate {g.get('id', '?')!r} is missing {missing}")
|
||||
for field in ("added", "review_by"):
|
||||
try:
|
||||
datetime.date.fromisoformat(g[field])
|
||||
except ValueError as e:
|
||||
raise Fail(f"gate {g['id']!r} has a bad {field}: {e}") from e
|
||||
return gates
|
||||
|
||||
|
||||
def make_targets(path=None):
|
||||
"""Targets declared in the Makefile, so an entry cannot name a
|
||||
command that does not exist."""
|
||||
path = path or os.path.join(ROOT, "Makefile")
|
||||
targets = set()
|
||||
for line in open(path):
|
||||
if line and not line[0].isspace() and ":" in line and not line.startswith("."):
|
||||
name = line.split(":", 1)[0].strip()
|
||||
if name and " " not in name:
|
||||
targets.add(name)
|
||||
return targets
|
||||
|
||||
|
||||
def report(today=None):
|
||||
today = today or datetime.date.today()
|
||||
gates = load()
|
||||
targets = make_targets()
|
||||
|
||||
overdue, quiet, broken = [], [], []
|
||||
for g in gates:
|
||||
if datetime.date.fromisoformat(g["review_by"]) <= today:
|
||||
overdue.append(g)
|
||||
if not g.get("caught"):
|
||||
quiet.append(g)
|
||||
target = g.get("target") or ""
|
||||
if target and target not in targets:
|
||||
broken.append((g, target))
|
||||
|
||||
print("gate review — every gate is an experiment (ADR-0006 D3)")
|
||||
print(f" registry {len(gates)} gate(s), {REGISTRY.split('/')[-1]}")
|
||||
print(f" today {today.isoformat()}")
|
||||
|
||||
print("\n gates")
|
||||
for g in gates:
|
||||
due = datetime.date.fromisoformat(g["review_by"])
|
||||
days = (due - today).days
|
||||
mark = "DUE " if days <= 0 else ("soon" if days <= 30 else "ok ")
|
||||
print(f" [{mark}] {g['id']:<14} {g['name']:<28} "
|
||||
f"review {g['review_by']} ({days:+}d) caught {len(g.get('caught', []))}")
|
||||
|
||||
if quiet:
|
||||
print("\n caught nothing yet — not a verdict, an argument that is owed")
|
||||
for g in quiet:
|
||||
print(f" {g['id']:<14} retire if: {g['retire_if']}")
|
||||
|
||||
if overdue:
|
||||
print("\n DUE for a keep-or-kill argument")
|
||||
for g in overdue:
|
||||
print(f" {g['id']:<14} {g['retire_if']}")
|
||||
|
||||
if broken:
|
||||
print("\n registry drift — entry names a target the Makefile lacks")
|
||||
for g, target in broken:
|
||||
print(f" {g['id']:<14} target {target!r}")
|
||||
|
||||
print(f"\n {len(overdue)} due, {len(quiet)} silent, {len(broken)} drifted")
|
||||
print(" reporting only — never fails the build (CB-RES-0005 §4)")
|
||||
return 0
|
||||
|
||||
|
||||
def self_test():
|
||||
"""Each check pins a way this tool could report a comfortable lie."""
|
||||
results = []
|
||||
|
||||
def check(name, ok, detail=""):
|
||||
results.append((name, ok, detail))
|
||||
|
||||
gates = load()
|
||||
check("the real registry loads", len(gates) >= 5, f"{len(gates)} gate(s)")
|
||||
check("every gate names what would retire it",
|
||||
all(len(g["retire_if"]) > 20 for g in gates))
|
||||
# Registry drift is the failure this exists to prevent: a gate added
|
||||
# to the Makefile with no entry, or an entry for a deleted target.
|
||||
targets = make_targets()
|
||||
named = [(g["id"], g["target"]) for g in gates if g.get("target")]
|
||||
check("every named target exists in the Makefile",
|
||||
all(t in targets for _i, t in named),
|
||||
", ".join(f"{i}:{t}" for i, t in named if t not in targets) or "all present")
|
||||
check("Makefile targets were actually parsed", len(targets) >= 15,
|
||||
f"{len(targets)} target(s)")
|
||||
|
||||
import tempfile
|
||||
|
||||
def registry(text):
|
||||
fh = tempfile.NamedTemporaryFile("w", suffix=".toml", delete=False)
|
||||
fh.write(text)
|
||||
fh.close()
|
||||
return fh.name
|
||||
|
||||
# A registry that parses to nothing must abort, not report "0 due".
|
||||
empty = registry("# no gates here\n")
|
||||
try:
|
||||
load(empty)
|
||||
check("an empty registry aborts", False, "reported a clean bill")
|
||||
except Fail:
|
||||
check("an empty registry aborts", True)
|
||||
finally:
|
||||
os.unlink(empty)
|
||||
|
||||
# A missing required field must abort rather than be treated as absent
|
||||
# evidence — `caught` is optional, the rest are not.
|
||||
partial = registry('[[gate]]\nid = "X"\nname = "n"\nchecks = "c"\n'
|
||||
'added = "2026-01-01"\nreview_by = "2026-02-01"\n')
|
||||
try:
|
||||
load(partial)
|
||||
check("a gate with no retire_if aborts", False, "accepted")
|
||||
except Fail:
|
||||
check("a gate with no retire_if aborts", True)
|
||||
finally:
|
||||
os.unlink(partial)
|
||||
|
||||
# The overdue arithmetic must actually fire.
|
||||
import io
|
||||
from contextlib import redirect_stdout
|
||||
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
report(today=datetime.date(2099, 1, 1))
|
||||
late = buf.getvalue()
|
||||
check("a far-future date marks every gate due",
|
||||
late.count("[DUE ]") == len(gates),
|
||||
f"{late.count('[DUE ]')} of {len(gates)}")
|
||||
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
report(today=datetime.date(2020, 1, 1))
|
||||
early = buf.getvalue()
|
||||
check("a far-past date marks none due", "[DUE ]" not in early)
|
||||
check("silent gates are named either way", "caught nothing yet" in early)
|
||||
|
||||
print("gate-review self-test (positive control)")
|
||||
ok = True
|
||||
for name, passed, det in results:
|
||||
print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {det}" if det else ""))
|
||||
ok &= passed
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
enter_root()
|
||||
try:
|
||||
if "--self-test" in sys.argv:
|
||||
return self_test()
|
||||
return report()
|
||||
except Fail as e:
|
||||
print(f"gate-review: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -162,12 +162,61 @@ def check_reporting_tools_self_test(root=REPO):
|
|||
return out
|
||||
|
||||
|
||||
def check_gate_registry(root=REPO):
|
||||
"""ADR-0006 D3 — every control gate is in `gates.toml`, and every
|
||||
entry names a real target.
|
||||
|
||||
The failure this prevents is drift in the direction nobody notices: a
|
||||
gate added to `make all` with no registry entry never acquires a
|
||||
review date, which is how five mechanisms accumulated with no way to
|
||||
retire any of them.
|
||||
"""
|
||||
out = []
|
||||
registry = os.path.join(root, "gates.toml")
|
||||
makefile = os.path.join(root, "Makefile")
|
||||
if not (os.path.exists(registry) and os.path.exists(makefile)):
|
||||
return out
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError: # pragma: no cover
|
||||
return out
|
||||
|
||||
with open(registry, "rb") as fh:
|
||||
data = tomllib.load(fh)
|
||||
gates = data.get("gate") or []
|
||||
if not gates:
|
||||
return [Finding("gates", "gates.toml", "registry contains no gates")]
|
||||
registered = {g.get("target") for g in gates if g.get("target")}
|
||||
exempt = set(data.get("not_control_gates") or [])
|
||||
|
||||
text = open(makefile).read()
|
||||
m = re.search(r"^all:(.*)$", text, re.M)
|
||||
deps = m.group(1).split() if m else []
|
||||
targets = {ln.split(":", 1)[0].strip() for ln in text.splitlines()
|
||||
if ln and not ln[0].isspace() and ":" in ln and not ln.startswith(".")}
|
||||
|
||||
for dep in deps:
|
||||
if dep not in registered and dep not in exempt:
|
||||
out.append(Finding(
|
||||
"gates", "gates.toml",
|
||||
f"`make all` runs {dep!r}, which is neither a registered "
|
||||
f"control gate nor listed in not_control_gates — classify it, "
|
||||
f"so it cannot acquire permanence without a review date"))
|
||||
for target in sorted(registered):
|
||||
if target not in targets:
|
||||
out.append(Finding(
|
||||
"gates", "gates.toml",
|
||||
f"entry names target {target!r}, which the Makefile lacks"))
|
||||
return out
|
||||
|
||||
|
||||
CHECKS = (
|
||||
check_loadability,
|
||||
check_evidence_no_unmeasured,
|
||||
check_survey_tier_and_chaos,
|
||||
check_review_trail,
|
||||
check_reporting_tools_self_test,
|
||||
check_gate_registry,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -196,6 +245,29 @@ def self_test():
|
|||
len(f) == 1 and "Big.md" in f[0].path,
|
||||
f"{len(f)} finding(s)")
|
||||
|
||||
# gates: an unclassified `all:` dependency trips, and so does an
|
||||
# entry naming a target the Makefile lacks.
|
||||
with open(os.path.join(tmp, "Makefile"), "w") as fh:
|
||||
fh.write("all: coverage newthing\ncoverage:\n\techo\n")
|
||||
with open(os.path.join(tmp, "gates.toml"), "w") as fh:
|
||||
fh.write('not_control_gates = []\n\n[[gate]]\nid = "G"\n'
|
||||
'name = "n"\ntarget = "coverage"\nchecks = "c"\n'
|
||||
'added = "2026-01-01"\nreview_by = "2026-02-01"\n'
|
||||
'retire_if = "r"\n')
|
||||
f = check_gate_registry(tmp)
|
||||
check("gate registry detects an unclassified all: dependency",
|
||||
len(f) == 1 and "newthing" in f[0].detail, f"{len(f)} finding(s)")
|
||||
with open(os.path.join(tmp, "gates.toml"), "w") as fh:
|
||||
fh.write('not_control_gates = ["newthing", "coverage"]\n\n[[gate]]\nid = "G"\n'
|
||||
'name = "n"\ntarget = "ghost"\nchecks = "c"\n'
|
||||
'added = "2026-01-01"\nreview_by = "2026-02-01"\n'
|
||||
'retire_if = "r"\n')
|
||||
f = check_gate_registry(tmp)
|
||||
check("gate registry detects an entry naming a missing target",
|
||||
len(f) == 1 and "ghost" in f[0].detail, f"{len(f)} finding(s)")
|
||||
os.unlink(os.path.join(tmp, "gates.toml"))
|
||||
os.unlink(os.path.join(tmp, "Makefile"))
|
||||
|
||||
# evidence: a table verdict trips; the word in prose does not.
|
||||
with open(os.path.join(tmp, "evidence", "E.md"), "w") as fh:
|
||||
fh.write("| AC-1 | x | unmeasured |\n"
|
||||
|
|
|
|||
137
tools/status.py
137
tools/status.py
|
|
@ -116,6 +116,90 @@ def cost_lines():
|
|||
return open_spend, rep["total"], mod
|
||||
|
||||
|
||||
# ------------------------------------------------------- the meta budget
|
||||
|
||||
# How many passes the budget looks back over. One pass is a coin flip —
|
||||
# a `meta` pass reads 100% and a `product` pass reads 0%, and neither
|
||||
# says anything about the balance the budget is about. Three is the
|
||||
# smallest window in which a ratio means something and still moves.
|
||||
TRAILING_PASSES = 3
|
||||
META_SOFT_PCT = 25
|
||||
|
||||
|
||||
def workplan_starts():
|
||||
"""[(id, kind, first_commit_iso)] — when each pass began.
|
||||
|
||||
A pass starts at the commit that *added* its workplan file, which is
|
||||
the only boundary git records and the same one `cb-cost --since` has
|
||||
been windowed on by hand since CB-WP-0004 T05.
|
||||
"""
|
||||
import datetime
|
||||
|
||||
out = []
|
||||
for path in sorted(glob.glob(os.path.join(ROOT, "workplans", "*.md"))):
|
||||
wid, _title, _status, _tasks, kind = parse_workplan(path)
|
||||
rel = os.path.relpath(path, ROOT)
|
||||
added = git("log", "--diff-filter=A", "--format=%cI", "--", rel)
|
||||
first = added.splitlines()[-1].strip() if added else ""
|
||||
if not (wid and first):
|
||||
continue
|
||||
# Transcript timestamps are UTC `...Z`; git prints a local offset.
|
||||
# Comparing the two as strings silently buckets everything into
|
||||
# `_before`, which is how this first read reported $0.
|
||||
first = (
|
||||
datetime.datetime.fromisoformat(first)
|
||||
.astimezone(datetime.timezone.utc)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
out.append((wid, kind, first))
|
||||
return sorted(out, key=lambda r: r[2])
|
||||
|
||||
|
||||
def meta_budget(plans):
|
||||
"""ADR-0006 D1: report the share for the window the budget governs."""
|
||||
_, _, mod = cost_lines()
|
||||
starts = workplan_starts()
|
||||
if not starts:
|
||||
print("\n meta budget UNAVAILABLE — no workplan start commits found")
|
||||
return
|
||||
costs = mod.pass_costs("-home-worsch-clay-borg",
|
||||
[(wid, when) for wid, _k, when in starts])
|
||||
|
||||
window = starts[-TRAILING_PASSES:]
|
||||
kinds = {wid: kind for wid, kind, _ in starts}
|
||||
total = sum(costs[wid]["cost"] for wid, _k, _w in window)
|
||||
# `mixed` splits evenly; stated rather than hidden.
|
||||
weight = {"product": 0.0, "meta": 1.0, "mixed": 0.5}
|
||||
meta = sum(costs[wid]["cost"] * weight.get(kinds[wid], 0.5)
|
||||
for wid, _k, _w in window)
|
||||
|
||||
if total <= 0:
|
||||
print("\n meta budget UNAVAILABLE — the trailing window measured $0")
|
||||
return
|
||||
share = 100 * meta / total
|
||||
mark = "ok " if share <= META_SOFT_PCT else "OVER"
|
||||
print(f"\n meta budget [{mark}] {share:.0f}% over the last "
|
||||
f"{len(window)} pass(es) (soft {META_SOFT_PCT}%, InnerLoop v1.6)")
|
||||
for wid, kind, _w in window:
|
||||
c = costs[wid]
|
||||
print(f" {wid} {kind:<8} ${c['cost']:>7,.2f} "
|
||||
f"{c['responses']:>4} response(s)")
|
||||
|
||||
# History, kept and labelled — the whole point of ADR-0006 D1 is that
|
||||
# this number is not what the target compares against.
|
||||
lifetime = sum(costs[wid]["cost"] for wid, _k, _w in starts)
|
||||
life_meta = sum(costs[wid]["cost"] * weight.get(kind, 0.5)
|
||||
for wid, kind, _ in starts)
|
||||
if lifetime > 0:
|
||||
print(f" history, all {len(starts)} passes: "
|
||||
f"{100 * life_meta / lifetime:.0f}% — NOT the metric "
|
||||
f"(ADR-0006 D1)")
|
||||
print(" NOTE: repairing the instrument that reports a breach is always")
|
||||
print(" in budget (ADR-0006 D2); other above-line meta work needs")
|
||||
print(" `authorized_above_budget:` in the workplan frontmatter.")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ report
|
||||
|
||||
|
||||
|
|
@ -162,31 +246,10 @@ def report():
|
|||
except Exception as e:
|
||||
print(f" UNAVAILABLE — {e}")
|
||||
|
||||
# InnerLoop v1.5 — soft 25% meta budget. Reported, never gated.
|
||||
# InnerLoop v1.6 — soft 25% meta budget over a trailing pass window.
|
||||
# Reported, never gated.
|
||||
try:
|
||||
_, _, mod = cost_lines()
|
||||
rep = mod.collect("-home-worsch-clay-borg", None)
|
||||
detail = rep["by_task_detail"]
|
||||
by_kind = {"product": 0.0, "meta": 0.0, "mixed": 0.0}
|
||||
attributed = 0.0
|
||||
for wid, _t, _s, tasks, kind in plans:
|
||||
for tid, _ts, _p in tasks:
|
||||
d = detail.get(tid)
|
||||
if d and kind in by_kind:
|
||||
by_kind[kind] += d["cost"]
|
||||
attributed += d["cost"]
|
||||
if attributed > 0:
|
||||
# `mixed` splits evenly; stated rather than hidden.
|
||||
meta = by_kind["meta"] + by_kind["mixed"] / 2
|
||||
share = 100 * meta / attributed
|
||||
mark = "ok " if share <= 25 else "OVER"
|
||||
print(f"\n meta budget [{mark}] {share:.0f}% of "
|
||||
f"${attributed:,.2f} attributed (soft 25%, InnerLoop v1.5)")
|
||||
print(f" product ${by_kind['product']:,.2f} "
|
||||
f"meta ${by_kind['meta']:,.2f} "
|
||||
f"mixed ${by_kind['mixed']:,.2f} (split 50/50)")
|
||||
print(" NOTE: attributed tasks only — early workplans used bare")
|
||||
print(" T## ids that collide across passes and are excluded.")
|
||||
meta_budget(plans)
|
||||
except Exception as e:
|
||||
print(f"\n meta budget UNAVAILABLE — {e}")
|
||||
|
||||
|
|
@ -279,6 +342,32 @@ def self_test():
|
|||
repr(h[:60]))
|
||||
check("task heading resolves to something", bool(h))
|
||||
|
||||
# ADR-0006 D1: the budget must measure the window it governs.
|
||||
starts = workplan_starts()
|
||||
check("every workplan has a start commit", len(starts) >= len(plans) - 1,
|
||||
f"{len(starts)} start(s) for {len(plans)} workplan(s)")
|
||||
check("start commits are UTC and comparable to transcript stamps",
|
||||
all(w.endswith("Z") for _i, _k, w in starts),
|
||||
# A local-offset stamp compares as a string against `...Z` and
|
||||
# silently buckets every response before the first boundary,
|
||||
# which is exactly how this reported $0 on its first run.
|
||||
", ".join(w for _i, _k, w in starts[:1]))
|
||||
check("start commits are in ascending order",
|
||||
[w for _i, _k, w in starts] == sorted(w for _i, _k, w in starts))
|
||||
|
||||
buf = io.StringIO()
|
||||
try:
|
||||
with redirect_stdout(buf):
|
||||
meta_budget(plans)
|
||||
text = buf.getvalue()
|
||||
check("budget reports a windowed share", "over the last" in text)
|
||||
check("budget labels the cumulative figure as history",
|
||||
"NOT the metric" in text)
|
||||
check("budget states the instrument-repair exemption",
|
||||
"in budget (ADR-0006 D2)" in text)
|
||||
except Exception as e:
|
||||
check("budget reports a windowed share", False, str(e))
|
||||
|
||||
# The report itself must run and produce substance, not a stub.
|
||||
buf = io.StringIO()
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
id: CB-WP-0009
|
||||
kind: meta
|
||||
title: "Make control gates experimental: window them, review them, cash them out"
|
||||
status: proposed
|
||||
status: done
|
||||
authorized_above_budget: "maintainer, 2026-08-01 — 'improve our policies where it seems promising; we will experiment our way to sensible task- and context-specific control and review gates'. Meta read 59% at the time."
|
||||
state_hub_workstream_id: "98c0af4a-415c-4cf5-92d0-a722eb70cb90"
|
||||
---
|
||||
|
|
@ -34,7 +34,7 @@ prose has not delivered.
|
|||
|
||||
```task
|
||||
id: CB-WP-0009-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "c6eefacd-5bb9-4640-8356-12e8112bcf11"
|
||||
```
|
||||
|
|
@ -53,11 +53,16 @@ it was a pure product pass — against 61% cumulative.
|
|||
**Refuted if** the two land within 20% of each other, in which case D1 is
|
||||
wrong and the machinery should be deleted rather than kept.
|
||||
|
||||
**Done 2026-08-01.** `cb-cost.pass_costs` buckets by pass window in one
|
||||
read of the transcripts; `make status` reports a trailing 3-pass share
|
||||
with the per-pass breakdown and the exemption. First reading **36% over
|
||||
the last 3 passes** against **49% lifetime** — CB-WP-0008 alone reads 0%.
|
||||
|
||||
## Task: `gates.toml` and `make gate-review`
|
||||
|
||||
```task
|
||||
id: CB-WP-0009-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "0fb6cfaf-9304-481a-a96f-1f0150f4d81d"
|
||||
```
|
||||
|
|
@ -77,11 +82,17 @@ behind the Makefile is worse than none, so `loop-lint` gains a check that
|
|||
every gate target has an entry and every entry names a real target. Its
|
||||
own self-test must fail when an entry is removed.
|
||||
|
||||
**Done 2026-08-01.** `gates.toml` (9 gates), `make gate-review`, and a
|
||||
`loop-lint` check in both directions — an unclassified `make all`
|
||||
dependency and an entry naming a target the Makefile lacks each produce
|
||||
a finding, both pinned by self-tests. First run: **0 due, 2 silent**
|
||||
(the chaos roll and `gate-review` itself).
|
||||
|
||||
## Task: InnerLoop v1.6
|
||||
|
||||
```task
|
||||
id: CB-WP-0009-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "0b1b33ac-8046-4b34-b287-7b32755d964a"
|
||||
```
|
||||
|
|
@ -100,11 +111,16 @@ Fold **D2**, **D4** and **D5** into `specs/InnerLoop.md`:
|
|||
pointer. Loadability is itself a gate; growing the file to fit a rule
|
||||
about gates would be the joke writing itself.
|
||||
|
||||
**Done 2026-08-01.** v1.6 in `specs/InnerLoop.md`, **386 lines** — the
|
||||
chaos-roll calibration rationale and the v1.0–v1.3 change log moved to
|
||||
`specs/InnerLoopReference.md`, so the file came in *smaller* than it
|
||||
started at 399.
|
||||
|
||||
## Task: control loop — did the changes fire?
|
||||
|
||||
```task
|
||||
id: CB-WP-0009-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "e06f57fc-147f-4cba-b398-0db6b060381f"
|
||||
```
|
||||
|
|
@ -123,3 +139,9 @@ answered from output, not argument:
|
|||
pass or needs the exception written down.
|
||||
4. **Cost**, against the 0.123–0.362 per-response range, and the meta
|
||||
share this pass itself consumed — reported under the rule it wrote.
|
||||
|
||||
**Done 2026-08-01.** [CB-EV-0008](../evidence/CB-EV-0008-adaptive-gates.md).
|
||||
The pass ran at **$0.177/response** — cheaper than every previous meta
|
||||
pass and still 1.4× the product pass, which is partial support for D4 and
|
||||
is reported as such. Meta share reads **45% over the trailing three**,
|
||||
over the line; the next pass should be product.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue