CB-WP-0013-T02/T03: retire SH-3 as a gate; correct AM-4a and its target
ADR-0008, tier M (survey and ADR merged).
D1 — SH-3 retired as a gate, kept as a diagnostic. Investigating it
found a third defect, deeper than the two this pass was declared on.
Re-deriving batching from the raw transcripts, independently of cb-cost:
CB-WP-0011 pass 54 with tools 0 batched 0.0%
gap -> next decl 16 with tools 6 batched 37.5%
CB-WP-0012 pass 86 with tools 0 batched 0.0%
gap -> next decl 10 with tools 1 batched 10.0%
CB-WP-0013 so far 10 with tools 0 batched 0.0%
Zero batched turns in 150 in-pass responses; 37.5% in one gap, above the
20% floor. Batching needs two calls whose inputs are known at once —
orientation work. Implementation consumes each step's result before the
next. SH-3's window is since the last commit, which during a pass is
always implementation. The metric could not read above ~0% in the window
it was gated on. A floor the window structurally excludes is not a
target.
This pass's own declaration was also wrong: it claimed batching "has got
worse" (7.8-8.6% vs 1.1-6.3%). Differently-placed windows, not different
behaviour. Withdrawn — the same class of error, in the pass written to
correct it.
Not retargeting to match the measurement: the floor was not moved to 6%,
the gate was removed on an argument about what the quantity is worth.
The number is still reported; only the verdict is gone.
D2/D3 — AM-4a counts --edges normal,no-proc-macro: 157,202, not 246,250.
The target moves down with it, 250,000 -> 161,000, so the correction
hands back essentially nothing (headroom 3,750 -> 3,798). Three controls:
the exclusion drops exactly the five expected crates, only removes and
never adds, and is not a no-op.
The DFD gate then caught the follow-on it exists for — three historical
documents carrying live fact tags for a number that had changed. Not
rewritten; untagged, with a supersession banner.
AM-4b is deliberately not corrected: its proc-macro share is unmeasured.
gate-review now reads 0 due, 0 silent, 0 drifted — GATE-REVIEW earns its
first caught entry by forcing SH-3's re-justification, and the registry
has no silent gates left.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
6080c4cf60
commit
d2c3f340f9
9 changed files with 306 additions and 24 deletions
|
|
@ -632,7 +632,7 @@ def render(rep, by_task=False, composition=False):
|
|||
f"need {SH3_MIN_SAMPLE}]")
|
||||
else:
|
||||
print(f"{indent}SH-3 batching rate {100*sh['SH-3_batching_rate']:>11.1f}% "
|
||||
f"[{'ok ' if sh['SH-3_batching_rate']>=0.20 else 'FAIL'} target 20.0%]")
|
||||
f"[diagnostic, no floor — ADR-0008 D1]")
|
||||
print(f"{indent} {sh['tool_calls']} tool calls in "
|
||||
f"{sh['responses_with_tools']} responses; "
|
||||
f"{sh['calls_in_batched_turns']} in batched turns")
|
||||
|
|
@ -808,8 +808,14 @@ def self_test():
|
|||
below.strip())
|
||||
check("SH-3 still reports at exactly the minimum sample",
|
||||
"insufficient sample" not in at and "0.0%" in at, at.strip())
|
||||
check("SH-3 reports a real rate above the floor",
|
||||
"40.0%" in good and "ok" in good, good.strip())
|
||||
check("SH-3 reports a real rate, with no verdict attached",
|
||||
"40.0%" in good and "no floor" in good and "FAIL" not in good,
|
||||
good.strip())
|
||||
# ADR-0008 D1: the gate is gone. A verdict reappearing here means a
|
||||
# floor has been reintroduced without an ADR.
|
||||
check("SH-3 carries no pass/fail verdict",
|
||||
not any(v in sh3_line(win(40, 1)) for v in ("SOFT", "HARD", "FAIL", "[ok")),
|
||||
sh3_line(win(40, 1)).strip())
|
||||
# The one that matters: a refusal must not read as a measured zero.
|
||||
check("a refusal is distinguishable from a genuine 0.0%",
|
||||
below != sh3_line(win(SH3_MIN_SAMPLE, 0)))
|
||||
|
|
@ -853,9 +859,20 @@ SH3_MIN_SAMPLE = 14
|
|||
|
||||
|
||||
def sh3_line(win, min_sample=SH3_MIN_SAMPLE):
|
||||
"""SH-3, or an explicit refusal when the window cannot support a rate.
|
||||
"""SH-3 as a DIAGNOSTIC — no floor, no verdict (ADR-0008 D1).
|
||||
|
||||
Returns the line rather than printing it so the refusal is testable
|
||||
Retired as a gate because the window and the metric are systematically
|
||||
anti-correlated: batching needs two tool calls whose inputs are known
|
||||
at once, which is orientation work, and this window is *since the last
|
||||
commit*, which during a pass is implementation work. Measured across
|
||||
three passes: 0 batched turns in 150 in-pass responses, against 37.5%
|
||||
in the gap between two of them. A floor the window structurally
|
||||
excludes is not a target.
|
||||
|
||||
The minimum-sample refusal is kept. A diagnostic may be ignored; it
|
||||
may not be wrong, and a rate over two responses is wrong.
|
||||
|
||||
Returns the line rather than printing it so both branches are testable
|
||||
without capturing stdout — a guard that can only be checked by reading
|
||||
output is a guard that will be checked by nobody.
|
||||
"""
|
||||
|
|
@ -865,7 +882,7 @@ def sh3_line(win, min_sample=SH3_MIN_SAMPLE):
|
|||
f"[----] {n} response(s) with tool calls, need {min_sample}")
|
||||
rate = win["SH-3_batching_rate"]
|
||||
return (f" SH-3 batching {100*rate:>9.1f}% "
|
||||
f"[{'ok ' if rate >= 0.20 else 'SOFT'}] floor 20.0%")
|
||||
f"[diag] no floor — retired as a gate, ADR-0008 D1")
|
||||
|
||||
|
||||
def shape_budget(slug):
|
||||
|
|
|
|||
|
|
@ -28,15 +28,30 @@ import sys
|
|||
from repo import cargo_bin, enter_root
|
||||
|
||||
PACKAGE = "games-ground"
|
||||
# ADR-0008 D2. `--edges normal` includes proc-macro crates, which run in
|
||||
# the compiler and never reach a shipped binary — 89,048 lines, 36.2% of
|
||||
# what this tool used to call "what a game ships", `syn` alone 66,916. The
|
||||
# shipped-runtime configuration now excludes them.
|
||||
#
|
||||
# AM-4b is deliberately NOT corrected here: its proc-macro share has not
|
||||
# been measured, and correcting a second instrument on the strength of the
|
||||
# first one's ratio is the error this change exists to fix.
|
||||
PROC_MACRO_EXCLUDED = ["--edges", "normal,no-proc-macro"]
|
||||
CONFIGS = {
|
||||
"shipped-runtime": ["--no-default-features"],
|
||||
"shipped-runtime": ["--no-default-features"] + PROC_MACRO_EXCLUDED,
|
||||
"dev-toolchain": [],
|
||||
}
|
||||
|
||||
# AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails
|
||||
# the build: a gate that only reports is a suggestion.
|
||||
# ADR-0008 D3: the target moves down with the instrument. Leaving it at
|
||||
# 250,000 against a corrected 157,202 would hand this project 89,048 lines
|
||||
# of headroom it did not earn, in the same change that revealed the error.
|
||||
# 161,000 keeps ~2.4% of room where 250,000 kept ~1.5% — the small
|
||||
# rounding up is the only thing this decision gives back, because a target
|
||||
# with 1.5% of room fails on a dependency's patch release.
|
||||
TARGETS = {
|
||||
"shipped-runtime": 250_000,
|
||||
"shipped-runtime": 161_000,
|
||||
"dev-toolchain": 350_000,
|
||||
}
|
||||
|
||||
|
|
@ -55,8 +70,8 @@ def crates(extra_args):
|
|||
)
|
||||
sys.exit(1)
|
||||
out = subprocess.run(
|
||||
[cargo, "tree", "-p", PACKAGE, "--edges", "normal", "--prefix", "none"]
|
||||
+ extra_args,
|
||||
[cargo, "tree", "-p", PACKAGE, "--prefix", "none"]
|
||||
+ (extra_args if "--edges" in extra_args else ["--edges", "normal"] + extra_args),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
|
|
@ -125,6 +140,23 @@ def self_test():
|
|||
check("cargo resolves without caller PATH setup", bool(cargo_bin()),
|
||||
cargo_bin() or "NOT FOUND")
|
||||
|
||||
# ADR-0008 D2. The exclusion must remove exactly the proc-macro crates
|
||||
# and nothing else — a flag that quietly dropped a runtime dependency
|
||||
# would shrink the number in the direction this metric can be gamed.
|
||||
with_pm = crates(["--no-default-features"])
|
||||
without_pm = crates(["--no-default-features"] + PROC_MACRO_EXCLUDED)
|
||||
dropped = set(with_pm) - set(without_pm)
|
||||
check("the proc-macro exclusion drops exactly the expected crates",
|
||||
dropped == {"syn", "quote", "proc-macro2", "unicode-ident", "serde_derive"},
|
||||
f"dropped {sorted(dropped)}")
|
||||
check("the exclusion only ever removes crates, never adds",
|
||||
set(without_pm) <= set(with_pm),
|
||||
f"{len(with_pm)} -> {len(without_pm)}")
|
||||
# And it must actually remove something: an exclusion that excluded
|
||||
# nothing would leave the old figure while claiming the new meaning.
|
||||
check("the exclusion is not a no-op",
|
||||
len(dropped) > 0, f"{len(dropped)} crate(s) dropped")
|
||||
|
||||
check("targets defined for every configuration",
|
||||
set(TARGETS) == set(CONFIGS) and all(
|
||||
isinstance(v, int) and v > 0 for v in TARGETS.values()),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue