clay-borg/.forgejo/workflows/ci.yml

52 lines
2 KiB
YAML
Raw Normal View History

name: ci
on:
push:
branches: [main]
pull_request:
jobs:
check:
runs-on: docker
container:
image: rust:1.97
steps:
- uses: actions/checkout@v4
- run: rustup component add rustfmt clippy
- run: cargo fmt --all --check
- run: cargo clippy --workspace --all-targets -- -D warnings
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
# The shipped runtime is a distinct configuration (AM-4a); it must
# compile and lint on its own, or the feature split rots.
- run: cargo clippy -p games-ground --no-default-features -- -D warnings
- run: cargo test --workspace
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
# No `|| test $? -eq 2`: cb-sim now fails on an unregistered game
# prefix and on a run that executed nothing. A silent skip is the
# failure this step exists to catch.
- run: cargo run -q -p cb-sim -- scenarios/ground/*.yaml
# AM-1: every numbered GR-rule is claimed by a scenario, and no
# scenario claims a rule the spec does not define.
- run: make coverage
# AM-4a/AM-4b: third-party source under audit, per configuration.
- run: make dep-weight
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
# Positive control for the cost collector (AC-5..AC-9). Does not
# gate on a dollar figure — transcripts are not present in CI — but
# proves the collector still detects the failures it claims to.
T01: audit every InnerLoop rule, and make the checkable ones executable 41 rules classified executable / checkable / decorative, each tagged with the failure class it catches. Counts: 11 executable, 22 checkable, 4 decorative (one of them dead policy). Audit: history/260731-inner-loop-rule-audit.md New tools/loop-lint.py makes 7 rules executable (tier declared, chaos roll recorded, tier-L review trail, unmeasured-in-evidence, whole-file loadability, reporting tools expose --self-test). It found three real violations on its first run, none previously visible: - specs/ArchitectureBlueprint.md was 543 lines against a ~400 limit the loop has stated since v0.2 and never measured. Split at its own section boundaries into Blueprint (1-8) + Runtime (9-15). - tools/dep-weight.py and tools/rule-coverage.py had positive-control logic and no --self-test, so nothing verified the control worked. Adding rule-coverage's self-test exposed a latent instance of the exact class this workplan is about: if the spec regex stopped matching, rules was empty, missing was empty, and the tool exited 0 reporting "0/0" -- a silent pass, in the tool that reports our headline AM-1 number. Both tools now assert they found something before reporting. Two demotions applied in the spec rather than left implicit: "structured over prose" is marked guidance (nothing can check it), and the 8k/10k token budget is struck through and marked DEAD POLICY pointing at T05. The audit's uncomfortable finding: rule 13 (re-derive inherited numbers) has no mechanical form, is deliberately left decorative, and caught the LARGEST error in CB-WP-0002. That is a counter-example to this workplan's own hypothesis. "A rule that cannot be executed is not a rule" is wrong as stated; the defensible version is that such a rule cannot be relied on to fire, so it must not be the only defence for a class that matters. Class coverage: harness-does-nothing has five executable rules; trusted-arithmetic has ZERO and produced the largest single error. make loop-lint and make self-tests wired into `make all` and CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 09:16:00 +02:00
- run: make self-tests
# InnerLoop rules made executable (CB-WP-0003 T01). Fails on an
# overlong artifact, a survey missing its tier/chaos declaration, an
# approved tier-L survey with no review trail, `unmeasured` in an
# evidence table, or a reporting tool with no --self-test.
- run: make loop-lint
T04: tools/cb-cost.py — and its positive control fires on first contact Collector per ADR-0003: enumerates every transcript including the subagents/ tree, dedups by requestId, prices per model and per cache TTL, attributes on (prev_commit, this_commit] intervals, and reconciles to the cent or aborts. The positive control caught a real defect on its very first run against real data, which is the entire argument for writing it: CA-02 assumed usage is identical across the lines of one requestId. True in the main transcript (206/206 groups, verified twice — by the survey and by the adversarial reviewer). FALSE in the subagents/ tree, where output_tokens is a running count: one response reads 5, 5, 195 across its three lines. First-wins scored it at 5. So CA-02 now splits: input-side counters are charged once and must be identical (assertion retained); output_tokens resolves to the max (CA-02a). AC-9 pins the exact 5,5,195 case as a regression test. The acceptance target moved again as a result, for the third time: $248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same first-wins method the tool just disproved, so the tool failing its target was the tool being correct. Target updated, not the tool. Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual $0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10. make cost / cost-test / cost-pin wired; cost-test added to `make all` and to CI, where it gates the collector's assertions without needing transcripts present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
CI: enforce every gate; close the silent-skip holes The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:02:33 +02:00
# InnerLoop v1.0 positive control, enforced rather than asserted in
# prose: --test runs every benchmark once, so a workload that
# stalls or produces the wrong event count fails the build instead
# of silently reporting throughput for work that never happened.
- run: cargo bench -p games-ground --bench synthetic -- --test