state-hub/docs/evidence/2026-09-14-release-headroom.md
tegwick 23f2038c5b
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Point STATE-WP-0091 residual at RAIL-BS-WP-0014.
Assistant: grok
Assistant-Session: 01a09dc1-b21e-77e1-919e-fcad2f82b267
2026-09-14 10:17:40 +02:00

2 KiB

STATE-WP-0091 — release headroom and demand handoff

Date: 2026-09-14

T01 — preflight

scripts/release_headroom_preflight.py is on the promotion path (make railiance-state-hub-headroom, PROMOTE.md step 0). It is read-only.

Proved against fixtures of the 2026-09-10 incident:

Case Remaining Result
Revision 61 65m refuse API 100m surge
Revision 63 105m narrowly sufficient; not factory admission
105m plus 50m unrelated pending 55m effective refuse

Atomic rollback stays --atomic. Requests are not lowered by this tool.

Live kubectl on 2026-09-14 refused: remaining 15m, pending unrelated 25m, migrate hook 50m unmet. The preflight is doing its job on the current node.

T02 — MCP demand and margin

STATE-WP-0090 recorded 1,078 five-minute samples in a seven-day window (partial coverage): MCP p99 2.35m, peak 30.45m. The MCP request was cut 50m → 10m; limit 500m, memory and replicas unchanged.

That sample set is still not a week of complete coverage. It is enough to say:

  • 10m request is above p99 and below peak; bursts still use the 500m limit.
  • 105m node remainder is only 5m above the API surge. Concurrent MCP surge (10m) or any unrelated pending pod ≥ 6m makes it insufficient.
  • Durable release margin for State Hub: remaining CPU after current allocations must cover API surge 100m + migrate hook 50m sequenced separately, and must refuse when unrelated pending demand would consume the 5m sliver. Do not treat 105m as shared-capacity admission.

Shared allocation and provisioning stay with other owners. Receiving records:

  • RAIL-BS-WP-0014 (railiance-cluster) — node remaining vs surge, pending pods
  • RESOURCE-WP-0007 (resource-control) — reef-railiance-k3s portfolio, not admission
  • RPF-WP-0041 (railiance-platform) — no platform provisioning change from 105m
  • CUST-WP-0071 already owns weekly fleet allocation review

HFACT T01/T04 factory demand is not admitted by this 105m figure.