--- id: STATE-WP-0099 type: workplan title: "Apply the wave-1 node allocation change for the State Hub API, database and release preflight" domain: infotech repo: state-hub status: active owner: claude-code topic_slug: infotech created: "2026-10-21" updated: "2026-10-21" flavor: implementation related: [CUST-WP-0071, RESOURCE-WP-0008, STATE-WP-0091] state_hub_workstream_id: "c883eb1e-bfe5-5f93-9e4a-bd5b32cc4502" --- This workplan receives the State Hub rows of the 2026-10-21 node allocation review (resource-control `docs/evidence/2026-10-21-node-allocation-review.md` and `2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}`). It also covers wave 1 of the CUST-WP-0071 T03/T04 rollout (`the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`). The founder's answer "I may apply wave 1" lets the Claude session push and apply these reviewed diffs through this repo's existing deploy path, one owner at a time, with before/after evidence and a rollback receipt. ## Change | Item | Before (live, 2026-10-21 06:39Z) | After | Source | | --- | --- | --- | --- | | API `state-hub` CPU request | 250m | **390m** | `deploy/railiance/apps/helm/state-hub-values.yaml` | | API memory request / limits | 512Mi / 1000m, 2Gi | unchanged | same | | MCP `state-hub-mcp` | 10m / 128Mi | unchanged | same | | `databases/state-hub-db` (CNPG) requests | none | **70m / 416Mi**, no limits | `deploy/railiance/platform/state-hub-db-cluster.yaml` | | Preflight API surge | constant 100m | **derived from the values the release deploys** (390m); fallback constant 390m | `scripts/release_headroom_preflight.py` | Reasons: - API: 7d p95 258m sits above the 250m request and the 975m peak comes from release and migration bursts. The review sets 390m (p95 x1.5, rounded up) as an override and leaves the limit unchanged. The budget row `state-hub/railiance-apps` reads "260→390" for two pods. That row adds the MCP 10m into the current figure but not into the proposed one. This change sets the API to 390m and keeps MCP at 10m, so the workload goes 260m→400m (+140m) and the review's node budget is 10m short. The 258.5m p95 is the workload figure. MCP p95 is about 2m, so the API's own p95 is about 256m and the result rounds to 390m either way. - DB: the pod had no requests. 7d CPU p95 is 36.7m and peak 133.2m, so max(50m DB floor, p95 x1.5 = 55m, peak x0.5 = 67m) gives 70m. The memory working set p95 is 349Mi, x1.15 gives 416Mi. shared_buffers is the CNPG default (no override in the Cluster spec). The 416Mi figure is a request only and lowers nothing. - Preflight: STATE-WP-0091 modelled a 100m API surge (revision 63). The live API pod has requested 250m since 2026-09-21. With maxSurge=1 and maxUnavailable=0 the surge pod carries the request of the release being deployed. The preflight now reads that request from the chart values merged with the production values (`--values` is repeatable in Helm `-f` order, and `--release` JSON still overrides). It also reads the live API request and reports `remaining_after_rollout_cpu_m`. When the release changes the request it notes the steady remainder, and it notes when the next release would be refused. A test keeps the fallback constants equal to the shipped values. ## Required ordering (node has 395m unrequested) Before this change the node had 4000m allocatable and 3605m requested, which leaves 395m. The API change needs a **390m surge pod** while the old 250m pod is still running. Its net steady cost is +140m. The DB change restarts the single instance in place, with a net cost of +70m. Pod-effective deltas of the other wave-1 owners, taken from the review rows: rail-knative −160m, railiance-forge +225m (gitea 100→200 counting its init container, runner 0→125), rapp-telemetry +10m. | Sequence | Free before the state-hub API rollout | API surge 390m | | --- | ---: | --- | | state-hub first (today) | 395m | fits by only 5m. Any CronJob pod (10–25m) or pending pod breaks it. **Not acceptable** | | rail-knative reductions first | 555m | **fits, 165m margin** | | knative + forge + telemetry first | 320m | **refused** (preflight: needs 390m) | The order is therefore fixed: 1. **rail-knative** wave-1 reductions applied and settled (pods Running, no Pending pods). Free is then about 555m. 2. **state-hub API 250m→390m** (T03). The live preflight must report `ok: true` with `api_surge_cpu_m: 390` immediately before Helm starts. Avoid the 02:40–04:00 CronJob window. Free afterwards is about 415m. 3. **state-hub-db 0→70m** (T04). Free afterwards is about 345m. 4. Then rapp-telemetry (+10m) and railiance-forge (+225m). Free afterwards is about 110m. T03 must run **after the wave-1 reductions and before the wave-1 increases (railiance-forge, rapp-telemetry)**. If forge lands first, the API change waits until wave 2 frees headroom. Consequence: after the whole of wave 1 the node has about 110m free, which is below the 390m surge. **Routine State Hub releases will be refused by the preflight until wave 2 decreases land** (the review's proposed steady total is 2400m). That is the preflight working as intended, not a defect (T06). ## Evidence - BEFORE (read-only): `docs/evidence/2026-10-21-resource-wave1/before.json`. It holds the live Deployment and CNPG specs, pod readiness and restarts, node allocation, Helm revision 77, hub health and the preflight as it ran before this change (old 100m model, ok). - Fixtures for the four ordering cases: `tests/fixtures/release_headroom/2026-10-21-*.json`. - Baselines for "no new restarts": API pod `state-hub-6c5b9c676-8chmx` 0, MCP pod `state-hub-mcp-b955b86b4-x849p` 0, and DB pod `state-hub-db-1` at 214 lifetime restarts. The last DB termination was on 2026-10-14 with exit 255, reason Unknown, from a node restart. The DB pod is recreated by T04, so its count resets. ## Deploy path The State Hub runs in-cluster on railiance01 (k3s single node). The API and MCP are deployed by **Helm from this repo checkout** (release `state-hub`, ns `state-hub`, not ArgoCD). The DB Cluster is **applied manually** from `deploy/railiance/platform/state-hub-db-cluster.yaml` (`app.kubernetes.io/managed-by: manual`). Its live last-applied spec equals the source spec before this change. Run from the workstation with `KUBECONFIG=~/.kube/config-railiance01`, after pushing this commit to Forgejo. ## Pre-check: read-only evidence and review against live ```task id: STATE-WP-0099-T01 status: done priority: high state_hub_task_id: "d55a86fa-b08b-58ab-ab9d-4caaa7648187" ``` BEFORE state captured read-only. Review figures checked against the live specs: API 250m, MCP 10m, DB without requests, 395m free, and the preflight still on its 100m constant. ## Source change: values, DB Cluster, derived preflight surge, tests ```task id: STATE-WP-0099-T02 status: done priority: high state_hub_task_id: "f6bd513f-c2b9-5a44-9e52-2ff6c94a7514" ``` `state-hub-values.yaml` sets the API to 390m. `state-hub-db-cluster.yaml` sets 70m/416Mi. `release_headroom_preflight.py` derives the surge from the values. `tests/test_release_headroom_preflight.py` gains six tests. The historical 100m cases are kept explicitly through `historical-100m-release.json`. `uv run pytest -q tests/test_release_headroom_preflight.py` gives 13 passed. ## Apply the API request (after rail-knative reductions, before forge/telemetry increases) ```task id: STATE-WP-0099-T03 status: todo priority: high state_hub_task_id: "61a60518-4607-5831-acd0-0f648f3a94b4" ``` ```bash cd ~/state-hub && export KUBECONFIG=~/.kube/config-railiance01 git push origin main # Forgejo; then: kubectl get pods -A --field-selector=status.phase=Pending # must be empty make railiance-state-hub-headroom # must print "ok": true, api_surge_cpu_m 390 make railiance-state-hub-server-dry-run helm upgrade state-hub deploy/railiance/apps/charts/state-hub -n state-hub \ -f deploy/railiance/apps/helm/state-hub-values.yaml --atomic --wait --timeout 5m ``` The image tag is unchanged: the values pin `main-c103cbf@sha256:4cc4…`, which is what runs live. The upgrade still runs the pre-upgrade migrate hook (50m, no-op schema). MCP does not roll because its spec is unchanged. Verify: ```bash curl -s http://127.0.0.1:8000/state/health # status ok, db connected, instance_role primary kubectl -n state-hub get deploy state-hub -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}' # 390m kubectl -n state-hub get pods # all 1/1 Running, RESTARTS 0 kubectl -n state-hub get endpoints state-hub-mcp # one address ssh railiance01 'curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://10.43.110.80:8001/sse' # 200 (stream cut by -m) kubectl describe node | grep -A3 "Allocated resources" # cpu requests ≈ prior +140m ``` Rollback (restores 250m. A rollback surge needs 250m free, which the order above leaves): ```bash helm rollback state-hub 77 --wait --timeout 5m # revision 77 = pre-change (live 2026-10-17) ``` Also revert the values line in git if the rollback is to be kept. ## Apply the DB request (after T03) ```task id: STATE-WP-0099-T04 status: todo priority: high state_hub_task_id: "08ca83df-cf37-502f-9a8c-48c3ef67fae7" ``` ```bash kubectl apply --dry-run=server -f deploy/railiance/platform/state-hub-db-cluster.yaml kubectl diff -f deploy/railiance/platform/state-hub-db-cluster.yaml # only spec.resources kubectl apply -f deploy/railiance/platform/state-hub-db-cluster.yaml kubectl -n databases get cluster state-hub-db -w # until "Cluster in healthy state" ``` There is a single instance (`primaryUpdateStrategy: unsupervised`, `primaryUpdateMethod: restart`). CNPG recreates `state-hub-db-1`, so expect a short window, typically under a minute, in which the hub reports DB errors. Announce it before applying, and run it outside the CronJob window. Verify: ```bash kubectl -n databases get pod state-hub-db-1 -o jsonpath='{.spec.containers[0].resources}' # requests 70m/416Mi kubectl -n databases get pod state-hub-db-1 # 1/1 Running, RESTARTS 0 after 15 min curl -s http://127.0.0.1:8000/state/health # db connected, schema ok kubectl -n state-hub get pods # API/MCP RESTARTS still 0 ``` Rollback: ```bash git show 39b51a2:deploy/railiance/platform/state-hub-db-cluster.yaml | kubectl apply -f - ``` The last-applied three-way merge removes `spec.resources`, and the instance restarts again. ## After evidence and receipt ```task id: STATE-WP-0099-T05 status: todo priority: medium state_hub_task_id: "6c1e9863-b7d6-565c-b3b9-8a0e6a690e2e" ``` Re-run the BEFORE collection (same fields) as `docs/evidence/2026-10-21-resource-wave1/after.json`. Add the live preflight output, the Helm revision number, the restart counts 15 minutes after each step, and the rollback commands with revision 77 and the base commit. Hand the receipt to CUST-WP-0071-T04. ## Release headroom after wave 1 ```task id: STATE-WP-0099-T06 status: wait priority: medium depends_on: [CUST-WP-0071-T04] blocking_reason: "After wave 1 ~110m stays free, below the 390m surge; routine State Hub releases are refused until wave-2 decreases land" state_hub_task_id: "0a8220c1-7543-58bf-b31b-c37d2433478b" ``` Once wave 2 has been applied, run `make railiance-state-hub-headroom` and record whether a routine release (live 390m equal to target 390m) passes. Do not lower the API request and do not drop the preflight to get a release through. If an urgent release is needed earlier, Bernd decides between waiting for wave 2 and an attended release with a brief API outage.