state-hub/workplans/STATE-WP-0099-node-allocation-wave1.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	flavor	related
STATE-WP-0099	workplan	Apply the wave-1 node allocation change for the State Hub API, database and release preflight	infotech	state-hub	active	claude-code	infotech	2026-10-21	2026-10-21	implementation	CUST-WP-0071 RESOURCE-WP-0008 STATE-WP-0091

This workplan receives the State Hub rows of the 2026-10-21 node allocation review (resource-control docs/evidence/2026-10-21-node-allocation-review.md and 2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}). It also covers wave 1 of the CUST-WP-0071 T03/T04 rollout (the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md). The founder's answer "I may apply wave 1" lets the Claude session push and apply these reviewed diffs through this repo's existing deploy path, one owner at a time, with before/after evidence and a rollback receipt.

Change

Item	Before (live, 2026-10-21 06:39Z)	After	Source
API state-hub CPU request	250m	390m	deploy/railiance/apps/helm/state-hub-values.yaml
API memory request / limits	512Mi / 1000m, 2Gi	unchanged	same
MCP state-hub-mcp	10m / 128Mi	unchanged	same
databases/state-hub-db (CNPG) requests	none	70m / 416Mi, no limits	deploy/railiance/platform/state-hub-db-cluster.yaml
Preflight API surge	constant 100m	derived from the values the release deploys (390m); fallback constant 390m	scripts/release_headroom_preflight.py

Reasons:

• API: 7d p95 258m sits above the 250m request and the 975m peak comes from release and migration bursts. The review sets 390m (p95 x1.5, rounded up) as an override and leaves the limit unchanged. The budget row state-hub/railiance-apps reads "260→390" for two pods. That row adds the MCP 10m into the current figure but not into the proposed one. This change sets the API to 390m and keeps MCP at 10m, so the workload goes 260m→400m (+140m) and the review's node budget is 10m short. The 258.5m p95 is the workload figure. MCP p95 is about 2m, so the API's own p95 is about 256m and the result rounds to 390m either way.
• DB: the pod had no requests. 7d CPU p95 is 36.7m and peak 133.2m, so max(50m DB floor, p95 x1.5 = 55m, peak x0.5 = 67m) gives 70m. The memory working set p95 is 349Mi, x1.15 gives 416Mi. shared_buffers is the CNPG default (no override in the Cluster spec). The 416Mi figure is a request only and lowers nothing.
• Preflight: STATE-WP-0091 modelled a 100m API surge (revision 63). The live API pod has requested 250m since 2026-09-21. With maxSurge=1 and maxUnavailable=0 the surge pod carries the request of the release being deployed. The preflight now reads that request from the chart values merged with the production values (--values is repeatable in Helm -f order, and --release JSON still overrides). It also reads the live API request and reports remaining_after_rollout_cpu_m. When the release changes the request it notes the steady remainder, and it notes when the next release would be refused. A test keeps the fallback constants equal to the shipped values.

Required ordering (node has 395m unrequested)

Before this change the node had 4000m allocatable and 3605m requested, which leaves 395m. The API change needs a 390m surge pod while the old 250m pod is still running. Its net steady cost is +140m. The DB change restarts the single instance in place, with a net cost of +70m. Pod-effective deltas of the other wave-1 owners, taken from the review rows: rail-knative −160m, railiance-forge +225m (gitea 100→200 counting its init container, runner 0→125), rapp-telemetry +10m.

Sequence	Free before the state-hub API rollout	API surge 390m
state-hub first (today)	395m	fits by only 5m. Any CronJob pod (10–25m) or pending pod breaks it. Not acceptable
rail-knative reductions first	555m	fits, 165m margin
knative + forge + telemetry first	320m	refused (preflight: needs 390m)

The order is therefore fixed:

1. rail-knative wave-1 reductions applied and settled (pods Running, no Pending pods). Free is then about 555m.
2. state-hub API 250m→390m (T03). The live preflight must report ok: true with api_surge_cpu_m: 390 immediately before Helm starts. Avoid the 02:40–04:00 CronJob window. Free afterwards is about 415m.
3. state-hub-db 0→70m (T04). Free afterwards is about 345m.
4. Then rapp-telemetry (+10m) and railiance-forge (+225m). Free afterwards is about 110m.

T03 must run after the wave-1 reductions and before the wave-1 increases (railiance-forge, rapp-telemetry). If forge lands first, the API change waits until wave 2 frees headroom.

Consequence: after the whole of wave 1 the node has about 110m free, which is below the 390m surge. Routine State Hub releases will be refused by the preflight until wave 2 decreases land (the review's proposed steady total is 2400m). That is the preflight working as intended, not a defect (T06).

Evidence

• BEFORE (read-only): docs/evidence/2026-10-21-resource-wave1/before.json. It holds the live Deployment and CNPG specs, pod readiness and restarts, node allocation, Helm revision 77, hub health and the preflight as it ran before this change (old 100m model, ok).
• Fixtures for the four ordering cases: tests/fixtures/release_headroom/2026-10-21-*.json.
• Baselines for "no new restarts": API pod state-hub-6c5b9c676-8chmx 0, MCP pod state-hub-mcp-b955b86b4-x849p 0, and DB pod state-hub-db-1 at 214 lifetime restarts. The last DB termination was on 2026-10-14 with exit 255, reason Unknown, from a node restart. The DB pod is recreated by T04, so its count resets.

Deploy path

The State Hub runs in-cluster on railiance01 (k3s single node). The API and MCP are deployed by Helm from this repo checkout (release state-hub, ns state-hub, not ArgoCD). The DB Cluster is applied manually from deploy/railiance/platform/state-hub-db-cluster.yaml (app.kubernetes.io/managed-by: manual). Its live last-applied spec equals the source spec before this change. Run from the workstation with KUBECONFIG=~/.kube/config-railiance01, after pushing this commit to Forgejo.

Pre-check: read-only evidence and review against live

id: STATE-WP-0099-T01
status: done
priority: high

BEFORE state captured read-only. Review figures checked against the live specs: API 250m, MCP 10m, DB without requests, 395m free, and the preflight still on its 100m constant.

Source change: values, DB Cluster, derived preflight surge, tests

id: STATE-WP-0099-T02
status: done
priority: high

state-hub-values.yaml sets the API to 390m. state-hub-db-cluster.yaml sets 70m/416Mi. release_headroom_preflight.py derives the surge from the values. tests/test_release_headroom_preflight.py gains six tests. The historical 100m cases are kept explicitly through historical-100m-release.json. uv run pytest -q tests/test_release_headroom_preflight.py gives 13 passed.

Apply the API request (after rail-knative reductions, before forge/telemetry increases)

id: STATE-WP-0099-T03
status: todo
priority: high

cd ~/state-hub && export KUBECONFIG=~/.kube/config-railiance01
git push origin main                                   # Forgejo; then:
kubectl get pods -A --field-selector=status.phase=Pending   # must be empty
make railiance-state-hub-headroom                      # must print "ok": true, api_surge_cpu_m 390
make railiance-state-hub-server-dry-run
helm upgrade state-hub deploy/railiance/apps/charts/state-hub -n state-hub \
  -f deploy/railiance/apps/helm/state-hub-values.yaml --atomic --wait --timeout 5m

The image tag is unchanged: the values pin main-c103cbf@sha256:4cc4…, which is what runs live. The upgrade still runs the pre-upgrade migrate hook (50m, no-op schema). MCP does not roll because its spec is unchanged.

Verify:

curl -s http://127.0.0.1:8000/state/health          # status ok, db connected, instance_role primary
kubectl -n state-hub get deploy state-hub -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}'   # 390m
kubectl -n state-hub get pods                        # all 1/1 Running, RESTARTS 0
kubectl -n state-hub get endpoints state-hub-mcp     # one address
ssh railiance01 'curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://10.43.110.80:8001/sse'   # 200 (stream cut by -m)
kubectl describe node | grep -A3 "Allocated resources"   # cpu requests ≈ prior +140m

Rollback (restores 250m. A rollback surge needs 250m free, which the order above leaves):

helm rollback state-hub 77 --wait --timeout 5m       # revision 77 = pre-change (live 2026-10-17)

Also revert the values line in git if the rollback is to be kept.

Apply the DB request (after T03)

id: STATE-WP-0099-T04
status: todo
priority: high

kubectl apply --dry-run=server -f deploy/railiance/platform/state-hub-db-cluster.yaml
kubectl diff -f deploy/railiance/platform/state-hub-db-cluster.yaml   # only spec.resources
kubectl apply -f deploy/railiance/platform/state-hub-db-cluster.yaml
kubectl -n databases get cluster state-hub-db -w     # until "Cluster in healthy state"

There is a single instance (primaryUpdateStrategy: unsupervised, primaryUpdateMethod: restart). CNPG recreates state-hub-db-1, so expect a short window, typically under a minute, in which the hub reports DB errors. Announce it before applying, and run it outside the CronJob window.

Verify:

kubectl -n databases get pod state-hub-db-1 -o jsonpath='{.spec.containers[0].resources}'   # requests 70m/416Mi
kubectl -n databases get pod state-hub-db-1          # 1/1 Running, RESTARTS 0 after 15 min
curl -s http://127.0.0.1:8000/state/health           # db connected, schema ok
kubectl -n state-hub get pods                        # API/MCP RESTARTS still 0

Rollback:

git show 39b51a2:deploy/railiance/platform/state-hub-db-cluster.yaml | kubectl apply -f -

The last-applied three-way merge removes spec.resources, and the instance restarts again.

After evidence and receipt

id: STATE-WP-0099-T05
status: todo
priority: medium

Re-run the BEFORE collection (same fields) as docs/evidence/2026-10-21-resource-wave1/after.json. Add the live preflight output, the Helm revision number, the restart counts 15 minutes after each step, and the rollback commands with revision 77 and the base commit. Hand the receipt to CUST-WP-0071-T04.

Release headroom after wave 1

id: STATE-WP-0099-T06
status: wait
priority: medium
depends_on: [CUST-WP-0071-T04]
blocking_reason: "After wave 1 ~110m stays free, below the 390m surge; routine State Hub releases are refused until wave-2 decreases land"

Once wave 2 has been applied, run make railiance-state-hub-headroom and record whether a routine release (live 390m equal to target 390m) passes. Do not lower the API request and do not drop the preflight to get a release through. If an urgent release is needed earlier, Bernd decides between waiting for wave 2 and an attended release with a brief API outage.