state-hub/workplans/STATE-WP-0099-node-allocation-wave1.md

---
id: STATE-WP-0099
type: workplan
title: "Apply the wave-1 node allocation change for the State Hub API, database and release preflight"
domain: infotech
repo: state-hub
status: active
owner: claude-code
topic_slug: infotech
created: "2026-10-21"
updated: "2026-10-21"
flavor: implementation
related: [CUST-WP-0071, RESOURCE-WP-0008, STATE-WP-0091]
state_hub_workstream_id: "c883eb1e-bfe5-5f93-9e4a-bd5b32cc4502"
---

This workplan receives the State Hub rows of the 2026-10-21 node allocation
review (resource-control `docs/evidence/2026-10-21-node-allocation-review.md`
and `2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}`).
It also covers wave 1 of the CUST-WP-0071 T03/T04 rollout
(`the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`).
The founder's answer "I may apply wave 1" lets the Claude session push and
apply these reviewed diffs through this repo's existing deploy path, one owner
at a time, with before/after evidence and a rollback receipt.

## Change

| Item | Before (live, 2026-10-21 06:39Z) | After | Source |
| --- | --- | --- | --- |
| API `state-hub` CPU request | 250m | **390m** | `deploy/railiance/apps/helm/state-hub-values.yaml` |
| API memory request / limits | 512Mi / 1000m, 2Gi | unchanged | same |
| MCP `state-hub-mcp` | 10m / 128Mi | unchanged | same |
| `databases/state-hub-db` (CNPG) requests | none | **70m / 416Mi**, no limits | `deploy/railiance/platform/state-hub-db-cluster.yaml` |
| Preflight API surge | constant 100m | **derived from the values the release deploys** (390m); fallback constant 390m | `scripts/release_headroom_preflight.py` |

Reasons:

- API: 7d p95 258m sits above the 250m request and the 975m peak comes from
  release and migration bursts. The review sets 390m (p95 x1.5, rounded up)
  as an override and leaves the limit unchanged. The budget row
  `state-hub/railiance-apps` reads "260→390" for two pods. That row adds the
  MCP 10m into the current figure but not into the proposed one. This change
  sets the API to 390m and keeps MCP at 10m, so the workload goes 260m→400m
  (+140m) and the review's node budget is 10m short. The 258.5m p95 is the
  workload figure. MCP p95 is about 2m, so the API's own p95 is about 256m and
  the result rounds to 390m either way.
- DB: the pod had no requests. 7d CPU p95 is 36.7m and peak 133.2m, so
  max(50m DB floor, p95 x1.5 = 55m, peak x0.5 = 67m) gives 70m. The memory
  working set p95 is 349Mi, x1.15 gives 416Mi. shared_buffers is the CNPG
  default (no override in the Cluster spec). The 416Mi figure is a request
  only and lowers nothing.
- Preflight: STATE-WP-0091 modelled a 100m API surge (revision 63). The live
  API pod has requested 250m since 2026-09-21. With maxSurge=1 and
  maxUnavailable=0 the surge pod carries the request of the release being
  deployed. The preflight now reads that request from the chart values merged
  with the production values (`--values` is repeatable in Helm `-f` order, and
  `--release` JSON still overrides). It also reads the live API request and
  reports `remaining_after_rollout_cpu_m`. When the release changes the request
  it notes the steady remainder, and it notes when the next release would be
  refused. A test keeps the fallback constants equal to the shipped values.

## Required ordering (node has 395m unrequested)

Before this change the node had 4000m allocatable and 3605m requested, which
leaves 395m. The API change needs a **390m surge pod** while the old 250m pod
is still running. Its net steady cost is +140m. The DB change restarts the
single instance in place, with a net cost of +70m.
Pod-effective deltas of the other wave-1 owners, taken from the review rows:
rail-knative −160m, railiance-forge +225m (gitea 100→200 counting its init
container, runner 0→125), rapp-telemetry +10m.

| Sequence | Free before the state-hub API rollout | API surge 390m |
| --- | ---: | --- |
| state-hub first (today) | 395m | fits by only 5m. Any CronJob pod (10–25m) or pending pod breaks it. **Not acceptable** |
| rail-knative reductions first | 555m | **fits, 165m margin** |
| knative + forge + telemetry first | 320m | **refused** (preflight: needs 390m) |

The order is therefore fixed:

1. **rail-knative** wave-1 reductions applied and settled (pods Running, no
   Pending pods). Free is then about 555m.
2. **state-hub API 250m→390m** (T03). The live preflight must report `ok: true`
   with `api_surge_cpu_m: 390` immediately before Helm starts. Avoid the
   02:40–04:00 CronJob window. Free afterwards is about 415m.
3. **state-hub-db 0→70m** (T04). Free afterwards is about 345m.
4. Then rapp-telemetry (+10m) and railiance-forge (+225m). Free afterwards is
   about 110m.

T03 must run **after the wave-1 reductions and before the wave-1 increases
(railiance-forge, rapp-telemetry)**. If forge lands first, the API change
waits until wave 2 frees headroom.

Consequence: after the whole of wave 1 the node has about 110m free, which is
below the 390m surge. **Routine State Hub releases will be refused by the
preflight until wave 2 decreases land** (the review's proposed steady total
is 2400m). That is the preflight working as intended, not a defect (T06).

## Evidence

- BEFORE (read-only): `docs/evidence/2026-10-21-resource-wave1/before.json`.
  It holds the live Deployment and CNPG specs, pod readiness and restarts,
  node allocation, Helm revision 77, hub health and the preflight as it ran
  before this change (old 100m model, ok).
- Fixtures for the four ordering cases:
  `tests/fixtures/release_headroom/2026-10-21-*.json`.
- Baselines for "no new restarts": API pod `state-hub-6c5b9c676-8chmx` 0, MCP
  pod `state-hub-mcp-b955b86b4-x849p` 0, and DB pod `state-hub-db-1` at 214
  lifetime restarts. The last DB termination was on 2026-10-14 with exit 255,
  reason Unknown, from a node restart. The DB pod is recreated by T04, so its
  count resets.

## Deploy path

The State Hub runs in-cluster on railiance01 (k3s single node). The API and
MCP are deployed by **Helm from this repo checkout** (release `state-hub`, ns
`state-hub`, not ArgoCD). The DB Cluster is **applied manually** from
`deploy/railiance/platform/state-hub-db-cluster.yaml`
(`app.kubernetes.io/managed-by: manual`). Its live last-applied spec equals
the source spec before this change. Run from the workstation with
`KUBECONFIG=~/.kube/config-railiance01`, after pushing this commit to Forgejo.

## Pre-check: read-only evidence and review against live

```task
id: STATE-WP-0099-T01
status: done
priority: high
state_hub_task_id: "d55a86fa-b08b-58ab-ab9d-4caaa7648187"
```

BEFORE state captured read-only. Review figures checked against the live
specs: API 250m, MCP 10m, DB without requests, 395m free, and the preflight
still on its 100m constant.

## Source change: values, DB Cluster, derived preflight surge, tests

```task
id: STATE-WP-0099-T02
status: done
priority: high
state_hub_task_id: "f6bd513f-c2b9-5a44-9e52-2ff6c94a7514"
```

`state-hub-values.yaml` sets the API to 390m. `state-hub-db-cluster.yaml`
sets 70m/416Mi. `release_headroom_preflight.py` derives the surge from the
values. `tests/test_release_headroom_preflight.py` gains six tests. The
historical 100m cases are kept explicitly through `historical-100m-release.json`.
`uv run pytest -q tests/test_release_headroom_preflight.py` gives 13 passed.

## Apply the API request (after rail-knative reductions, before forge/telemetry increases)

```task
id: STATE-WP-0099-T03
status: done
priority: high
state_hub_task_id: "61a60518-4607-5831-acd0-0f648f3a94b4"
```

```bash
cd ~/state-hub && export KUBECONFIG=~/.kube/config-railiance01
git push origin main                                   # Forgejo; then:
kubectl get pods -A --field-selector=status.phase=Pending   # must be empty
make railiance-state-hub-headroom                      # must print "ok": true, api_surge_cpu_m 390
make railiance-state-hub-server-dry-run
helm upgrade state-hub deploy/railiance/apps/charts/state-hub -n state-hub \
  -f deploy/railiance/apps/helm/state-hub-values.yaml --atomic --wait --timeout 5m
```

The image tag is unchanged: the values pin `main-c103cbf@sha256:4cc4…`, which
is what runs live. The upgrade still runs the pre-upgrade migrate hook (50m,
no-op schema). MCP does not roll because its spec is unchanged.

Verify:

```bash
curl -s http://127.0.0.1:8000/state/health          # status ok, db connected, instance_role primary
kubectl -n state-hub get deploy state-hub -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}'   # 390m
kubectl -n state-hub get pods                        # all 1/1 Running, RESTARTS 0
kubectl -n state-hub get endpoints state-hub-mcp     # one address
ssh railiance01 'curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://10.43.110.80:8001/sse'   # 200 (stream cut by -m)
kubectl describe node | grep -A3 "Allocated resources"   # cpu requests ≈ prior +140m
```

Rollback (restores 250m. A rollback surge needs 250m free, which the order
above leaves):

```bash
helm rollback state-hub 77 --wait --timeout 5m       # revision 77 = pre-change (live 2026-10-17)
```

Also revert the values line in git if the rollback is to be kept.

## Apply the DB request (after T03)

```task
id: STATE-WP-0099-T04
status: done
priority: high
state_hub_task_id: "08ca83df-cf37-502f-9a8c-48c3ef67fae7"
```

```bash
kubectl apply --dry-run=server -f deploy/railiance/platform/state-hub-db-cluster.yaml
kubectl diff -f deploy/railiance/platform/state-hub-db-cluster.yaml   # only spec.resources
kubectl apply -f deploy/railiance/platform/state-hub-db-cluster.yaml
kubectl -n databases get cluster state-hub-db -w     # until "Cluster in healthy state"
```

There is a single instance (`primaryUpdateStrategy: unsupervised`,
`primaryUpdateMethod: restart`). CNPG recreates `state-hub-db-1`, so expect a
short window, typically under a minute, in which the hub reports DB errors.
Announce it before applying, and run it outside the CronJob window.

Verify:

```bash
kubectl -n databases get pod state-hub-db-1 -o jsonpath='{.spec.containers[0].resources}'   # requests 70m/416Mi
kubectl -n databases get pod state-hub-db-1          # 1/1 Running, RESTARTS 0 after 15 min
curl -s http://127.0.0.1:8000/state/health           # db connected, schema ok
kubectl -n state-hub get pods                        # API/MCP RESTARTS still 0
```

Rollback:

```bash
git show 39b51a2:deploy/railiance/platform/state-hub-db-cluster.yaml | kubectl apply -f -
```

The last-applied three-way merge removes `spec.resources`, and the instance
restarts again.

## After evidence and receipt

```task
id: STATE-WP-0099-T05
status: done
priority: medium
state_hub_task_id: "6c1e9863-b7d6-565c-b3b9-8a0e6a690e2e"
```

Re-run the BEFORE collection (same fields) as
`docs/evidence/2026-10-21-resource-wave1/after.json`. Add the live preflight
output, the Helm revision number, the restart counts 15 minutes after each
step, and the rollback commands with revision 77 and the base commit. Hand the
receipt to CUST-WP-0071-T04.

## Release headroom after wave 1

```task
id: STATE-WP-0099-T06
status: wait
priority: medium
depends_on: [CUST-WP-0071-T04]
blocking_reason: "After wave 1 ~110m stays free, below the 390m surge; routine State Hub releases are refused until wave-2 decreases land"
state_hub_task_id: "0a8220c1-7543-58bf-b31b-c37d2433478b"
```

Once wave 2 has been applied, run `make railiance-state-hub-headroom` and
record whether a routine release (live 390m equal to target 390m) passes.
Do not lower the API request and do not drop the preflight to get a release
through. If an urgent release is needed earlier, Bernd decides between waiting
for wave 2 and an attended release with a brief API outage.

## Applied — 2026-10-21

- Preflight: ok, 605m remaining against a 390m surge (surge read from the values).
- Helm revision 78: the API requests 390m, health is ok and the schema is current.
- The upgrade also rolled out image main-c103cbf. The values bump bb525bc was committed on 17 Oct and never deployed; revision 77 ran main-5dd3df9. Those commits were already in main, and their migration c4d5e6f7a8b9 had already been applied. Rollback: `helm rollback state-hub 77`.
- state-hub-db (namespace databases) requests 70m/416Mi. The CNPG cluster is healthy and the hub stayed connected.
- Node requests after the probe removal: 3605m. Evidence: the-custodian docs/evidence/2026-10-21-wave1-apply.md.