rail-knative/workplans/RAIL-KNATIVE-WP-0003-single-node-hpa-caps-and-request-trim.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	related	state_hub_workstream_id
RAIL-KNATIVE-WP-0003	workplan	Cap Knative HPA ceilings for one node and trim substrate CPU requests to 60m	financials	rail-knative	finished	codex	railiance	2026-10-21	2026-10-21		CUST-WP-0071 RESOURCE-WP-0008	4bf822ea-55a3-5990-9967-6684cdc74fa2

RAIL-KNATIVE-WP-0003 - Single-node HPA caps and CPU request trim

Receiving workplan for the three rail-knative rows of the resource-control node budget (CUST-WP-0071 T03/T04, RESOURCE-WP-0008).

Evidence and rationale

• Review: resource-control/docs/evidence/2026-10-21-node-allocation-review.md (Knative/Kourier item; "Knative HPA scale-out headroom" 1520m -> 230m).
• Budget rows: resource-control/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}
  • knative-serving/knative-serving (4 pods): 140m -> 40m, 7d p95 7.2m, peak 9.9m, throttle 0
  • knative-serving/net-kourier-controller: 30m -> 10m, p95 0.8m, peak 1.6m
  • kourier-system/3scale-kourier-gateway: 50m -> 10m, p95 3.6m, peak 3.8m
• Delivery decision: the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md.
• BEFORE state (read-only, 2026-10-21): docs/evidence/2026-10-21-hpa-caps-before/. Live HPAs: activator 1/20 (cpu 2%/100%), webhook 1/5 (10%/100%), 3scale-kourier-gateway 1/10 (8%/100%). All six Deployments 1/1 Ready. Node CPU requests 3605m (90%).

railiance01 is one node. The upstream HPA ceilings (20+5+10 pods) can request up to 1520m on it, and since the HPAs scale on percent of request, every request trim makes them scale out sooner. Capping maxReplicas makes the trim safe; minReplicas stays 1 (upstream). Knative recommends >= 2 activator replicas for HA across nodes; with one node that buys nothing, so min 1 stays. Limits stay upstream (1 CPU, webhook 500m), so the request trim cannot cause CPU throttling; only scheduling weight and HPA percent change. Memory requests stay upstream in this workplan (the budget also proposes memory down; that is a separate, later step).

Object	Before (live)	After (declared)
HPA knative-serving/activator max	20	3
HPA knative-serving/webhook max	5	2
HPA kourier-system/3scale-kourier-gateway max	10	3
activator cpu request	50m	10m
autoscaler cpu request	30m	10m
controller cpu request	30m	10m
webhook cpu request	30m	10m
net-kourier-controller cpu request	30m	10m
3scale-kourier-gateway cpu request	50m	10m
total cpu request	220m	60m

T01 - Declare caps and trimmed requests over the pinned v1.22.0 assets

id: RAIL-KNATIVE-WP-0003-T01
status: done
priority: high
state_hub_task_id: "b70fc781-dba2-54f6-9e70-f39ea86320ce"

2026-10-21: substrate/v1.22.0/cpu-requests.patch.yaml now declares 10m for all six containers; new substrate/v1.22.0/hpa-caps.patch.yaml sets only maxReplicas on the three upstream HPAs; the kustomization lists both; tests/test_substrate_requests.py and docs/substrate-runbook.md updated. Validation: python3 -m pytest -q -p no:cacheprovider tests 5 passed; validate_contracts.py --rail declarations/rail.yaml passed. Rendered with kubectl kustomize over the checksum-verified upstream core.yaml and kourier.yaml (staged by railiance-cluster/install/knative/render.sh): exactly 9 of 72 documents differ from upstream - six Deployment CPU requests and three HPA maxReplicas; memory, limits and minReplicas unchanged.

T02 - Owner handoff, apply and verify on railiance01

id: RAIL-KNATIVE-WP-0003-T02
status: done
priority: high
state_hub_task_id: "1e205e27-1c3a-5a16-a787-0afb3b35f75a"

Cross-repo coupling. The deploy path is railiance-cluster/install/knative/install.sh (render.sh + kubectl apply over ssh; no ArgoCD Application). Its overlays still carry the 2026-09-21 values, and railiance-cluster/tests/test_knative_render.py::test_matches_rail_knative_declaration compares them with this repository, so it fails until railiance-cluster follows: update install/knative/overlays/{serving-core,kourier}/cpu-requests.patch.yaml to 10m, add the HPA caps patch to both overlays (activator and webhook in serving-core, 3scale-kourier-gateway in kourier), update the DECLARED table in the test and the request list in install/knative/verify.sh, and add HPA maxReplicas checks to the verifier. Until then, a re-run of the installer reverts the live change to 220m and 20/5/10.

Apply (coordinator, in this order: caps first, so the lower requests never meet the upstream ceilings). Each request patch rolls one pod (maxSurge 1); node has ~395m request headroom, each new pod asks 10m.

ssh railiance01 'set -e
kubectl patch hpa activator -n knative-serving --type merge -p "{\"spec\":{\"maxReplicas\":3}}"
kubectl patch hpa webhook -n knative-serving --type merge -p "{\"spec\":{\"maxReplicas\":2}}"
kubectl patch hpa 3scale-kourier-gateway -n kourier-system --type merge -p "{\"spec\":{\"maxReplicas\":3}}"
for d in activator autoscaler controller webhook; do
  kubectl set resources deployment/$d -n knative-serving -c $d --requests=cpu=10m
done
kubectl set resources deployment/net-kourier-controller -n knative-serving -c controller --requests=cpu=10m
kubectl set resources deployment/3scale-kourier-gateway -n kourier-system -c kourier-gateway --requests=cpu=10m
for d in activator autoscaler controller webhook net-kourier-controller; do
  kubectl rollout status deployment/$d -n knative-serving --timeout=300s
done
kubectl rollout status deployment/3scale-kourier-gateway -n kourier-system --timeout=300s'

Owner path once railiance-cluster has followed (same result, durable):

cd ~/railiance-cluster && python3 -m pytest -q tests/test_knative_render.py \
  && install/knative/install.sh railiance01 && install/knative/verify.sh railiance01

Verify (all must hold).

ssh railiance01 'kubectl get hpa -n knative-serving; kubectl get hpa -n kourier-system
kubectl get deploy -n knative-serving; kubectl get deploy -n kourier-system
kubectl get deploy -n knative-serving -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"
kubectl get deploy -n kourier-system -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"
kubectl get ksvc -A; kubectl top pods -n knative-serving; kubectl top pods -n kourier-system'

• HPA max 3/2/3, min 1; six Deployments Available, pods Ready, no restarts.
• All six requests read 10m.
• ksvc rapp-qonto stays Ready=True and answers a request through Kourier (scale-from-zero via the activator included) — e.g. an in-cluster curl -sS -o /dev/null -w '%{http_code}' http://rapp-qonto.rapp-qonto.svc.cluster.local from a debug pod.
• No throttling regression: resource-control 7d throttle_ratio_7d stays 0 for the three workloads at the next weekly allocation report; HPA replicas stay at 1 at idle (activator ~10%, webhook ~30%, gateway ~40% of request).

Rollback (restores the BEFORE state exactly).

ssh railiance01 'set -e
kubectl set resources deployment/activator -n knative-serving -c activator --requests=cpu=50m
for d in autoscaler controller webhook; do
  kubectl set resources deployment/$d -n knative-serving -c $d --requests=cpu=30m
done
kubectl set resources deployment/net-kourier-controller -n knative-serving -c controller --requests=cpu=30m
kubectl set resources deployment/3scale-kourier-gateway -n kourier-system -c kourier-gateway --requests=cpu=50m
kubectl patch hpa activator -n knative-serving --type merge -p "{\"spec\":{\"maxReplicas\":20}}"
kubectl patch hpa webhook -n knative-serving --type merge -p "{\"spec\":{\"maxReplicas\":5}}"
kubectl patch hpa 3scale-kourier-gateway -n kourier-system --type merge -p "{\"spec\":{\"maxReplicas\":10}}"'

and git revert this workplan's declaration commit (plus the railiance-cluster follow-up if it landed). Full BEFORE objects: docs/evidence/2026-10-21-hpa-caps-before/.

Risks: a 10m request gives these pods little CPU weight under node contention (limits still allow bursts); the caps limit burst capacity of the activator and gateway to 3 pods, which on one node is the practical ceiling anyway; the 80% PDBs on activator, webhook and gateway already allow 0 disruptions at 1 replica (unchanged, but it blocks node drain).

Applied — 2026-10-21

Applied with the workstation kubeconfig under the founder's wave-1 authorization (the-custodian docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md), using the output of the railiance-cluster renderer at 3b18b84.

• HPA max is now 3/2/3, min 1.
• All six deployments request 10m CPU and are Ready 1/1.
• Kourier is still ClusterIP, and the rapp-qonto Knative service is Ready.
• Node CPU requests fell from 3605m to 3445m (−160m), as planned.

The cluster policy reject-secret-last-applied refused the Knative webhook certificate object in the render. Knative generates its contents at runtime, so it stayed unchanged; the fix is RCLUSTER-WP-0020-T04. Evidence: the-custodian docs/evidence/2026-10-21-wave1-apply.md.