coulomb/rail-knative
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Declare the Knative v1.22.0 substrate CPU requests set live on railiance01

Browse source 
Strategic-merge patches and a kustomization over the checksum-pinned
upstream assets, a runbook with the rollout-deadlock and HPA lessons,
offline tests, and RAIL-KNATIVE-WP-0002 with the railiance-cluster
installer handoff as a wait task. Verified read-only against live.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 63291@bnt-lap001
Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703
This commit is contained in:
tegwick 2026-09-21 18:40:08 +02:00
parent 3b2d230923
commit fd9b848b63
5 changed files with 264 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

50
docs/substrate-runbook.md Normal file
View file

@ -0,0 +1,50 @@
# Knative substrate runbook: declared CPU requests
The Knative Serving and Kourier v1.22.0 install on railiance01 is performed by
`railiance-cluster/install/knative/install.sh` from checksum-pinned upstream
assets. rail-knative declares the CPU requests that install must carry, in
`substrate/v1.22.0/cpu-requests.patch.yaml` (strategic-merge patches) and
`substrate/v1.22.0/kustomization.yaml` (overlay for the staged `core.yaml` and
`kourier.yaml`; `crds.yaml` stays a separate first apply).
| Deployment | Container | Upstream v1.22.0 | Declared |
|---|---|---|---|
| knative-serving/activator | activator | 300m | 50m |
| kourier-system/3scale-kourier-gateway | kourier-gateway | 200m | 50m |
| knative-serving/net-kourier-controller | controller | 200m | 30m |
| knative-serving/controller | controller | 100m | 30m |
| knative-serving/webhook | webhook | 100m | 30m |
| knative-serving/autoscaler | autoscaler | 100m | 30m |
Only CPU requests differ from upstream. Memory requests and all limits are
upstream's. These values were set live on 2026-09-21 as
`ADMINISTER @ realm:kubernetes/railiance01`, activation=APPROVED by the founder,
because the node had 100% of its allocatable CPU requested and backups could
not be scheduled. Record: `the-custodian/docs/kubernetes-change-gate-decision.md`.
A plain re-apply or upgrade of the upstream manifests restores the upstream
column. Every apply or upgrade must apply these patches afterwards, and an
upgrade to a new version needs a new `substrate/<version>/` with container
names re-checked against that release.
## Lessons
**A request cut cannot roll out on a node whose requests are exhausted.** A
rolling update creates the new pod before the old one stops. When no CPU is
left to request, the new pod stays Pending, so the old pod never stops, even
though the change would free capacity. On 2026-09-21 all six rollouts
deadlocked until 50m was freed elsewhere. Free capacity *before* changing a
Deployment's requests; then the first rollout frees more for the next.
**The HPAs scale on percent of request.** The HPAs on the activator, the
Kourier gateway and the webhook target 100% CPU utilisation *of the request*.
Lowering a request makes the same load read as a higher percentage, so the
HPA scales out sooner (at about 50m or 30m of use, not 300m or 100m). At
2026-09-21 use they read about 2-10%. If the request is raised or lowered
again, check the HPA targets in the same change.
## Read-only drift check
```bash
ssh railiance01 'kubectl get deploy -n knative-serving -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get deploy -n kourier-system -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get hpa -n knative-serving; kubectl get hpa -n kourier-system'
```