From fd9b848b632e5c9b565de7d17935246a104662bb Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 21 Sep 2026 18:40:08 +0200 Subject: [PATCH] Declare the Knative v1.22.0 substrate CPU requests set live on railiance01 Strategic-merge patches and a kustomization over the checksum-pinned upstream assets, a runbook with the rollout-deadlock and HPA lessons, offline tests, and RAIL-KNATIVE-WP-0002 with the railiance-cluster installer handoff as a wait task. Verified read-only against live. Co-Authored-By: Claude Opus 5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 63291@bnt-lap001 Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703 --- docs/substrate-runbook.md | 50 +++++++++ substrate/v1.22.0/cpu-requests.patch.yaml | 100 ++++++++++++++++++ substrate/v1.22.0/kustomization.yaml | 12 +++ tests/test_substrate_requests.py | 33 ++++++ ...-WP-0002-declare-substrate-cpu-requests.md | 69 ++++++++++++ 5 files changed, 264 insertions(+) create mode 100644 docs/substrate-runbook.md create mode 100644 substrate/v1.22.0/cpu-requests.patch.yaml create mode 100644 substrate/v1.22.0/kustomization.yaml create mode 100644 tests/test_substrate_requests.py create mode 100644 workplans/RAIL-KNATIVE-WP-0002-declare-substrate-cpu-requests.md diff --git a/docs/substrate-runbook.md b/docs/substrate-runbook.md new file mode 100644 index 0000000..82609e8 --- /dev/null +++ b/docs/substrate-runbook.md @@ -0,0 +1,50 @@ +# Knative substrate runbook: declared CPU requests + +The Knative Serving and Kourier v1.22.0 install on railiance01 is performed by +`railiance-cluster/install/knative/install.sh` from checksum-pinned upstream +assets. rail-knative declares the CPU requests that install must carry, in +`substrate/v1.22.0/cpu-requests.patch.yaml` (strategic-merge patches) and +`substrate/v1.22.0/kustomization.yaml` (overlay for the staged `core.yaml` and +`kourier.yaml`; `crds.yaml` stays a separate first apply). + +| Deployment | Container | Upstream v1.22.0 | Declared | +|---|---|---|---| +| knative-serving/activator | activator | 300m | 50m | +| kourier-system/3scale-kourier-gateway | kourier-gateway | 200m | 50m | +| knative-serving/net-kourier-controller | controller | 200m | 30m | +| knative-serving/controller | controller | 100m | 30m | +| knative-serving/webhook | webhook | 100m | 30m | +| knative-serving/autoscaler | autoscaler | 100m | 30m | + +Only CPU requests differ from upstream. Memory requests and all limits are +upstream's. These values were set live on 2026-09-21 as +`ADMINISTER @ realm:kubernetes/railiance01`, activation=APPROVED by the founder, +because the node had 100% of its allocatable CPU requested and backups could +not be scheduled. Record: `the-custodian/docs/kubernetes-change-gate-decision.md`. + +A plain re-apply or upgrade of the upstream manifests restores the upstream +column. Every apply or upgrade must apply these patches afterwards, and an +upgrade to a new version needs a new `substrate//` with container +names re-checked against that release. + +## Lessons + +**A request cut cannot roll out on a node whose requests are exhausted.** A +rolling update creates the new pod before the old one stops. When no CPU is +left to request, the new pod stays Pending, so the old pod never stops, even +though the change would free capacity. On 2026-09-21 all six rollouts +deadlocked until 50m was freed elsewhere. Free capacity *before* changing a +Deployment's requests; then the first rollout frees more for the next. + +**The HPAs scale on percent of request.** The HPAs on the activator, the +Kourier gateway and the webhook target 100% CPU utilisation *of the request*. +Lowering a request makes the same load read as a higher percentage, so the +HPA scales out sooner (at about 50m or 30m of use, not 300m or 100m). At +2026-09-21 use they read about 2-10%. If the request is raised or lowered +again, check the HPA targets in the same change. + +## Read-only drift check + +```bash +ssh railiance01 'kubectl get deploy -n knative-serving -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get deploy -n kourier-system -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get hpa -n knative-serving; kubectl get hpa -n kourier-system' +``` diff --git a/substrate/v1.22.0/cpu-requests.patch.yaml b/substrate/v1.22.0/cpu-requests.patch.yaml new file mode 100644 index 0000000..83a8295 --- /dev/null +++ b/substrate/v1.22.0/cpu-requests.patch.yaml @@ -0,0 +1,100 @@ +# Declared CPU requests for the Knative Serving + Kourier v1.22.0 substrate. +# Strategic-merge patches over the upstream release manifests (serving-core.yaml, +# kourier.yaml), checksum-pinned in railiance-cluster/install/knative/release-lock.env. +# Only CPU requests are set; memory requests and all limits stay upstream. +# Live on railiance01 since 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01, +# activation=APPROVED by the founder). Re-applying upstream without these patches +# restores the upstream values noted per document and re-exhausts the node. +# Apply after each upstream apply, e.g. for each document: +# kubectl patch deployment -n --type strategic --patch-file +# or list this file under `patches:` in a kustomization over the staged assets. +--- +# upstream v1.22.0 request: 300m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: activator + namespace: knative-serving +spec: + template: + spec: + containers: + - name: activator + resources: + requests: + cpu: 50m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: autoscaler + namespace: knative-serving +spec: + template: + spec: + containers: + - name: autoscaler + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller + namespace: knative-serving +spec: + template: + spec: + containers: + - name: controller + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: webhook + namespace: knative-serving +spec: + template: + spec: + containers: + - name: webhook + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 200m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: net-kourier-controller + namespace: knative-serving +spec: + template: + spec: + containers: + - name: controller + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 200m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 3scale-kourier-gateway + namespace: kourier-system +spec: + template: + spec: + containers: + - name: kourier-gateway + resources: + requests: + cpu: 50m diff --git a/substrate/v1.22.0/kustomization.yaml b/substrate/v1.22.0/kustomization.yaml new file mode 100644 index 0000000..44ffd58 --- /dev/null +++ b/substrate/v1.22.0/kustomization.yaml @@ -0,0 +1,12 @@ +# Overlay for the checksum-verified upstream assets staged by +# railiance-cluster/install/knative/install.sh (core.yaml, kourier.yaml). +# crds.yaml stays a separate first apply: core.yaml repeats one CRD, so kustomize +# refuses both in one build. Copy the staged files next to this kustomization, +# then `kubectl kustomize .`. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - core.yaml + - kourier.yaml +patches: + - path: cpu-requests.patch.yaml diff --git a/tests/test_substrate_requests.py b/tests/test_substrate_requests.py new file mode 100644 index 0000000..cc128a6 --- /dev/null +++ b/tests/test_substrate_requests.py @@ -0,0 +1,33 @@ +from pathlib import Path +import yaml + +ROOT = Path(__file__).parents[1] +SUBSTRATE = ROOT / "substrate/v1.22.0" + +# Live on railiance01 since 2026-09-21; see docs/substrate-runbook.md. +DECLARED = { + ("knative-serving", "activator", "activator"): "50m", + ("knative-serving", "autoscaler", "autoscaler"): "30m", + ("knative-serving", "controller", "controller"): "30m", + ("knative-serving", "webhook", "webhook"): "30m", + ("knative-serving", "net-kourier-controller", "controller"): "30m", + ("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "50m", +} + + +def test_patches_declare_exactly_the_cpu_requests(): + found = {} + for doc in yaml.safe_load_all((SUBSTRATE / "cpu-requests.patch.yaml").read_text()): + assert doc["kind"] == "Deployment" + (container,) = doc["spec"]["template"]["spec"]["containers"] + # Only CPU requests: no limits, no memory, nothing else. + assert container["resources"] == {"requests": {"cpu": container["resources"]["requests"]["cpu"]}} + key = (doc["metadata"]["namespace"], doc["metadata"]["name"], container["name"]) + found[key] = container["resources"]["requests"]["cpu"] + assert found == DECLARED + + +def test_kustomization_patches_the_pinned_assets(): + k = yaml.safe_load((SUBSTRATE / "kustomization.yaml").read_text()) + assert k["resources"] == ["core.yaml", "kourier.yaml"] + assert k["patches"] == [{"path": "cpu-requests.patch.yaml"}] diff --git a/workplans/RAIL-KNATIVE-WP-0002-declare-substrate-cpu-requests.md b/workplans/RAIL-KNATIVE-WP-0002-declare-substrate-cpu-requests.md new file mode 100644 index 0000000..828bc3c --- /dev/null +++ b/workplans/RAIL-KNATIVE-WP-0002-declare-substrate-cpu-requests.md @@ -0,0 +1,69 @@ +--- +id: RAIL-KNATIVE-WP-0002 +type: workplan +title: "Declare the Knative substrate CPU requests set live on railiance01" +domain: financials +repo: rail-knative +status: active +owner: codex +topic_slug: railiance +created: "2026-09-21" +updated: "2026-09-21" +depends_on: [] +--- + +# RAIL-KNATIVE-WP-0002 - Declare the Knative substrate CPU requests + +On 2026-09-21 the CPU requests of the Serving and Kourier v1.22.0 install on +railiance01 were lowered live (`ADMINISTER @ realm:kubernetes/railiance01`, +activation=APPROVED by the founder). Limits are unchanged. Record: +`the-custodian/docs/kubernetes-change-gate-decision.md`. Runbook: +`docs/substrate-runbook.md`. + +## T01 - Declare the requests over the pinned v1.22.0 assets + +```task +id: RAIL-KNATIVE-WP-0002-T01 +status: done +priority: high +``` + +2026-09-21: Added `substrate/v1.22.0/cpu-requests.patch.yaml` and a +kustomization over the staged `core.yaml` and `kourier.yaml`, with offline +tests. Rendered against the checksum-verified upstream assets (hashes from +`railiance-cluster/install/knative/release-lock.env`): the six Deployments +carry exactly the declared CPU requests, with upstream memory and limits. + +## T02 - Verify the declaration against live, read-only + +```task +id: RAIL-KNATIVE-WP-0002-T02 +status: done +priority: high +``` + +2026-09-21: `kubectl get deploy -o jsonpath` over `ssh railiance01` read +activator 50m, autoscaler 30m, controller 30m, webhook 30m, +net-kourier-controller 30m, 3scale-kourier-gateway 50m; limits 1 (webhook +500m), matching upstream. The namespace version label is 1.22.0. HPAs: +activator 2%/100%, webhook 10%/100%, gateway 10%/100%. Declared equals live. + +## T03 - Make the railiance-cluster installer apply the declaration + +```task +id: RAIL-KNATIVE-WP-0002-T03 +status: wait +priority: high +``` + +Waits on railiance-cluster, which owns the install (`install/knative/`). +rail-knative does not edit that repository. The installer must apply +`substrate/v1.22.0/cpu-requests.patch.yaml` after each upstream apply, for +example by building the kustomization over its staged files, or by running +`kubectl patch deployment -n --type strategic --patch-file ` +per document; `verify.sh` should assert the six requests. + +Because the values are already live, re-running the patched installer changes +no running state. Running the *unpatched* installer before this lands reverts +the requests and should not be done. Rollback of the installer change is a +revert of its commit; the live values need no rollback.