Declare the Knative v1.22.0 substrate CPU requests set live on railiance01

Strategic-merge patches and a kustomization over the checksum-pinned
upstream assets, a runbook with the rollout-deadlock and HPA lessons,
offline tests, and RAIL-KNATIVE-WP-0002 with the railiance-cluster
installer handoff as a wait task. Verified read-only against live.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 63291@bnt-lap001
Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703

docs/substrate-runbook.md

@@ -0,0 +1,50 @@
+# Knative substrate runbook: declared CPU requests
+
+The Knative Serving and Kourier v1.22.0 install on railiance01 is performed by
+`railiance-cluster/install/knative/install.sh` from checksum-pinned upstream
+assets. rail-knative declares the CPU requests that install must carry, in
+`substrate/v1.22.0/cpu-requests.patch.yaml` (strategic-merge patches) and
+`substrate/v1.22.0/kustomization.yaml` (overlay for the staged `core.yaml` and
+`kourier.yaml`; `crds.yaml` stays a separate first apply).
+
+| Deployment | Container | Upstream v1.22.0 | Declared |
+|---|---|---|---|
+| knative-serving/activator | activator | 300m | 50m |
+| kourier-system/3scale-kourier-gateway | kourier-gateway | 200m | 50m |
+| knative-serving/net-kourier-controller | controller | 200m | 30m |
+| knative-serving/controller | controller | 100m | 30m |
+| knative-serving/webhook | webhook | 100m | 30m |
+| knative-serving/autoscaler | autoscaler | 100m | 30m |
+
+Only CPU requests differ from upstream. Memory requests and all limits are
+upstream's. These values were set live on 2026-09-21 as
+`ADMINISTER @ realm:kubernetes/railiance01`, activation=APPROVED by the founder,
+because the node had 100% of its allocatable CPU requested and backups could
+not be scheduled. Record: `the-custodian/docs/kubernetes-change-gate-decision.md`.
+
+A plain re-apply or upgrade of the upstream manifests restores the upstream
+column. Every apply or upgrade must apply these patches afterwards, and an
+upgrade to a new version needs a new `substrate/<version>/` with container
+names re-checked against that release.
+
+## Lessons
+
+**A request cut cannot roll out on a node whose requests are exhausted.** A
+rolling update creates the new pod before the old one stops. When no CPU is
+left to request, the new pod stays Pending, so the old pod never stops, even
+though the change would free capacity. On 2026-09-21 all six rollouts
+deadlocked until 50m was freed elsewhere. Free capacity *before* changing a
+Deployment's requests; then the first rollout frees more for the next.
+
+**The HPAs scale on percent of request.** The HPAs on the activator, the
+Kourier gateway and the webhook target 100% CPU utilisation *of the request*.
+Lowering a request makes the same load read as a higher percentage, so the
+HPA scales out sooner (at about 50m or 30m of use, not 300m or 100m). At
+2026-09-21 use they read about 2-10%. If the request is raised or lowered
+again, check the HPA targets in the same change.
+
+## Read-only drift check
+
+```bash
+ssh railiance01 'kubectl get deploy -n knative-serving -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get deploy -n kourier-system -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get hpa -n knative-serving; kubectl get hpa -n kourier-system'
+```

substrate/v1.22.0/cpu-requests.patch.yaml

@@ -0,0 +1,100 @@
+# Declared CPU requests for the Knative Serving + Kourier v1.22.0 substrate.
+# Strategic-merge patches over the upstream release manifests (serving-core.yaml,
+# kourier.yaml), checksum-pinned in railiance-cluster/install/knative/release-lock.env.
+# Only CPU requests are set; memory requests and all limits stay upstream.
+# Live on railiance01 since 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01,
+# activation=APPROVED by the founder). Re-applying upstream without these patches
+# restores the upstream values noted per document and re-exhausts the node.
+# Apply after each upstream apply, e.g. for each document:
+#   kubectl patch deployment <name> -n <ns> --type strategic --patch-file <doc>
+# or list this file under `patches:` in a kustomization over the staged assets.
+---
+# upstream v1.22.0 request: 300m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: activator
+  namespace: knative-serving
+spec:
+  template:
+    spec:
+      containers:
+        - name: activator
+          resources:
+            requests:
+              cpu: 50m
+---
+# upstream v1.22.0 request: 100m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: autoscaler
+  namespace: knative-serving
+spec:
+  template:
+    spec:
+      containers:
+        - name: autoscaler
+          resources:
+            requests:
+              cpu: 30m
+---
+# upstream v1.22.0 request: 100m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller
+  namespace: knative-serving
+spec:
+  template:
+    spec:
+      containers:
+        - name: controller
+          resources:
+            requests:
+              cpu: 30m
+---
+# upstream v1.22.0 request: 100m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: webhook
+  namespace: knative-serving
+spec:
+  template:
+    spec:
+      containers:
+        - name: webhook
+          resources:
+            requests:
+              cpu: 30m
+---
+# upstream v1.22.0 request: 200m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: net-kourier-controller
+  namespace: knative-serving
+spec:
+  template:
+    spec:
+      containers:
+        - name: controller
+          resources:
+            requests:
+              cpu: 30m
+---
+# upstream v1.22.0 request: 200m
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: 3scale-kourier-gateway
+  namespace: kourier-system
+spec:
+  template:
+    spec:
+      containers:
+        - name: kourier-gateway
+          resources:
+            requests:
+              cpu: 50m

substrate/v1.22.0/kustomization.yaml

@@ -0,0 +1,12 @@
+# Overlay for the checksum-verified upstream assets staged by
+# railiance-cluster/install/knative/install.sh (core.yaml, kourier.yaml).
+# crds.yaml stays a separate first apply: core.yaml repeats one CRD, so kustomize
+# refuses both in one build. Copy the staged files next to this kustomization,
+# then `kubectl kustomize .`.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - core.yaml
+  - kourier.yaml
+patches:
+  - path: cpu-requests.patch.yaml

tests/test_substrate_requests.py

@@ -0,0 +1,33 @@
+from pathlib import Path
+import yaml
+
+ROOT = Path(__file__).parents[1]
+SUBSTRATE = ROOT / "substrate/v1.22.0"
+
+# Live on railiance01 since 2026-09-21; see docs/substrate-runbook.md.
+DECLARED = {
+    ("knative-serving", "activator", "activator"): "50m",
+    ("knative-serving", "autoscaler", "autoscaler"): "30m",
+    ("knative-serving", "controller", "controller"): "30m",
+    ("knative-serving", "webhook", "webhook"): "30m",
+    ("knative-serving", "net-kourier-controller", "controller"): "30m",
+    ("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "50m",
+}
+
+
+def test_patches_declare_exactly_the_cpu_requests():
+    found = {}
+    for doc in yaml.safe_load_all((SUBSTRATE / "cpu-requests.patch.yaml").read_text()):
+        assert doc["kind"] == "Deployment"
+        (container,) = doc["spec"]["template"]["spec"]["containers"]
+        # Only CPU requests: no limits, no memory, nothing else.
+        assert container["resources"] == {"requests": {"cpu": container["resources"]["requests"]["cpu"]}}
+        key = (doc["metadata"]["namespace"], doc["metadata"]["name"], container["name"])
+        found[key] = container["resources"]["requests"]["cpu"]
+    assert found == DECLARED
+
+
+def test_kustomization_patches_the_pinned_assets():
+    k = yaml.safe_load((SUBSTRATE / "kustomization.yaml").read_text())
+    assert k["resources"] == ["core.yaml", "kourier.yaml"]
+    assert k["patches"] == [{"path": "cpu-requests.patch.yaml"}]

workplans/RAIL-KNATIVE-WP-0002-declare-substrate-cpu-requests.md

@@ -0,0 +1,69 @@
+---
+id: RAIL-KNATIVE-WP-0002
+type: workplan
+title: "Declare the Knative substrate CPU requests set live on railiance01"
+domain: financials
+repo: rail-knative
+status: active
+owner: codex
+topic_slug: railiance
+created: "2026-09-21"
+updated: "2026-09-21"
+depends_on: []
+---
+
+# RAIL-KNATIVE-WP-0002 - Declare the Knative substrate CPU requests
+
+On 2026-09-21 the CPU requests of the Serving and Kourier v1.22.0 install on
+railiance01 were lowered live (`ADMINISTER @ realm:kubernetes/railiance01`,
+activation=APPROVED by the founder). Limits are unchanged. Record:
+`the-custodian/docs/kubernetes-change-gate-decision.md`. Runbook:
+`docs/substrate-runbook.md`.
+
+## T01 - Declare the requests over the pinned v1.22.0 assets
+
+```task
+id: RAIL-KNATIVE-WP-0002-T01
+status: done
+priority: high
+```
+
+2026-09-21: Added `substrate/v1.22.0/cpu-requests.patch.yaml` and a
+kustomization over the staged `core.yaml` and `kourier.yaml`, with offline
+tests. Rendered against the checksum-verified upstream assets (hashes from
+`railiance-cluster/install/knative/release-lock.env`): the six Deployments
+carry exactly the declared CPU requests, with upstream memory and limits.
+
+## T02 - Verify the declaration against live, read-only
+
+```task
+id: RAIL-KNATIVE-WP-0002-T02
+status: done
+priority: high
+```
+
+2026-09-21: `kubectl get deploy -o jsonpath` over `ssh railiance01` read
+activator 50m, autoscaler 30m, controller 30m, webhook 30m,
+net-kourier-controller 30m, 3scale-kourier-gateway 50m; limits 1 (webhook
+500m), matching upstream. The namespace version label is 1.22.0. HPAs:
+activator 2%/100%, webhook 10%/100%, gateway 10%/100%. Declared equals live.
+
+## T03 - Make the railiance-cluster installer apply the declaration
+
+```task
+id: RAIL-KNATIVE-WP-0002-T03
+status: wait
+priority: high
+```
+
+Waits on railiance-cluster, which owns the install (`install/knative/`).
+rail-knative does not edit that repository. The installer must apply
+`substrate/v1.22.0/cpu-requests.patch.yaml` after each upstream apply, for
+example by building the kustomization over its staged files, or by running
+`kubectl patch deployment <name> -n <ns> --type strategic --patch-file <doc>`
+per document; `verify.sh` should assert the six requests.
+
+Because the values are already live, re-running the patched installer changes
+no running state. Running the *unpatched* installer before this lands reverts
+the requests and should not be done. Rollback of the installer change is a
+revert of its commit; the live values need no rollback.