Declare the Knative v1.22.0 substrate CPU requests set live on railiance01
Strategic-merge patches and a kustomization over the checksum-pinned upstream assets, a runbook with the rollout-deadlock and HPA lessons, offline tests, and RAIL-KNATIVE-WP-0002 with the railiance-cluster installer handoff as a wait task. Verified read-only against live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 63291@bnt-lap001 Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703
This commit is contained in:
parent
3b2d230923
commit
fd9b848b63
5 changed files with 264 additions and 0 deletions
50
docs/substrate-runbook.md
Normal file
50
docs/substrate-runbook.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
# Knative substrate runbook: declared CPU requests
|
||||
|
||||
The Knative Serving and Kourier v1.22.0 install on railiance01 is performed by
|
||||
`railiance-cluster/install/knative/install.sh` from checksum-pinned upstream
|
||||
assets. rail-knative declares the CPU requests that install must carry, in
|
||||
`substrate/v1.22.0/cpu-requests.patch.yaml` (strategic-merge patches) and
|
||||
`substrate/v1.22.0/kustomization.yaml` (overlay for the staged `core.yaml` and
|
||||
`kourier.yaml`; `crds.yaml` stays a separate first apply).
|
||||
|
||||
| Deployment | Container | Upstream v1.22.0 | Declared |
|
||||
|---|---|---|---|
|
||||
| knative-serving/activator | activator | 300m | 50m |
|
||||
| kourier-system/3scale-kourier-gateway | kourier-gateway | 200m | 50m |
|
||||
| knative-serving/net-kourier-controller | controller | 200m | 30m |
|
||||
| knative-serving/controller | controller | 100m | 30m |
|
||||
| knative-serving/webhook | webhook | 100m | 30m |
|
||||
| knative-serving/autoscaler | autoscaler | 100m | 30m |
|
||||
|
||||
Only CPU requests differ from upstream. Memory requests and all limits are
|
||||
upstream's. These values were set live on 2026-09-21 as
|
||||
`ADMINISTER @ realm:kubernetes/railiance01`, activation=APPROVED by the founder,
|
||||
because the node had 100% of its allocatable CPU requested and backups could
|
||||
not be scheduled. Record: `the-custodian/docs/kubernetes-change-gate-decision.md`.
|
||||
|
||||
A plain re-apply or upgrade of the upstream manifests restores the upstream
|
||||
column. Every apply or upgrade must apply these patches afterwards, and an
|
||||
upgrade to a new version needs a new `substrate/<version>/` with container
|
||||
names re-checked against that release.
|
||||
|
||||
## Lessons
|
||||
|
||||
**A request cut cannot roll out on a node whose requests are exhausted.** A
|
||||
rolling update creates the new pod before the old one stops. When no CPU is
|
||||
left to request, the new pod stays Pending, so the old pod never stops, even
|
||||
though the change would free capacity. On 2026-09-21 all six rollouts
|
||||
deadlocked until 50m was freed elsewhere. Free capacity *before* changing a
|
||||
Deployment's requests; then the first rollout frees more for the next.
|
||||
|
||||
**The HPAs scale on percent of request.** The HPAs on the activator, the
|
||||
Kourier gateway and the webhook target 100% CPU utilisation *of the request*.
|
||||
Lowering a request makes the same load read as a higher percentage, so the
|
||||
HPA scales out sooner (at about 50m or 30m of use, not 300m or 100m). At
|
||||
2026-09-21 use they read about 2-10%. If the request is raised or lowered
|
||||
again, check the HPA targets in the same change.
|
||||
|
||||
## Read-only drift check
|
||||
|
||||
```bash
|
||||
ssh railiance01 'kubectl get deploy -n knative-serving -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get deploy -n kourier-system -o jsonpath="{range .items[*]}{.metadata.name} {.spec.template.spec.containers[0].resources.requests.cpu}{\"\n\"}{end}"; kubectl get hpa -n knative-serving; kubectl get hpa -n kourier-system'
|
||||
```
|
||||
100
substrate/v1.22.0/cpu-requests.patch.yaml
Normal file
100
substrate/v1.22.0/cpu-requests.patch.yaml
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
# Declared CPU requests for the Knative Serving + Kourier v1.22.0 substrate.
|
||||
# Strategic-merge patches over the upstream release manifests (serving-core.yaml,
|
||||
# kourier.yaml), checksum-pinned in railiance-cluster/install/knative/release-lock.env.
|
||||
# Only CPU requests are set; memory requests and all limits stay upstream.
|
||||
# Live on railiance01 since 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01,
|
||||
# activation=APPROVED by the founder). Re-applying upstream without these patches
|
||||
# restores the upstream values noted per document and re-exhausts the node.
|
||||
# Apply after each upstream apply, e.g. for each document:
|
||||
# kubectl patch deployment <name> -n <ns> --type strategic --patch-file <doc>
|
||||
# or list this file under `patches:` in a kustomization over the staged assets.
|
||||
---
|
||||
# upstream v1.22.0 request: 300m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: activator
|
||||
namespace: knative-serving
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: activator
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
---
|
||||
# upstream v1.22.0 request: 100m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: autoscaler
|
||||
namespace: knative-serving
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: autoscaler
|
||||
resources:
|
||||
requests:
|
||||
cpu: 30m
|
||||
---
|
||||
# upstream v1.22.0 request: 100m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: controller
|
||||
namespace: knative-serving
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: controller
|
||||
resources:
|
||||
requests:
|
||||
cpu: 30m
|
||||
---
|
||||
# upstream v1.22.0 request: 100m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: webhook
|
||||
namespace: knative-serving
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: webhook
|
||||
resources:
|
||||
requests:
|
||||
cpu: 30m
|
||||
---
|
||||
# upstream v1.22.0 request: 200m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: net-kourier-controller
|
||||
namespace: knative-serving
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: controller
|
||||
resources:
|
||||
requests:
|
||||
cpu: 30m
|
||||
---
|
||||
# upstream v1.22.0 request: 200m
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: 3scale-kourier-gateway
|
||||
namespace: kourier-system
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: kourier-gateway
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
12
substrate/v1.22.0/kustomization.yaml
Normal file
12
substrate/v1.22.0/kustomization.yaml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# Overlay for the checksum-verified upstream assets staged by
|
||||
# railiance-cluster/install/knative/install.sh (core.yaml, kourier.yaml).
|
||||
# crds.yaml stays a separate first apply: core.yaml repeats one CRD, so kustomize
|
||||
# refuses both in one build. Copy the staged files next to this kustomization,
|
||||
# then `kubectl kustomize .`.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- core.yaml
|
||||
- kourier.yaml
|
||||
patches:
|
||||
- path: cpu-requests.patch.yaml
|
||||
33
tests/test_substrate_requests.py
Normal file
33
tests/test_substrate_requests.py
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
from pathlib import Path
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).parents[1]
|
||||
SUBSTRATE = ROOT / "substrate/v1.22.0"
|
||||
|
||||
# Live on railiance01 since 2026-09-21; see docs/substrate-runbook.md.
|
||||
DECLARED = {
|
||||
("knative-serving", "activator", "activator"): "50m",
|
||||
("knative-serving", "autoscaler", "autoscaler"): "30m",
|
||||
("knative-serving", "controller", "controller"): "30m",
|
||||
("knative-serving", "webhook", "webhook"): "30m",
|
||||
("knative-serving", "net-kourier-controller", "controller"): "30m",
|
||||
("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "50m",
|
||||
}
|
||||
|
||||
|
||||
def test_patches_declare_exactly_the_cpu_requests():
|
||||
found = {}
|
||||
for doc in yaml.safe_load_all((SUBSTRATE / "cpu-requests.patch.yaml").read_text()):
|
||||
assert doc["kind"] == "Deployment"
|
||||
(container,) = doc["spec"]["template"]["spec"]["containers"]
|
||||
# Only CPU requests: no limits, no memory, nothing else.
|
||||
assert container["resources"] == {"requests": {"cpu": container["resources"]["requests"]["cpu"]}}
|
||||
key = (doc["metadata"]["namespace"], doc["metadata"]["name"], container["name"])
|
||||
found[key] = container["resources"]["requests"]["cpu"]
|
||||
assert found == DECLARED
|
||||
|
||||
|
||||
def test_kustomization_patches_the_pinned_assets():
|
||||
k = yaml.safe_load((SUBSTRATE / "kustomization.yaml").read_text())
|
||||
assert k["resources"] == ["core.yaml", "kourier.yaml"]
|
||||
assert k["patches"] == [{"path": "cpu-requests.patch.yaml"}]
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
id: RAIL-KNATIVE-WP-0002
|
||||
type: workplan
|
||||
title: "Declare the Knative substrate CPU requests set live on railiance01"
|
||||
domain: financials
|
||||
repo: rail-knative
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-09-21"
|
||||
updated: "2026-09-21"
|
||||
depends_on: []
|
||||
---
|
||||
|
||||
# RAIL-KNATIVE-WP-0002 - Declare the Knative substrate CPU requests
|
||||
|
||||
On 2026-09-21 the CPU requests of the Serving and Kourier v1.22.0 install on
|
||||
railiance01 were lowered live (`ADMINISTER @ realm:kubernetes/railiance01`,
|
||||
activation=APPROVED by the founder). Limits are unchanged. Record:
|
||||
`the-custodian/docs/kubernetes-change-gate-decision.md`. Runbook:
|
||||
`docs/substrate-runbook.md`.
|
||||
|
||||
## T01 - Declare the requests over the pinned v1.22.0 assets
|
||||
|
||||
```task
|
||||
id: RAIL-KNATIVE-WP-0002-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
2026-09-21: Added `substrate/v1.22.0/cpu-requests.patch.yaml` and a
|
||||
kustomization over the staged `core.yaml` and `kourier.yaml`, with offline
|
||||
tests. Rendered against the checksum-verified upstream assets (hashes from
|
||||
`railiance-cluster/install/knative/release-lock.env`): the six Deployments
|
||||
carry exactly the declared CPU requests, with upstream memory and limits.
|
||||
|
||||
## T02 - Verify the declaration against live, read-only
|
||||
|
||||
```task
|
||||
id: RAIL-KNATIVE-WP-0002-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
2026-09-21: `kubectl get deploy -o jsonpath` over `ssh railiance01` read
|
||||
activator 50m, autoscaler 30m, controller 30m, webhook 30m,
|
||||
net-kourier-controller 30m, 3scale-kourier-gateway 50m; limits 1 (webhook
|
||||
500m), matching upstream. The namespace version label is 1.22.0. HPAs:
|
||||
activator 2%/100%, webhook 10%/100%, gateway 10%/100%. Declared equals live.
|
||||
|
||||
## T03 - Make the railiance-cluster installer apply the declaration
|
||||
|
||||
```task
|
||||
id: RAIL-KNATIVE-WP-0002-T03
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Waits on railiance-cluster, which owns the install (`install/knative/`).
|
||||
rail-knative does not edit that repository. The installer must apply
|
||||
`substrate/v1.22.0/cpu-requests.patch.yaml` after each upstream apply, for
|
||||
example by building the kustomization over its staged files, or by running
|
||||
`kubectl patch deployment <name> -n <ns> --type strategic --patch-file <doc>`
|
||||
per document; `verify.sh` should assert the six requests.
|
||||
|
||||
Because the values are already live, re-running the patched installer changes
|
||||
no running state. Running the *unpatched* installer before this lands reverts
|
||||
the requests and should not be done. Rollback of the installer change is a
|
||||
revert of its commit; the live values need no rollback.
|
||||
Loading…
Add table
Add a link
Reference in a new issue