diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 471a244..fa87332 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -23,6 +23,7 @@ | workplan | RCLUSTER-WP-0017 | active | — | workplans/RCLUSTER-WP-0017-netkingdom-signer-runtime-storage-review.md | | workplan | RCLUSTER-WP-0018 | active | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md | | workplan | RCLUSTER-WP-0019 | active | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | +| workplan | RCLUSTER-WP-0020 | active | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md | | task | RCLUSTER-WP-ADHOC-2026-07-27-T01 | done | — | workplans/ADHOC-2026-07-27.md | | task | RAIL-BS-WP-0008-T01 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md | | task | RAIL-BS-WP-0008-T02 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md | @@ -74,3 +75,6 @@ | task | RCLUSTER-WP-0019-T01 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | | task | RCLUSTER-WP-0019-T02 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | | task | RCLUSTER-WP-0019-T03 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | +| task | RCLUSTER-WP-0020-T01 | done | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md | +| task | RCLUSTER-WP-0020-T02 | wait | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md | +| task | RCLUSTER-WP-0020-T03 | todo | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md | diff --git a/install/knative/README.md b/install/knative/README.md index 87f4228..bc8d147 100644 --- a/install/knative/README.md +++ b/install/knative/README.md @@ -7,13 +7,15 @@ idempotent. It does not apply the upstream manifests as published. `render.sh` (also usable on its own, with no cluster contact) renders them through the kustomize overlays in `overlays/`: CRDs verbatim and first, because `serving-core.yaml` -repeats one of them; then serving-core and kourier with the CPU requests that -rail-knative declares in `substrate/v1.22.0/cpu-requests.patch.yaml` -(activator 50m, autoscaler/controller/webhook/net-kourier-controller 30m, -3scale-kourier-gateway 50m), the Kourier Service as `ClusterIP` and the Envoy -gateway image pinned to `ENVOY_IMAGE`. Memory requests and all limits stay +repeats one of them; then serving-core and kourier with the CPU requests and +HPA ceilings that rail-knative declares in `substrate/v1.22.0/` +(`cpu-requests.patch.yaml`: 10m for activator, autoscaler, controller, webhook, +net-kourier-controller and 3scale-kourier-gateway; `hpa-caps.patch.yaml`: +maxReplicas activator 3, webhook 2, 3scale-kourier-gateway 3; RAIL-KNATIVE-WP-0003), +the Kourier Service as `ClusterIP` and the Envoy +gateway image pinned to `ENVOY_IMAGE`. Memory requests, limits, minReplicas and HPA targets stay upstream. Applying the unpatched upstream files restores 300m/200m/100m and -exhausts railiance01's CPU requests again. `tests/test_knative_render.py` +HPA ceilings 20/5/10, and exhausts railiance01's CPU requests again. `tests/test_knative_render.py` proves the render offline and fails if these overlays drift from rail-knative's declaration. `install.sh` needs a local `kubectl` for `kubectl kustomize`. Kourier is kept `ClusterIP`; public entry through Traefik, DNS, and TLS requires separate reef admission evidence. @@ -24,7 +26,7 @@ containers for fail-closed admission checks such as verifying that egress policy has reconciled before application code starts. Run `install.sh railiance01`, then `verify.sh railiance01`; `verify.sh` is -read-only and also checks the six CPU requests and the Envoy pin. +read-only and also checks the six CPU requests, the three HPA ceilings and the Envoy pin. Before workload admission, rollback deletes Kourier, Serving core, then CRDs using the same verified assets. After Knative Services exist, removal requires diff --git a/install/knative/overlays/kourier/cpu-requests.patch.yaml b/install/knative/overlays/kourier/cpu-requests.patch.yaml index 9245114..2cf1b88 100644 --- a/install/knative/overlays/kourier/cpu-requests.patch.yaml +++ b/install/knative/overlays/kourier/cpu-requests.patch.yaml @@ -1,8 +1,9 @@ # CPU requests declared by rail-knative (substrate/v1.22.0/cpu-requests.patch.yaml), # the documents of that file that target kourier.yaml. Only CPU requests are set; -# memory requests and all limits stay upstream. Live on railiance01 since +# memory requests and all limits stay upstream. First set live on railiance01 # 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by -# the founder). Keep in step with rail-knative: tests/test_knative_render.py +# the founder); trimmed to 10m each by RAIL-KNATIVE-WP-0003 (2026-10-21). +# Keep in step with rail-knative: tests/test_knative_render.py # fails if the two declarations disagree. --- # upstream v1.22.0 request: 200m @@ -18,7 +19,7 @@ spec: - name: controller resources: requests: - cpu: 30m + cpu: 10m --- # upstream v1.22.0 request: 200m apiVersion: apps/v1 @@ -33,4 +34,4 @@ spec: - name: kourier-gateway resources: requests: - cpu: 50m + cpu: 10m diff --git a/install/knative/overlays/kourier/hpa-caps.patch.yaml b/install/knative/overlays/kourier/hpa-caps.patch.yaml new file mode 100644 index 0000000..a1b8c5b --- /dev/null +++ b/install/knative/overlays/kourier/hpa-caps.patch.yaml @@ -0,0 +1,13 @@ +# HPA ceilings declared by rail-knative (substrate/v1.22.0/hpa-caps.patch.yaml), +# the documents of that file that target kourier.yaml. Only maxReplicas changes; +# minReplicas and the CPU target stay upstream (RAIL-KNATIVE-WP-0003). Keep in +# step with rail-knative: tests/test_knative_render.py fails if they disagree. +--- +# upstream v1.22.0: minReplicas 1, maxReplicas 10 +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: 3scale-kourier-gateway + namespace: kourier-system +spec: + maxReplicas: 3 diff --git a/install/knative/overlays/kourier/kustomization.yaml b/install/knative/overlays/kourier/kustomization.yaml index 4a5501c..743124b 100644 --- a/install/knative/overlays/kourier/kustomization.yaml +++ b/install/knative/overlays/kourier/kustomization.yaml @@ -7,4 +7,5 @@ resources: - upstream.yaml patches: - path: cpu-requests.patch.yaml + - path: hpa-caps.patch.yaml - path: service-clusterip.patch.yaml diff --git a/install/knative/overlays/serving-core/cpu-requests.patch.yaml b/install/knative/overlays/serving-core/cpu-requests.patch.yaml index 8676ab8..2d38376 100644 --- a/install/knative/overlays/serving-core/cpu-requests.patch.yaml +++ b/install/knative/overlays/serving-core/cpu-requests.patch.yaml @@ -1,8 +1,9 @@ # CPU requests declared by rail-knative (substrate/v1.22.0/cpu-requests.patch.yaml), # the documents of that file that target serving-core.yaml. Only CPU requests are set; -# memory requests and all limits stay upstream. Live on railiance01 since +# memory requests and all limits stay upstream. First set live on railiance01 # 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by -# the founder). Keep in step with rail-knative: tests/test_knative_render.py +# the founder); trimmed to 10m each by RAIL-KNATIVE-WP-0003 (2026-10-21). +# Keep in step with rail-knative: tests/test_knative_render.py # fails if the two declarations disagree. --- # upstream v1.22.0 request: 300m @@ -18,7 +19,7 @@ spec: - name: activator resources: requests: - cpu: 50m + cpu: 10m --- # upstream v1.22.0 request: 100m apiVersion: apps/v1 @@ -33,7 +34,7 @@ spec: - name: autoscaler resources: requests: - cpu: 30m + cpu: 10m --- # upstream v1.22.0 request: 100m apiVersion: apps/v1 @@ -48,7 +49,7 @@ spec: - name: controller resources: requests: - cpu: 30m + cpu: 10m --- # upstream v1.22.0 request: 100m apiVersion: apps/v1 @@ -63,4 +64,4 @@ spec: - name: webhook resources: requests: - cpu: 30m + cpu: 10m diff --git a/install/knative/overlays/serving-core/hpa-caps.patch.yaml b/install/knative/overlays/serving-core/hpa-caps.patch.yaml new file mode 100644 index 0000000..877b11a --- /dev/null +++ b/install/knative/overlays/serving-core/hpa-caps.patch.yaml @@ -0,0 +1,22 @@ +# HPA ceilings declared by rail-knative (substrate/v1.22.0/hpa-caps.patch.yaml), +# the documents of that file that target serving-core.yaml. Only maxReplicas changes; +# minReplicas and the CPU target stay upstream (RAIL-KNATIVE-WP-0003). Keep in +# step with rail-knative: tests/test_knative_render.py fails if they disagree. +--- +# upstream v1.22.0: minReplicas 1, maxReplicas 20 +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: activator + namespace: knative-serving +spec: + maxReplicas: 3 +--- +# upstream v1.22.0: minReplicas 1, maxReplicas 5 +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: webhook + namespace: knative-serving +spec: + maxReplicas: 2 diff --git a/install/knative/overlays/serving-core/kustomization.yaml b/install/knative/overlays/serving-core/kustomization.yaml index 6e1be69..3bb91b1 100644 --- a/install/knative/overlays/serving-core/kustomization.yaml +++ b/install/knative/overlays/serving-core/kustomization.yaml @@ -7,3 +7,4 @@ resources: - upstream.yaml patches: - path: cpu-requests.patch.yaml + - path: hpa-caps.patch.yaml diff --git a/install/knative/verify.sh b/install/knative/verify.sh index 49b0786..a6195f9 100755 --- a/install/knative/verify.sh +++ b/install/knative/verify.sh @@ -18,11 +18,20 @@ while read -r ns deploy container cpu; do -o "jsonpath={.spec.template.spec.containers[?(@.name==\"$container\")].resources.requests.cpu}")" [ "$got" = "$cpu" ] || { echo "cpu request $ns/$deploy/$container: want $cpu, got ${got:-}" >&2; exit 1; } done <<'LIST' -knative-serving activator activator 50m -knative-serving autoscaler autoscaler 30m -knative-serving controller controller 30m -knative-serving webhook webhook 30m -knative-serving net-kourier-controller controller 30m -kourier-system 3scale-kourier-gateway kourier-gateway 50m +knative-serving activator activator 10m +knative-serving autoscaler autoscaler 10m +knative-serving controller controller 10m +knative-serving webhook webhook 10m +knative-serving net-kourier-controller controller 10m +kourier-system 3scale-kourier-gateway kourier-gateway 10m LIST +# Declared HPA ceilings (rail-knative substrate/v1.22.0/hpa-caps.patch.yaml). +while read -r ns hpa max; do + got="$(ssh "$target" kubectl get hpa "$hpa" -n "$ns" -o 'jsonpath={.spec.maxReplicas}')" + [ "$got" = "$max" ] || { echo "hpa maxReplicas $ns/$hpa: want $max, got ${got:-}" >&2; exit 1; } +done <<'HPAS' +knative-serving activator 3 +knative-serving webhook 2 +kourier-system 3scale-kourier-gateway 3 +HPAS echo "knative substrate verified" diff --git a/tests/test_knative_render.py b/tests/test_knative_render.py index fc295bd..88f4ef5 100644 --- a/tests/test_knative_render.py +++ b/tests/test_knative_render.py @@ -1,8 +1,8 @@ -"""Offline proof that install/knative renders the declared CPU requests. +"""Offline proof that install/knative renders the declared CPU requests and HPA caps. Runs install/knative/render.sh (download + SHA-256 verification + kustomize; no cluster contact) and compares the result with the upstream assets: the six -declared CPU requests change, the Kourier Service is ClusterIP, the gateway +declared CPU requests and HPA maxReplicas change, the Kourier Service is ClusterIP, the gateway image is the pinned Envoy digest, and nothing else differs. Skips when the upstream assets cannot be fetched or kubectl is missing. """ @@ -27,12 +27,17 @@ ROOT = Path(__file__).resolve().parents[1] KNATIVE = ROOT / "install" / "knative" RAIL_KNATIVE = Path(os.environ.get("RAIL_KNATIVE_DIR", ROOT.parent / "rail-knative")) DECLARED = { # (namespace, deployment, container) -> cpu request - ("knative-serving", "activator", "activator"): "50m", - ("knative-serving", "autoscaler", "autoscaler"): "30m", - ("knative-serving", "controller", "controller"): "30m", - ("knative-serving", "webhook", "webhook"): "30m", - ("knative-serving", "net-kourier-controller", "controller"): "30m", - ("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "50m", + ("knative-serving", "activator", "activator"): "10m", + ("knative-serving", "autoscaler", "autoscaler"): "10m", + ("knative-serving", "controller", "controller"): "10m", + ("knative-serving", "webhook", "webhook"): "10m", + ("knative-serving", "net-kourier-controller", "controller"): "10m", + ("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "10m", +} +DECLARED_HPA = { # (namespace, hpa) -> maxReplicas + ("knative-serving", "activator"): 3, + ("knative-serving", "webhook"): 2, + ("kourier-system", "3scale-kourier-gateway"): 3, } @@ -58,6 +63,10 @@ def patch_requests(path: Path) -> dict[tuple, str]: return out +def patch_hpa_caps(path: Path) -> dict[tuple, int]: + return {(d["metadata"]["namespace"], d["metadata"]["name"]): d["spec"]["maxReplicas"] for d in load(path)} + + @unittest.skipIf(yaml is None, "PyYAML not installed") class DeclarationTests(unittest.TestCase): def test_overlays_carry_the_declared_requests(self) -> None: @@ -66,16 +75,28 @@ class DeclarationTests(unittest.TestCase): got.update(patch_requests(KNATIVE / "overlays" / overlay / "cpu-requests.patch.yaml")) self.assertEqual(DECLARED, got) + def test_overlays_carry_the_declared_hpa_caps(self) -> None: + got = {} + for overlay in ("serving-core", "kourier"): + got.update(patch_hpa_caps(KNATIVE / "overlays" / overlay / "hpa-caps.patch.yaml")) + self.assertEqual(DECLARED_HPA, got) + def test_verify_checks_the_declared_requests(self) -> None: rows = re.search(r"<<'LIST'\n(.*?)\nLIST", (KNATIVE / "verify.sh").read_text(), re.S).group(1) got = {tuple(r.split()[:3]): r.split()[3] for r in rows.splitlines()} self.assertEqual(DECLARED, got) + def test_verify_checks_the_declared_hpa_caps(self) -> None: + rows = re.search(r"<<'HPAS'\n(.*?)\nHPAS", (KNATIVE / "verify.sh").read_text(), re.S).group(1) + got = {tuple(r.split()[:2]): int(r.split()[2]) for r in rows.splitlines()} + self.assertEqual(DECLARED_HPA, got) + def test_matches_rail_knative_declaration(self) -> None: source = RAIL_KNATIVE / "substrate" / "v1.22.0" / "cpu-requests.patch.yaml" if not source.exists(): self.skipTest(f"rail-knative not checked out at {RAIL_KNATIVE}") self.assertEqual(DECLARED, patch_requests(source)) + self.assertEqual(DECLARED_HPA, patch_hpa_caps(source.with_name("hpa-caps.patch.yaml"))) @unittest.skipIf(yaml is None or shutil.which("kubectl") is None, "needs PyYAML and kubectl") @@ -110,6 +131,9 @@ class RenderTests(unittest.TestCase): seen.add(ident) if c["image"].startswith("docker.io/envoyproxy/envoy:"): c["image"] = envoy + if k[1] == "HorizontalPodAutoscaler" and k[2:] in DECLARED_HPA: + expected["spec"]["maxReplicas"] = DECLARED_HPA[k[2:]] + seen.add(k[2:]) if k[1:] == ("Service", "kourier-system", "kourier"): expected["spec"]["type"] = "ClusterIP" self.assertEqual(expected, out, f"{k} differs beyond the declared changes") @@ -124,7 +148,7 @@ class RenderTests(unittest.TestCase): def test_rendered_carries_declared_requests_and_nothing_else_changes(self) -> None: seen = self.check("core.yaml", "serving-core.rendered.yaml") seen |= self.check("kourier.yaml", "kourier.rendered.yaml") - self.assertEqual(set(DECLARED), seen) + self.assertEqual(set(DECLARED) | set(DECLARED_HPA), seen) if __name__ == "__main__": diff --git a/workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md b/workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md new file mode 100644 index 0000000..c9bff05 --- /dev/null +++ b/workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md @@ -0,0 +1,92 @@ +--- +id: RCLUSTER-WP-0020 +type: workplan +title: "Receive the 2026-10-21 node budget: Knative HPA caps/requests and wave-2 allocation" +domain: financials +repo: railiance-cluster +status: active +owner: claude +topic_slug: railiance +created: "2026-10-21" +updated: "2026-10-21" +flavor: implementation +depends_on: [] +related: [CUST-WP-0071, RESOURCE-WP-0008, RAIL-KNATIVE-WP-0003] +--- + +# RCLUSTER-WP-0020 - Receive the 2026-10-21 node budget + +resource-control's node allocation review of 2026-10-21 +(`resource-control/docs/evidence/2026-10-21-node-allocation-review.md`, per-workload +table in `.../2026-10-21-node-allocation-review/2026-10-21-node-budget.md`) +reconciles railiance01 only conditionally: lean decreases accepted, releases, +builds and Knative scale-out serialized, and Knative HPA ceilings capped. +This plan receives the railiance-cluster parts. + +## T01 - Installer renders rail-knative's RAIL-KNATIVE-WP-0003 declaration + +```task +id: RCLUSTER-WP-0020-T01 +status: done +priority: high +``` + +rail-knative `76cb9c7` declares 10m CPU requests for the six substrate +containers (was 50/30/30/30/30/50m) and HPA `maxReplicas` activator 20→3, +webhook 5→2, 3scale-kourier-gateway 10→3. `install/knative/overlays/` now +carries both (`cpu-requests.patch.yaml`, new `hpa-caps.patch.yaml`, split by +upstream asset), `verify.sh` checks the six requests and the three ceilings, +and `tests/test_knative_render.py` asserts the overlays, verify.sh and the +rail-knative checkout agree and that the render differs from upstream only in +the declared fields. A read-only render-vs-live comparison showed exactly the +9 intended fields differing. Not applied. + +## T02 - Attended apply and verify on railiance01 + +```task +id: RCLUSTER-WP-0020-T02 +status: wait +priority: high +needs_human: true +depends_on: [RCLUSTER-WP-0020-T01] +blocking_reason: "Coordinator applies wave-1 changes one owner at a time after review (RAIL-KNATIVE-WP-0003-T02)" +``` + +`install/knative/install.sh railiance01`, then `install/knative/verify.sh railiance01`. + +## T03 - Wave-2 allocation for railiance-cluster workloads (prepare only) + +```task +id: RCLUSTER-WP-0020-T03 +status: todo +priority: medium +``` + +Receiving task for the seven railiance-cluster rows of the node budget. +Requests only; limits unchanged. Locate the source of each (k3s packaged +manifests / HelmChartConfig under `install/`, cert-manager install) before +changing anything; k3s-packaged components need a HelmChartConfig or +`--disable` + owned manifest so the change survives k3s restarts. + +| Workload | CPU req→prop (m) | Mem req→prop (Mi) | p95/peak CPU 7d (m) | Action | +| --- | ---: | ---: | ---: | --- | +| cert-manager/cainjector | 0→10 | 0→96 | 1.1/2.8 | increase + memory up | +| cert-manager/cert-manager | 0→10 | 0→80 | 1.4/3.5 | increase + memory up | +| cert-manager/webhook | 0→10 | 0→0 | 0.4/0.9 | increase | +| kube-system/coredns | 100→50 | 70→70 | 7.8/10.3 | decrease (DNS floor 50m) | +| kube-system/local-path-provisioner | 0→10 | 0→0 | 0.1/0.3 | increase | +| kube-system/metrics-server | 100→10 | 70→70 | 4.9/5.1 | decrease | +| kube-system/traefik | 0→10 | 0→80 | 3.8/16.0 | increase + memory up | + +Measurement-only items in the same task, no manifest changes yet: + +- Produce a 7-day host/control-plane series (non-pod cgroup CPU/memory); the + 900m host figure is a 70-second kubelet sample. Coordinate with rapp-telemetry + (node-exporter is disabled, Prometheus has no non-pod series). +- `kube-reserved`/`system-reserved` (~900m/2.5Gi) are set **only after** the + wave-2 request decreases across owners have landed; applying them first makes + the current requests unschedulable. + +Done when the seven request changes are declared in this repo with a render or +diff proof, applied attended, and the host series plus the reserved-resources +decision are recorded.