diff --git a/install/knative/README.md b/install/knative/README.md index 898ae3b..87f4228 100644 --- a/install/knative/README.md +++ b/install/knative/README.md @@ -2,7 +2,20 @@ `install.sh` verifies repository-pinned SHA-256 checksums for the upstream Serving and Kourier v1.22.0 assets before applying them over SSH. It is -idempotent. Kourier is kept `ClusterIP`; public entry through Traefik, DNS, and +idempotent. + +It does not apply the upstream manifests as published. `render.sh` (also +usable on its own, with no cluster contact) renders them through the kustomize +overlays in `overlays/`: CRDs verbatim and first, because `serving-core.yaml` +repeats one of them; then serving-core and kourier with the CPU requests that +rail-knative declares in `substrate/v1.22.0/cpu-requests.patch.yaml` +(activator 50m, autoscaler/controller/webhook/net-kourier-controller 30m, +3scale-kourier-gateway 50m), the Kourier Service as `ClusterIP` and the Envoy +gateway image pinned to `ENVOY_IMAGE`. Memory requests and all limits stay +upstream. Applying the unpatched upstream files restores 300m/200m/100m and +exhausts railiance01's CPU requests again. `tests/test_knative_render.py` +proves the render offline and fails if these overlays drift from rail-knative's +declaration. `install.sh` needs a local `kubectl` for `kubectl kustomize`. Kourier is kept `ClusterIP`; public entry through Traefik, DNS, and TLS requires separate reef admission evidence. The installer enables only Knative's @@ -10,7 +23,8 @@ The installer enables only Knative's containers for fail-closed admission checks such as verifying that egress policy has reconciled before application code starts. -Run `install.sh railiance01`, then `verify.sh railiance01`. +Run `install.sh railiance01`, then `verify.sh railiance01`; `verify.sh` is +read-only and also checks the six CPU requests and the Envoy pin. Before workload admission, rollback deletes Kourier, Serving core, then CRDs using the same verified assets. After Knative Services exist, removal requires diff --git a/install/knative/install.sh b/install/knative/install.sh index 3939376..0d6797c 100755 --- a/install/knative/install.sh +++ b/install/knative/install.sh @@ -1,25 +1,17 @@ #!/usr/bin/env bash set -euo pipefail root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -source "$root/release-lock.env" target="${1:-railiance01}" stage="$(mktemp -d)" trap 'rm -rf "$stage"' EXIT -download() { - curl -fsSL --retry 3 "$1" -o "$2" - printf '%s %s\n' "$3" "$2" | sha256sum --check --status -} -serving="https://github.com/knative/serving/releases/download/knative-v${KNATIVE_VERSION}" -kourier="https://github.com/knative-extensions/net-kourier/releases/download/knative-v${KNATIVE_VERSION}" -download "$serving/serving-crds.yaml" "$stage/crds.yaml" "$SERVING_CRDS_SHA256" -download "$serving/serving-core.yaml" "$stage/core.yaml" "$SERVING_CORE_SHA256" -download "$kourier/kourier.yaml" "$stage/kourier.yaml" "$KOURIER_SHA256" +# Checksum-verified upstream assets, rendered with the declared CPU requests +# (rail-knative substrate/v1.22.0), Kourier ClusterIP and the Envoy pin. +"$root/render.sh" "$stage" +# CRDs first and separately: serving-core.yaml repeats one of them. ssh "$target" kubectl apply -f - < "$stage/crds.yaml" -ssh "$target" kubectl apply -f - < "$stage/core.yaml" +ssh "$target" kubectl apply -f - < "$stage/serving-core.rendered.yaml" ssh "$target" kubectl wait --for=condition=Available deployment --all -n knative-serving --timeout=300s -ssh "$target" kubectl apply -f - < "$stage/kourier.yaml" -ssh "$target" kubectl set image deployment/3scale-kourier-gateway -n kourier-system "kourier-gateway=$ENVOY_IMAGE" +ssh "$target" kubectl apply -f - < "$stage/kourier.rendered.yaml" ssh "$target" kubectl patch configmap/config-network -n knative-serving --type merge -p '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}' ssh "$target" kubectl patch configmap/config-features -n knative-serving --type merge -p '{"data":{"kubernetes.podspec-init-containers":"enabled"}}' -ssh "$target" kubectl patch service/kourier -n kourier-system --type merge -p '{"spec":{"type":"ClusterIP"}}' ssh "$target" kubectl wait --for=condition=Available deployment --all -n kourier-system --timeout=300s diff --git a/install/knative/overlays/kourier/cpu-requests.patch.yaml b/install/knative/overlays/kourier/cpu-requests.patch.yaml new file mode 100644 index 0000000..9245114 --- /dev/null +++ b/install/knative/overlays/kourier/cpu-requests.patch.yaml @@ -0,0 +1,36 @@ +# CPU requests declared by rail-knative (substrate/v1.22.0/cpu-requests.patch.yaml), +# the documents of that file that target kourier.yaml. Only CPU requests are set; +# memory requests and all limits stay upstream. Live on railiance01 since +# 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by +# the founder). Keep in step with rail-knative: tests/test_knative_render.py +# fails if the two declarations disagree. +--- +# upstream v1.22.0 request: 200m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: net-kourier-controller + namespace: knative-serving +spec: + template: + spec: + containers: + - name: controller + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 200m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 3scale-kourier-gateway + namespace: kourier-system +spec: + template: + spec: + containers: + - name: kourier-gateway + resources: + requests: + cpu: 50m diff --git a/install/knative/overlays/kourier/kustomization.yaml b/install/knative/overlays/kourier/kustomization.yaml new file mode 100644 index 0000000..4a5501c --- /dev/null +++ b/install/knative/overlays/kourier/kustomization.yaml @@ -0,0 +1,10 @@ +# Overlay over the checksum-verified upstream kourier.yaml, which render.sh +# stages next to this file as upstream.yaml. render.sh appends the `images:` +# pin for the Envoy gateway from ENVOY_IMAGE in release-lock.env. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - upstream.yaml +patches: + - path: cpu-requests.patch.yaml + - path: service-clusterip.patch.yaml diff --git a/install/knative/overlays/kourier/service-clusterip.patch.yaml b/install/knative/overlays/kourier/service-clusterip.patch.yaml new file mode 100644 index 0000000..cdac368 --- /dev/null +++ b/install/knative/overlays/kourier/service-clusterip.patch.yaml @@ -0,0 +1,10 @@ +# Kourier stays ClusterIP; public entry needs separate reef admission evidence. +# Declared here rather than patched after apply, so a re-run never flips the +# live Service to the upstream LoadBalancer type, even briefly. +apiVersion: v1 +kind: Service +metadata: + name: kourier + namespace: kourier-system +spec: + type: ClusterIP diff --git a/install/knative/overlays/serving-core/cpu-requests.patch.yaml b/install/knative/overlays/serving-core/cpu-requests.patch.yaml new file mode 100644 index 0000000..8676ab8 --- /dev/null +++ b/install/knative/overlays/serving-core/cpu-requests.patch.yaml @@ -0,0 +1,66 @@ +# CPU requests declared by rail-knative (substrate/v1.22.0/cpu-requests.patch.yaml), +# the documents of that file that target serving-core.yaml. Only CPU requests are set; +# memory requests and all limits stay upstream. Live on railiance01 since +# 2026-09-21 (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by +# the founder). Keep in step with rail-knative: tests/test_knative_render.py +# fails if the two declarations disagree. +--- +# upstream v1.22.0 request: 300m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: activator + namespace: knative-serving +spec: + template: + spec: + containers: + - name: activator + resources: + requests: + cpu: 50m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: autoscaler + namespace: knative-serving +spec: + template: + spec: + containers: + - name: autoscaler + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller + namespace: knative-serving +spec: + template: + spec: + containers: + - name: controller + resources: + requests: + cpu: 30m +--- +# upstream v1.22.0 request: 100m +apiVersion: apps/v1 +kind: Deployment +metadata: + name: webhook + namespace: knative-serving +spec: + template: + spec: + containers: + - name: webhook + resources: + requests: + cpu: 30m diff --git a/install/knative/overlays/serving-core/kustomization.yaml b/install/knative/overlays/serving-core/kustomization.yaml new file mode 100644 index 0000000..6e1be69 --- /dev/null +++ b/install/knative/overlays/serving-core/kustomization.yaml @@ -0,0 +1,9 @@ +# Overlay over the checksum-verified upstream serving-core.yaml, which render.sh +# stages next to this file as upstream.yaml. Do not add the CRDs here: core.yaml +# repeats one of them, so they are applied separately and first. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - upstream.yaml +patches: + - path: cpu-requests.patch.yaml diff --git a/install/knative/render.sh b/install/knative/render.sh new file mode 100755 index 0000000..5e5fd23 --- /dev/null +++ b/install/knative/render.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Render exactly what install.sh applies, without touching any cluster. +# render.sh +# Downloads the upstream v1.22.0 assets (reusing files already in +# only when their pinned SHA-256 matches), verifies every checksum, and writes: +# /crds.yaml upstream CRDs, verbatim (applied first) +# /serving-core.rendered.yaml serving-core + declared CPU requests +# /kourier.rendered.yaml kourier + CPU requests, ClusterIP, Envoy pin +# Needs curl, sha256sum and a local kubectl with built-in kustomize. +set -euo pipefail +root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$root/release-lock.env" +stage="${1:?usage: render.sh }" +mkdir -p "$stage" + +fetch() { + local url="$1" out="$2" sum="$3" + if ! printf '%s %s\n' "$sum" "$out" | sha256sum --check --status 2>/dev/null; then + curl -fsSL --retry 3 "$url" -o "$out" + printf '%s %s\n' "$sum" "$out" | sha256sum --check --status \ + || { echo "checksum mismatch: $url" >&2; rm -f "$out"; exit 1; } + fi +} +serving="https://github.com/knative/serving/releases/download/knative-v${KNATIVE_VERSION}" +kourier="https://github.com/knative-extensions/net-kourier/releases/download/knative-v${KNATIVE_VERSION}" +fetch "$serving/serving-crds.yaml" "$stage/crds.yaml" "$SERVING_CRDS_SHA256" +fetch "$serving/serving-core.yaml" "$stage/core.yaml" "$SERVING_CORE_SHA256" +fetch "$kourier/kourier.yaml" "$stage/kourier.yaml" "$KOURIER_SHA256" + +build() { + local overlay="$1" asset="$2" out="$3" dir="$stage/overlay-$1" + rm -rf "$dir" + cp -r "$root/overlays/$overlay" "$dir" + cp "$stage/$asset" "$dir/upstream.yaml" + if [ "$overlay" = kourier ]; then + printf 'images:\n - name: docker.io/envoyproxy/envoy\n newName: %s\n digest: %s\n' \ + "${ENVOY_IMAGE%@*}" "${ENVOY_IMAGE#*@}" >> "$dir/kustomization.yaml" + fi + kubectl kustomize "$dir" > "$stage/$out" +} +build serving-core core.yaml serving-core.rendered.yaml +build kourier kourier.yaml kourier.rendered.yaml diff --git a/install/knative/verify.sh b/install/knative/verify.sh index b631ab3..49b0786 100755 --- a/install/knative/verify.sh +++ b/install/knative/verify.sh @@ -1,10 +1,28 @@ #!/usr/bin/env bash +# Read-only checks of the installed Knative substrate. set -euo pipefail +root="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$root/release-lock.env" target="${1:-railiance01}" ssh "$target" 'set -e test "$(kubectl get namespace knative-serving -o go-template="{{index .metadata.labels \"app.kubernetes.io/version\"}}")" = "1.22.0" test "$(kubectl get service kourier -n kourier-system -o jsonpath="{.spec.type}")" = "ClusterIP" test "$(kubectl get configmap config-network -n knative-serving -o jsonpath="{.data.ingress-class}")" = "kourier.ingress.networking.knative.dev" test "$(kubectl get configmap config-features -n knative-serving -o jsonpath="{.data.kubernetes\\.podspec-init-containers}")" = "enabled" +test "$(kubectl get deployment 3scale-kourier-gateway -n kourier-system -o jsonpath="{.spec.template.spec.containers[?(@.name==\"kourier-gateway\")].image}")" = "'"$ENVOY_IMAGE"'" kubectl wait --for=condition=Available deployment --all -n knative-serving --timeout=120s kubectl wait --for=condition=Available deployment --all -n kourier-system --timeout=120s' +# Declared CPU requests (rail-knative substrate/v1.22.0/cpu-requests.patch.yaml). +while read -r ns deploy container cpu; do + got="$(ssh "$target" kubectl get deployment "$deploy" -n "$ns" \ + -o "jsonpath={.spec.template.spec.containers[?(@.name==\"$container\")].resources.requests.cpu}")" + [ "$got" = "$cpu" ] || { echo "cpu request $ns/$deploy/$container: want $cpu, got ${got:-}" >&2; exit 1; } +done <<'LIST' +knative-serving activator activator 50m +knative-serving autoscaler autoscaler 30m +knative-serving controller controller 30m +knative-serving webhook webhook 30m +knative-serving net-kourier-controller controller 30m +kourier-system 3scale-kourier-gateway kourier-gateway 50m +LIST +echo "knative substrate verified" diff --git a/tests/test_knative_render.py b/tests/test_knative_render.py new file mode 100644 index 0000000..fc295bd --- /dev/null +++ b/tests/test_knative_render.py @@ -0,0 +1,131 @@ +"""Offline proof that install/knative renders the declared CPU requests. + +Runs install/knative/render.sh (download + SHA-256 verification + kustomize; +no cluster contact) and compares the result with the upstream assets: the six +declared CPU requests change, the Kourier Service is ClusterIP, the gateway +image is the pinned Envoy digest, and nothing else differs. Skips when the +upstream assets cannot be fetched or kubectl is missing. +""" +from __future__ import annotations + +import copy +import hashlib +import os +import re +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + yaml = None + +ROOT = Path(__file__).resolve().parents[1] +KNATIVE = ROOT / "install" / "knative" +RAIL_KNATIVE = Path(os.environ.get("RAIL_KNATIVE_DIR", ROOT.parent / "rail-knative")) +DECLARED = { # (namespace, deployment, container) -> cpu request + ("knative-serving", "activator", "activator"): "50m", + ("knative-serving", "autoscaler", "autoscaler"): "30m", + ("knative-serving", "controller", "controller"): "30m", + ("knative-serving", "webhook", "webhook"): "30m", + ("knative-serving", "net-kourier-controller", "controller"): "30m", + ("kourier-system", "3scale-kourier-gateway", "kourier-gateway"): "50m", +} + + +def lock() -> dict[str, str]: + lines = (KNATIVE / "release-lock.env").read_text().splitlines() + return dict(line.split("=", 1) for line in lines if "=" in line) + + +def load(path: Path) -> list[dict]: + return [d for d in yaml.safe_load_all(path.read_text()) if d] + + +def key(doc: dict) -> tuple: + meta = doc["metadata"] + return (doc["apiVersion"], doc["kind"], meta.get("namespace"), meta["name"]) + + +def patch_requests(path: Path) -> dict[tuple, str]: + out = {} + for doc in load(path): + for c in doc["spec"]["template"]["spec"]["containers"]: + out[(doc["metadata"]["namespace"], doc["metadata"]["name"], c["name"])] = c["resources"]["requests"]["cpu"] + return out + + +@unittest.skipIf(yaml is None, "PyYAML not installed") +class DeclarationTests(unittest.TestCase): + def test_overlays_carry_the_declared_requests(self) -> None: + got = {} + for overlay in ("serving-core", "kourier"): + got.update(patch_requests(KNATIVE / "overlays" / overlay / "cpu-requests.patch.yaml")) + self.assertEqual(DECLARED, got) + + def test_verify_checks_the_declared_requests(self) -> None: + rows = re.search(r"<<'LIST'\n(.*?)\nLIST", (KNATIVE / "verify.sh").read_text(), re.S).group(1) + got = {tuple(r.split()[:3]): r.split()[3] for r in rows.splitlines()} + self.assertEqual(DECLARED, got) + + def test_matches_rail_knative_declaration(self) -> None: + source = RAIL_KNATIVE / "substrate" / "v1.22.0" / "cpu-requests.patch.yaml" + if not source.exists(): + self.skipTest(f"rail-knative not checked out at {RAIL_KNATIVE}") + self.assertEqual(DECLARED, patch_requests(source)) + + +@unittest.skipIf(yaml is None or shutil.which("kubectl") is None, "needs PyYAML and kubectl") +class RenderTests(unittest.TestCase): + stage: Path + + @classmethod + def setUpClass(cls) -> None: + cls.stage = Path(os.environ.get("KNATIVE_STAGE_DIR") or tempfile.mkdtemp(prefix="knative-render-")) + result = subprocess.run([str(KNATIVE / "render.sh"), str(cls.stage)], capture_output=True, text=True) + if result.returncode != 0: + if "checksum mismatch" in result.stderr: + raise AssertionError(result.stderr) + raise unittest.SkipTest(f"render.sh failed (offline?): {result.stderr.strip()[:200]}") + + def compare(self, upstream_file: str, rendered_file: str) -> dict[tuple, dict]: + upstream = {key(d): d for d in load(self.stage / upstream_file)} + rendered = {key(d): d for d in load(self.stage / rendered_file)} + self.assertEqual(set(upstream), set(rendered)) + return {k: (upstream[k], rendered[k]) for k in upstream} + + def check(self, upstream_file: str, rendered_file: str) -> set: + seen = set() + envoy = lock()["ENVOY_IMAGE"] + for k, (up, out) in self.compare(upstream_file, rendered_file).items(): + expected = copy.deepcopy(up) + if k[1] == "Deployment": + for c in expected["spec"]["template"]["spec"]["containers"]: + ident = (k[2], k[3], c["name"]) + if ident in DECLARED: + c["resources"]["requests"]["cpu"] = DECLARED[ident] + seen.add(ident) + if c["image"].startswith("docker.io/envoyproxy/envoy:"): + c["image"] = envoy + if k[1:] == ("Service", "kourier-system", "kourier"): + expected["spec"]["type"] = "ClusterIP" + self.assertEqual(expected, out, f"{k} differs beyond the declared changes") + return seen + + def test_crds_are_the_pinned_upstream_file(self) -> None: + digest = hashlib.sha256((self.stage / "crds.yaml").read_bytes()).hexdigest() + self.assertEqual(lock()["SERVING_CRDS_SHA256"], digest) + for doc in load(self.stage / "crds.yaml"): + self.assertEqual("CustomResourceDefinition", doc["kind"]) + + def test_rendered_carries_declared_requests_and_nothing_else_changes(self) -> None: + seen = self.check("core.yaml", "serving-core.rendered.yaml") + seen |= self.check("kourier.yaml", "kourier.rendered.yaml") + self.assertEqual(set(DECLARED), seen) + + +if __name__ == "__main__": + unittest.main() diff --git a/workplans/RAIL-BS-WP-0015-knative-declared-cpu-requests.md b/workplans/RAIL-BS-WP-0015-knative-declared-cpu-requests.md new file mode 100644 index 0000000..00223e1 --- /dev/null +++ b/workplans/RAIL-BS-WP-0015-knative-declared-cpu-requests.md @@ -0,0 +1,121 @@ +--- +id: RAIL-BS-WP-0015 +type: workplan +title: "Make the Knative installer apply rail-knative's declared CPU requests" +domain: financials +repo: railiance-cluster +status: finished +flavor: implementation +owner: codex +topic_slug: railiance +created: "2026-09-21" +updated: "2026-09-21" +related: + - RAIL-BS-WP-0013 + - RAIL-KNATIVE-WP-0002 +--- + +# RAIL-BS-WP-0015 - Knative installer applies the declared CPU requests + +On 2026-09-21 the CPU requests of six Knative Deployments were lowered live on +railiance01 (`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED` +by the founder), to free CPU requests on a node where the identity and +user-engine backups could not be scheduled. The record is +`the-custodian/docs/kubernetes-change-gate-decision.md`. rail-knative declares +the values in `substrate/v1.22.0/cpu-requests.patch.yaml` +(RAIL-KNATIVE-WP-0002). `install/knative/install.sh` still applied the +unpatched upstream manifests, so re-running it would have restored +300m/200m/100m and stalled the backups again. + +| Deployment | upstream | declared | +| --- | --- | --- | +| knative-serving/activator | 300m | 50m | +| knative-serving/autoscaler | 100m | 30m | +| knative-serving/controller | 100m | 30m | +| knative-serving/webhook | 100m | 30m | +| knative-serving/net-kourier-controller | 200m | 30m | +| kourier-system/3scale-kourier-gateway | 200m | 50m | + +rail-knative's declaration, the change-gate record and the live cluster agree +on all six values. + +## T01 - Render the upstream assets through overlays + +```task +id: RAIL-BS-WP-0015-T01 +status: done +priority: high +``` + +`install/knative/render.sh ` downloads the three upstream assets, keeps +the SHA-256 check against `release-lock.env`, and renders with the local +`kubectl kustomize`: `crds.yaml` verbatim (applied first and separately, since +`serving-core.yaml` repeats a CRD), `serving-core.rendered.yaml` and +`kourier.rendered.yaml`. The overlays in `install/knative/overlays/` split +rail-knative's patch file by asset (kustomize refuses a patch document with no +matching resource) and keep each document as rail-knative wrote it. Only CPU +requests change; memory requests and all limits stay upstream. + +The kourier overlay also declares the Service as `ClusterIP` and pins the +Envoy image from `ENVOY_IMAGE`. The old installer applied the upstream +`LoadBalancer` Service and `envoy:v1.37-latest`, then patched both back; on a +re-run that flipped the live Service and rolled the gateway twice. + +## T02 - install.sh and verify.sh + +```task +id: RAIL-BS-WP-0015-T02 +status: done +priority: high +``` + +`install.sh` applies the three rendered files in the old order and keeps the +two ConfigMap patches. It drops `kubectl set image` and the Service patch, +because the render now carries both. `verify.sh` stays read-only and +additionally checks the six CPU requests and the Envoy digest. + +## T03 - Offline proof and a test + +```task +id: RAIL-BS-WP-0015-T03 +status: done +priority: high +``` + +`tests/test_knative_render.py` (in `make test-unit`) runs `render.sh` and +compares every rendered object with upstream. The only differences are the six +CPU requests, the Kourier Service type and the gateway image. The test also +checks the CRD file's checksum, checks that `verify.sh` lists the same six +values, and checks that the overlays match +`rail-knative/substrate/v1.22.0/cpu-requests.patch.yaml` when rail-knative is +checked out next to this repository. It skips when offline. + +Read-only evidence, 2026-09-21: `ssh railiance01 'kubectl diff -f -'` against +each rendered file. +- `crds.yaml` and `kourier.rendered.yaml` show no diff. +- `serving-core.rendered.yaml` shows no diff on any Deployment. It does show a + diff on the `webhook.serving.knative.dev` and + `validation.webhook.serving.knative.dev` webhook configurations. Their + `rules` are filled at runtime by the Knative webhook, and the unpatched + upstream file shows the same diff, so the change here did not cause it. +- For contrast, the unpatched upstream `serving-core.yaml` diffs the four + serving Deployments back to 300m/100m. + +## T04 - Re-running the installer (not run) + +```task +id: RAIL-BS-WP-0015-T04 +status: done +priority: medium +``` + +This session did not run the installer. The values are already live, so +running the patched installer changes none of the six Deployments, the CRDs, +Kourier, the Service type or the gateway image. The one exception is the two +webhook configurations: apply resets them to the upstream skeleton and the +Knative webhook fills them in again, exactly as any earlier re-run did. There +is nothing to apply now. Any future run is +`ADMINISTER @ realm:kubernetes/railiance01` and needs the founder's go-ahead: +`install/knative/install.sh railiance01`, then `install/knative/verify.sh railiance01`. +Rollback of this change is a revert of its commit; the live values need no +rollback.