coulomb/the-custodian
4
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Record the released CPU requests on railiance01: backups running again.
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 7s
Details

Browse source 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
codex 2026-09-21 18:31:34 +02:00
parent bc0963868a
commit 79de54cab1
1 changed files with 31 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/kubernetes-change-gate-decision.md
View file

@ -132,3 +132,34 @@ Before the install, railiance01 already had **100% of its allocatable CPU reques
| activity-core cert-manager ACME solver | 2026-09-19 |
ArgoCD did not cause this: it requests no CPU. It is an existing capacity defect with a direct consequence: **identity and user-engine backups have not run.** The custodian has not changed anything; it is reported to the founder.
## CPU requests released, 2026-09-21
**Founder go-ahead, 2026-09-21.** The mode was `ADMINISTER @ realm:kubernetes/railiance01` with `activation=APPROVED`, performed by the custodian session over SSH. Both targets are non-production tier: rail-knative's readiness is `verified`, and the stuck drill pod belongs to a drill.
**Changed.** Only CPU requests were changed; limits are unchanged.
| Deployment | Before | After |
|---|---|---|
| knative-serving/activator | 300m | 50m |
| kourier-system/3scale-kourier-gateway | 200m | 50m |
| knative-serving/net-kourier-controller | 200m | 30m |
| knative-serving/controller | 100m | 30m |
| knative-serving/webhook | 100m | 30m |
| knative-serving/autoscaler | 100m | 30m |
The HPAs on the activator, gateway and webhook target 100% of the request. At observed use they now read about 2–10%. The stuck pod `platform-pg-drill/platform-pg-drill-1` was force-deleted; its deletion had been requested on 2026-08-13 and it had no finalizers.
**What happened.** The rolling updates deadlocked at first. The new pods are created before the old ones stop, and on a node with no requestable CPU none of the new pods could be scheduled. Force-deleting the drill pod freed 50m, which let the first new pod schedule; its old pod then stopped, and the rest followed. **Lesson:** on a node whose requests are exhausted, free capacity *before* changing a Deployment's requests, or a request cut cannot roll out.
**Verified afterwards:**
- all six rollouts are complete, with no old pods left;
- `ksvc/rapp-qonto` is Ready;
- 0 Pending pods cluster-wide;
- the authelia, lldap, privacyidea and user-engine backups and audit-core's attest-chain have Completed, and fresh backups ran within minutes;
- node CPU is at 69% actual use.
**Open:**
- rail-knative must declare these requests, or the next upstream apply resets them.
- Actual CPU use is about 70% on a single node. That is a capacity question for the railiance02 move, not something to settle here.
- state-hub uses about 447m against a 100m request. It is *under*-requested.