tegwick
e1f7491e31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 276069@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c
93 lines
6.4 KiB
Markdown
93 lines
6.4 KiB
Markdown
---
|
|
id: NK-WP-0052
|
|
type: workplan
|
|
title: "Wave-2 resource right-sizing from the 2026-10-21 allocation review"
|
|
domain: infotech
|
|
repo: net-kingdom
|
|
status: finished
|
|
flavor: implementation
|
|
owner: claude
|
|
topic_slug: netkingdom
|
|
created: "2026-10-21"
|
|
updated: "2026-10-21"
|
|
related:
|
|
- CUST-WP-0071
|
|
- RESOURCE-WP-0008
|
|
state_hub_workstream_id: "7eaae0ab-19e6-533a-b3d1-11d11fa56210"
|
|
---
|
|
|
|
# NK-WP-0052 — Wave-2 resource right-sizing from the 2026-10-21 allocation review
|
|
|
|
Receiving workplan in `net-kingdom` for the wave-2 request changes below. Login path for the whole fleet; apply one Deployment at a time and test sign-in after each.
|
|
|
|
Origin: the 2026-10-21 node allocation review (`RESOURCE-WP-0008-T03`, feeding `CUST-WP-0071-T03`). Per the founder's 2026-10-21 delivery interview (`the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`) the small adjustments form wave 2: one receiving task per owner repo, applied later in a batch. Applying wave 2 needs a separate founder approval, so every cluster-write task here is a human-gated `wait`. This file changes no manifest.
|
|
|
|
## Apply the reviewed request changes
|
|
|
|
```task
|
|
id: NK-WP-0052-T01
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "a69cd984-3c32-5087-9438-913c37e309f6"
|
|
```
|
|
|
|
Values are pod-aggregate figures from the budget file (7-day window ending 2026-10-21T06:08Z). Limits stay unchanged unless stated.
|
|
|
|
| Workload (ns/name) | Pods | CPU request | CPU limit | CPU p95/peak 7d (m) | Memory request | Memory limit | Mem p95/peak (Mi) | Throttle | Action |
|
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
|
| `mfa/net-kingdom-sso-mfa` | 1 | 110m → **20m** | 1000m | 12.2/23.1 | 384Mi → **272Mi** | 896Mi | 225.6/225.8 | 0.049 | decrease + memory down |
|
|
| `sso/net-kingdom-sso-mfa` | 4 | 150m → **40m** | 1150m | 22.4/34.5 | 256Mi → **176Mi** | 640Mi | 140.5/179.1 | 0.537 | decrease + review CPU limit (throttled) + memory down |
|
|
|
|
Source and container detail:
|
|
|
|
- `mfa/net-kingdom-sso-mfa`: namespace `mfa`: Deployment `privacyidea`, container `privacyidea` (`sso-mfa/k8s/privacyidea/deployment.yaml`, 100m/256Mi) plus the remaining 10m/128Mi in the same pod (confirm which container) — split the 20m/272Mi proposal across them. Also ~5% throttling at the 500m container limit.
|
|
- `sso/net-kingdom-sso-mfa`: namespace `sso`, four Deployments: `keycape` (25m/32Mi, limit 200m, `sso-mfa/k8s/keycape/deployment.yaml`), `lldap` (50m/64Mi, limit 200m), `authelia` (50m/128Mi, limit 500m), `identity-provisioner` (25m/32Mi, limit 250m). 53.7% of CFS periods throttled on the login path: find the throttled container(s) and review their CPU limit independently of the request decrease.
|
|
|
|
Evidence: resource-control `docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md` (+ `.json`, commit `3d9b1bd`). Rule: CPU = max(floor, p95x1.5, peakx0.5) rounded up to 5m; memory request = 7d p95 working set x1.15 (only when the change is at least 64Mi).
|
|
|
|
Steps:
|
|
|
|
1. Map the pod-aggregate proposal onto the named container(s) in the source file; where a workload has several containers or pods, split by the per-pod figures in `2026-10-21-workload-usage.json` and keep the sum at the proposed value.
|
|
2. Prepare the source change as a reviewed commit (no push, no sync) and record the before values here.
|
|
3. Apply only inside the batched wave-2 rollout, after founder approval, through the owner's normal deploy path (ArgoCD sync or documented helm/kubectl apply).
|
|
|
|
Verification: `kubectl -n <ns> get pod -o jsonpath='{..resources}'` shows the new requests; pods Ready with no new restarts or OOMKilled over 24h; for CPU decreases the throttle ratio and request latency stay at or below the 7d baseline; node request total drops by the expected delta.
|
|
|
|
Prepared 2026-10-22 (wave-2 worker W3; no apply, cluster read-only). Live
|
|
objects are client-side-applied from `sso-mfa/k8s/` (plus out-of-band
|
|
`kubectl patch/set` history); each edited file was diffed against live with
|
|
`kubectl diff`: only `resources` keys change, no other drift. The
|
|
`factor-recovery` sidecar (added by `privacyidea/deploy-factor-recovery.py`,
|
|
not in `deployment.yaml`) is kept by the client-side merge and left at 10m/128Mi.
|
|
|
|
The `sso` row (4 pods) splits per container; the throttled pod is **lldap**
|
|
(53.7% of CFS periods at its 200m limit; authelia/keycape/provisioner ~0%):
|
|
|
|
| Container | CPU req | CPU limit | Mem req | Basis |
|
|
|---|---|---|---|---|
|
|
| sso/lldap | 50m -> 25m | 200m -> **1000m** | 64Mi (keep) | p95/peak 20.4/27.0m; limit rule max(2x current, 1000m) |
|
|
| sso/authelia | 50m -> 5m | 500m | 128Mi -> 64Mi | p95/peak 0.5/6.0m, mem p95 50Mi |
|
|
| sso/keycape | 25m -> 5m | 200m | 32Mi | p95 ~1m |
|
|
| sso/identity-provisioner | 25m -> 5m | 250m | 32Mi | p95 0.5m |
|
|
| mfa/privacyidea (main) | 100m -> 10m | 500m | 256Mi -> 160Mi | pod p95/peak 12.2/23.1m; now 125Mi |
|
|
| databases/net-kingdom-pg | 100m -> 50m | 1000m | 256Mi (keep) | DB floor 50m |
|
|
|
|
Totals: sso 150m -> 40m (as proposed); sso memory 256Mi -> 192Mi (budget 176Mi;
|
|
lldap kept at 64Mi over its 59Mi p95); mfa 110m -> 20m, 384Mi -> 288Mi (budget
|
|
272Mi; sidecar untouched). net-kingdom-pg: `shared_buffers` unset (default
|
|
128MB) — no conflict with the unchanged 256Mi request / 1Gi limit.
|
|
|
|
Apply (local kubectl, client-side as live, one at a time):
|
|
`kubectl apply -f sso-mfa/k8s/{lldap,authelia,keycape,identity-provisioner}/deployment.yaml`,
|
|
`kubectl apply -f sso-mfa/k8s/privacyidea/deployment.yaml`,
|
|
`kubectl apply -f sso-mfa/k8s/postgresql/cluster.yaml` (DB window).
|
|
Impact: every Deployment is 1 replica with strategy Recreate, so each apply is a
|
|
short outage of that component: authelia/lldap/keycape = SSO login path,
|
|
privacyidea = MFA; net-kingdom-pg restart (CNPG single instance) takes
|
|
privacyIDEA/authelia storage down briefly. Run off-peak, health-check between.
|
|
|
|
Rollback: revert the source commit and re-sync/re-apply; the before values in the table above are the rollback target. Keep before/after `kubectl get -o yaml` captures as the receipt.
|
|
|
|
## Applied — 2026-10-22 (NK-WP-0052-T01)
|
|
|
|
Applied under the founder's wave-2 authorization (the-custodian docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md), after a full-render diff against live that showed resources only. lldap CPU limit raised to 1 core (it had been throttled); authelia, keycape, identity-provisioner and privacyIDEA right-sized; net-kingdom-pg restarted at 22:40Z at 50m and is healthy. KeyCape discovery returns 200 and the forge sign-in redirect works. Evidence: the-custodian docs/evidence/2026-10-22-wave2-apply.md.
|