resource-control/docs/evidence/2026-10-21-node-allocation-review.md

Node-level allocation review — 2026-10-21

Provider: resource-control, RESOURCE-WP-0008-T02/T03; beneficiary CUST-WP-0071-T03. This is a review proposal for owners. It applies nothing, admits no capacity and proposes no procurement. Application stays with each runtime owner (CUST-WP-0071-T04, separate founder approval).

Sources and method

Target identity verified before capture: context default, single k3s node 239.62.205.92.host.secureserver.net (92.205.62.239, control-plane/etcd, v1.35.1+k3s1). All access was read-only (kubectl get, get --raw GETs).

Retained under docs/evidence/2026-10-21-node-allocation-review/:

File	Content
2026-10-21-allocation-reconcile.{json,md}	Existing Custodian collector scripts/reconcile_railiance_allocation.py (railiance-cluster observe + STATE-WP-0091 preflight), captured 06:08:20Z, written here via --output-dir
2026-10-21-telemetry-7d.json	Exact PromQL and responses, 7d window, 5m step, in-cluster telemetry-prometheus via the API service proxy (oldest sample 7.26 days)
2026-10-21-workload-usage.json	Per-workload join of live pod requests/limits with per-pod 7d p95/peak CPU, memory, throttling, restarts (join.py)
2026-10-21-supplementary-observation.json	kubelet summary samples, kubelet reservations, HPA, rollout surge, CronJobs, stale pods, empty namespaces, concurrent benchmark
2026-10-21-node-budget.{json,md}	Generated proposals, owners, receiving records and node budget (build_review.py, no cluster access)

Workload p95 values are sums of per-pod p95 (an upper bound for multi-pod workloads). Terminated Job pods appear only in namespace totals.

Current state

• 4000m allocatable, 3605m scheduled requests, 0 pending, 395m arithmetic residual (unchanged since 2026-10-19). Memory: 9856Mi requested of 15993Mi.
• Measured demand: all containers 7d p95 966m, peak 1740m. Host/k3s outside pods 677–871m across eight kubelet samples (k3s server 415–504m, 2604Mi). Measured demand is ~1.8 cores at p95 and ~2.6 cores at peak: it fits the node.
• The kubelet sets no systemReserved/kubeReserved, so the scheduler may hand all 4000m to pods while k3s itself needs ~0.9 core. Requests total 3.7× measured pod p95. The shortfall below is reservation inflation, not missing CPU.

Priority findings

1. Knative/Kourier. Requests are already 220m in total (RAIL-KNATIVE-WP-0002, finished); the "~1000m" figure in CUST-WP-0071 is stale. Usage is p95 ~12m. The open risk is HPA ceilings on a single node: activator max 20, gateway 10 and webhook 5 can claim up to 1520m more requests. Proposal: cap them at 3/3/2 (230m) and trim requests to the rule floor (220m→60m). Owner rail-knative.
2. Forgejo. The gitea main container has no requests or limits. Its 100m is the init container's. Usage is p95 133m, peak 377m, memory p95 956Mi against a 128Mi request. The runner (runner + dind) has no requests or limits; namespace peak is 1033m. Proposal: gitea 200m/1104Mi plus a memory limit, runner 125m/496Mi with a dind limit (~1500m/2Gi) and one concurrent job. The Prometheus request recording for forgejo still returns no series (missing since 2026-09-11). Owners railiance-forge and rapp-telemetry.
3. Zero-request workloads (13). Eleven get small requests: state-hub-db 70m (p95 37m), target-revenue-pg 50m, forgejo-runner 125m, traefik 10m, cert-manager 3×10m, ESO 3×10m, local-path 10m. The two k3s svclb DaemonSets (~0m) stay at zero as an accepted, documented exception.
4. Stale probe/drill allocations. coulomb/ihp-railiance-probe reserves 50m while in ImagePullBackOff for 21 days from legacy registry 92.205.130.254, with two more stale pods. Proposal: retire or scale to zero by owner decision. The owner is unconfirmed: namespace-ownership says unknown, and the ihp-railiance-probe repo's workplans are archived. platform-pg-drill no longer exists. openbao-migration-drill and cert-manager-test are empty. platform-pg-2 is an overflow admission cluster, not a drill. Failed pods in forgejo (105d/109d) and telemetry (smtp-proof, 22d) reserve nothing; they are hygiene only. inter-hub sits at 0 replicas with a latent 250m request (Haskell Inter-Hub is retired).
5. Under-reserved and throttled. state-hub API p95 258m at a 250m request: proposal 390m (p95×1.5). Prometheus p95 134m at 100m: proposal 205m/1152Mi. activity-core and grafana memory requests sit far below p95. sso is 53.7% CFS-throttled and playwright 23.8%: the owners should review those CPU limits.
6. Release surge model. STATE-WP-0091 preflight models a 100m API surge, but the live API pod requests 250m. A state-hub release needs ~310m, not 160m. Owner state-hub (STATE-WP-0091 is finished).

Rule and assumptions

CPU proposal = max(floor, p95×1.5, peak×0.5), rounded up to 5m. Floors are 50m per database pod, 10m per critical-path pod and 5m otherwise. A change is proposed only when |Δ| ≥ 15m or the current request is zero. Overrides: coredns 50m, state-hub 390m, the probe 0m, the svclb DaemonSets 0m, and Vergabe 60m. Memory request = p95 working set ×1.15 when |Δ| ≥ 64Mi; limits are unchanged. Requests set CFS weight and placement only, so limits keep burst room. A lower request reduces a workload's share only under full contention. Database owners check shared_buffers before lowering a database memory request.

Concurrency: one Forgejo build job at a time. One release at a time (largest: state-hub). CronJobs overlap 02:40–04:00. Two Vergabe pilot users (capped comparison bfd1a2d). Rollback for any applied change is the owner's previous manifest revision (CUST-WP-0071-T04 retains the receipts).

Node budget (CPU, m)

Item	Current	Proposed
Steady pod requests (incl. Vergabe 60m provisional)	3605	2400
Host / k3s control plane (no kube/system-reserved)	900	900
Release surge + migration hook (state-hub, one at a time)	310	310
Admitted factory/build execution (one runner job burst)	250	250
Maintenance CronJobs	50	50
Knative HPA scale-out headroom	1520	230
Pilot growth reserve — SLOT, pending RAPPS-WP-0014-T04	0	200
Total, all events concurrent	6635	4340
Residual vs 4000m, all concurrent	−2635	−340
Total, one disruptive event at a time (max of surge/build/HPA)	6075	3860
Residual, serialized	−2075	+140

Proposed changes: +660m in increases and −1865m in decreases, across 55 non-keep rows (54 owner changes plus the Vergabe slot).

Memory (Mi): proposed requests 11704 + k3s 2604 + surge 864 + build 512 + pilot slot 256 leaves +53 of 15993. It fits, but only just. The k3s figure is a single sample.

Reconciliation verdict: the budget reconciles only conditionally, at +140m. That needs three things: the lean decreases are accepted, releases, builds and Knative scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before releases), and the HPA ceilings are capped. With every event concurrent it is −340m short. Measured demand plus margin does fit the 4 vCPU. Procurement is not indicated; the remedy is to correct reservations. Once requests have come down, railiance-cluster should make host operations explicit with kube-reserved/system-reserved (~900m/2.5Gi). Applying that first would make the current 3605m unschedulable.

Vergabe slot

Vergabe stays at 60m request, provisional, pending the target replay (RAPPS-WP-0014-T04, separate worker). The replay namespace vergabe-sizing-20261021 (temporary 260m) appeared after the 06:08Z capture and is excluded from the steady budget. Insert the replay result into vergabe_slot.replay_result and the pilot-growth row of build_review.py, then regenerate. A larger Vergabe request must fit inside the +140m serialized residual together with the pilot-growth slot.

Receiving owners

None of the 54 non-keep rows has a live receiving task. RAIL-KNATIVE-WP-0002, RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover admission and release, not sizing. The only live slot is RAPPS-WP-0014-T04 for Vergabe. Per-repo lists are in 2026-10-21-node-budget.md ("Missing receiving tasks by owner repo"). Each repo needs a receiving task:

activity-core, audit-core, canned-prompts, core-hub, coulomb-social, email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative, railiance-cluster, railiance-enablement, railiance-forge, railiance-platform, rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface, sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.

Owners without a row in this table also need tasks for the measurement work: rapp-telemetry (Forgejo recording rule; host/node series), railiance-cluster (host/control-plane 7d series, kube/system-reserved), state-hub (preflight surge constant) and railiance-apps (inter-hub latent 250m).

Open gaps

• No 7-day host/control-plane series. The 900m figure is a 70-second sample.
• Single-node HPA ceilings are not reviewed by rail-knative.
• Vergabe target replay result (slot).
• Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
• Owner for coulomb namespace unconfirmed; flex-auth deployment source not available locally.
• Startup/restart and rollout behaviour at the lowered requests is unverified (CUST-WP-0071-T04 acceptance).