resource-control/docs/evidence/2026-10-21-node-allocation-review.md

Node-level allocation review — 2026-10-21

Provider: resource-control, RESOURCE-WP-0008-T02/T03; beneficiary CUST-WP-0071-T03. This is a review proposal for owners. It applies nothing, admits no capacity and proposes no procurement. Application stays with each runtime owner (CUST-WP-0071-T04, separate founder approval).

Sources and method

Target identity verified before capture: context default, single k3s node 239.62.205.92.host.secureserver.net (92.205.62.239, control-plane/etcd, v1.35.1+k3s1). All access was read-only (kubectl get, get --raw GETs).

Retained under docs/evidence/2026-10-21-node-allocation-review/:

File	Content
2026-10-21-allocation-reconcile.{json,md}	Existing Custodian collector scripts/reconcile_railiance_allocation.py (railiance-cluster observe + STATE-WP-0091 preflight), captured 06:08:20Z, written here via --output-dir
2026-10-21-telemetry-7d.json	Exact PromQL and responses, 7d window, 5m step, in-cluster telemetry-prometheus via the API service proxy (oldest sample 7.26 days)
2026-10-21-workload-usage.json	Per-workload join of live pod requests/limits with per-pod 7d p95/peak CPU, memory, throttling, restarts (join.py)
2026-10-21-supplementary-observation.json	kubelet summary samples, kubelet reservations, HPA, rollout surge, CronJobs, stale pods, empty namespaces, concurrent benchmark
2026-10-21-node-budget.{json,md}	Generated proposals, owners, receiving records and node budget (build_review.py, no cluster access)

Workload p95 values are sums of per-pod p95 (an upper bound for multi-pod workloads). Terminated Job pods appear only in namespace totals.

Current state

• 4000m allocatable, 3605m scheduled requests, 0 pending, 395m arithmetic residual (unchanged since 2026-10-19). Memory: 9856Mi requested of 15993Mi.
• Measured demand: all containers 7d p95 966m, peak 1740m. Host/k3s outside pods 677–871m across eight kubelet samples (k3s server 415–504m, 2604Mi). Measured demand is ~1.8 cores at p95 and ~2.6 cores at peak: it fits the node.
• The kubelet sets no systemReserved/kubeReserved, so the scheduler may hand all 4000m to pods while k3s itself needs ~0.9 core. Requests total 3.7× measured pod p95. The shortfall below is reservation inflation, not missing CPU.

Priority findings

1. Knative/Kourier. Requests are already 220m in total (RAIL-KNATIVE-WP-0002, finished); the "~1000m" figure in CUST-WP-0071 is stale. Usage is p95 ~12m. The open risk is HPA ceilings on a single node: activator max 20, gateway 10 and webhook 5 can claim up to 1520m more requests. Proposal: cap them at 3/3/2 (230m) and trim requests to the rule floor (220m→60m). Owner rail-knative.
2. Forgejo. The gitea main container has no requests or limits. Its 100m is the init container's. Usage is p95 133m, peak 377m, memory p95 956Mi against a 128Mi request. The runner (runner + dind) has no requests or limits; namespace peak is 1033m. Proposal: gitea 200m/1104Mi plus a memory limit, runner 125m/496Mi with a dind limit (~1500m/2Gi) and one concurrent job. The Prometheus request recording for forgejo still returns no series (missing since 2026-09-11). Owners railiance-forge and rapp-telemetry.
3. Zero-request workloads (13). Eleven get small requests: state-hub-db 70m (p95 37m), target-revenue-pg 50m, forgejo-runner 125m, traefik 10m, cert-manager 3×10m, ESO 3×10m, local-path 10m. The two k3s svclb DaemonSets (~0m) stay at zero as an accepted, documented exception.
4. Stale probe/drill allocations. coulomb/ihp-railiance-probe reserves 50m while in ImagePullBackOff for 21 days from legacy registry 92.205.130.254, with two more stale pods. Proposal: retire or scale to zero by owner decision. The owner is unconfirmed: namespace-ownership says unknown, and the ihp-railiance-probe repo's workplans are archived. platform-pg-drill no longer exists. openbao-migration-drill and cert-manager-test are empty. platform-pg-2 is an overflow admission cluster, not a drill. Failed pods in forgejo (105d/109d) and telemetry (smtp-proof, 22d) reserve nothing; they are hygiene only. inter-hub sits at 0 replicas with a latent 250m request (Haskell Inter-Hub is retired).
5. Under-reserved and throttled. state-hub API p95 258m at a 250m request: proposal 390m (p95×1.5). Prometheus p95 134m at 100m: proposal 205m/1152Mi. activity-core and grafana memory requests sit far below p95. sso is 53.7% CFS-throttled and playwright 23.8%: the owners should review those CPU limits.
6. Release surge model. STATE-WP-0091 preflight models a 100m API surge, but the live API pod requests 250m. A state-hub release needs ~310m, not 160m. Owner state-hub (STATE-WP-0091 is finished).

Rule and assumptions

CPU proposal = max(floor, p95×1.5, peak×0.5), rounded up to 5m. Floors are 50m per database pod, 10m per critical-path pod and 5m otherwise. A change is proposed only when |Δ| ≥ 15m or the current request is zero. Overrides: coredns 50m, state-hub 390m, the probe 0m, the svclb DaemonSets 0m, and Vergabe 60m. Memory request = p95 working set ×1.15 when |Δ| ≥ 64Mi; limits are unchanged. Requests set CFS weight and placement only, so limits keep burst room. A lower request reduces a workload's share only under full contention. Database owners check shared_buffers before lowering a database memory request.

Concurrency: one Forgejo build job at a time. One release at a time (largest: state-hub). CronJobs overlap 02:40–04:00. Two Vergabe pilot users (capped comparison bfd1a2d). Rollback for any applied change is the owner's previous manifest revision (CUST-WP-0071-T04 retains the receipts).

Node budget (CPU, m)

Item	Current	Proposed
Steady pod requests (incl. Vergabe 60m, apps-pg kept at 100m)	3605	2450
Host / k3s control plane (no kube/system-reserved)	900	900
Release surge + migration hook (state-hub, one at a time)	310	310
Admitted factory/build execution (one runner job burst)	250	250
Maintenance CronJobs	50	50
Knative HPA scale-out headroom	1520	230
Pilot growth reserve (next tenant/user growth; Vergabe is in steady requests)	0	200
Total, all events concurrent	6635	4390
Residual vs 4000m, all concurrent	−2635	−390
Total, one disruptive event at a time (max of surge/build/HPA)	6075	3910
Residual, serialized	−2075	+90

Proposed changes: +660m increases and −1815m decreases, across 54 non-keep rows.

Memory (Mi, allocatable 15993; k3s 2604 single sample, surge 864, build 512, pilot growth 256):

Memory	Current	Proposed
Pod requests	9856	12104
Residual, all concurrent	+2157	−347
Residual, serialized (max of surge/build)	+2669	+165

The current memory residual looks comfortable only because 13 zero-request workloads hide about 1.3Gi of working set. The proposed requests include the Vergabe raise (+256Mi), and apps-pg keeps its 256Mi.

Reconciliation verdict: the budget reconciles only conditionally, at +90m CPU and +165Mi memory. That needs three things: the lean decreases are accepted, releases, builds and Knative scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before releases), and the HPA ceilings are capped. With every event concurrent it is −390m CPU and −347Mi memory short. Measured demand plus margin does fit the 4 vCPU. Procurement is not indicated; the remedy is to correct reservations. Once requests have come down, railiance-cluster should make host operations explicit with kube-reserved/system-reserved (~900m/2.5Gi). Applying that first would make the current 3605m unschedulable.

Vergabe slot (filled 2026-10-21 from the target replay)

Source: railiance-apps 1af6d82, docs/evidence/2026-10-21-vergabe-target-replay/ (RAPPS-WP-0014-T04). The replay ran the deployed digest on the target node: 3 workers, 60m/256Mi request and 1000m/1Gi limit, in an isolated namespace that has since been torn down.

• 3675 ordinary operations with 0 errors; ordinary p95 0.192s; transfer p95 0.076s. The restart recovered in 19s.
• App CPU was 775m mean, 998m p95 and 1020m peak, throttled in 36% of periods (66% during the burst phase). Memory peaked at 483MiB with no OOM.
• Incremental PostgreSQL demand was about 320m mean and 480m p95, measured on an isolated PostgreSQL pod rather than shared apps-pg.
• The load was a ~25 req/s closed loop, far above what two human users generate.

Decision (T02):

• Keep the 60m CPU request and the 1000m/1Gi limits.
• Raise the memory request from 256Mi to 512Mi in the next wave; the owner is railiance-apps.
• Keep apps-pg at 100m. The rule would have proposed 50m; the replay justifies keeping it.
• The ~0.3–0.5 core of incremental apps-pg load at synthetic rates is burst usage under apps-pg's 1000m limit, not a reservation. Revisit if apps-pg CPU stays above 600m.

At the synthetic rate the pilot used about 1.1 core inside its limits, and node free CPU at p5 fell to about 150m. That was a stress case, not the two-user demand.

The replay does not establish:

• production settings (SSO, ingress/TLS)
• contention on the shared apps-pg cluster
• observed human demand
• busy-hour node contention (CI, backups)
• human/recovery acceptance (RAPPS-WP-0014-T03)

Receiving owners

None of the 54 non-keep rows has a live receiving task yet. Wave 1 tasks (founder decision 2026-10-21) are being prepared in rail-knative, railiance-forge and state-hub, with rapp-telemetry following. Wave 2 is one task per remaining repo. RAIL-KNATIVE-WP-0002, RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover admission and release, not sizing. RAPPS-WP-0014-T04 delivered the Vergabe replay; its memory raise needs a railiance-apps receiving task. Per-repo lists are in 2026-10-21-node-budget.md ("Missing receiving tasks by owner repo"). Each repo needs a receiving task:

activity-core, audit-core, canned-prompts, core-hub, coulomb-social, email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative, railiance-apps, railiance-cluster, railiance-enablement, railiance-forge, railiance-platform, rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface, sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.

Owners without a row in this table also need tasks for the measurement work: rapp-telemetry (Forgejo recording rule; host/node series), railiance-cluster (host/control-plane 7d series, kube/system-reserved), state-hub (preflight surge constant) and railiance-apps (inter-hub latent 250m).

Open gaps

• No 7-day host/control-plane series. The 900m figure is a 70-second sample.
• Single-node HPA ceilings are not reviewed by rail-knative.
• Shared apps-pg behaviour under pilot load (the replay used an isolated database).
• Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
• Owner for coulomb namespace unconfirmed; flex-auth deployment source not available locally.
• Startup/restart and rollout behaviour at the lowered requests is unverified (CUST-WP-0071-T04 acceptance).