resource-control/docs/evidence/2026-10-21-node-allocation-review.md

# Node-level allocation review — 2026-10-21

Provider: resource-control, RESOURCE-WP-0008-T02/T03; beneficiary CUST-WP-0071-T03.
This is a review proposal for owners. It applies nothing, admits no capacity and
proposes no procurement. Application stays with each runtime owner (CUST-WP-0071-T04,
separate founder approval).

## Sources and method

Target identity verified before capture: context `default`, single k3s node
`239.62.205.92.host.secureserver.net` (92.205.62.239, control-plane/etcd,
v1.35.1+k3s1). All access was read-only (`kubectl get`, `get --raw` GETs).

Retained under `docs/evidence/2026-10-21-node-allocation-review/`:

| File | Content |
| --- | --- |
| `2026-10-21-allocation-reconcile.{json,md}` | Existing Custodian collector `scripts/reconcile_railiance_allocation.py` (railiance-cluster observe + STATE-WP-0091 preflight), captured 06:08:20Z, written here via `--output-dir` |
| `2026-10-21-telemetry-7d.json` | Exact PromQL and responses, 7d window, 5m step, in-cluster `telemetry-prometheus` via the API service proxy (oldest sample 7.26 days) |
| `2026-10-21-workload-usage.json` | Per-workload join of live pod requests/limits with per-pod 7d p95/peak CPU, memory, throttling, restarts (`join.py`) |
| `2026-10-21-supplementary-observation.json` | kubelet summary samples, kubelet reservations, HPA, rollout surge, CronJobs, stale pods, empty namespaces, concurrent benchmark |
| `2026-10-21-node-budget.{json,md}` | Generated proposals, owners, receiving records and node budget (`build_review.py`, no cluster access) |

Workload p95 values are sums of per-pod p95 (an upper bound for multi-pod
workloads). Terminated Job pods appear only in namespace totals.

## Current state

- 4000m allocatable, 3605m scheduled requests, 0 pending, 395m arithmetic residual
  (unchanged since 2026-10-19). Memory: 9856Mi requested of 15993Mi.
- Measured demand: all containers 7d p95 966m, peak 1740m. Host/k3s outside pods
  677–871m across eight kubelet samples (k3s server 415–504m, 2604Mi). Measured
  demand is ~1.8 cores at p95 and ~2.6 cores at peak: **it fits the node.**
- The kubelet sets no `systemReserved`/`kubeReserved`, so the scheduler may hand
  all 4000m to pods while k3s itself needs ~0.9 core. Requests total 3.7× measured
  pod p95. The shortfall below is reservation inflation, not missing CPU.

## Priority findings

1. **Knative/Kourier.** Requests are already 220m in total (RAIL-KNATIVE-WP-0002,
   finished); the "~1000m" figure in CUST-WP-0071 is stale. Usage is p95 ~12m.
   The open risk is HPA ceilings on a single node: activator max 20, gateway 10 and
   webhook 5 can claim up to 1520m more requests. Proposal: cap them at 3/3/2
   (230m) and trim requests to the rule floor (220m→60m). Owner rail-knative.
2. **Forgejo.** The gitea main container has no requests or limits. Its 100m is
   the init container's. Usage is p95 133m, peak 377m, memory p95 956Mi against a
   128Mi request. The runner (runner + dind) has no requests or limits; namespace
   peak is 1033m. Proposal: gitea 200m/1104Mi plus a memory limit, runner
   125m/496Mi with a dind limit (~1500m/2Gi) and one concurrent job. The Prometheus
   request recording for `forgejo` still returns no series (missing since
   2026-09-11). Owners railiance-forge and rapp-telemetry.
3. **Zero-request workloads (13).** Eleven get small requests: state-hub-db 70m
   (p95 37m), target-revenue-pg 50m, forgejo-runner 125m, traefik 10m, cert-manager
   3×10m, ESO 3×10m, local-path 10m. The two k3s svclb DaemonSets (~0m) stay at
   zero as an accepted, documented exception.
4. **Stale probe/drill allocations.** `coulomb/ihp-railiance-probe` reserves 50m
   while in ImagePullBackOff for 21 days from legacy registry 92.205.130.254, with
   two more stale pods. Proposal: retire or scale to zero by owner decision. The
   owner is unconfirmed: namespace-ownership says unknown, and the
   ihp-railiance-probe repo's workplans are archived. `platform-pg-drill` no longer
   exists. `openbao-migration-drill` and `cert-manager-test` are empty. platform-pg-2
   is an overflow admission cluster, not a drill. Failed pods in forgejo
   (105d/109d) and telemetry (smtp-proof, 22d) reserve nothing; they are hygiene
   only. `inter-hub` sits at 0 replicas with a latent 250m request (Haskell
   Inter-Hub is retired).
5. **Under-reserved and throttled.** state-hub API p95 258m at a 250m request:
   proposal 390m (p95×1.5). Prometheus p95 134m at 100m: proposal 205m/1152Mi.
   activity-core and grafana memory requests sit far below p95. sso is 53.7%
   CFS-throttled and playwright 23.8%: the owners should review those CPU limits.
6. **Release surge model.** STATE-WP-0091 preflight models a 100m API surge, but
   the live API pod requests 250m. A state-hub release needs ~310m, not 160m.
   Owner state-hub (STATE-WP-0091 is finished).

## Rule and assumptions

CPU proposal = max(floor, p95×1.5, peak×0.5), rounded up to 5m. Floors are 50m
per database pod, 10m per critical-path pod and 5m otherwise. A change is
proposed only when |Δ| ≥ 15m or the current request is zero. Overrides: coredns
50m, state-hub 390m, the probe 0m, the svclb DaemonSets 0m, and Vergabe 60m.
Memory request = p95 working set ×1.15 when |Δ| ≥ 64Mi; limits are unchanged.
Requests set CFS weight and placement only, so limits keep burst room. A lower
request reduces a workload's share only under full contention. Database owners
check shared_buffers before lowering a database memory request.

Concurrency: one Forgejo build job at a time. One release at a time (largest:
state-hub). CronJobs overlap 02:40–04:00. Two Vergabe pilot users (capped
comparison `bfd1a2d`). Rollback for any applied change is the owner's previous
manifest revision (CUST-WP-0071-T04 retains the receipts).

## Node budget (CPU, m)

| Item | Current | Proposed |
| --- | ---: | ---: |
| Steady pod requests (incl. Vergabe 60m provisional) | 3605 | 2400 |
| Host / k3s control plane (no kube/system-reserved) | 900 | 900 |
| Release surge + migration hook (state-hub, one at a time) | 310 | 310 |
| Admitted factory/build execution (one runner job burst) | 250 | 250 |
| Maintenance CronJobs | 50 | 50 |
| Knative HPA scale-out headroom | 1520 | 230 |
| Pilot growth reserve — **SLOT, pending RAPPS-WP-0014-T04** | 0 | 200 |
| **Total, all events concurrent** | **6635** | **4340** |
| **Residual vs 4000m, all concurrent** | **−2635** | **−340** |
| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3860 |
| Residual, serialized | −2075 | **+140** |

Proposed changes: +660m in increases and −1865m in decreases, across 55 non-keep
rows (54 owner changes plus the Vergabe slot).

Memory (Mi): proposed requests 11704 + k3s 2604 + surge 864 + build 512 + pilot
slot 256 leaves +53 of 15993. It fits, but only just. The k3s figure is a single
sample.

**Reconciliation verdict:** the budget reconciles only conditionally, at +140m. That
needs three things: the lean decreases are accepted, releases, builds and Knative
scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before
releases), and the HPA ceilings are capped. With every event concurrent it is
−340m short. Measured demand plus margin does fit the 4 vCPU. **Procurement is
not indicated**; the remedy is to correct reservations. Once requests have come
down, railiance-cluster should make host operations explicit with
`kube-reserved`/`system-reserved` (~900m/2.5Gi). Applying that first would make
the current 3605m unschedulable.

## Vergabe slot

Vergabe stays at **60m request, provisional, pending the target replay**
(RAPPS-WP-0014-T04, separate worker). The replay namespace
`vergabe-sizing-20261021` (temporary 260m) appeared after the 06:08Z capture and
is excluded from the steady budget. Insert the replay result into
`vergabe_slot.replay_result` and the pilot-growth row of `build_review.py`, then
regenerate. A larger Vergabe request must fit inside the +140m serialized
residual together with the pilot-growth slot.

## Receiving owners

None of the 54 non-keep rows has a live receiving task. RAIL-KNATIVE-WP-0002,
RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are
finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover
admission and release, not sizing. The only live slot is RAPPS-WP-0014-T04 for
Vergabe. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving
tasks by owner repo"). Each repo needs a receiving task:

activity-core, audit-core, canned-prompts, core-hub, coulomb-social,
email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner
unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative,
railiance-cluster, railiance-enablement, railiance-forge, railiance-platform,
rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface,
sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.

Owners without a row in this table also need tasks for the measurement work:
rapp-telemetry (Forgejo recording rule; host/node series), railiance-cluster
(host/control-plane 7d series, kube/system-reserved), state-hub (preflight surge
constant) and railiance-apps (inter-hub latent 250m).

## Open gaps

- No 7-day host/control-plane series. The 900m figure is a 70-second sample.
- Single-node HPA ceilings are not reviewed by rail-knative.
- Vergabe target replay result (slot).
- Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
- Owner for `coulomb` namespace unconfirmed; flex-auth deployment source not
  available locally.
- Startup/restart and rollout behaviour at the lowered requests is unverified
  (CUST-WP-0071-T04 acceptance).