resource-control/docs/evidence/2026-10-21-node-allocation-review.md

# Node-level allocation review — 2026-10-21

Provider: resource-control, RESOURCE-WP-0008-T02/T03; beneficiary CUST-WP-0071-T03.
This is a review proposal for owners. It applies nothing, admits no capacity and
proposes no procurement. Application stays with each runtime owner (CUST-WP-0071-T04,
separate founder approval).

## Sources and method

Target identity verified before capture: context `default`, single k3s node
`239.62.205.92.host.secureserver.net` (92.205.62.239, control-plane/etcd,
v1.35.1+k3s1). All access was read-only (`kubectl get`, `get --raw` GETs).

Retained under `docs/evidence/2026-10-21-node-allocation-review/`:

| File | Content |
| --- | --- |
| `2026-10-21-allocation-reconcile.{json,md}` | Existing Custodian collector `scripts/reconcile_railiance_allocation.py` (railiance-cluster observe + STATE-WP-0091 preflight), captured 06:08:20Z, written here via `--output-dir` |
| `2026-10-21-telemetry-7d.json` | Exact PromQL and responses, 7d window, 5m step, in-cluster `telemetry-prometheus` via the API service proxy (oldest sample 7.26 days) |
| `2026-10-21-workload-usage.json` | Per-workload join of live pod requests/limits with per-pod 7d p95/peak CPU, memory, throttling, restarts (`join.py`) |
| `2026-10-21-supplementary-observation.json` | kubelet summary samples, kubelet reservations, HPA, rollout surge, CronJobs, stale pods, empty namespaces, concurrent benchmark |
| `2026-10-21-node-budget.{json,md}` | Generated proposals, owners, receiving records and node budget (`build_review.py`, no cluster access) |

Workload p95 values are sums of per-pod p95 (an upper bound for multi-pod
workloads). Terminated Job pods appear only in namespace totals.

## Current state

- 4000m allocatable, 3605m scheduled requests, 0 pending, 395m arithmetic residual
  (unchanged since 2026-10-19). Memory: 9856Mi requested of 15993Mi.
- Measured demand: all containers 7d p95 966m, peak 1740m. Host/k3s outside pods
  677–871m across eight kubelet samples (k3s server 415–504m, 2604Mi). Measured
  demand is ~1.8 cores at p95 and ~2.6 cores at peak: **it fits the node.**
- The kubelet sets no `systemReserved`/`kubeReserved`, so the scheduler may hand
  all 4000m to pods while k3s itself needs ~0.9 core. Requests total 3.7× measured
  pod p95. The shortfall below is reservation inflation, not missing CPU.

## Priority findings

1. **Knative/Kourier.** Requests are already 220m in total (RAIL-KNATIVE-WP-0002,
   finished); the "~1000m" figure in CUST-WP-0071 is stale. Usage is p95 ~12m.
   The open risk is HPA ceilings on a single node: activator max 20, gateway 10 and
   webhook 5 can claim up to 1520m more requests. Proposal: cap them at 3/3/2
   (230m) and trim requests to the rule floor (220m→60m). Owner rail-knative.
2. **Forgejo.** The gitea main container has no requests or limits. Its 100m is
   the init container's. Usage is p95 133m, peak 377m, memory p95 956Mi against a
   128Mi request. The runner (runner + dind) has no requests or limits; namespace
   peak is 1033m. Proposal: gitea 200m/1104Mi plus a memory limit, runner
   125m/496Mi with a dind limit (~1500m/2Gi) and one concurrent job. The Prometheus
   request recording for `forgejo` still returns no series (missing since
   2026-09-11). Owners railiance-forge and rapp-telemetry.
3. **Zero-request workloads (13).** Eleven get small requests: state-hub-db 70m
   (p95 37m), target-revenue-pg 50m, forgejo-runner 125m, traefik 10m, cert-manager
   3×10m, ESO 3×10m, local-path 10m. The two k3s svclb DaemonSets (~0m) stay at
   zero as an accepted, documented exception.
4. **Stale probe/drill allocations.** `coulomb/ihp-railiance-probe` reserves 50m
   while in ImagePullBackOff for 21 days from legacy registry 92.205.130.254, with
   two more stale pods. Proposal: retire or scale to zero by owner decision. The
   owner is unconfirmed: namespace-ownership says unknown, and the
   ihp-railiance-probe repo's workplans are archived. `platform-pg-drill` no longer
   exists. `openbao-migration-drill` and `cert-manager-test` are empty. platform-pg-2
   is an overflow admission cluster, not a drill. Failed pods in forgejo
   (105d/109d) and telemetry (smtp-proof, 22d) reserve nothing; they are hygiene
   only. `inter-hub` sits at 0 replicas with a latent 250m request (Haskell
   Inter-Hub is retired).
5. **Under-reserved and throttled.** state-hub API p95 258m at a 250m request:
   proposal 390m (p95×1.5). Prometheus p95 134m at 100m: proposal 205m/1152Mi.
   activity-core and grafana memory requests sit far below p95. sso is 53.7%
   CFS-throttled and playwright 23.8%: the owners should review those CPU limits.
6. **Release surge model.** STATE-WP-0091 preflight models a 100m API surge, but
   the live API pod requests 250m. A state-hub release needs ~310m, not 160m.
   Owner state-hub (STATE-WP-0091 is finished).

## Rule and assumptions

CPU proposal = max(floor, p95×1.5, peak×0.5), rounded up to 5m. Floors are 50m
per database pod, 10m per critical-path pod and 5m otherwise. A change is
proposed only when |Δ| ≥ 15m or the current request is zero. Overrides: coredns
50m, state-hub 390m, the probe 0m, the svclb DaemonSets 0m, and Vergabe 60m.
Memory request = p95 working set ×1.15 when |Δ| ≥ 64Mi; limits are unchanged.
Requests set CFS weight and placement only, so limits keep burst room. A lower
request reduces a workload's share only under full contention. Database owners
check shared_buffers before lowering a database memory request.

Concurrency: one Forgejo build job at a time. One release at a time (largest:
state-hub). CronJobs overlap 02:40–04:00. Two Vergabe pilot users (capped
comparison `bfd1a2d`). Rollback for any applied change is the owner's previous
manifest revision (CUST-WP-0071-T04 retains the receipts).

## Node budget (CPU, m)

| Item | Current | Proposed |
| --- | ---: | ---: |
| Steady pod requests (incl. Vergabe 60m, apps-pg kept at 100m) | 3605 | 2450 |
| Host / k3s control plane (no kube/system-reserved) | 900 | 900 |
| Release surge + migration hook (state-hub, one at a time) | 310 | 310 |
| Admitted factory/build execution (one runner job burst) | 250 | 250 |
| Maintenance CronJobs | 50 | 50 |
| Knative HPA scale-out headroom | 1520 | 230 |
| Pilot growth reserve (next tenant/user growth; Vergabe is in steady requests) | 0 | 200 |
| **Total, all events concurrent** | **6635** | **4390** |
| **Residual vs 4000m, all concurrent** | **−2635** | **−390** |
| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3910 |
| Residual, serialized | −2075 | **+90** |

Proposed changes: +660m increases and −1815m decreases, across 54 non-keep rows.

Memory (Mi, allocatable 15993; k3s 2604 single sample, surge 864, build 512,
pilot growth 256):

| Memory | Current | Proposed |
| --- | ---: | ---: |
| Pod requests | 9856 | 12104 |
| Residual, all concurrent | +2157 | **−347** |
| Residual, serialized (max of surge/build) | +2669 | **+165** |

The current memory residual looks comfortable only because 13 zero-request
workloads hide about 1.3Gi of working set. The proposed requests include the
Vergabe raise (+256Mi), and apps-pg keeps its 256Mi.

**Reconciliation verdict:** the budget reconciles only conditionally, at +90m CPU and +165Mi memory. That
needs three things: the lean decreases are accepted, releases, builds and Knative
scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before
releases), and the HPA ceilings are capped. With every event concurrent it is
−390m CPU and −347Mi memory short. Measured demand plus margin does fit the 4 vCPU. **Procurement is
not indicated**; the remedy is to correct reservations. Once requests have come
down, railiance-cluster should make host operations explicit with
`kube-reserved`/`system-reserved` (~900m/2.5Gi). Applying that first would make
the current 3605m unschedulable.

## Vergabe slot (filled 2026-10-21 from the target replay)

Source: railiance-apps `1af6d82`,
`docs/evidence/2026-10-21-vergabe-target-replay/` (RAPPS-WP-0014-T04). The replay
ran the deployed digest on the target node: 3 workers, 60m/256Mi request and
1000m/1Gi limit, in an isolated namespace that has since been torn down.

- 3675 ordinary operations with 0 errors; ordinary p95 0.192s; transfer p95 0.076s.
  The restart recovered in 19s.
- App CPU was 775m mean, 998m p95 and 1020m peak, throttled in 36% of periods
  (66% during the burst phase). Memory peaked at 483MiB with no OOM.
- Incremental PostgreSQL demand was about 320m mean and 480m p95, measured on an
  isolated PostgreSQL pod rather than shared apps-pg.
- The load was a ~25 req/s closed loop, far above what two human users generate.

Decision (T02):
- **Keep the 60m CPU request and the 1000m/1Gi limits.**
- **Raise the memory request from 256Mi to 512Mi** in the next wave; the owner is
  railiance-apps.
- **Keep apps-pg at 100m.** The rule would have proposed 50m; the replay justifies
  keeping it.
- The ~0.3–0.5 core of incremental apps-pg load at synthetic rates is burst usage
  under apps-pg's 1000m limit, not a reservation. Revisit if apps-pg CPU stays
  above 600m.

At the synthetic rate the pilot used about 1.1 core inside its limits, and node
free CPU at p5 fell to about 150m. That was a stress case, not the two-user
demand.

The replay does not establish:
- production settings (SSO, ingress/TLS)
- contention on the shared apps-pg cluster
- observed human demand
- busy-hour node contention (CI, backups)
- human/recovery acceptance (RAPPS-WP-0014-T03)

## Receiving owners

None of the 54 non-keep rows has a live receiving task yet. Wave 1 tasks (founder
decision 2026-10-21) are being prepared in rail-knative, railiance-forge and
state-hub, with rapp-telemetry following. Wave 2 is one task per remaining repo. RAIL-KNATIVE-WP-0002,
RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are
finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover
admission and release, not sizing. RAPPS-WP-0014-T04 delivered the Vergabe replay; its memory raise needs a
railiance-apps receiving task. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving
tasks by owner repo"). Each repo needs a receiving task:

activity-core, audit-core, canned-prompts, core-hub, coulomb-social,
email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner
unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative,
railiance-apps, railiance-cluster, railiance-enablement, railiance-forge, railiance-platform,
rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface,
sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.

Owners without a row in this table also need tasks for the measurement work:
rapp-telemetry (Forgejo recording rule; host/node series), railiance-cluster
(host/control-plane 7d series, kube/system-reserved), state-hub (preflight surge
constant) and railiance-apps (inter-hub latent 250m).

## Open gaps

- No 7-day host/control-plane series. The 900m figure is a 70-second sample.
- Single-node HPA ceilings are not reviewed by rail-knative.
- Shared apps-pg behaviour under pilot load (the replay used an isolated database).
- Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
- Owner for `coulomb` namespace unconfirmed; flex-auth deployment source not
  available locally.
- Startup/restart and rollout behaviour at the lowered requests is unverified
  (CUST-WP-0071-T04 acceptance).