Review node-level allocation budget for CUST-WP-0071-T03
Fresh read-only railiance01 capture (06:08Z) via the Custodian collector, 7d Prometheus usage, per-workload proposals with owners, and a node budget. The rule-based proposal reconciles at +140m only with serialized release/build/HPA events (-340m all concurrent); no procurement. Vergabe stays 60m provisional pending RAPPS-WP-0014-T04. T03 stays in progress: 54 changes lack receiving tasks in 26 owner repos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 8340@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c
This commit is contained in:
parent
e3e996fc95
commit
e72c5de92f
13 changed files with 26468 additions and 3 deletions
|
|
@ -52,4 +52,4 @@
|
|||
| task | RESOURCE-WP-0007-T01 | done | — | workplans/RESOURCE-WP-0007-state-hub-release-margin.md |
|
||||
| task | RESOURCE-WP-0008-T01 | done | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T02 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T03 | todo | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T03 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
|
|
|
|||
163
docs/evidence/2026-10-21-node-allocation-review.md
Normal file
163
docs/evidence/2026-10-21-node-allocation-review.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# Node-level allocation review — 2026-10-21
|
||||
|
||||
Provider: resource-control, RESOURCE-WP-0008-T02/T03; beneficiary CUST-WP-0071-T03.
|
||||
This is a review proposal for owners. It applies nothing, admits no capacity and
|
||||
proposes no procurement. Application stays with each runtime owner (CUST-WP-0071-T04,
|
||||
separate founder approval).
|
||||
|
||||
## Sources and method
|
||||
|
||||
Target identity verified before capture: context `default`, single k3s node
|
||||
`239.62.205.92.host.secureserver.net` (92.205.62.239, control-plane/etcd,
|
||||
v1.35.1+k3s1). All access was read-only (`kubectl get`, `get --raw` GETs).
|
||||
|
||||
Retained under `docs/evidence/2026-10-21-node-allocation-review/`:
|
||||
|
||||
| File | Content |
|
||||
| --- | --- |
|
||||
| `2026-10-21-allocation-reconcile.{json,md}` | Existing Custodian collector `scripts/reconcile_railiance_allocation.py` (railiance-cluster observe + STATE-WP-0091 preflight), captured 06:08:20Z, written here via `--output-dir` |
|
||||
| `2026-10-21-telemetry-7d.json` | Exact PromQL and responses, 7d window, 5m step, in-cluster `telemetry-prometheus` via the API service proxy (oldest sample 7.26 days) |
|
||||
| `2026-10-21-workload-usage.json` | Per-workload join of live pod requests/limits with per-pod 7d p95/peak CPU, memory, throttling, restarts (`join.py`) |
|
||||
| `2026-10-21-supplementary-observation.json` | kubelet summary samples, kubelet reservations, HPA, rollout surge, CronJobs, stale pods, empty namespaces, concurrent benchmark |
|
||||
| `2026-10-21-node-budget.{json,md}` | Generated proposals, owners, receiving records and node budget (`build_review.py`, no cluster access) |
|
||||
|
||||
Workload p95 values are sums of per-pod p95 (an upper bound for multi-pod
|
||||
workloads). Terminated Job pods appear only in namespace totals.
|
||||
|
||||
## Current state
|
||||
|
||||
- 4000m allocatable, 3605m scheduled requests, 0 pending, 395m arithmetic residual
|
||||
(unchanged since 2026-10-19). Memory: 9856Mi requested of 15993Mi.
|
||||
- Measured demand: all containers 7d p95 966m, peak 1740m. Host/k3s outside pods
|
||||
677–871m across eight kubelet samples (k3s server 415–504m, 2604Mi). Measured
|
||||
demand is ~1.8 cores at p95 and ~2.6 cores at peak: **it fits the node.**
|
||||
- The kubelet sets no `systemReserved`/`kubeReserved`, so the scheduler may hand
|
||||
all 4000m to pods while k3s itself needs ~0.9 core. Requests total 3.7× measured
|
||||
pod p95. The shortfall below is reservation inflation, not missing CPU.
|
||||
|
||||
## Priority findings
|
||||
|
||||
1. **Knative/Kourier.** Requests are already 220m in total (RAIL-KNATIVE-WP-0002,
|
||||
finished); the "~1000m" figure in CUST-WP-0071 is stale. Usage is p95 ~12m.
|
||||
The open risk is HPA ceilings on a single node: activator max 20, gateway 10 and
|
||||
webhook 5 can claim up to 1520m more requests. Proposal: cap them at 3/3/2
|
||||
(230m) and trim requests to the rule floor (220m→60m). Owner rail-knative.
|
||||
2. **Forgejo.** The gitea main container has no requests or limits. Its 100m is
|
||||
the init container's. Usage is p95 133m, peak 377m, memory p95 956Mi against a
|
||||
128Mi request. The runner (runner + dind) has no requests or limits; namespace
|
||||
peak is 1033m. Proposal: gitea 200m/1104Mi plus a memory limit, runner
|
||||
125m/496Mi with a dind limit (~1500m/2Gi) and one concurrent job. The Prometheus
|
||||
request recording for `forgejo` still returns no series (missing since
|
||||
2026-09-11). Owners railiance-forge and rapp-telemetry.
|
||||
3. **Zero-request workloads (13).** Eleven get small requests: state-hub-db 70m
|
||||
(p95 37m), target-revenue-pg 50m, forgejo-runner 125m, traefik 10m, cert-manager
|
||||
3×10m, ESO 3×10m, local-path 10m. The two k3s svclb DaemonSets (~0m) stay at
|
||||
zero as an accepted, documented exception.
|
||||
4. **Stale probe/drill allocations.** `coulomb/ihp-railiance-probe` reserves 50m
|
||||
while in ImagePullBackOff for 21 days from legacy registry 92.205.130.254, with
|
||||
two more stale pods. Proposal: retire or scale to zero by owner decision. The
|
||||
owner is unconfirmed: namespace-ownership says unknown, and the
|
||||
ihp-railiance-probe repo's workplans are archived. `platform-pg-drill` no longer
|
||||
exists. `openbao-migration-drill` and `cert-manager-test` are empty. platform-pg-2
|
||||
is an overflow admission cluster, not a drill. Failed pods in forgejo
|
||||
(105d/109d) and telemetry (smtp-proof, 22d) reserve nothing; they are hygiene
|
||||
only. `inter-hub` sits at 0 replicas with a latent 250m request (Haskell
|
||||
Inter-Hub is retired).
|
||||
5. **Under-reserved and throttled.** state-hub API p95 258m at a 250m request:
|
||||
proposal 390m (p95×1.5). Prometheus p95 134m at 100m: proposal 205m/1152Mi.
|
||||
activity-core and grafana memory requests sit far below p95. sso is 53.7%
|
||||
CFS-throttled and playwright 23.8%: the owners should review those CPU limits.
|
||||
6. **Release surge model.** STATE-WP-0091 preflight models a 100m API surge, but
|
||||
the live API pod requests 250m. A state-hub release needs ~310m, not 160m.
|
||||
Owner state-hub (STATE-WP-0091 is finished).
|
||||
|
||||
## Rule and assumptions
|
||||
|
||||
CPU proposal = max(floor, p95×1.5, peak×0.5), rounded up to 5m. Floors are 50m
|
||||
per database pod, 10m per critical-path pod and 5m otherwise. A change is
|
||||
proposed only when |Δ| ≥ 15m or the current request is zero. Overrides: coredns
|
||||
50m, state-hub 390m, the probe 0m, the svclb DaemonSets 0m, and Vergabe 60m.
|
||||
Memory request = p95 working set ×1.15 when |Δ| ≥ 64Mi; limits are unchanged.
|
||||
Requests set CFS weight and placement only, so limits keep burst room. A lower
|
||||
request reduces a workload's share only under full contention. Database owners
|
||||
check shared_buffers before lowering a database memory request.
|
||||
|
||||
Concurrency: one Forgejo build job at a time. One release at a time (largest:
|
||||
state-hub). CronJobs overlap 02:40–04:00. Two Vergabe pilot users (capped
|
||||
comparison `bfd1a2d`). Rollback for any applied change is the owner's previous
|
||||
manifest revision (CUST-WP-0071-T04 retains the receipts).
|
||||
|
||||
## Node budget (CPU, m)
|
||||
|
||||
| Item | Current | Proposed |
|
||||
| --- | ---: | ---: |
|
||||
| Steady pod requests (incl. Vergabe 60m provisional) | 3605 | 2400 |
|
||||
| Host / k3s control plane (no kube/system-reserved) | 900 | 900 |
|
||||
| Release surge + migration hook (state-hub, one at a time) | 310 | 310 |
|
||||
| Admitted factory/build execution (one runner job burst) | 250 | 250 |
|
||||
| Maintenance CronJobs | 50 | 50 |
|
||||
| Knative HPA scale-out headroom | 1520 | 230 |
|
||||
| Pilot growth reserve — **SLOT, pending RAPPS-WP-0014-T04** | 0 | 200 |
|
||||
| **Total, all events concurrent** | **6635** | **4340** |
|
||||
| **Residual vs 4000m, all concurrent** | **−2635** | **−340** |
|
||||
| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3860 |
|
||||
| Residual, serialized | −2075 | **+140** |
|
||||
|
||||
Proposed changes: +660m in increases and −1865m in decreases, across 55 non-keep
|
||||
rows (54 owner changes plus the Vergabe slot).
|
||||
|
||||
Memory (Mi): proposed requests 11704 + k3s 2604 + surge 864 + build 512 + pilot
|
||||
slot 256 leaves +53 of 15993. It fits, but only just. The k3s figure is a single
|
||||
sample.
|
||||
|
||||
**Reconciliation verdict:** the budget reconciles only conditionally, at +140m. That
|
||||
needs three things: the lean decreases are accepted, releases, builds and Knative
|
||||
scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before
|
||||
releases), and the HPA ceilings are capped. With every event concurrent it is
|
||||
−340m short. Measured demand plus margin does fit the 4 vCPU. **Procurement is
|
||||
not indicated**; the remedy is to correct reservations. Once requests have come
|
||||
down, railiance-cluster should make host operations explicit with
|
||||
`kube-reserved`/`system-reserved` (~900m/2.5Gi). Applying that first would make
|
||||
the current 3605m unschedulable.
|
||||
|
||||
## Vergabe slot
|
||||
|
||||
Vergabe stays at **60m request, provisional, pending the target replay**
|
||||
(RAPPS-WP-0014-T04, separate worker). The replay namespace
|
||||
`vergabe-sizing-20261021` (temporary 260m) appeared after the 06:08Z capture and
|
||||
is excluded from the steady budget. Insert the replay result into
|
||||
`vergabe_slot.replay_result` and the pilot-growth row of `build_review.py`, then
|
||||
regenerate. A larger Vergabe request must fit inside the +140m serialized
|
||||
residual together with the pilot-growth slot.
|
||||
|
||||
## Receiving owners
|
||||
|
||||
None of the 54 non-keep rows has a live receiving task. RAIL-KNATIVE-WP-0002,
|
||||
RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are
|
||||
finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover
|
||||
admission and release, not sizing. The only live slot is RAPPS-WP-0014-T04 for
|
||||
Vergabe. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving
|
||||
tasks by owner repo"). Each repo needs a receiving task:
|
||||
|
||||
activity-core, audit-core, canned-prompts, core-hub, coulomb-social,
|
||||
email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner
|
||||
unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative,
|
||||
railiance-cluster, railiance-enablement, railiance-forge, railiance-platform,
|
||||
rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface,
|
||||
sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.
|
||||
|
||||
Owners without a row in this table also need tasks for the measurement work:
|
||||
rapp-telemetry (Forgejo recording rule; host/node series), railiance-cluster
|
||||
(host/control-plane 7d series, kube/system-reserved), state-hub (preflight surge
|
||||
constant) and railiance-apps (inter-hub latent 250m).
|
||||
|
||||
## Open gaps
|
||||
|
||||
- No 7-day host/control-plane series. The 900m figure is a 70-second sample.
|
||||
- Single-node HPA ceilings are not reviewed by rail-knative.
|
||||
- Vergabe target replay result (slot).
|
||||
- Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
|
||||
- Owner for `coulomb` namespace unconfirmed; flex-auth deployment source not
|
||||
available locally.
|
||||
- Startup/restart and rollout behaviour at the lowered requests is unverified
|
||||
(CUST-WP-0071-T04 acceptance).
|
||||
|
|
@ -0,0 +1,757 @@
|
|||
{
|
||||
"schema": "custodian.allocation-reconcile.v1",
|
||||
"workplan_id": "CUST-WP-0071-T01",
|
||||
"captured_at": "2026-10-21T06:08:20Z",
|
||||
"cluster_observation_schema": "railiance.cluster-resource-observation.v1",
|
||||
"count_host_and_cluster_cpus_once": true,
|
||||
"host": {
|
||||
"owner": "railiance-cluster",
|
||||
"resource_id": "resource:hosteurope:railiance01",
|
||||
"same_cpus_as_cluster": true,
|
||||
"cluster_resource_id": "resource:railiance:reef-railiance:k3s"
|
||||
},
|
||||
"capacity_cpu_m": 4000,
|
||||
"scheduled_request_cpu_m": 3605,
|
||||
"pending_unscheduled_cpu_m": 0,
|
||||
"residual_cpu_m": 395,
|
||||
"not_a_scheduling_guarantee": true,
|
||||
"workloads": [
|
||||
{
|
||||
"namespace": "state-hub",
|
||||
"workload": "railiance-apps",
|
||||
"pods": 2,
|
||||
"cpu_request_m": 260,
|
||||
"memory_request_bytes": 671088640,
|
||||
"owner": "state-hub",
|
||||
"service": "state-hub",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "core-hub",
|
||||
"workload": "rapp-core-hub",
|
||||
"pods": 3,
|
||||
"cpu_request_m": 200,
|
||||
"memory_request_bytes": 939524096,
|
||||
"owner": "core-hub",
|
||||
"service": "core-hub",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "forgejo-db",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 200,
|
||||
"memory_request_bytes": 536870912,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "sso",
|
||||
"workload": "net-kingdom-sso-mfa",
|
||||
"pods": 4,
|
||||
"cpu_request_m": 150,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "net-kingdom",
|
||||
"service": "keycape-authelia",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "flex-auth",
|
||||
"workload": "flex-auth",
|
||||
"pods": 9,
|
||||
"cpu_request_m": 145,
|
||||
"memory_request_bytes": 301989888,
|
||||
"owner": "flex-auth",
|
||||
"service": "flex-auth",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "knative-serving",
|
||||
"workload": "knative-serving",
|
||||
"pods": 4,
|
||||
"cpu_request_m": 140,
|
||||
"memory_request_bytes": 377487360,
|
||||
"owner": "rail-knative",
|
||||
"service": "knative-serving",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "mfa",
|
||||
"workload": "net-kingdom-sso-mfa",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 110,
|
||||
"memory_request_bytes": 402653184,
|
||||
"owner": "net-kingdom",
|
||||
"service": "lldap-mfa",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "reuse",
|
||||
"workload": "reuse-surface",
|
||||
"pods": 2,
|
||||
"cpu_request_m": 110,
|
||||
"memory_request_bytes": 301989888,
|
||||
"owner": "reuse-surface",
|
||||
"service": "reuse-surface",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "activity-core",
|
||||
"workload": "activity-core",
|
||||
"pods": 10,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "activity-core",
|
||||
"service": "activity-core",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "cnpg-system",
|
||||
"workload": "cloudnative-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 104857600,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "cloudnative-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "coulomb-social",
|
||||
"workload": "coulomb-social",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "coulomb-social",
|
||||
"service": "coulomb-social",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "apps-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "net-kingdom-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "platform-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "platform-pg-2",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "forgejo",
|
||||
"workload": "gitea",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 134217728,
|
||||
"owner": "railiance-forge",
|
||||
"service": "forgejo",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "coredns-7bdb54f89",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 73400320,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "metrics-server-786d997795",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 73400320,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "openbao",
|
||||
"workload": "openbao",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "railiance-platform",
|
||||
"service": "openbao",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "playwright",
|
||||
"workload": "rapp-playwright",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "railiance-platform",
|
||||
"service": "rapp-playwright",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "telemetry",
|
||||
"workload": "prometheus",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 536870912,
|
||||
"owner": "rapp-telemetry",
|
||||
"service": "prometheus-grafana",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "user-engine",
|
||||
"workload": "user-engine-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "user-engine",
|
||||
"service": "user-engine",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "telemetry",
|
||||
"workload": "grafana",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 70,
|
||||
"memory_request_bytes": 201326592,
|
||||
"owner": "rapp-telemetry",
|
||||
"service": "prometheus-grafana",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "vergabe-demo-company",
|
||||
"workload": "vergabe-teilnahme",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 60,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "railiance-apps",
|
||||
"service": "vergabe-teilnahme",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "argocd",
|
||||
"workload": "argocd-application-controller",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "railiance-enablement",
|
||||
"service": "argocd",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "audit-core",
|
||||
"workload": "audit-core",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "audit-core",
|
||||
"service": "audit-core",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "coulomb",
|
||||
"workload": "ihp-railiance-probe",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 134217728,
|
||||
"owner": "unknown",
|
||||
"service": "ihp-railiance-probe",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "issue-core",
|
||||
"workload": "issue-core",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 134217728,
|
||||
"owner": "issue-core",
|
||||
"service": "issue-core",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kourier-system",
|
||||
"workload": "3scale-kourier-gateway",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 209715200,
|
||||
"owner": "rail-knative",
|
||||
"service": "kourier",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "target-revenue",
|
||||
"workload": "target-revenue",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 268435456,
|
||||
"owner": "target-revenue",
|
||||
"service": "target-revenue",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "user-engine",
|
||||
"workload": "user-engine",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 50,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "user-engine",
|
||||
"service": "user-engine",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "knative-serving",
|
||||
"workload": "net-kourier-controller",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 30,
|
||||
"memory_request_bytes": 209715200,
|
||||
"owner": "rail-knative",
|
||||
"service": "knative-serving",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "argocd",
|
||||
"workload": "argocd-repo-server",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 134217728,
|
||||
"owner": "railiance-enablement",
|
||||
"service": "argocd",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "canned-prompts",
|
||||
"workload": "canned-prompts",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 100663296,
|
||||
"owner": "canned-prompts",
|
||||
"service": "canned-prompts",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "email-connect",
|
||||
"workload": "email-connect",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "email-connect",
|
||||
"service": "email-connect",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "openbao",
|
||||
"workload": "railiance-platform",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 33554432,
|
||||
"owner": "railiance-platform",
|
||||
"service": "openbao",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "rein-aharness",
|
||||
"workload": "rein-aharness",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "rein-aharness",
|
||||
"service": "rein-aharness",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "sbom-nexus",
|
||||
"workload": "sbom-nexus",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "sbom-nexus",
|
||||
"service": "sbom-nexus",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "telemetry",
|
||||
"workload": "alertmanager",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "rapp-telemetry",
|
||||
"service": "prometheus-grafana",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "telemetry",
|
||||
"workload": "kube-state-metrics",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "rapp-telemetry",
|
||||
"service": "prometheus-grafana",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "telemetry",
|
||||
"workload": "rapp-telemetry",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 134217728,
|
||||
"owner": "rapp-telemetry",
|
||||
"service": "prometheus-grafana",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "tenant-engine",
|
||||
"workload": "tenant-engine",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 25,
|
||||
"memory_request_bytes": 50331648,
|
||||
"owner": "tenant-engine",
|
||||
"service": "tenant-engine",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "mas",
|
||||
"workload": "mas-mx",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 20,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "mail-address-semantics",
|
||||
"service": "mas-live-test",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "rapp-qonto",
|
||||
"workload": "rapp-qonto-audit-collector",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 20,
|
||||
"memory_request_bytes": 33554432,
|
||||
"owner": "rapp-qonto",
|
||||
"service": "qonto",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "rapp-qonto-egress",
|
||||
"workload": "qonto-egress-proxy",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 20,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "rapp-qonto",
|
||||
"service": "qonto-egress",
|
||||
"tenant": "tenant:friendly:binky"
|
||||
},
|
||||
{
|
||||
"namespace": "activity-core",
|
||||
"workload": "actcore-temporal-ui-tls-2-1888679036-1756117428",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 10,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "activity-core",
|
||||
"service": "activity-core",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "approval-engine",
|
||||
"workload": "approval-engine",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 10,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "approval-engine",
|
||||
"service": "approval-engine",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "argocd",
|
||||
"workload": "argocd-applicationset-controller",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 10,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "railiance-enablement",
|
||||
"service": "argocd",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "argocd",
|
||||
"workload": "argocd-redis",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 10,
|
||||
"memory_request_bytes": 33554432,
|
||||
"owner": "railiance-enablement",
|
||||
"service": "argocd",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "policy-nexus",
|
||||
"workload": "policy-nexus",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 10,
|
||||
"memory_request_bytes": 33554432,
|
||||
"owner": "policy-nexus",
|
||||
"service": "policy-nexus",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "bao-notice",
|
||||
"workload": "bao-notice",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 5,
|
||||
"memory_request_bytes": 16777216,
|
||||
"owner": "railiance-platform",
|
||||
"service": "bao-notice",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "informed-decision",
|
||||
"workload": "informed-decision",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 5,
|
||||
"memory_request_bytes": 67108864,
|
||||
"owner": "informed-decision",
|
||||
"service": "informed-decision",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "mas",
|
||||
"workload": "mas-statute",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 5,
|
||||
"memory_request_bytes": 33554432,
|
||||
"owner": "mail-address-semantics",
|
||||
"service": "mas-live-test",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "state-hub",
|
||||
"workload": "state-hub",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 5,
|
||||
"memory_request_bytes": 25165824,
|
||||
"owner": "state-hub",
|
||||
"service": "state-hub",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "cert-manager",
|
||||
"workload": "cainjector",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "cert-manager",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "cert-manager",
|
||||
"workload": "cert-manager",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "cert-manager",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "cert-manager",
|
||||
"workload": "webhook",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "cert-manager",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "state-hub-db",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "rapp-postgres",
|
||||
"service": "apps-pg",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "external-secrets",
|
||||
"workload": "external-secrets",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-platform",
|
||||
"service": "external-secrets",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "external-secrets",
|
||||
"workload": "external-secrets-cert-controller",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-platform",
|
||||
"service": "external-secrets",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "external-secrets",
|
||||
"workload": "external-secrets-webhook",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-platform",
|
||||
"service": "external-secrets",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "forgejo",
|
||||
"workload": "forgejo-runner",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-forge",
|
||||
"service": "forgejo",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "local-path-provisioner",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "svclb-mas-smtp-d86db2b1",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "svclb-traefik-0c8aecaf",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "kube-system",
|
||||
"workload": "traefik",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "railiance-cluster",
|
||||
"service": "k3s-control-plane",
|
||||
"tenant": "unknown"
|
||||
},
|
||||
{
|
||||
"namespace": "target-revenue",
|
||||
"workload": "target-revenue-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 0,
|
||||
"memory_request_bytes": 0,
|
||||
"owner": "target-revenue",
|
||||
"service": "target-revenue",
|
||||
"tenant": "unknown"
|
||||
}
|
||||
],
|
||||
"unknown_namespaces": [],
|
||||
"zero_request_workloads": [
|
||||
"cert-manager/cainjector",
|
||||
"cert-manager/cert-manager",
|
||||
"cert-manager/webhook",
|
||||
"databases/state-hub-db",
|
||||
"external-secrets/external-secrets",
|
||||
"external-secrets/external-secrets-cert-controller",
|
||||
"external-secrets/external-secrets-webhook",
|
||||
"forgejo/forgejo-runner",
|
||||
"kube-system/local-path-provisioner",
|
||||
"kube-system/svclb-mas-smtp-d86db2b1",
|
||||
"kube-system/svclb-traefik-0c8aecaf",
|
||||
"kube-system/traefik",
|
||||
"target-revenue/target-revenue-pg"
|
||||
],
|
||||
"pending_unscheduled": [],
|
||||
"measurement_gaps": [
|
||||
"node 239.62.205.92.host.secureserver.net lacks independent region/zone labels"
|
||||
],
|
||||
"state_hub_preflight_observation": {
|
||||
"schema": "state-hub.release-headroom-preflight.v1-input",
|
||||
"observed_at": "2026-10-21T06:08:20Z",
|
||||
"freshness_seconds": 0,
|
||||
"nodes": [
|
||||
{
|
||||
"name": "239.62.205.92.host.secureserver.net",
|
||||
"ready": true,
|
||||
"unschedulable": false,
|
||||
"allocatable_cpu_m": 4000,
|
||||
"allocatable_memory_bytes": 16770080768,
|
||||
"allocated_cpu_m": 3605,
|
||||
"allocated_memory_bytes": 10334765056
|
||||
}
|
||||
],
|
||||
"pending_unrelated": []
|
||||
},
|
||||
"signal_notes": [
|
||||
"Host resource:hosteurope:railiance01 and cluster resource:railiance:reef-railiance:k3s are the same 4 vCPU; do not add them.",
|
||||
"Prometheus namespace_cpu:kube_pod_container_resource_requests:sum omitted Forgejo 100m on 2026-09-11; Kubernetes API is the scheduler-effective source.",
|
||||
"node-exporter was disabled; host CPU usage is not a reservation and is not treated as spare capacity.",
|
||||
"Zero-request pods are demand, not zero. Missing metrics are listed as gaps, not zeros.",
|
||||
"STATE-WP-0091 preflight consumes state_hub_preflight_observation; residual millicores are not admission."
|
||||
],
|
||||
"state_hub_preflight": {
|
||||
"schema": "state-hub.release-headroom-preflight.v1",
|
||||
"ok": true,
|
||||
"observed_at": "2026-10-21T06:08:20Z",
|
||||
"freshness_seconds": 0,
|
||||
"remaining_cpu_m": 395,
|
||||
"remaining_memory_bytes": 6435315712,
|
||||
"pending_unrelated_cpu_m": 0,
|
||||
"api_surge_cpu_m": 100,
|
||||
"mcp_surge_cpu_m": 10,
|
||||
"migrate_cpu_m": 50,
|
||||
"atomic": true,
|
||||
"reasons": [],
|
||||
"notes": [
|
||||
"Aggregate remaining millicores is not a kube-scheduler guarantee.",
|
||||
"Preflight does not lower requests and does not start Helm."
|
||||
],
|
||||
"hook_order": [
|
||||
"pre-upgrade migrate job (helm.sh/hook-weight -5)",
|
||||
"API RollingUpdate maxSurge=1 maxUnavailable=0",
|
||||
"MCP RollingUpdate maxSurge=1 maxUnavailable=0"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# Railiance allocation reconcile — 2026-10-21T06:08:20Z
|
||||
|
||||
CUST-WP-0071-T01. Scheduler-effective requests from the Kubernetes API
|
||||
via `railiance-cluster` observe (RCLUSTER-WP-0014 / RAIL-BS-WP-0014).
|
||||
Host and cluster are the same 4 vCPU and are counted once.
|
||||
|
||||
- Capacity: 4000m
|
||||
- Scheduled requests: 3605m
|
||||
- Pending unscheduled: 0m
|
||||
- Residual (not a guarantee): 395m
|
||||
- Unknown namespaces: none
|
||||
- Zero-request workloads: 13
|
||||
|
||||
| Namespace | Workload | Owner | Tenant | CPU request (m) | Pods |
|
||||
|---|---|---|---|---:|---:|
|
||||
| state-hub | railiance-apps | state-hub | unknown | 260 | 2 |
|
||||
| core-hub | rapp-core-hub | core-hub | unknown | 200 | 3 |
|
||||
| databases | forgejo-db | rapp-postgres | unknown | 200 | 1 |
|
||||
| sso | net-kingdom-sso-mfa | net-kingdom | unknown | 150 | 4 |
|
||||
| flex-auth | flex-auth | flex-auth | unknown | 145 | 9 |
|
||||
| knative-serving | knative-serving | rail-knative | unknown | 140 | 4 |
|
||||
| mfa | net-kingdom-sso-mfa | net-kingdom | unknown | 110 | 1 |
|
||||
| reuse | reuse-surface | reuse-surface | unknown | 110 | 2 |
|
||||
| activity-core | activity-core | activity-core | unknown | 100 | 10 |
|
||||
| cnpg-system | cloudnative-pg | rapp-postgres | unknown | 100 | 1 |
|
||||
| coulomb-social | coulomb-social | coulomb-social | unknown | 100 | 1 |
|
||||
| databases | apps-pg | rapp-postgres | unknown | 100 | 1 |
|
||||
| databases | net-kingdom-pg | rapp-postgres | unknown | 100 | 1 |
|
||||
| databases | platform-pg | rapp-postgres | unknown | 100 | 1 |
|
||||
| databases | platform-pg-2 | rapp-postgres | unknown | 100 | 1 |
|
||||
| forgejo | gitea | railiance-forge | unknown | 100 | 1 |
|
||||
| kube-system | coredns-7bdb54f89 | railiance-cluster | unknown | 100 | 1 |
|
||||
| kube-system | metrics-server-786d997795 | railiance-cluster | unknown | 100 | 1 |
|
||||
| openbao | openbao | railiance-platform | unknown | 100 | 1 |
|
||||
| playwright | rapp-playwright | railiance-platform | unknown | 100 | 1 |
|
||||
| telemetry | prometheus | rapp-telemetry | unknown | 100 | 1 |
|
||||
| user-engine | user-engine-pg | user-engine | unknown | 100 | 1 |
|
||||
| telemetry | grafana | rapp-telemetry | unknown | 70 | 1 |
|
||||
| vergabe-demo-company | vergabe-teilnahme | railiance-apps | unknown | 60 | 1 |
|
||||
| argocd | argocd-application-controller | railiance-enablement | unknown | 50 | 1 |
|
||||
| audit-core | audit-core | audit-core | unknown | 50 | 1 |
|
||||
| coulomb | ihp-railiance-probe | unknown | unknown | 50 | 1 |
|
||||
| issue-core | issue-core | issue-core | unknown | 50 | 1 |
|
||||
| kourier-system | 3scale-kourier-gateway | rail-knative | unknown | 50 | 1 |
|
||||
| target-revenue | target-revenue | target-revenue | unknown | 50 | 1 |
|
||||
| user-engine | user-engine | user-engine | unknown | 50 | 1 |
|
||||
| knative-serving | net-kourier-controller | rail-knative | unknown | 30 | 1 |
|
||||
| argocd | argocd-repo-server | railiance-enablement | unknown | 25 | 1 |
|
||||
| canned-prompts | canned-prompts | canned-prompts | unknown | 25 | 1 |
|
||||
| email-connect | email-connect | email-connect | unknown | 25 | 1 |
|
||||
| openbao | railiance-platform | railiance-platform | unknown | 25 | 1 |
|
||||
| rein-aharness | rein-aharness | rein-aharness | unknown | 25 | 1 |
|
||||
| sbom-nexus | sbom-nexus | sbom-nexus | unknown | 25 | 1 |
|
||||
| telemetry | alertmanager | rapp-telemetry | unknown | 25 | 1 |
|
||||
| telemetry | kube-state-metrics | rapp-telemetry | unknown | 25 | 1 |
|
||||
| telemetry | rapp-telemetry | rapp-telemetry | unknown | 25 | 1 |
|
||||
| tenant-engine | tenant-engine | tenant-engine | unknown | 25 | 1 |
|
||||
| mas | mas-mx | mail-address-semantics | unknown | 20 | 1 |
|
||||
| rapp-qonto | rapp-qonto-audit-collector | rapp-qonto | unknown | 20 | 1 |
|
||||
| rapp-qonto-egress | qonto-egress-proxy | rapp-qonto | tenant:friendly:binky | 20 | 1 |
|
||||
| activity-core | actcore-temporal-ui-tls-2-1888679036-1756117428 | activity-core | unknown | 10 | 1 |
|
||||
| approval-engine | approval-engine | approval-engine | unknown | 10 | 1 |
|
||||
| argocd | argocd-applicationset-controller | railiance-enablement | unknown | 10 | 1 |
|
||||
| argocd | argocd-redis | railiance-enablement | unknown | 10 | 1 |
|
||||
| policy-nexus | policy-nexus | policy-nexus | unknown | 10 | 1 |
|
||||
| bao-notice | bao-notice | railiance-platform | unknown | 5 | 1 |
|
||||
| informed-decision | informed-decision | informed-decision | unknown | 5 | 1 |
|
||||
| mas | mas-statute | mail-address-semantics | unknown | 5 | 1 |
|
||||
| state-hub | state-hub | state-hub | unknown | 5 | 1 |
|
||||
| cert-manager | cainjector | railiance-cluster | unknown | 0 | 1 |
|
||||
| cert-manager | cert-manager | railiance-cluster | unknown | 0 | 1 |
|
||||
| cert-manager | webhook | railiance-cluster | unknown | 0 | 1 |
|
||||
| databases | state-hub-db | rapp-postgres | unknown | 0 | 1 |
|
||||
| external-secrets | external-secrets | railiance-platform | unknown | 0 | 1 |
|
||||
| external-secrets | external-secrets-cert-controller | railiance-platform | unknown | 0 | 1 |
|
||||
| external-secrets | external-secrets-webhook | railiance-platform | unknown | 0 | 1 |
|
||||
| forgejo | forgejo-runner | railiance-forge | unknown | 0 | 1 |
|
||||
| kube-system | local-path-provisioner | railiance-cluster | unknown | 0 | 1 |
|
||||
| kube-system | svclb-mas-smtp-d86db2b1 | railiance-cluster | unknown | 0 | 1 |
|
||||
| kube-system | svclb-traefik-0c8aecaf | railiance-cluster | unknown | 0 | 1 |
|
||||
| kube-system | traefik | railiance-cluster | unknown | 0 | 1 |
|
||||
| target-revenue | target-revenue-pg | target-revenue | unknown | 0 | 1 |
|
||||
|
||||
## STATE-WP-0091 preflight
|
||||
|
||||
- ok: `True`
|
||||
- remaining_cpu_m: 395
|
||||
- note: Aggregate remaining millicores is not a kube-scheduler guarantee.
|
||||
- note: Preflight does not lower requests and does not start Helm.
|
||||
|
||||
## Signal notes
|
||||
|
||||
- Host resource:hosteurope:railiance01 and cluster resource:railiance:reef-railiance:k3s are the same 4 vCPU; do not add them.
|
||||
- Prometheus namespace_cpu:kube_pod_container_resource_requests:sum omitted Forgejo 100m on 2026-09-11; Kubernetes API is the scheduler-effective source.
|
||||
- node-exporter was disabled; host CPU usage is not a reservation and is not treated as spare capacity.
|
||||
- Zero-request pods are demand, not zero. Missing metrics are listed as gaps, not zeros.
|
||||
- STATE-WP-0091 preflight consumes state_hub_preflight_observation; residual millicores are not admission.
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,163 @@
|
|||
# Per-workload allocation proposals — 2026-10-21 (generated)
|
||||
|
||||
Generated by `build_review.py` from `2026-10-21-node-budget.json`. Review inputs for owners, not applied changes.
|
||||
CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databases, 10m/pod critical path, 5m/pod other; change only if |delta|>=15m or request is zero. Memory rule: request = 7d p95 working set x1.15 rounded to 16Mi when |delta|>=64Mi; limits unchanged (requests only affect placement/eviction ranking).
|
||||
|
||||
| Namespace/workload | Pods | CPU req→prop (m) | CPU lim | p95/peak 7d (m) | Mem req→prop (Mi) | Mem p95/peak (Mi) | Throttle | Action | Owner | Receiving record |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---|---|---|
|
||||
| activity-core/activity-core | 10 | 100→100 | 1000* | 62.8/132.5 | 256→944 | 811/1011 | 0.0 | keep + memory up | activity-core | needs receiving task in activity-core |
|
||||
| argocd/argocd-application-controller | 1 | 50→30 | 1000 | 19.0/35.5 | 256→432 | 366/438 | 0.005 | decrease + memory up | railiance-enablement | needs receiving task in railiance-enablement |
|
||||
| argocd/argocd-repo-server | 1 | 25→10 | 1000 | 0.9/4.9 | 128→128 | 63/73 | 0.0 | decrease | railiance-enablement | needs receiving task in railiance-enablement |
|
||||
| audit-core/audit-core | 1 | 50→10 | 500 | 1.5/12.2 | 64→64 | 26/27 | 0.001 | decrease | audit-core | needs receiving task in audit-core |
|
||||
| canned-prompts/canned-prompts | 1 | 25→5 | 500 | 2.5/6.6 | 96→96 | 63/65 | 0.0 | decrease | canned-prompts | needs receiving task in canned-prompts |
|
||||
| cert-manager/cainjector | 1 | 0→10 | 0* | 1.1/2.8 | 0→96 | 71/94 | None | increase + memory up | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| cert-manager/cert-manager | 1 | 0→10 | 0* | 1.4/3.5 | 0→80 | 56/90 | None | increase + memory up | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| cert-manager/webhook | 1 | 0→10 | 0* | 0.4/0.9 | 0→0 | 29/57 | None | increase | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| cnpg-system/cloudnative-pg | 1 | 100→50 | 100 | 2.5/4.4 | 100→100 | 72/86 | 0.005 | decrease | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| core-hub/rapp-core-hub | 3 | 200→60 | 2500 | 39.2/63.5 | 896→288 | 247/292 | 0.002 | decrease + memory down | core-hub | needs receiving task in core-hub |
|
||||
| coulomb/ihp-railiance-probe | 1 | 50→0 | 200 | 0.0/0.0 | 128→128 | 0/0 | None | retire/scale-to-zero (owner decision) | ihp-railiance-probe (unconfirmed; namespace owner unknown) | needs receiving task in ihp-railiance-probe |
|
||||
| coulomb-social/coulomb-social | 1 | 100→15 | 1000 | 0.4/24.6 | 256→176 | 146/146 | 0.0 | decrease + memory down | coulomb-social | needs receiving task in coulomb-social |
|
||||
| databases/apps-pg | 1 | 100→50 | 1000 | 8.0/55.6 | 256→112 | 89/116 | 0.0 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/forgejo-db | 1 | 200→105 | 1000 | 68.8/160.9 | 512→288 | 249/286 | 0.07 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/net-kingdom-pg | 1 | 100→50 | 1000 | 10.3/48.0 | 256→256 | 229/497 | 0.001 | decrease | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/platform-pg | 1 | 100→50 | 1000 | 13.0/68.7 | 256→192 | 160/264 | 0.001 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/platform-pg-2 | 1 | 100→50 | 1000 | 12.0/62.3 | 256→256 | 199/268 | 0.001 | decrease | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/state-hub-db | 1 | 0→70 | 0* | 36.7/133.2 | 0→416 | 349/359 | None | increase + memory up | state-hub | needs receiving task in state-hub |
|
||||
| email-connect/email-connect | 1 | 25→5 | 250 | 0.3/0.8 | 64→64 | 13/13 | 0.001 | decrease | email-connect | needs receiving task in email-connect |
|
||||
| external-secrets/external-secrets | 1 | 0→10 | 0* | 4.1/5.5 | 0→64 | 51/53 | None | increase + memory up | railiance-platform | needs receiving task in railiance-platform |
|
||||
| external-secrets/external-secrets-cert-controller | 1 | 0→10 | 0* | 1.0/1.5 | 0→64 | 42/51 | None | increase + memory up | railiance-platform | needs receiving task in railiance-platform |
|
||||
| external-secrets/external-secrets-webhook | 1 | 0→10 | 0* | 0.5/0.6 | 0→80 | 63/112 | None | increase + memory up | railiance-platform | needs receiving task in railiance-platform |
|
||||
| flex-auth/flex-auth | 9 | 145→45 | 2700 | 2.2/18.6 | 288→176 | 140/191 | 0.002 | decrease + memory down | flex-auth (no local checkout) | needs receiving task in flex-auth |
|
||||
| forgejo/forgejo-runner | 1 | 0→125 | 0* | 13.1/243.2 | 0→496 | 428/734 | None | increase + memory up | railiance-forge | needs receiving task in railiance-forge |
|
||||
| forgejo/gitea | 1 | 100→200 | 0* | 133.2/376.7 | 128→1104 | 956/1005 | None | increase + memory up | railiance-forge | needs receiving task in railiance-forge |
|
||||
| issue-core/issue-core | 1 | 50→5 | 500 | 2.0/3.4 | 128→48 | 41/51 | 0.0 | decrease + memory down | issue-core | needs receiving task in issue-core |
|
||||
| knative-serving/knative-serving | 4 | 140→40 | 3500 | 7.2/9.9 | 360→176 | 146/211 | 0 | decrease + memory down | rail-knative | needs receiving task in rail-knative |
|
||||
| knative-serving/net-kourier-controller | 1 | 30→10 | 1000 | 0.8/1.6 | 200→64 | 53/80 | 0 | decrease + memory down | rail-knative | needs receiving task in rail-knative |
|
||||
| kourier-system/3scale-kourier-gateway | 1 | 50→10 | 1000 | 3.6/3.8 | 200→48 | 29/33 | 0 | decrease + memory down | rail-knative | needs receiving task in rail-knative |
|
||||
| kube-system/coredns-7bdb54f89 | 1 | 100→50 | 0* | 7.8/10.3 | 70→70 | 35/77 | None | decrease | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| kube-system/local-path-provisioner | 1 | 0→10 | 0* | 0.1/0.3 | 0→0 | 22/24 | None | increase | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| kube-system/metrics-server-786d997795 | 1 | 100→10 | 0* | 4.9/5.1 | 70→70 | 47/48 | None | decrease | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| kube-system/traefik | 1 | 0→10 | 0* | 3.8/16.0 | 0→80 | 69/149 | None | increase + memory up | railiance-cluster | needs receiving task in railiance-cluster |
|
||||
| mas/mas-mx | 1 | 20→5 | 0* | 0.6/3.6 | 64→64 | 43/48 | None | decrease | mail-address-semantics | needs receiving task in mail-address-semantics |
|
||||
| mfa/net-kingdom-sso-mfa | 1 | 110→20 | 1000 | 12.2/23.1 | 384→272 | 226/226 | 0.049 | decrease + memory down | net-kingdom | needs receiving task in net-kingdom |
|
||||
| openbao/railiance-platform | 1 | 25→10 | 200 | 0.2/0.5 | 32→32 | 5/5 | 0.0 | decrease | railiance-platform | needs receiving task in railiance-platform |
|
||||
| playwright/rapp-playwright | 1 | 100→65 | 1000 | 41.5/62.7 | 256→336 | 285/437 | 0.238 | decrease + review CPU limit (throttled) + memory up | railiance-platform | needs receiving task in railiance-platform |
|
||||
| rapp-qonto/rapp-qonto-audit-collector | 1 | 20→5 | 200 | 0.4/3.5 | 32→32 | 14/14 | 0.0 | decrease | rapp-qonto | needs receiving task in rapp-qonto |
|
||||
| rapp-qonto-egress/qonto-egress-proxy | 1 | 20→5 | 200 | 0.4/0.4 | 64→192 | 162/190 | 0 | decrease + memory up | rapp-qonto | needs receiving task in rapp-qonto |
|
||||
| rein-aharness/rein-aharness | 1 | 25→5 | 500 | 0.0/0.0 | 64→64 | 0/0 | 0 | decrease | rein-aharness | needs receiving task in rein-aharness |
|
||||
| reuse/reuse-surface | 2 | 110→10 | 550 | 2.1/10.2 | 288→80 | 56/57 | 0.0 | decrease + memory down | reuse-surface | needs receiving task in reuse-surface |
|
||||
| sbom-nexus/sbom-nexus | 1 | 25→10 | 500 | 3.8/5.8 | 64→64 | 80/80 | 0.0 | decrease | sbom-nexus | needs receiving task in sbom-nexus |
|
||||
| sso/net-kingdom-sso-mfa | 4 | 150→40 | 1150 | 22.4/34.5 | 256→176 | 140/179 | 0.537 | decrease + review CPU limit (throttled) + memory down | net-kingdom | needs receiving task in net-kingdom |
|
||||
| state-hub/railiance-apps | 2 | 260→390 | 1500 | 258.5/975.0 | 640→640 | 744/798 | 0.0 | increase | state-hub | needs receiving task in state-hub |
|
||||
| target-revenue/target-revenue | 1 | 50→5 | 1000 | 2.5/2.6 | 256→64 | 52/71 | 0 | decrease + memory down | target-revenue | needs receiving task in target-revenue |
|
||||
| target-revenue/target-revenue-pg | 1 | 0→50 | 0* | 7.6/9.2 | 0→80 | 56/63 | None | increase + memory up | target-revenue | needs receiving task in target-revenue |
|
||||
| telemetry/alertmanager | 1 | 25→5 | 250* | 0.7/0.8 | 64→64 | 39/57 | 0 | decrease | rapp-telemetry | needs receiving task in rapp-telemetry |
|
||||
| telemetry/grafana | 1 | 70→35 | 700 | 12.5/61.0 | 192→512 | 436/520 | 0.007 | decrease + memory up | rapp-telemetry | needs receiving task in rapp-telemetry |
|
||||
| telemetry/kube-state-metrics | 1 | 25→5 | 250 | 2.7/4.5 | 64→64 | 34/64 | 0.0 | decrease | rapp-telemetry | needs receiving task in rapp-telemetry |
|
||||
| telemetry/prometheus | 1 | 100→205 | 1000* | 134.5/192.4 | 512→1152 | 992/1089 | 0.002 | increase + memory up | rapp-telemetry | needs receiving task in rapp-telemetry |
|
||||
| telemetry/rapp-telemetry | 1 | 25→5 | 500 | 2.6/4.4 | 128→48 | 30/38 | 0.0 | decrease + memory down | rapp-telemetry | needs receiving task in rapp-telemetry |
|
||||
| tenant-engine/tenant-engine | 1 | 25→5 | 300 | 2.4/2.6 | 48→48 | 53/68 | 0.0 | decrease | tenant-engine | needs receiving task in tenant-engine |
|
||||
| user-engine/user-engine | 1 | 50→5 | 500 | 0.4/4.2 | 64→64 | 56/57 | 0.0 | decrease | user-engine | needs receiving task in user-engine |
|
||||
| user-engine/user-engine-pg | 1 | 100→50 | 1000 | 7.5/9.7 | 256→96 | 78/103 | 0.0 | decrease + memory down | user-engine | needs receiving task in user-engine |
|
||||
| vergabe-demo-company/vergabe-teilnahme | 1 | 60→60 | 1000 | 0.5/20.3 | 256→256 | 172/172 | 0.0 | keep (provisional) | railiance-apps | RAPPS-WP-0014-T04 target replay (SLOT, result pending) |
|
||||
| activity-core/actcore-temporal-ui-tls-2-1888679036-1756117428 | 1 | 10→10 | 100 | 0.0/0.1 | 64→64 | 13/18 | 0 | keep | activity-core | — |
|
||||
| approval-engine/approval-engine | 1 | 10→10 | 500 | 0.7/16.3 | 64→64 | 30/31 | 0.0 | keep | approval-engine | — |
|
||||
| argocd/argocd-applicationset-controller | 1 | 10→10 | 250 | 0.7/1.0 | 64→64 | 84/113 | 0 | keep | railiance-enablement | — |
|
||||
| argocd/argocd-redis | 1 | 10→10 | 200 | 5.0/5.1 | 32→32 | 6/27 | 0 | keep | railiance-enablement | — |
|
||||
| bao-notice/bao-notice | 1 | 5→5 | 100 | 0.0/0.1 | 16→16 | 4/4 | 0 | keep | railiance-platform | — |
|
||||
| informed-decision/informed-decision | 1 | 5→5 | 500 | 1.2/23.2 | 64→64 | 39/47 | 0.0 | keep | informed-decision | — |
|
||||
| kube-system/svclb-mas-smtp-d86db2b1 | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 0/2 | None | keep | railiance-cluster | — |
|
||||
| kube-system/svclb-traefik-0c8aecaf | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 1/2 | None | keep | railiance-cluster | — |
|
||||
| mas/mas-statute | 1 | 5→5 | 0* | 0.3/0.5 | 32→32 | 10/10 | None | keep | mail-address-semantics | — |
|
||||
| openbao/openbao | 1 | 100→100 | 500 | 13.1/181.3 | 256→256 | 177/399 | 0.0 | keep | railiance-platform | — |
|
||||
| policy-nexus/policy-nexus | 1 | 10→10 | 100 | 0.1/0.4 | 32→32 | 11/11 | 0.0 | keep | policy-nexus | — |
|
||||
| state-hub/state-hub | 1 | 5→5 | 100 | 2.0/6.1 | 24→24 | 26/26 | 0.001 | keep | state-hub | — |
|
||||
|
||||
`*` = at least one container has no CPU limit.
|
||||
|
||||
## Rationale for non-keep rows
|
||||
|
||||
- `activity-core/activity-core`: Within rule tolerance; keep. Memory p95 811Mi vs 256Mi request; one container without CPU limit.
|
||||
- `argocd/argocd-application-controller`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `argocd/argocd-repo-server`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `audit-core/audit-core`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `canned-prompts/canned-prompts`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `cert-manager/cainjector`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `cert-manager/cert-manager`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `cert-manager/webhook`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `cnpg-system/cloudnative-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `core-hub/rapp-core-hub`: Rule p95x1.5 / peakx0.5 / floor. Production hub.coulomb.social: verify startup and rollout before lowering.
|
||||
- `coulomb/ihp-railiance-probe`: ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision.
|
||||
- `coulomb-social/coulomb-social`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/apps-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/forgejo-db`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/net-kingdom-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/platform-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/platform-pg-2`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/state-hub-db`: Rule p95x1.5 / peakx0.5 / floor. Zero-request CNPG cluster, source state-hub/deploy/railiance/platform/state-hub-db-cluster.yaml; memory p95 349Mi.
|
||||
- `email-connect/email-connect`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `external-secrets/external-secrets`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `external-secrets/external-secrets-cert-controller`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `external-secrets/external-secrets-webhook`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `flex-auth/flex-auth`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `forgejo/forgejo-runner`: Rule p95x1.5 / peakx0.5 / floor. Zero request, dind unbounded. Admit one concurrent job: request ~125m/512Mi, dind limit ~1500m/2Gi; job bursts above the request are the build reserve.
|
||||
- `forgejo/gitea`: Rule p95x1.5 / peakx0.5 / floor. Main container has no requests/limits (100m effective comes from the init container); memory p95 956Mi vs 128Mi. Add memory request ~1Gi and limit ~1.5Gi.
|
||||
- `issue-core/issue-core`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `knative-serving/knative-serving`: Rule p95x1.5 / peakx0.5 / floor. Requests already reduced by RAIL-KNATIVE-WP-0002 (finished); the open risk is HPA maxReplicas (see budget).
|
||||
- `knative-serving/net-kourier-controller`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `kourier-system/3scale-kourier-gateway`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `kube-system/coredns-7bdb54f89`: DNS for every workload: floor 50m regardless of p95 7.8m.
|
||||
- `kube-system/local-path-provisioner`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `kube-system/metrics-server-786d997795`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `kube-system/traefik`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `mas/mas-mx`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `mfa/net-kingdom-sso-mfa`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `openbao/railiance-platform`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `playwright/rapp-playwright`: Rule p95x1.5 / peakx0.5 / floor. 23.8% throttled; RPF-WP-0056 covers admission, not sizing.
|
||||
- `rapp-qonto/rapp-qonto-audit-collector`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `rapp-qonto-egress/qonto-egress-proxy`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `rein-aharness/rein-aharness`: Rule p95x1.5 / peakx0.5 / floor. 0m/0Mi recorded over 7d; confirm the service is meant to run.
|
||||
- `reuse/reuse-surface`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `sbom-nexus/sbom-nexus`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `sso/net-kingdom-sso-mfa`: Rule p95x1.5 / peakx0.5 / floor. 53.7% of CFS periods throttled: review the CPU limit on the login path, independent of the request.
|
||||
- `state-hub/railiance-apps`: API p95 258m at 250m request = p95x1.5; the 975m peak is release/migration bursts carried by the surge reserve. Preflight models API surge 100m but live API pod requests 250m.
|
||||
- `target-revenue/target-revenue`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `target-revenue/target-revenue-pg`: Rule p95x1.5 / peakx0.5 / floor. Zero-request CNPG cluster, source target-revenue/k8s/railiance/cnpg-cluster.yaml.
|
||||
- `telemetry/alertmanager`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `telemetry/grafana`: Rule p95x1.5 / peakx0.5 / floor. Memory p95 436Mi vs 192Mi request; 3 restarts in 7d (cause not verified).
|
||||
- `telemetry/kube-state-metrics`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `telemetry/prometheus`: Rule p95x1.5 / peakx0.5 / floor. Memory p95 992Mi vs 512Mi request. Forgejo request recording rule still returns no series.
|
||||
- `telemetry/rapp-telemetry`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `tenant-engine/tenant-engine`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `user-engine/user-engine`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `user-engine/user-engine-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `vergabe-demo-company/vergabe-teilnahme`: 60m provisional, pending target replay (RAPPS-WP-0014-T04).
|
||||
|
||||
## Missing receiving tasks by owner repo
|
||||
|
||||
- **activity-core** (1): activity-core/activity-core
|
||||
- **audit-core** (1): audit-core/audit-core
|
||||
- **canned-prompts** (1): canned-prompts/canned-prompts
|
||||
- **core-hub** (1): core-hub/rapp-core-hub
|
||||
- **coulomb-social** (1): coulomb-social/coulomb-social
|
||||
- **email-connect** (1): email-connect/email-connect
|
||||
- **flex-auth** (1): flex-auth/flex-auth
|
||||
- **ihp-railiance-probe** (1): coulomb/ihp-railiance-probe
|
||||
- **issue-core** (1): issue-core/issue-core
|
||||
- **mail-address-semantics** (1): mas/mas-mx
|
||||
- **net-kingdom** (2): mfa/net-kingdom-sso-mfa, sso/net-kingdom-sso-mfa
|
||||
- **rail-knative** (3): knative-serving/knative-serving, knative-serving/net-kourier-controller, kourier-system/3scale-kourier-gateway
|
||||
- **railiance-cluster** (7): cert-manager/cainjector, cert-manager/cert-manager, cert-manager/webhook, kube-system/coredns-7bdb54f89, kube-system/local-path-provisioner, kube-system/metrics-server-786d997795, kube-system/traefik
|
||||
- **railiance-enablement** (2): argocd/argocd-application-controller, argocd/argocd-repo-server
|
||||
- **railiance-forge** (2): forgejo/forgejo-runner, forgejo/gitea
|
||||
- **railiance-platform** (5): external-secrets/external-secrets, external-secrets/external-secrets-cert-controller, external-secrets/external-secrets-webhook, openbao/railiance-platform, playwright/rapp-playwright
|
||||
- **rapp-postgres** (6): cnpg-system/cloudnative-pg, databases/apps-pg, databases/forgejo-db, databases/net-kingdom-pg, databases/platform-pg, databases/platform-pg-2
|
||||
- **rapp-qonto** (2): rapp-qonto/rapp-qonto-audit-collector, rapp-qonto-egress/qonto-egress-proxy
|
||||
- **rapp-telemetry** (5): telemetry/alertmanager, telemetry/grafana, telemetry/kube-state-metrics, telemetry/prometheus, telemetry/rapp-telemetry
|
||||
- **rein-aharness** (1): rein-aharness/rein-aharness
|
||||
- **reuse-surface** (1): reuse/reuse-surface
|
||||
- **sbom-nexus** (1): sbom-nexus/sbom-nexus
|
||||
- **state-hub** (2): databases/state-hub-db, state-hub/railiance-apps
|
||||
- **target-revenue** (2): target-revenue/target-revenue, target-revenue/target-revenue-pg
|
||||
- **tenant-engine** (1): tenant-engine/tenant-engine
|
||||
- **user-engine** (2): user-engine/user-engine, user-engine/user-engine-pg
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
{
|
||||
"captured_between": ["2026-10-21T06:08:20Z", "2026-10-21T06:30:00Z"],
|
||||
"target": {"context": "default", "node": "239.62.205.92.host.secureserver.net", "internal_ip": "92.205.62.239", "roles": "control-plane,etcd", "kubelet": "v1.35.1+k3s1"},
|
||||
"access": "read-only GET (kubectl get / get --raw); no resource was created, changed or deleted",
|
||||
"kubelet_config": {"systemReserved": null, "kubeReserved": null, "evictionHard": {"imagefs.available": "5%", "nodefs.available": "5%"}, "maxPods": 110,
|
||||
"note": "allocatable == capacity (4000m); host and k3s control plane consume from the same 4000m"},
|
||||
"kubelet_summary_samples": {
|
||||
"columns": ["time", "node_cpu_m", "pods_cpu_m", "k3s_kubelet_container_cpu_m", "non_pod_cpu_m"],
|
||||
"rows": [["06:25:01Z",1153,365,436,787],["06:25:12Z",1360,511,424,849],["06:25:22Z",1312,539,424,773],["06:25:32Z",1174,437,457,736],
|
||||
["06:25:42Z",1228,430,457,798],["06:25:52Z",1097,414,415,683],["06:26:03Z",2034,1163,415,871],["06:26:13Z",1985,1308,504,677]],
|
||||
"single_sample_memory_mi": {"node_working_set": 9578, "pods": 7639, "k3s_kubelet_container": 2604},
|
||||
"gap": "No 7-day host/control-plane series: Prometheus has no non-pod cgroup series and node-exporter is disabled."
|
||||
},
|
||||
"knative_hpa": {"knative-serving/activator": {"min": 1, "max": 20, "pod_cpu_request_m": 50, "current": 1},
|
||||
"knative-serving/webhook": {"min": 1, "max": 5, "pod_cpu_request_m": 30, "current": 1},
|
||||
"kourier-system/3scale-kourier-gateway": {"min": 1, "max": 10, "pod_cpu_request_m": 50, "current": 1},
|
||||
"worst_case_extra_requests_m": 1520},
|
||||
"knative_requests_m": {"activator": 50, "autoscaler": 30, "controller": 30, "webhook": 30, "net-kourier-controller": 30, "3scale-kourier-gateway": 50, "total": 220,
|
||||
"note": "The ~1000m Knative/Kourier reservation in CUST-WP-0071 predates RAIL-KNATIVE-WP-0002 (finished); it is stale."},
|
||||
"release_surge": {"state-hub/state-hub": {"strategy": "RollingUpdate", "maxSurge": 1, "api_pod_requests": {"cpu": "250m", "memory": "512Mi"}},
|
||||
"state_hub_preflight_model": {"api_surge_cpu_m": 100, "mcp_surge_cpu_m": 10, "migrate_cpu_m": 50},
|
||||
"finding": "STATE-WP-0091 preflight models a 100m API surge; the live API pod requests 250m, so a release needs ~310m, not 160m."},
|
||||
"latent_requests": {"inter-hub/inter-hub": {"replicas": 0, "pod_cpu_request_m": 250, "source": "railiance-apps/charts/inter-hub",
|
||||
"note": "Haskell Inter-Hub retired (CORE-WP-0007); scaling up would claim 250m. No current reservation."}},
|
||||
"cronjobs": [["audit-core","audit-core-attest-chain","17 3 * * *","10m"],["external-secrets","eso-token-renewer","40 2 * * *","10m"],
|
||||
["mfa","privacyidea-backup","30 3 * * *","10m"],["sso","authelia-backup","15 3 * * *","10m"],["sso","keycape-factor-renewer","*/10 * * * *","10m"],
|
||||
["sso","lldap-backup","0 3 * * *","10m"],["user-engine","user-engine-backup","0 4 * * *","25m"]],
|
||||
"stale_non_running_pods": [["coulomb","ihp-railiance-probe-7c99c6fcc4-jwhnb","ContainerStatusUnknown","166d"],["coulomb","ihp-railiance-probe-7c99c6fcc4-l9f2z","ContainerStatusUnknown","22d"],
|
||||
["coulomb","ihp-railiance-probe-7c99c6fcc4-nmtlq","ImagePullBackOff (image 92.205.130.254:32166/coulomb/ihp-railiance-probe:d0162d5)","21d"],
|
||||
["forgejo","forgejo-gitea-7d94988c79-krqdr","ContainerStatusUnknown","105d"],["forgejo","forgejo-runner-6fcb669668-7g8m7","Error","109d"],
|
||||
["telemetry","telemetry-smtp-proof-20260928-ltd7q","Error","22d"]],
|
||||
"empty_namespaces": ["openbao-migration-drill","cert-manager-test","whitehat","inter-hub (deployment at 0 replicas)","ops-warden","custodian-agents"],
|
||||
"drill_note": "platform-pg-drill namespace no longer exists; platform-pg-2 is the overflow admission cluster (RAPP-POSTGRES-WP-0005/0006), not a drill.",
|
||||
"concurrent_benchmark": {"namespace": "vergabe-sizing-20261021", "requests_m": {"synthetic-driver": 100, "vergabe-app": 60, "vergabe-db": 100},
|
||||
"note": "RAPPS-WP-0014-T04 target replay appeared after the 06:08Z capture; temporary 260m, excluded from the steady budget."},
|
||||
"forgejo_recording_rule": "namespace_cpu:kube_pod_container_resource_requests:sum{namespace=\"forgejo\"} returned no series (still missing since 2026-09-11)."
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
142
docs/evidence/2026-10-21-node-allocation-review/build_review.py
Normal file
142
docs/evidence/2026-10-21-node-allocation-review/build_review.py
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
#!/usr/bin/env python3
|
||||
"""RESOURCE-WP-0008-T02/T03: per-workload proposals and node budget (no cluster access).
|
||||
|
||||
Inputs (same directory): 2026-10-21-allocation-reconcile.json (Custodian collector),
|
||||
2026-10-21-workload-usage.json (7d per-pod join). Output: 2026-10-21-node-budget.json.
|
||||
Proposals are review inputs for owners; nothing here applies a change.
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
usage = {(w["namespace"], w["workload"]): w for w in json.loads((HERE / "2026-10-21-workload-usage.json").read_text())["workloads"]}
|
||||
NEEDS = "needs receiving task in {}"
|
||||
|
||||
import math
|
||||
import yaml
|
||||
OWNERS = yaml.safe_load(Path("/home/worsch/the-custodian/docs/evidence/namespace-ownership.yaml").read_text())["namespaces"]
|
||||
OWNER_OVERRIDE = {("databases", "state-hub-db"): "state-hub", ("target-revenue", "target-revenue-pg"): "target-revenue",
|
||||
("coulomb", "ihp-railiance-probe"): "ihp-railiance-probe (unconfirmed; namespace owner unknown)",
|
||||
("flex-auth", "flex-auth"): "flex-auth (no local checkout)"}
|
||||
DB_NS = {"databases"}
|
||||
CRITICAL_NS = {"kube-system", "knative-serving", "kourier-system", "sso", "mfa", "openbao", "external-secrets", "cert-manager", "cnpg-system", "argocd", "state-hub"}
|
||||
SAFETY_P95, SAFETY_PEAK, MIN_DELTA_M, MIN_DELTA_MI = 1.5, 0.5, 15, 64
|
||||
RECEIVING = {("vergabe-demo-company", "vergabe-teilnahme"): "RAPPS-WP-0014-T04 target replay (SLOT, result pending)"}
|
||||
OVERRIDES = { # (ns, workload): (cpu_m, reason)
|
||||
("coulomb", "ihp-railiance-probe"): (0, "ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision."),
|
||||
("vergabe-demo-company", "vergabe-teilnahme"): (60, "60m provisional, pending target replay (RAPPS-WP-0014-T04)."),
|
||||
("kube-system", "svclb-traefik-0c8aecaf"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."),
|
||||
("kube-system", "svclb-mas-smtp-d86db2b1"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."),
|
||||
("kube-system", "coredns-7bdb54f89"): (50, "DNS for every workload: floor 50m regardless of p95 7.8m."),
|
||||
("state-hub", "railiance-apps"): (390, "API p95 258m at 250m request = p95x1.5; the 975m peak is release/migration bursts carried by the surge reserve. Preflight models API surge 100m but live API pod requests 250m."),
|
||||
}
|
||||
NOTES = {
|
||||
("forgejo", "gitea"): "Main container has no requests/limits (100m effective comes from the init container); memory p95 956Mi vs 128Mi. Add memory request ~1Gi and limit ~1.5Gi.",
|
||||
("forgejo", "forgejo-runner"): "Zero request, dind unbounded. Admit one concurrent job: request ~125m/512Mi, dind limit ~1500m/2Gi; job bursts above the request are the build reserve.",
|
||||
("telemetry", "prometheus"): "Memory p95 992Mi vs 512Mi request. Forgejo request recording rule still returns no series.",
|
||||
("activity-core", "activity-core"): "Memory p95 811Mi vs 256Mi request; one container without CPU limit.",
|
||||
("databases", "state-hub-db"): "Zero-request CNPG cluster, source state-hub/deploy/railiance/platform/state-hub-db-cluster.yaml; memory p95 349Mi.",
|
||||
("target-revenue", "target-revenue-pg"): "Zero-request CNPG cluster, source target-revenue/k8s/railiance/cnpg-cluster.yaml.",
|
||||
("sso", "net-kingdom-sso-mfa"): "53.7% of CFS periods throttled: review the CPU limit on the login path, independent of the request.",
|
||||
("playwright", "rapp-playwright"): "23.8% throttled; RPF-WP-0056 covers admission, not sizing.",
|
||||
("telemetry", "grafana"): "Memory p95 436Mi vs 192Mi request; 3 restarts in 7d (cause not verified).",
|
||||
("core-hub", "rapp-core-hub"): "Production hub.coulomb.social: verify startup and rollout before lowering.",
|
||||
("rein-aharness", "rein-aharness"): "0m/0Mi recorded over 7d; confirm the service is meant to run.",
|
||||
("knative-serving", "knative-serving"): "Requests already reduced by RAIL-KNATIVE-WP-0002 (finished); the open risk is HPA maxReplicas (see budget).",
|
||||
}
|
||||
|
||||
def rule(ns, wl, w):
|
||||
pods = len(w["pods"])
|
||||
floor = (50 if ns in DB_NS or wl.endswith("-pg") else 10 if ns in CRITICAL_NS else 5) * pods
|
||||
c5 = lambda x: int(math.ceil(x / 5) * 5)
|
||||
return max(floor, c5(w["cpu_p95_m_sum"] * SAFETY_P95), c5(w["cpu_peak_m_sum"] * SAFETY_PEAK))
|
||||
|
||||
rows, cur_total, prop_total = [], 0, 0
|
||||
for key, w in usage.items():
|
||||
ns, wl = key
|
||||
target = rule(ns, wl, w)
|
||||
cur = w["cpu_req_m"]
|
||||
if key in OVERRIDES:
|
||||
req, why = OVERRIDES[key]
|
||||
elif cur == 0 or abs(target - cur) >= MIN_DELTA_M:
|
||||
req, why = target, "Rule p95x1.5 / peakx0.5 / floor."
|
||||
else:
|
||||
req, why = cur, "Within rule tolerance; keep."
|
||||
if key in NOTES: why += " " + NOTES[key]
|
||||
action = "keep" if req == cur else ("increase" if req > cur else "decrease")
|
||||
if key == ("coulomb", "ihp-railiance-probe"): action = "retire/scale-to-zero (owner decision)"
|
||||
if key == ("vergabe-demo-company", "vergabe-teilnahme"): action = "keep (provisional)"
|
||||
if key in {("sso", "net-kingdom-sso-mfa"), ("playwright", "rapp-playwright")}: action += " + review CPU limit (throttled)"
|
||||
mem_target = max(16, int(math.ceil(w["mem_p95_mi_sum"] * 1.15 / 16) * 16))
|
||||
mem = mem_target if abs(mem_target - w["mem_req_mi"]) >= MIN_DELTA_MI and key not in OVERRIDES else w["mem_req_mi"]
|
||||
if mem != w["mem_req_mi"]: action += " + memory " + ("up" if mem > w["mem_req_mi"] else "down")
|
||||
owner = OWNER_OVERRIDE.get(key) or (OWNERS.get(ns) or {}).get("owner", "unknown")
|
||||
record = RECEIVING.get(key) or (None if action == "keep" else f"needs receiving task in {owner.split(' ')[0]}")
|
||||
rows.append({"namespace": ns, "workload": wl, "pods": len(w["pods"]), "cpu_request_m": cur, "cpu_limit_m": w["cpu_lim_m"],
|
||||
"containers_without_cpu_limit": w["containers_without_cpu_limit"], "memory_request_mi": w["mem_req_mi"], "memory_limit_mi": w["mem_lim_mi"],
|
||||
"cpu_p95_m_7d": w["cpu_p95_m_sum"], "cpu_peak_m_7d": w["cpu_peak_m_sum"], "memory_p95_mi_7d": w["mem_p95_mi_sum"], "memory_peak_mi_7d": w["mem_peak_mi_sum"],
|
||||
"throttle_ratio_7d": w["throttle_max"], "restarts_7d": w["restarts_7d"], "rule_cpu_m": target,
|
||||
"action": action, "proposed_cpu_request_m": req, "proposed_memory_request_mi": mem,
|
||||
"owner_repo": owner, "receiving_record": record, "rationale": why})
|
||||
cur_total += cur; prop_total += req
|
||||
KNATIVE_HPA = {"activator": (20, 3, 50), "3scale-kourier-gateway": (10, 3, 50), "webhook": (5, 2, 30)}
|
||||
|
||||
cur_mem = sum(r["memory_request_mi"] for r in rows); prop_mem = sum(r["proposed_memory_request_mi"] for r in rows)
|
||||
hpa_now = sum((mx - 1) * c for mx, _, c in KNATIVE_HPA.values()); hpa_capped = sum((cap - 1) * c for _, cap, c in KNATIVE_HPA.values())
|
||||
budget = [
|
||||
("Proposed steady pod requests (incl. Vergabe 60m provisional)", cur_total, prop_total, "Kubernetes API 06:08Z + proposals"),
|
||||
("Host / k3s control plane (no system/kube-reserved; allocatable == capacity)", 900, 900, "kubelet summary 8 samples 06:25Z: non-pod 677-871m, k3s 415-504m; no 7d series"),
|
||||
("Release surge + migration hook (one release at a time)", 310, 310, "state-hub API 250m surge + MCP 10m + migrate 50m; largest live RollingUpdate"),
|
||||
("Admitted factory/build execution (one runner job burst above runner request)", 250, 250, "forgejo-runner pod peak 243m; namespace peak 1033m incl. gitea"),
|
||||
("Maintenance CronJobs (7 jobs, 10-25m, overlapping 02:40-04:00)", 50, 50, "kubectl cronjobs; keycape-factor-renewer every 10 min"),
|
||||
("Knative HPA scale-out headroom (requests at max replicas minus 1)", hpa_now, hpa_capped, "activator 20/gateway 10/webhook 5 -> proposed caps 3/3/2"),
|
||||
("Pilot growth reserve (Vergabe replay result + next tenant) — SLOT", 0, 200, "placeholder pending RAPPS-WP-0014-T04; not measured"),
|
||||
]
|
||||
cur_sum = sum(b[1] for b in budget); prop_sum = sum(b[2] for b in budget)
|
||||
# Serialized: one disruptive event (release surge, build burst or HPA step) at a time.
|
||||
ser = lambda i: budget[0][i] + budget[1][i] + max(budget[2][i], budget[3][i], budget[5][i]) + budget[4][i] + budget[6][i]
|
||||
out = {"schema": "resource-control.node-allocation-review.v1", "workplan": "RESOURCE-WP-0008", "beneficiary": "CUST-WP-0071-T03",
|
||||
"observed_at": "2026-10-21T06:08:20Z", "allocatable_cpu_m": 4000, "allocatable_memory_mi": round(16770080768 / 2**20),
|
||||
"rule": {"cpu": f"max(floor, p95x{SAFETY_P95}, peakx{SAFETY_PEAK}) rounded up to 5m; floors 50m/pod databases, 10m/pod critical path, 5m/pod other; change only if |delta|>={MIN_DELTA_M}m or request is zero",
|
||||
"memory": "request = 7d p95 working set x1.15 rounded to 16Mi when |delta|>=64Mi; limits unchanged (requests only affect placement/eviction ranking)"},
|
||||
"budget_rows": [dict(item=b[0], current_m=b[1], proposed_m=b[2], basis=b[3]) for b in budget],
|
||||
"budget_total_all_concurrent": {"current_m": cur_sum, "proposed_m": prop_sum, "residual_current_m": 4000 - cur_sum, "residual_proposed_m": 4000 - prop_sum},
|
||||
"budget_total_serialized_events": {"current_m": ser(1), "proposed_m": ser(2), "residual_current_m": 4000 - ser(1), "residual_proposed_m": 4000 - ser(2)},
|
||||
"measured_demand_7d": {"containers_cpu_p95_m": 966, "containers_cpu_peak_m": 1740, "host_non_pod_cpu_m_sample": [677, 871],
|
||||
"note": "Measured demand (containers p95 + host) ~1.8 cores, peaks ~2.6 cores; the deficit is reservation, not demand."},
|
||||
"memory_budget_mi": {"allocatable": round(16770080768 / 2**20), "requests_current": round(cur_mem), "requests_proposed": round(prop_mem),
|
||||
"host_k3s_sample": 2604, "release_surge_state_hub_api": 864, "build_runner_burst": 512, "pilot_growth_slot": 256,
|
||||
"residual_current": round(16770080768 / 2**20 - cur_mem - 2604 - 512 - 512),
|
||||
"residual_proposed": round(16770080768 / 2**20 - prop_mem - 2604 - 864 - 512 - 256)},
|
||||
"vergabe_slot": {"current_request_m": 60, "status": "provisional, pending target replay (RAPPS-WP-0014-T04)", "replay_result": None},
|
||||
"workloads": sorted(rows, key=lambda r: (r["action"] == "keep", r["namespace"], r["workload"]))}
|
||||
(HERE / "2026-10-21-node-budget.json").write_text(json.dumps(out, indent=1) + "\n")
|
||||
for b in out["budget_rows"]: print(f"{b['item'][:70]:70} {b['current_m']:6} {b['proposed_m']:6}")
|
||||
print(json.dumps({k: out[k] for k in ("budget_total_all_concurrent", "budget_total_serialized_events", "memory_budget_mi")}))
|
||||
|
||||
# Markdown table for owners (generated; edit build_review.py, not the .md).
|
||||
from collections import defaultdict
|
||||
L = ["# Per-workload allocation proposals — 2026-10-21 (generated)", "",
|
||||
"Generated by `build_review.py` from `2026-10-21-node-budget.json`. Review inputs for owners, not applied changes.",
|
||||
f"CPU rule: {out['rule']['cpu']}. Memory rule: {out['rule']['memory']}.", "",
|
||||
"| Namespace/workload | Pods | CPU req→prop (m) | CPU lim | p95/peak 7d (m) | Mem req→prop (Mi) | Mem p95/peak (Mi) | Throttle | Action | Owner | Receiving record |",
|
||||
"|---|---:|---:|---:|---:|---:|---:|---:|---|---|---|"]
|
||||
for r in out["workloads"]:
|
||||
lim = f"{r['cpu_limit_m']}" + ("*" if r["containers_without_cpu_limit"] else "")
|
||||
L.append(f"| {r['namespace']}/{r['workload']} | {r['pods']} | {r['cpu_request_m']}→{r['proposed_cpu_request_m']} | {lim} | {r['cpu_p95_m_7d']}/{r['cpu_peak_m_7d']} | "
|
||||
f"{r['memory_request_mi']:.0f}→{r['proposed_memory_request_mi']:.0f} | {r['memory_p95_mi_7d']:.0f}/{r['memory_peak_mi_7d']:.0f} | {r['throttle_ratio_7d']} | "
|
||||
f"{r['action']} | {r['owner_repo']} | {r['receiving_record'] or '—'} |")
|
||||
L += ["", "`*` = at least one container has no CPU limit.", "", "## Rationale for non-keep rows", ""]
|
||||
for r in out["workloads"]:
|
||||
if r["action"] != "keep":
|
||||
L.append(f"- `{r['namespace']}/{r['workload']}`: {r['rationale']}")
|
||||
need = defaultdict(list)
|
||||
for r in out["workloads"]:
|
||||
if (r["receiving_record"] or "").startswith("needs receiving task"):
|
||||
need[r["owner_repo"].split(" ")[0]].append(f"{r['namespace']}/{r['workload']}")
|
||||
L += ["", "## Missing receiving tasks by owner repo", ""]
|
||||
for repo in sorted(need):
|
||||
L.append(f"- **{repo}** ({len(need[repo])}): " + ", ".join(need[repo]))
|
||||
(HERE / "2026-10-21-node-budget.md").write_text("\n".join(L) + "\n")
|
||||
out["missing_receiving_tasks"] = {k: v for k, v in sorted(need.items())}
|
||||
(HERE / "2026-10-21-node-budget.json").write_text(json.dumps(out, indent=1) + "\n")
|
||||
45
docs/evidence/2026-10-21-node-allocation-review/join.py
Normal file
45
docs/evidence/2026-10-21-node-allocation-review/join.py
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
import json, subprocess, sys, importlib.util
|
||||
from collections import defaultdict
|
||||
spec = importlib.util.spec_from_file_location("o", "/home/worsch/railiance-cluster/tools/observe_cluster_resources.py")
|
||||
o = importlib.util.module_from_spec(spec); spec.loader.exec_module(o)
|
||||
tele = json.load(open(sys.argv[1])); out = sys.argv[2]
|
||||
pods = json.loads(subprocess.run(["kubectl","get","pods","-A","-o","json"],capture_output=True,text=True,check=True).stdout)["items"]
|
||||
def vec(k):
|
||||
d = {}
|
||||
for r in tele["queries"][k]["response"]["data"]["result"]:
|
||||
m = r["metric"]; d[(m.get("namespace"), m.get("pod"))] = float(r["value"][1])
|
||||
return d
|
||||
V = {k: vec(k) for k in ["cpu_p95_m_by_pod","cpu_peak_m_by_pod","mem_peak_bytes_by_pod","mem_p95_bytes_by_pod","throttle_ratio_by_pod","restarts_by_pod"]}
|
||||
W = defaultdict(lambda: {"pods":[], "cpu_req_m":0,"mem_req_mi":0.0,"cpu_lim_m":0,"mem_lim_mi":0.0,"containers_without_cpu_limit":0,"containers_without_mem_limit":0,"containers_without_cpu_request":0,
|
||||
"cpu_p95_m_sum":0.0,"cpu_peak_m_sum":0.0,"mem_peak_mi_sum":0.0,"mem_p95_mi_sum":0.0,"throttle_max":None,"restarts_7d":0.0,"pods_without_metrics":[],"phases":set(),"kinds":set()})
|
||||
for p in pods:
|
||||
md, sp, st = p["metadata"], p["spec"], p.get("status",{})
|
||||
if st.get("phase") in ("Succeeded","Failed"): continue
|
||||
ns, name = md["namespace"], md["name"]; wl = o.workload_name(p); w = W[(ns, wl)]
|
||||
w["pods"].append(name); w["phases"].add(st.get("phase")); w["kinds"].update(r["kind"] for r in md.get("ownerReferences") or [])
|
||||
c, m = o.pod_effective_requests(p); w["cpu_req_m"] += c; w["mem_req_mi"] += m/2**20
|
||||
for ct in sp.get("containers", []):
|
||||
lim = (ct.get("resources") or {}).get("limits") or {}; req = (ct.get("resources") or {}).get("requests") or {}
|
||||
if "cpu" in lim: w["cpu_lim_m"] += o.cpu_millicores(lim["cpu"])
|
||||
else: w["containers_without_cpu_limit"] += 1
|
||||
if "memory" in lim: w["mem_lim_mi"] += o.bytes_value(lim["memory"])/2**20
|
||||
else: w["containers_without_mem_limit"] += 1
|
||||
if not req.get("cpu"): w["containers_without_cpu_request"] += 1
|
||||
k = (ns, name)
|
||||
if k not in V["cpu_p95_m_by_pod"]: w["pods_without_metrics"].append(name)
|
||||
w["cpu_p95_m_sum"] += V["cpu_p95_m_by_pod"].get(k,0); w["cpu_peak_m_sum"] += V["cpu_peak_m_by_pod"].get(k,0)
|
||||
w["mem_peak_mi_sum"] += V["mem_peak_bytes_by_pod"].get(k,0)/2**20; w["mem_p95_mi_sum"] += V["mem_p95_bytes_by_pod"].get(k,0)/2**20
|
||||
t = V["throttle_ratio_by_pod"].get(k)
|
||||
if t is not None and t == t: w["throttle_max"] = max(w["throttle_max"] or 0, t)
|
||||
w["restarts_7d"] += V["restarts_by_pod"].get(k,0)
|
||||
rows = []
|
||||
for (ns, wl), w in W.items():
|
||||
w = {**w, "namespace": ns, "workload": wl, "phases": sorted(x for x in w["phases"] if x), "kinds": sorted(w["kinds"])}
|
||||
for f in ("cpu_p95_m_sum","cpu_peak_m_sum","mem_peak_mi_sum","mem_p95_mi_sum","restarts_7d","mem_req_mi","mem_lim_mi"): w[f] = round(w[f],1)
|
||||
if w["throttle_max"] is not None: w["throttle_max"] = round(w["throttle_max"],3)
|
||||
rows.append(w)
|
||||
rows.sort(key=lambda r: (-r["cpu_req_m"], r["namespace"]))
|
||||
json.dump({"captured_at": tele["captured_at"], "note": "Per-workload sums of per-pod 7d p95/peak (upper bound for multi-pod workloads; sum of p95 >= p95 of sum). Current running pods only; terminated Job pods appear only in namespace totals.", "workloads": rows}, open(out,"w"), indent=1)
|
||||
print(f"{'ns/workload':55} req lim p95 peak memreq memlim mempk thr rst")
|
||||
for r in rows: print(f"{r['namespace']+'/'+r['workload']:55} {r['cpu_req_m']:4} {r['cpu_lim_m']:5}{'*' if r['containers_without_cpu_limit'] else ' '}{r['cpu_p95_m_sum']:6} {r['cpu_peak_m_sum']:6} {r['mem_req_mi']:6.0f} {r['mem_lim_mi']:6.0f} {r['mem_peak_mi_sum']:6.0f} {r['throttle_max']} {r['restarts_7d']} {'NOMET' if r['pods_without_metrics'] else ''}")
|
||||
print("TOTAL req", sum(r['cpu_req_m'] for r in rows), "p95sum", round(sum(r['cpu_p95_m_sum'] for r in rows)), "peaksum", round(sum(r['cpu_peak_m_sum'] for r in rows)), "memreq", round(sum(r['mem_req_mi'] for r in rows)), "mempk", round(sum(r['mem_peak_mi_sum'] for r in rows)))
|
||||
35
docs/evidence/2026-10-21-node-allocation-review/prom7d.py
Normal file
35
docs/evidence/2026-10-21-node-allocation-review/prom7d.py
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
"""Read-only 7-day Prometheus capture via kube API service proxy (GET only)."""
|
||||
import json, subprocess, sys, urllib.parse
|
||||
from datetime import UTC, datetime
|
||||
out = sys.argv[1]
|
||||
SEL = 'container!="",container!="POD",namespace!=""'
|
||||
Q = {
|
||||
"cpu_p95_m_by_pod": f'quantile_over_time(0.95, (sum by(namespace,pod) (rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"cpu_peak_m_by_pod": f'max_over_time((sum by(namespace,pod) (rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"mem_peak_bytes_by_pod": f'max_over_time((sum by(namespace,pod) (container_memory_working_set_bytes{{{SEL}}}))[7d:5m])',
|
||||
"mem_p95_bytes_by_pod": f'quantile_over_time(0.95, (sum by(namespace,pod) (container_memory_working_set_bytes{{{SEL}}}))[7d:5m])',
|
||||
"throttle_ratio_by_pod": f'sum by(namespace,pod) (increase(container_cpu_cfs_throttled_periods_total{{{SEL}}}[7d])) / sum by(namespace,pod) (increase(container_cpu_cfs_periods_total{{{SEL}}}[7d]))',
|
||||
"restarts_by_pod": 'sum by(namespace,pod) (increase(kube_pod_container_status_restarts_total[7d]))',
|
||||
"cpu_p95_m_by_ns": f'quantile_over_time(0.95, (sum by(namespace) (rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"cpu_peak_m_by_ns": f'max_over_time((sum by(namespace) (rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"mem_peak_bytes_by_ns": f'max_over_time((sum by(namespace) (container_memory_working_set_bytes{{{SEL}}}))[7d:5m])',
|
||||
"node_cpu_p95_m": f'quantile_over_time(0.95, (sum(rate(container_cpu_usage_seconds_total{{id="/"}}[5m])) * 1000)[7d:5m])',
|
||||
"node_cpu_peak_m": f'max_over_time((sum(rate(container_cpu_usage_seconds_total{{id="/"}}[5m])) * 1000)[7d:5m])',
|
||||
"node_cpu_p50_m": f'quantile_over_time(0.50, (sum(rate(container_cpu_usage_seconds_total{{id="/"}}[5m])) * 1000)[7d:5m])',
|
||||
"containers_cpu_p95_m": f'quantile_over_time(0.95, (sum(rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"containers_cpu_peak_m": f'max_over_time((sum(rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"node_mem_peak_bytes": 'max_over_time(container_memory_working_set_bytes{id="/"}[7d])',
|
||||
"cpu_sample_count_by_ns": f'count_over_time((sum by(namespace) (rate(container_cpu_usage_seconds_total{{{SEL}}}[5m])) * 1000)[7d:5m])',
|
||||
"forgejo_recording": 'namespace_cpu:kube_pod_container_resource_requests:sum{namespace="forgejo"}',
|
||||
"prometheus_oldest_sample_age_s": 'time() - min(prometheus_tsdb_lowest_timestamp_seconds)',
|
||||
}
|
||||
res = {"captured_at": datetime.now(UTC).isoformat(), "window": "7d", "step": "5m",
|
||||
"source": "railiance01 telemetry/telemetry-prometheus via kube-apiserver service proxy (GET, read-only)", "queries": {}}
|
||||
for k, q in Q.items():
|
||||
path = "/api/v1/namespaces/telemetry/services/telemetry-prometheus:9090/proxy/api/v1/query?" + urllib.parse.urlencode({"query": q})
|
||||
p = subprocess.run(["kubectl", "get", "--raw", path], capture_output=True, text=True, timeout=180)
|
||||
try: r = json.loads(p.stdout)
|
||||
except Exception: r = {"status": "error", "error": p.stderr.strip()[:500]}
|
||||
res["queries"][k] = {"promql": q, "response": r}
|
||||
print(k, r.get("status"), len((r.get("data") or {}).get("result") or []), file=sys.stderr)
|
||||
json.dump(res, open(out, "w"), indent=1)
|
||||
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-10-19"
|
||||
updated: "2026-10-19"
|
||||
updated: "2026-10-21"
|
||||
flavor: implementation
|
||||
related: [CUST-WP-0071, VERGABE-WP-0019, RAPPS-WP-0014, RCLUSTER-WP-0014]
|
||||
depends_on: []
|
||||
|
|
@ -44,6 +44,7 @@ refreshing stale utilization metrics or inventing financial attribution.
|
|||
id: RESOURCE-WP-0008-T02
|
||||
status: progress
|
||||
priority: high
|
||||
blocking_reason: "Node budget and per-workload comparison done 2026-10-21; only the Vergabe target replay result (RAPPS-WP-0014-T04) remains to be inserted into the 60m provisional slot."
|
||||
state_hub_task_id: "1f538441-8b45-5d37-aeb3-345204b75117"
|
||||
```
|
||||
|
||||
|
|
@ -63,8 +64,9 @@ whole node budget fits. Do not infer human workflow acceptance.
|
|||
|
||||
```task
|
||||
id: RESOURCE-WP-0008-T03
|
||||
status: todo
|
||||
status: progress
|
||||
priority: medium
|
||||
blocking_reason: "Proposal and node budget published 2026-10-21, but none of the 54 owner changes has a receiving task in its owner repo (26 repos listed in the review), the coulomb probe owner is unconfirmed, and the Vergabe slot awaits RAPPS-WP-0014-T04."
|
||||
state_hub_task_id: "3e6e6a59-71a5-5048-b7a5-ed61416a442c"
|
||||
```
|
||||
|
||||
|
|
@ -75,3 +77,28 @@ finished while actionable leftovers exist only in prose. All application of
|
|||
accepted sizing stays with existing runtime owners and their reviewed deployment
|
||||
paths. This record does not create a parallel scheduler or change financial
|
||||
authority.
|
||||
|
||||
|
||||
## Node-level review — 2026-10-21
|
||||
|
||||
`docs/evidence/2026-10-21-node-allocation-review.md` and its directory carry a
|
||||
fresh read-only capture (06:08Z; identity 92.205.62.239 verified), a 7d Prometheus
|
||||
capture, per-workload proposals and the node budget. The capture reused the
|
||||
Custodian collector with `--output-dir`. Requests are 3605m, while measured
|
||||
demand is container p95 966m/peak 1740m plus host/k3s at 677–871m. Demand fits;
|
||||
reservations do not. The kubelet has no kube/system-reserved, the Knative HPA
|
||||
ceilings could claim 1520m, and zero-request workloads hide demand.
|
||||
|
||||
The rule-based proposal (p95×1.5 / peak×0.5 / floors) lowers steady requests to
|
||||
2400m. It reconciles at +140m only when releases, builds and Knative scale-out
|
||||
are serialized; it is −340m short when they all coincide. No procurement. The
|
||||
Vergabe line stays at 60m provisional, pending the target replay
|
||||
(RAPPS-WP-0014-T04 slot).
|
||||
|
||||
T02 holds the evidence-backed comparison. It stays in progress only for the
|
||||
Vergabe replay result: insert it into `build_review.py` and regenerate.
|
||||
|
||||
T03 cannot finish because every non-Vergabe change still needs a receiving task in
|
||||
its owner repo. The per-repo list is in `2026-10-21-node-budget.md`. Creating
|
||||
those records is the owners' or the Custodian's decision; this record does not
|
||||
create them.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue