From 3d9b1bd08043735e71246213e46c91f98f41d6ce Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 21 Oct 2026 08:40:36 +0200 Subject: [PATCH] Consume Vergabe target replay into node allocation budget; finish T02 Fill the Vergabe slot from railiance-apps 1af6d82 (RAPPS-WP-0014-T04): keep 60m request and 1000m/1Gi limits, raise memory request 256Mi->512Mi next wave, keep apps-pg at 100m for incremental pilot DB demand. Regenerated budget: serialized CPU +90m / memory +165Mi; all-concurrent CPU -390m / memory -347Mi. T03 stays in progress pending wave-1/wave-2 receiving tasks. Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 8340@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c --- WORK-RECORDS.md | 2 +- .../2026-10-21-node-allocation-review.md | 88 ++++++++---- .../2026-10-21-node-budget.json | 129 ++++++++++++------ .../2026-10-21-node-budget.md | 10 +- .../build_review.py | 37 +++-- ...RCE-WP-0008-custodian-allocation-review.md | 28 +++- 6 files changed, 208 insertions(+), 86 deletions(-) diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 1394c1a..6d13e58 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -51,5 +51,5 @@ | task | RESOURCE-WP-0006-T04 | done | — | workplans/RESOURCE-WP-0006-resource-object-and-reef-refs.md | | task | RESOURCE-WP-0007-T01 | done | — | workplans/RESOURCE-WP-0007-state-hub-release-margin.md | | task | RESOURCE-WP-0008-T01 | done | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md | -| task | RESOURCE-WP-0008-T02 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md | +| task | RESOURCE-WP-0008-T02 | done | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md | | task | RESOURCE-WP-0008-T03 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md | diff --git a/docs/evidence/2026-10-21-node-allocation-review.md b/docs/evidence/2026-10-21-node-allocation-review.md index 5314265..92f6c94 100644 --- a/docs/evidence/2026-10-21-node-allocation-review.md +++ b/docs/evidence/2026-10-21-node-allocation-review.md @@ -91,58 +91,94 @@ manifest revision (CUST-WP-0071-T04 retains the receipts). | Item | Current | Proposed | | --- | ---: | ---: | -| Steady pod requests (incl. Vergabe 60m provisional) | 3605 | 2400 | +| Steady pod requests (incl. Vergabe 60m, apps-pg kept at 100m) | 3605 | 2450 | | Host / k3s control plane (no kube/system-reserved) | 900 | 900 | | Release surge + migration hook (state-hub, one at a time) | 310 | 310 | | Admitted factory/build execution (one runner job burst) | 250 | 250 | | Maintenance CronJobs | 50 | 50 | | Knative HPA scale-out headroom | 1520 | 230 | -| Pilot growth reserve — **SLOT, pending RAPPS-WP-0014-T04** | 0 | 200 | -| **Total, all events concurrent** | **6635** | **4340** | -| **Residual vs 4000m, all concurrent** | **−2635** | **−340** | -| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3860 | -| Residual, serialized | −2075 | **+140** | +| Pilot growth reserve (next tenant/user growth; Vergabe is in steady requests) | 0 | 200 | +| **Total, all events concurrent** | **6635** | **4390** | +| **Residual vs 4000m, all concurrent** | **−2635** | **−390** | +| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3910 | +| Residual, serialized | −2075 | **+90** | -Proposed changes: +660m in increases and −1865m in decreases, across 55 non-keep -rows (54 owner changes plus the Vergabe slot). +Proposed changes: +660m increases and −1815m decreases, across 54 non-keep rows. -Memory (Mi): proposed requests 11704 + k3s 2604 + surge 864 + build 512 + pilot -slot 256 leaves +53 of 15993. It fits, but only just. The k3s figure is a single -sample. +Memory (Mi, allocatable 15993; k3s 2604 single sample, surge 864, build 512, +pilot growth 256): -**Reconciliation verdict:** the budget reconciles only conditionally, at +140m. That +| Memory | Current | Proposed | +| --- | ---: | ---: | +| Pod requests | 9856 | 12104 | +| Residual, all concurrent | +2157 | **−347** | +| Residual, serialized (max of surge/build) | +2669 | **+165** | + +The current memory residual looks comfortable only because 13 zero-request +workloads hide about 1.3Gi of working set. The proposed requests include the +Vergabe raise (+256Mi), and apps-pg keeps its 256Mi. + +**Reconciliation verdict:** the budget reconciles only conditionally, at +90m CPU and +165Mi memory. That needs three things: the lean decreases are accepted, releases, builds and Knative scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before releases), and the HPA ceilings are capped. With every event concurrent it is -−340m short. Measured demand plus margin does fit the 4 vCPU. **Procurement is +−390m CPU and −347Mi memory short. Measured demand plus margin does fit the 4 vCPU. **Procurement is not indicated**; the remedy is to correct reservations. Once requests have come down, railiance-cluster should make host operations explicit with `kube-reserved`/`system-reserved` (~900m/2.5Gi). Applying that first would make the current 3605m unschedulable. -## Vergabe slot +## Vergabe slot (filled 2026-10-21 from the target replay) -Vergabe stays at **60m request, provisional, pending the target replay** -(RAPPS-WP-0014-T04, separate worker). The replay namespace -`vergabe-sizing-20261021` (temporary 260m) appeared after the 06:08Z capture and -is excluded from the steady budget. Insert the replay result into -`vergabe_slot.replay_result` and the pilot-growth row of `build_review.py`, then -regenerate. A larger Vergabe request must fit inside the +140m serialized -residual together with the pilot-growth slot. +Source: railiance-apps `1af6d82`, +`docs/evidence/2026-10-21-vergabe-target-replay/` (RAPPS-WP-0014-T04). The replay +ran the deployed digest on the target node: 3 workers, 60m/256Mi request and +1000m/1Gi limit, in an isolated namespace that has since been torn down. + +- 3675 ordinary operations with 0 errors; ordinary p95 0.192s; transfer p95 0.076s. + The restart recovered in 19s. +- App CPU was 775m mean, 998m p95 and 1020m peak, throttled in 36% of periods + (66% during the burst phase). Memory peaked at 483MiB with no OOM. +- Incremental PostgreSQL demand was about 320m mean and 480m p95, measured on an + isolated PostgreSQL pod rather than shared apps-pg. +- The load was a ~25 req/s closed loop, far above what two human users generate. + +Decision (T02): +- **Keep the 60m CPU request and the 1000m/1Gi limits.** +- **Raise the memory request from 256Mi to 512Mi** in the next wave; the owner is + railiance-apps. +- **Keep apps-pg at 100m.** The rule would have proposed 50m; the replay justifies + keeping it. +- The ~0.3–0.5 core of incremental apps-pg load at synthetic rates is burst usage + under apps-pg's 1000m limit, not a reservation. Revisit if apps-pg CPU stays + above 600m. + +At the synthetic rate the pilot used about 1.1 core inside its limits, and node +free CPU at p5 fell to about 150m. That was a stress case, not the two-user +demand. + +The replay does not establish: +- production settings (SSO, ingress/TLS) +- contention on the shared apps-pg cluster +- observed human demand +- busy-hour node contention (CI, backups) +- human/recovery acceptance (RAPPS-WP-0014-T03) ## Receiving owners -None of the 54 non-keep rows has a live receiving task. RAIL-KNATIVE-WP-0002, +None of the 54 non-keep rows has a live receiving task yet. Wave 1 tasks (founder +decision 2026-10-21) are being prepared in rail-knative, railiance-forge and +state-hub, with rapp-telemetry following. Wave 2 is one task per remaining repo. RAIL-KNATIVE-WP-0002, RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover -admission and release, not sizing. The only live slot is RAPPS-WP-0014-T04 for -Vergabe. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving +admission and release, not sizing. RAPPS-WP-0014-T04 delivered the Vergabe replay; its memory raise needs a +railiance-apps receiving task. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving tasks by owner repo"). Each repo needs a receiving task: activity-core, audit-core, canned-prompts, core-hub, coulomb-social, email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative, -railiance-cluster, railiance-enablement, railiance-forge, railiance-platform, +railiance-apps, railiance-cluster, railiance-enablement, railiance-forge, railiance-platform, rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface, sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine. @@ -155,7 +191,7 @@ constant) and railiance-apps (inter-hub latent 250m). - No 7-day host/control-plane series. The 900m figure is a 70-second sample. - Single-node HPA ceilings are not reviewed by rail-knative. -- Vergabe target replay result (slot). +- Shared apps-pg behaviour under pilot load (the replay used an isolated database). - Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache. - Owner for `coulomb` namespace unconfirmed; flex-auth deployment source not available locally. diff --git a/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.json b/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.json index d6270c8..7504648 100644 --- a/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.json +++ b/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.json @@ -13,7 +13,7 @@ { "item": "Proposed steady pod requests (incl. Vergabe 60m provisional)", "current_m": 3605, - "proposed_m": 2400, + "proposed_m": 2450, "basis": "Kubernetes API 06:08Z + proposals" }, { @@ -47,23 +47,23 @@ "basis": "activator 20/gateway 10/webhook 5 -> proposed caps 3/3/2" }, { - "item": "Pilot growth reserve (Vergabe replay result + next tenant) \u2014 SLOT", + "item": "Pilot growth reserve (next tenant / user growth; Vergabe itself is in steady requests)", "current_m": 0, "proposed_m": 200, - "basis": "placeholder pending RAPPS-WP-0014-T04; not measured" + "basis": "Replay: two-user pilot needs no CPU request increase; reserve is unmeasured headroom for growth and human-paced apps-pg demand" } ], "budget_total_all_concurrent": { "current_m": 6635, - "proposed_m": 4340, + "proposed_m": 4390, "residual_current_m": -2635, - "residual_proposed_m": -340 + "residual_proposed_m": -390 }, "budget_total_serialized_events": { "current_m": 6075, - "proposed_m": 3860, + "proposed_m": 3910, "residual_current_m": -2075, - "residual_proposed_m": 140 + "residual_proposed_m": 90 }, "measured_demand_7d": { "containers_cpu_p95_m": 966, @@ -76,20 +76,63 @@ }, "memory_budget_mi": { "allocatable": 15993, - "requests_current": 9856, - "requests_proposed": 11704, "host_k3s_sample": 2604, "release_surge_state_hub_api": 864, "build_runner_burst": 512, - "pilot_growth_slot": 256, - "residual_current": 2509, - "residual_proposed": 53 + "pilot_growth_reserve": 256, + "current": { + "requests": 9856, + "total_all_concurrent": 13836, + "residual_all_concurrent": 2157, + "total_serialized": 13324, + "residual_serialized": 2669 + }, + "proposed": { + "requests": 12104, + "total_all_concurrent": 16340, + "residual_all_concurrent": -347, + "total_serialized": 15828, + "residual_serialized": 165 + } }, "vergabe_slot": { "current_request_m": 60, - "status": "provisional, pending target replay (RAPPS-WP-0014-T04)", - "replay_result": null + "proposed_request_m": 60, + "current_memory_request_mi": 256, + "proposed_memory_request_mi": 512, + "status": "replay-backed keep (CPU), memory request raise in next wave", + "replay_result": { + "source": "railiance-apps 1af6d82 docs/evidence/2026-10-21-vergabe-target-replay/ (RAPPS-WP-0014-T04)", + "image": "deployed digest sha256:a26444f5..., 3 gunicorn workers, 60m/256Mi request, 1000m/1Gi limit, target node", + "ordinary_ops": 3675, + "errors": 0, + "ordinary_p95_s": 0.192, + "transfer_p95_s": 0.076, + "restart_to_ready_s": 18.98, + "app_cpu_mean_p95_peak_m": [ + 775, + 998, + 1020 + ], + "app_throttle_ratio": 0.357, + "app_memory_peak_mib": 483, + "db_incremental_cpu_mean_p95_m": [ + 320, + 480 + ], + "db_note": "isolated PostgreSQL 16 pod, not shared CNPG apps-pg", + "rate": "~25 req/s closed loop, far above two human users", + "recommendation": "keep 60m request and 1000m/1Gi limits; raise memory request 256Mi->512Mi next wave; budget 0.3-0.5 core incremental apps-pg demand at synthetic load", + "not_established": [ + "production settings (SSO, ingress/TLS)", + "shared apps-pg contention", + "observed human demand", + "busy-hour node contention (CI, backups)", + "human/recovery acceptance (RAPPS-WP-0014-T03)" + ] + } }, + "pilot_stress_note": "At the synthetic rate the pilot used ~1.1 core (app ~775m + db ~320m) inside limits; that is burst usage, not reservation. During the replay node free CPU p5 fell to ~150m.", "workloads": [ { "namespace": "activity-core", @@ -367,29 +410,6 @@ "receiving_record": "needs receiving task in coulomb-social", "rationale": "Rule p95x1.5 / peakx0.5 / floor." }, - { - "namespace": "databases", - "workload": "apps-pg", - "pods": 1, - "cpu_request_m": 100, - "cpu_limit_m": 1000, - "containers_without_cpu_limit": 0, - "memory_request_mi": 256.0, - "memory_limit_mi": 1024.0, - "cpu_p95_m_7d": 8.0, - "cpu_peak_m_7d": 55.6, - "memory_p95_mi_7d": 89.1, - "memory_peak_mi_7d": 116.1, - "throttle_ratio_7d": 0.0, - "restarts_7d": 0.0, - "rule_cpu_m": 50, - "action": "decrease + memory down", - "proposed_cpu_request_m": 50, - "proposed_memory_request_mi": 112, - "owner_repo": "rapp-postgres", - "receiving_record": "needs receiving task in rapp-postgres", - "rationale": "Rule p95x1.5 / peakx0.5 / floor." - }, { "namespace": "databases", "workload": "forgejo-db", @@ -1349,12 +1369,12 @@ "throttle_ratio_7d": 0.0, "restarts_7d": 0.0, "rule_cpu_m": 15, - "action": "keep (provisional)", + "action": "keep CPU (replay-backed) + memory up", "proposed_cpu_request_m": 60, - "proposed_memory_request_mi": 256.0, + "proposed_memory_request_mi": 512, "owner_repo": "railiance-apps", - "receiving_record": "RAPPS-WP-0014-T04 target replay (SLOT, result pending)", - "rationale": "60m provisional, pending target replay (RAPPS-WP-0014-T04)." + "receiving_record": "needs receiving task in railiance-apps", + "rationale": "Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi." }, { "namespace": "activity-core", @@ -1471,6 +1491,29 @@ "receiving_record": null, "rationale": "Within rule tolerance; keep." }, + { + "namespace": "databases", + "workload": "apps-pg", + "pods": 1, + "cpu_request_m": 100, + "cpu_limit_m": 1000, + "containers_without_cpu_limit": 0, + "memory_request_mi": 256.0, + "memory_limit_mi": 1024.0, + "cpu_p95_m_7d": 8.0, + "cpu_peak_m_7d": 55.6, + "memory_p95_mi_7d": 89.1, + "memory_peak_mi_7d": 116.1, + "throttle_ratio_7d": 0.0, + "restarts_7d": 0.0, + "rule_cpu_m": 50, + "action": "keep", + "proposed_cpu_request_m": 100, + "proposed_memory_request_mi": 256.0, + "owner_repo": "rapp-postgres", + "receiving_record": null, + "rationale": "Keep 100m (rule would give 50m): Vergabe replay adds ~320m mean / ~480m p95 incremental PostgreSQL CPU at synthetic load; shared apps-pg behaviour unproven." + }, { "namespace": "informed-decision", "workload": "informed-decision", @@ -1673,6 +1716,9 @@ "knative-serving/net-kourier-controller", "kourier-system/3scale-kourier-gateway" ], + "railiance-apps": [ + "vergabe-demo-company/vergabe-teilnahme" + ], "railiance-cluster": [ "cert-manager/cainjector", "cert-manager/cert-manager", @@ -1699,7 +1745,6 @@ ], "rapp-postgres": [ "cnpg-system/cloudnative-pg", - "databases/apps-pg", "databases/forgejo-db", "databases/net-kingdom-pg", "databases/platform-pg", diff --git a/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md b/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md index 7913bc7..71b28ac 100644 --- a/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md +++ b/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md @@ -17,7 +17,6 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas | core-hub/rapp-core-hub | 3 | 200→60 | 2500 | 39.2/63.5 | 896→288 | 247/292 | 0.002 | decrease + memory down | core-hub | needs receiving task in core-hub | | coulomb/ihp-railiance-probe | 1 | 50→0 | 200 | 0.0/0.0 | 128→128 | 0/0 | None | retire/scale-to-zero (owner decision) | ihp-railiance-probe (unconfirmed; namespace owner unknown) | needs receiving task in ihp-railiance-probe | | coulomb-social/coulomb-social | 1 | 100→15 | 1000 | 0.4/24.6 | 256→176 | 146/146 | 0.0 | decrease + memory down | coulomb-social | needs receiving task in coulomb-social | -| databases/apps-pg | 1 | 100→50 | 1000 | 8.0/55.6 | 256→112 | 89/116 | 0.0 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres | | databases/forgejo-db | 1 | 200→105 | 1000 | 68.8/160.9 | 512→288 | 249/286 | 0.07 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres | | databases/net-kingdom-pg | 1 | 100→50 | 1000 | 10.3/48.0 | 256→256 | 229/497 | 0.001 | decrease | rapp-postgres | needs receiving task in rapp-postgres | | databases/platform-pg | 1 | 100→50 | 1000 | 13.0/68.7 | 256→192 | 160/264 | 0.001 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres | @@ -59,12 +58,13 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas | tenant-engine/tenant-engine | 1 | 25→5 | 300 | 2.4/2.6 | 48→48 | 53/68 | 0.0 | decrease | tenant-engine | needs receiving task in tenant-engine | | user-engine/user-engine | 1 | 50→5 | 500 | 0.4/4.2 | 64→64 | 56/57 | 0.0 | decrease | user-engine | needs receiving task in user-engine | | user-engine/user-engine-pg | 1 | 100→50 | 1000 | 7.5/9.7 | 256→96 | 78/103 | 0.0 | decrease + memory down | user-engine | needs receiving task in user-engine | -| vergabe-demo-company/vergabe-teilnahme | 1 | 60→60 | 1000 | 0.5/20.3 | 256→256 | 172/172 | 0.0 | keep (provisional) | railiance-apps | RAPPS-WP-0014-T04 target replay (SLOT, result pending) | +| vergabe-demo-company/vergabe-teilnahme | 1 | 60→60 | 1000 | 0.5/20.3 | 256→512 | 172/172 | 0.0 | keep CPU (replay-backed) + memory up | railiance-apps | needs receiving task in railiance-apps | | activity-core/actcore-temporal-ui-tls-2-1888679036-1756117428 | 1 | 10→10 | 100 | 0.0/0.1 | 64→64 | 13/18 | 0 | keep | activity-core | — | | approval-engine/approval-engine | 1 | 10→10 | 500 | 0.7/16.3 | 64→64 | 30/31 | 0.0 | keep | approval-engine | — | | argocd/argocd-applicationset-controller | 1 | 10→10 | 250 | 0.7/1.0 | 64→64 | 84/113 | 0 | keep | railiance-enablement | — | | argocd/argocd-redis | 1 | 10→10 | 200 | 5.0/5.1 | 32→32 | 6/27 | 0 | keep | railiance-enablement | — | | bao-notice/bao-notice | 1 | 5→5 | 100 | 0.0/0.1 | 16→16 | 4/4 | 0 | keep | railiance-platform | — | +| databases/apps-pg | 1 | 100→100 | 1000 | 8.0/55.6 | 256→256 | 89/116 | 0.0 | keep | rapp-postgres | — | | informed-decision/informed-decision | 1 | 5→5 | 500 | 1.2/23.2 | 64→64 | 39/47 | 0.0 | keep | informed-decision | — | | kube-system/svclb-mas-smtp-d86db2b1 | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 0/2 | None | keep | railiance-cluster | — | | kube-system/svclb-traefik-0c8aecaf | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 1/2 | None | keep | railiance-cluster | — | @@ -89,7 +89,6 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas - `core-hub/rapp-core-hub`: Rule p95x1.5 / peakx0.5 / floor. Production hub.coulomb.social: verify startup and rollout before lowering. - `coulomb/ihp-railiance-probe`: ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision. - `coulomb-social/coulomb-social`: Rule p95x1.5 / peakx0.5 / floor. -- `databases/apps-pg`: Rule p95x1.5 / peakx0.5 / floor. - `databases/forgejo-db`: Rule p95x1.5 / peakx0.5 / floor. - `databases/net-kingdom-pg`: Rule p95x1.5 / peakx0.5 / floor. - `databases/platform-pg`: Rule p95x1.5 / peakx0.5 / floor. @@ -131,7 +130,7 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas - `tenant-engine/tenant-engine`: Rule p95x1.5 / peakx0.5 / floor. - `user-engine/user-engine`: Rule p95x1.5 / peakx0.5 / floor. - `user-engine/user-engine-pg`: Rule p95x1.5 / peakx0.5 / floor. -- `vergabe-demo-company/vergabe-teilnahme`: 60m provisional, pending target replay (RAPPS-WP-0014-T04). +- `vergabe-demo-company/vergabe-teilnahme`: Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi. ## Missing receiving tasks by owner repo @@ -147,11 +146,12 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas - **mail-address-semantics** (1): mas/mas-mx - **net-kingdom** (2): mfa/net-kingdom-sso-mfa, sso/net-kingdom-sso-mfa - **rail-knative** (3): knative-serving/knative-serving, knative-serving/net-kourier-controller, kourier-system/3scale-kourier-gateway +- **railiance-apps** (1): vergabe-demo-company/vergabe-teilnahme - **railiance-cluster** (7): cert-manager/cainjector, cert-manager/cert-manager, cert-manager/webhook, kube-system/coredns-7bdb54f89, kube-system/local-path-provisioner, kube-system/metrics-server-786d997795, kube-system/traefik - **railiance-enablement** (2): argocd/argocd-application-controller, argocd/argocd-repo-server - **railiance-forge** (2): forgejo/forgejo-runner, forgejo/gitea - **railiance-platform** (5): external-secrets/external-secrets, external-secrets/external-secrets-cert-controller, external-secrets/external-secrets-webhook, openbao/railiance-platform, playwright/rapp-playwright -- **rapp-postgres** (6): cnpg-system/cloudnative-pg, databases/apps-pg, databases/forgejo-db, databases/net-kingdom-pg, databases/platform-pg, databases/platform-pg-2 +- **rapp-postgres** (5): cnpg-system/cloudnative-pg, databases/forgejo-db, databases/net-kingdom-pg, databases/platform-pg, databases/platform-pg-2 - **rapp-qonto** (2): rapp-qonto/rapp-qonto-audit-collector, rapp-qonto-egress/qonto-egress-proxy - **rapp-telemetry** (5): telemetry/alertmanager, telemetry/grafana, telemetry/kube-state-metrics, telemetry/prometheus, telemetry/rapp-telemetry - **rein-aharness** (1): rein-aharness/rein-aharness diff --git a/docs/evidence/2026-10-21-node-allocation-review/build_review.py b/docs/evidence/2026-10-21-node-allocation-review/build_review.py index af13031..98c9ac0 100644 --- a/docs/evidence/2026-10-21-node-allocation-review/build_review.py +++ b/docs/evidence/2026-10-21-node-allocation-review/build_review.py @@ -21,12 +21,22 @@ OWNER_OVERRIDE = {("databases", "state-hub-db"): "state-hub", ("target-revenue", DB_NS = {"databases"} CRITICAL_NS = {"kube-system", "knative-serving", "kourier-system", "sso", "mfa", "openbao", "external-secrets", "cert-manager", "cnpg-system", "argocd", "state-hub"} SAFETY_P95, SAFETY_PEAK, MIN_DELTA_M, MIN_DELTA_MI = 1.5, 0.5, 15, 64 -RECEIVING = {("vergabe-demo-company", "vergabe-teilnahme"): "RAPPS-WP-0014-T04 target replay (SLOT, result pending)"} +RECEIVING = {} +REPLAY = {"source": "railiance-apps 1af6d82 docs/evidence/2026-10-21-vergabe-target-replay/ (RAPPS-WP-0014-T04)", + "image": "deployed digest sha256:a26444f5..., 3 gunicorn workers, 60m/256Mi request, 1000m/1Gi limit, target node", + "ordinary_ops": 3675, "errors": 0, "ordinary_p95_s": 0.192, "transfer_p95_s": 0.076, "restart_to_ready_s": 18.98, + "app_cpu_mean_p95_peak_m": [775, 998, 1020], "app_throttle_ratio": 0.357, "app_memory_peak_mib": 483, + "db_incremental_cpu_mean_p95_m": [320, 480], "db_note": "isolated PostgreSQL 16 pod, not shared CNPG apps-pg", + "rate": "~25 req/s closed loop, far above two human users", + "recommendation": "keep 60m request and 1000m/1Gi limits; raise memory request 256Mi->512Mi next wave; budget 0.3-0.5 core incremental apps-pg demand at synthetic load", + "not_established": ["production settings (SSO, ingress/TLS)", "shared apps-pg contention", "observed human demand", "busy-hour node contention (CI, backups)", "human/recovery acceptance (RAPPS-WP-0014-T03)"]} +MEM_OVERRIDES = {("vergabe-demo-company", "vergabe-teilnahme"): 512} OVERRIDES = { # (ns, workload): (cpu_m, reason) ("coulomb", "ihp-railiance-probe"): (0, "ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision."), - ("vergabe-demo-company", "vergabe-teilnahme"): (60, "60m provisional, pending target replay (RAPPS-WP-0014-T04)."), + ("vergabe-demo-company", "vergabe-teilnahme"): (60, "Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi."), ("kube-system", "svclb-traefik-0c8aecaf"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."), ("kube-system", "svclb-mas-smtp-d86db2b1"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."), + ("databases", "apps-pg"): (100, "Keep 100m (rule would give 50m): Vergabe replay adds ~320m mean / ~480m p95 incremental PostgreSQL CPU at synthetic load; shared apps-pg behaviour unproven."), ("kube-system", "coredns-7bdb54f89"): (50, "DNS for every workload: floor 50m regardless of p95 7.8m."), ("state-hub", "railiance-apps"): (390, "API p95 258m at 250m request = p95x1.5; the 975m peak is release/migration bursts carried by the surge reserve. Preflight models API surge 100m but live API pod requests 250m."), } @@ -65,10 +75,11 @@ for key, w in usage.items(): if key in NOTES: why += " " + NOTES[key] action = "keep" if req == cur else ("increase" if req > cur else "decrease") if key == ("coulomb", "ihp-railiance-probe"): action = "retire/scale-to-zero (owner decision)" - if key == ("vergabe-demo-company", "vergabe-teilnahme"): action = "keep (provisional)" + if key == ("vergabe-demo-company", "vergabe-teilnahme"): action = "keep CPU (replay-backed)" if key in {("sso", "net-kingdom-sso-mfa"), ("playwright", "rapp-playwright")}: action += " + review CPU limit (throttled)" mem_target = max(16, int(math.ceil(w["mem_p95_mi_sum"] * 1.15 / 16) * 16)) mem = mem_target if abs(mem_target - w["mem_req_mi"]) >= MIN_DELTA_MI and key not in OVERRIDES else w["mem_req_mi"] + mem = MEM_OVERRIDES.get(key, mem) if mem != w["mem_req_mi"]: action += " + memory " + ("up" if mem > w["mem_req_mi"] else "down") owner = OWNER_OVERRIDE.get(key) or (OWNERS.get(ns) or {}).get("owner", "unknown") record = RECEIVING.get(key) or (None if action == "keep" else f"needs receiving task in {owner.split(' ')[0]}") @@ -90,10 +101,19 @@ budget = [ ("Admitted factory/build execution (one runner job burst above runner request)", 250, 250, "forgejo-runner pod peak 243m; namespace peak 1033m incl. gitea"), ("Maintenance CronJobs (7 jobs, 10-25m, overlapping 02:40-04:00)", 50, 50, "kubectl cronjobs; keycape-factor-renewer every 10 min"), ("Knative HPA scale-out headroom (requests at max replicas minus 1)", hpa_now, hpa_capped, "activator 20/gateway 10/webhook 5 -> proposed caps 3/3/2"), - ("Pilot growth reserve (Vergabe replay result + next tenant) — SLOT", 0, 200, "placeholder pending RAPPS-WP-0014-T04; not measured"), + ("Pilot growth reserve (next tenant / user growth; Vergabe itself is in steady requests)", 0, 200, "Replay: two-user pilot needs no CPU request increase; reserve is unmeasured headroom for growth and human-paced apps-pg demand"), ] cur_sum = sum(b[1] for b in budget); prop_sum = sum(b[2] for b in budget) # Serialized: one disruptive event (release surge, build burst or HPA step) at a time. +ALLOC_MI = round(16770080768 / 2**20) +HOST_MI, SURGE_MI, BUILD_MI, PILOT_MI = 2604, 864, 512, 256 +def _mem(req, pilot): + allc = req + HOST_MI + SURGE_MI + BUILD_MI + pilot + ser_ = req + HOST_MI + max(SURGE_MI, BUILD_MI) + pilot + return {"requests": round(req), "total_all_concurrent": round(allc), "residual_all_concurrent": ALLOC_MI - round(allc), + "total_serialized": round(ser_), "residual_serialized": ALLOC_MI - round(ser_)} +mem_budget = {"allocatable": ALLOC_MI, "host_k3s_sample": HOST_MI, "release_surge_state_hub_api": SURGE_MI, "build_runner_burst": BUILD_MI, + "pilot_growth_reserve": PILOT_MI, "current": _mem(cur_mem, 0), "proposed": _mem(prop_mem, PILOT_MI)} ser = lambda i: budget[0][i] + budget[1][i] + max(budget[2][i], budget[3][i], budget[5][i]) + budget[4][i] + budget[6][i] out = {"schema": "resource-control.node-allocation-review.v1", "workplan": "RESOURCE-WP-0008", "beneficiary": "CUST-WP-0071-T03", "observed_at": "2026-10-21T06:08:20Z", "allocatable_cpu_m": 4000, "allocatable_memory_mi": round(16770080768 / 2**20), @@ -104,11 +124,10 @@ out = {"schema": "resource-control.node-allocation-review.v1", "workplan": "RESO "budget_total_serialized_events": {"current_m": ser(1), "proposed_m": ser(2), "residual_current_m": 4000 - ser(1), "residual_proposed_m": 4000 - ser(2)}, "measured_demand_7d": {"containers_cpu_p95_m": 966, "containers_cpu_peak_m": 1740, "host_non_pod_cpu_m_sample": [677, 871], "note": "Measured demand (containers p95 + host) ~1.8 cores, peaks ~2.6 cores; the deficit is reservation, not demand."}, - "memory_budget_mi": {"allocatable": round(16770080768 / 2**20), "requests_current": round(cur_mem), "requests_proposed": round(prop_mem), - "host_k3s_sample": 2604, "release_surge_state_hub_api": 864, "build_runner_burst": 512, "pilot_growth_slot": 256, - "residual_current": round(16770080768 / 2**20 - cur_mem - 2604 - 512 - 512), - "residual_proposed": round(16770080768 / 2**20 - prop_mem - 2604 - 864 - 512 - 256)}, - "vergabe_slot": {"current_request_m": 60, "status": "provisional, pending target replay (RAPPS-WP-0014-T04)", "replay_result": None}, + "memory_budget_mi": mem_budget, + "vergabe_slot": {"current_request_m": 60, "proposed_request_m": 60, "current_memory_request_mi": 256, "proposed_memory_request_mi": 512, + "status": "replay-backed keep (CPU), memory request raise in next wave", "replay_result": REPLAY}, + "pilot_stress_note": "At the synthetic rate the pilot used ~1.1 core (app ~775m + db ~320m) inside limits; that is burst usage, not reservation. During the replay node free CPU p5 fell to ~150m.", "workloads": sorted(rows, key=lambda r: (r["action"] == "keep", r["namespace"], r["workload"]))} (HERE / "2026-10-21-node-budget.json").write_text(json.dumps(out, indent=1) + "\n") for b in out["budget_rows"]: print(f"{b['item'][:70]:70} {b['current_m']:6} {b['proposed_m']:6}") diff --git a/workplans/RESOURCE-WP-0008-custodian-allocation-review.md b/workplans/RESOURCE-WP-0008-custodian-allocation-review.md index d7f17e2..0b2d3f3 100644 --- a/workplans/RESOURCE-WP-0008-custodian-allocation-review.md +++ b/workplans/RESOURCE-WP-0008-custodian-allocation-review.md @@ -42,9 +42,8 @@ refreshing stale utilization metrics or inventing financial attribution. ```task id: RESOURCE-WP-0008-T02 -status: progress +status: done priority: high -blocking_reason: "Node budget and per-workload comparison done 2026-10-21; only the Vergabe target replay result (RAPPS-WP-0014-T04) remains to be inserted into the 60m provisional slot." state_hub_task_id: "1f538441-8b45-5d37-aeb3-345204b75117" ``` @@ -66,7 +65,7 @@ whole node budget fits. Do not infer human workflow acceptance. id: RESOURCE-WP-0008-T03 status: progress priority: medium -blocking_reason: "Proposal and node budget published 2026-10-21, but none of the 54 owner changes has a receiving task in its owner repo (26 repos listed in the review), the coulomb probe owner is unconfirmed, and the Vergabe slot awaits RAPPS-WP-0014-T04." +blocking_reason: "Proposal and node budget published 2026-10-21 (Vergabe replay consumed). 54 owner changes in 27 repos still lack receiving tasks: wave 1 (rail-knative, railiance-forge, state-hub, then rapp-telemetry) is being prepared, wave 2 per-repo tasks follow, and the coulomb probe owner must be found first." state_hub_task_id: "3e6e6a59-71a5-5048-b7a5-ed61416a442c" ``` @@ -102,3 +101,26 @@ T03 cannot finish because every non-Vergabe change still needs a receiving task its owner repo. The per-repo list is in `2026-10-21-node-budget.md`. Creating those records is the owners' or the Custodian's decision; this record does not create them. + +## Vergabe replay consumed — 2026-10-21 + +Source: railiance-apps `1af6d82` (RAPPS-WP-0014-T04), which replayed the deployed +digest on the target node. Results: 0 errors in 3675 ordinary operations, ordinary +p95 0.192s, app CPU p95 998m under 36% throttling, memory peak 483MiB, and +incremental database CPU of about 320m mean / 480m p95 on an isolated PostgreSQL. + +T02 is done, with this qualified allocation: +- Keep the 60m request and the 1000m/1Gi limits. +- Raise the memory request from 256Mi to 512Mi in the next wave (receiving owner + railiance-apps). +- Keep apps-pg at 100m rather than the rule's 50m. + +Regenerated budget, all events concurrent: CPU −390m, memory −347Mi. +Regenerated budget, one disruptive event at a time: CPU +90m, memory +165Mi. + +The replay does not establish shared-apps-pg contention, production settings, +human demand or human acceptance (RAPPS-WP-0014-T03). + +T03 stays in progress under the founder's wave rollout. Wave 1 is rail-knative, +railiance-forge, state-hub and rapp-telemetry; wave 2 is batched per-repo tasks. +The ihp-railiance-probe owner must be confirmed first.