Consume Vergabe target replay into node allocation budget; finish T02
Fill the Vergabe slot from railiance-apps 1af6d82 (RAPPS-WP-0014-T04): keep 60m request and 1000m/1Gi limits, raise memory request 256Mi->512Mi next wave, keep apps-pg at 100m for incremental pilot DB demand. Regenerated budget: serialized CPU +90m / memory +165Mi; all-concurrent CPU -390m / memory -347Mi. T03 stays in progress pending wave-1/wave-2 receiving tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 8340@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c
This commit is contained in:
parent
e72c5de92f
commit
3d9b1bd080
6 changed files with 208 additions and 86 deletions
|
|
@ -51,5 +51,5 @@
|
|||
| task | RESOURCE-WP-0006-T04 | done | — | workplans/RESOURCE-WP-0006-resource-object-and-reef-refs.md |
|
||||
| task | RESOURCE-WP-0007-T01 | done | — | workplans/RESOURCE-WP-0007-state-hub-release-margin.md |
|
||||
| task | RESOURCE-WP-0008-T01 | done | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T02 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T02 | done | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
| task | RESOURCE-WP-0008-T03 | progress | — | workplans/RESOURCE-WP-0008-custodian-allocation-review.md |
|
||||
|
|
|
|||
|
|
@ -91,58 +91,94 @@ manifest revision (CUST-WP-0071-T04 retains the receipts).
|
|||
|
||||
| Item | Current | Proposed |
|
||||
| --- | ---: | ---: |
|
||||
| Steady pod requests (incl. Vergabe 60m provisional) | 3605 | 2400 |
|
||||
| Steady pod requests (incl. Vergabe 60m, apps-pg kept at 100m) | 3605 | 2450 |
|
||||
| Host / k3s control plane (no kube/system-reserved) | 900 | 900 |
|
||||
| Release surge + migration hook (state-hub, one at a time) | 310 | 310 |
|
||||
| Admitted factory/build execution (one runner job burst) | 250 | 250 |
|
||||
| Maintenance CronJobs | 50 | 50 |
|
||||
| Knative HPA scale-out headroom | 1520 | 230 |
|
||||
| Pilot growth reserve — **SLOT, pending RAPPS-WP-0014-T04** | 0 | 200 |
|
||||
| **Total, all events concurrent** | **6635** | **4340** |
|
||||
| **Residual vs 4000m, all concurrent** | **−2635** | **−340** |
|
||||
| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3860 |
|
||||
| Residual, serialized | −2075 | **+140** |
|
||||
| Pilot growth reserve (next tenant/user growth; Vergabe is in steady requests) | 0 | 200 |
|
||||
| **Total, all events concurrent** | **6635** | **4390** |
|
||||
| **Residual vs 4000m, all concurrent** | **−2635** | **−390** |
|
||||
| Total, one disruptive event at a time (max of surge/build/HPA) | 6075 | 3910 |
|
||||
| Residual, serialized | −2075 | **+90** |
|
||||
|
||||
Proposed changes: +660m in increases and −1865m in decreases, across 55 non-keep
|
||||
rows (54 owner changes plus the Vergabe slot).
|
||||
Proposed changes: +660m increases and −1815m decreases, across 54 non-keep rows.
|
||||
|
||||
Memory (Mi): proposed requests 11704 + k3s 2604 + surge 864 + build 512 + pilot
|
||||
slot 256 leaves +53 of 15993. It fits, but only just. The k3s figure is a single
|
||||
sample.
|
||||
Memory (Mi, allocatable 15993; k3s 2604 single sample, surge 864, build 512,
|
||||
pilot growth 256):
|
||||
|
||||
**Reconciliation verdict:** the budget reconciles only conditionally, at +140m. That
|
||||
| Memory | Current | Proposed |
|
||||
| --- | ---: | ---: |
|
||||
| Pod requests | 9856 | 12104 |
|
||||
| Residual, all concurrent | +2157 | **−347** |
|
||||
| Residual, serialized (max of surge/build) | +2669 | **+165** |
|
||||
|
||||
The current memory residual looks comfortable only because 13 zero-request
|
||||
workloads hide about 1.3Gi of working set. The proposed requests include the
|
||||
Vergabe raise (+256Mi), and apps-pg keeps its 256Mi.
|
||||
|
||||
**Reconciliation verdict:** the budget reconciles only conditionally, at +90m CPU and +165Mi memory. That
|
||||
needs three things: the lean decreases are accepted, releases, builds and Knative
|
||||
scale-out are serialized (runner concurrency 1, STATE-WP-0091 preflight before
|
||||
releases), and the HPA ceilings are capped. With every event concurrent it is
|
||||
−340m short. Measured demand plus margin does fit the 4 vCPU. **Procurement is
|
||||
−390m CPU and −347Mi memory short. Measured demand plus margin does fit the 4 vCPU. **Procurement is
|
||||
not indicated**; the remedy is to correct reservations. Once requests have come
|
||||
down, railiance-cluster should make host operations explicit with
|
||||
`kube-reserved`/`system-reserved` (~900m/2.5Gi). Applying that first would make
|
||||
the current 3605m unschedulable.
|
||||
|
||||
## Vergabe slot
|
||||
## Vergabe slot (filled 2026-10-21 from the target replay)
|
||||
|
||||
Vergabe stays at **60m request, provisional, pending the target replay**
|
||||
(RAPPS-WP-0014-T04, separate worker). The replay namespace
|
||||
`vergabe-sizing-20261021` (temporary 260m) appeared after the 06:08Z capture and
|
||||
is excluded from the steady budget. Insert the replay result into
|
||||
`vergabe_slot.replay_result` and the pilot-growth row of `build_review.py`, then
|
||||
regenerate. A larger Vergabe request must fit inside the +140m serialized
|
||||
residual together with the pilot-growth slot.
|
||||
Source: railiance-apps `1af6d82`,
|
||||
`docs/evidence/2026-10-21-vergabe-target-replay/` (RAPPS-WP-0014-T04). The replay
|
||||
ran the deployed digest on the target node: 3 workers, 60m/256Mi request and
|
||||
1000m/1Gi limit, in an isolated namespace that has since been torn down.
|
||||
|
||||
- 3675 ordinary operations with 0 errors; ordinary p95 0.192s; transfer p95 0.076s.
|
||||
The restart recovered in 19s.
|
||||
- App CPU was 775m mean, 998m p95 and 1020m peak, throttled in 36% of periods
|
||||
(66% during the burst phase). Memory peaked at 483MiB with no OOM.
|
||||
- Incremental PostgreSQL demand was about 320m mean and 480m p95, measured on an
|
||||
isolated PostgreSQL pod rather than shared apps-pg.
|
||||
- The load was a ~25 req/s closed loop, far above what two human users generate.
|
||||
|
||||
Decision (T02):
|
||||
- **Keep the 60m CPU request and the 1000m/1Gi limits.**
|
||||
- **Raise the memory request from 256Mi to 512Mi** in the next wave; the owner is
|
||||
railiance-apps.
|
||||
- **Keep apps-pg at 100m.** The rule would have proposed 50m; the replay justifies
|
||||
keeping it.
|
||||
- The ~0.3–0.5 core of incremental apps-pg load at synthetic rates is burst usage
|
||||
under apps-pg's 1000m limit, not a reservation. Revisit if apps-pg CPU stays
|
||||
above 600m.
|
||||
|
||||
At the synthetic rate the pilot used about 1.1 core inside its limits, and node
|
||||
free CPU at p5 fell to about 150m. That was a stress case, not the two-user
|
||||
demand.
|
||||
|
||||
The replay does not establish:
|
||||
- production settings (SSO, ingress/TLS)
|
||||
- contention on the shared apps-pg cluster
|
||||
- observed human demand
|
||||
- busy-hour node contention (CI, backups)
|
||||
- human/recovery acceptance (RAPPS-WP-0014-T03)
|
||||
|
||||
## Receiving owners
|
||||
|
||||
None of the 54 non-keep rows has a live receiving task. RAIL-KNATIVE-WP-0002,
|
||||
None of the 54 non-keep rows has a live receiving task yet. Wave 1 tasks (founder
|
||||
decision 2026-10-21) are being prepared in rail-knative, railiance-forge and
|
||||
state-hub, with rapp-telemetry following. Wave 2 is one task per remaining repo. RAIL-KNATIVE-WP-0002,
|
||||
RFORGE-WP-0002, RCLUSTER-WP-0014, RAIL-BS-WP-0014 and STATE-WP-0091 are
|
||||
finished. RPF-WP-0056 (playwright) and RAPPS-WP-0015 (coulomb-social) cover
|
||||
admission and release, not sizing. The only live slot is RAPPS-WP-0014-T04 for
|
||||
Vergabe. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving
|
||||
admission and release, not sizing. RAPPS-WP-0014-T04 delivered the Vergabe replay; its memory raise needs a
|
||||
railiance-apps receiving task. Per-repo lists are in `2026-10-21-node-budget.md` ("Missing receiving
|
||||
tasks by owner repo"). Each repo needs a receiving task:
|
||||
|
||||
activity-core, audit-core, canned-prompts, core-hub, coulomb-social,
|
||||
email-connect, flex-auth (no local checkout), ihp-railiance-probe (owner
|
||||
unconfirmed), issue-core, mail-address-semantics, net-kingdom, rail-knative,
|
||||
railiance-cluster, railiance-enablement, railiance-forge, railiance-platform,
|
||||
railiance-apps, railiance-cluster, railiance-enablement, railiance-forge, railiance-platform,
|
||||
rapp-postgres, rapp-qonto, rapp-telemetry, rein-aharness, reuse-surface,
|
||||
sbom-nexus, state-hub, target-revenue, tenant-engine, user-engine.
|
||||
|
||||
|
|
@ -155,7 +191,7 @@ constant) and railiance-apps (inter-hub latent 250m).
|
|||
|
||||
- No 7-day host/control-plane series. The 900m figure is a 70-second sample.
|
||||
- Single-node HPA ceilings are not reviewed by rail-knative.
|
||||
- Vergabe target replay result (slot).
|
||||
- Shared apps-pg behaviour under pilot load (the replay used an isolated database).
|
||||
- Per-pod p95 sums overstate multi-pod workloads. Memory p95 includes page cache.
|
||||
- Owner for `coulomb` namespace unconfirmed; flex-auth deployment source not
|
||||
available locally.
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
{
|
||||
"item": "Proposed steady pod requests (incl. Vergabe 60m provisional)",
|
||||
"current_m": 3605,
|
||||
"proposed_m": 2400,
|
||||
"proposed_m": 2450,
|
||||
"basis": "Kubernetes API 06:08Z + proposals"
|
||||
},
|
||||
{
|
||||
|
|
@ -47,23 +47,23 @@
|
|||
"basis": "activator 20/gateway 10/webhook 5 -> proposed caps 3/3/2"
|
||||
},
|
||||
{
|
||||
"item": "Pilot growth reserve (Vergabe replay result + next tenant) \u2014 SLOT",
|
||||
"item": "Pilot growth reserve (next tenant / user growth; Vergabe itself is in steady requests)",
|
||||
"current_m": 0,
|
||||
"proposed_m": 200,
|
||||
"basis": "placeholder pending RAPPS-WP-0014-T04; not measured"
|
||||
"basis": "Replay: two-user pilot needs no CPU request increase; reserve is unmeasured headroom for growth and human-paced apps-pg demand"
|
||||
}
|
||||
],
|
||||
"budget_total_all_concurrent": {
|
||||
"current_m": 6635,
|
||||
"proposed_m": 4340,
|
||||
"proposed_m": 4390,
|
||||
"residual_current_m": -2635,
|
||||
"residual_proposed_m": -340
|
||||
"residual_proposed_m": -390
|
||||
},
|
||||
"budget_total_serialized_events": {
|
||||
"current_m": 6075,
|
||||
"proposed_m": 3860,
|
||||
"proposed_m": 3910,
|
||||
"residual_current_m": -2075,
|
||||
"residual_proposed_m": 140
|
||||
"residual_proposed_m": 90
|
||||
},
|
||||
"measured_demand_7d": {
|
||||
"containers_cpu_p95_m": 966,
|
||||
|
|
@ -76,20 +76,63 @@
|
|||
},
|
||||
"memory_budget_mi": {
|
||||
"allocatable": 15993,
|
||||
"requests_current": 9856,
|
||||
"requests_proposed": 11704,
|
||||
"host_k3s_sample": 2604,
|
||||
"release_surge_state_hub_api": 864,
|
||||
"build_runner_burst": 512,
|
||||
"pilot_growth_slot": 256,
|
||||
"residual_current": 2509,
|
||||
"residual_proposed": 53
|
||||
"pilot_growth_reserve": 256,
|
||||
"current": {
|
||||
"requests": 9856,
|
||||
"total_all_concurrent": 13836,
|
||||
"residual_all_concurrent": 2157,
|
||||
"total_serialized": 13324,
|
||||
"residual_serialized": 2669
|
||||
},
|
||||
"proposed": {
|
||||
"requests": 12104,
|
||||
"total_all_concurrent": 16340,
|
||||
"residual_all_concurrent": -347,
|
||||
"total_serialized": 15828,
|
||||
"residual_serialized": 165
|
||||
}
|
||||
},
|
||||
"vergabe_slot": {
|
||||
"current_request_m": 60,
|
||||
"status": "provisional, pending target replay (RAPPS-WP-0014-T04)",
|
||||
"replay_result": null
|
||||
"proposed_request_m": 60,
|
||||
"current_memory_request_mi": 256,
|
||||
"proposed_memory_request_mi": 512,
|
||||
"status": "replay-backed keep (CPU), memory request raise in next wave",
|
||||
"replay_result": {
|
||||
"source": "railiance-apps 1af6d82 docs/evidence/2026-10-21-vergabe-target-replay/ (RAPPS-WP-0014-T04)",
|
||||
"image": "deployed digest sha256:a26444f5..., 3 gunicorn workers, 60m/256Mi request, 1000m/1Gi limit, target node",
|
||||
"ordinary_ops": 3675,
|
||||
"errors": 0,
|
||||
"ordinary_p95_s": 0.192,
|
||||
"transfer_p95_s": 0.076,
|
||||
"restart_to_ready_s": 18.98,
|
||||
"app_cpu_mean_p95_peak_m": [
|
||||
775,
|
||||
998,
|
||||
1020
|
||||
],
|
||||
"app_throttle_ratio": 0.357,
|
||||
"app_memory_peak_mib": 483,
|
||||
"db_incremental_cpu_mean_p95_m": [
|
||||
320,
|
||||
480
|
||||
],
|
||||
"db_note": "isolated PostgreSQL 16 pod, not shared CNPG apps-pg",
|
||||
"rate": "~25 req/s closed loop, far above two human users",
|
||||
"recommendation": "keep 60m request and 1000m/1Gi limits; raise memory request 256Mi->512Mi next wave; budget 0.3-0.5 core incremental apps-pg demand at synthetic load",
|
||||
"not_established": [
|
||||
"production settings (SSO, ingress/TLS)",
|
||||
"shared apps-pg contention",
|
||||
"observed human demand",
|
||||
"busy-hour node contention (CI, backups)",
|
||||
"human/recovery acceptance (RAPPS-WP-0014-T03)"
|
||||
]
|
||||
}
|
||||
},
|
||||
"pilot_stress_note": "At the synthetic rate the pilot used ~1.1 core (app ~775m + db ~320m) inside limits; that is burst usage, not reservation. During the replay node free CPU p5 fell to ~150m.",
|
||||
"workloads": [
|
||||
{
|
||||
"namespace": "activity-core",
|
||||
|
|
@ -367,29 +410,6 @@
|
|||
"receiving_record": "needs receiving task in coulomb-social",
|
||||
"rationale": "Rule p95x1.5 / peakx0.5 / floor."
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "apps-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"cpu_limit_m": 1000,
|
||||
"containers_without_cpu_limit": 0,
|
||||
"memory_request_mi": 256.0,
|
||||
"memory_limit_mi": 1024.0,
|
||||
"cpu_p95_m_7d": 8.0,
|
||||
"cpu_peak_m_7d": 55.6,
|
||||
"memory_p95_mi_7d": 89.1,
|
||||
"memory_peak_mi_7d": 116.1,
|
||||
"throttle_ratio_7d": 0.0,
|
||||
"restarts_7d": 0.0,
|
||||
"rule_cpu_m": 50,
|
||||
"action": "decrease + memory down",
|
||||
"proposed_cpu_request_m": 50,
|
||||
"proposed_memory_request_mi": 112,
|
||||
"owner_repo": "rapp-postgres",
|
||||
"receiving_record": "needs receiving task in rapp-postgres",
|
||||
"rationale": "Rule p95x1.5 / peakx0.5 / floor."
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "forgejo-db",
|
||||
|
|
@ -1349,12 +1369,12 @@
|
|||
"throttle_ratio_7d": 0.0,
|
||||
"restarts_7d": 0.0,
|
||||
"rule_cpu_m": 15,
|
||||
"action": "keep (provisional)",
|
||||
"action": "keep CPU (replay-backed) + memory up",
|
||||
"proposed_cpu_request_m": 60,
|
||||
"proposed_memory_request_mi": 256.0,
|
||||
"proposed_memory_request_mi": 512,
|
||||
"owner_repo": "railiance-apps",
|
||||
"receiving_record": "RAPPS-WP-0014-T04 target replay (SLOT, result pending)",
|
||||
"rationale": "60m provisional, pending target replay (RAPPS-WP-0014-T04)."
|
||||
"receiving_record": "needs receiving task in railiance-apps",
|
||||
"rationale": "Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi."
|
||||
},
|
||||
{
|
||||
"namespace": "activity-core",
|
||||
|
|
@ -1471,6 +1491,29 @@
|
|||
"receiving_record": null,
|
||||
"rationale": "Within rule tolerance; keep."
|
||||
},
|
||||
{
|
||||
"namespace": "databases",
|
||||
"workload": "apps-pg",
|
||||
"pods": 1,
|
||||
"cpu_request_m": 100,
|
||||
"cpu_limit_m": 1000,
|
||||
"containers_without_cpu_limit": 0,
|
||||
"memory_request_mi": 256.0,
|
||||
"memory_limit_mi": 1024.0,
|
||||
"cpu_p95_m_7d": 8.0,
|
||||
"cpu_peak_m_7d": 55.6,
|
||||
"memory_p95_mi_7d": 89.1,
|
||||
"memory_peak_mi_7d": 116.1,
|
||||
"throttle_ratio_7d": 0.0,
|
||||
"restarts_7d": 0.0,
|
||||
"rule_cpu_m": 50,
|
||||
"action": "keep",
|
||||
"proposed_cpu_request_m": 100,
|
||||
"proposed_memory_request_mi": 256.0,
|
||||
"owner_repo": "rapp-postgres",
|
||||
"receiving_record": null,
|
||||
"rationale": "Keep 100m (rule would give 50m): Vergabe replay adds ~320m mean / ~480m p95 incremental PostgreSQL CPU at synthetic load; shared apps-pg behaviour unproven."
|
||||
},
|
||||
{
|
||||
"namespace": "informed-decision",
|
||||
"workload": "informed-decision",
|
||||
|
|
@ -1673,6 +1716,9 @@
|
|||
"knative-serving/net-kourier-controller",
|
||||
"kourier-system/3scale-kourier-gateway"
|
||||
],
|
||||
"railiance-apps": [
|
||||
"vergabe-demo-company/vergabe-teilnahme"
|
||||
],
|
||||
"railiance-cluster": [
|
||||
"cert-manager/cainjector",
|
||||
"cert-manager/cert-manager",
|
||||
|
|
@ -1699,7 +1745,6 @@
|
|||
],
|
||||
"rapp-postgres": [
|
||||
"cnpg-system/cloudnative-pg",
|
||||
"databases/apps-pg",
|
||||
"databases/forgejo-db",
|
||||
"databases/net-kingdom-pg",
|
||||
"databases/platform-pg",
|
||||
|
|
|
|||
|
|
@ -17,7 +17,6 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas
|
|||
| core-hub/rapp-core-hub | 3 | 200→60 | 2500 | 39.2/63.5 | 896→288 | 247/292 | 0.002 | decrease + memory down | core-hub | needs receiving task in core-hub |
|
||||
| coulomb/ihp-railiance-probe | 1 | 50→0 | 200 | 0.0/0.0 | 128→128 | 0/0 | None | retire/scale-to-zero (owner decision) | ihp-railiance-probe (unconfirmed; namespace owner unknown) | needs receiving task in ihp-railiance-probe |
|
||||
| coulomb-social/coulomb-social | 1 | 100→15 | 1000 | 0.4/24.6 | 256→176 | 146/146 | 0.0 | decrease + memory down | coulomb-social | needs receiving task in coulomb-social |
|
||||
| databases/apps-pg | 1 | 100→50 | 1000 | 8.0/55.6 | 256→112 | 89/116 | 0.0 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/forgejo-db | 1 | 200→105 | 1000 | 68.8/160.9 | 512→288 | 249/286 | 0.07 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/net-kingdom-pg | 1 | 100→50 | 1000 | 10.3/48.0 | 256→256 | 229/497 | 0.001 | decrease | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
| databases/platform-pg | 1 | 100→50 | 1000 | 13.0/68.7 | 256→192 | 160/264 | 0.001 | decrease + memory down | rapp-postgres | needs receiving task in rapp-postgres |
|
||||
|
|
@ -59,12 +58,13 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas
|
|||
| tenant-engine/tenant-engine | 1 | 25→5 | 300 | 2.4/2.6 | 48→48 | 53/68 | 0.0 | decrease | tenant-engine | needs receiving task in tenant-engine |
|
||||
| user-engine/user-engine | 1 | 50→5 | 500 | 0.4/4.2 | 64→64 | 56/57 | 0.0 | decrease | user-engine | needs receiving task in user-engine |
|
||||
| user-engine/user-engine-pg | 1 | 100→50 | 1000 | 7.5/9.7 | 256→96 | 78/103 | 0.0 | decrease + memory down | user-engine | needs receiving task in user-engine |
|
||||
| vergabe-demo-company/vergabe-teilnahme | 1 | 60→60 | 1000 | 0.5/20.3 | 256→256 | 172/172 | 0.0 | keep (provisional) | railiance-apps | RAPPS-WP-0014-T04 target replay (SLOT, result pending) |
|
||||
| vergabe-demo-company/vergabe-teilnahme | 1 | 60→60 | 1000 | 0.5/20.3 | 256→512 | 172/172 | 0.0 | keep CPU (replay-backed) + memory up | railiance-apps | needs receiving task in railiance-apps |
|
||||
| activity-core/actcore-temporal-ui-tls-2-1888679036-1756117428 | 1 | 10→10 | 100 | 0.0/0.1 | 64→64 | 13/18 | 0 | keep | activity-core | — |
|
||||
| approval-engine/approval-engine | 1 | 10→10 | 500 | 0.7/16.3 | 64→64 | 30/31 | 0.0 | keep | approval-engine | — |
|
||||
| argocd/argocd-applicationset-controller | 1 | 10→10 | 250 | 0.7/1.0 | 64→64 | 84/113 | 0 | keep | railiance-enablement | — |
|
||||
| argocd/argocd-redis | 1 | 10→10 | 200 | 5.0/5.1 | 32→32 | 6/27 | 0 | keep | railiance-enablement | — |
|
||||
| bao-notice/bao-notice | 1 | 5→5 | 100 | 0.0/0.1 | 16→16 | 4/4 | 0 | keep | railiance-platform | — |
|
||||
| databases/apps-pg | 1 | 100→100 | 1000 | 8.0/55.6 | 256→256 | 89/116 | 0.0 | keep | rapp-postgres | — |
|
||||
| informed-decision/informed-decision | 1 | 5→5 | 500 | 1.2/23.2 | 64→64 | 39/47 | 0.0 | keep | informed-decision | — |
|
||||
| kube-system/svclb-mas-smtp-d86db2b1 | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 0/2 | None | keep | railiance-cluster | — |
|
||||
| kube-system/svclb-traefik-0c8aecaf | 1 | 0→0 | 0* | 0.0/0.0 | 0→0 | 1/2 | None | keep | railiance-cluster | — |
|
||||
|
|
@ -89,7 +89,6 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas
|
|||
- `core-hub/rapp-core-hub`: Rule p95x1.5 / peakx0.5 / floor. Production hub.coulomb.social: verify startup and rollout before lowering.
|
||||
- `coulomb/ihp-railiance-probe`: ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision.
|
||||
- `coulomb-social/coulomb-social`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/apps-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/forgejo-db`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/net-kingdom-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `databases/platform-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
|
|
@ -131,7 +130,7 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas
|
|||
- `tenant-engine/tenant-engine`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `user-engine/user-engine`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `user-engine/user-engine-pg`: Rule p95x1.5 / peakx0.5 / floor.
|
||||
- `vergabe-demo-company/vergabe-teilnahme`: 60m provisional, pending target replay (RAPPS-WP-0014-T04).
|
||||
- `vergabe-demo-company/vergabe-teilnahme`: Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi.
|
||||
|
||||
## Missing receiving tasks by owner repo
|
||||
|
||||
|
|
@ -147,11 +146,12 @@ CPU rule: max(floor, p95x1.5, peakx0.5) rounded up to 5m; floors 50m/pod databas
|
|||
- **mail-address-semantics** (1): mas/mas-mx
|
||||
- **net-kingdom** (2): mfa/net-kingdom-sso-mfa, sso/net-kingdom-sso-mfa
|
||||
- **rail-knative** (3): knative-serving/knative-serving, knative-serving/net-kourier-controller, kourier-system/3scale-kourier-gateway
|
||||
- **railiance-apps** (1): vergabe-demo-company/vergabe-teilnahme
|
||||
- **railiance-cluster** (7): cert-manager/cainjector, cert-manager/cert-manager, cert-manager/webhook, kube-system/coredns-7bdb54f89, kube-system/local-path-provisioner, kube-system/metrics-server-786d997795, kube-system/traefik
|
||||
- **railiance-enablement** (2): argocd/argocd-application-controller, argocd/argocd-repo-server
|
||||
- **railiance-forge** (2): forgejo/forgejo-runner, forgejo/gitea
|
||||
- **railiance-platform** (5): external-secrets/external-secrets, external-secrets/external-secrets-cert-controller, external-secrets/external-secrets-webhook, openbao/railiance-platform, playwright/rapp-playwright
|
||||
- **rapp-postgres** (6): cnpg-system/cloudnative-pg, databases/apps-pg, databases/forgejo-db, databases/net-kingdom-pg, databases/platform-pg, databases/platform-pg-2
|
||||
- **rapp-postgres** (5): cnpg-system/cloudnative-pg, databases/forgejo-db, databases/net-kingdom-pg, databases/platform-pg, databases/platform-pg-2
|
||||
- **rapp-qonto** (2): rapp-qonto/rapp-qonto-audit-collector, rapp-qonto-egress/qonto-egress-proxy
|
||||
- **rapp-telemetry** (5): telemetry/alertmanager, telemetry/grafana, telemetry/kube-state-metrics, telemetry/prometheus, telemetry/rapp-telemetry
|
||||
- **rein-aharness** (1): rein-aharness/rein-aharness
|
||||
|
|
|
|||
|
|
@ -21,12 +21,22 @@ OWNER_OVERRIDE = {("databases", "state-hub-db"): "state-hub", ("target-revenue",
|
|||
DB_NS = {"databases"}
|
||||
CRITICAL_NS = {"kube-system", "knative-serving", "kourier-system", "sso", "mfa", "openbao", "external-secrets", "cert-manager", "cnpg-system", "argocd", "state-hub"}
|
||||
SAFETY_P95, SAFETY_PEAK, MIN_DELTA_M, MIN_DELTA_MI = 1.5, 0.5, 15, 64
|
||||
RECEIVING = {("vergabe-demo-company", "vergabe-teilnahme"): "RAPPS-WP-0014-T04 target replay (SLOT, result pending)"}
|
||||
RECEIVING = {}
|
||||
REPLAY = {"source": "railiance-apps 1af6d82 docs/evidence/2026-10-21-vergabe-target-replay/ (RAPPS-WP-0014-T04)",
|
||||
"image": "deployed digest sha256:a26444f5..., 3 gunicorn workers, 60m/256Mi request, 1000m/1Gi limit, target node",
|
||||
"ordinary_ops": 3675, "errors": 0, "ordinary_p95_s": 0.192, "transfer_p95_s": 0.076, "restart_to_ready_s": 18.98,
|
||||
"app_cpu_mean_p95_peak_m": [775, 998, 1020], "app_throttle_ratio": 0.357, "app_memory_peak_mib": 483,
|
||||
"db_incremental_cpu_mean_p95_m": [320, 480], "db_note": "isolated PostgreSQL 16 pod, not shared CNPG apps-pg",
|
||||
"rate": "~25 req/s closed loop, far above two human users",
|
||||
"recommendation": "keep 60m request and 1000m/1Gi limits; raise memory request 256Mi->512Mi next wave; budget 0.3-0.5 core incremental apps-pg demand at synthetic load",
|
||||
"not_established": ["production settings (SSO, ingress/TLS)", "shared apps-pg contention", "observed human demand", "busy-hour node contention (CI, backups)", "human/recovery acceptance (RAPPS-WP-0014-T03)"]}
|
||||
MEM_OVERRIDES = {("vergabe-demo-company", "vergabe-teilnahme"): 512}
|
||||
OVERRIDES = { # (ns, workload): (cpu_m, reason)
|
||||
("coulomb", "ihp-railiance-probe"): (0, "ImagePullBackOff 21d from legacy registry 92.205.130.254; 50m reserved, 0 usage, two more stale pods. Scale to zero/retire only by owner decision."),
|
||||
("vergabe-demo-company", "vergabe-teilnahme"): (60, "60m provisional, pending target replay (RAPPS-WP-0014-T04)."),
|
||||
("vergabe-demo-company", "vergabe-teilnahme"): (60, "Target replay (RAPPS-WP-0014-T04, railiance-apps 1af6d82): keep 60m request and 1000m/1Gi limits; 0 errors, ordinary p95 0.192s under 36% throttling at ~25 req/s; loaded working set 320-480MiB exceeds the 256Mi request -> 512Mi."),
|
||||
("kube-system", "svclb-traefik-0c8aecaf"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."),
|
||||
("kube-system", "svclb-mas-smtp-d86db2b1"): (0, "k3s klipper DaemonSet, ~0m measured; accepted documented zero."),
|
||||
("databases", "apps-pg"): (100, "Keep 100m (rule would give 50m): Vergabe replay adds ~320m mean / ~480m p95 incremental PostgreSQL CPU at synthetic load; shared apps-pg behaviour unproven."),
|
||||
("kube-system", "coredns-7bdb54f89"): (50, "DNS for every workload: floor 50m regardless of p95 7.8m."),
|
||||
("state-hub", "railiance-apps"): (390, "API p95 258m at 250m request = p95x1.5; the 975m peak is release/migration bursts carried by the surge reserve. Preflight models API surge 100m but live API pod requests 250m."),
|
||||
}
|
||||
|
|
@ -65,10 +75,11 @@ for key, w in usage.items():
|
|||
if key in NOTES: why += " " + NOTES[key]
|
||||
action = "keep" if req == cur else ("increase" if req > cur else "decrease")
|
||||
if key == ("coulomb", "ihp-railiance-probe"): action = "retire/scale-to-zero (owner decision)"
|
||||
if key == ("vergabe-demo-company", "vergabe-teilnahme"): action = "keep (provisional)"
|
||||
if key == ("vergabe-demo-company", "vergabe-teilnahme"): action = "keep CPU (replay-backed)"
|
||||
if key in {("sso", "net-kingdom-sso-mfa"), ("playwright", "rapp-playwright")}: action += " + review CPU limit (throttled)"
|
||||
mem_target = max(16, int(math.ceil(w["mem_p95_mi_sum"] * 1.15 / 16) * 16))
|
||||
mem = mem_target if abs(mem_target - w["mem_req_mi"]) >= MIN_DELTA_MI and key not in OVERRIDES else w["mem_req_mi"]
|
||||
mem = MEM_OVERRIDES.get(key, mem)
|
||||
if mem != w["mem_req_mi"]: action += " + memory " + ("up" if mem > w["mem_req_mi"] else "down")
|
||||
owner = OWNER_OVERRIDE.get(key) or (OWNERS.get(ns) or {}).get("owner", "unknown")
|
||||
record = RECEIVING.get(key) or (None if action == "keep" else f"needs receiving task in {owner.split(' ')[0]}")
|
||||
|
|
@ -90,10 +101,19 @@ budget = [
|
|||
("Admitted factory/build execution (one runner job burst above runner request)", 250, 250, "forgejo-runner pod peak 243m; namespace peak 1033m incl. gitea"),
|
||||
("Maintenance CronJobs (7 jobs, 10-25m, overlapping 02:40-04:00)", 50, 50, "kubectl cronjobs; keycape-factor-renewer every 10 min"),
|
||||
("Knative HPA scale-out headroom (requests at max replicas minus 1)", hpa_now, hpa_capped, "activator 20/gateway 10/webhook 5 -> proposed caps 3/3/2"),
|
||||
("Pilot growth reserve (Vergabe replay result + next tenant) — SLOT", 0, 200, "placeholder pending RAPPS-WP-0014-T04; not measured"),
|
||||
("Pilot growth reserve (next tenant / user growth; Vergabe itself is in steady requests)", 0, 200, "Replay: two-user pilot needs no CPU request increase; reserve is unmeasured headroom for growth and human-paced apps-pg demand"),
|
||||
]
|
||||
cur_sum = sum(b[1] for b in budget); prop_sum = sum(b[2] for b in budget)
|
||||
# Serialized: one disruptive event (release surge, build burst or HPA step) at a time.
|
||||
ALLOC_MI = round(16770080768 / 2**20)
|
||||
HOST_MI, SURGE_MI, BUILD_MI, PILOT_MI = 2604, 864, 512, 256
|
||||
def _mem(req, pilot):
|
||||
allc = req + HOST_MI + SURGE_MI + BUILD_MI + pilot
|
||||
ser_ = req + HOST_MI + max(SURGE_MI, BUILD_MI) + pilot
|
||||
return {"requests": round(req), "total_all_concurrent": round(allc), "residual_all_concurrent": ALLOC_MI - round(allc),
|
||||
"total_serialized": round(ser_), "residual_serialized": ALLOC_MI - round(ser_)}
|
||||
mem_budget = {"allocatable": ALLOC_MI, "host_k3s_sample": HOST_MI, "release_surge_state_hub_api": SURGE_MI, "build_runner_burst": BUILD_MI,
|
||||
"pilot_growth_reserve": PILOT_MI, "current": _mem(cur_mem, 0), "proposed": _mem(prop_mem, PILOT_MI)}
|
||||
ser = lambda i: budget[0][i] + budget[1][i] + max(budget[2][i], budget[3][i], budget[5][i]) + budget[4][i] + budget[6][i]
|
||||
out = {"schema": "resource-control.node-allocation-review.v1", "workplan": "RESOURCE-WP-0008", "beneficiary": "CUST-WP-0071-T03",
|
||||
"observed_at": "2026-10-21T06:08:20Z", "allocatable_cpu_m": 4000, "allocatable_memory_mi": round(16770080768 / 2**20),
|
||||
|
|
@ -104,11 +124,10 @@ out = {"schema": "resource-control.node-allocation-review.v1", "workplan": "RESO
|
|||
"budget_total_serialized_events": {"current_m": ser(1), "proposed_m": ser(2), "residual_current_m": 4000 - ser(1), "residual_proposed_m": 4000 - ser(2)},
|
||||
"measured_demand_7d": {"containers_cpu_p95_m": 966, "containers_cpu_peak_m": 1740, "host_non_pod_cpu_m_sample": [677, 871],
|
||||
"note": "Measured demand (containers p95 + host) ~1.8 cores, peaks ~2.6 cores; the deficit is reservation, not demand."},
|
||||
"memory_budget_mi": {"allocatable": round(16770080768 / 2**20), "requests_current": round(cur_mem), "requests_proposed": round(prop_mem),
|
||||
"host_k3s_sample": 2604, "release_surge_state_hub_api": 864, "build_runner_burst": 512, "pilot_growth_slot": 256,
|
||||
"residual_current": round(16770080768 / 2**20 - cur_mem - 2604 - 512 - 512),
|
||||
"residual_proposed": round(16770080768 / 2**20 - prop_mem - 2604 - 864 - 512 - 256)},
|
||||
"vergabe_slot": {"current_request_m": 60, "status": "provisional, pending target replay (RAPPS-WP-0014-T04)", "replay_result": None},
|
||||
"memory_budget_mi": mem_budget,
|
||||
"vergabe_slot": {"current_request_m": 60, "proposed_request_m": 60, "current_memory_request_mi": 256, "proposed_memory_request_mi": 512,
|
||||
"status": "replay-backed keep (CPU), memory request raise in next wave", "replay_result": REPLAY},
|
||||
"pilot_stress_note": "At the synthetic rate the pilot used ~1.1 core (app ~775m + db ~320m) inside limits; that is burst usage, not reservation. During the replay node free CPU p5 fell to ~150m.",
|
||||
"workloads": sorted(rows, key=lambda r: (r["action"] == "keep", r["namespace"], r["workload"]))}
|
||||
(HERE / "2026-10-21-node-budget.json").write_text(json.dumps(out, indent=1) + "\n")
|
||||
for b in out["budget_rows"]: print(f"{b['item'][:70]:70} {b['current_m']:6} {b['proposed_m']:6}")
|
||||
|
|
|
|||
|
|
@ -42,9 +42,8 @@ refreshing stale utilization metrics or inventing financial attribution.
|
|||
|
||||
```task
|
||||
id: RESOURCE-WP-0008-T02
|
||||
status: progress
|
||||
status: done
|
||||
priority: high
|
||||
blocking_reason: "Node budget and per-workload comparison done 2026-10-21; only the Vergabe target replay result (RAPPS-WP-0014-T04) remains to be inserted into the 60m provisional slot."
|
||||
state_hub_task_id: "1f538441-8b45-5d37-aeb3-345204b75117"
|
||||
```
|
||||
|
||||
|
|
@ -66,7 +65,7 @@ whole node budget fits. Do not infer human workflow acceptance.
|
|||
id: RESOURCE-WP-0008-T03
|
||||
status: progress
|
||||
priority: medium
|
||||
blocking_reason: "Proposal and node budget published 2026-10-21, but none of the 54 owner changes has a receiving task in its owner repo (26 repos listed in the review), the coulomb probe owner is unconfirmed, and the Vergabe slot awaits RAPPS-WP-0014-T04."
|
||||
blocking_reason: "Proposal and node budget published 2026-10-21 (Vergabe replay consumed). 54 owner changes in 27 repos still lack receiving tasks: wave 1 (rail-knative, railiance-forge, state-hub, then rapp-telemetry) is being prepared, wave 2 per-repo tasks follow, and the coulomb probe owner must be found first."
|
||||
state_hub_task_id: "3e6e6a59-71a5-5048-b7a5-ed61416a442c"
|
||||
```
|
||||
|
||||
|
|
@ -102,3 +101,26 @@ T03 cannot finish because every non-Vergabe change still needs a receiving task
|
|||
its owner repo. The per-repo list is in `2026-10-21-node-budget.md`. Creating
|
||||
those records is the owners' or the Custodian's decision; this record does not
|
||||
create them.
|
||||
|
||||
## Vergabe replay consumed — 2026-10-21
|
||||
|
||||
Source: railiance-apps `1af6d82` (RAPPS-WP-0014-T04), which replayed the deployed
|
||||
digest on the target node. Results: 0 errors in 3675 ordinary operations, ordinary
|
||||
p95 0.192s, app CPU p95 998m under 36% throttling, memory peak 483MiB, and
|
||||
incremental database CPU of about 320m mean / 480m p95 on an isolated PostgreSQL.
|
||||
|
||||
T02 is done, with this qualified allocation:
|
||||
- Keep the 60m request and the 1000m/1Gi limits.
|
||||
- Raise the memory request from 256Mi to 512Mi in the next wave (receiving owner
|
||||
railiance-apps).
|
||||
- Keep apps-pg at 100m rather than the rule's 50m.
|
||||
|
||||
Regenerated budget, all events concurrent: CPU −390m, memory −347Mi.
|
||||
Regenerated budget, one disruptive event at a time: CPU +90m, memory +165Mi.
|
||||
|
||||
The replay does not establish shared-apps-pg contention, production settings,
|
||||
human demand or human acceptance (RAPPS-WP-0014-T03).
|
||||
|
||||
T03 stays in progress under the founder's wave rollout. Wave 1 is rail-knative,
|
||||
railiance-forge, state-hub and rapp-telemetry; wave 2 is batched per-repo tasks.
|
||||
The ihp-railiance-probe owner must be confirmed first.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue