diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index dee3908..c8a83eb 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -70,6 +70,7 @@ | workplan | CUST-WP-0067 | finished | — | workplans/CUST-WP-0067-hub-authority-target-resolution.md | | workplan | CUST-WP-0068 | finished | — | workplans/CUST-WP-0068-cache-only-work-record-recovery.md | | workplan | CUST-WP-0070 | finished | — | workplans/CUST-WP-0070-publication-repo-category.md | +| workplan | CUST-WP-0071 | ready | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | | workplan | THE-WP-0001 | finished | — | workplans/THE-WP-0001-federation-interface.md | | task | CUST-WP-ADHOC-2026-05-02-T01 | done | — | workplans/ADHOC-2026-05-02.md | | task | CUST-WP-ADHOC-2026-07-02-T01 | done | — | workplans/ADHOC-2026-07-02.md | @@ -449,6 +450,11 @@ | task | CUST-WP-0070-T01 | done | — | workplans/CUST-WP-0070-publication-repo-category.md | | task | CUST-WP-0070-T02 | done | — | workplans/CUST-WP-0070-publication-repo-category.md | | task | CUST-WP-0070-T03 | done | — | workplans/CUST-WP-0070-publication-repo-category.md | +| task | CUST-WP-0071-T01 | todo | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | +| task | CUST-WP-0071-T02 | wait | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | +| task | CUST-WP-0071-T03 | wait | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | +| task | CUST-WP-0071-T04 | wait | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | +| task | CUST-WP-0071-T05 | wait | — | workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md | | task | THE-WP-0001-T01 | done | — | workplans/THE-WP-0001-federation-interface.md | | task | THE-WP-0001-T02 | done | — | workplans/THE-WP-0001-federation-interface.md | | intake | CUST-IN-0001 | closed | green | intake-legacy-suggestions-migration.md | diff --git a/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md b/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md new file mode 100644 index 0000000..1c4bc0b --- /dev/null +++ b/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md @@ -0,0 +1,198 @@ +--- +id: CUST-WP-0071 +type: workplan +title: "Size Railiance workloads from actual demand and establish a weekly allocation review" +domain: infotech +repo: the-custodian +status: ready +owner: the-custodian +topic_slug: custodian +created: "2026-09-11" +updated: "2026-09-11" +related: [STATE-WP-0091, RCLUSTER-WP-0014, RESOURCE-WP-0003, RAPP-TELEMETRY-WP-0001, RAIL-FAB-WP-0028, RAPPS-WP-0014, VERGABE-WP-0019, HFACT-WP-0001] +state_hub_workstream_id: "2249bddb-7524-5add-bd5c-c4163a6ca0f3" +--- + +# Measured sizing and recurring allocation review + +User instruction, 2026-09-11: persist and register this work for later follow-up, +then continue the invited Vergabe pilot at an explicitly accepted 60m CPU +request. This ready workplan is not a prerequisite to deploying that prototype. +It is not an assertion that 60m or the inherited 100m is a measured requirement. + +The objective is an explainable, repeatable allocation process across Railiance +workloads, starting with vergabe-teilnahme. Use existing telemetry, cluster +evidence, resource-control portfolio and workload deployment paths. The +Custodian coordinates; participating repositories retain implementation and +operating authority. Fabric supplies ownership/dependency context when its +accepted endpoint is available; its hosted rollout is not a prerequisite to +collecting measurements or reviewing allocations. + +Starting evidence: + +- `docs/evidence/2026-09-11-railiance-cpu-overview.md` and its JSON snapshot: + 4,000m allocatable, 3,905m requested, 95m available; 20 active pods with zero + CPU request, Knative/Kourier requesting 1,000m, and two stale/unhealthy pods + retaining 100m. These observations must be refreshed before action. +- Grafana/Prometheus is installed privately. Its CPU request recording omits + Forgejo's 100m, and its cluster CPU utilization panel lacks the required + series. Resource-control uses August capacity evidence. A finished collector + implementation does not prove fresh or scheduler-correct consumer data. +- Vergabe's 100m CPU request originated in the May 19 initial chart commit + `962c5a1b3692fb27f003bda2612ffd5ed38d7ffb`; no measured sizing rationale was + found. The user now accepts a 60m request for one tenant with very few users, + prioritizing deployment and onboarding learning over production-grade sizing. + +The accepted prototype keeps its configured 1,000m CPU limit and 256Mi memory +request/1Gi limit unless separately changed. The database, shared services, +build workers and host/control-plane demand remain separate capacity consumers. +Record actual deployed values and image revisions; do not infer them from this +plan. Do not treat reservation headroom as measured spare processing capacity. + +## Reconcile live capacity, demand and ownership evidence + +```task +id: CUST-WP-0071-T01 +status: todo +priority: high +assignee: the-custodian +state_hub_task_id: "4ae8eb8d-f8c5-5256-ae50-c323cfd73d7a" +``` + +With railiance-cluster, rapp-telemetry/railiance-telemetry and resource-control, +produce a timestamped observation joining cluster/node, namespace/workload, +owner/service and tenant where known. Keep unknown allocations explicit. +Count host and cluster representations of the same CPUs only once. + +Reconcile scheduler-effective requests against the Kubernetes API, including +scheduled nonterminal and terminating pods, pending demand, init/sidecar/pod +resources and overhead. Explain differences from dashboard/collector totals; +repair the Forgejo omission and absent host CPU signal in their owning sources. +Account for zero-request containers, host/control-plane demand and data gaps. +Capture CPU usage, requests/limits, throttling and contention, memory working +set/peaks, OOM/restarts, storage exposure and workload activity where available. +Preserve query/collector revisions, units, timestamps and coverage. Reuse +RCLUSTER-WP-0014 and return the release-specific contract to STATE-WP-0091. + +Done when a repeatable observation reconciles current allocation totals, +identifies missing/stale signals without reporting them as zero demand, and +updates the existing portfolio evidence with accountable owner references. + +## Measure the Vergabe prototype under representative pilot use + +```task +id: CUST-WP-0071-T02 +status: wait +priority: high +assignee: the-custodian +depends_on: [CUST-WP-0071-T01] +blocking_reason: "Await reliable measurements and the current invited-pilot deployment or an equivalent isolated fixture." +state_hub_task_id: "82192370-2fd7-5363-88d2-3c67889d3d68" +``` + +VERGABE-WP-0019/RAPPS-WP-0014 supply exact release, tenant/user assumptions, +runtime configuration and deployment. Observe the accepted 60m prototype and +exercise an isolated synthetic-data fixture with representative concurrent +users: login, list/search/detail, tender/lot/task edits and document upload/ +download. Include startup, idle periods, ordinary usage, bursts and restart. +Measure API/UI latency percentiles and errors alongside CPU/memory, throttling, +database demand and request volume. Distinguish container CPU from incremental +database/shared-service demand. No benchmark writes to existing customer data. + +Record workload sizes, concurrency, hardware/image/workers, duration, coverage +and limitations so results are reproducible. Agree response-time/error targets +with the product owner before claiming adequacy. Recommend request/limit and +memory values with a stated margin and revisit trigger; label an incomplete +pilot sample provisional. A successful smoke test alone is not sizing proof. + +## Review ecosystem allocations and reserve operating room + +```task +id: CUST-WP-0071-T03 +status: wait +priority: high +assignee: the-custodian +depends_on: [CUST-WP-0071-T01, CUST-WP-0071-T02] +blocking_reason: "Await reconciled demand evidence and a measured pilot recommendation." +state_hub_task_id: "6dc67558-eb1e-5bb6-a667-986f884dd495" +``` + +Resource-control and the owning workload/cluster repositories compare actual +demand with reservations. Prioritize Knative/Kourier's large reservations, +Forgejo's higher observed demand, zero-request workloads and the stale probe/ +database-drill allocations. Review each service's startup, burst, autoscaling, +latency, availability and storage requirements before proposing changes. + +Publish proposed per-workload requests/limits and a node-level budget including +host operations, pilot growth, deployment surges/migration hooks and admitted +factory/build execution. Declare margin and concurrency assumptions, current +constraints, owners and rollback. If demand plus margin does not fit, compare +capacity options through resource-control using current provider evidence. +STATE-WP-0091 retains its implementation of release preflight; this task consumes +and supplies that contract rather than replacing its workplan. + +Done when the recommended allocation reconciles to admitted capacity and every +remaining change has a live receiving owner/task. This plan does not grant +unrelated deletion, procurement or automatic resource changes. + +## Apply accepted sizing changes and verify their effects + +```task +id: CUST-WP-0071-T04 +status: wait +priority: medium +assignee: the-custodian +depends_on: [CUST-WP-0071-T03] +blocking_reason: "Await reviewed allocation proposals and applicable workload-owner authority." +state_hub_task_id: "944f98e2-7e42-5894-925b-752423f3a1cc" +``` + +Apply accepted changes in the existing owner manifests and deployment paths. +Use staged changes and retain exact prior configuration and rollback receipts. +Compare before/after traffic, latency/errors, resource consumption, throttling, +OOM/restarts and release headroom over a representative observation window. +Verify startup and a subsequent release at the new allocation. Revert or revise +changes that violate the agreed product/service targets. Update resource-control +and task evidence with deployed revisions, measured outcome and residual risks. + +Do not close based only on successful scheduling or smaller requests. Acceptance +requires useful operation and a reconciled budget; unimplemented proposals or +unresolved measurement gaps remain live work records with owners. + +## Establish the regular weekly workload-demand and allocation assessment + +```task +id: CUST-WP-0071-T05 +status: wait +priority: medium +assignee: the-custodian +depends_on: [CUST-WP-0071-T04] +blocking_reason: "Await the repeatable observation and accepted sizing/review procedure." +state_hub_task_id: "20ddd4d7-d85d-5b95-bb0a-7b0ca8af76ca" +``` + +As the final step, install a weekly assessment on an existing durable Railiance +executor with scoped read access. Record the executor owner, schedule and time +zone; proposed cadence is Monday 08:00 Europe/Berlin. Use the existing telemetry +and resource-control surfaces, not a workstation reminder or a parallel +monitoring service. Persist weekly aggregates so comparisons survive the current +seven-day raw-metric retention, and compare the last seven days with prior weeks. + +Each run records per-workload demand and activity, CPU/memory p95 and peaks, +requests/limits, throttling/latency/errors/restarts, pending demand, node and +release margin, coverage/freshness, ownership and week-on-week changes. Produce +explicit keep/increase/decrease/investigate recommendations with evidence and +confidence. Deduplicate open recommendations into live owner work records and +review their results in the following run. Resource allocation adapts through +the accepted owner change procedure; no unrestricted automatic resizing. + +Name the weekly review owner and persist delivery/acknowledgment through an +accepted internal channel. Define and exercise stale/failed/missed-run handling, +retry/idempotency and an accountable recovery route. Preserve scheduler and +assessment configuration in source, with non-secret execution receipts. + +Done when the durable schedule is installed, the first unattended scheduled +assessment has produced its retained report and owner receipt, missed-run +handling is demonstrated, and ownership for the ongoing weekly operation is +registered before this workplan finishes. Hand off residual recommendations as +live records; the recurring assessment continues after this plan is finished.