Record live Railiance CPU reservations and overview gaps

docs/evidence/2026-09-11-railiance-cpu-overview.md

@@ -0,0 +1,89 @@
+# Railiance CPU reservations and ecosystem overview — 2026-09-11
+
+Observed 2026-09-11T15:23:49.132186+00:00; cluster UID `a553c742-0115-43d4-99a4-a5ca56fe0786`. Read-only cluster assessment; no allocations, deployments, credentials, or admission decisions changed.
+
+One primary node: 4,000m capacity and allocatable CPU, 3,905m effective scheduled requests (97.625%), 95m remaining. Limits sum to 29,950m and do not reserve CPU; some workloads have no CPU limit. Memory requests are 8,368Mi against about 15.62GiB. Host and k3s inventory entries refer to the same capacity and must not be added together. The documented frozen CoulombCore cluster was not measured and is not counted as usable spare capacity.
+
+76 active scheduled pods; 21 terminal pods excluded. No unscheduled active pods. Effective requests include regular containers, ordered init/restartable-init behavior, pod-level overrides and overhead; no pod-level or restartable-init CPU cases were present. Status allocated CPU fields were inspected; the total reconciles exactly with `kubectl describe nodes`. Non-atomic API and metrics snapshots are timestamped in the JSON.
+
+Node usage changed from 1,625m to 2,828m during the check (about 41–71%). The retained snapshot has 2,828m node CPU and 1,898m summed pod CPU in different measurement windows. The difference is not an exact measurement of host overhead. Host/control-plane load is outside pod reservations; allocatable CPU shows no explicit deduction from capacity.
+
+| Namespace | CPU request (m) | Share of 4,000m |
+|---|---:|---:|
+| knative-serving | 800 | 20.000% |
+| databases | 600 | 15.000% |
+| telemetry | 245 | 6.125% |
+| core-hub | 200 | 5.000% |
+| kourier-system | 200 | 5.000% |
+| kube-system | 200 | 5.000% |
+| sso | 150 | 3.750% |
+| user-engine | 150 | 3.750% |
+| openbao | 125 | 3.125% |
+| reuse | 110 | 2.750% |
+| state-hub | 110 | 2.750% |
+| activity-core | 100 | 2.500% |
+| cnpg-system | 100 | 2.500% |
+| coulomb-social | 100 | 2.500% |
+| flex-auth | 100 | 2.500% |
+| forgejo | 100 | 2.500% |
+| mfa | 100 | 2.500% |
+| audit-core | 50 | 1.250% |
+| coulomb | 50 | 1.250% |
+| issue-core | 50 | 1.250% |
+| platform-pg-drill | 50 | 1.250% |
+| target-revenue | 50 | 1.250% |
+| canned-prompts | 25 | 0.625% |
+| email-connect | 25 | 0.625% |
+| rein-aharness | 25 | 0.625% |
+| sbom-nexus | 25 | 0.625% |
+| tenant-engine | 25 | 0.625% |
+| rapp-qonto-egress | 20 | 0.500% |
+| informed-decision | 10 | 0.250% |
+| policy-nexus | 10 | 0.250% |
+| cert-manager | 0 | 0.000% |
+| external-secrets | 0 | 0.000% |
+| **Total** | **3905** | **97.625%** |
+
+Twenty active pods have zero effective CPU request; 25 regular containers have zero request. Examples include the Forgejo runner, State Hub database, cert-manager and external-secrets. Their demand and host processes are not represented by the reservation sum. No ResourceQuota/LimitRange objects or VPA API were found; three Knative/Kourier HPAs exist. There is no general quota reservation for the future factory or invited pilot.
+
+## Four-day demand evidence
+
+Prometheus returned 1,152 five-minute evaluation points over four days for every namespace below. Values are five-minute CPU-rate p95 and maximum, not instantaneous peaks or a startup/load-test sizing promise. Full query text and per-namespace coverage are retained in the JSON.
+
+| Namespace | Requested (m) | Four-day p95 (m) | Largest sampled five-minute rate (m) |
+|---|---:|---:|---:|
+| knative-serving | 800 | 7.6 | 8.0 |
+| kourier-system | 200 | 3.8 | 4.0 |
+| databases | 600 | 138.9 | 226.5 |
+| telemetry | 245 | 131.4 | 177.4 |
+| forgejo | 100 | 1014.8 | 1246.9 |
+| state-hub | 110 | 183.2 | 935.5 |
+
+Knative/Kourier reserves 1,000m in six control/gateway pods: activator 300m, net-kourier-controller 200m, gateway 200m, autoscaler/controller/webhook 100m each. Review startup, routing and HPA behavior before changing these requests. Forgejo’s 100m request is substantially below observed demand; lowering all requests would make accounting less representative.
+
+Two owner-review candidates hold 100m combined: `coulomb/ihp-railiance-probe` (50m, ImagePullBackOff against the legacy registry), and `platform-pg-drill/platform-pg-drill-1` (50m, deletion timestamp August 13, still scheduled). Their retirement/termination has not been authorized by this assessment; neither pod nor storage was changed.
+
+## Existing ecosystem surfaces
+
+- **rapp-telemetry / railiance-telemetry:** Live private Grafana, Prometheus and Alertmanager; all five workloads ready and 12 current scrape targets up. Twenty dashboard ConfigMaps include Kubernetes Compute Resources Cluster, Namespace, Workload and Pod. Native operator bootstrap login was proven in the package acceptance record; authenticated public operator access remains RAPP-TELEMETRY-WP-0001-T04. There is no live Grafana ingress for telemetry.coulomb.social. Notifications still have an empty receiver; independent watchdog and recurring backup/operator admission work remain in that existing plan.
+- **Dashboard accuracy gaps:** `sum(namespace_cpu:kube_pod_container_resource_requests:sum)` reports 3.805 CPUs, while the Kubernetes API reports 3.905. Forgejo has two kube_pod_info series but no CPU-request series; its 100m reservation is missing from this metric. The Cluster CPU Utilisation panel uses `cluster:node_cpu:ratio_rate5m`, currently absent. node-exporter is disabled. Other workload CPU series are available. Multi-cluster dashboard provisioning does not establish multi-cluster data coverage.
+- **railiance-cluster:** RCLUSTER-WP-0014 is finished; `make cluster-observe REMOTE=railiance01` produces machine-readable capacity, demand, storage and failure-domain evidence. The latest checked-in observation is August 22. Collector source currently sums regular containers for all returned pods without terminal/scheduling filters or effective init/overhead accounting, so it should not be used as a scheduler admission check without reconciliation.
+- **resource-control:** `make portfolio-report` works and covers seven resource records, ownership, allocation/cost gaps and optimization cases. Its cluster utilization still comes from August 11 (0.564 CPU, labeled idle), and portfolio coverage is August 14. It is a file/CLI portfolio surface, not a verified live allocation console. The host and cluster allocation method and booked host price are still missing in these records.
+- **railiance-fabric:** Provides ownership/dependency graphs and a local graph explorer (`make graph-explorer`, default 127.0.0.1:8765/ui/graph-explorer). It can join workloads, services, tenants and resource ownership. Durable hosted authority and consumer cutover remain proposed RAIL-FAB-WP-0028. This assessment did not start or verify a live Fabric UI and found no basis to claim a unified hosted CPU management console.
+
+## Recommended continuation using existing work
+
+1. STATE-WP-0091: use exact effective requests in the release preflight and hand off shared allocation work to the existing cluster/platform/resource-control owners. Resolve the 100m Prometheus discrepancy and reconcile all overview totals to the same live observation.
+2. Review the two stale/unhealthy 50m reservations with their workload/storage owners, then rightsize the Knative/Kourier control/gateway requests from observed demand plus startup, burst and HPA requirements. Account for Forgejo, zero-request pods and host usage at the same time.
+3. Establish an explicit margin for the 100m vergabe pilot, releases/migration hooks and admitted factory execution. At 95m remaining, the pilot currently fails the CPU request fit by 5m; reclaiming only 5m is not a durable operating margin. If representative demand plus margin does not fit the node, resource-control should compare added capacity with allocation changes.
+4. Finish the existing telemetry operator-access path and connect fresh cluster evidence into resource-control, with Fabric owner/tenant links when available. Add freshness and scheduling-margin signals to the existing dashboards. Preserve workload manifests as the change authority, with reviewed changes executed by the owning Railiance repos.
+
+Source references:
+- `/home/worsch/state-hub/workplans/STATE-WP-0091-release-headroom.md`
+- `/home/worsch/rapp-telemetry/workplans/RAPP-TELEMETRY-WP-0001-foundation.md`
+- `/home/worsch/railiance-cluster/workplans/RCLUSTER-WP-0014-capacity-utilization-and-allocation-evidence.md`
+- `/home/worsch/resource-control/data/resources/reef-railiance-k3s.json`
+- `/home/worsch/railiance-fabric/workplans/RAIL-FAB-WP-0028-hosted-financial-fabric-authority.md`
+- [Kubernetes resource requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+
+Machine-readable live evidence: [2026-09-11-railiance-cpu-overview.json](2026-09-11-railiance-cpu-overview.json).