diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 94d107d..f7c5fa8 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -34,7 +34,7 @@ | task | RAPPS-WP-0014-T01 | done | — | workplans/RAPPS-WP-0014-vergabe-invited-pilot.md | | task | RAPPS-WP-0014-T02 | done | — | workplans/RAPPS-WP-0014-vergabe-invited-pilot.md | | task | RAPPS-WP-0014-T03 | wait | — | workplans/RAPPS-WP-0014-vergabe-invited-pilot.md | -| task | RAPPS-WP-0014-T04 | todo | — | workplans/RAPPS-WP-0014-vergabe-invited-pilot.md | +| task | RAPPS-WP-0014-T04 | done | — | workplans/RAPPS-WP-0014-vergabe-invited-pilot.md | | task | RAPPS-WP-0015-T01 | done | — | workplans/RAPPS-WP-0015-coulomb-personal-content-release.md | | task | RAPPS-WP-0015-T02 | done | — | workplans/RAPPS-WP-0015-coulomb-personal-content-release.md | | task | RAPPS-WP-0015-T03 | done | — | workplans/RAPPS-WP-0015-coulomb-personal-content-release.md | diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/README.md b/docs/evidence/2026-10-21-vergabe-target-replay/README.md new file mode 100644 index 0000000..ef9c237 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/README.md @@ -0,0 +1,155 @@ +# Vergabe target-hardware synthetic replay — 2026-10-21 + +Task: `RAPPS-WP-0014-T04` (feeds `CUST-WP-0071-T02`, `RESOURCE-WP-0008-T02`). +Authority: founder decision of 2026-10-21 +(`the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`): +isolated replay in a throwaway namespace on railiance01, reads elsewhere only. +No customer data was read or written; the live `vergabe-demo-company` deployment, +its database (`apps-pg`), quotas, policies and nodes were not modified (live +Deployment generation 4, same digest, Ready 1/1 before and after). No Secret was +read. This is synthetic sizing evidence, not human/recovery acceptance (T03). + +## Deployed release (read-only capture, 2026-10-21) + +| Item | Value | +| --- | --- | +| Cluster / node | context `default` → 92.205.62.239, single k3s v1.35.1 control-plane node | +| Node CPU / memory | 4 vCPU `AMD EPYC Processor (with IBPB)`, 16 GiB; requests 3605m (90%) / limits 37500m; reservation residual 395m | +| Image | `forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91` (source `c067993`, per `docs/vergabe-demo-company-binding.md`) | +| Chart / rollout | `vergabe-teilnahme-0.2.1`, one replica, Recreate, init `migrate`; Deployment revision 4 (rollout restart 2026-09-25). Helm release revision not re-read (release storage is a Secret) | +| Gunicorn | no command override → image CMD `--workers 3` (sync), default 30s timeout | +| Resources (app and migrate) | requests 60m CPU / 256Mi, limits 1000m CPU / 1Gi | +| Settings | `vergabe_teilnahme.settings.prod`, SSO via NetKingdom, `envFrom` Secret reference `vergabe-demo-env` (not read) | +| Database binding | namespace label `railiance.io/postgres-client=apps-pg`: shared CNPG `databases/apps-pg`, PostgreSQL 16, 100m/256Mi request, 1000m/1Gi limit, WAL archiving + `wal_level=logical` | +| Storage | media 5Gi and app-state 1Gi PVCs, `local-path` | + +Drift vs. the 2026-10-19 local test image (`vergabe-synthetic:local-11f7c8a`, +`sha256:57da46…`): source `11f7c8a` is 16 commits ahead of deployed `c067993`; +9 application files differ (+181/−10: ausschreibungen forms/views/templates, +core views, search partial); Dockerfile, `uv.lock`, `pyproject.toml` identical. +The local runs used 2 workers; the deployed image runs 3. This replay used the +deployed digest itself, so the drift does not affect these results. + +## Replay configuration (pinned) + +Namespace `vergabe-sizing-20261021` (manifests in `harness/bench.yaml`, +`harness/driver.yaml`), all pods on the same node: + +- **app**: deployed digest, image CMD (3 workers), init `migrate`, 60m/256Mi request, + 1000m/1Gi limit, live securityContext/probes, Recreate; media on a fresh + `local-path` PVC, issue state emptyDir. Settings = harness synthetic settings + (`harness/synthetic_settings.py`: base settings, SSO off, password login). +- **db**: fresh `docker.io/library/postgres@sha256:4b7183ac…` (PostgreSQL 16), + emptyDir, 100m/256Mi request, 1000m/1Gi limit (matching apps-pg resources). +- **driver**: deployed digest running `vergabe-teilnahme/scripts/synthetic_pilot.py` + `runtime: remote` (commit `dca6683`; executed copy and config retained, digests + `c8da867f…` / `9045f2c7…` match `raw/result.json.gz`), 100m request/1000m limit. + In-cluster HTTP via ClusterIP Service — chosen over port-forward so latency + excludes the WSL/SSH tunnel. Restart deletes the app pod through a namespaced + Role (pods get/list/delete) and waits for a different ready pod. +- NetworkPolicy: ingress/egress within the namespace, DNS, API server only; + preflight confirmed the live Service was unreachable. +- Profile `config/synthetic-pilot-phases.json` unchanged except runtime fields + (`synthetic-pilot-target-remote.json`): 2 users, 100 seed tenders, 1 MiB + documents, 30s idle, 90s ordinary (0.2s think), 30s burst (0 think), restart, + 60s recovery; 11 named operations including edits. +- Sampling: 1s cgroup v2 `cpu.stat`/`memory.current`/`cpu.pressure` inside app, + db and driver containers plus node-wide `/proc/stat` (`harness/sampler.sh`); + `kubectl top` every 15s; Prometheus (cadvisor) per-namespace CPU/throttling. + The sampler shell itself costs ~11–12m per container and is included. + +Timeline (UTC): namespace created 06:26; pre-run idle baseline 06:27:14–06:29:26; +harness idle 06:29:31–06:30:01; ordinary 06:30:01–06:31:32; burst –06:32:03; +restart –06:32:22 (18.98s to ready incl. migrate); recovery –06:33:23; post +idle –06:34:28; namespace deletion requested 06:35:25, completed 06:36:18; namespace and its PV verified absent (`raw/teardown.txt`). + +## Results + +Driver: complete, `provisional_pass: true`, `measurement_complete: true`, no gaps. + +| Phase | Ordinary req / err / p95 | Transfer req / err / p95 | +| --- | --- | --- | +| ordinary | 1903 / 0 / 0.142s | 422 / 0 / 0.085s | +| burst | 697 / 0 / 0.192s | 154 / 0 / 0.069s | +| recovery | 1075 / 0 / 0.214s | 238 / 0 / 0.073s | +| **all** | **3675 / 0 / 0.192s** | **814 / 0 / 0.076s** | + +Per operation p95 (all phases): list 0.262s, search 0.255s, detail 0.146s, +creates/edits 0.059–0.066s, upload 0.088s, download 0.050s; login 1.27s and +login form 0.62s (n=6, first requests after idle). ~24.7 requests/s closed loop — +far above two human users' pace. + +| Window | App CPU mean / p95 / peak | Throttled periods | App memory peak | DB CPU mean / p95 | Node busy mean (free mean / free p5) | +| --- | --- | --- | --- | --- | --- | +| pre-run baseline | 12m / 15m / 16m | 0% | 166 MiB | 22m / 66m | 1348m (2652m / 1499m) | +| ordinary | 711m / 845m / 1003m | 20.4% | 403 MiB | 388m / 501m | 2979m (1021m / 78m) | +| burst | 953m / 1004m / 1011m | 65.8% | 482 MiB | 430m / 539m | 2806m (1194m / 278m) | +| recovery | 861m / 957m / 982m | 41.5% | 322 MiB | 335m / 399m | 2624m (1376m / 392m) | +| all load phases | 775m / 998m / 1020m | 35.7% (22.8s throttled) | 483 MiB | 341m / 500m | 2790m (1210m / 147m) | +| post | 12m | 0% | 323 MiB | 22m | 1413m (2587m) | + +App memory is cgroup `memory.current` (includes page cache); no OOM, no restarts. +CPU pressure (PSI some) on the app was 15–21% of wall time under load. +Incremental PostgreSQL CPU ≈ 320m mean / ~480m p95 over its idle baseline, on +the isolated DB only (live `apps-pg` stayed at ~8m during the window). Driver +added ~84m mean. + +Competing workloads (Prometheus, workload window): all containers 1622m mean, of +which the replay namespace 915m; others led by databases 121m, state-hub 120m, +telemetry 102m, forgejo 78m (max 385m), activity-core 59m, core-hub 43m; live +vergabe ~0m. Node-wide busy includes ~0.7 core of k3s/system outside containers. +Pre-run competing load left ~2.65 cores free on average; during the replay the +node reached near-saturation at peaks (free p5 ~150m) largely through the replay +itself (app + db + driver ≈ 1.2 cores). Requests during the run were 3865m of +4000m (replay 260m of the 395m residual, released at teardown). + +## Coverage and gaps + +- Representative: exact deployed image/digest, worker count, CPU/memory + requests and limits (CFS quota 100000/100000 verified), Recreate + init migrate + restart path, target hardware and real shared node contention. +- Not representative: synthetic settings instead of prod (no SSO/OIDC, insecure + cookies, no whitenoise manifest storage); no Traefik ingress/TLS/browser; DB is a + standalone PostgreSQL 16 pod, not shared CNPG `apps-pg` (WAL archiving, logical + WAL, other tenants) — the ~320m incremental DB demand would land on apps-pg's + 1000m limit/100m request in production and is not proven there. +- Load is a synthetic closed loop (~25 req/s), not observed human demand; one run + with ~3.5 min of load (pinned profile), ~9 min namespace lifetime. +- Node contention was partly self-induced; contention from other tenants was + light at this hour. A busier moment (forgejo CI, backups) is not covered. +- The 60m request is a scheduling weight/reservation, not a quota; the 1000m + limit is the quota that throttled. Sampler exec overhead is in the app cgroup. + +## Bounded recommendation (for RESOURCE-WP-0008-T02 / CUST-WP-0071-T02) + +- **CPU limit 1000m: keep.** Under an aggressive synthetic mix it throttles + (36% of periods, 66% in burst) yet ordinary p95 stays ≤0.22s per phase — ~9× + margin to the 2s target, zero failed operations, including restart recovery. +- **CPU request 60m: keep for the two-user invited pilot**, provisionally. Even + with the node near saturation and only a 60m weight, latency stayed far inside + target; two humans generate a small fraction of this load. Raising it (e.g. + 250m) is not required by these results and would consume most of the 395m + residual — an allocation-owner decision, not made here. +- **Memory limit 1Gi: keep** (peak 483 MiB incl. page cache, ~2× margin). + **Memory request 256Mi: keep, consider 512Mi** at the next allocation review, + since loaded working set (320–480 MiB) exceeds the request; node memory requests + are 61%, so this is low-risk but not urgent. +- **Database**: budget ~0.3–0.5 core incremental on apps-pg at this synthetic + rate; acceptable against its 1000m limit for a two-user pilot, unverified on the + shared cluster. +- **Revisit when**: user count/companies or corpus grow, image or worker count + changes, live p95 >1s or any 5xx, node requests >95% or sustained node free CPU + <500m outside benchmarks, apps-pg CPU >600m sustained, or the pilot moves to + railiance02/other hardware. + +## Files + +- `analysis.json` — per-window cgroup statistics, per-operation p95, phase results + (`harness/analyze.py raw/ analysis.json` after gunzip) +- `raw/result.json.gz` — driver output (all operations, phase UTC windows, host facts) +- `raw/{app,db,driver}-cgroup.txt.gz` — 1s cgroup/node samples (`POD` lines mark + the restart re-attach; the `exit code 137` exec termination is that restart) +- `raw/kubectl-top.txt.gz`, `raw/prometheus-{baseline,workload}.json.gz` +- `raw/windows.txt`, `raw/pod-provenance.txt`, `raw/namespace-events.txt`, + `raw/teardown.txt`, `raw/driver-stdout.txt` +- `harness/` — manifests, settings, samplers, orchestrator, executed driver/config diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/analysis.json b/docs/evidence/2026-10-21-vergabe-target-replay/analysis.json new file mode 100644 index 0000000..06ef31f --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/analysis.json @@ -0,0 +1,923 @@ +{ + "windows_utc": { + "baseline_pre": [ + "2026-10-21T06:27:14.447540Z", + "2026-10-21T06:29:26.175778Z" + ], + "harness_idle": [ + "2026-10-21T06:29:31.522662Z", + "2026-10-21T06:30:01.522800Z" + ], + "ordinary": [ + "2026-10-21T06:30:01.523056Z", + "2026-10-21T06:31:32.589152Z" + ], + "burst": [ + "2026-10-21T06:31:32.589194Z", + "2026-10-21T06:32:03.275085Z" + ], + "restart": [ + "2026-10-21T06:32:03.275117Z", + "2026-10-21T06:32:22.256585Z" + ], + "recovery": [ + "2026-10-21T06:32:22.256609Z", + "2026-10-21T06:33:23.057045Z" + ], + "workload_all_load_phases": [ + "2026-10-21T06:30:01.523056Z", + "2026-10-21T06:33:23.057045Z" + ], + "post": [ + "2026-10-21T06:33:22.885983Z", + "2026-10-21T06:34:28.753426Z" + ] + }, + "cgroup": { + "app": { + "pods": [ + "vergabe-app-579448bf49-2fsn2", + "vergabe-app-579448bf49-plp4j" + ], + "baseline_pre": { + "samples": 126, + "seconds": 128.0, + "cpu_mean_m": 12.2, + "cpu_p95_m": 14.7, + "cpu_peak_m": 16.2, + "cfs_periods": 317, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 165.6, + "mem_mean_mib": 165.2, + "node_busy_mean_m": 1348, + "node_busy_p95_m": 2501, + "node_free_mean_m": 2652, + "node_free_p5_m": 1499, + "load1_max": 4.91 + }, + "harness_idle": { + "samples": 28, + "seconds": 28.4, + "cpu_mean_m": 11.5, + "cpu_p95_m": 13.4, + "cpu_peak_m": 13.6, + "cfs_periods": 76, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 165.3, + "mem_mean_mib": 165.1, + "node_busy_mean_m": 1538, + "node_busy_p95_m": 2800, + "node_free_mean_m": 2462, + "node_free_p5_m": 1200, + "load1_max": 2.91 + }, + "ordinary": { + "samples": 88, + "seconds": 90.2, + "cpu_mean_m": 711.4, + "cpu_p95_m": 845.4, + "cpu_peak_m": 1003.2, + "cfs_periods": 902, + "cfs_throttled_periods": 184, + "throttle_ratio": 0.204, + "throttled_seconds": 4.85, + "psi_cpu_some_share": 0.15, + "mem_peak_mib": 402.5, + "mem_mean_mib": 295.2, + "node_busy_mean_m": 2979, + "node_busy_p95_m": 3922, + "node_free_mean_m": 1021, + "node_free_p5_m": 78, + "load1_max": 7.83 + }, + "burst": { + "samples": 29, + "seconds": 29.9, + "cpu_mean_m": 952.9, + "cpu_p95_m": 1003.7, + "cpu_peak_m": 1011.1, + "cfs_periods": 298, + "cfs_throttled_periods": 196, + "throttle_ratio": 0.658, + "throttled_seconds": 8.94, + "psi_cpu_some_share": 0.211, + "mem_peak_mib": 481.8, + "mem_mean_mib": 443.4, + "node_busy_mean_m": 2806, + "node_busy_p95_m": 3722, + "node_free_mean_m": 1194, + "node_free_p5_m": 278, + "load1_max": 8.19 + }, + "restart": { + "samples": 10, + "seconds": 10.3, + "cpu_mean_m": 403.3, + "cpu_p95_m": 1019.7, + "cpu_peak_m": 1019.7, + "cfs_periods": 54, + "cfs_throttled_periods": 39, + "throttle_ratio": 0.722, + "throttled_seconds": 2.37, + "psi_cpu_some_share": 0.235, + "mem_peak_mib": 483.3, + "mem_mean_mib": 154.0, + "node_busy_mean_m": 1908, + "node_busy_p95_m": 3359, + "node_free_mean_m": 2092, + "node_free_p5_m": 641, + "load1_max": 7.84 + }, + "recovery": { + "samples": 59, + "seconds": 60.6, + "cpu_mean_m": 860.6, + "cpu_p95_m": 956.8, + "cpu_peak_m": 982.3, + "cfs_periods": 605, + "cfs_throttled_periods": 251, + "throttle_ratio": 0.415, + "throttled_seconds": 6.17, + "psi_cpu_some_share": 0.152, + "mem_peak_mib": 322.2, + "mem_mean_mib": 251.0, + "node_busy_mean_m": 2624, + "node_busy_p95_m": 3608, + "node_free_mean_m": 1376, + "node_free_p5_m": 392, + "load1_max": 6.66 + }, + "workload_all_load_phases": { + "samples": 189, + "seconds": 194.0, + "cpu_mean_m": 775.3, + "cpu_p95_m": 998.3, + "cpu_peak_m": 1019.7, + "cfs_periods": 1889, + "cfs_throttled_periods": 675, + "throttle_ratio": 0.357, + "throttled_seconds": 22.77, + "psi_cpu_some_share": 0.166, + "mem_peak_mib": 483.3, + "mem_mean_mib": 295.7, + "node_busy_mean_m": 2790, + "node_busy_p95_m": 3853, + "node_free_mean_m": 1210, + "node_free_p5_m": 147, + "load1_max": 8.19 + }, + "post": { + "samples": 63, + "seconds": 63.9, + "cpu_mean_m": 12.3, + "cpu_p95_m": 14.3, + "cpu_peak_m": 15.1, + "cfs_periods": 161, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.001, + "mem_peak_mib": 322.5, + "mem_mean_mib": 322.2, + "node_busy_mean_m": 1413, + "node_busy_p95_m": 2414, + "node_free_mean_m": 2587, + "node_free_p5_m": 1586, + "load1_max": 5.75 + } + }, + "db": { + "pods": [ + "vergabe-db-7c4bffdd48-w4jz4" + ], + "baseline_pre": { + "samples": 127, + "seconds": 128.5, + "cpu_mean_m": 21.5, + "cpu_p95_m": 65.7, + "cpu_peak_m": 71.7, + "cfs_periods": 333, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 91.3, + "mem_mean_mib": 90.4, + "node_busy_mean_m": 1346, + "node_busy_p95_m": 2448, + "node_free_mean_m": 2654, + "node_free_p5_m": 1552, + "load1_max": 4.91 + }, + "harness_idle": { + "samples": 29, + "seconds": 29.3, + "cpu_mean_m": 20.9, + "cpu_p95_m": 64.5, + "cpu_peak_m": 64.9, + "cfs_periods": 75, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.003, + "mem_peak_mib": 93.0, + "mem_mean_mib": 92.7, + "node_busy_mean_m": 1522, + "node_busy_p95_m": 2680, + "node_free_mean_m": 2478, + "node_free_p5_m": 1320, + "load1_max": 3.15 + }, + "ordinary": { + "samples": 88, + "seconds": 89.9, + "cpu_mean_m": 387.8, + "cpu_p95_m": 501.1, + "cpu_peak_m": 532.2, + "cfs_periods": 892, + "cfs_throttled_periods": 2, + "throttle_ratio": 0.002, + "throttled_seconds": 0.02, + "psi_cpu_some_share": 0.067, + "mem_peak_mib": 101.4, + "mem_mean_mib": 95.7, + "node_busy_mean_m": 2974, + "node_busy_p95_m": 3895, + "node_free_mean_m": 1026, + "node_free_p5_m": 105, + "load1_max": 8.19 + }, + "burst": { + "samples": 29, + "seconds": 29.6, + "cpu_mean_m": 430.1, + "cpu_p95_m": 539.1, + "cpu_peak_m": 544.4, + "cfs_periods": 293, + "cfs_throttled_periods": 2, + "throttle_ratio": 0.007, + "throttled_seconds": 0.06, + "psi_cpu_some_share": 0.062, + "mem_peak_mib": 113.1, + "mem_mean_mib": 97.1, + "node_busy_mean_m": 2798, + "node_busy_p95_m": 3664, + "node_free_mean_m": 1202, + "node_free_p5_m": 336, + "load1_max": 8.19 + }, + "restart": { + "samples": 18, + "seconds": 18.3, + "cpu_mean_m": 24.3, + "cpu_p95_m": 67.7, + "cpu_peak_m": 67.7, + "cfs_periods": 55, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.009, + "mem_peak_mib": 103.3, + "mem_mean_mib": 101.4, + "node_busy_mean_m": 2650, + "node_busy_p95_m": 3961, + "node_free_mean_m": 1350, + "node_free_p5_m": 39, + "load1_max": 7.84 + }, + "recovery": { + "samples": 58, + "seconds": 58.9, + "cpu_mean_m": 335.0, + "cpu_p95_m": 398.7, + "cpu_peak_m": 408.1, + "cfs_periods": 581, + "cfs_throttled_periods": 1, + "throttle_ratio": 0.002, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.041, + "mem_peak_mib": 111.3, + "mem_mean_mib": 106.9, + "node_busy_mean_m": 2621, + "node_busy_p95_m": 3524, + "node_free_mean_m": 1379, + "node_free_p5_m": 476, + "load1_max": 6.66 + }, + "workload_all_load_phases": { + "samples": 196, + "seconds": 199.8, + "cpu_mean_m": 340.8, + "cpu_p95_m": 500.0, + "cpu_peak_m": 544.4, + "cfs_periods": 1837, + "cfs_throttled_periods": 5, + "throttle_ratio": 0.003, + "throttled_seconds": 0.08, + "psi_cpu_some_share": 0.052, + "mem_peak_mib": 113.1, + "mem_mean_mib": 99.8, + "node_busy_mean_m": 2813, + "node_busy_p95_m": 3894, + "node_free_mean_m": 1187, + "node_free_p5_m": 106, + "load1_max": 8.19 + }, + "post": { + "samples": 64, + "seconds": 64.8, + "cpu_mean_m": 21.7, + "cpu_p95_m": 65.1, + "cpu_peak_m": 79.6, + "cfs_periods": 161, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 101.3, + "mem_mean_mib": 98.1, + "node_busy_mean_m": 1420, + "node_busy_p95_m": 2509, + "node_free_mean_m": 2580, + "node_free_p5_m": 1491, + "load1_max": 5.75 + } + }, + "driver": { + "pods": [ + "synthetic-driver" + ], + "baseline_pre": { + "samples": 126, + "seconds": 127.9, + "cpu_mean_m": 11.6, + "cpu_p95_m": 13.2, + "cpu_peak_m": 13.7, + "cfs_periods": 267, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 6.6, + "mem_mean_mib": 6.3, + "node_busy_mean_m": 1348, + "node_busy_p95_m": 2510, + "node_free_mean_m": 2652, + "node_free_p5_m": 1490, + "load1_max": 4.91 + }, + "harness_idle": { + "samples": 28, + "seconds": 28.4, + "cpu_mean_m": 11.6, + "cpu_p95_m": 13.1, + "cpu_peak_m": 13.1, + "cfs_periods": 59, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 63.3, + "mem_mean_mib": 63.0, + "node_busy_mean_m": 1537, + "node_busy_p95_m": 2829, + "node_free_mean_m": 2463, + "node_free_p5_m": 1171, + "load1_max": 2.91 + }, + "ordinary": { + "samples": 88, + "seconds": 90.0, + "cpu_mean_m": 91.7, + "cpu_p95_m": 117.2, + "cpu_peak_m": 124.8, + "cfs_periods": 891, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.029, + "mem_peak_mib": 75.9, + "mem_mean_mib": 72.4, + "node_busy_mean_m": 2977, + "node_busy_p95_m": 3912, + "node_free_mean_m": 1023, + "node_free_p5_m": 88, + "load1_max": 7.83 + }, + "burst": { + "samples": 29, + "seconds": 29.6, + "cpu_mean_m": 98.5, + "cpu_p95_m": 124.8, + "cpu_peak_m": 129.3, + "cfs_periods": 289, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.026, + "mem_peak_mib": 78.4, + "mem_mean_mib": 76.4, + "node_busy_mean_m": 2790, + "node_busy_p95_m": 3387, + "node_free_mean_m": 1210, + "node_free_p5_m": 613, + "load1_max": 8.19 + }, + "restart": { + "samples": 18, + "seconds": 18.3, + "cpu_mean_m": 22.4, + "cpu_p95_m": 25.5, + "cpu_peak_m": 25.5, + "cfs_periods": 112, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.009, + "mem_peak_mib": 73.7, + "mem_mean_mib": 71.3, + "node_busy_mean_m": 2650, + "node_busy_p95_m": 3990, + "node_free_mean_m": 1350, + "node_free_p5_m": 10, + "load1_max": 7.84 + }, + "recovery": { + "samples": 59, + "seconds": 60.1, + "cpu_mean_m": 82.2, + "cpu_p95_m": 95.4, + "cpu_peak_m": 139.7, + "cfs_periods": 594, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.02, + "mem_peak_mib": 80.9, + "mem_mean_mib": 77.2, + "node_busy_mean_m": 2635, + "node_busy_p95_m": 3540, + "node_free_mean_m": 1365, + "node_free_p5_m": 460, + "load1_max": 6.66 + }, + "workload_all_load_phases": { + "samples": 197, + "seconds": 201.1, + "cpu_mean_m": 84.4, + "cpu_p95_m": 117.7, + "cpu_peak_m": 200.8, + "cfs_periods": 1911, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.024, + "mem_peak_mib": 80.9, + "mem_mean_mib": 74.4, + "node_busy_mean_m": 2820, + "node_busy_p95_m": 3912, + "node_free_mean_m": 1180, + "node_free_p5_m": 88, + "load1_max": 8.19 + }, + "post": { + "samples": 64, + "seconds": 64.9, + "cpu_mean_m": 16.3, + "cpu_p95_m": 13.2, + "cpu_peak_m": 317.3, + "cfs_periods": 137, + "cfs_throttled_periods": 0, + "throttle_ratio": 0.0, + "throttled_seconds": 0.0, + "psi_cpu_some_share": 0.002, + "mem_peak_mib": 79.1, + "mem_mean_mib": 19.2, + "node_busy_mean_m": 1423, + "node_busy_p95_m": 2535, + "node_free_mean_m": 2577, + "node_free_p5_m": 1465, + "load1_max": 5.75 + } + } + }, + "per_operation": { + "burst/detail": { + "n": 77, + "errors": 0, + "p95_s": 0.1132, + "max_s": 0.1706 + }, + "burst/document-download": { + "n": 77, + "errors": 0, + "p95_s": 0.0447, + "max_s": 0.0592 + }, + "burst/document-upload": { + "n": 77, + "errors": 0, + "p95_s": 0.075, + "max_s": 0.1055 + }, + "burst/list": { + "n": 77, + "errors": 0, + "p95_s": 0.2974, + "max_s": 0.3233 + }, + "burst/login": { + "n": 2, + "errors": 0, + "p95_s": 1.1775, + "max_s": 1.1775 + }, + "burst/login-form": { + "n": 2, + "errors": 0, + "p95_s": 0.0281, + "max_s": 0.0281 + }, + "burst/lot-create": { + "n": 77, + "errors": 0, + "p95_s": 0.0581, + "max_s": 0.0971 + }, + "burst/lot-edit": { + "n": 77, + "errors": 0, + "p95_s": 0.0536, + "max_s": 0.0743 + }, + "burst/search": { + "n": 77, + "errors": 0, + "p95_s": 0.2652, + "max_s": 0.3359 + }, + "burst/task-create": { + "n": 77, + "errors": 0, + "p95_s": 0.0599, + "max_s": 0.1778 + }, + "burst/task-edit": { + "n": 77, + "errors": 0, + "p95_s": 0.0597, + "max_s": 0.0759 + }, + "burst/tender-create": { + "n": 77, + "errors": 0, + "p95_s": 0.0772, + "max_s": 0.0845 + }, + "burst/tender-edit": { + "n": 77, + "errors": 0, + "p95_s": 0.0623, + "max_s": 0.0732 + }, + "ordinary/detail": { + "n": 211, + "errors": 0, + "p95_s": 0.1758, + "max_s": 0.3877 + }, + "ordinary/document-download": { + "n": 211, + "errors": 0, + "p95_s": 0.0537, + "max_s": 0.11 + }, + "ordinary/document-upload": { + "n": 211, + "errors": 0, + "p95_s": 0.1051, + "max_s": 0.4418 + }, + "ordinary/list": { + "n": 211, + "errors": 0, + "p95_s": 0.194, + "max_s": 0.3779 + }, + "ordinary/login": { + "n": 2, + "errors": 0, + "p95_s": 1.2574, + "max_s": 1.2574 + }, + "ordinary/login-form": { + "n": 2, + "errors": 0, + "p95_s": 0.3288, + "max_s": 0.3288 + }, + "ordinary/lot-create": { + "n": 211, + "errors": 0, + "p95_s": 0.0696, + "max_s": 0.1603 + }, + "ordinary/lot-edit": { + "n": 211, + "errors": 0, + "p95_s": 0.0763, + "max_s": 0.1017 + }, + "ordinary/search": { + "n": 211, + "errors": 0, + "p95_s": 0.2294, + "max_s": 0.6217 + }, + "ordinary/task-create": { + "n": 211, + "errors": 0, + "p95_s": 0.0697, + "max_s": 0.1338 + }, + "ordinary/task-edit": { + "n": 211, + "errors": 0, + "p95_s": 0.0686, + "max_s": 0.1146 + }, + "ordinary/tender-create": { + "n": 211, + "errors": 0, + "p95_s": 0.0626, + "max_s": 0.1187 + }, + "ordinary/tender-edit": { + "n": 211, + "errors": 0, + "p95_s": 0.0633, + "max_s": 0.1171 + }, + "recovery/detail": { + "n": 119, + "errors": 0, + "p95_s": 0.131, + "max_s": 0.1379 + }, + "recovery/document-download": { + "n": 119, + "errors": 0, + "p95_s": 0.0475, + "max_s": 0.0818 + }, + "recovery/document-upload": { + "n": 119, + "errors": 0, + "p95_s": 0.08, + "max_s": 0.1587 + }, + "recovery/list": { + "n": 119, + "errors": 0, + "p95_s": 0.3302, + "max_s": 0.3757 + }, + "recovery/login": { + "n": 2, + "errors": 0, + "p95_s": 1.2653, + "max_s": 1.2653 + }, + "recovery/login-form": { + "n": 2, + "errors": 0, + "p95_s": 0.6157, + "max_s": 0.6157 + }, + "recovery/lot-create": { + "n": 119, + "errors": 0, + "p95_s": 0.0526, + "max_s": 0.0945 + }, + "recovery/lot-edit": { + "n": 119, + "errors": 0, + "p95_s": 0.0602, + "max_s": 0.1186 + }, + "recovery/search": { + "n": 119, + "errors": 0, + "p95_s": 0.3193, + "max_s": 0.4237 + }, + "recovery/task-create": { + "n": 119, + "errors": 0, + "p95_s": 0.0558, + "max_s": 0.0888 + }, + "recovery/task-edit": { + "n": 119, + "errors": 0, + "p95_s": 0.0577, + "max_s": 0.0959 + }, + "recovery/tender-create": { + "n": 119, + "errors": 0, + "p95_s": 0.0489, + "max_s": 0.0661 + }, + "recovery/tender-edit": { + "n": 119, + "errors": 0, + "p95_s": 0.0556, + "max_s": 0.0753 + } + }, + "per_operation_all_phases": { + "detail": { + "n": 407, + "errors": 0, + "p95_s": 0.1463 + }, + "document-download": { + "n": 407, + "errors": 0, + "p95_s": 0.0498 + }, + "document-upload": { + "n": 407, + "errors": 0, + "p95_s": 0.0878 + }, + "list": { + "n": 407, + "errors": 0, + "p95_s": 0.2624 + }, + "login": { + "n": 6, + "errors": 0, + "p95_s": 1.2653 + }, + "login-form": { + "n": 6, + "errors": 0, + "p95_s": 0.6157 + }, + "lot-create": { + "n": 407, + "errors": 0, + "p95_s": 0.0613 + }, + "lot-edit": { + "n": 407, + "errors": 0, + "p95_s": 0.0621 + }, + "search": { + "n": 407, + "errors": 0, + "p95_s": 0.2553 + }, + "task-create": { + "n": 407, + "errors": 0, + "p95_s": 0.0621 + }, + "task-edit": { + "n": 407, + "errors": 0, + "p95_s": 0.0657 + }, + "tender-create": { + "n": 407, + "errors": 0, + "p95_s": 0.0661 + }, + "tender-edit": { + "n": 407, + "errors": 0, + "p95_s": 0.0587 + } + }, + "summary": { + "ordinary": { + "requests": 3675, + "errors": 0, + "p95_seconds": 0.19182809395715594 + }, + "document_transfer": { + "requests": 814, + "errors": 0, + "p95_seconds": 0.07621732295956463 + }, + "complete": true, + "provisional_pass": true, + "measurement_complete": true, + "measurement_gaps": [], + "startup_seconds": 0.04661138809751719, + "duration_seconds": 201.53947580303065, + "completed_iterations": { + "ordinary": [ + 106, + 105 + ], + "burst": [ + 39, + 38 + ], + "recovery": [ + 59, + 60 + ] + }, + "host": { + "logical_cpus": 4, + "affinity_cpus": 4, + "cpu_model": "AMD EPYC Processor (with IBPB)", + "memory_total": "MemTotal: 16377032 kB", + "cgroup_cpu_max": "100000 100000", + "app_cpu_quota": 1.0 + } + }, + "phase_results": [ + { + "name": "ordinary", + "kind": "load", + "seconds": 91.06608834001236, + "started_at": "2026-10-21T06:30:01.523056+00:00", + "finished_at": "2026-10-21T06:31:32.589152+00:00", + "expected_requests": 2325, + "actual_requests": 2325, + "complete": true, + "ordinary": { + "requests": 1903, + "errors": 0, + "p95_seconds": 0.14162223995663226 + }, + "document_transfer": { + "requests": 422, + "errors": 0, + "p95_seconds": 0.08520571992266923 + } + }, + { + "name": "burst", + "kind": "load", + "seconds": 30.685886312043294, + "started_at": "2026-10-21T06:31:32.589194+00:00", + "finished_at": "2026-10-21T06:32:03.275085+00:00", + "expected_requests": 851, + "actual_requests": 851, + "complete": true, + "ordinary": { + "requests": 697, + "errors": 0, + "p95_seconds": 0.19206618797034025 + }, + "document_transfer": { + "requests": 154, + "errors": 0, + "p95_seconds": 0.06853019201662391 + } + }, + { + "name": "restart", + "kind": "restart", + "seconds": 18.981462653027847, + "started_at": "2026-10-21T06:32:03.275117+00:00", + "finished_at": "2026-10-21T06:32:22.256585+00:00", + "ready": true + }, + { + "name": "recovery", + "kind": "load", + "seconds": 60.80042844603304, + "started_at": "2026-10-21T06:32:22.256609+00:00", + "finished_at": "2026-10-21T06:33:23.057045+00:00", + "expected_requests": 1313, + "actual_requests": 1313, + "complete": true, + "ordinary": { + "requests": 1075, + "errors": 0, + "p95_seconds": 0.21410848898813128 + }, + "document_transfer": { + "requests": 238, + "errors": 0, + "p95_seconds": 0.07255190098658204 + } + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/analyze.py b/docs/evidence/2026-10-21-vergabe-target-replay/harness/analyze.py new file mode 100644 index 0000000..32435ba --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/analyze.py @@ -0,0 +1,73 @@ +import json, math, sys, gzip, collections +from datetime import datetime +R = sys.argv[1] +def ts(x): return datetime.fromisoformat(x.replace("Z", "+00:00").replace(".", ".", 1)[:26] + ("+00:00" if "+" not in x[19:] and not x.endswith("Z") else "")).timestamp() if False else _ts(x) +def _ts(x): + x = x.rstrip("Z") + if "+" in x[19:]: x = x[:x.index("+", 19)] + if "." in x: a, b = x.split("."); x = a + "." + b[:6] + return datetime.fromisoformat(x + "+00:00").timestamp() +def p95(v): + v = sorted(v); return v[max(0, math.ceil(0.95 * len(v)) - 1)] if v else None +def parse(path): + segs, cur, s = [], None, None + for line in open(path): + line = line.rstrip("\n") + if line.startswith("POD "): cur = {"pod": line.split()[1], "samples": []}; segs.append(cur) + elif line.startswith("S "): + s = {"t": float(line[2:])}; cur["samples"].append(s) + elif s is None: continue + elif line.startswith("M "): s["mem"] = int(line[2:]) + elif line.startswith("P "): s["psi_some_total"] = int(line.split("total=")[1]) + elif line.startswith("N "): + f = [int(x) for x in line.split()[2:]]; s["node_total"] = sum(f[:8]); s["node_idle"] = f[3] + f[4] + elif line.startswith("L "): s["load1"] = float(line.split()[1]) + elif line.split() and line.split()[0] in ("usage_usec","nr_periods","nr_throttled","throttled_usec"): + k, v = line.split(); s[k] = int(v) + for seg in segs: seg["samples"] = [x for x in seg["samples"] if "mem" in x and "node_total" in x and "usage_usec" in x] + return segs +def window(segs, a, b, cores=4): + rates, mems, thr_p, per_p, thr_us, nodebusy, psi, load = [], [], 0, 0, 0, [], 0, [] + span = 0; used = 0 + for seg in segs: + xs = [x for x in seg["samples"] if a <= x["t"] <= b] + for x in xs: mems.append(x["mem"]); load.append(x["load1"]) + for p, q in zip(xs, xs[1:]): + dt = q["t"] - p["t"] + if dt <= 0 or dt > 5: continue + du = q["usage_usec"] - p["usage_usec"]; used += du; span += dt + rates.append(du / 1e3 / dt) + per_p += q.get("nr_periods",0) - p.get("nr_periods",0); thr_p += q.get("nr_throttled",0) - p.get("nr_throttled",0) + thr_us += q.get("throttled_usec",0) - p.get("throttled_usec",0) + psi += q["psi_some_total"] - p["psi_some_total"] + tot = q["node_total"] - p["node_total"]; idle = q["node_idle"] - p["node_idle"] + if tot > 0: nodebusy.append(cores * (tot - idle) / tot * 1000) + if not rates: return None + return {"samples": len(rates), "seconds": round(span,1), "cpu_mean_m": round(used/1e3/span,1), "cpu_p95_m": round(p95(rates),1), "cpu_peak_m": round(max(rates),1), + "cfs_periods": per_p, "cfs_throttled_periods": thr_p, "throttle_ratio": round(thr_p/per_p,3) if per_p else None, "throttled_seconds": round(thr_us/1e6,2), + "psi_cpu_some_share": round(psi/1e6/span,3), "mem_peak_mib": round(max(mems)/2**20,1), "mem_mean_mib": round(sum(mems)/len(mems)/2**20,1), + "node_busy_mean_m": round(sum(nodebusy)/len(nodebusy)), "node_busy_p95_m": round(p95(nodebusy)), "node_free_mean_m": round(cores*1000-sum(nodebusy)/len(nodebusy)), + "node_free_p5_m": round(cores*1000-p95(nodebusy)), "load1_max": max(load)} +win = {} +for line in open(f"{R}/windows.txt"): + k, *v = line.split(); win[k] = v +res = json.load(open(f"{R}/result.json")) +W = {"baseline_pre": (_ts(win["baseline_start"][0]), _ts(win["driver_start"][0])), + "harness_idle": (_ts(res["idle_window"]["started_at"]), _ts(res["idle_window"]["finished_at"]))} +for p in res["phase_results"]: W[p["name"]] = (_ts(p["started_at"]), _ts(p["finished_at"])) +loads = [p for p in res["phase_results"] if p["kind"] == "load"] +W["workload_all_load_phases"] = (_ts(loads[0]["started_at"]), _ts(loads[-1]["finished_at"])) +W["post"] = (_ts(win["driver_exit"][1]), _ts(win["post_end"][0])) +out = {"windows_utc": {k: [datetime.utcfromtimestamp(a).isoformat()+"Z", datetime.utcfromtimestamp(b).isoformat()+"Z"] for k,(a,b) in W.items()}, "cgroup": {}} +for name in ("app", "db", "driver"): + segs = parse(f"{R}/{name}-cgroup.txt") + out["cgroup"][name] = {"pods": [s["pod"] for s in segs], **{k: window(segs, a, b) for k, (a, b) in W.items() if k != "workload_all_load_phases" or True}} +ops = collections.defaultdict(list) +for r in res["operations"]: ops[(r["phase"], r["operation"])].append(r) +out["per_operation"] = {f"{ph}/{op}": {"n": len(v), "errors": sum(not x["ok"] for x in v), "p95_s": round(p95([x["seconds"] for x in v]),4), "max_s": round(max(x["seconds"] for x in v),4)} for (ph, op), v in sorted(ops.items())} +allops = collections.defaultdict(list) +for r in res["operations"]: allops[r["operation"]].append(r) +out["per_operation_all_phases"] = {op: {"n": len(v), "errors": sum(not x["ok"] for x in v), "p95_s": round(p95([x["seconds"] for x in v]),4)} for op, v in sorted(allops.items())} +out["summary"] = {k: res[k] for k in ("ordinary","document_transfer","complete","provisional_pass","measurement_complete","measurement_gaps","startup_seconds","duration_seconds","completed_iterations","host")} +out["phase_results"] = [{k: v for k, v in p.items() if k != "operations"} for p in res["phase_results"]] +json.dump(out, open(sys.argv[2], "w"), indent=1) diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/bench.yaml b/docs/evidence/2026-10-21-vergabe-target-replay/harness/bench.yaml new file mode 100644 index 0000000..573740b --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/bench.yaml @@ -0,0 +1,125 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: vergabe-sizing-20261021 + labels: + railiance.io/purpose: synthetic-sizing + railiance.io/task: rapps-wp-0014-t04 + railiance.io/ephemeral: "true" +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: {name: bench-media, namespace: vergabe-sizing-20261021} +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: {requests: {storage: 5Gi}} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: vergabe-db, namespace: vergabe-sizing-20261021} +spec: + replicas: 1 + strategy: {type: Recreate} + selector: {matchLabels: {app: vergabe-sizing-db}} + template: + metadata: {labels: {app: vergabe-sizing-db}} + spec: + automountServiceAccountToken: false + containers: + - name: postgres + image: docker.io/library/postgres@sha256:4b7183ac05f8ef417db21fd72d71047a4238340c261d3cc3ddb6d579ab5071ae + imagePullPolicy: IfNotPresent + env: + - {name: POSTGRES_DB, value: synthetic} + - {name: POSTGRES_USER, value: synthetic} + - {name: POSTGRES_PASSWORD, value: synthetic-local-only} + - {name: PGDATA, value: /var/lib/postgresql/data/pgdata} + ports: [{containerPort: 5432}] + readinessProbe: {exec: {command: [pg_isready, -U, synthetic]}, periodSeconds: 5} + resources: {requests: {cpu: 100m, memory: 256Mi}, limits: {cpu: "1", memory: 1Gi}} + volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}] + volumes: [{name: data, emptyDir: {}}] +--- +apiVersion: v1 +kind: Service +metadata: {name: vergabe-db, namespace: vergabe-sizing-20261021} +spec: {selector: {app: vergabe-sizing-db}, ports: [{port: 5432, targetPort: 5432}]} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: vergabe-app, namespace: vergabe-sizing-20261021} +spec: + replicas: 1 + strategy: {type: Recreate} + selector: {matchLabels: {app: vergabe-sizing-app}} + template: + metadata: {labels: {app: vergabe-sizing-app}} + spec: + automountServiceAccountToken: false + securityContext: {fsGroup: 999, runAsGroup: 999, runAsNonRoot: true, runAsUser: 999} + initContainers: + - name: migrate + image: forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 + imagePullPolicy: IfNotPresent + command: [python, manage.py, migrate, --noinput] + env: &env + - {name: DJANGO_SETTINGS_MODULE, value: synthetic_settings} + - {name: PYTHONPATH, value: /synthetic} + - {name: SECRET_KEY, value: synthetic-disposable-not-a-deployment-key} + - {name: DATABASE_URL, value: "postgres://synthetic:synthetic-local-only@vergabe-db:5432/synthetic"} + resources: &res {requests: {cpu: 60m, memory: 256Mi}, limits: {cpu: "1", memory: 1Gi}} + securityContext: &sc {allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}, readOnlyRootFilesystem: false} + volumeMounts: [{name: settings, mountPath: /synthetic}] + containers: + - name: app + image: forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 + imagePullPolicy: IfNotPresent + env: *env + ports: [{containerPort: 8000, name: http}] + readinessProbe: {httpGet: {path: /health/, port: 8000, httpHeaders: [{name: Host, value: vergabe-app}]}, initialDelaySeconds: 5, periodSeconds: 10, timeoutSeconds: 5, failureThreshold: 3} + livenessProbe: {httpGet: {path: /health/, port: 8000, httpHeaders: [{name: Host, value: vergabe-app}]}, initialDelaySeconds: 30, periodSeconds: 30, timeoutSeconds: 5, failureThreshold: 3} + resources: *res + securityContext: *sc + volumeMounts: + - {name: settings, mountPath: /synthetic} + - {name: media, mountPath: /app/media} + - {name: app-state, mountPath: /app/.issue-facade} + volumes: + - {name: settings, configMap: {name: synthetic-settings}} + - {name: media, persistentVolumeClaim: {claimName: bench-media}} + - {name: app-state, emptyDir: {}} +--- +apiVersion: v1 +kind: Service +metadata: {name: vergabe-app, namespace: vergabe-sizing-20261021} +spec: {selector: {app: vergabe-sizing-app}, ports: [{port: 80, targetPort: 8000}]} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: {name: synthetic-driver, namespace: vergabe-sizing-20261021} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: {name: synthetic-driver-restart, namespace: vergabe-sizing-20261021} +rules: [{apiGroups: [""], resources: [pods], verbs: [get, list, delete]}] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: {name: synthetic-driver-restart, namespace: vergabe-sizing-20261021} +roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: synthetic-driver-restart} +subjects: [{kind: ServiceAccount, name: synthetic-driver, namespace: vergabe-sizing-20261021}] +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: {name: synthetic-isolation, namespace: vergabe-sizing-20261021} +spec: + podSelector: {} + policyTypes: [Ingress, Egress] + ingress: [{from: [{podSelector: {}}]}] + egress: + - to: [{podSelector: {}}] + - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}] + ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}] + - to: [{ipBlock: {cidr: 92.205.62.239/32}}, {ipBlock: {cidr: 10.43.0.1/32}}] + ports: [{port: 6443, protocol: TCP}, {port: 443, protocol: TCP}] diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver-config.json b/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver-config.json new file mode 100644 index 0000000..25241c5 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver-config.json @@ -0,0 +1,53 @@ +{ + "users": 2, + "workers": 3, + "iterations": 100000, + "tenders": 100, + "document_bytes": 1048576, + "think_seconds": 0.2, + "timeout_seconds": 30, + "ordinary_p95_seconds": 2.0, + "duration_seconds": 180, + "idle_seconds": 30, + "runtime": "remote", + "app_image": "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91", + "app_cpu_limit": 1.0, + "app_memory_limit": "1Gi", + "operations": [ + "list", + "search", + "tender-create", + "detail", + "tender-edit", + "lot-create", + "lot-edit", + "task-create", + "task-edit", + "document-upload", + "document-download" + ], + "phases": [ + { + "name": "ordinary", + "kind": "load", + "duration_seconds": 90, + "think_seconds": 0.2 + }, + { + "name": "burst", + "kind": "load", + "duration_seconds": 30, + "think_seconds": 0 + }, + { + "name": "restart", + "kind": "restart" + }, + { + "name": "recovery", + "kind": "load", + "duration_seconds": 60, + "think_seconds": 0.2 + } + ] +} diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver.yaml b/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver.yaml new file mode 100644 index 0000000..db2991a --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/driver.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Pod +metadata: {name: synthetic-driver, namespace: vergabe-sizing-20261021, labels: {app: vergabe-sizing-driver}} +spec: + serviceAccountName: synthetic-driver + restartPolicy: Never + securityContext: {runAsGroup: 999, runAsNonRoot: true, runAsUser: 999} + containers: + - name: driver + image: forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 + imagePullPolicy: IfNotPresent + workingDir: /app + command: [sleep, "7200"] + env: + - {name: PYTHONPATH, value: /app} + - {name: SYNTHETIC_BASE_URL, value: "http://vergabe-app"} + - {name: SYNTHETIC_DATABASE_URL, value: "postgres://synthetic:synthetic-local-only@vergabe-db:5432/synthetic"} + - {name: SYNTHETIC_RESTART_SELECTOR, value: "app=vergabe-sizing-app"} + - {name: SYNTHETIC_SOURCE_REVISION, value: "deployed image c067993037f1cdeeb7b22d5600b51c649bc239db; driver from vergabe-teilnahme working tree"} + - {name: SYNTHETIC_DATABASE_DESCRIPTION, value: "fresh PostgreSQL 16 pod in throwaway namespace (docker.io/library/postgres:16 pinned digest)"} + - {name: SYNTHETIC_TRANSPORT_DESCRIPTION, value: "in-cluster HTTP via ClusterIP Service to Gunicorn (image CMD, 3 workers)"} + resources: {requests: {cpu: 100m, memory: 256Mi}, limits: {cpu: "1", memory: 1Gi}} + securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}} + volumeMounts: + - {name: driver, mountPath: /driver} + - {name: out, mountPath: /out} + volumes: + - {name: driver, configMap: {name: synthetic-driver}} + - {name: out, emptyDir: {}} diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/executed-synthetic_pilot.py b/docs/evidence/2026-10-21-vergabe-target-replay/harness/executed-synthetic_pilot.py new file mode 100644 index 0000000..0ef5b91 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/executed-synthetic_pilot.py @@ -0,0 +1,1091 @@ +#!/usr/bin/env python3 +"""Provisional HTTP workload against Gunicorn and a disposable PostgreSQL. + +Runtimes: ``host`` and ``container`` start everything locally. ``remote`` drives +an already running isolated synthetic deployment (for example a throwaway +Kubernetes namespace): the base URL, database URL and optional restart pod +selector come from ``SYNTHETIC_BASE_URL``, ``SYNTHETIC_DATABASE_URL`` and +``SYNTHETIC_RESTART_SELECTOR``. Remote resource sampling is external; phase UTC +windows are retained so samples can be aligned afterwards. +""" + +import argparse +import concurrent.futures +import hashlib +import http.cookiejar +import json +import math +import os +import platform +import signal +import socket +import subprocess +import sys +import tempfile +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +import uuid +from datetime import UTC, datetime +from pathlib import Path + + +def summarize(rows): + durations = sorted(row["seconds"] for row in rows) + return { + "requests": len(rows), + "errors": sum(not r["ok"] for r in rows), + "p95_seconds": durations[math.ceil(len(durations) * 0.95) - 1] if durations else None, + } + + +LEGACY_OPERATIONS = [ + "list", + "search", + "tender-create", + "detail", + "lot-create", + "task-create", + "document-upload", + "document-download", +] +DEPENDENCIES = { + "list": None, + "search": None, + "tender-create": None, + "detail": "tender-create", + "tender-edit": "tender-create", + "lot-create": "tender-create", + "lot-edit": "lot-create", + "task-create": "tender-create", + "task-edit": "task-create", + "document-upload": "tender-create", + "document-download": "document-upload", +} + + +def operation_plan(operations): + if not isinstance(operations, list) or not operations: + raise ValueError("operations must be a nonempty ordered list") + seen = set() + for operation in operations: + if operation not in DEPENDENCIES: + raise ValueError(f"unknown operation: {operation}") + dependency = DEPENDENCIES[operation] + if dependency and dependency not in seen: + raise ValueError(f"{operation} requires earlier {dependency}") + if operation in seen and (operation.endswith("-create") or operation == "document-upload"): + raise ValueError(f"only one {operation} per cycle; repeat reads/edits to tune mix") + seen.add(operation) + return list(operations) + + +def phase_plan(config): + phases = config.get( + "phases", + [ + { + "name": "ordinary", + "kind": "load", + "duration_seconds": config["duration_seconds"], + "think_seconds": config["think_seconds"], + } + ], + ) + if not isinstance(phases, list) or not phases: + raise ValueError("phases must be a nonempty list") + names = set() + result = [] + for item in phases: + item = dict(item) + name = item.get("name") + if not isinstance(name, str) or not name or name in names or name in ("idle", "startup"): + raise ValueError("phase names must be unique and not idle/startup") + names.add(name) + if item.get("kind") == "restart": + if config.get("runtime") not in ("container", "remote"): + raise ValueError("restart phase requires container or remote runtime") + elif item.get("kind") == "load": + for key in ("duration_seconds", "think_seconds"): + value = item.get(key) + if type(value) not in (int, float) or not math.isfinite(value) or value < 0: + raise ValueError(f"invalid phase {key}") + if not item["duration_seconds"]: + raise ValueError("load duration must be positive") + item["operations"] = operation_plan( + item.get("operations", config.get("operations", LEGACY_OPERATIONS)) + ) + else: + raise ValueError("phase kind must be load or restart") + result.append(item) + if not any(p["kind"] == "load" for p in result): + raise ValueError("at least one load phase required") + return result + + +def expected_requests(completed, operations, users): + return sum(completed) * len(operations) + users * 2 + + +def wait_ready(base_url, timeout=30): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(base_url + "/health/", timeout=1) as response: + if response.status == 200: + return + except OSError: + pass + time.sleep(0.1) + raise RuntimeError("application readiness timeout") + + +def container_url(name): + binding = subprocess.check_output( + ["docker", "port", name, "8000/tcp"], text=True, timeout=10 + ).strip() + host, port = binding.rsplit(":", 1) + if host != "127.0.0.1" or not port.isdigit(): + raise ValueError("synthetic runtime must have one loopback HTTP binding") + return f"http://127.0.0.1:{port}" + + +KUBE_SA = Path("/var/run/secrets/kubernetes.io/serviceaccount") + + +def kube_request(method, path, sa_dir=KUBE_SA): + """Minimal in-cluster API call using the pod's own namespaced ServiceAccount.""" + host = os.environ["KUBERNETES_SERVICE_HOST"] + port = os.environ.get("KUBERNETES_SERVICE_PORT", "443") + import ssl + + context = ssl.create_default_context(cafile=str(sa_dir / "ca.crt")) + req = urllib.request.Request( + f"https://{host}:{port}{path}", + method=method, + headers={"Authorization": "Bearer " + (sa_dir / "token").read_text().strip()}, + ) + with urllib.request.urlopen(req, timeout=10, context=context) as response: + return json.loads(response.read()) + + +def ready_pods(listing): + return [ + item["metadata"]["name"] + for item in listing.get("items", []) + if not item["metadata"].get("deletionTimestamp") + and any( + c["type"] == "Ready" and c["status"] == "True" + for c in item.get("status", {}).get("conditions", []) + ) + ] + + +def restart_kubernetes_pod(base_url, selector, timeout=180, sa_dir=KUBE_SA, request=None): + """Delete the single selected pod in this namespace and wait for a new ready one.""" + request = request or (lambda method, path: kube_request(method, path, sa_dir)) + namespace = (sa_dir / "namespace").read_text().strip() + query = urllib.parse.urlencode({"labelSelector": selector}) + pods = f"/api/v1/namespaces/{namespace}/pods" + before = ready_pods(request("GET", f"{pods}?{query}")) + if len(before) != 1: + raise RuntimeError("restart requires exactly one ready selected pod") + request("DELETE", f"{pods}/{before[0]}") + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + listing = request("GET", f"{pods}?{query}") + names = [i["metadata"]["name"] for i in listing.get("items", [])] + if before[0] not in names and ready_pods(listing): + wait_ready(base_url, timeout=max(1, deadline - time.monotonic())) + return base_url + time.sleep(0.5) + raise RuntimeError("replacement pod readiness timeout") + + +def settings_text(media, issue_db, allowed_hosts=("127.0.0.1",)): + return ( + "from vergabe_teilnahme.settings.base import *\n" + f"DEBUG = False\nALLOWED_HOSTS = {list(allowed_hosts)!r}\nNETKINGDOM_ENABLED = False\n" + f"MEDIA_ROOT = {media!r}\n" + f"ISSUE_FACADE_LOCAL_DB = {issue_db!r}\n" + 'ISSUE_FACADE_GITEA = None\nFORCE_SCRIPT_NAME = None\nMEDIA_URL = "/media/"\n' + 'EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"\n' + "SESSION_COOKIE_SECURE = False\nCSRF_COOKIE_SECURE = False\n" + ) + + +def restart_container(name): + subprocess.run( + ["docker", "restart", "--time", "5", name], + check=True, + timeout=20, + stdout=subprocess.DEVNULL, + ) + # Docker can allocate a different random published port after restart. + endpoint = container_url(name) + wait_ready(endpoint) + return endpoint + + +def validate(config): + for key in ( + "users", + "iterations", + "tenders", + "document_bytes", + "workers", + "timeout_seconds", + ): + if type(config[key]) is not int or config[key] < 1: + raise ValueError(f"{key} must be a positive integer") + for key in ( + "think_seconds", + "ordinary_p95_seconds", + "duration_seconds", + "idle_seconds", + ): + if not isinstance(config[key], (int, float)) or not math.isfinite(config[key]): + raise ValueError(f"{key} must be finite") + if config["think_seconds"] < 0 or config["ordinary_p95_seconds"] <= 0: + raise ValueError("invalid timing parameter") + if config["duration_seconds"] <= 0 or config["idle_seconds"] < 0: + raise ValueError("duration must be positive and idle nonnegative") + if config["document_bytes"] < 8: + raise ValueError("document_bytes must be at least 8") + if config.get("runtime", "host") not in ("host", "container", "remote"): + raise ValueError("unsupported runtime") + if config.get("runtime") == "container": + if ( + not isinstance(config["app_cpu_limit"], (int, float)) + or not 0 < config["app_cpu_limit"] <= 8 + ): + raise ValueError("app_cpu_limit must be positive and at most 8") + if not config.get("app_image") or not config.get("app_memory_limit"): + raise ValueError("container image and memory limit required") + phase_plan(config) + return config + + +def stop_process(server): + """Bounded teardown of a process group, including Gunicorn workers.""" + if server is None: + return + try: + os.killpg(server.pid, signal.SIGTERM) + except ProcessLookupError: + pass + try: + server.wait(timeout=5) + except subprocess.TimeoutExpired: + try: + os.killpg(server.pid, signal.SIGKILL) + except ProcessLookupError: + pass + server.wait(timeout=5) + + +def guarded_sampler(function, errors): + try: + function() + except Exception as exc: + errors.append({"sampler": function.__name__, "error_type": type(exc).__name__}) + + +def sampling_gaps(samples, db_samples, errors, remote=False): + gaps = list(errors) + if remote: + return gaps + if any(s.get("sample_error") for s in samples): + gaps.append({"sampler": "application", "reason": "sample failed"}) + good = [s for s in samples if "monotonic_seconds" in s] + if len(good) < 2 or not db_samples: + gaps.append({"sampler": "coverage", "reason": "insufficient samples"}) + if any( + b["monotonic_seconds"] - a["monotonic_seconds"] > 3 + for a, b in zip(good, good[1:]) + if a.get("phase") == b.get("phase") + ): + gaps.append({"sampler": "application", "reason": "sampling interval exceeds 3 seconds"}) + return gaps + + +def run(config, root, failure_output=None): + # Never consume a deployment database, media tree, session or credential. + os.environ["DATABASE_URL"] = os.environ["SYNTHETIC_DATABASE_URL"] + os.environ["SECRET_KEY"] = "synthetic-disposable-not-a-deployment-key" + os.environ["DJANGO_SETTINGS_MODULE"] = "synthetic_settings" + container_mode = config.get("runtime", "host") == "container" + remote_mode = config.get("runtime") == "remote" + media = "/tmp/synthetic-media" if container_mode else str(root / "media") + issue_db = "/tmp/synthetic-issues.db" if container_mode else str(root / "issues.db") + (root / "synthetic_settings.py").write_text(settings_text(media, issue_db)) + sys.path.insert(0, str(root)) + import django + + django.setup() + from django.core.management import call_command + from django.db import connections + + from vergabe_teilnahme.apps.accounts.models import Mitarbeiter + from vergabe_teilnahme.apps.aufgaben.models import Aufgabe + from vergabe_teilnahme.apps.ausschreibungen.models import Ausschreibung + from vergabe_teilnahme.apps.dokumente.models import Dokument + from vergabe_teilnahme.apps.lose.models import Los + + call_command("migrate", verbosity=0) + for i in range(config["tenders"]): + Ausschreibung.objects.create(titel=f"Synthetic tender {i}", ausschreiber="Synthetic") + users = [] + password = uuid.uuid4().hex + for i in range(config["users"]): + user = Mitarbeiter.objects.create_user(username=f"synthetic-{i}", password=password) + users.append(user.username) + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + port = sock.getsockname()[1] + base_url = f"http://127.0.0.1:{port}" + env = dict(os.environ, PYTHONPATH=str(root) + os.pathsep + str(Path.cwd())) + server_log = (root / "server.log").open("w") + server = None + app_container = None + sampling_done = threading.Event() + sampler_threads = [] + sampler_errors = [] + sampling_lock = threading.Lock() + try: + startup_started = time.monotonic() + if remote_mode: + base_url = os.environ["SYNTHETIC_BASE_URL"].rstrip("/") + elif container_mode: + app_container = os.environ["SYNTHETIC_CONTAINER"] + "-app" + root.chmod(0o755) + subprocess.run( + [ + "docker", + "run", + "--detach", + "--pull=never", + "--name", + app_container, + "--network", + os.environ["SYNTHETIC_NETWORK"], + "--publish", + "127.0.0.1::8000", + "--cpus", + str(config["app_cpu_limit"]), + "--memory", + config["app_memory_limit"], + "--pids-limit", + "256", + "--mount", + ( + f"type=bind,src={root / 'synthetic_settings.py'}," + "dst=/synthetic_settings.py,readonly" + ), + "--env", + "PYTHONPATH=/", + "--env", + "DJANGO_SETTINGS_MODULE=synthetic_settings", + "--env", + "SECRET_KEY=synthetic-disposable-not-a-deployment-key", + "--env", + "DATABASE_URL=postgres://synthetic:synthetic-local-only@db:5432/synthetic", + config["app_image"], + "gunicorn", + "vergabe_teilnahme.wsgi:application", + "--bind", + "0.0.0.0:8000", + "--workers", + str(config["workers"]), + "--timeout", + "30", + ], + check=True, + timeout=30, + stdout=subprocess.DEVNULL, + ) + base_url = container_url(app_container) + else: + server = subprocess.Popen( + [ + sys.executable, + "-m", + "gunicorn", + "vergabe_teilnahme.wsgi:application", + "--bind", + f"127.0.0.1:{port}", + "--workers", + str(config["workers"]), + "--timeout", + "30", + ], + start_new_session=True, + env=env, + stdout=server_log, + stderr=server_log, + ) + samples = [] + db_samples = [] + phase = {"name": "startup", "kind": "startup"} + + def sample(): + ticks_per_second = os.sysconf("SC_CLK_TCK") + while not sampling_done.is_set(): + if phase.get("kind") == "restart": + samples.append( + { + "at": datetime.now(UTC).isoformat(), + "phase": phase["name"], + "expected_unavailable": True, + "reason": "planned restart", + } + ) + sampling_done.wait(0.5) + continue + if container_mode: + with sampling_lock: + value = subprocess.check_output( + [ + "docker", + "exec", + app_container, + "sh", + "-c", + ( + "cat /sys/fs/cgroup/cpu.stat; echo MEMORY; " + "cat /sys/fs/cgroup/memory.current" + ), + ], + text=True, + timeout=5, + ) + cpu_text, memory_text = value.split("MEMORY\n") + counters = { + k: int(v) for k, v in (line.split() for line in cpu_text.splitlines()) + } + samples.append( + { + "at": datetime.now(UTC).isoformat(), + "monotonic_seconds": time.monotonic(), + "phase": phase["name"], + "cpu_seconds": counters["usage_usec"] / 1e6, + "rss_bytes": int(memory_text.strip()), + "memory_kind": "cgroup-memory-current", + "cpu_stat": counters, + "pids": [app_container], + } + ) + sampling_done.wait(0.5) + continue + pids = [server.pid] + try: + pids += [ + int(x) + for x in Path(f"/proc/{server.pid}/task/{server.pid}/children") + .read_text() + .split() + ] + cpu, rss = 0, 0 + for pid in pids: + fields = Path(f"/proc/{pid}/stat").read_text().split(") ", 1)[1].split() + cpu += (int(fields[11]) + int(fields[12])) / ticks_per_second + rss += int(fields[21]) * os.sysconf("SC_PAGE_SIZE") + samples.append( + { + "at": datetime.now(UTC).isoformat(), + "monotonic_seconds": time.monotonic(), + "phase": phase["name"], + "cpu_seconds": cpu, + "rss_bytes": rss, + "pids": pids, + } + ) + except (OSError, ValueError): + samples.append({"at": datetime.now(UTC).isoformat(), "sample_error": True}) + sampling_done.wait(0.5) + + def sample_db(): + while not sampling_done.is_set(): + value = subprocess.check_output( + [ + "docker", + "stats", + "--no-stream", + "--format", + "{{json .}}", + os.environ["SYNTHETIC_CONTAINER"], + ], + text=True, + timeout=5, + ) + db_samples.append( + { + "at": datetime.now(UTC).isoformat(), + "phase": phase["name"], + "docker_stats": json.loads(value), + } + ) + sampling_done.wait(1) + + sampler = threading.Thread( + target=guarded_sampler, args=(sample, sampler_errors), daemon=True + ) + db_sampler = threading.Thread( + target=guarded_sampler, args=(sample_db, sampler_errors), daemon=True + ) + if not remote_mode: + sampler_threads.extend([sampler, db_sampler]) + sampler.start() + db_sampler.start() + wait_ready(base_url) + startup_seconds = time.monotonic() - startup_started + phase.update(name="idle", kind="idle") + idle_window = {"name": "idle", "kind": "idle", "started_at": datetime.now(UTC).isoformat()} + time.sleep(config["idle_seconds"]) + idle_window["finished_at"] = datetime.now(UTC).isoformat() + plan = phase_plan(config) + active_plan = [] + phase_results = [] + transport_rebinds = [] + phase_completed = {} + + def worker(i): + jar = http.cookiejar.CookieJar() + + class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + + opener = urllib.request.build_opener( + urllib.request.HTTPCookieProcessor(jar), NoRedirect() + ) + rows = [] + + def request(operation, method, path, data=None, expected=200, transfer=False): + started = time.perf_counter() + try: + headers = {} + body = None + if method == "post": + token = next((c.value for c in jar if c.name == "csrftoken"), "") + fields = dict(data or {}, csrfmiddlewaretoken=token) + if transfer: + boundary = "synthetic-" + uuid.uuid4().hex + body = b"" + for name, value in fields.items(): + body += ( + f"--{boundary}\r\nContent-Disposition: form-data; " + f'name="{name}"\r\n\r\n{value}\r\n' + ).encode() + body += ( + ( + f"--{boundary}\r\nContent-Disposition: form-data; " + 'name="datei"; filename="synthetic.pdf"\r\n' + "Content-Type: application/pdf\r\n\r\n" + ).encode() + + b"%PDF-1.4" + + b"x" * (config["document_bytes"] - 8) + + f"\r\n--{boundary}--\r\n".encode() + ) + headers["Content-Type"] = f"multipart/form-data; boundary={boundary}" + else: + body = urllib.parse.urlencode(fields).encode() + headers["Referer"] = base_url + path + req = urllib.request.Request( + base_url + path, data=body, headers=headers, method=method.upper() + ) + try: + response = opener.open(req, timeout=config["timeout_seconds"]) + except urllib.error.HTTPError as exc: + response = exc + received = len(response.read()) + status = response.status + ok = status == expected + if transfer and method == "get": + ok = ok and received == config["document_bytes"] + response.close() + except Exception: + status, ok, received = None, False, 0 + rows.append( + { + "phase": phase["name"], + "user": i, + "operation": operation, + "transfer": transfer, + "seconds": time.perf_counter() - started, + "status": status, + "ok": ok, + "response_bytes": received, + } + ) + return ok + + request("login-form", "get", "/accounts/login/") + barrier.wait(timeout=30) + request( + "login", + "post", + "/accounts/login/", + {"username": users[i], "password": password}, + 302, + ) + for n in range(config["iterations"]): + if time.monotonic() >= deadline: + break + title = f"Workload-{phase['name']}-{i}-{n}" + tender = lot = task = document = None + base = None + for index, operation in enumerate(active_plan): + dependency = DEPENDENCIES[operation] + entity = { + "tender-create": tender, + "lot-create": lot, + "task-create": task, + "document-upload": document, + } + if dependency and entity[dependency] is None: + rows.append( + { + "phase": phase["name"], + "user": i, + "operation": operation, + "transfer": operation.startswith("document-"), + "seconds": 0, + "status": None, + "ok": False, + "response_bytes": 0, + "reason": "dependency operation failed", + } + ) + continue + if operation == "list": + request(operation, "get", "/ausschreibungen/") + elif operation == "search": + request(operation, "get", "/ausschreibungen/?q=Synthetic") + elif operation == "tender-create": + if request( + operation, + "post", + "/ausschreibungen/neu/", + {"titel": title, "ausschreiber": "Synthetic"}, + 302, + ): + tender = Ausschreibung.objects.filter(titel=title).first() + base = f"/ausschreibungen/{tender.pk}/" if tender else None + elif operation == "detail": + request(operation, "get", base) + elif operation == "tender-edit": + changed = f"{title} edited-{index}" + request( + operation, + "post", + base + "bearbeiten/", + {"titel": changed, "ausschreiber": "Synthetic edited"}, + 302, + ) + tender.refresh_from_db() + rows[-1]["ok"] &= tender.titel == changed + elif operation == "lot-create": + request( + operation, + "post", + base + "lose/neu/", + {"losnummer": "1", "lostitel": "Synthetic lot"}, + 302, + ) + lot = Los.objects.filter(ausschreibung=tender).first() + elif operation == "lot-edit": + changed = f"Edited lot-{index}" + request( + operation, + "post", + base + f"lose/{lot.pk}/bearbeiten/", + {"losnummer": "1", "lostitel": changed}, + 302, + ) + lot.refresh_from_db() + rows[-1]["ok"] &= lot.lostitel == changed + elif operation == "task-create": + request( + operation, + "post", + base + "aufgaben/neu/", + {"titel": "Synthetic task", "typ": "fachlich", "prioritaet": 2}, + 302, + ) + task = Aufgabe.objects.filter(ausschreibung=tender).first() + elif operation == "task-edit": + changed = f"Edited task-{index}" + request( + operation, + "post", + base + f"aufgaben/{task.pk}/bearbeiten/", + {"titel": changed, "typ": "fachlich", "prioritaet": 1}, + 302, + ) + task.refresh_from_db() + rows[-1]["ok"] &= task.titel == changed and task.prioritaet == 1 + elif operation == "document-upload": + request( + operation, + "post", + base + "dokumente/hochladen/", + {"kategorie": "intern", "version": "1.0"}, + 302, + True, + ) + document = Dokument.objects.filter(ausschreibung=tender).first() + elif operation == "document-download": + request(operation, "get", document.datei.url, transfer=True) + completed[i] += 1 + time.sleep(active_phase["think_seconds"]) + connections.close_all() + return rows + + started = datetime.now(UTC).isoformat() + before = time.perf_counter() + rows = [] + for active_phase in plan: + phase.update(name=active_phase["name"], kind=active_phase["kind"]) + phase_started = time.monotonic() + phase_started_at = datetime.now(UTC).isoformat() + if active_phase["kind"] == "restart": + old_url = base_url + with sampling_lock: + if remote_mode: + base_url = restart_kubernetes_pod( + base_url, os.environ["SYNTHETIC_RESTART_SELECTOR"] + ) + else: + base_url = restart_container(app_container) + transport_rebinds.append( + {"phase": active_phase["name"], "before": old_url, "after": base_url} + ) + phase_results.append( + { + "name": active_phase["name"], + "kind": "restart", + "seconds": time.monotonic() - phase_started, + "started_at": phase_started_at, + "finished_at": datetime.now(UTC).isoformat(), + "ready": True, + } + ) + continue + active_plan = active_phase["operations"] + deadline = time.monotonic() + active_phase["duration_seconds"] + completed = [0] * config["users"] + barrier = threading.Barrier(config["users"]) + with concurrent.futures.ThreadPoolExecutor(max_workers=config["users"]) as pool: + phase_rows = [ + r for group in pool.map(worker, range(config["users"])) for r in group + ] + phase_completed[active_phase["name"]] = completed + phase_ordinary = summarize([r for r in phase_rows if not r["transfer"]]) + phase_transfers = summarize([r for r in phase_rows if r["transfer"]]) + expected = expected_requests(completed, active_plan, config["users"]) + phase_results.append( + { + "name": active_phase["name"], + "kind": "load", + "seconds": time.monotonic() - phase_started, + "started_at": phase_started_at, + "finished_at": datetime.now(UTC).isoformat(), + "operations": active_plan, + "expected_requests": expected, + "actual_requests": len(phase_rows), + "complete": all(completed) and len(phase_rows) == expected, + "ordinary": phase_ordinary, + "document_transfer": phase_transfers, + } + ) + rows.extend(phase_rows) + ordinary = summarize([r for r in rows if not r["transfer"]]) + transfers = summarize([r for r in rows if r["transfer"]]) + complete = all(p.get("complete", p.get("ready", False)) for p in phase_results) + result = { + "schema_version": 2, + "startup_seconds": startup_seconds, + "idle_window": idle_window, + "transport_rebinds": transport_rebinds, + "phase_results": phase_results, + "started_at": started, + "finished_at": datetime.now(UTC).isoformat(), + "duration_seconds": time.perf_counter() - before, + "config": config, + "environment": { + "python": platform.python_version(), + "django": django.get_version(), + "platform": platform.platform(), + "database": os.environ.get( + "SYNTHETIC_DATABASE_DESCRIPTION", "disposable PostgreSQL 16 container" + ), + "transport": os.environ.get( + "SYNTHETIC_TRANSPORT_DESCRIPTION", "loopback HTTP Gunicorn" + ), + "workers": config["workers"], + }, + "ordinary": ordinary, + "document_transfer": transfers, + "complete": complete, + "provisional_pass": complete + and ordinary["errors"] == 0 + and transfers["errors"] == 0 + and all( + p["ordinary"]["p95_seconds"] <= config["ordinary_p95_seconds"] + for p in phase_results + if p["kind"] == "load" + ), + "limitations": [ + "No browser, ingress, TLS or SSO", + "No CPU/memory allocation recommendation or human acceptance", + "No deployed-release hardware or shared-service contention acceptance", + ], + "operations": rows, + "runtime_samples": samples, + "database_samples": db_samples, + "completed_iterations": phase_completed, + } + sampling_done.set() + for thread in sampler_threads: + thread.join(timeout=7) + gaps = sampling_gaps(samples, db_samples, sampler_errors, remote=remote_mode) + if remote_mode: + result["resource_sampling"] = "external; align with phase UTC windows" + if any(thread.is_alive() for thread in sampler_threads): + gaps.append({"sampler": "cleanup", "reason": "thread did not stop"}) + result["measurement_gaps"] = gaps + result["measurement_complete"] = not gaps + if container_mode: + result["app_container"] = json.loads( + subprocess.check_output( + ["docker", "inspect", "--format", "{{json .}}", app_container], + text=True, + timeout=10, + ) + ) + # Retain only public runtime identifiers/limits, never full container environment. + info = result.pop("app_container") + result["app_runtime"] = { + "image_id": info["Image"], + "nano_cpus": info["HostConfig"]["NanoCpus"], + "memory_bytes": info["HostConfig"]["Memory"], + "memory_kind": "cgroup-memory-current includes page cache", + "oom_killed": info["State"]["OOMKilled"], + "dockerfile_sha256": hashlib.sha256(Path("Dockerfile").read_bytes()).hexdigest(), + } + return result + except Exception as exc: + if failure_output is not None: + failure = { + "schema_version": 2, + "status": "failed", + "complete": False, + "provisional_pass": False, + "measurement_complete": False, + "error_type": type(exc).__name__, + "phase": locals().get("phase", {"name": "startup"}), + "config": config, + "phase_results": locals().get("phase_results", []), + "operations": locals().get("rows", []), + "runtime_samples": locals().get("samples", []), + "database_samples": locals().get("db_samples", []), + "measurement_gaps": [{"reason": "runtime failure; partial evidence only"}], + "driver_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + } + failure_output.write_text(json.dumps(failure, indent=2) + "\n") + raise + finally: + sampling_done.set() + for thread in sampler_threads: + if thread.ident is not None: + thread.join(timeout=7) + stop_process(server) + if app_container is not None: + subprocess.run( + ["docker", "rm", "--force", "--volumes", app_container], + timeout=15, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + server_log.close() + connections.close_all() + + +def host_facts(config): + return { + "logical_cpus": os.cpu_count(), + "affinity_cpus": len(os.sched_getaffinity(0)), + "cpu_model": next( + ( + x.split(":", 1)[1].strip() + for x in Path("/proc/cpuinfo").read_text().splitlines() + if x.startswith("model name") + ), + "unknown", + ), + "memory_total": next( + x for x in Path("/proc/meminfo").read_text().splitlines() if x.startswith("MemTotal:") + ), + "cgroup_cpu_max": Path("/sys/fs/cgroup/cpu.max").read_text().strip() + if Path("/sys/fs/cgroup/cpu.max").exists() + else "unavailable", + "app_cpu_quota": config.get("app_cpu_limit", "not imposed by harness"), + } + + +def main_remote(config, args): + """Drive an existing isolated synthetic deployment; never a shared database.""" + for name in ("SYNTHETIC_BASE_URL", "SYNTHETIC_DATABASE_URL"): + if not os.environ.get(name): + raise SystemExit(f"{name} is required for remote runtime") + with tempfile.TemporaryDirectory(prefix="vergabe-synthetic-") as directory: + result = run(config, Path(directory), args.output) + result["source_revision"] = os.environ.get("SYNTHETIC_SOURCE_REVISION", "unrecorded") + result["driver_sha256"] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + result["host"] = host_facts(config) + result["config_sha256"] = hashlib.sha256(args.config.read_bytes()).hexdigest() + args.output.write_text(json.dumps(result, indent=2) + "\n") + keys = ("ordinary", "document_transfer", "complete", "provisional_pass") + print(json.dumps({k: result[k] for k in keys})) + return 0 if result["provisional_pass"] and result["measurement_complete"] else 1 + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--config", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + config = validate(json.loads(args.config.read_text())) + sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + if config.get("runtime") == "remote": + sys.exit(main_remote(config, args)) + name = "vergabe-synthetic-" + uuid.uuid4().hex[:12] + network = name + "-network" + network_args = [] + try: + if config.get("runtime") == "container": + subprocess.run( + ["docker", "network", "create", network], + check=True, + timeout=15, + stdout=subprocess.DEVNULL, + ) + os.environ["SYNTHETIC_NETWORK"] = network + network_args = ["--network", network, "--network-alias", "db"] + # Cached image only, loopback random host port, no existing volume or database. + subprocess.run( + [ + "docker", + "run", + "--detach", + "--pull=never", + "--name", + name, + "--publish", + "127.0.0.1::5432", + "--env", + "POSTGRES_DB=synthetic", + "--env", + "POSTGRES_USER=synthetic", + "--env", + "POSTGRES_PASSWORD=synthetic-local-only", + *network_args, + "postgres:16-alpine", + ], + check=True, + stdout=subprocess.DEVNULL, + ) + for _ in range(100): + ready = subprocess.run( + ["docker", "exec", name, "pg_isready", "-U", "synthetic"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if ready.returncode == 0: + break + time.sleep(0.1) + else: + raise RuntimeError("isolated PostgreSQL startup failed") + port = ( + subprocess.check_output(["docker", "port", name, "5432/tcp"], text=True) + .strip() + .split(":")[-1] + ) + os.environ["SYNTHETIC_CONTAINER"] = name + os.environ["SYNTHETIC_DATABASE_URL"] = ( + f"postgres://synthetic:synthetic-local-only@127.0.0.1:{port}/synthetic" + ) + with tempfile.TemporaryDirectory(prefix="vergabe-synthetic-") as directory: + result = run(config, Path(directory), args.output) + result["postgres_image_id"] = subprocess.check_output( + ["docker", "inspect", "--format", "{{.Image}}", name], text=True + ).strip() + result["postgres_final_stats"] = json.loads( + subprocess.check_output( + ["docker", "stats", "--no-stream", "--format", "{{json .}}", name], + text=True, + ) + ) + finally: + subprocess.run( + ["docker", "rm", "--force", "--volumes", name], + timeout=15, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if network_args: + subprocess.run( + ["docker", "network", "rm", network], + check=True, + timeout=15, + stdout=subprocess.DEVNULL, + ) + result["source_revision"] = subprocess.check_output( + ["git", "rev-parse", "HEAD"], text=True + ).strip() + result["driver_sha256"] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + app_files = sorted(Path("vergabe_teilnahme").rglob("*.py")) + [ + Path("uv.lock"), + Path("pyproject.toml"), + ] + result["app_source_sha256"] = hashlib.sha256( + b"".join(str(p).encode() + b"\0" + p.read_bytes() for p in app_files) + ).hexdigest() + result["host"] = { + "logical_cpus": os.cpu_count(), + "affinity_cpus": len(os.sched_getaffinity(0)), + "cpu_model": next( + ( + x.split(":", 1)[1].strip() + for x in Path("/proc/cpuinfo").read_text().splitlines() + if x.startswith("model name") + ), + "unknown", + ), + "memory_total": next( + x for x in Path("/proc/meminfo").read_text().splitlines() if x.startswith("MemTotal:") + ), + "cgroup_cpu_max": Path("/sys/fs/cgroup/cpu.max").read_text().strip() + if Path("/sys/fs/cgroup/cpu.max").exists() + else "unavailable", + "app_cpu_quota": config.get("app_cpu_limit", "not imposed by harness"), + } + result["config_sha256"] = hashlib.sha256(args.config.read_bytes()).hexdigest() + args.output.write_text(json.dumps(result, indent=2) + "\n") + print( + json.dumps( + { + k: result[k] + for k in ( + "ordinary", + "document_transfer", + "complete", + "provisional_pass", + ) + } + ) + ) + raise SystemExit(0 if result["provisional_pass"] and result["measurement_complete"] else 1) diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/orchestrate.sh b/docs/evidence/2026-10-21-vergabe-target-replay/harness/orchestrate.sh new file mode 100755 index 0000000..5bbc6e7 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/orchestrate.sh @@ -0,0 +1,15 @@ +#!/bin/bash +NS=vergabe-sizing-20261021; W=$SCRATCH; R=$W/raw; STOP=$R/stop; rm -f $STOP +$SCRATCH/sampler.sh app=vergabe-sizing-app app $R/app-cgroup.txt $STOP & +$SCRATCH/sampler.sh app=vergabe-sizing-db postgres $R/db-cgroup.txt $STOP & +$SCRATCH/sampler.sh app=vergabe-sizing-driver driver $R/driver-cgroup.txt $STOP & +$SCRATCH/top.sh $R/kubectl-top.txt $STOP & +echo "baseline_start $(date -u +%FT%T.%NZ)" > $R/windows.txt +sleep 120 +echo "driver_start $(date -u +%FT%T.%NZ)" >> $R/windows.txt +kubectl -n $NS exec synthetic-driver -- python /driver/synthetic_pilot.py --config /driver/config.json --output /out/result.json > $R/driver-stdout.txt 2>&1 +echo "driver_exit $? $(date -u +%FT%T.%NZ)" >> $R/windows.txt +sleep 60 +echo "post_end $(date -u +%FT%T.%NZ)" >> $R/windows.txt +touch $STOP; sleep 3; wait +echo done diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/prom.py b/docs/evidence/2026-10-21-vergabe-target-replay/harness/prom.py new file mode 100644 index 0000000..2dde2be --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/prom.py @@ -0,0 +1,25 @@ +import json, subprocess, sys, urllib.parse +start, end, out = sys.argv[1], sys.argv[2], sys.argv[3] +P = "/api/v1/namespaces/telemetry/services/telemetry-prometheus:9090/proxy/api/v1/query_range?" +def q(expr, step="15s"): + url = P + urllib.parse.urlencode({"query": expr, "start": start, "end": end, "step": step}) + return json.loads(subprocess.check_output(["kubectl", "get", "--raw", url])) +res = {"window": [start, end], "queries": {}} +for name, expr in { + "cpu_by_namespace_cores": 'sum by (namespace) (rate(container_cpu_usage_seconds_total{container!="",image!=""}[1m]))', + "throttled_ratio_by_namespace": 'sum by (namespace) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[1m])) / sum by (namespace) (rate(container_cpu_cfs_periods_total{container!=""}[1m]))', + "memory_working_set_by_namespace_bytes": 'sum by (namespace) (container_memory_working_set_bytes{container!="",image!=""})', + "apps_pg_cpu_cores": 'sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="databases",container!="",image!=""}[1m]))', + "live_vergabe_cpu_cores": 'sum (rate(container_cpu_usage_seconds_total{namespace="vergabe-demo-company",container!="",image!=""}[1m]))', +}.items(): + r = q(expr) + res["queries"][name] = {"expr": expr, "result": r["data"]["result"]} +json.dump(res, open(out, "w")) +# summary: mean per namespace cpu +s = {} +for series in res["queries"]["cpu_by_namespace_cores"]["result"]: + v = [float(x[1]) for x in series["values"]] + s[series["metric"].get("namespace", "?")] = (round(1000*sum(v)/len(v)), round(1000*max(v))) +tot = sum(a for a, b in s.values()) +print("container_total_mean_m", tot) +for k, v in sorted(s.items(), key=lambda kv: -kv[1][0])[:15]: print(k, "mean_m", v[0], "max_m", v[1]) diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/sampler.sh b/docs/evidence/2026-10-21-vergabe-target-replay/harness/sampler.sh new file mode 100755 index 0000000..dd469c6 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/sampler.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# usage: sampler.sh +NS=vergabe-sizing-20261021; SEL=$1; C=$2; OUT=$3; STOP=$4 +LOOP='while :; do echo "S $(date +%s.%N)"; cat /sys/fs/cgroup/cpu.stat; echo "M $(cat /sys/fs/cgroup/memory.current)"; echo "P $(head -1 /sys/fs/cgroup/cpu.pressure)"; echo "N $(head -1 /proc/stat)"; echo "L $(cat /proc/loadavg)"; sleep 1; done' +while [ ! -e "$STOP" ]; do + POD=$(kubectl -n $NS get pod -l "$SEL" --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) + if [ -n "$POD" ]; then + echo "POD $POD $(date +%s.%N)" >> "$OUT" + timeout 1500 kubectl -n $NS exec "$POD" -c "$C" -- sh -c "$LOOP" >> "$OUT" 2>>"$OUT.err" & + PID=$! + while kill -0 $PID 2>/dev/null && [ ! -e "$STOP" ]; do sleep 1; done + kill $PID 2>/dev/null + echo "DETACH $(date +%s.%N)" >> "$OUT" + fi + sleep 0.5 +done diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/synthetic_settings.py b/docs/evidence/2026-10-21-vergabe-target-replay/harness/synthetic_settings.py new file mode 100644 index 0000000..2260675 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/synthetic_settings.py @@ -0,0 +1,12 @@ +from vergabe_teilnahme.settings.base import * +DEBUG = False +ALLOWED_HOSTS = ['vergabe-app'] +NETKINGDOM_ENABLED = False +MEDIA_ROOT = '/app/media' +ISSUE_FACADE_LOCAL_DB = '/app/.issue-facade/issues.db' +ISSUE_FACADE_GITEA = None +FORCE_SCRIPT_NAME = None +MEDIA_URL = "/media/" +EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend" +SESSION_COOKIE_SECURE = False +CSRF_COOKIE_SECURE = False diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/harness/top.sh b/docs/evidence/2026-10-21-vergabe-target-replay/harness/top.sh new file mode 100755 index 0000000..8366b81 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/harness/top.sh @@ -0,0 +1,3 @@ +#!/bin/bash +OUT=$1; STOP=$2 +while [ ! -e "$STOP" ]; do echo "T $(date -u +%s)"; kubectl top pods -A --no-headers 2>&1; kubectl top node --no-headers 2>&1; sleep 15; done >> $OUT diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/app-cgroup.txt.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/app-cgroup.txt.gz new file mode 100644 index 0000000..7a1b12b Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/app-cgroup.txt.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/db-cgroup.txt.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/db-cgroup.txt.gz new file mode 100644 index 0000000..839d6da Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/db-cgroup.txt.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-cgroup.txt.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-cgroup.txt.gz new file mode 100644 index 0000000..1676805 Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-cgroup.txt.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-stdout.txt b/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-stdout.txt new file mode 100644 index 0000000..3aab4e1 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/raw/driver-stdout.txt @@ -0,0 +1 @@ +{"ordinary": {"requests": 3675, "errors": 0, "p95_seconds": 0.19182809395715594}, "document_transfer": {"requests": 814, "errors": 0, "p95_seconds": 0.07621732295956463}, "complete": true, "provisional_pass": true} diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/kubectl-top.txt.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/kubectl-top.txt.gz new file mode 100644 index 0000000..edbdace Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/kubectl-top.txt.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/namespace-events.txt b/docs/evidence/2026-10-21-vergabe-target-replay/raw/namespace-events.txt new file mode 100644 index 0000000..f7c67b9 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/raw/namespace-events.txt @@ -0,0 +1,36 @@ +T R O M + Scheduled vergabe-app-579448bf49-2fsn2 Successfully assigned vergabe-sizing-20261021/vergabe-app-579448bf49-2fsn2 to 239.62.205.92.host.secureserver.net + Scheduled vergabe-db-7c4bffdd48-w4jz4 Successfully assigned vergabe-sizing-20261021/vergabe-db-7c4bffdd48-w4jz4 to 239.62.205.92.host.secureserver.net + Scheduled synthetic-driver Successfully assigned vergabe-sizing-20261021/synthetic-driver to 239.62.205.92.host.secureserver.net + Scheduled vergabe-app-579448bf49-plp4j Successfully assigned vergabe-sizing-20261021/vergabe-app-579448bf49-plp4j to 239.62.205.92.host.secureserver.net +2026-10-21T06:26:14Z ScalingReplicaSet vergabe-db Scaled up replica set vergabe-db-7c4bffdd48 from 0 to 1 +2026-10-21T06:26:14Z SuccessfulCreate vergabe-db-7c4bffdd48 Created pod: vergabe-db-7c4bffdd48-w4jz4 +2026-10-21T06:26:14Z WaitForFirstConsumer bench-media waiting for first consumer to be created before binding +2026-10-21T06:26:16Z SuccessfulCreate vergabe-app-579448bf49 Created pod: vergabe-app-579448bf49-2fsn2 +2026-10-21T06:26:16Z ScalingReplicaSet vergabe-app Scaled up replica set vergabe-app-579448bf49 from 0 to 1 +2026-10-21T06:26:16Z Pulled vergabe-db-7c4bffdd48-w4jz4 Container image "docker.io/library/postgres@sha256:4b7183ac05f8ef417db21fd72d71047a4238340c261d3cc3ddb6d579ab5071ae" already present on machine and can be accessed by the pod +2026-10-21T06:26:16Z Created vergabe-db-7c4bffdd48-w4jz4 Container created +2026-10-21T06:26:16Z Provisioning bench-media External provisioner is provisioning volume for claim "vergabe-sizing-20261021/bench-media" +2026-10-21T06:26:16Z Started vergabe-db-7c4bffdd48-w4jz4 Container started +2026-10-21T06:26:19Z Unhealthy vergabe-db-7c4bffdd48-w4jz4 Readiness probe failed: /var/run/postgresql:5432 - no response + +2026-10-21T06:26:21Z Created synthetic-driver Container created +2026-10-21T06:26:21Z ExternalProvisioning bench-media Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. +2026-10-21T06:26:21Z Pulled synthetic-driver Container image "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91" already present on machine and can be accessed by the pod +2026-10-21T06:26:22Z Started synthetic-driver Container started +2026-10-21T06:26:25Z ProvisioningSucceeded bench-media Successfully provisioned volume pvc-a23ccb84-ecd8-47af-9832-6d8b09c48e38 +2026-10-21T06:26:26Z Started vergabe-app-579448bf49-2fsn2 Container started +2026-10-21T06:26:26Z Created vergabe-app-579448bf49-2fsn2 Container created +2026-10-21T06:26:26Z Pulled vergabe-app-579448bf49-2fsn2 Container image "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91" already present on machine and can be accessed by the pod +2026-10-21T06:26:32Z Started vergabe-app-579448bf49-2fsn2 Container started +2026-10-21T06:26:32Z Created vergabe-app-579448bf49-2fsn2 Container created +2026-10-21T06:26:32Z Pulled vergabe-app-579448bf49-2fsn2 Container image "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91" already present on machine and can be accessed by the pod +2026-10-21T06:32:03Z SuccessfulCreate vergabe-app-579448bf49 Created pod: vergabe-app-579448bf49-plp4j +2026-10-21T06:32:03Z Killing vergabe-app-579448bf49-2fsn2 Stopping container app +2026-10-21T06:32:04Z Started vergabe-app-579448bf49-plp4j Container started +2026-10-21T06:32:04Z Created vergabe-app-579448bf49-plp4j Container created +2026-10-21T06:32:04Z Pulled vergabe-app-579448bf49-plp4j Container image "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91" already present on machine and can be accessed by the pod +2026-10-21T06:32:04Z Unhealthy vergabe-app-579448bf49-2fsn2 Readiness probe failed: Get "http://10.42.0.96:8000/health/": dial tcp 10.42.0.96:8000: connect: connection refused +2026-10-21T06:32:09Z Started vergabe-app-579448bf49-plp4j Container started +2026-10-21T06:32:09Z Created vergabe-app-579448bf49-plp4j Container created +2026-10-21T06:32:09Z Pulled vergabe-app-579448bf49-plp4j Container image "forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91" already present on machine and can be accessed by the pod diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/pod-provenance.txt b/docs/evidence/2026-10-21-vergabe-target-replay/raw/pod-provenance.txt new file mode 100644 index 0000000..0ca72f5 --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/raw/pod-provenance.txt @@ -0,0 +1,4 @@ +synthetic-driver driver forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 0 {} ['running'] +vergabe-app-579448bf49-plp4j migrate forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 0 {} ['terminated'] +vergabe-app-579448bf49-plp4j app forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 0 {} ['running'] +vergabe-db-7c4bffdd48-w4jz4 postgres docker.io/library/postgres@sha256:4b7183ac05f8ef417db21fd72d71047a4238340c261d3cc3ddb6d579ab5071ae 0 {} ['running'] diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-baseline.json.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-baseline.json.gz new file mode 100644 index 0000000..f246eec Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-baseline.json.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-workload.json.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-workload.json.gz new file mode 100644 index 0000000..c589397 Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/prometheus-workload.json.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/result.json.gz b/docs/evidence/2026-10-21-vergabe-target-replay/raw/result.json.gz new file mode 100644 index 0000000..7128c41 Binary files /dev/null and b/docs/evidence/2026-10-21-vergabe-target-replay/raw/result.json.gz differ diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/teardown.txt b/docs/evidence/2026-10-21-vergabe-target-replay/raw/teardown.txt new file mode 100644 index 0000000..85aa06d --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/raw/teardown.txt @@ -0,0 +1,7 @@ +delete_requested 2026-10-21T06:35:25Z +namespace "vergabe-sizing-20261021" deleted +delete_returned 2026-10-21T06:36:18Z +Error from server (NotFound): namespaces "vergabe-sizing-20261021" not found +Error from server (NotFound): persistentvolumes "pvc-a23ccb84-ecd8-47af-9832-6d8b09c48e38" not found +0 +live generation=4 image=forgejo.coulomb.social/coulomb/vergabe-teilnahme@sha256:a26444f59c259698159c69ccb96f73dc648a261ece4c86bb2037a9d977870d91 ready=1 diff --git a/docs/evidence/2026-10-21-vergabe-target-replay/raw/windows.txt b/docs/evidence/2026-10-21-vergabe-target-replay/raw/windows.txt new file mode 100644 index 0000000..8e3d24d --- /dev/null +++ b/docs/evidence/2026-10-21-vergabe-target-replay/raw/windows.txt @@ -0,0 +1,4 @@ +baseline_start 2026-10-21T06:27:14.447540583Z +driver_start 2026-10-21T06:29:26.175778371Z +driver_exit 0 2026-10-21T06:33:22.885983171Z +post_end 2026-10-21T06:34:28.753426744Z diff --git a/workplans/RAPPS-WP-0014-vergabe-invited-pilot.md b/workplans/RAPPS-WP-0014-vergabe-invited-pilot.md index d1cfe51..af15267 100644 --- a/workplans/RAPPS-WP-0014-vergabe-invited-pilot.md +++ b/workplans/RAPPS-WP-0014-vergabe-invited-pilot.md @@ -9,7 +9,7 @@ flavor: implementation owner: the-custodian topic_slug: railiance created: "2026-09-11" -updated: "2026-10-19" +updated: "2026-10-21" related: [VERGABE-WP-0019, VERGABE-WP-0018, HFACT-WP-0001, CUST-WP-0071] state_hub_workstream_id: "c7fdaa7e-cab8-5d1d-86c2-f1aad7927c57" --- @@ -252,7 +252,7 @@ all remaining work; no task or workplan was opened. ```task id: RAPPS-WP-0014-T04 -status: todo +status: done priority: high assignee: railiance-apps state_hub_task_id: "c53fcaa9-859c-5b37-a17b-0a4cd2fb0e26" @@ -293,3 +293,18 @@ coverage/gaps and a bounded recommendation to `RESOURCE-WP-0008-T02` for headroo and allocation review and `CUST-WP-0071-T02` for sizing acceptance. T03 remains unchanged: operator/customer recovery and human acceptance are not inferred from these synthetic results. + +2026-10-21 result: isolated replay run on railiance01 under the founder's +2026-10-21 decision, in throwaway namespace `vergabe-sizing-20261021` (deleted +06:36:18Z, PV removed; live deployment unchanged). Deployed digest +`sha256:a26444f5…` (source `c067993`, chart 0.2.1, 3 Gunicorn workers, +60m/256Mi request, 1000m/1Gi limit) against a fresh PostgreSQL 16 pod, phased +profile with in-cluster driver. 3675 ordinary requests p95 0.192s and 814 +transfers p95 0.076s, zero errors, restart ready in 19s. App CPU 775m mean under +load, 36% of CFS periods throttled (66% in burst), memory peak 483 MiB; isolated +DB +~320m mean; node free CPU 1.2 cores mean, ~150m at p5 (largely self-induced). +Recommendation: keep 1000m/1Gi limits and the 60m request for the two-user pilot; +consider 512Mi memory request at allocation review. Gaps: synthetic settings (no +SSO/ingress/TLS), standalone DB rather than shared `apps-pg`, synthetic closed +loop, one run at a quiet hour. Evidence and revisit triggers: +`docs/evidence/2026-10-21-vergabe-target-replay/README.md`. T03 is unchanged.