feat: measure configurable synthetic pilot workload in isolated PostgreSQL runtime
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a15374-f59f-7a31-86d2-d23b5d2ed181
This commit is contained in:
parent
a1793dbe12
commit
11f7c8ae83
12 changed files with 1134 additions and 0 deletions
13
SCOPE.md
13
SCOPE.md
|
|
@ -189,3 +189,16 @@ existing bytes remain available for offline session startup; refreshing it does
|
|||
not require a Git commit. Authored workplans and the generated WORK-RECORDS index
|
||||
retain their existing source and staged-generation contracts. No runtime or
|
||||
credential change follows from this repository hygiene policy.
|
||||
|
||||
|
||||
## Provisional synthetic pilot sizing
|
||||
|
||||
`scripts/synthetic_pilot.py` drives real loopback HTTP through an isolated Gunicorn
|
||||
instance and disposable PostgreSQL16. `config/synthetic-pilot.json` tunes workload
|
||||
assumptions, and `config/synthetic-sizing-margins.json` tunes the conservative local
|
||||
resource-envelope calculation. Per-operation latency/errors and separate document
|
||||
transfer, process CPU/RSS and database snapshots are retained. The 2026-10-19 local
|
||||
sample passes the two-user ordinary p95<=2s/zero-error criterion; it does not establish
|
||||
human acceptance or the production60m CPU request. Coverage and provisional values
|
||||
are in `docs/synthetic-pilot.md` and `docs/evidence/2026-10-19-synthetic-sizing/`.
|
||||
No deployment or publication is performed by these tools.
|
||||
|
|
|
|||
12
config/synthetic-pilot.json
Normal file
12
config/synthetic-pilot.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"users": 2,
|
||||
"workers": 2,
|
||||
"iterations": 100000,
|
||||
"tenders": 100,
|
||||
"document_bytes": 1048576,
|
||||
"think_seconds": 0.2,
|
||||
"timeout_seconds": 30,
|
||||
"ordinary_p95_seconds": 2.0,
|
||||
"duration_seconds": 180,
|
||||
"idle_seconds": 30
|
||||
}
|
||||
6
config/synthetic-sizing-margins.json
Normal file
6
config/synthetic-sizing-margins.json
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"cpu_request_multiplier": 1.5,
|
||||
"cpu_limit_multiplier": 1.5,
|
||||
"memory_request_multiplier": 1.25,
|
||||
"memory_limit_multiplier": 1.5
|
||||
}
|
||||
44
docs/evidence/2026-10-19-synthetic-sizing/README.md
Normal file
44
docs/evidence/2026-10-19-synthetic-sizing/README.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# Isolated synthetic pilot sizing — 2026-10-19
|
||||
|
||||
The user authorized provisional synthetic sizing with adjustable assumptions.
|
||||
Run locally against fresh PostgreSQL16 and loopback Gunicorn, not production.
|
||||
Two users, two workers,100 initial tenders,1MiB documents,0.2-second think time.
|
||||
30s idle then180.889s monotonic ordinary duration;378 complete cycles.
|
||||
|
||||
| Result | Value |
|
||||
| --- | --- |
|
||||
| Ordinary requests / errors / p95 | 2272 / 0 / 0.226793s |
|
||||
| Document transfer requests / errors / p95 | 756 / 0 / 0.095466s |
|
||||
| Aggregate app CPU p95 / observed peak | 1298.30m / 1916.96m |
|
||||
| Peak summed app RSS | 193.84MiB |
|
||||
| Process samples / database interval snapshots | 422 / 103 |
|
||||
| Conservative local CPU request / limit candidate | 1950m / 2900m |
|
||||
| Conservative local memory request / limit candidate | 256Mi / 320Mi |
|
||||
|
||||
This ~16.7-requests/second synthetic mix is deliberately aggressive. It is not
|
||||
measured human demand. This unbounded local envelope exceeds the accepted production1000m CPU limit
|
||||
and the current395m reservation residual reported by the allocation owner. It is
|
||||
not a recommendation to raise that limit. The next comparison is an isolated
|
||||
application-container run at the existing1000m/1Gi limit;60m is a request, not a quota.
|
||||
The candidate envelope derives from configurable margins
|
||||
in `config/synthetic-sizing-margins.json`; no deployment change is made or implied.
|
||||
The eight-logical-CPU Ryzen5 PRO3500U host had no harness app CPU quota. Neither
|
||||
these results nor the passing latency criterion prove the existing60m production
|
||||
request. Database snapshots are retained separately and not charged to app demand.
|
||||
RSS sums shared resident pages more than once; per-worker CPU is aggregate/2,
|
||||
not independently observed worker load. Idle is measured; startup timing, restart,
|
||||
editing, separate bursts, browser/ingress/SSO and production quota/throttling are not.
|
||||
Repeat with revised assumptions or approved target-runtime constraints before
|
||||
production right-sizing. Human pilot acceptance remains open.
|
||||
|
||||
`sizing-run.json.gz` preserves raw per-operation and resource samples, source/config/
|
||||
image digests and hardware. `sizing-envelope.json` is reproducible with the summary
|
||||
command in `docs/synthetic-pilot.md`. `executed-sizing-driver.py` is the exact run
|
||||
snapshot matching its retained digest; the maintained script subsequently received
|
||||
only import/format lint cleanup. `exploratory.json.gz` retains the earlier15.7s
|
||||
sample, not used as the sizing recommendation. Use monotonic duration for rates;
|
||||
UTC boundaries are retained as observed and may differ due to wall-clock adjustment.
|
||||
|
||||
Validation: six focused configuration/percentile/sizing tests; Ruff clean for
|
||||
maintained scripts; complete real HTTP run passed all3028 operations. No customer
|
||||
records, external identity, live credentials or production writes were used.
|
||||
|
|
@ -0,0 +1,292 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Provisional loopback HTTP workload using Gunicorn and disposable PostgreSQL."""
|
||||
import argparse
|
||||
import atexit
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
import http.cookiejar
|
||||
import uuid
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def summarize(rows):
|
||||
durations = sorted(row['seconds'] for row in rows)
|
||||
return {'requests': len(rows), 'errors': sum(not r['ok'] for r in rows),
|
||||
'p95_seconds': durations[math.ceil(len(durations) * .95) - 1]
|
||||
if durations else None}
|
||||
|
||||
|
||||
def validate(config):
|
||||
for key in ('users', 'iterations', 'tenders', 'document_bytes', 'workers', 'timeout_seconds'):
|
||||
if type(config[key]) is not int or config[key] < 1:
|
||||
raise ValueError(f'{key} must be a positive integer')
|
||||
for key in ('think_seconds', 'ordinary_p95_seconds', 'duration_seconds', 'idle_seconds'):
|
||||
if not isinstance(config[key], (int, float)) or not math.isfinite(config[key]):
|
||||
raise ValueError(f'{key} must be finite')
|
||||
if config['think_seconds'] < 0 or config['ordinary_p95_seconds'] <= 0:
|
||||
raise ValueError('invalid timing parameter')
|
||||
if config['duration_seconds'] <= 0 or config['idle_seconds'] < 0:
|
||||
raise ValueError('duration must be positive and idle nonnegative')
|
||||
if config['document_bytes'] < 8:
|
||||
raise ValueError('document_bytes must be at least 8')
|
||||
return config
|
||||
|
||||
|
||||
def run(config, root):
|
||||
# Never consume a deployment database, media tree, session or credential.
|
||||
os.environ['DATABASE_URL'] = os.environ['SYNTHETIC_DATABASE_URL']
|
||||
os.environ['SECRET_KEY'] = 'synthetic-disposable-not-a-deployment-key'
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'synthetic_settings'
|
||||
(root / 'synthetic_settings.py').write_text(
|
||||
'from vergabe_teilnahme.settings.base import *\n'
|
||||
'DEBUG = False\nALLOWED_HOSTS = ["127.0.0.1"]\nNETKINGDOM_ENABLED = False\n'
|
||||
f'MEDIA_ROOT = {str(root / "media")!r}\n'
|
||||
f'ISSUE_FACADE_LOCAL_DB = {str(root / "issues.db")!r}\n'
|
||||
'ISSUE_FACADE_GITEA = None\nFORCE_SCRIPT_NAME = None\nMEDIA_URL = "/media/"\n'
|
||||
'EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"\n'
|
||||
'SESSION_COOKIE_SECURE = False\nCSRF_COOKIE_SECURE = False\n'
|
||||
)
|
||||
sys.path.insert(0, str(root))
|
||||
import django
|
||||
django.setup()
|
||||
from django.core.management import call_command
|
||||
from django.db import connections, connection
|
||||
from vergabe_teilnahme.apps.accounts.models import Mitarbeiter
|
||||
from vergabe_teilnahme.apps.ausschreibungen.models import Ausschreibung
|
||||
from vergabe_teilnahme.apps.dokumente.models import Dokument
|
||||
call_command('migrate', verbosity=0)
|
||||
for i in range(config['tenders']):
|
||||
Ausschreibung.objects.create(titel=f'Synthetic tender {i}', ausschreiber='Synthetic')
|
||||
users = []
|
||||
password = uuid.uuid4().hex
|
||||
for i in range(config['users']):
|
||||
user = Mitarbeiter.objects.create_user(username=f'synthetic-{i}', password=password)
|
||||
users.append(user.username)
|
||||
with socket.socket() as sock:
|
||||
sock.bind(('127.0.0.1', 0))
|
||||
port = sock.getsockname()[1]
|
||||
base_url = f'http://127.0.0.1:{port}'
|
||||
env = dict(os.environ, PYTHONPATH=str(root) + os.pathsep + str(Path.cwd()))
|
||||
server_log = (root / 'server.log').open('w')
|
||||
server = subprocess.Popen([sys.executable, '-m', 'gunicorn',
|
||||
'vergabe_teilnahme.wsgi:application', '--bind', f'127.0.0.1:{port}',
|
||||
'--workers', str(config['workers']), '--timeout', '30'], env=env,
|
||||
stdout=server_log, stderr=server_log)
|
||||
atexit.register(lambda: server.terminate() if server.poll() is None else None)
|
||||
for _ in range(100):
|
||||
try:
|
||||
with urllib.request.urlopen(base_url + '/health/', timeout=1) as response:
|
||||
if response.status == 200:
|
||||
break
|
||||
except OSError:
|
||||
if server.poll() is not None:
|
||||
raise RuntimeError('Gunicorn startup failed; see isolated server log')
|
||||
time.sleep(.1)
|
||||
else:
|
||||
server.terminate()
|
||||
raise RuntimeError('Gunicorn readiness timeout')
|
||||
samples = []
|
||||
db_samples = []
|
||||
sampling_done = threading.Event()
|
||||
phase = {'name': 'idle'}
|
||||
def sample():
|
||||
ticks_per_second = os.sysconf('SC_CLK_TCK')
|
||||
while not sampling_done.is_set():
|
||||
pids = [server.pid]
|
||||
try:
|
||||
pids += [int(x) for x in Path(f'/proc/{server.pid}/task/{server.pid}/children').read_text().split()]
|
||||
cpu, rss = 0, 0
|
||||
for pid in pids:
|
||||
fields = Path(f'/proc/{pid}/stat').read_text().split(') ', 1)[1].split()
|
||||
cpu += (int(fields[11]) + int(fields[12])) / ticks_per_second
|
||||
rss += int(fields[21]) * os.sysconf('SC_PAGE_SIZE')
|
||||
samples.append({'at': datetime.now(timezone.utc).isoformat(),
|
||||
'monotonic_seconds': time.monotonic(), 'phase': phase['name'],
|
||||
'cpu_seconds': cpu, 'rss_bytes': rss, 'pids': pids})
|
||||
except (OSError, ValueError):
|
||||
samples.append({'at': datetime.now(timezone.utc).isoformat(), 'sample_error': True})
|
||||
sampling_done.wait(.5)
|
||||
def sample_db():
|
||||
while not sampling_done.is_set():
|
||||
value = subprocess.check_output(['docker', 'stats', '--no-stream', '--format',
|
||||
'{{json .}}', os.environ['SYNTHETIC_CONTAINER']], text=True)
|
||||
db_samples.append({'at': datetime.now(timezone.utc).isoformat(),
|
||||
'phase': phase['name'], 'docker_stats': json.loads(value)})
|
||||
sampling_done.wait(1)
|
||||
sampler = threading.Thread(target=sample, daemon=True)
|
||||
db_sampler = threading.Thread(target=sample_db, daemon=True)
|
||||
sampler.start()
|
||||
db_sampler.start()
|
||||
time.sleep(config['idle_seconds'])
|
||||
phase['name'] = 'ordinary'
|
||||
deadline = time.monotonic() + config['duration_seconds']
|
||||
completed = [0] * config['users']
|
||||
barrier = threading.Barrier(config['users'])
|
||||
def worker(i):
|
||||
jar = http.cookiejar.CookieJar()
|
||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, *args, **kwargs):
|
||||
return None
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar), NoRedirect())
|
||||
rows = []
|
||||
def request(operation, method, path, data=None, expected=200, transfer=False):
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
headers = {}
|
||||
body = None
|
||||
if method == 'post':
|
||||
token = next((c.value for c in jar if c.name == 'csrftoken'), '')
|
||||
fields = dict(data or {}, csrfmiddlewaretoken=token)
|
||||
if transfer:
|
||||
boundary = 'synthetic-' + uuid.uuid4().hex
|
||||
body = b''
|
||||
for name, value in fields.items():
|
||||
body += f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\n\r\n{value}\r\n'.encode()
|
||||
body += (f'--{boundary}\r\nContent-Disposition: form-data; name="datei"; filename="synthetic.pdf"\r\nContent-Type: application/pdf\r\n\r\n'.encode()
|
||||
+ b'%PDF-1.4' + b'x' * (config['document_bytes'] - 8)
|
||||
+ f'\r\n--{boundary}--\r\n'.encode())
|
||||
headers['Content-Type'] = f'multipart/form-data; boundary={boundary}'
|
||||
else:
|
||||
body = urllib.parse.urlencode(fields).encode()
|
||||
headers['Referer'] = base_url + path
|
||||
req = urllib.request.Request(base_url + path, data=body, headers=headers,
|
||||
method=method.upper())
|
||||
try:
|
||||
response = opener.open(req, timeout=config['timeout_seconds'])
|
||||
except urllib.error.HTTPError as exc:
|
||||
response = exc
|
||||
received = len(response.read())
|
||||
status = response.status
|
||||
ok = status == expected
|
||||
response.close()
|
||||
except Exception:
|
||||
status, ok, received = None, False, 0
|
||||
rows.append({'user': i, 'operation': operation, 'transfer': transfer,
|
||||
'seconds': time.perf_counter() - started, 'status': status,
|
||||
'ok': ok, 'response_bytes': received})
|
||||
return ok
|
||||
request('login-form', 'get', '/accounts/login/')
|
||||
barrier.wait(timeout=30)
|
||||
request('login', 'post', '/accounts/login/',
|
||||
{'username': users[i], 'password': password}, 302)
|
||||
for n in range(config['iterations']):
|
||||
if time.monotonic() >= deadline:
|
||||
break
|
||||
request('list', 'get', '/ausschreibungen/')
|
||||
request('search', 'get', '/ausschreibungen/?q=Synthetic')
|
||||
title = f'Workload-{i}-{n}'
|
||||
created = request('tender-create', 'post', '/ausschreibungen/neu/',
|
||||
{'titel': title, 'ausschreiber': 'Synthetic'}, 302)
|
||||
tender = Ausschreibung.objects.filter(titel=title).first() if created else None
|
||||
if tender:
|
||||
base = f'/ausschreibungen/{tender.pk}/'
|
||||
request('detail', 'get', base)
|
||||
request('lot-create', 'post', base + 'lose/neu/',
|
||||
{'losnummer': '1', 'lostitel': 'Synthetic lot'}, 302)
|
||||
request('task-create', 'post', base + 'aufgaben/neu/',
|
||||
{'titel': 'Synthetic task', 'typ': 'fachlich', 'prioritaet': 2}, 302)
|
||||
request('document-upload', 'post', base + 'dokumente/hochladen/',
|
||||
{'kategorie': 'intern',
|
||||
'version': '1.0'}, 302, True)
|
||||
document = Dokument.objects.filter(ausschreibung=tender).first()
|
||||
if document:
|
||||
request('document-download', 'get', document.datei.url, transfer=True)
|
||||
completed[i] += 1
|
||||
time.sleep(config['think_seconds'])
|
||||
connections.close_all()
|
||||
return rows
|
||||
started = datetime.now(timezone.utc).isoformat()
|
||||
before = time.perf_counter()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=config['users']) as pool:
|
||||
rows = [r for group in pool.map(worker, range(config['users'])) for r in group]
|
||||
ordinary = summarize([r for r in rows if not r['transfer']])
|
||||
transfers = summarize([r for r in rows if r['transfer']])
|
||||
complete = all(completed) and len(rows) == sum(completed) * 8 + config['users'] * 2
|
||||
result = {'schema_version': 1, 'started_at': started,
|
||||
'finished_at': datetime.now(timezone.utc).isoformat(),
|
||||
'duration_seconds': time.perf_counter() - before, 'config': config,
|
||||
'environment': {'python': platform.python_version(), 'django': django.get_version(),
|
||||
'platform': platform.platform(), 'database': 'disposable PostgreSQL 16 container',
|
||||
'transport': 'loopback HTTP Gunicorn',
|
||||
'workers': config['workers']},
|
||||
'ordinary': ordinary, 'document_transfer': transfers, 'complete': complete,
|
||||
'provisional_pass': complete and ordinary['errors'] == 0
|
||||
and transfers['errors'] == 0
|
||||
and ordinary['p95_seconds'] <= config['ordinary_p95_seconds'],
|
||||
'limitations': ['No browser, ingress, TLS, SSO or application container quota',
|
||||
'No CPU/memory allocation recommendation or human acceptance',
|
||||
'No timed startup/restart or distinct burst; configured idle and ordinary phases only'],
|
||||
'operations': rows, 'runtime_samples': samples, 'database_samples': db_samples,
|
||||
'completed_iterations': completed}
|
||||
sampling_done.set()
|
||||
sampler.join()
|
||||
db_sampler.join()
|
||||
server.terminate()
|
||||
server.wait(timeout=10)
|
||||
server_log.close()
|
||||
connections.close_all()
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--config', type=Path, required=True)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
config = validate(json.loads(args.config.read_text()))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
name = 'vergabe-synthetic-' + uuid.uuid4().hex[:12]
|
||||
# Cached image only, loopback random host port, no existing volume or database.
|
||||
subprocess.run(['docker', 'run', '--detach', '--pull=never', '--name', name,
|
||||
'--publish', '127.0.0.1::5432', '--env', 'POSTGRES_DB=synthetic',
|
||||
'--env', 'POSTGRES_USER=synthetic', '--env', 'POSTGRES_PASSWORD=synthetic-local-only',
|
||||
'postgres:16-alpine'], check=True, stdout=subprocess.DEVNULL)
|
||||
try:
|
||||
for _ in range(100):
|
||||
ready = subprocess.run(['docker', 'exec', name, 'pg_isready', '-U', 'synthetic'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
if ready.returncode == 0:
|
||||
break
|
||||
time.sleep(.1)
|
||||
else:
|
||||
raise RuntimeError('isolated PostgreSQL startup failed')
|
||||
port = subprocess.check_output(['docker', 'port', name, '5432/tcp'], text=True).strip().split(':')[-1]
|
||||
os.environ['SYNTHETIC_CONTAINER'] = name
|
||||
os.environ['SYNTHETIC_DATABASE_URL'] = f'postgres://synthetic:synthetic-local-only@127.0.0.1:{port}/synthetic'
|
||||
with tempfile.TemporaryDirectory(prefix='vergabe-synthetic-') as directory:
|
||||
result = run(config, Path(directory))
|
||||
result['postgres_image_id'] = subprocess.check_output(
|
||||
['docker', 'inspect', '--format', '{{.Image}}', name], text=True).strip()
|
||||
result['postgres_final_stats'] = json.loads(subprocess.check_output(
|
||||
['docker', 'stats', '--no-stream', '--format', '{{json .}}', name], text=True))
|
||||
finally:
|
||||
subprocess.run(['docker', 'rm', '--force', '--volumes', name], check=True,
|
||||
stdout=subprocess.DEVNULL)
|
||||
result['source_revision'] = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip()
|
||||
result['driver_sha256'] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
|
||||
app_files = sorted(Path('vergabe_teilnahme').rglob('*.py')) + [Path('uv.lock'), Path('pyproject.toml')]
|
||||
result['app_source_sha256'] = hashlib.sha256(b''.join(str(p).encode() + b'\0' + p.read_bytes() for p in app_files)).hexdigest()
|
||||
result['host'] = {'logical_cpus': os.cpu_count(), 'affinity_cpus': len(os.sched_getaffinity(0)),
|
||||
'cpu_model': next((x.split(':', 1)[1].strip() for x in Path('/proc/cpuinfo').read_text().splitlines() if x.startswith('model name')), 'unknown'),
|
||||
'memory_total': next(x for x in Path('/proc/meminfo').read_text().splitlines() if x.startswith('MemTotal:')),
|
||||
'cgroup_cpu_max': Path('/sys/fs/cgroup/cpu.max').read_text().strip() if Path('/sys/fs/cgroup/cpu.max').exists() else 'unavailable',
|
||||
'app_cpu_quota': 'not imposed by harness'}
|
||||
result['config_sha256'] = hashlib.sha256(args.config.read_bytes()).hexdigest()
|
||||
args.output.write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(json.dumps({k: result[k] for k in ('ordinary', 'document_transfer', 'complete',
|
||||
'provisional_pass')}))
|
||||
raise SystemExit(0 if result['provisional_pass'] else 1)
|
||||
BIN
docs/evidence/2026-10-19-synthetic-sizing/exploratory.json.gz
Normal file
BIN
docs/evidence/2026-10-19-synthetic-sizing/exploratory.json.gz
Normal file
Binary file not shown.
|
|
@ -0,0 +1,30 @@
|
|||
{
|
||||
"status": "provisional-local-host-only",
|
||||
"application_observed": {
|
||||
"cpu_p95_m": 1298.300558321979,
|
||||
"cpu_peak_m": 1916.960754568632,
|
||||
"rss_sum_peak_mib": 193.8359375,
|
||||
"sample_intervals": 361,
|
||||
"normalized_cpu_p95_m_per_worker": 649.1502791609895
|
||||
},
|
||||
"suggested_application": {
|
||||
"cpu_request_m": 1950,
|
||||
"cpu_limit_m": 2900,
|
||||
"memory_request_mib": 256,
|
||||
"memory_limit_mib": 320
|
||||
},
|
||||
"margins": {
|
||||
"cpu_request_multiplier": 1.5,
|
||||
"cpu_limit_multiplier": 1.5,
|
||||
"memory_request_multiplier": 1.25,
|
||||
"memory_limit_multiplier": 1.5
|
||||
},
|
||||
"limits": [
|
||||
"Per-worker number is normalized aggregate, not an independently sampled worker.",
|
||||
"RSS sum double-counts shared pages; conservative upper bound, not unique memory.",
|
||||
"Observed local CPU is hardware/think-time dependent; quota effects untested.",
|
||||
"Database/container demand is separate and not included in application request.",
|
||||
"No production apply or human acceptance; validate target runtime before right-sizing."
|
||||
],
|
||||
"revisit": "Changed concurrency, corpus, document size, workers, image, hardware or quota; then repeat under approved target runtime including bursts and restart."
|
||||
}
|
||||
BIN
docs/evidence/2026-10-19-synthetic-sizing/sizing-run.json.gz
Normal file
BIN
docs/evidence/2026-10-19-synthetic-sizing/sizing-run.json.gz
Normal file
Binary file not shown.
82
docs/synthetic-pilot.md
Normal file
82
docs/synthetic-pilot.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# Provisional synthetic pilot workload
|
||||
|
||||
The owner implementation is `scripts/synthetic_pilot.py`, configured by
|
||||
`config/synthetic-pilot.json`. It reuses the application routes/forms exercised
|
||||
by existing workflow tests. There was no existing performance driver in this
|
||||
repository or railiance-apps. The driver uses real loopback HTTP, cookie sessions,
|
||||
CSRF, Gunicorn and a fresh PostgreSQL 16 container; it never points to an existing
|
||||
service/database. No live credentials, customer records or publication are needed.
|
||||
|
||||
Run from the application repository with its installed locked environment and
|
||||
cached `postgres:16-alpine` Docker image:
|
||||
|
||||
```sh
|
||||
.venv/bin/python scripts/test_synthetic_pilot.py
|
||||
.venv/bin/python scripts/synthetic_pilot.py \
|
||||
--config config/synthetic-pilot.json --output /tmp/vergabe-sizing.json
|
||||
```
|
||||
|
||||
The container uses a unique name, random loopback database port and disposable
|
||||
volume; cleanup removes only that container and its volume. Gunicorn binds a
|
||||
random loopback port. Database, media and issue storage are isolated; temporary
|
||||
passwords exist only for generated local users. The local settings disable SSO;
|
||||
this measures local password login, not the invited deployment's identity flow.
|
||||
No `seed_dev` command or existing PostgreSQL volume is reused.
|
||||
|
||||
## Adjustable assumptions
|
||||
|
||||
Defaults are two users, two synchronous Gunicorn workers, 100 initial tenders,
|
||||
180 seconds of ordinary work after 30 seconds idle, and 0.2 seconds think time
|
||||
per cycle. Each cycle creates a tender, lot and task, and reads list, search and
|
||||
detail pages; it uploads then downloads one 1 MiB synthetic PDF. Thus the corpus
|
||||
grows during the run. This is an explicit provisional operation mix, not observed
|
||||
customer demand. `iterations` is a per-user safety ceiling; duration normally
|
||||
ends the run first. User/worker count, duration, idle, iteration ceiling, corpus,
|
||||
think time, document size, HTTP timeout and ordinary p95 threshold are tunable.
|
||||
The PDF bytes are a transfer fixture, not a claim of document parser fidelity.
|
||||
|
||||
Pass requires all expected operations, ordinary p95 <= 2 seconds and zero failed
|
||||
ordinary/transfer requests. Redirects are not automatically followed: rejected
|
||||
login redirects cannot count as successful data-page responses. Document transfer
|
||||
samples are reported separately. Ordinary samples include the login page/login;
|
||||
fixture creation/migration and readiness are outside the measured workload.
|
||||
|
||||
## Coverage and interpretation
|
||||
|
||||
The result retains individual status/latency/response-size records, phase-tagged
|
||||
0.5-second Gunicorn process CPU-time/RSS samples, separate interval Docker database
|
||||
stats, driver/config/application source digests, runtime versions, image identity,
|
||||
hardware and UTC boundaries. App RSS sums process resident sets and can double-count
|
||||
shared pages; CPU reflects this host and has no app quota. Database snapshots are
|
||||
separate demand, not allocated to the app by inference. A shared deployment database
|
||||
requires a baseline and attribution before using these observations for capacity.
|
||||
|
||||
Startup is readiness-checked, not timed. Idle and fixed-think-time ordinary phases
|
||||
are measured; restart and a distinct burst phase are not. Ingress/TLS/browser/SSO,
|
||||
production quotas, throttling and shared services are unmeasured. Local success does
|
||||
not establish human acceptance or prove the existing 60m production request.
|
||||
Revisit when user count, corpus, document sizes, workers, image, database, hardware
|
||||
or quota changes, and after an owner-approved target-runtime observation window.
|
||||
The existing VERGABE-WP-0019 / RAPPS-WP-0014 human acceptance remains separate.
|
||||
|
||||
## Provisional allocation calculation
|
||||
|
||||
`config/synthetic-sizing-margins.json` keeps safety factors tunable. After a
|
||||
complete passing run, derive a candidate envelope without applying it:
|
||||
|
||||
```sh
|
||||
.venv/bin/python scripts/summarize_synthetic_sizing.py /tmp/vergabe-sizing.json \
|
||||
--margins config/synthetic-sizing-margins.json --output /tmp/vergabe-sizing-envelope.json
|
||||
```
|
||||
|
||||
CPU request is measured aggregate ordinary p95 times 1.5, rounded up to 25m;
|
||||
CPU limit is observed sample peak times 1.5, rounded up to 100m. Memory request is
|
||||
peak summed RSS times 1.25 and limit times 1.5, each rounded up to 64Mi. These
|
||||
are conservative test candidates for this configuration, not measured production
|
||||
requirements. The normalized per-worker CPU value divides aggregate by configured
|
||||
workers; it is not a claim that worker load is equal. No database capacity is
|
||||
folded into the app recommendation. No resource setting is applied by this tool.
|
||||
|
||||
The configured cycle creates rather than edits tender/lot/task records. Editing,
|
||||
startup timing, restart, a distinct burst and target-platform quota/throttling
|
||||
remain coverage gaps before full pilot sizing acceptance.
|
||||
78
scripts/summarize_synthetic_sizing.py
Normal file
78
scripts/summarize_synthetic_sizing.py
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
"""Derive a provisional local-host envelope, never apply deployment resources."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def percentile(values):
|
||||
values = sorted(values)
|
||||
if not values:
|
||||
raise ValueError("measurement samples missing")
|
||||
return values[math.ceil(0.95 * len(values)) - 1]
|
||||
|
||||
|
||||
def recommend(result, margin):
|
||||
samples = result["runtime_samples"]
|
||||
cpu = []
|
||||
memory = []
|
||||
for before, after in zip(samples, samples[1:]):
|
||||
if before.get("phase") != "ordinary" or after.get("phase") != "ordinary":
|
||||
continue
|
||||
if before.get("pids") != after.get("pids"):
|
||||
continue
|
||||
elapsed = after["monotonic_seconds"] - before["monotonic_seconds"]
|
||||
used = after["cpu_seconds"] - before["cpu_seconds"]
|
||||
if elapsed > 0 and used >= 0:
|
||||
cpu.append(used / elapsed * 1000)
|
||||
memory.append(after["rss_bytes"] / 1024**2)
|
||||
if not cpu or not memory or not result["complete"] or not result["provisional_pass"]:
|
||||
raise ValueError("incomplete or failed workload cannot establish a sizing envelope")
|
||||
cpu_p95, cpu_peak, memory_peak = percentile(cpu), max(cpu), max(memory)
|
||||
|
||||
def rounded(value, quantum):
|
||||
return math.ceil(value / quantum) * quantum
|
||||
|
||||
return {
|
||||
"status": "provisional-local-host-only",
|
||||
"application_observed": {
|
||||
"cpu_p95_m": cpu_p95,
|
||||
"cpu_peak_m": cpu_peak,
|
||||
"rss_sum_peak_mib": memory_peak,
|
||||
"sample_intervals": len(cpu),
|
||||
"normalized_cpu_p95_m_per_worker": cpu_p95 / result["config"]["workers"],
|
||||
},
|
||||
"suggested_application": {
|
||||
"cpu_request_m": rounded(cpu_p95 * margin["cpu_request_multiplier"], 25),
|
||||
"cpu_limit_m": rounded(cpu_peak * margin["cpu_limit_multiplier"], 100),
|
||||
"memory_request_mib": rounded(memory_peak * margin["memory_request_multiplier"], 64),
|
||||
"memory_limit_mib": rounded(memory_peak * margin["memory_limit_multiplier"], 64),
|
||||
},
|
||||
"margins": margin,
|
||||
"limits": [
|
||||
"Per-worker number is normalized aggregate, not an independently sampled worker.",
|
||||
"RSS sum double-counts shared pages; conservative upper bound, not unique memory.",
|
||||
"Observed local CPU is hardware/think-time dependent; quota effects untested.",
|
||||
"Database/container demand is separate and not included in application request.",
|
||||
"No production apply or human acceptance; validate target runtime before right-sizing.",
|
||||
],
|
||||
"revisit": "Changed concurrency, corpus, document size, workers, image, hardware or quota; "
|
||||
"then repeat under approved target runtime including bursts and restart.",
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("result", type=Path)
|
||||
parser.add_argument("--margins", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
margins = json.loads(args.margins.read_text())
|
||||
if any(
|
||||
not isinstance(v, (int, float)) or not math.isfinite(v) or v < 1 for v in margins.values()
|
||||
):
|
||||
raise ValueError("all margins must be finite and at least one")
|
||||
args.output.write_text(
|
||||
json.dumps(recommend(json.loads(args.result.read_text()), margins), indent=2) + "\n"
|
||||
)
|
||||
479
scripts/synthetic_pilot.py
Normal file
479
scripts/synthetic_pilot.py
Normal file
|
|
@ -0,0 +1,479 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Provisional loopback HTTP workload using Gunicorn and disposable PostgreSQL."""
|
||||
|
||||
import argparse
|
||||
import atexit
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import http.cookiejar
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def summarize(rows):
|
||||
durations = sorted(row["seconds"] for row in rows)
|
||||
return {
|
||||
"requests": len(rows),
|
||||
"errors": sum(not r["ok"] for r in rows),
|
||||
"p95_seconds": durations[math.ceil(len(durations) * 0.95) - 1] if durations else None,
|
||||
}
|
||||
|
||||
|
||||
def validate(config):
|
||||
for key in (
|
||||
"users",
|
||||
"iterations",
|
||||
"tenders",
|
||||
"document_bytes",
|
||||
"workers",
|
||||
"timeout_seconds",
|
||||
):
|
||||
if type(config[key]) is not int or config[key] < 1:
|
||||
raise ValueError(f"{key} must be a positive integer")
|
||||
for key in (
|
||||
"think_seconds",
|
||||
"ordinary_p95_seconds",
|
||||
"duration_seconds",
|
||||
"idle_seconds",
|
||||
):
|
||||
if not isinstance(config[key], (int, float)) or not math.isfinite(config[key]):
|
||||
raise ValueError(f"{key} must be finite")
|
||||
if config["think_seconds"] < 0 or config["ordinary_p95_seconds"] <= 0:
|
||||
raise ValueError("invalid timing parameter")
|
||||
if config["duration_seconds"] <= 0 or config["idle_seconds"] < 0:
|
||||
raise ValueError("duration must be positive and idle nonnegative")
|
||||
if config["document_bytes"] < 8:
|
||||
raise ValueError("document_bytes must be at least 8")
|
||||
return config
|
||||
|
||||
|
||||
def run(config, root):
|
||||
# Never consume a deployment database, media tree, session or credential.
|
||||
os.environ["DATABASE_URL"] = os.environ["SYNTHETIC_DATABASE_URL"]
|
||||
os.environ["SECRET_KEY"] = "synthetic-disposable-not-a-deployment-key"
|
||||
os.environ["DJANGO_SETTINGS_MODULE"] = "synthetic_settings"
|
||||
(root / "synthetic_settings.py").write_text(
|
||||
"from vergabe_teilnahme.settings.base import *\n"
|
||||
'DEBUG = False\nALLOWED_HOSTS = ["127.0.0.1"]\nNETKINGDOM_ENABLED = False\n'
|
||||
f"MEDIA_ROOT = {str(root / 'media')!r}\n"
|
||||
f"ISSUE_FACADE_LOCAL_DB = {str(root / 'issues.db')!r}\n"
|
||||
'ISSUE_FACADE_GITEA = None\nFORCE_SCRIPT_NAME = None\nMEDIA_URL = "/media/"\n'
|
||||
'EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"\n'
|
||||
"SESSION_COOKIE_SECURE = False\nCSRF_COOKIE_SECURE = False\n"
|
||||
)
|
||||
sys.path.insert(0, str(root))
|
||||
import django
|
||||
|
||||
django.setup()
|
||||
from django.core.management import call_command
|
||||
from django.db import connections
|
||||
|
||||
from vergabe_teilnahme.apps.accounts.models import Mitarbeiter
|
||||
from vergabe_teilnahme.apps.ausschreibungen.models import Ausschreibung
|
||||
from vergabe_teilnahme.apps.dokumente.models import Dokument
|
||||
|
||||
call_command("migrate", verbosity=0)
|
||||
for i in range(config["tenders"]):
|
||||
Ausschreibung.objects.create(titel=f"Synthetic tender {i}", ausschreiber="Synthetic")
|
||||
users = []
|
||||
password = uuid.uuid4().hex
|
||||
for i in range(config["users"]):
|
||||
user = Mitarbeiter.objects.create_user(username=f"synthetic-{i}", password=password)
|
||||
users.append(user.username)
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
base_url = f"http://127.0.0.1:{port}"
|
||||
env = dict(os.environ, PYTHONPATH=str(root) + os.pathsep + str(Path.cwd()))
|
||||
server_log = (root / "server.log").open("w")
|
||||
server = subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"gunicorn",
|
||||
"vergabe_teilnahme.wsgi:application",
|
||||
"--bind",
|
||||
f"127.0.0.1:{port}",
|
||||
"--workers",
|
||||
str(config["workers"]),
|
||||
"--timeout",
|
||||
"30",
|
||||
],
|
||||
env=env,
|
||||
stdout=server_log,
|
||||
stderr=server_log,
|
||||
)
|
||||
atexit.register(lambda: server.terminate() if server.poll() is None else None)
|
||||
for _ in range(100):
|
||||
try:
|
||||
with urllib.request.urlopen(base_url + "/health/", timeout=1) as response:
|
||||
if response.status == 200:
|
||||
break
|
||||
except OSError:
|
||||
if server.poll() is not None:
|
||||
raise RuntimeError("Gunicorn startup failed; see isolated server log")
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
server.terminate()
|
||||
raise RuntimeError("Gunicorn readiness timeout")
|
||||
samples = []
|
||||
db_samples = []
|
||||
sampling_done = threading.Event()
|
||||
phase = {"name": "idle"}
|
||||
|
||||
def sample():
|
||||
ticks_per_second = os.sysconf("SC_CLK_TCK")
|
||||
while not sampling_done.is_set():
|
||||
pids = [server.pid]
|
||||
try:
|
||||
pids += [
|
||||
int(x)
|
||||
for x in Path(f"/proc/{server.pid}/task/{server.pid}/children")
|
||||
.read_text()
|
||||
.split()
|
||||
]
|
||||
cpu, rss = 0, 0
|
||||
for pid in pids:
|
||||
fields = Path(f"/proc/{pid}/stat").read_text().split(") ", 1)[1].split()
|
||||
cpu += (int(fields[11]) + int(fields[12])) / ticks_per_second
|
||||
rss += int(fields[21]) * os.sysconf("SC_PAGE_SIZE")
|
||||
samples.append(
|
||||
{
|
||||
"at": datetime.now(UTC).isoformat(),
|
||||
"monotonic_seconds": time.monotonic(),
|
||||
"phase": phase["name"],
|
||||
"cpu_seconds": cpu,
|
||||
"rss_bytes": rss,
|
||||
"pids": pids,
|
||||
}
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
samples.append({"at": datetime.now(UTC).isoformat(), "sample_error": True})
|
||||
sampling_done.wait(0.5)
|
||||
|
||||
def sample_db():
|
||||
while not sampling_done.is_set():
|
||||
value = subprocess.check_output(
|
||||
[
|
||||
"docker",
|
||||
"stats",
|
||||
"--no-stream",
|
||||
"--format",
|
||||
"{{json .}}",
|
||||
os.environ["SYNTHETIC_CONTAINER"],
|
||||
],
|
||||
text=True,
|
||||
)
|
||||
db_samples.append(
|
||||
{
|
||||
"at": datetime.now(UTC).isoformat(),
|
||||
"phase": phase["name"],
|
||||
"docker_stats": json.loads(value),
|
||||
}
|
||||
)
|
||||
sampling_done.wait(1)
|
||||
|
||||
sampler = threading.Thread(target=sample, daemon=True)
|
||||
db_sampler = threading.Thread(target=sample_db, daemon=True)
|
||||
sampler.start()
|
||||
db_sampler.start()
|
||||
time.sleep(config["idle_seconds"])
|
||||
phase["name"] = "ordinary"
|
||||
deadline = time.monotonic() + config["duration_seconds"]
|
||||
completed = [0] * config["users"]
|
||||
barrier = threading.Barrier(config["users"])
|
||||
|
||||
def worker(i):
|
||||
jar = http.cookiejar.CookieJar()
|
||||
|
||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, *args, **kwargs):
|
||||
return None
|
||||
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar), NoRedirect())
|
||||
rows = []
|
||||
|
||||
def request(operation, method, path, data=None, expected=200, transfer=False):
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
headers = {}
|
||||
body = None
|
||||
if method == "post":
|
||||
token = next((c.value for c in jar if c.name == "csrftoken"), "")
|
||||
fields = dict(data or {}, csrfmiddlewaretoken=token)
|
||||
if transfer:
|
||||
boundary = "synthetic-" + uuid.uuid4().hex
|
||||
body = b""
|
||||
for name, value in fields.items():
|
||||
body += (f'--{boundary}\r\nContent-Disposition: form-data; '
|
||||
f'name="{name}"\r\n\r\n{value}\r\n').encode()
|
||||
body += (
|
||||
(f'--{boundary}\r\nContent-Disposition: form-data; '
|
||||
'name="datei"; filename="synthetic.pdf"\r\n'
|
||||
'Content-Type: application/pdf\r\n\r\n').encode()
|
||||
+ b"%PDF-1.4"
|
||||
+ b"x" * (config["document_bytes"] - 8)
|
||||
+ f"\r\n--{boundary}--\r\n".encode()
|
||||
)
|
||||
headers["Content-Type"] = f"multipart/form-data; boundary={boundary}"
|
||||
else:
|
||||
body = urllib.parse.urlencode(fields).encode()
|
||||
headers["Referer"] = base_url + path
|
||||
req = urllib.request.Request(
|
||||
base_url + path, data=body, headers=headers, method=method.upper()
|
||||
)
|
||||
try:
|
||||
response = opener.open(req, timeout=config["timeout_seconds"])
|
||||
except urllib.error.HTTPError as exc:
|
||||
response = exc
|
||||
received = len(response.read())
|
||||
status = response.status
|
||||
ok = status == expected
|
||||
response.close()
|
||||
except Exception:
|
||||
status, ok, received = None, False, 0
|
||||
rows.append(
|
||||
{
|
||||
"user": i,
|
||||
"operation": operation,
|
||||
"transfer": transfer,
|
||||
"seconds": time.perf_counter() - started,
|
||||
"status": status,
|
||||
"ok": ok,
|
||||
"response_bytes": received,
|
||||
}
|
||||
)
|
||||
return ok
|
||||
|
||||
request("login-form", "get", "/accounts/login/")
|
||||
barrier.wait(timeout=30)
|
||||
request(
|
||||
"login",
|
||||
"post",
|
||||
"/accounts/login/",
|
||||
{"username": users[i], "password": password},
|
||||
302,
|
||||
)
|
||||
for n in range(config["iterations"]):
|
||||
if time.monotonic() >= deadline:
|
||||
break
|
||||
request("list", "get", "/ausschreibungen/")
|
||||
request("search", "get", "/ausschreibungen/?q=Synthetic")
|
||||
title = f"Workload-{i}-{n}"
|
||||
created = request(
|
||||
"tender-create",
|
||||
"post",
|
||||
"/ausschreibungen/neu/",
|
||||
{"titel": title, "ausschreiber": "Synthetic"},
|
||||
302,
|
||||
)
|
||||
tender = Ausschreibung.objects.filter(titel=title).first() if created else None
|
||||
if tender:
|
||||
base = f"/ausschreibungen/{tender.pk}/"
|
||||
request("detail", "get", base)
|
||||
request(
|
||||
"lot-create",
|
||||
"post",
|
||||
base + "lose/neu/",
|
||||
{"losnummer": "1", "lostitel": "Synthetic lot"},
|
||||
302,
|
||||
)
|
||||
request(
|
||||
"task-create",
|
||||
"post",
|
||||
base + "aufgaben/neu/",
|
||||
{"titel": "Synthetic task", "typ": "fachlich", "prioritaet": 2},
|
||||
302,
|
||||
)
|
||||
request(
|
||||
"document-upload",
|
||||
"post",
|
||||
base + "dokumente/hochladen/",
|
||||
{"kategorie": "intern", "version": "1.0"},
|
||||
302,
|
||||
True,
|
||||
)
|
||||
document = Dokument.objects.filter(ausschreibung=tender).first()
|
||||
if document:
|
||||
request("document-download", "get", document.datei.url, transfer=True)
|
||||
completed[i] += 1
|
||||
time.sleep(config["think_seconds"])
|
||||
connections.close_all()
|
||||
return rows
|
||||
|
||||
started = datetime.now(UTC).isoformat()
|
||||
before = time.perf_counter()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=config["users"]) as pool:
|
||||
rows = [r for group in pool.map(worker, range(config["users"])) for r in group]
|
||||
ordinary = summarize([r for r in rows if not r["transfer"]])
|
||||
transfers = summarize([r for r in rows if r["transfer"]])
|
||||
complete = all(completed) and len(rows) == sum(completed) * 8 + config["users"] * 2
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"started_at": started,
|
||||
"finished_at": datetime.now(UTC).isoformat(),
|
||||
"duration_seconds": time.perf_counter() - before,
|
||||
"config": config,
|
||||
"environment": {
|
||||
"python": platform.python_version(),
|
||||
"django": django.get_version(),
|
||||
"platform": platform.platform(),
|
||||
"database": "disposable PostgreSQL 16 container",
|
||||
"transport": "loopback HTTP Gunicorn",
|
||||
"workers": config["workers"],
|
||||
},
|
||||
"ordinary": ordinary,
|
||||
"document_transfer": transfers,
|
||||
"complete": complete,
|
||||
"provisional_pass": complete
|
||||
and ordinary["errors"] == 0
|
||||
and transfers["errors"] == 0
|
||||
and ordinary["p95_seconds"] <= config["ordinary_p95_seconds"],
|
||||
"limitations": [
|
||||
"No browser, ingress, TLS, SSO or application container quota",
|
||||
"No CPU/memory allocation recommendation or human acceptance",
|
||||
"No timed startup/restart or distinct burst; configured idle and ordinary phases only",
|
||||
],
|
||||
"operations": rows,
|
||||
"runtime_samples": samples,
|
||||
"database_samples": db_samples,
|
||||
"completed_iterations": completed,
|
||||
}
|
||||
sampling_done.set()
|
||||
sampler.join()
|
||||
db_sampler.join()
|
||||
server.terminate()
|
||||
server.wait(timeout=10)
|
||||
server_log.close()
|
||||
connections.close_all()
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--config", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
config = validate(json.loads(args.config.read_text()))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
name = "vergabe-synthetic-" + uuid.uuid4().hex[:12]
|
||||
# Cached image only, loopback random host port, no existing volume or database.
|
||||
subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"run",
|
||||
"--detach",
|
||||
"--pull=never",
|
||||
"--name",
|
||||
name,
|
||||
"--publish",
|
||||
"127.0.0.1::5432",
|
||||
"--env",
|
||||
"POSTGRES_DB=synthetic",
|
||||
"--env",
|
||||
"POSTGRES_USER=synthetic",
|
||||
"--env",
|
||||
"POSTGRES_PASSWORD=synthetic-local-only",
|
||||
"postgres:16-alpine",
|
||||
],
|
||||
check=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
for _ in range(100):
|
||||
ready = subprocess.run(
|
||||
["docker", "exec", name, "pg_isready", "-U", "synthetic"],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
if ready.returncode == 0:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
raise RuntimeError("isolated PostgreSQL startup failed")
|
||||
port = (
|
||||
subprocess.check_output(["docker", "port", name, "5432/tcp"], text=True)
|
||||
.strip()
|
||||
.split(":")[-1]
|
||||
)
|
||||
os.environ["SYNTHETIC_CONTAINER"] = name
|
||||
os.environ["SYNTHETIC_DATABASE_URL"] = (
|
||||
f"postgres://synthetic:synthetic-local-only@127.0.0.1:{port}/synthetic"
|
||||
)
|
||||
with tempfile.TemporaryDirectory(prefix="vergabe-synthetic-") as directory:
|
||||
result = run(config, Path(directory))
|
||||
result["postgres_image_id"] = subprocess.check_output(
|
||||
["docker", "inspect", "--format", "{{.Image}}", name], text=True
|
||||
).strip()
|
||||
result["postgres_final_stats"] = json.loads(
|
||||
subprocess.check_output(
|
||||
["docker", "stats", "--no-stream", "--format", "{{json .}}", name],
|
||||
text=True,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
subprocess.run(
|
||||
["docker", "rm", "--force", "--volumes", name],
|
||||
check=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
)
|
||||
result["source_revision"] = subprocess.check_output(
|
||||
["git", "rev-parse", "HEAD"], text=True
|
||||
).strip()
|
||||
result["driver_sha256"] = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
|
||||
app_files = sorted(Path("vergabe_teilnahme").rglob("*.py")) + [
|
||||
Path("uv.lock"),
|
||||
Path("pyproject.toml"),
|
||||
]
|
||||
result["app_source_sha256"] = hashlib.sha256(
|
||||
b"".join(str(p).encode() + b"\0" + p.read_bytes() for p in app_files)
|
||||
).hexdigest()
|
||||
result["host"] = {
|
||||
"logical_cpus": os.cpu_count(),
|
||||
"affinity_cpus": len(os.sched_getaffinity(0)),
|
||||
"cpu_model": next(
|
||||
(
|
||||
x.split(":", 1)[1].strip()
|
||||
for x in Path("/proc/cpuinfo").read_text().splitlines()
|
||||
if x.startswith("model name")
|
||||
),
|
||||
"unknown",
|
||||
),
|
||||
"memory_total": next(
|
||||
x for x in Path("/proc/meminfo").read_text().splitlines() if x.startswith("MemTotal:")
|
||||
),
|
||||
"cgroup_cpu_max": Path("/sys/fs/cgroup/cpu.max").read_text().strip()
|
||||
if Path("/sys/fs/cgroup/cpu.max").exists()
|
||||
else "unavailable",
|
||||
"app_cpu_quota": "not imposed by harness",
|
||||
}
|
||||
result["config_sha256"] = hashlib.sha256(args.config.read_bytes()).hexdigest()
|
||||
args.output.write_text(json.dumps(result, indent=2) + "\n")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
k: result[k]
|
||||
for k in (
|
||||
"ordinary",
|
||||
"document_transfer",
|
||||
"complete",
|
||||
"provisional_pass",
|
||||
)
|
||||
}
|
||||
)
|
||||
)
|
||||
raise SystemExit(0 if result["provisional_pass"] else 1)
|
||||
98
scripts/test_synthetic_pilot.py
Normal file
98
scripts/test_synthetic_pilot.py
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"pilot", Path(__file__).with_name("synthetic_pilot.py")
|
||||
)
|
||||
pilot = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(pilot)
|
||||
|
||||
|
||||
class SyntheticContractTests(unittest.TestCase):
|
||||
def config(self):
|
||||
return dict(
|
||||
users=2,
|
||||
workers=2,
|
||||
iterations=20,
|
||||
tenders=100,
|
||||
document_bytes=1024,
|
||||
think_seconds=0.2,
|
||||
timeout_seconds=30,
|
||||
ordinary_p95_seconds=2,
|
||||
duration_seconds=180,
|
||||
idle_seconds=30,
|
||||
)
|
||||
|
||||
def test_invalid_workload_fails_before_runtime(self):
|
||||
for key, value in [
|
||||
("users", 0),
|
||||
("users", True),
|
||||
("workers", -1),
|
||||
("iterations", 1.2),
|
||||
("document_bytes", 7),
|
||||
("timeout_seconds", 0),
|
||||
("duration_seconds", 0),
|
||||
("idle_seconds", -1),
|
||||
("ordinary_p95_seconds", float("nan")),
|
||||
("think_seconds", -1),
|
||||
]:
|
||||
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
|
||||
pilot.validate(self.config() | {key: value})
|
||||
|
||||
def test_nearest_rank_p95_and_errors_include_failed_requests(self):
|
||||
rows = [{"seconds": n, "ok": n != 20} for n in range(1, 21)]
|
||||
self.assertEqual(pilot.summarize(rows), {"requests": 20, "errors": 1, "p95_seconds": 19})
|
||||
|
||||
def test_empty_measurement_is_unknown(self):
|
||||
self.assertIsNone(pilot.summarize([])["p95_seconds"])
|
||||
|
||||
def test_default_is_two_users_and_two_seconds(self):
|
||||
self.assertEqual(pilot.validate(self.config())["users"], 2)
|
||||
self.assertEqual(self.config()["ordinary_p95_seconds"], 2)
|
||||
|
||||
|
||||
class SizingTests(unittest.TestCase):
|
||||
def test_failed_measurement_does_not_recommend(self):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"sizing", Path(__file__).with_name("summarize_synthetic_sizing.py")
|
||||
)
|
||||
sizing = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(sizing)
|
||||
with self.assertRaises(ValueError):
|
||||
sizing.recommend({"runtime_samples": [], "complete": False}, {})
|
||||
|
||||
def test_cpu_delta_and_rounding(self):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"sizing", Path(__file__).with_name("summarize_synthetic_sizing.py")
|
||||
)
|
||||
sizing = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(sizing)
|
||||
rows = [
|
||||
dict(
|
||||
phase="ordinary",
|
||||
pids=[1],
|
||||
monotonic_seconds=i,
|
||||
cpu_seconds=i * 0.1,
|
||||
rss_bytes=100 * 1024**2,
|
||||
)
|
||||
for i in range(3)
|
||||
]
|
||||
margin = dict(
|
||||
cpu_request_multiplier=1.5,
|
||||
cpu_limit_multiplier=1.5,
|
||||
memory_request_multiplier=1.25,
|
||||
memory_limit_multiplier=1.5,
|
||||
)
|
||||
result = sizing.recommend(
|
||||
dict(runtime_samples=rows, complete=True, provisional_pass=True, config={"workers": 2}),
|
||||
margin,
|
||||
)
|
||||
self.assertEqual(
|
||||
result["suggested_application"],
|
||||
dict(cpu_request_m=150, cpu_limit_m=200, memory_request_mib=128, memory_limit_mib=192),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue