Wave-2 right-sizing applied 2026-10-22 (CUST-WP-0071)
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Reject generated-only commits / generated-commit-guard (push) Successful in 11s
Python Tests / pytest (push) Failing after 28s

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
codex 2026-10-22 01:05:32 +02:00
parent eda153c7a1
commit c0227108d2
3 changed files with 107 additions and 0 deletions

View file

@ -91,3 +91,18 @@ completion of T04.
- Policy: raise the CPU limits of the throttled sso and playwright workloads, and
remove the retired inter-hub release. rein-aharness is NOT scaled to zero; it
only gets the lower request.
## Wave 2 apply scope (after worker preparation)
- Drift approved to go in together with resources:
- the Vergabe `railiance.io/release-unit` label;
- the email-connect annotation and volume reordering;
- the rein-aharness labels (Deployment only, no smoke Job);
- the cert-manager `--acme-http01-solver-nameservers=1.1.1.1:53` flag, now
written into the values.
- activity-core: infrastructure (including its already-declared 1Gi DB memory
limits), relay and llm-connect now. Its ArgoCD part (api, worker, event-router)
waits, because the promotion check admits image changes only.
- flex-auth: only `flex-auth-ops-warden` now. The other 8 releases wait for owner
review of the `--caller-audience` auth change and the image change.
- target-revenue: allowed tonight; its sync re-runs the migrate and bootstrap hooks.

View file

@ -0,0 +1,74 @@
# CUST-WP-0071 wave 2 — apply evidence (2026-10-21/22, 22:00–22:50 UTC)
**Authority:** `docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`
(sections "Wave 2" and "Wave 2 apply scope").
**Method:**
- Four worker agents prepared the owner changes in disjoint repo sets.
- Each change was diffed against live as a full render, showing changed key paths
only, with no Secret values.
- Applies went one owner at a time through the owner deploy path. Where an owner's
`make` target goes through the broken ssh hop, the equivalent local command was
used.
- Databases were restarted one at a time, each followed by a health check.
## Node CPU requests (of 4000m allocatable)
| | CPU requested | Memory requested |
|---|---:|---:|
| Before wave 2 | 3840m (96%) | 12496Mi (78%) |
| After wave 2 | **2755m (68%)** | 12356Mi (77%) |
Afterwards, nothing was Pending, every Deployment and StatefulSet was Ready, and all
50 ExternalSecrets reported SecretSynced. Argo CD apps were Synced and Healthy,
except bao-notice and informed-decision, which were already OutOfSync before wave 2.
## Applied
- **Stateless applications:**
- core-hub (Helm 34), coulomb-social (Helm 17) and reuse-surface (Helm 12).
- issue-core: Argo CD pin 11af3cb.
- canned-prompts and sbom-nexus: runtime manifest only, so no migration Job re-ran.
- tenant-engine, audit-core (Deployment only), mail-address-semantics and rapp-qonto.
- email-connect, including the approved annotation drift.
- rein-aharness, including the approved labels; Deployment only.
- flex-auth-ops-warden (Helm 4).
- playwright: CPU limit 2 cores.
- Vergabe: memory 512Mi plus the approved release-unit label, Helm 4.
- inter-hub: release uninstalled.
- **Identity stack:**
- lldap CPU limit raised from 200m to 1 core; it had been throttled 54%.
- authelia, keycape, identity-provisioner and privacyIDEA right-sized.
- Checks: KeyCape discovery returns 200, and the forge sign-in redirect works.
- **Platform:**
- Argo CD controller and repo-server.
- ESO (Helm 9) and the openbao UI gateway.
- cert-manager (Helm 2): the ACME solver nameserver flag is now written into values.
- traefik, through its HelmChartConfig; ingress returned 200 throughout.
- **Databases** (single-instance CNPG; each restart confirmed with the new requests
and Cluster in healthy state):
| Cluster | Restarted (UTC) | Notes |
|---|---|---|
| platform-pg-2 | — | — |
| platform-pg | — | dependents logged connection refusals for about 1 min after "healthy", then recovered |
| user-engine-pg | 22:33 | — |
| forgejo-db | 22:37 | the forge stayed reachable |
| net-kingdom-pg | 22:40 | — |
| target-revenue-pg | 22:42 | the first Argo sync stopped mid-restart; a second sync completed |
- **activity-core:** the infrastructure, including the app-db and temporal-db
StatefulSets and their declared 1Gi limits, the relay and llm-connect.
## Not applied (open, with owner records)
- **flex-auth:** 8 of 9 releases. Chart HEAD would add `--caller-audience` and change
sitting's image; that needs owner review (FLEX-WP-0038-T01).
- **activity-core api, worker and event-router:** the promotion check admits image
changes only, so this needs an exception (ACTIVITY-WP-0049-T01).
- **CNPG operator (100m→50m):** it has no source file (RAPP-POSTGRES-WP-0008-T01).
- **k3s addons** (coredns, metrics-server, local-path-provisioner): founder-run with
sudo on railiance01 (RCLUSTER-WP-0020-T03). kube-reserved and system-reserved
follow only after that and a measurement.
- **Namespaces `coulomb` and `inter-hub`:** the founder checks leftover credential
objects before deleting them.

View file

@ -617,3 +617,21 @@ After the founder's redacted diff review, the Forgejo server Helm upgrade ran
server requests 200m/1104Mi. **Wave 1 is complete.** The node is at 3840m CPU (96%)
and 78% memory, so wave 2 is now the precondition for any further increase or for
routine state-hub release surges.
## Wave 2 applied — 2026-10-22
Evidence: `docs/evidence/2026-10-22-wave2-apply.md`. Node CPU requests went from
3840m (96%) to **2755m (68%)**, about 1.2 cores of headroom. Routine state-hub
release surges fit again (STATE-WP-0099-T06).
18 owner receiving workplans are finished. Four items remain open with owner
records:
- flex-auth: 8 releases;
- the activity-core gitops part;
- the CNPG operator;
- the founder-run k3s addons.
The CPU limits of the throttled lldap (1 core) and playwright (2 cores) were raised.
inter-hub was removed. Remaining for T03/T04: re-measure over a representative
window (RAPP-TELEMETRY-WP-0002-T05 after 7 days) before T04 closes. T05 activation
follows.