Wave-2 right-sizing applied 2026-10-22 (CUST-WP-0071)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
eda153c7a1
commit
c0227108d2
3 changed files with 107 additions and 0 deletions
|
|
@ -91,3 +91,18 @@ completion of T04.
|
|||
- Policy: raise the CPU limits of the throttled sso and playwright workloads, and
|
||||
remove the retired inter-hub release. rein-aharness is NOT scaled to zero; it
|
||||
only gets the lower request.
|
||||
|
||||
## Wave 2 apply scope (after worker preparation)
|
||||
|
||||
- Drift approved to go in together with resources:
|
||||
- the Vergabe `railiance.io/release-unit` label;
|
||||
- the email-connect annotation and volume reordering;
|
||||
- the rein-aharness labels (Deployment only, no smoke Job);
|
||||
- the cert-manager `--acme-http01-solver-nameservers=1.1.1.1:53` flag, now
|
||||
written into the values.
|
||||
- activity-core: infrastructure (including its already-declared 1Gi DB memory
|
||||
limits), relay and llm-connect now. Its ArgoCD part (api, worker, event-router)
|
||||
waits, because the promotion check admits image changes only.
|
||||
- flex-auth: only `flex-auth-ops-warden` now. The other 8 releases wait for owner
|
||||
review of the `--caller-audience` auth change and the image change.
|
||||
- target-revenue: allowed tonight; its sync re-runs the migrate and bootstrap hooks.
|
||||
|
|
|
|||
74
docs/evidence/2026-10-22-wave2-apply.md
Normal file
74
docs/evidence/2026-10-22-wave2-apply.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# CUST-WP-0071 wave 2 — apply evidence (2026-10-21/22, 22:00–22:50 UTC)
|
||||
|
||||
**Authority:** `docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`
|
||||
(sections "Wave 2" and "Wave 2 apply scope").
|
||||
|
||||
**Method:**
|
||||
- Four worker agents prepared the owner changes in disjoint repo sets.
|
||||
- Each change was diffed against live as a full render, showing changed key paths
|
||||
only, with no Secret values.
|
||||
- Applies went one owner at a time through the owner deploy path. Where an owner's
|
||||
`make` target goes through the broken ssh hop, the equivalent local command was
|
||||
used.
|
||||
- Databases were restarted one at a time, each followed by a health check.
|
||||
|
||||
## Node CPU requests (of 4000m allocatable)
|
||||
|
||||
| | CPU requested | Memory requested |
|
||||
|---|---:|---:|
|
||||
| Before wave 2 | 3840m (96%) | 12496Mi (78%) |
|
||||
| After wave 2 | **2755m (68%)** | 12356Mi (77%) |
|
||||
|
||||
Afterwards, nothing was Pending, every Deployment and StatefulSet was Ready, and all
|
||||
50 ExternalSecrets reported SecretSynced. Argo CD apps were Synced and Healthy,
|
||||
except bao-notice and informed-decision, which were already OutOfSync before wave 2.
|
||||
|
||||
## Applied
|
||||
|
||||
- **Stateless applications:**
|
||||
- core-hub (Helm 34), coulomb-social (Helm 17) and reuse-surface (Helm 12).
|
||||
- issue-core: Argo CD pin 11af3cb.
|
||||
- canned-prompts and sbom-nexus: runtime manifest only, so no migration Job re-ran.
|
||||
- tenant-engine, audit-core (Deployment only), mail-address-semantics and rapp-qonto.
|
||||
- email-connect, including the approved annotation drift.
|
||||
- rein-aharness, including the approved labels; Deployment only.
|
||||
- flex-auth-ops-warden (Helm 4).
|
||||
- playwright: CPU limit 2 cores.
|
||||
- Vergabe: memory 512Mi plus the approved release-unit label, Helm 4.
|
||||
- inter-hub: release uninstalled.
|
||||
- **Identity stack:**
|
||||
- lldap CPU limit raised from 200m to 1 core; it had been throttled 54%.
|
||||
- authelia, keycape, identity-provisioner and privacyIDEA right-sized.
|
||||
- Checks: KeyCape discovery returns 200, and the forge sign-in redirect works.
|
||||
- **Platform:**
|
||||
- Argo CD controller and repo-server.
|
||||
- ESO (Helm 9) and the openbao UI gateway.
|
||||
- cert-manager (Helm 2): the ACME solver nameserver flag is now written into values.
|
||||
- traefik, through its HelmChartConfig; ingress returned 200 throughout.
|
||||
- **Databases** (single-instance CNPG; each restart confirmed with the new requests
|
||||
and Cluster in healthy state):
|
||||
|
||||
| Cluster | Restarted (UTC) | Notes |
|
||||
|---|---|---|
|
||||
| platform-pg-2 | — | — |
|
||||
| platform-pg | — | dependents logged connection refusals for about 1 min after "healthy", then recovered |
|
||||
| user-engine-pg | 22:33 | — |
|
||||
| forgejo-db | 22:37 | the forge stayed reachable |
|
||||
| net-kingdom-pg | 22:40 | — |
|
||||
| target-revenue-pg | 22:42 | the first Argo sync stopped mid-restart; a second sync completed |
|
||||
|
||||
- **activity-core:** the infrastructure, including the app-db and temporal-db
|
||||
StatefulSets and their declared 1Gi limits, the relay and llm-connect.
|
||||
|
||||
## Not applied (open, with owner records)
|
||||
|
||||
- **flex-auth:** 8 of 9 releases. Chart HEAD would add `--caller-audience` and change
|
||||
sitting's image; that needs owner review (FLEX-WP-0038-T01).
|
||||
- **activity-core api, worker and event-router:** the promotion check admits image
|
||||
changes only, so this needs an exception (ACTIVITY-WP-0049-T01).
|
||||
- **CNPG operator (100m→50m):** it has no source file (RAPP-POSTGRES-WP-0008-T01).
|
||||
- **k3s addons** (coredns, metrics-server, local-path-provisioner): founder-run with
|
||||
sudo on railiance01 (RCLUSTER-WP-0020-T03). kube-reserved and system-reserved
|
||||
follow only after that and a measurement.
|
||||
- **Namespaces `coulomb` and `inter-hub`:** the founder checks leftover credential
|
||||
objects before deleting them.
|
||||
|
|
@ -617,3 +617,21 @@ After the founder's redacted diff review, the Forgejo server Helm upgrade ran
|
|||
server requests 200m/1104Mi. **Wave 1 is complete.** The node is at 3840m CPU (96%)
|
||||
and 78% memory, so wave 2 is now the precondition for any further increase or for
|
||||
routine state-hub release surges.
|
||||
|
||||
## Wave 2 applied — 2026-10-22
|
||||
|
||||
Evidence: `docs/evidence/2026-10-22-wave2-apply.md`. Node CPU requests went from
|
||||
3840m (96%) to **2755m (68%)**, about 1.2 cores of headroom. Routine state-hub
|
||||
release surges fit again (STATE-WP-0099-T06).
|
||||
|
||||
18 owner receiving workplans are finished. Four items remain open with owner
|
||||
records:
|
||||
- flex-auth: 8 releases;
|
||||
- the activity-core gitops part;
|
||||
- the CNPG operator;
|
||||
- the founder-run k3s addons.
|
||||
|
||||
The CPU limits of the throttled lldap (1 core) and playwright (2 cores) were raised.
|
||||
inter-hub was removed. Remaining for T03/T04: re-measure over a representative
|
||||
window (RAPP-TELEMETRY-WP-0002-T05 after 7 days) before T04 closes. T05 activation
|
||||
follows.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue