diff --git a/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md b/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md index b388cb6..f757c1b 100644 --- a/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md +++ b/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md @@ -91,3 +91,18 @@ completion of T04. - Policy: raise the CPU limits of the throttled sso and playwright workloads, and remove the retired inter-hub release. rein-aharness is NOT scaled to zero; it only gets the lower request. + +## Wave 2 apply scope (after worker preparation) + +- Drift approved to go in together with resources: + - the Vergabe `railiance.io/release-unit` label; + - the email-connect annotation and volume reordering; + - the rein-aharness labels (Deployment only, no smoke Job); + - the cert-manager `--acme-http01-solver-nameservers=1.1.1.1:53` flag, now + written into the values. +- activity-core: infrastructure (including its already-declared 1Gi DB memory + limits), relay and llm-connect now. Its ArgoCD part (api, worker, event-router) + waits, because the promotion check admits image changes only. +- flex-auth: only `flex-auth-ops-warden` now. The other 8 releases wait for owner + review of the `--caller-audience` auth change and the image change. +- target-revenue: allowed tonight; its sync re-runs the migrate and bootstrap hooks. diff --git a/docs/evidence/2026-10-22-wave2-apply.md b/docs/evidence/2026-10-22-wave2-apply.md new file mode 100644 index 0000000..a40ec64 --- /dev/null +++ b/docs/evidence/2026-10-22-wave2-apply.md @@ -0,0 +1,74 @@ +# CUST-WP-0071 wave 2 — apply evidence (2026-10-21/22, 22:00–22:50 UTC) + +**Authority:** `docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md` +(sections "Wave 2" and "Wave 2 apply scope"). + +**Method:** +- Four worker agents prepared the owner changes in disjoint repo sets. +- Each change was diffed against live as a full render, showing changed key paths + only, with no Secret values. +- Applies went one owner at a time through the owner deploy path. Where an owner's + `make` target goes through the broken ssh hop, the equivalent local command was + used. +- Databases were restarted one at a time, each followed by a health check. + +## Node CPU requests (of 4000m allocatable) + +| | CPU requested | Memory requested | +|---|---:|---:| +| Before wave 2 | 3840m (96%) | 12496Mi (78%) | +| After wave 2 | **2755m (68%)** | 12356Mi (77%) | + +Afterwards, nothing was Pending, every Deployment and StatefulSet was Ready, and all +50 ExternalSecrets reported SecretSynced. Argo CD apps were Synced and Healthy, +except bao-notice and informed-decision, which were already OutOfSync before wave 2. + +## Applied + +- **Stateless applications:** + - core-hub (Helm 34), coulomb-social (Helm 17) and reuse-surface (Helm 12). + - issue-core: Argo CD pin 11af3cb. + - canned-prompts and sbom-nexus: runtime manifest only, so no migration Job re-ran. + - tenant-engine, audit-core (Deployment only), mail-address-semantics and rapp-qonto. + - email-connect, including the approved annotation drift. + - rein-aharness, including the approved labels; Deployment only. + - flex-auth-ops-warden (Helm 4). + - playwright: CPU limit 2 cores. + - Vergabe: memory 512Mi plus the approved release-unit label, Helm 4. + - inter-hub: release uninstalled. +- **Identity stack:** + - lldap CPU limit raised from 200m to 1 core; it had been throttled 54%. + - authelia, keycape, identity-provisioner and privacyIDEA right-sized. + - Checks: KeyCape discovery returns 200, and the forge sign-in redirect works. +- **Platform:** + - Argo CD controller and repo-server. + - ESO (Helm 9) and the openbao UI gateway. + - cert-manager (Helm 2): the ACME solver nameserver flag is now written into values. + - traefik, through its HelmChartConfig; ingress returned 200 throughout. +- **Databases** (single-instance CNPG; each restart confirmed with the new requests + and Cluster in healthy state): + + | Cluster | Restarted (UTC) | Notes | + |---|---|---| + | platform-pg-2 | — | — | + | platform-pg | — | dependents logged connection refusals for about 1 min after "healthy", then recovered | + | user-engine-pg | 22:33 | — | + | forgejo-db | 22:37 | the forge stayed reachable | + | net-kingdom-pg | 22:40 | — | + | target-revenue-pg | 22:42 | the first Argo sync stopped mid-restart; a second sync completed | + +- **activity-core:** the infrastructure, including the app-db and temporal-db + StatefulSets and their declared 1Gi limits, the relay and llm-connect. + +## Not applied (open, with owner records) + +- **flex-auth:** 8 of 9 releases. Chart HEAD would add `--caller-audience` and change + sitting's image; that needs owner review (FLEX-WP-0038-T01). +- **activity-core api, worker and event-router:** the promotion check admits image + changes only, so this needs an exception (ACTIVITY-WP-0049-T01). +- **CNPG operator (100m→50m):** it has no source file (RAPP-POSTGRES-WP-0008-T01). +- **k3s addons** (coredns, metrics-server, local-path-provisioner): founder-run with + sudo on railiance01 (RCLUSTER-WP-0020-T03). kube-reserved and system-reserved + follow only after that and a measurement. +- **Namespaces `coulomb` and `inter-hub`:** the founder checks leftover credential + objects before deleting them. diff --git a/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md b/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md index d9e1e6d..3914e9a 100644 --- a/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md +++ b/workplans/CUST-WP-0071-measured-workload-sizing-and-weekly-review.md @@ -617,3 +617,21 @@ After the founder's redacted diff review, the Forgejo server Helm upgrade ran server requests 200m/1104Mi. **Wave 1 is complete.** The node is at 3840m CPU (96%) and 78% memory, so wave 2 is now the precondition for any further increase or for routine state-hub release surges. + +## Wave 2 applied — 2026-10-22 + +Evidence: `docs/evidence/2026-10-22-wave2-apply.md`. Node CPU requests went from +3840m (96%) to **2755m (68%)**, about 1.2 cores of headroom. Routine state-hub +release surges fit again (STATE-WP-0099-T06). + +18 owner receiving workplans are finished. Four items remain open with owner +records: +- flex-auth: 8 releases; +- the activity-core gitops part; +- the CNPG operator; +- the founder-run k3s addons. + +The CPU limits of the throttled lldap (1 core) and playwright (2 cores) were raised. +inter-hub was removed. Remaining for T03/T04: re-measure over a representative +window (RAPP-TELEMETRY-WP-0002-T05 after 7 days) before T04 closes. T05 activation +follows.