tegwick
b7cfe087b2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 276069@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c
71 lines
6.1 KiB
Markdown
71 lines
6.1 KiB
Markdown
---
|
|
id: ACTIVITY-WP-0049
|
|
type: workplan
|
|
title: "Wave-2 resource right-sizing from the 2026-10-21 allocation review"
|
|
domain: infotech
|
|
repo: activity-core
|
|
status: ready
|
|
flavor: implementation
|
|
owner: claude
|
|
topic_slug: activity-core
|
|
created: "2026-10-21"
|
|
updated: "2026-10-21"
|
|
related:
|
|
- CUST-WP-0071
|
|
- RESOURCE-WP-0008
|
|
state_hub_workstream_id: "de34ab8e-3f4c-514b-82b6-671588362a6c"
|
|
---
|
|
|
|
# ACTIVITY-WP-0049 — Wave-2 resource right-sizing from the 2026-10-21 allocation review
|
|
|
|
Receiving workplan in `activity-core` for the wave-2 request changes below.
|
|
|
|
Origin: the 2026-10-21 node allocation review (`RESOURCE-WP-0008-T03`, feeding `CUST-WP-0071-T03`). Per the founder's 2026-10-21 delivery interview (`the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`) the small adjustments form wave 2: one receiving task per owner repo, applied later in a batch. Applying wave 2 needs a separate founder approval, so every cluster-write task here is a human-gated `wait`. This file changes no manifest.
|
|
|
|
## Apply the reviewed request changes
|
|
|
|
```task
|
|
id: ACTIVITY-WP-0049-T01
|
|
status: wait
|
|
blocking_reason: "ArgoCD part (api/worker/event-router) needs an exception to the image-only promotion check; infra, relay and llm-connect applied."
|
|
priority: medium
|
|
needs_human: true
|
|
state_hub_task_id: "588ffa46-5368-5479-a2f2-df8419245a77"
|
|
```
|
|
|
|
Values are pod-aggregate figures from the budget file (7-day window ending 2026-10-21T06:08Z). Limits stay unchanged unless stated.
|
|
|
|
| Workload (ns/name) | Pods | CPU request | CPU limit | CPU p95/peak 7d (m) | Memory request | Memory limit | Mem p95/peak (Mi) | Throttle | Action |
|
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
|
| `activity-core/activity-core` | 10 | 100m → **100m** | 1000m (8 container(s) without CPU limit) | 62.8/132.5 | 256Mi → **944Mi** | 1024Mi | 811.0/1011.0 | 0.0 | keep + memory up |
|
|
|
|
Source and container detail:
|
|
|
|
- `activity-core/activity-core`: ten pods in `k8s/railiance/10-infrastructure.yaml` (StatefulSets `actcore-app-db`, `actcore-temporal-db` with 50m/256Mi each, `actcore-nats`; Deployments `actcore-temporal`, `actcore-temporal-ui`) and `k8s/railiance/20-runtime.yaml`. CPU stays at 100m (within tolerance). Memory p95 811Mi vs 256Mi requested: raise the requests to a 944Mi total, distributed by each pod's p95. Eight containers have no CPU request or limit; give each at least the 5m floor.
|
|
|
|
Evidence: resource-control `docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.md` (+ `.json`, commit `3d9b1bd`). Rule: CPU = max(floor, p95x1.5, peakx0.5) rounded up to 5m; memory request = 7d p95 working set x1.15 (only when the change is at least 64Mi).
|
|
|
|
Steps:
|
|
|
|
1. Map the pod-aggregate proposal onto the named container(s) in the source file; where a workload has several containers or pods, split by the per-pod figures in `2026-10-21-workload-usage.json` and keep the sum at the proposed value.
|
|
2. Prepare the source change as a reviewed commit (no push, no sync) and record the before values here.
|
|
3. Apply only inside the batched wave-2 rollout, after founder approval, through the owner's normal deploy path (ArgoCD sync or documented helm/kubectl apply).
|
|
|
|
Verification: `kubectl -n <ns> get pod -o jsonpath='{..resources}'` shows the new requests; pods Ready with no new restarts or OOMKilled over 24h; for CPU decreases the throttle ratio and request latency stay at or below the 7d baseline; node request total drops by the expected delta.
|
|
|
|
Rollback: revert the source commit and re-sync/re-apply; the before values in the table above are the rollback target. Keep before/after `kubectl get -o yaml` captures as the receipt.
|
|
|
|
## Prepared 2026-10-22 (wave-2 worker W1b)
|
|
|
|
Status stays `wait` / `needs_human`. Local commit only: not pushed, not applied. Diff = full owner render vs live (`kubectl diff`, object names and changed key paths only, Secret objects skipped).
|
|
|
|
- Totals kept at the proposal, 100m CPU / 944Mi memory over the 10 live pods (requests only, no new limits): api 5m/96Mi, worker 5m/176Mi, event-router 5m/80Mi, statehub-edge-relay 50m/128Mi -> 10m/96Mi, temporal 35m/240Mi, temporal-ui 5m/16Mi, nats 5m/16Mi, app-db 10m/48Mi, temporal-db 15m/128Mi, llm-connect 50m/128Mi -> 5m/48Mi.
|
|
- Commits: activity-core `d3cd1a2` (`k8s/gitops/runtime.yaml`, `k8s/railiance/10-infrastructure.yaml`, `k8s/railiance/20-runtime.yaml`) and llm-connect `19006db` (`deploy/k8s/activity-core-llm-connect/deployment.yaml`).
|
|
- Diff: resource fields only for all objects. The local diff of the gitops render also drops the ArgoCD tracking-id annotation, which is expected. Existing drift: the source already declared app-db/temporal-db 50m/256Mi requests and a 1Gi memory limit that were never applied. Applying 10-infrastructure therefore also adds that owner-declared 1Gi memory limit to both DBs, which the founder should confirm.
|
|
- Restart: yes. app-db and temporal-db (single-replica postgres StatefulSets), nats, temporal, temporal-ui, relay, api, worker, event-router and llm-connect all roll.
|
|
- Gates: (1) api/worker/event-router are ArgoCD-managed with `targetRevision` pinned in `railiance-platform/argocd/railiance01/applications/activity-core.application.yaml`, so after push the pin needs a bump. That repo is owned by another worker. (2) `scripts/check_gitops_promotion.py` is an image-only admission policy and refuses a resources-only gitops change. The founder needs to accept the exception.
|
|
- Apply: `kubectl apply -f k8s/railiance/10-infrastructure.yaml`; apply only the relay Deployment from `20-runtime.yaml`, not the whole file, because it holds immutable Jobs and ArgoCD-owned Deployments; `kubectl apply -k ~/llm-connect/deploy/k8s/activity-core-llm-connect`; then bump the ArgoCD pin and sync `activity-core`.
|
|
|
|
## Applied — 2026-10-22 (ACTIVITY-WP-0049-T01)
|
|
|
|
Partial: the infrastructure is applied (app-db and temporal-db restarted, including the already-declared 1Gi memory limits), and so are the relay and llm-connect. All are Ready. The ArgoCD part (api, worker, event-router) waits: `scripts/check_gitops_promotion.py` admits image changes only, so it needs an owner exception or a promotion-check change, then a pin bump in railiance-platform. Evidence: the-custodian docs/evidence/2026-10-22-wave2-apply.md.
|