diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 0dc9c3b..ccf91c5 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -13,6 +13,7 @@ | workplan | FORGE-WP-0008 | proposed | — | workplans/FORGE-WP-0008-secret-detection-residual-coverage.md | | workplan | FORGE-WP-0009 | proposed | — | workplans/FORGE-WP-0009-retire-stale-gitea-deploy-assets.md | | workplan | FORGE-WP-0010 | finished | — | workplans/FORGE-WP-0010-receive-scanner-budget.md | +| workplan | FORGE-WP-0011 | active | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md | | workplan | RFORGE-WP-0002 | finished | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md | | task | FORGE-WP-0006-T01 | done | — | workplans/FORGE-WP-0006-independent-secret-detection.md | | task | FORGE-WP-0006-T02 | done | — | workplans/FORGE-WP-0006-independent-secret-detection.md | @@ -31,6 +32,10 @@ | task | FORGE-WP-0010-T01 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md | | task | FORGE-WP-0010-T02 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md | | task | FORGE-WP-0010-T03 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md | +| task | FORGE-WP-0011-T01 | done | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md | +| task | FORGE-WP-0011-T02 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md | +| task | FORGE-WP-0011-T03 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md | +| task | FORGE-WP-0011-T04 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md | | task | RFORGE-WP-0002-T01 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md | | task | RFORGE-WP-0002-T02 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md | | task | RFORGE-WP-0002-T03 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md | diff --git a/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch b/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch new file mode 100644 index 0000000..7ce0532 --- /dev/null +++ b/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch @@ -0,0 +1,59 @@ +diff --git a/helm/forgejo-values.yaml b/helm/forgejo-values.yaml +index 3bb9812..08a219e 100644 +--- a/helm/forgejo-values.yaml ++++ b/helm/forgejo-values.yaml +@@ -4,6 +4,19 @@ + strategy: + type: Recreate + ++# Main Forgejo container sizing (FORGE-WP-0011, CUST-WP-0071, RESOURCE-WP-0008). ++# Evidence 2026-10-21 (7d): CPU p95 133-160m, peak ~1030m; memory working set ++# p95 ~956Mi, per-pod peak 1252Mi, RSS peak 500Mi; no OOM events. ++# Request = p95 rule (200m / 1104Mi). Memory limit 1536Mi = 1.23x working-set ++# peak and ~3x RSS peak (the rest is reclaimable page cache). No CPU limit: ++# bursts to ~1 core are normal for git pack/clone work and must not throttle. ++resources: ++ requests: ++ cpu: 200m ++ memory: 1104Mi ++ limits: ++ memory: 1536Mi ++ + service: + http: + type: ClusterIP +diff --git a/manifests/forgejo-runner.yaml b/manifests/forgejo-runner.yaml +index ccbb94b..a886adb 100644 +--- a/manifests/forgejo-runner.yaml ++++ b/manifests/forgejo-runner.yaml +@@ -100,6 +100,16 @@ spec: + - -H + - tcp://127.0.0.1:2375 + - --tls=false ++ # FORGE-WP-0011: job workload runs here (one job at a time, ++ # runner capacity: 1). Request covers idle p95 (391Mi); the limit ++ # bounds a build burst. Peak 7d: 200m / 693Mi working set. ++ resources: ++ requests: ++ cpu: 100m ++ memory: 432Mi ++ limits: ++ cpu: 1500m ++ memory: 2Gi + readinessProbe: + exec: + command: ["sh", "-c", "wget -q -O- -T 3 http://127.0.0.1:2375/_ping | grep -qx OK"] +@@ -108,6 +118,13 @@ spec: + - name: runner + image: code.forgejo.org/forgejo/runner@sha256:5071e6832313bafe71577e05631bece88caff08fcfb372193e4a21941f4ed54b + imagePullPolicy: IfNotPresent ++ # FORGE-WP-0011: runner daemon; p95 2m / 56Mi, peak 124m / 459Mi ++ # (host-label jobs run in this container). Pod request total ++ # 125m / 496Mi. No limit set in this change (see FORGE-WP-0011). ++ resources: ++ requests: ++ cpu: 25m ++ memory: 64Mi + env: + - name: DOCKER_HOST + value: tcp://127.0.0.1:2375 diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/README.md b/docs/evidence/2026-10-21-forgejo-resource-requests/README.md new file mode 100644 index 0000000..745379a --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/README.md @@ -0,0 +1,37 @@ +# Forgejo resource requests — BEFORE state (2026-10-21) + +Workplan: `workplans/FORGE-WP-0011-forgejo-resource-requests.md`. +Captured read-only from railiance01 (`kubectl get/top`, `helm list/history`, +Prometheus query API through the service proxy). No Secret objects or Helm +values were read. + +| File | Content | +| --- | --- | +| `before-deployments.json` | live `forgejo-gitea` / `forgejo-runner` container resources, strategy, PVCs | +| `before-pods.txt` | pods, restarts, QoS class | +| `before-top.txt` | point-in-time `kubectl top` | +| `before-quota.txt` | ResourceQuota / LimitRange (none) | +| `before-helm.txt` | Helm release `forgejo` (rev 9, chart gitea-12.5.0) | +| `before-node.txt` | node allocatable and allocated requests | +| `before-prometheus-7d.txt` | 7d working set, RSS, CPU p95/peak, OOM events | +| `after-render-resources.txt` | rendered resources from the proposed patch (not applied) | + +## Findings + +- `forgejo-gitea` main container `gitea`: **no requests or limits**. The + effective pod request of 100m/128Mi comes from the chart init containers. + QoS Burstable. +- `forgejo-runner` containers `dind` and `runner`: **no requests or limits**. + QoS BestEffort. Runner `config.yaml` on the PVC has `capacity: 1`, so one job + runs at a time already. +- 7d Prometheus (2026-10-14..21): gitea CPU p95 160m, peak about 1030m. Working + set p95 955Mi, per-pod peak 1252Mi, RSS peak 500Mi. dind working set p95 + 392Mi, peak 693Mi, CPU peak 200m. runner working set p95 56Mi, peak 459Mi, + RSS 429Mi, CPU peak 124m. No OOM events. +- Node: 4 CPU / about 15.6Gi allocatable. Requested 3605m (90%) / 9856Mi (61%). +- Drift: Helm history ends at rev 9 (2026-07-07), but the Deployment is at + revision 14 (2026-10-19). Live changes since July were applied outside + `helm upgrade`, though their content matches `railiance-apps` HEAD values. +- Stale pods: `forgejo-gitea-7d94988c79-krqdr` (ContainerStatusUnknown) and + `forgejo-runner-6fcb669668-7g8m7` (Error) are leftovers from 66 days ago. They + hold no requests but clutter status. They are out of scope here. diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/after-render-resources.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/after-render-resources.txt new file mode 100644 index 0000000..684d8b9 --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/after-render-resources.txt @@ -0,0 +1,10 @@ +# helm template (chart gitea 12.5.0) of patched railiance-apps values + registry values; sops layer not rendered +forgejo-gitea verify-secret-scanner {'limits': {'cpu': '100m', 'memory': '64Mi'}, 'requests': {'cpu': '10m', 'memory': '32Mi'}} +forgejo-gitea init-directories {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}} +forgejo-gitea init-app-ini {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}} +forgejo-gitea configure-gitea {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}} +forgejo-gitea gitea {'limits': {'memory': '1536Mi'}, 'requests': {'cpu': '200m', 'memory': '1104Mi'}} +# manifests/forgejo-runner.yaml (patched) +forgejo-runner register None +forgejo-runner dind {'requests': {'cpu': '100m', 'memory': '432Mi'}, 'limits': {'cpu': '1500m', 'memory': '2Gi'}} +forgejo-runner runner {'requests': {'cpu': '25m', 'memory': '64Mi'}} diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-deployments.json b/docs/evidence/2026-10-21-forgejo-resource-requests/before-deployments.json new file mode 100644 index 0000000..7912880 --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-deployments.json @@ -0,0 +1,115 @@ +[ + { + "deployment": "forgejo-gitea", + "generation": 16, + "replicas": 1, + "strategy": { + "type": "Recreate" + }, + "initContainers": [ + { + "name": "verify-secret-scanner", + "resources": { + "limits": { + "cpu": "100m", + "memory": "64Mi" + }, + "requests": { + "cpu": "10m", + "memory": "32Mi" + } + } + }, + { + "name": "init-directories", + "resources": { + "requests": { + "cpu": "100m", + "memory": "128Mi" + } + } + }, + { + "name": "init-app-ini", + "resources": { + "requests": { + "cpu": "100m", + "memory": "128Mi" + } + } + }, + { + "name": "configure-gitea", + "resources": { + "requests": { + "cpu": "100m", + "memory": "128Mi" + } + } + } + ], + "containers": [ + { + "name": "gitea", + "image": "code.forgejo.org/forgejo/forgejo:11.0.3-rootless", + "resources": {} + } + ], + "volumes": [ + { + "name": "init" + }, + { + "name": "config" + }, + { + "name": "inline-config-sources" + }, + { + "name": "temp" + }, + { + "name": "data", + "pvc": "gitea-shared-storage" + }, + { + "name": "secret-scan-template" + } + ] + }, + { + "deployment": "forgejo-runner", + "generation": 4, + "replicas": 1, + "strategy": { + "type": "Recreate" + }, + "initContainers": [ + { + "name": "register", + "resources": {} + } + ], + "containers": [ + { + "name": "dind", + "image": "docker.io/library/docker@sha256:aa3df78ecf320f5fafdce71c659f1629e96e9de0968305fe1de670e0ca9176ce", + "resources": {} + }, + { + "name": "runner", + "image": "code.forgejo.org/forgejo/runner@sha256:5071e6832313bafe71577e05631bece88caff08fcfb372193e4a21941f4ed54b", + "resources": {} + } + ], + "volumes": [ + { + "name": "data", + "pvc": "forgejo-runner-data" + }, + { + "name": "init-scripts" + } + ] + } +] diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-helm.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-helm.txt new file mode 100644 index 0000000..2c9ecd6 --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-helm.txt @@ -0,0 +1,2 @@ +NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION +forgejo forgejo 9 2026-07-07 20:30:46.946867268 +0200 CEST deployed gitea-12.5.0 1.25.4 diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-node.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-node.txt new file mode 100644 index 0000000..ab33635 --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-node.txt @@ -0,0 +1,12 @@ +NAME CPU MEM +239.62.205.92.host.secureserver.net 4 16377032Ki +Allocated resources: + (Total limits may be over 100 percent, i.e., overcommitted.) + Resource Requests Limits + -------- -------- ------ + cpu 3605m (90%) 37500m (937%) + memory 9856Mi (61%) 34562Mi (216%) + ephemeral-storage 256Mi (0%) 1Gi (0%) + hugepages-1Gi 0 (0%) 0 (0%) + hugepages-2Mi 0 (0%) 0 (0%) +Events: diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-pods.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-pods.txt new file mode 100644 index 0000000..f78c492 --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-pods.txt @@ -0,0 +1,5 @@ +NAME PHASE RESTARTS REASON QOS +forgejo-gitea-78bc5d85d7-8bxlz Running 0 Burstable +forgejo-gitea-7d94988c79-krqdr Failed 8 Unknown Burstable +forgejo-runner-6fcb669668-7g8m7 Failed 5,14 Unknown,Error BestEffort +forgejo-runner-8bf88cd84-g4n65 Running 1,4 Unknown,Error BestEffort diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-prometheus-7d.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-prometheus-7d.txt new file mode 100644 index 0000000..9cddccd --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-prometheus-7d.txt @@ -0,0 +1,29 @@ +## max by (pod,container)(max_over_time(container_memory_working_set_bytes{namespace="forgejo",container!="",container!="POD"}[7d]))/1048576 +{'container': 'configure-gitea', 'pod': 'forgejo-gitea-78bc5d85d7-8bxlz'} 2.9 +{'container': 'dind', 'pod': 'forgejo-runner-8bf88cd84-g4n65'} 692.9 +{'container': 'gitea', 'pod': 'forgejo-gitea-78bc5d85d7-8bxlz'} 1017.6 +{'container': 'gitea', 'pod': 'forgejo-gitea-745f685c97-b97ch'} 1180.2 +{'container': 'gitea', 'pod': 'forgejo-gitea-7474684b6-lnr9w'} 1252.0 +{'container': 'runner', 'pod': 'forgejo-runner-8bf88cd84-g4n65'} 458.7 +## quantile_over_time(0.95,sum by (container)(container_memory_working_set_bytes{namespace="forgejo",container!="",container!="POD"})[7d:5m])/1048576 +{'container': 'dind'} 391.5 +{'container': 'gitea'} 955.3 +{'container': 'runner'} 55.5 +## max by (container)(max_over_time(container_memory_rss{namespace="forgejo",container!="",container!="POD"}[7d]))/1048576 +{'container': 'configure-gitea'} 2.4 +{'container': 'dind'} 81.8 +{'container': 'gitea'} 500.4 +{'container': 'runner'} 428.7 +## quantile_over_time(0.95,sum by (container)(rate(container_cpu_usage_seconds_total{namespace="forgejo",container!="",container!="POD"}[5m]))[7d:5m])*1000 +{'container': 'dind'} 10.0 +{'container': 'gitea'} 159.8 +{'container': 'runner'} 1.9 +## max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="forgejo",container!="",container!="POD"}[5m]))[7d:5m])*1000 +{'container': 'dind'} 199.5 +{'container': 'gitea'} 1030.0 +{'container': 'runner'} 124.2 +## sum by (container)(increase(container_oom_events_total{namespace="forgejo"}[7d])) +{'container': 'dind'} 0.0 +{'container': 'gitea'} 0.0 +{'container': 'runner'} 0.0 +{} 0.0 diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-quota.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-quota.txt new file mode 100644 index 0000000..c96fbda --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-quota.txt @@ -0,0 +1 @@ +No resources found in forgejo namespace. diff --git a/docs/evidence/2026-10-21-forgejo-resource-requests/before-top.txt b/docs/evidence/2026-10-21-forgejo-resource-requests/before-top.txt new file mode 100644 index 0000000..704382f --- /dev/null +++ b/docs/evidence/2026-10-21-forgejo-resource-requests/before-top.txt @@ -0,0 +1,4 @@ +POD NAME CPU(cores) MEMORY(bytes) +forgejo-gitea-78bc5d85d7-8bxlz gitea 6m 241Mi +forgejo-runner-8bf88cd84-g4n65 dind 2m 261Mi +forgejo-runner-8bf88cd84-g4n65 runner 1m 33Mi diff --git a/workplans/FORGE-WP-0011-forgejo-resource-requests.md b/workplans/FORGE-WP-0011-forgejo-resource-requests.md new file mode 100644 index 0000000..92ce44c --- /dev/null +++ b/workplans/FORGE-WP-0011-forgejo-resource-requests.md @@ -0,0 +1,225 @@ +--- +id: FORGE-WP-0011 +type: workplan +title: "Size Forgejo and Actions runner requests from the 2026-10-21 allocation review" +domain: financials +repo: railiance-forge +status: active +owner: claude +topic_slug: railiance-forge +created: "2026-10-21" +updated: "2026-10-21" +flavor: implementation +depends_on: [] +related: + - CUST-WP-0071 + - RESOURCE-WP-0008 + - RFORGE-WP-0002 + - FORGE-WP-0009 +--- + +# FORGE-WP-0011 — Forgejo and runner resource requests + +## Goal + +This plan receives the wave-1 railiance-forge rows of the 2026-10-21 node +allocation review (CUST-WP-0071 T03/T04, RESOURCE-WP-0008). It gives the +Forgejo main container and the in-cluster Actions runner explicit requests and +bounds that match measured use. Today both run without any declared resources. + +Sources: + +- `resource-control/docs/evidence/2026-10-21-node-allocation-review.md` +- `resource-control/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}` + (rows `forgejo/gitea`, `forgejo/forgejo-runner`) +- `the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md` + (wave 1 apply approval, one owner at a time, before/after plus rollback receipt) +- BEFORE evidence: `docs/evidence/2026-10-21-forgejo-resource-requests/` + +## Deployment path (important) + +The live Forgejo release and runner are deployed from **railiance-apps**, not +from this repo: + +- Helm release `forgejo` (chart `gitea-charts/gitea` 12.5.0) ← + `railiance-apps/helm/forgejo-values.yaml` via `make forgejo-deploy` +- `deployment/forgejo-runner` ← `railiance-apps/manifests/forgejo-runner.yaml` + via `make forgejo-runner-deploy` + +`railiance-forge/helm/gitea-values.sops.yaml` and `make gitea-deploy` here are +stale. They carry the read-only Gitea mirror settings and +`ROOT_URL=gitea.coulomb.social`; FORGE-WP-0009 retires them. **Never use +`make gitea-deploy` from this repo for this change.** It would overwrite live +Forgejo configuration. + +The change is therefore prepared here as a reviewed patch against +railiance-apps: +`docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch` +(sha256 `5496ceeb7634e7e70634376c3c4967890a39d56f8017b5c9d107bd5c3ff789f9`, +applies cleanly at railiance-apps `10465cb`). Landing it in railiance-apps is +part of T02. + +## Values + +| Workload / container | Before (live) | After | Basis (7d to 2026-10-21) | +| --- | --- | --- | --- | +| forgejo-gitea / gitea | none (pod effective 100m/128Mi from init containers) | req 200m / 1104Mi, limit mem 1536Mi, no CPU limit | CPU p95 133–160m, peak ~1030m; WS p95 956Mi, peak 1252Mi, RSS peak 500Mi | +| forgejo-runner / dind | none (BestEffort) | req 100m / 432Mi, limit 1500m / 2Gi | WS p95 392Mi, peak 693Mi; CPU peak 200m | +| forgejo-runner / runner | none (BestEffort) | req 25m / 64Mi, no limit | WS p95 56Mi, peak 459Mi; CPU peak 124m | +| runner capacity | `capacity: 1` (live config.yaml) | unchanged | one job at a time already holds | + +Pod totals: Forgejo 200m / 1104Mi (QoS Burstable). Runner 125m / 496Mi +(Burstable instead of BestEffort), which matches the review. + +Decisions and deviations from the proposal: + +- **Gitea memory limit 1536Mi.** This is 1.23x the highest per-pod working set + (1252Mi, 2026-10-13 pod) and about 3x the peak RSS. The rest of the working + set is page cache, which the kernel reclaims before an OOM kill. The review + suggested "~1.5Gi" and the evidence supports it. Raise it to 2Gi if T03 shows + a working set that stays at the limit, or any OOM. +- **Gitea CPU request stays 200m.** The independent 7d p95 here is 160m + (review: 133m). 200m still covers it, so the reviewed value is kept. No CPU + limit is set: clone/pack bursts reach ~1 core, and repo convention does not + require CPU limits. +- **The runner pod request is split per container** (the review gives only the + pod total): dind takes most of it because the job containers run inside + dind. +- **No limit on the `runner` container** (out of reviewed scope). Jobs with + `:host` labels (`container-build`, `registry-publish`, …) run inside the + runner container itself, which explains its 459Mi peak. A memory limit there + (suggest 1Gi) needs a separate review, because an OOM would kill the runner + daemon in the middle of a job. + +## Tasks + +```task +id: FORGE-WP-0011-T01 +status: done +priority: high +``` + +Capture BEFORE state read-only and prepare the reviewed change. + +Done 2026-10-21: evidence in `docs/evidence/2026-10-21-forgejo-resource-requests/`. +The patch was rendered on a scratch copy of railiance-apps with +`helm template forgejo gitea-charts/gitea --version 12.5.0 -f helm/forgejo-values.yaml -f helm/forgejo-registry-values.yaml` +(rc 0; the SOPS layer was not rendered) and with +`kubectl apply --dry-run=client -f manifests/forgejo-runner.yaml`. The rendered +resources are in `after-render-resources.txt`. + +```task +id: FORGE-WP-0011-T02 +status: todo +priority: high +``` + +Land the patch in railiance-apps and apply, one workload at a time, under the +wave-1 approval in the delivery interview. + +**Downtime:** the Forgejo Deployment uses `strategy: Recreate` on an RWO +`local-path` PVC. A rollout therefore stops Forgejo before the new pod starts, +so git/web/API/registry are down for about 1–3 minutes (init containers plus +startup). This is acceptable. Pick a quiet moment, with no running Actions job +and no release in flight. **No PVC or StatefulSet recreation is needed.** Both +are plain Deployment template changes, and the PVCs `gitea-shared-storage` and +`forgejo-runner-data` are reused. The runner Deployment is also `Recreate`, so +a running job would be killed. Check first that no job is running. + +Node headroom: the change adds +225m CPU and +1472Mi memory of requests. That +takes the node to about 3830m / 4000m (96%) CPU and about 11.3Gi / 15.6Gi +memory, as of the BEFORE snapshot. Recreate frees the old pod's request first, +so scheduling will fit. Still, recheck `kubectl describe node` right before +apply: if other wave-1 increases landed first and allocated CPU is above +~3775m, the new pod can stay Pending. In that case, apply wave-1 decreases +first or roll back. + +Deploy (railiance-apps checkout, operator kubeconfig): + +```bash +cd ~/railiance-apps +git pull --ff-only +git apply ~/railiance-forge/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch +git diff --stat # helm/forgejo-values.yaml, manifests/forgejo-runner.yaml +git commit -am "FORGE-WP-0011: size Forgejo and runner requests (CUST-WP-0071)" +git push + +# 0. preflight: no running Actions job, node headroom +KUBECONFIG=~/.kube/config-railiance01 kubectl describe node | sed -n '/Allocated resources/,/Events/p' +KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo logs deploy/forgejo-runner -c runner --tail=20 + +# 1. Forgejo (Helm upgrade; restarts Forgejo; --wait 10m) +make forgejo-deploy + +# 2. Runner (kubectl apply; Recreate) +make forgejo-runner-deploy +``` + +Note on Helm drift: the release history ends at rev 9 (2026-07-07), but the +Deployment has had 5 later rollouts that were not made through Helm. +`make forgejo-deploy` renders the full railiance-apps HEAD values. Review +`helm diff` (if installed) or compare the +`helm template ... | yq 'select(.kind=="Deployment")'` output with the live +Deployment before upgrading. Expect only the `resources` delta plus any +Helm-label churn. + +Rollback receipt (record the Helm revision created in step 1): + +```bash +# Forgejo: back to the previous release revision (restarts Forgejo again) +KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo history forgejo --max 3 +KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo rollback forgejo --wait --timeout 10m +# Runner: revert the manifest and re-apply +cd ~/railiance-apps && git revert --no-edit && git push && make forgejo-runner-deploy +# Emergency runner-only revert without git: +KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo rollout undo deployment/forgejo-runner +``` + +`helm rollback` to rev 9 restores the July-rendered template, which is older +than the October live state (secret-scanner init container, PKCE). **Prefer +`git revert` + `make forgejo-deploy`** for Forgejo. Use `helm rollback` only to +the revision created immediately before this change. + +```task +id: FORGE-WP-0011-T03 +status: todo +priority: high +depends_on: [FORGE-WP-0011-T02] +``` + +Verify right after apply, and again after 24h and 7d. Store the outputs as +`docs/evidence/2026-10-21-forgejo-resource-requests/after-*.txt`. + +```bash +K="kubectl --kubeconfig ~/.kube/config-railiance01 -n forgejo" +$K get deploy forgejo-gitea forgejo-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]} {.name} {.resources}{"\n"}{end}{end}' +$K get pods -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,QOS:.status.qosClass,RESTARTS:.status.containerStatuses[*].restartCount,LAST:.status.containerStatuses[*].lastState.terminated.reason +# Forgejo UI / API / git +curl -fsS https://forgejo.coulomb.social/api/healthz +curl -fsS https://forgejo.coulomb.social/api/v1/version +git ls-remote https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git HEAD +git clone --depth 1 https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git /tmp/fwp11-clone && rm -rf /tmp/fwp11-clone +# Runner picks a job: re-run the probe workflow (container-probe / image-build) in +# coulomb/forgejo-actions-probe from the UI, then: +$K logs deploy/forgejo-runner -c runner --tail=50 | grep -iE "task|job" +# No OOM / throttling (Prometheus, 24h and 7d after) +# increase(container_oom_events_total{namespace="forgejo"}[24h]) == 0 +# max_over_time(container_memory_working_set_bytes{namespace="forgejo",container="gitea"}[24h]) < 1536Mi +# rate(container_cpu_cfs_throttled_periods_total{namespace="forgejo",container="dind"}[5m]) during a build +``` + +Done when Forgejo answers on UI/API/git clone, a runner job completes, and no +OOMKilled appears after 7d. Then report the after-values back to +resource-control RESOURCE-WP-0008 and CUST-WP-0071 T04. + +```task +id: FORGE-WP-0011-T04 +status: todo +priority: low +``` + +Follow-ups outside the reviewed wave-1 diff (each needs its own approval): +a memory limit for the `runner` container (host-label jobs), removal of the +two stale Failed pods from 66 days ago, and reconciling Helm release drift +(rev 9 vs Deployment rev 14). The Prometheus `forgejo` request recording rule +that returns no series is owned by rapp-telemetry.