FORGE-WP-0011: Forgejo and runner resource requests (CUST-WP-0071, RESOURCE-WP-0008)
Receive the wave-1 allocation review rows: BEFORE evidence captured read-only, reviewed patch against the live railiance-apps deploy path, deploy/verify/ rollback commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 8340@bnt-lap001 Assistant-Session: 2e192b3d-b3e8-4403-94d4-d55a2c8b476c
This commit is contained in:
parent
256da9fb70
commit
6d795677b8
12 changed files with 504 additions and 0 deletions
|
|
@ -13,6 +13,7 @@
|
|||
| workplan | FORGE-WP-0008 | proposed | — | workplans/FORGE-WP-0008-secret-detection-residual-coverage.md |
|
||||
| workplan | FORGE-WP-0009 | proposed | — | workplans/FORGE-WP-0009-retire-stale-gitea-deploy-assets.md |
|
||||
| workplan | FORGE-WP-0010 | finished | — | workplans/FORGE-WP-0010-receive-scanner-budget.md |
|
||||
| workplan | FORGE-WP-0011 | active | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md |
|
||||
| workplan | RFORGE-WP-0002 | finished | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md |
|
||||
| task | FORGE-WP-0006-T01 | done | — | workplans/FORGE-WP-0006-independent-secret-detection.md |
|
||||
| task | FORGE-WP-0006-T02 | done | — | workplans/FORGE-WP-0006-independent-secret-detection.md |
|
||||
|
|
@ -31,6 +32,10 @@
|
|||
| task | FORGE-WP-0010-T01 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md |
|
||||
| task | FORGE-WP-0010-T02 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md |
|
||||
| task | FORGE-WP-0010-T03 | done | — | workplans/FORGE-WP-0010-receive-scanner-budget.md |
|
||||
| task | FORGE-WP-0011-T01 | done | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md |
|
||||
| task | FORGE-WP-0011-T02 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md |
|
||||
| task | FORGE-WP-0011-T03 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md |
|
||||
| task | FORGE-WP-0011-T04 | todo | — | workplans/FORGE-WP-0011-forgejo-resource-requests.md |
|
||||
| task | RFORGE-WP-0002-T01 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md |
|
||||
| task | RFORGE-WP-0002-T02 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md |
|
||||
| task | RFORGE-WP-0002-T03 | done | — | workplans/RFORGE-WP-0002-forge-resource-economics-evidence.md |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,59 @@
|
|||
diff --git a/helm/forgejo-values.yaml b/helm/forgejo-values.yaml
|
||||
index 3bb9812..08a219e 100644
|
||||
--- a/helm/forgejo-values.yaml
|
||||
+++ b/helm/forgejo-values.yaml
|
||||
@@ -4,6 +4,19 @@
|
||||
strategy:
|
||||
type: Recreate
|
||||
|
||||
+# Main Forgejo container sizing (FORGE-WP-0011, CUST-WP-0071, RESOURCE-WP-0008).
|
||||
+# Evidence 2026-10-21 (7d): CPU p95 133-160m, peak ~1030m; memory working set
|
||||
+# p95 ~956Mi, per-pod peak 1252Mi, RSS peak 500Mi; no OOM events.
|
||||
+# Request = p95 rule (200m / 1104Mi). Memory limit 1536Mi = 1.23x working-set
|
||||
+# peak and ~3x RSS peak (the rest is reclaimable page cache). No CPU limit:
|
||||
+# bursts to ~1 core are normal for git pack/clone work and must not throttle.
|
||||
+resources:
|
||||
+ requests:
|
||||
+ cpu: 200m
|
||||
+ memory: 1104Mi
|
||||
+ limits:
|
||||
+ memory: 1536Mi
|
||||
+
|
||||
service:
|
||||
http:
|
||||
type: ClusterIP
|
||||
diff --git a/manifests/forgejo-runner.yaml b/manifests/forgejo-runner.yaml
|
||||
index ccbb94b..a886adb 100644
|
||||
--- a/manifests/forgejo-runner.yaml
|
||||
+++ b/manifests/forgejo-runner.yaml
|
||||
@@ -100,6 +100,16 @@ spec:
|
||||
- -H
|
||||
- tcp://127.0.0.1:2375
|
||||
- --tls=false
|
||||
+ # FORGE-WP-0011: job workload runs here (one job at a time,
|
||||
+ # runner capacity: 1). Request covers idle p95 (391Mi); the limit
|
||||
+ # bounds a build burst. Peak 7d: 200m / 693Mi working set.
|
||||
+ resources:
|
||||
+ requests:
|
||||
+ cpu: 100m
|
||||
+ memory: 432Mi
|
||||
+ limits:
|
||||
+ cpu: 1500m
|
||||
+ memory: 2Gi
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "wget -q -O- -T 3 http://127.0.0.1:2375/_ping | grep -qx OK"]
|
||||
@@ -108,6 +118,13 @@ spec:
|
||||
- name: runner
|
||||
image: code.forgejo.org/forgejo/runner@sha256:5071e6832313bafe71577e05631bece88caff08fcfb372193e4a21941f4ed54b
|
||||
imagePullPolicy: IfNotPresent
|
||||
+ # FORGE-WP-0011: runner daemon; p95 2m / 56Mi, peak 124m / 459Mi
|
||||
+ # (host-label jobs run in this container). Pod request total
|
||||
+ # 125m / 496Mi. No limit set in this change (see FORGE-WP-0011).
|
||||
+ resources:
|
||||
+ requests:
|
||||
+ cpu: 25m
|
||||
+ memory: 64Mi
|
||||
env:
|
||||
- name: DOCKER_HOST
|
||||
value: tcp://127.0.0.1:2375
|
||||
37
docs/evidence/2026-10-21-forgejo-resource-requests/README.md
Normal file
37
docs/evidence/2026-10-21-forgejo-resource-requests/README.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Forgejo resource requests — BEFORE state (2026-10-21)
|
||||
|
||||
Workplan: `workplans/FORGE-WP-0011-forgejo-resource-requests.md`.
|
||||
Captured read-only from railiance01 (`kubectl get/top`, `helm list/history`,
|
||||
Prometheus query API through the service proxy). No Secret objects or Helm
|
||||
values were read.
|
||||
|
||||
| File | Content |
|
||||
| --- | --- |
|
||||
| `before-deployments.json` | live `forgejo-gitea` / `forgejo-runner` container resources, strategy, PVCs |
|
||||
| `before-pods.txt` | pods, restarts, QoS class |
|
||||
| `before-top.txt` | point-in-time `kubectl top` |
|
||||
| `before-quota.txt` | ResourceQuota / LimitRange (none) |
|
||||
| `before-helm.txt` | Helm release `forgejo` (rev 9, chart gitea-12.5.0) |
|
||||
| `before-node.txt` | node allocatable and allocated requests |
|
||||
| `before-prometheus-7d.txt` | 7d working set, RSS, CPU p95/peak, OOM events |
|
||||
| `after-render-resources.txt` | rendered resources from the proposed patch (not applied) |
|
||||
|
||||
## Findings
|
||||
|
||||
- `forgejo-gitea` main container `gitea`: **no requests or limits**. The
|
||||
effective pod request of 100m/128Mi comes from the chart init containers.
|
||||
QoS Burstable.
|
||||
- `forgejo-runner` containers `dind` and `runner`: **no requests or limits**.
|
||||
QoS BestEffort. Runner `config.yaml` on the PVC has `capacity: 1`, so one job
|
||||
runs at a time already.
|
||||
- 7d Prometheus (2026-10-14..21): gitea CPU p95 160m, peak about 1030m. Working
|
||||
set p95 955Mi, per-pod peak 1252Mi, RSS peak 500Mi. dind working set p95
|
||||
392Mi, peak 693Mi, CPU peak 200m. runner working set p95 56Mi, peak 459Mi,
|
||||
RSS 429Mi, CPU peak 124m. No OOM events.
|
||||
- Node: 4 CPU / about 15.6Gi allocatable. Requested 3605m (90%) / 9856Mi (61%).
|
||||
- Drift: Helm history ends at rev 9 (2026-07-07), but the Deployment is at
|
||||
revision 14 (2026-10-19). Live changes since July were applied outside
|
||||
`helm upgrade`, though their content matches `railiance-apps` HEAD values.
|
||||
- Stale pods: `forgejo-gitea-7d94988c79-krqdr` (ContainerStatusUnknown) and
|
||||
`forgejo-runner-6fcb669668-7g8m7` (Error) are leftovers from 66 days ago. They
|
||||
hold no requests but clutter status. They are out of scope here.
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
# helm template (chart gitea 12.5.0) of patched railiance-apps values + registry values; sops layer not rendered
|
||||
forgejo-gitea verify-secret-scanner {'limits': {'cpu': '100m', 'memory': '64Mi'}, 'requests': {'cpu': '10m', 'memory': '32Mi'}}
|
||||
forgejo-gitea init-directories {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}}
|
||||
forgejo-gitea init-app-ini {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}}
|
||||
forgejo-gitea configure-gitea {'limits': {}, 'requests': {'cpu': '100m', 'memory': '128Mi'}}
|
||||
forgejo-gitea gitea {'limits': {'memory': '1536Mi'}, 'requests': {'cpu': '200m', 'memory': '1104Mi'}}
|
||||
# manifests/forgejo-runner.yaml (patched)
|
||||
forgejo-runner register None
|
||||
forgejo-runner dind {'requests': {'cpu': '100m', 'memory': '432Mi'}, 'limits': {'cpu': '1500m', 'memory': '2Gi'}}
|
||||
forgejo-runner runner {'requests': {'cpu': '25m', 'memory': '64Mi'}}
|
||||
|
|
@ -0,0 +1,115 @@
|
|||
[
|
||||
{
|
||||
"deployment": "forgejo-gitea",
|
||||
"generation": 16,
|
||||
"replicas": 1,
|
||||
"strategy": {
|
||||
"type": "Recreate"
|
||||
},
|
||||
"initContainers": [
|
||||
{
|
||||
"name": "verify-secret-scanner",
|
||||
"resources": {
|
||||
"limits": {
|
||||
"cpu": "100m",
|
||||
"memory": "64Mi"
|
||||
},
|
||||
"requests": {
|
||||
"cpu": "10m",
|
||||
"memory": "32Mi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "init-directories",
|
||||
"resources": {
|
||||
"requests": {
|
||||
"cpu": "100m",
|
||||
"memory": "128Mi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "init-app-ini",
|
||||
"resources": {
|
||||
"requests": {
|
||||
"cpu": "100m",
|
||||
"memory": "128Mi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "configure-gitea",
|
||||
"resources": {
|
||||
"requests": {
|
||||
"cpu": "100m",
|
||||
"memory": "128Mi"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "gitea",
|
||||
"image": "code.forgejo.org/forgejo/forgejo:11.0.3-rootless",
|
||||
"resources": {}
|
||||
}
|
||||
],
|
||||
"volumes": [
|
||||
{
|
||||
"name": "init"
|
||||
},
|
||||
{
|
||||
"name": "config"
|
||||
},
|
||||
{
|
||||
"name": "inline-config-sources"
|
||||
},
|
||||
{
|
||||
"name": "temp"
|
||||
},
|
||||
{
|
||||
"name": "data",
|
||||
"pvc": "gitea-shared-storage"
|
||||
},
|
||||
{
|
||||
"name": "secret-scan-template"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"deployment": "forgejo-runner",
|
||||
"generation": 4,
|
||||
"replicas": 1,
|
||||
"strategy": {
|
||||
"type": "Recreate"
|
||||
},
|
||||
"initContainers": [
|
||||
{
|
||||
"name": "register",
|
||||
"resources": {}
|
||||
}
|
||||
],
|
||||
"containers": [
|
||||
{
|
||||
"name": "dind",
|
||||
"image": "docker.io/library/docker@sha256:aa3df78ecf320f5fafdce71c659f1629e96e9de0968305fe1de670e0ca9176ce",
|
||||
"resources": {}
|
||||
},
|
||||
{
|
||||
"name": "runner",
|
||||
"image": "code.forgejo.org/forgejo/runner@sha256:5071e6832313bafe71577e05631bece88caff08fcfb372193e4a21941f4ed54b",
|
||||
"resources": {}
|
||||
}
|
||||
],
|
||||
"volumes": [
|
||||
{
|
||||
"name": "data",
|
||||
"pvc": "forgejo-runner-data"
|
||||
},
|
||||
{
|
||||
"name": "init-scripts"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
|
||||
forgejo forgejo 9 2026-07-07 20:30:46.946867268 +0200 CEST deployed gitea-12.5.0 1.25.4
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
NAME CPU MEM
|
||||
239.62.205.92.host.secureserver.net 4 16377032Ki
|
||||
Allocated resources:
|
||||
(Total limits may be over 100 percent, i.e., overcommitted.)
|
||||
Resource Requests Limits
|
||||
-------- -------- ------
|
||||
cpu 3605m (90%) 37500m (937%)
|
||||
memory 9856Mi (61%) 34562Mi (216%)
|
||||
ephemeral-storage 256Mi (0%) 1Gi (0%)
|
||||
hugepages-1Gi 0 (0%) 0 (0%)
|
||||
hugepages-2Mi 0 (0%) 0 (0%)
|
||||
Events: <none>
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
NAME PHASE RESTARTS REASON QOS
|
||||
forgejo-gitea-78bc5d85d7-8bxlz Running 0 <none> Burstable
|
||||
forgejo-gitea-7d94988c79-krqdr Failed 8 Unknown Burstable
|
||||
forgejo-runner-6fcb669668-7g8m7 Failed 5,14 Unknown,Error BestEffort
|
||||
forgejo-runner-8bf88cd84-g4n65 Running 1,4 Unknown,Error BestEffort
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
## max by (pod,container)(max_over_time(container_memory_working_set_bytes{namespace="forgejo",container!="",container!="POD"}[7d]))/1048576
|
||||
{'container': 'configure-gitea', 'pod': 'forgejo-gitea-78bc5d85d7-8bxlz'} 2.9
|
||||
{'container': 'dind', 'pod': 'forgejo-runner-8bf88cd84-g4n65'} 692.9
|
||||
{'container': 'gitea', 'pod': 'forgejo-gitea-78bc5d85d7-8bxlz'} 1017.6
|
||||
{'container': 'gitea', 'pod': 'forgejo-gitea-745f685c97-b97ch'} 1180.2
|
||||
{'container': 'gitea', 'pod': 'forgejo-gitea-7474684b6-lnr9w'} 1252.0
|
||||
{'container': 'runner', 'pod': 'forgejo-runner-8bf88cd84-g4n65'} 458.7
|
||||
## quantile_over_time(0.95,sum by (container)(container_memory_working_set_bytes{namespace="forgejo",container!="",container!="POD"})[7d:5m])/1048576
|
||||
{'container': 'dind'} 391.5
|
||||
{'container': 'gitea'} 955.3
|
||||
{'container': 'runner'} 55.5
|
||||
## max by (container)(max_over_time(container_memory_rss{namespace="forgejo",container!="",container!="POD"}[7d]))/1048576
|
||||
{'container': 'configure-gitea'} 2.4
|
||||
{'container': 'dind'} 81.8
|
||||
{'container': 'gitea'} 500.4
|
||||
{'container': 'runner'} 428.7
|
||||
## quantile_over_time(0.95,sum by (container)(rate(container_cpu_usage_seconds_total{namespace="forgejo",container!="",container!="POD"}[5m]))[7d:5m])*1000
|
||||
{'container': 'dind'} 10.0
|
||||
{'container': 'gitea'} 159.8
|
||||
{'container': 'runner'} 1.9
|
||||
## max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="forgejo",container!="",container!="POD"}[5m]))[7d:5m])*1000
|
||||
{'container': 'dind'} 199.5
|
||||
{'container': 'gitea'} 1030.0
|
||||
{'container': 'runner'} 124.2
|
||||
## sum by (container)(increase(container_oom_events_total{namespace="forgejo"}[7d]))
|
||||
{'container': 'dind'} 0.0
|
||||
{'container': 'gitea'} 0.0
|
||||
{'container': 'runner'} 0.0
|
||||
{} 0.0
|
||||
|
|
@ -0,0 +1 @@
|
|||
No resources found in forgejo namespace.
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
POD NAME CPU(cores) MEMORY(bytes)
|
||||
forgejo-gitea-78bc5d85d7-8bxlz gitea 6m 241Mi
|
||||
forgejo-runner-8bf88cd84-g4n65 dind 2m 261Mi
|
||||
forgejo-runner-8bf88cd84-g4n65 runner 1m 33Mi
|
||||
225
workplans/FORGE-WP-0011-forgejo-resource-requests.md
Normal file
225
workplans/FORGE-WP-0011-forgejo-resource-requests.md
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
---
|
||||
id: FORGE-WP-0011
|
||||
type: workplan
|
||||
title: "Size Forgejo and Actions runner requests from the 2026-10-21 allocation review"
|
||||
domain: financials
|
||||
repo: railiance-forge
|
||||
status: active
|
||||
owner: claude
|
||||
topic_slug: railiance-forge
|
||||
created: "2026-10-21"
|
||||
updated: "2026-10-21"
|
||||
flavor: implementation
|
||||
depends_on: []
|
||||
related:
|
||||
- CUST-WP-0071
|
||||
- RESOURCE-WP-0008
|
||||
- RFORGE-WP-0002
|
||||
- FORGE-WP-0009
|
||||
---
|
||||
|
||||
# FORGE-WP-0011 — Forgejo and runner resource requests
|
||||
|
||||
## Goal
|
||||
|
||||
This plan receives the wave-1 railiance-forge rows of the 2026-10-21 node
|
||||
allocation review (CUST-WP-0071 T03/T04, RESOURCE-WP-0008). It gives the
|
||||
Forgejo main container and the in-cluster Actions runner explicit requests and
|
||||
bounds that match measured use. Today both run without any declared resources.
|
||||
|
||||
Sources:
|
||||
|
||||
- `resource-control/docs/evidence/2026-10-21-node-allocation-review.md`
|
||||
- `resource-control/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json}`
|
||||
(rows `forgejo/gitea`, `forgejo/forgejo-runner`)
|
||||
- `the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md`
|
||||
(wave 1 apply approval, one owner at a time, before/after plus rollback receipt)
|
||||
- BEFORE evidence: `docs/evidence/2026-10-21-forgejo-resource-requests/`
|
||||
|
||||
## Deployment path (important)
|
||||
|
||||
The live Forgejo release and runner are deployed from **railiance-apps**, not
|
||||
from this repo:
|
||||
|
||||
- Helm release `forgejo` (chart `gitea-charts/gitea` 12.5.0) ←
|
||||
`railiance-apps/helm/forgejo-values.yaml` via `make forgejo-deploy`
|
||||
- `deployment/forgejo-runner` ← `railiance-apps/manifests/forgejo-runner.yaml`
|
||||
via `make forgejo-runner-deploy`
|
||||
|
||||
`railiance-forge/helm/gitea-values.sops.yaml` and `make gitea-deploy` here are
|
||||
stale. They carry the read-only Gitea mirror settings and
|
||||
`ROOT_URL=gitea.coulomb.social`; FORGE-WP-0009 retires them. **Never use
|
||||
`make gitea-deploy` from this repo for this change.** It would overwrite live
|
||||
Forgejo configuration.
|
||||
|
||||
The change is therefore prepared here as a reviewed patch against
|
||||
railiance-apps:
|
||||
`docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch`
|
||||
(sha256 `5496ceeb7634e7e70634376c3c4967890a39d56f8017b5c9d107bd5c3ff789f9`,
|
||||
applies cleanly at railiance-apps `10465cb`). Landing it in railiance-apps is
|
||||
part of T02.
|
||||
|
||||
## Values
|
||||
|
||||
| Workload / container | Before (live) | After | Basis (7d to 2026-10-21) |
|
||||
| --- | --- | --- | --- |
|
||||
| forgejo-gitea / gitea | none (pod effective 100m/128Mi from init containers) | req 200m / 1104Mi, limit mem 1536Mi, no CPU limit | CPU p95 133–160m, peak ~1030m; WS p95 956Mi, peak 1252Mi, RSS peak 500Mi |
|
||||
| forgejo-runner / dind | none (BestEffort) | req 100m / 432Mi, limit 1500m / 2Gi | WS p95 392Mi, peak 693Mi; CPU peak 200m |
|
||||
| forgejo-runner / runner | none (BestEffort) | req 25m / 64Mi, no limit | WS p95 56Mi, peak 459Mi; CPU peak 124m |
|
||||
| runner capacity | `capacity: 1` (live config.yaml) | unchanged | one job at a time already holds |
|
||||
|
||||
Pod totals: Forgejo 200m / 1104Mi (QoS Burstable). Runner 125m / 496Mi
|
||||
(Burstable instead of BestEffort), which matches the review.
|
||||
|
||||
Decisions and deviations from the proposal:
|
||||
|
||||
- **Gitea memory limit 1536Mi.** This is 1.23x the highest per-pod working set
|
||||
(1252Mi, 2026-10-13 pod) and about 3x the peak RSS. The rest of the working
|
||||
set is page cache, which the kernel reclaims before an OOM kill. The review
|
||||
suggested "~1.5Gi" and the evidence supports it. Raise it to 2Gi if T03 shows
|
||||
a working set that stays at the limit, or any OOM.
|
||||
- **Gitea CPU request stays 200m.** The independent 7d p95 here is 160m
|
||||
(review: 133m). 200m still covers it, so the reviewed value is kept. No CPU
|
||||
limit is set: clone/pack bursts reach ~1 core, and repo convention does not
|
||||
require CPU limits.
|
||||
- **The runner pod request is split per container** (the review gives only the
|
||||
pod total): dind takes most of it because the job containers run inside
|
||||
dind.
|
||||
- **No limit on the `runner` container** (out of reviewed scope). Jobs with
|
||||
`:host` labels (`container-build`, `registry-publish`, …) run inside the
|
||||
runner container itself, which explains its 459Mi peak. A memory limit there
|
||||
(suggest 1Gi) needs a separate review, because an OOM would kill the runner
|
||||
daemon in the middle of a job.
|
||||
|
||||
## Tasks
|
||||
|
||||
```task
|
||||
id: FORGE-WP-0011-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Capture BEFORE state read-only and prepare the reviewed change.
|
||||
|
||||
Done 2026-10-21: evidence in `docs/evidence/2026-10-21-forgejo-resource-requests/`.
|
||||
The patch was rendered on a scratch copy of railiance-apps with
|
||||
`helm template forgejo gitea-charts/gitea --version 12.5.0 -f helm/forgejo-values.yaml -f helm/forgejo-registry-values.yaml`
|
||||
(rc 0; the SOPS layer was not rendered) and with
|
||||
`kubectl apply --dry-run=client -f manifests/forgejo-runner.yaml`. The rendered
|
||||
resources are in `after-render-resources.txt`.
|
||||
|
||||
```task
|
||||
id: FORGE-WP-0011-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Land the patch in railiance-apps and apply, one workload at a time, under the
|
||||
wave-1 approval in the delivery interview.
|
||||
|
||||
**Downtime:** the Forgejo Deployment uses `strategy: Recreate` on an RWO
|
||||
`local-path` PVC. A rollout therefore stops Forgejo before the new pod starts,
|
||||
so git/web/API/registry are down for about 1–3 minutes (init containers plus
|
||||
startup). This is acceptable. Pick a quiet moment, with no running Actions job
|
||||
and no release in flight. **No PVC or StatefulSet recreation is needed.** Both
|
||||
are plain Deployment template changes, and the PVCs `gitea-shared-storage` and
|
||||
`forgejo-runner-data` are reused. The runner Deployment is also `Recreate`, so
|
||||
a running job would be killed. Check first that no job is running.
|
||||
|
||||
Node headroom: the change adds +225m CPU and +1472Mi memory of requests. That
|
||||
takes the node to about 3830m / 4000m (96%) CPU and about 11.3Gi / 15.6Gi
|
||||
memory, as of the BEFORE snapshot. Recreate frees the old pod's request first,
|
||||
so scheduling will fit. Still, recheck `kubectl describe node` right before
|
||||
apply: if other wave-1 increases landed first and allocated CPU is above
|
||||
~3775m, the new pod can stay Pending. In that case, apply wave-1 decreases
|
||||
first or roll back.
|
||||
|
||||
Deploy (railiance-apps checkout, operator kubeconfig):
|
||||
|
||||
```bash
|
||||
cd ~/railiance-apps
|
||||
git pull --ff-only
|
||||
git apply ~/railiance-forge/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch
|
||||
git diff --stat # helm/forgejo-values.yaml, manifests/forgejo-runner.yaml
|
||||
git commit -am "FORGE-WP-0011: size Forgejo and runner requests (CUST-WP-0071)"
|
||||
git push
|
||||
|
||||
# 0. preflight: no running Actions job, node headroom
|
||||
KUBECONFIG=~/.kube/config-railiance01 kubectl describe node | sed -n '/Allocated resources/,/Events/p'
|
||||
KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo logs deploy/forgejo-runner -c runner --tail=20
|
||||
|
||||
# 1. Forgejo (Helm upgrade; restarts Forgejo; --wait 10m)
|
||||
make forgejo-deploy
|
||||
|
||||
# 2. Runner (kubectl apply; Recreate)
|
||||
make forgejo-runner-deploy
|
||||
```
|
||||
|
||||
Note on Helm drift: the release history ends at rev 9 (2026-07-07), but the
|
||||
Deployment has had 5 later rollouts that were not made through Helm.
|
||||
`make forgejo-deploy` renders the full railiance-apps HEAD values. Review
|
||||
`helm diff` (if installed) or compare the
|
||||
`helm template ... | yq 'select(.kind=="Deployment")'` output with the live
|
||||
Deployment before upgrading. Expect only the `resources` delta plus any
|
||||
Helm-label churn.
|
||||
|
||||
Rollback receipt (record the Helm revision created in step 1):
|
||||
|
||||
```bash
|
||||
# Forgejo: back to the previous release revision (restarts Forgejo again)
|
||||
KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo history forgejo --max 3
|
||||
KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo rollback forgejo <previous-rev> --wait --timeout 10m
|
||||
# Runner: revert the manifest and re-apply
|
||||
cd ~/railiance-apps && git revert --no-edit <commit> && git push && make forgejo-runner-deploy
|
||||
# Emergency runner-only revert without git:
|
||||
KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo rollout undo deployment/forgejo-runner
|
||||
```
|
||||
|
||||
`helm rollback` to rev 9 restores the July-rendered template, which is older
|
||||
than the October live state (secret-scanner init container, PKCE). **Prefer
|
||||
`git revert` + `make forgejo-deploy`** for Forgejo. Use `helm rollback` only to
|
||||
the revision created immediately before this change.
|
||||
|
||||
```task
|
||||
id: FORGE-WP-0011-T03
|
||||
status: todo
|
||||
priority: high
|
||||
depends_on: [FORGE-WP-0011-T02]
|
||||
```
|
||||
|
||||
Verify right after apply, and again after 24h and 7d. Store the outputs as
|
||||
`docs/evidence/2026-10-21-forgejo-resource-requests/after-*.txt`.
|
||||
|
||||
```bash
|
||||
K="kubectl --kubeconfig ~/.kube/config-railiance01 -n forgejo"
|
||||
$K get deploy forgejo-gitea forgejo-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]} {.name} {.resources}{"\n"}{end}{end}'
|
||||
$K get pods -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,QOS:.status.qosClass,RESTARTS:.status.containerStatuses[*].restartCount,LAST:.status.containerStatuses[*].lastState.terminated.reason
|
||||
# Forgejo UI / API / git
|
||||
curl -fsS https://forgejo.coulomb.social/api/healthz
|
||||
curl -fsS https://forgejo.coulomb.social/api/v1/version
|
||||
git ls-remote https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git HEAD
|
||||
git clone --depth 1 https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git /tmp/fwp11-clone && rm -rf /tmp/fwp11-clone
|
||||
# Runner picks a job: re-run the probe workflow (container-probe / image-build) in
|
||||
# coulomb/forgejo-actions-probe from the UI, then:
|
||||
$K logs deploy/forgejo-runner -c runner --tail=50 | grep -iE "task|job"
|
||||
# No OOM / throttling (Prometheus, 24h and 7d after)
|
||||
# increase(container_oom_events_total{namespace="forgejo"}[24h]) == 0
|
||||
# max_over_time(container_memory_working_set_bytes{namespace="forgejo",container="gitea"}[24h]) < 1536Mi
|
||||
# rate(container_cpu_cfs_throttled_periods_total{namespace="forgejo",container="dind"}[5m]) during a build
|
||||
```
|
||||
|
||||
Done when Forgejo answers on UI/API/git clone, a runner job completes, and no
|
||||
OOMKilled appears after 7d. Then report the after-values back to
|
||||
resource-control RESOURCE-WP-0008 and CUST-WP-0071 T04.
|
||||
|
||||
```task
|
||||
id: FORGE-WP-0011-T04
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
Follow-ups outside the reviewed wave-1 diff (each needs its own approval):
|
||||
a memory limit for the `runner` container (host-label jobs), removal of the
|
||||
two stale Failed pods from 66 days ago, and reconciling Helm release drift
|
||||
(rev 9 vs Deployment rev 14). The Prometheus `forgejo` request recording rule
|
||||
that returns no series is owned by rapp-telemetry.
|
||||
Loading…
Add table
Add a link
Reference in a new issue