railiance-forge/workplans/FORGE-WP-0011-forgejo-resource-requests.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	flavor	depends_on	related	state_hub_workstream_id
FORGE-WP-0011	workplan	Size Forgejo and Actions runner requests from the 2026-10-21 allocation review	financials	railiance-forge	active	claude	railiance-forge	2026-10-21	2026-10-21	implementation		CUST-WP-0071 RESOURCE-WP-0008 RFORGE-WP-0002 FORGE-WP-0009	8b5f2b9e-bf68-5052-8162-46fe907068be

FORGE-WP-0011 — Forgejo and runner resource requests

Goal

This plan receives the wave-1 railiance-forge rows of the 2026-10-21 node allocation review (CUST-WP-0071 T03/T04, RESOURCE-WP-0008). It gives the Forgejo main container and the in-cluster Actions runner explicit requests and bounds that match measured use. Today both run without any declared resources.

Sources:

• resource-control/docs/evidence/2026-10-21-node-allocation-review.md
• resource-control/docs/evidence/2026-10-21-node-allocation-review/2026-10-21-node-budget.{md,json} (rows forgejo/gitea, forgejo/forgejo-runner)
• the-custodian/docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md (wave 1 apply approval, one owner at a time, before/after plus rollback receipt)
• BEFORE evidence: docs/evidence/2026-10-21-forgejo-resource-requests/

Deployment path (important)

The live Forgejo release and runner are deployed from railiance-apps, not from this repo:

• Helm release forgejo (chart gitea-charts/gitea 12.5.0) ← railiance-apps/helm/forgejo-values.yaml via make forgejo-deploy
• deployment/forgejo-runner ← railiance-apps/manifests/forgejo-runner.yaml via make forgejo-runner-deploy

railiance-forge/helm/gitea-values.sops.yaml and make gitea-deploy here are stale. They carry the read-only Gitea mirror settings and ROOT_URL=gitea.coulomb.social; FORGE-WP-0009 retires them. Never use make gitea-deploy from this repo for this change. It would overwrite live Forgejo configuration.

The change is therefore prepared here as a reviewed patch against railiance-apps: docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch (sha256 5496ceeb7634e7e70634376c3c4967890a39d56f8017b5c9d107bd5c3ff789f9, applies cleanly at railiance-apps 10465cb). Landing it in railiance-apps is part of T02.

Values

Workload / container	Before (live)	After	Basis (7d to 2026-10-21)
forgejo-gitea / gitea	none (pod effective 100m/128Mi from init containers)	req 200m / 1104Mi, limit mem 1536Mi, no CPU limit	CPU p95 133–160m, peak ~1030m; WS p95 956Mi, peak 1252Mi, RSS peak 500Mi
forgejo-runner / dind	none (BestEffort)	req 100m / 432Mi, limit 1500m / 2Gi	WS p95 392Mi, peak 693Mi; CPU peak 200m
forgejo-runner / runner	none (BestEffort)	req 25m / 64Mi, no limit	WS p95 56Mi, peak 459Mi; CPU peak 124m
runner capacity	capacity: 1 (live config.yaml)	unchanged	one job at a time already holds

Pod totals: Forgejo 200m / 1104Mi (QoS Burstable). Runner 125m / 496Mi (Burstable instead of BestEffort), which matches the review.

Decisions and deviations from the proposal:

• Gitea memory limit 1536Mi. This is 1.23x the highest per-pod working set (1252Mi, 2026-10-13 pod) and about 3x the peak RSS. The rest of the working set is page cache, which the kernel reclaims before an OOM kill. The review suggested "~1.5Gi" and the evidence supports it. Raise it to 2Gi if T03 shows a working set that stays at the limit, or any OOM.
• Gitea CPU request stays 200m. The independent 7d p95 here is 160m (review: 133m). 200m still covers it, so the reviewed value is kept. No CPU limit is set: clone/pack bursts reach ~1 core, and repo convention does not require CPU limits.
• The runner pod request is split per container (the review gives only the pod total): dind takes most of it because the job containers run inside dind.
• No limit on the runner container (out of reviewed scope). Jobs with :host labels (container-build, registry-publish, …) run inside the runner container itself, which explains its 459Mi peak. A memory limit there (suggest 1Gi) needs a separate review, because an OOM would kill the runner daemon in the middle of a job.

Tasks

id: FORGE-WP-0011-T01
status: done
priority: high
state_hub_task_id: "d570265b-d4d5-5dfa-91d3-070c91479cd7"

Capture BEFORE state read-only and prepare the reviewed change.

Done 2026-10-21: evidence in docs/evidence/2026-10-21-forgejo-resource-requests/. The patch was rendered on a scratch copy of railiance-apps with helm template forgejo gitea-charts/gitea --version 12.5.0 -f helm/forgejo-values.yaml -f helm/forgejo-registry-values.yaml (rc 0; the SOPS layer was not rendered) and with kubectl apply --dry-run=client -f manifests/forgejo-runner.yaml. The rendered resources are in after-render-resources.txt.

id: FORGE-WP-0011-T02
status: done
priority: high
state_hub_task_id: "d4162309-9869-5534-adf2-0417c40ce5d6"

Land the patch in railiance-apps and apply, one workload at a time, under the wave-1 approval in the delivery interview.

Downtime: the Forgejo Deployment uses strategy: Recreate on an RWO local-path PVC. A rollout therefore stops Forgejo before the new pod starts, so git/web/API/registry are down for about 1–3 minutes (init containers plus startup). This is acceptable. Pick a quiet moment, with no running Actions job and no release in flight. No PVC or StatefulSet recreation is needed. Both are plain Deployment template changes, and the PVCs gitea-shared-storage and forgejo-runner-data are reused. The runner Deployment is also Recreate, so a running job would be killed. Check first that no job is running.

Node headroom: the change adds +225m CPU and +1472Mi memory of requests. That takes the node to about 3830m / 4000m (96%) CPU and about 11.3Gi / 15.6Gi memory, as of the BEFORE snapshot. Recreate frees the old pod's request first, so scheduling will fit. Still, recheck kubectl describe node right before apply: if other wave-1 increases landed first and allocated CPU is above ~3775m, the new pod can stay Pending. In that case, apply wave-1 decreases first or roll back.

Deploy (railiance-apps checkout, operator kubeconfig):

cd ~/railiance-apps
git pull --ff-only
git apply ~/railiance-forge/docs/changes/FORGE-WP-0011/railiance-apps-forgejo-resources.patch
git diff --stat   # helm/forgejo-values.yaml, manifests/forgejo-runner.yaml
git commit -am "FORGE-WP-0011: size Forgejo and runner requests (CUST-WP-0071)"
git push

# 0. preflight: no running Actions job, node headroom
KUBECONFIG=~/.kube/config-railiance01 kubectl describe node | sed -n '/Allocated resources/,/Events/p'
KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo logs deploy/forgejo-runner -c runner --tail=20

# 1. Forgejo (Helm upgrade; restarts Forgejo; --wait 10m)
make forgejo-deploy

# 2. Runner (kubectl apply; Recreate)
make forgejo-runner-deploy

Note on Helm drift: the release history ends at rev 9 (2026-07-07), but the Deployment has had 5 later rollouts that were not made through Helm. make forgejo-deploy renders the full railiance-apps HEAD values. Review helm diff (if installed) or compare the helm template ... | yq 'select(.kind=="Deployment")' output with the live Deployment before upgrading. Expect only the resources delta plus any Helm-label churn.

Rollback receipt (record the Helm revision created in step 1):

# Forgejo: back to the previous release revision (restarts Forgejo again)
KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo history forgejo --max 3
KUBECONFIG=~/.kube/config-railiance01 helm -n forgejo rollback forgejo <previous-rev> --wait --timeout 10m
# Runner: revert the manifest and re-apply
cd ~/railiance-apps && git revert --no-edit <commit> && git push && make forgejo-runner-deploy
# Emergency runner-only revert without git:
KUBECONFIG=~/.kube/config-railiance01 kubectl -n forgejo rollout undo deployment/forgejo-runner

helm rollback to rev 9 restores the July-rendered template, which is older than the October live state (secret-scanner init container, PKCE). Prefer git revert + make forgejo-deploy for Forgejo. Use helm rollback only to the revision created immediately before this change.

id: FORGE-WP-0011-T03
status: done
priority: high
depends_on: [FORGE-WP-0011-T02]
state_hub_task_id: "1fe5a35d-c9e0-562c-9873-951943c36268"

Verify right after apply, and again after 24h and 7d. Store the outputs as docs/evidence/2026-10-21-forgejo-resource-requests/after-*.txt.

K="kubectl --kubeconfig ~/.kube/config-railiance01 -n forgejo"
$K get deploy forgejo-gitea forgejo-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]}  {.name} {.resources}{"\n"}{end}{end}'
$K get pods -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,QOS:.status.qosClass,RESTARTS:.status.containerStatuses[*].restartCount,LAST:.status.containerStatuses[*].lastState.terminated.reason
# Forgejo UI / API / git
curl -fsS https://forgejo.coulomb.social/api/healthz
curl -fsS https://forgejo.coulomb.social/api/v1/version
git ls-remote https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git HEAD
git clone --depth 1 https://forgejo.coulomb.social/coulomb/forgejo-actions-probe.git /tmp/fwp11-clone && rm -rf /tmp/fwp11-clone
# Runner picks a job: re-run the probe workflow (container-probe / image-build) in
# coulomb/forgejo-actions-probe from the UI, then:
$K logs deploy/forgejo-runner -c runner --tail=50 | grep -iE "task|job"
# No OOM / throttling (Prometheus, 24h and 7d after)
#   increase(container_oom_events_total{namespace="forgejo"}[24h]) == 0
#   max_over_time(container_memory_working_set_bytes{namespace="forgejo",container="gitea"}[24h]) < 1536Mi
#   rate(container_cpu_cfs_throttled_periods_total{namespace="forgejo",container="dind"}[5m]) during a build

Done when Forgejo answers on UI/API/git clone, a runner job completes, and no OOMKilled appears after 7d. Then report the after-values back to resource-control RESOURCE-WP-0008 and CUST-WP-0071 T04.

id: FORGE-WP-0011-T04
status: todo
priority: low
state_hub_task_id: "851f2ffe-3d66-51fd-8c40-83c34b117713"

Follow-ups outside the reviewed wave-1 diff (each needs its own approval): a memory limit for the runner container (host-label jobs), removal of the two stale Failed pods from 66 days ago, and reconciling Helm release drift (rev 9 vs Deployment rev 14). The Prometheus forgejo request recording rule that returns no series is owned by rapp-telemetry.

Runner applied, server deferred — 2026-10-21

The founder ran a diff in their own terminal, rendered with the same inputs as make forgejo-deploy and compared with live. The agent reviewed only a redacted summary of it.

• Runner: the diff showed only resources. Applied with make forgejo-runner-deploy (railiance-apps 4a1d653); the runner daemon and poller are up.
• Server: the Helm upgrade would also apply the NetKingdom sign-in declaration from railiance-apps 4d6a8dd (the oauth source and the oauth2_client app.ini section). That source was activated natively on 19 Oct (CUST-WP-0080 T42, source ID 1), and founder linking (T43) is still open. The founder chose to defer the server upgrade to a Helm reconciliation coordinated with the sign-in owners (RAPPS-WP-0016 / CUST-WP-0080 T43). Until then, the server requests declared in helm/forgejo-values.yaml are not live.
• Naming: forgejo-gitea and the gitea container come from the upstream chart gitea-charts/gitea, not the retired Gitea. Docs should say Forgejo. Renaming the live objects means recreating them, so it is a follow-up for a maintenance window. Evidence: the-custodian docs/evidence/2026-10-21-wave1-apply.md.

Server reconciled and verified — 2026-10-21

The founder ran the final step of make forgejo-deploy in their own terminal: helm upgrade --install forgejo gitea-charts/gitea 12.5.0 with the three values inputs. It skipped the eso-deploy chain, which would also have upgraded the External Secrets Operator. Its prerequisite manifests (mailer ExternalSecret, secret-scan template, OpenBao store) were diffed first and matched live.

• Helm revision 10, deployed. Helm and live now match, closing the drift between revision 9 (July) and the out-of-band rollouts.
• Deployment revision 15, Ready. Forgejo 11.0.3; requests 200m/1104Mi, limit 1536Mi.
• /api/healthz returns 200, the version API answers, and git ls-remote works.
• Sign-in: still exactly one source (ID 1 netkingdom, OAuth2, enabled). The chart init step took the update path ("sync settings done") with values identical to the 19 Oct native add.
• The login page offers local sign-in and netkingdom. The redirect goes to kc.coulomb.social with client forgejo, the HTTPS callback, S256 PKCE and scope openid profile email groups, listed once.
• Node requests: 3840m (96%) CPU, 78% memory.

Rollback reference, if ever needed: kubectl rollout undo deployment/forgejo-gitea -n forgejo --to-revision=14 (never helm rollback forgejo 9). The founder's own NetKingdom sign-in check is optional confirmation. Re-measurement after 24h/7d follows in the weekly allocation report.

Naming follow-up resolved by RAPPS-WP-0019 — 2026-10-21

The forge objects are now named forge, forge-http and forge-ssh, with pod label forge (railiance-apps RAPPS-WP-0019, Helm revision 11). Still left in T04: a runner container memory limit and the stale Error runner pod from 66 days ago. The data volume and the container names keep their chart-imposed legacy names.