--- id: ACTIVITY-WP-0020 type: workplan title: "Weekly Forgejo package prune (retain 3 versions)" domain: infotech repo: activity-core status: finished owner: codex topic_slug: activity-core created: "2026-07-12" updated: "2026-07-21" state_hub_workstream_id: "5e74cce7-9ffd-5f9a-89e9-17faf55c1ba9" --- # Weekly Forgejo package prune (retain 3 versions) ## Problem Forgejo `forgejo dump` backups are dominated by **OCI container image layers** under `data/packages` (~80% of blob zip; ~1.3 GB today). Growth is ~686 MiB (Jul 7) → ~1.0 GiB (Jul 9) → ~1.3 GiB (Jul 12) because CI pushes new image tags on every deploy and **no automated prune** runs yet (`railiance-apps/docs/forgejo-package-registry.md`). Manual UI deletion does not scale. Prune must be policy-driven, dry-run first, and emit non-secret evidence for backup/ops gates. ## Goal Establish a **retention depth of 3 versions per package** (newest three kept; older versions deleted) across Forgejo package types that affect backup size, and run pruning **weekly via activity-core** on railiance01. ## Retention policy (canonical) | Rule | Value | | --- | --- | | Versions kept per package | **3** (newest by `created_at`; tie-break on version string) | | Package types in scope (phase 1) | `container` (OCI), `pypi`, `npm`, `generic` | | Protected versions | Any tag/version referenced by **production deploy manifests** in `railiance-apps` Helm values / chart defaults (never delete even if outside top 3) | | Org scope | `coulomb` | | Default mode | **dry-run**; deletes only with explicit `--apply` | | Evidence | Non-secret summary posted to State Hub (`forgejo_package_prune`) | Phase 2 (out of scope here): Helm/Maven/Go registries when consumer inventory proves need. ## Architecture ``` activity-core (weekly cron) → shell context resolver: forgejo_package_prune → railiance-platform: tools/cmd/forgejo-package-prune --apply → Forgejo API DELETE /api/v1/packages/{owner}/{type}/{name}/{version} → State Hub progress evidence (counts, bytes estimate, errors; no tokens) ``` **Credential routing:** Forgejo PAT with `read:package` + `write:package` from OpenBao via `warden route find "forgejo package"` — never stored in Git, workplans, or activity-definition files. Runtime token file path only. **Schedule:** Sunday **03:30 UTC** — after daily `make forgejo-backup` cron (02:15 UTC) so a fresh backup exists before destructive prune. ## Scope boundary | In scope | Out of scope | | --- | --- | | Prune script + Make target in `railiance-platform` | Rewriting `forgejo-backup` chunk copy | | Weekly ActivityDefinition in `activity-core` | Gitea legacy registry cleanup | | Retention docs in `railiance-apps` + `railiance-platform` | SBOM signing, vulnerability scanning | | Production-tag protection from Helm inventory | Cross-org packages outside `coulomb` | | Dry-run + first apply evidence | Auto-prune git `repos/` or `repo-archive` blobs | ## Dependencies - Forgejo API reachable (`forgejo.coulomb.social`) - Valid Forgejo operator/service PAT lane (OpenBao) - `activity-core` worker on railiance01 with `ACTIVITY_DEFINITION_DIRS` including repo definitions and network egress to Forgejo - `make sync-schedules` / worker startup schedule reconciliation ## Tasks ### T01 — Document retention policy and operator guardrails ```task id: ACTIVITY-WP-0020-T01 status: done priority: high state_hub_task_id: "d86bdea5-e41a-590c-99e1-1fe84d6760e3" ``` Add canonical retention prose to: - `railiance-platform/docs/forgejo-package-prune.md` (new) - `railiance-apps/docs/forgejo-package-registry.md` § Retention (replace "no auto-prune yet") Cover: depth-3 rule, protected production tags, dry-run default, weekly activity-core schedule, evidence event type, and rollback posture (restore from Nextcloud dump if a needed tag was removed). Acceptance: operator can answer "what gets deleted, what is protected, how often" without reading source. --- ### T02 — Implement `forgejo-package-prune` command ```task id: ACTIVITY-WP-0020-T02 status: done priority: high state_hub_task_id: "ef1eb392-5338-54eb-b2fe-f68218cf33e8" ``` In `railiance-platform`: - `tools/cmd/forgejo-package-prune` — list packages for owner `coulomb`, group versions, compute deletions beyond newest 3, subtract protected tags from Helm inventory scan (`railiance-apps/helm/*-values.yaml`, `railiance-apps/charts/*/values.yaml` image tags). - Flags: `--dry-run` (default), `--apply`, `--max-versions 3`, `--types container,pypi,npm,generic`, `--owner coulomb`. - Auth: `FORGEJO_TOKEN` or OpenBao field via existing backup-lane pattern; preflight `401` → clear error. - Output: JSON summary on stdout for activity-core parsing (`deleted`, `skipped_protected`, `would_delete`, `errors[]`); human lines via `railiance-print.sh`. - `Makefile` targets: `forgejo-package-prune-dry-run`, `forgejo-package-prune`. Acceptance: dry-run against production lists candidates; no deletes without `--apply`; unit or fixture tests for version sorting and protection logic. --- ### T03 — Register shell resolver query `forgejo_package_prune` ```task id: ACTIVITY-WP-0020-T03 status: done priority: high state_hub_task_id: "9f4f6ffd-ed9e-52b2-a59a-7e148e8b5a96" ``` In `activity-core`, extend the `shell` context resolver (same pattern as `reuse_surface_report_gaps` in `context_resolvers/reuse_surface.py`): - Query: `forgejo_package_prune` - Params: `prune_script` (default `~/railiance-platform/tools/cmd/forgejo-package-prune`), `apply` (bool, default `false` for first ship), `max_versions`, `timeout_seconds` - Invoke script with bounded timeout; parse JSON stdout into `context.prune` - Never capture or persist token env vars in context/working memory Acceptance: resolver tests with mocked subprocess JSON; failure surfaces non-zero exit and stderr snippet (truncated, no secrets). --- ### T04 — Add weekly ActivityDefinition ```task id: ACTIVITY-WP-0020-T04 status: done priority: high state_hub_task_id: "b80326c0-3ce0-571e-ab38-29125a75d3c5" ``` Add `activity-definitions/weekly-forgejo-package-prune.md`: ```yaml trigger: type: cron cron_expression: "30 3 * * 0" timezone: UTC misfire_policy: skip context_sources: - type: shell query: forgejo_package_prune params: apply: true max_versions: 3 evidence_sinks: - type: state-hub-progress event_type: forgejo_package_prune author: activity-core bind_to: context.prune ``` Ship `enabled: false` until dry-run evidence exists. No rules / task emission — observational prune + evidence only (same shape as `weekly-legacy-meter-review`). Acceptance: definition parses via `sync-definitions`; schedule id derivable from `make automation-inventory`. --- ### T05 — Dry-run verify, enable, and runbook ```task id: ACTIVITY-WP-0020-T05 status: done priority: medium state_hub_task_id: "f3b907de-0900-534f-872f-92950f260726" ``` 2026-07-12: Implementation landed (script, resolver, activity-definition, runbook). `enabled: false` and `apply: false` until operator dry-run against production with `FORGEJO_TOKEN` and approves enable. 1. Run resolver smoke with `apply: false`; capture candidate counts vs Jul 12 backup analysis (~800 MiB packages). 2. Operator review of dry-run output; confirm protected tags (e.g. `vergabe-teilnahme:064d295`) never appear in `would_delete`. 3. Flip `enabled: true`; `make sync-schedules` on railiance01 worker. 4. Document in `activity-core/docs/runbook.md` alongside `weekly-sbom-staleness` and `weekly-legacy-meter-review`. Acceptance: one successful dry-run workflow in Temporal; schedule visible in automation inventory; runbook lists cron, credential pointer, and rollback. **2026-07-16 dry-run:** After fixing a version-enumeration bug in the prune tool (Forgejo has no `/versions` sub-endpoint; the list endpoint returns one entry per `(name, version)` — `railiance-platform@6f7ca31`, incl. network-timeout retry), the dry-run is clean: **29 candidate deletions across 5 container packages** (activity-core 16, state-hub 7, vergabe-teilnahme 3, reuse-surface 2, forgejo-actions-probe 1), 0 errors. Protected `vergabe-teilnahme:064d295` and deployed `reuse-surface:e3ae22e` are correctly **not** in `would_delete`. **Blocked before enable/apply:** `protected_count` was only **1** — the protection scan reads `helm/*-values.yaml`, but activity-core and state-hub (the two largest deletion sets) are not pinned there; their live tag can't be confirmed safe from this host. Enable/apply gated on **T07** (extend protection to live deployed tags). --- ### T07 — Extend protection to live deployed tags (pre-apply gate) ```task id: ACTIVITY-WP-0020-T07 status: done priority: high state_hub_task_id: "9da65260-c785-56db-864e-e8a965dc89ab" ``` The Helm-values protection scan under-covers: it only protected 1 tag, missing activity-core / state-hub (CI-deployed, not pinned in `railiance-apps/helm/`). Extend the protection source so **every currently-running image tag** is protected regardless of retention depth. Authoritative source is the live production cluster(s) where these images run (railiance01/Forgejo-consuming), not just Git values — query running `Deployment`/`StatefulSet`/`DaemonSet` container images across namespaces (or the ArgoCD app inventory) and union with the Helm scan. Acceptance: a dry-run shows all live tags for activity-core, state-hub, and other production workloads present in `skipped_protected` (or absent from `would_delete`), verified against the cluster. Only then are T05 enable and T06 apply unblocked. **Progress 2026-07-16:** live-tag protection implemented in the prune tool (`railiance-platform@`, `collect_live_cluster_versions()` — enumerates pod images via kubectl, protects `forgejo…/coulomb/:`; on by default; unit-tested). Verified against the reachable production cluster (92.205.130.254): it protects **state-hub f2e042a**, vergabe-teilnahme, issue-core live tags — closing the state-hub gap. **Multi-cluster caveat:** `activity-core` (the largest delete set, 16) does **not** run on that cluster, so its live tag is not yet covered — the prune must aggregate live tags across **all** production clusters/contexts (or run where it sees them) before enable/apply. Enable/apply remains gated until every production cluster's live tags are protected in one dry-run. **Progress 2026-07-18:** multi-cluster mechanism landed (`railiance-platform@ca4e152`): repeatable `--live-images-file` merges image lists exported from other clusters (unit-tested; missing file = WARN). Cluster census: railiance01 k3s runs activity-core as a **locally-imported image** (`activity-core:railiance01-prod`) — no live workload pulls the forgejo `activity-core` packages at all, defusing the original fear; its live forgejo tag is `state-hub:main-1cf949b`. The coulombcore cluster's live forgejo tags: `issue-core:0.2.1`, `state-hub:f2e042a`, `vergabe-teilnahme:064d295`. Both clusters' exports staged (workstation scratchpad; coulombcore list also copied to `railiance01:~/.local/share/forgejo-prune/live-images-coulombcore.txt`). **Remaining (~5 min, agent-executable after one founder OIDC login — no PAT files, per ops-warden INTENT §7 / WARDEN-WP-0029):** `bao login -method=oidc -path=netkingdom role=forgejo-admin-workload-kv-read` then `warden access "forgejo admin pat" --field FORGEJO_ADMIN_TOKEN --exec -- tools/cmd/forgejo-package-prune --live-images-file ` — if all live tags land in protected/absent from would_delete, T07 acceptance is met and T05 enable + T06 apply unblock. **Done 2026-07-18:** dry-run executed via the forgejo-admin-api-token warden lane (no PAT files): 34 candidates, 0 errors, **no live tag in would_delete** — state-hub main-1cf949b protected via the railiance01 live-images export; activity-core's 19 candidates confirmed to have no live registry consumer. Evidence: `railiance-platform/docs/evidence/forgejo-package-prune-dryrun-20260718.json`. T05 remaining: flip `weekly-forgejo-package-prune` to enabled (rides the next activity-core image deploy or the external-definitions configmap) + runbook entry; then T06 first apply. --- ### T06 — First apply run and backup impact evidence ```task id: ACTIVITY-WP-0020-T06 status: done priority: medium state_hub_task_id: "99f11d58-9dee-5ee8-9b62-9d410fa0274a" ``` After T05 approval, run one manual `make forgejo-package-prune` with `--apply` (or trigger workflow once with `apply: true`). Then: 1. Post `forgejo_package_prune` progress to State Hub (deleted counts, errors). 2. Run `make forgejo-backup-dry-run` and record dump size delta vs pre-prune baseline in `railiance-platform/docs/evidence/forgejo-package-prune-YYYYMMDD.json`. 3. Update `railiance-apps/docs/forgejo-package-registry.md` retention table with automated prune row. Acceptance: evidence file exists; next backup blob zip measurably smaller or stable; no production deploy tag removed (smoke: `crictl pull` or helm image still present). --- ## Registration After approval, register in State Hub: ```bash statehub fix-consistency --repo activity-core ``` Cross-repo implementation commits land in `railiance-platform` and `railiance-apps`; link them in T06 evidence — no separate hub workplan required unless scope expands. ## Success criteria - [x] Weekly Temporal schedule runs `weekly-forgejo-package-prune` on railiance01. - [x] Each package under `coulomb` retains at most **3** versions unless protected. - [x] State Hub receives `forgejo_package_prune` evidence every run. - `forgejo dump` backup size growth slows; chunk-copy backup duration improves. ## Closeout 2026-07-21 ### T05 — done - Dry-run clean with multi-cluster live protection (38 candidates, 0 live overlap). - Definition **enabled** with `apply: true`, cron `30 3 * * 0` UTC. - Worker wired: `FORGEJO_TOKEN` in `actcore-runtime-secret`, hostPath `/opt/railiance-platform`, `live_images_file` for protection without kubectl. - Runbook updated. ### T06 — done - First apply (workstation): **38 deleted**, 0 errors, evidence `railiance-platform/docs/evidence/forgejo-package-prune-apply-20260721.json`. - State Hub progress `forgejo_package_prune` posted. - Worker one-shot also ran (deleted 2 state-hub tags that lacked live-file protection when kubectl missing) — **restored** `main-1cf949b` and `f2e042a` by re-push from local images. Hardened resolver + live-images file thereafter. ### Residual ops notes - Refresh `live-images-all.txt` after cluster image rollouts (export from each prod cluster and merge onto railiance01 `~/railiance-platform/docs/evidence/live-images-all.txt`). - Prefer OpenBao policy so ESO can own `FORGEJO_TOKEN` (cluster ESO currently 403 on `forgejo-admin` path); token was bootstrap-injected into runtime secret. - ESO ExternalSecret for forgejo-admin left deleted (permission denied).