feat(forgejo): add package prune script with retention depth 3
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

List and optionally delete package versions beyond the newest three,
protecting production Helm image tags. Adds Make targets and unit tests
for ACTIVITY-WP-0020.
This commit is contained in:
tegwick 2026-07-12 11:35:04 +02:00
parent 2806f2cdda
commit 715631dedd
5 changed files with 564 additions and 1 deletions

View file

@ -375,6 +375,12 @@ forgejo-backup-status: ## Show last Forgejo backup success and 7-day gate
echo "Recent successes:"; tail -7 "$$STAMP"; \
echo "7-day gate: $$(tail -7 "$$STAMP" | wc -l)/7 consecutive days logged (verify cron separately)"
forgejo-package-prune-dry-run: ## List Forgejo package versions beyond retention depth (no deletes)
tools/cmd/forgejo-package-prune
forgejo-package-prune: ## Prune Forgejo packages — keep newest 3 versions per package
tools/cmd/forgejo-package-prune --apply
##@ Help
help: ## Show this help
@ -382,4 +388,4 @@ help: ## Show this help
/^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-22s\033[0m %s\n", $$1, $$2 } \
/^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST)
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status help
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run help

View file

@ -0,0 +1,46 @@
# Forgejo package prune (retention)
Workplan: `ACTIVITY-WP-0020` · complements `docs/forgejo-backup.md`
## Policy
| Rule | Value |
| --- | --- |
| Versions kept per package | **3** (newest by `created_at`) |
| Package types (phase 1) | `container`, `pypi`, `npm`, `generic` |
| Org scope | `coulomb` |
| Protected versions | Image tags referenced in `railiance-apps` production Helm values / chart defaults |
| Default mode | dry-run (no deletes) |
| Schedule | Weekly Sunday 03:30 UTC via `activity-core` (`weekly-forgejo-package-prune`) |
OCI container layers dominate `forgejo dump` size (~80% of blob zip). Pruning old
tags slows backup growth and shortens chunked-copy duration.
## Operator commands
```bash
cd ~/railiance-platform
export FORGEJO_TOKEN=... # or FORGEJO_TOKEN_FILE=/path/to/token
make forgejo-package-prune-dry-run # list would-delete candidates
make forgejo-package-prune # delete beyond retention depth
```
Requires a Forgejo PAT with `read:package` and `write:package`. Resolve custody via
`warden route find "forgejo package token"` — do not store tokens in Git.
## Rollback
If a needed tag was removed, restore from the latest Nextcloud `forgejo-dump-*.zip.age`
(`make forgejo-backup` evidence) using `railiance-infra/tools/forgejo-restore-drill.sh`,
or re-push the image from CI.
## Evidence
activity-core posts `forgejo_package_prune` progress to State Hub with non-secret
counts (`deleted_count`, `candidate_count`, `skipped_protected_count`, `errors`).
## Related
- `railiance-apps/docs/forgejo-package-registry.md`
- `docs/forgejo-backup.md`

View file

@ -0,0 +1,439 @@
#!/usr/bin/env python3
"""Forgejo package retention prune — keep newest N versions per package."""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
DEFAULT_BASE = "https://forgejo.coulomb.social"
DEFAULT_OWNER = "coulomb"
DEFAULT_TYPES = ("container", "pypi", "npm", "generic")
DEFAULT_MAX_VERSIONS = 3
DEFAULT_APPS_ROOT = Path.home() / "railiance-apps"
FORGEJO_IMAGE_RE = re.compile(
r"^forgejo\.coulomb\.social/(?:coulomb/)?(?P<name>[^:/]+)(?::(?P<tag>[^/\s]+))?$",
re.IGNORECASE,
)
@dataclass(frozen=True)
class VersionRef:
package_type: str
name: str
version: str
def key(self) -> tuple[str, str, str]:
return (self.package_type, self.name, self.version)
@dataclass(frozen=True)
class DeletePlan:
package_type: str
name: str
version: str
created_at: str
protected: bool
reason: str
def _parse_created_at(value: str | None) -> datetime:
if not value:
return datetime.min
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return datetime.min
def collect_protected_versions(apps_root: Path) -> set[tuple[str, str, str]]:
protected: set[tuple[str, str, str]] = set()
if not apps_root.is_dir():
return protected
patterns = [
apps_root / "helm" / "*-values.yaml",
apps_root / "charts" / "*" / "values.yaml",
]
paths: list[Path] = []
for pattern in patterns:
paths.extend(sorted(pattern.parent.glob(pattern.name)))
try:
import yaml # type: ignore
except ImportError:
yaml = None
for path in paths:
text = path.read_text(encoding="utf-8")
if yaml is not None:
try:
data = yaml.safe_load(text) or {}
except Exception:
data = {}
image = data.get("image") if isinstance(data, dict) else None
if isinstance(image, dict):
repo = str(image.get("repository") or "").strip()
tag = str(image.get("tag") or "").strip()
if repo and tag:
match = FORGEJO_IMAGE_RE.match(repo) or FORGEJO_IMAGE_RE.match(
f"{repo}:{tag}"
)
if match:
name = match.group("name")
protected.add(("container", name, tag))
continue
repo_match = re.search(
r"repository:\s*forgejo\.coulomb\.social/coulomb/([^\s]+)",
text,
re.IGNORECASE,
)
tag_match = re.search(r'^\s*tag:\s*"?([^"\s#]+)"?\s*$', text, re.MULTILINE)
if repo_match and tag_match:
protected.add(("container", repo_match.group(1), tag_match.group(1)))
return protected
def _api_request(
method: str,
url: str,
token: str,
*,
timeout: float = 30.0,
) -> Any:
req = urllib.request.Request(
url,
method=method,
headers={
"Authorization": f"token {token}",
"Accept": "application/json",
},
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read().decode("utf-8")
return json.loads(body) if body else None
def _paginate(
base_url: str,
token: str,
path: str,
*,
page_size: int = 50,
) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
page = 1
while True:
query = urllib.parse.urlencode({"limit": page_size, "page": page})
url = f"{base_url.rstrip('/')}{path}?{query}"
payload = _api_request("GET", url, token)
if not payload:
break
if isinstance(payload, list):
batch = payload
else:
batch = payload.get("data") or payload.get("items") or []
if not batch:
break
items.extend(batch)
if len(batch) < page_size:
break
page += 1
return items
def list_packages(
base_url: str,
token: str,
owner: str,
package_type: str,
) -> list[dict[str, Any]]:
owner_q = urllib.parse.quote(owner)
items: list[dict[str, Any]] = []
page = 1
page_size = 50
while True:
query = urllib.parse.urlencode(
{"limit": page_size, "page": page, "type": package_type}
)
url = f"{base_url.rstrip('/')}/api/v1/packages/{owner_q}?{query}"
payload = _api_request("GET", url, token)
batch = payload if isinstance(payload, list) else []
if not batch:
break
items.extend(batch)
if len(batch) < page_size:
break
page += 1
return items
def list_versions(
base_url: str,
token: str,
owner: str,
package_type: str,
name: str,
) -> list[dict[str, Any]]:
owner_q = urllib.parse.quote(owner)
type_q = urllib.parse.quote(package_type)
name_q = urllib.parse.quote(name, safe="")
return _paginate(
base_url,
token,
f"/api/v1/packages/{owner_q}/{type_q}/{name_q}/versions",
)
def delete_version(
base_url: str,
token: str,
owner: str,
package_type: str,
name: str,
version: str,
*,
dry_run: bool,
) -> None:
owner_q = urllib.parse.quote(owner)
type_q = urllib.parse.quote(package_type)
name_q = urllib.parse.quote(name, safe="")
version_q = urllib.parse.quote(version, safe="")
path = f"/api/v1/packages/{owner_q}/{type_q}/{name_q}/{version_q}"
if dry_run:
return
url = f"{base_url.rstrip('/')}{path}"
_api_request("DELETE", url, token)
def build_delete_plans(
*,
base_url: str,
token: str,
owner: str,
package_types: list[str],
max_versions: int,
protected: set[tuple[str, str, str]],
) -> tuple[list[DeletePlan], list[str]]:
plans: list[DeletePlan] = []
errors: list[str] = []
for package_type in package_types:
try:
packages = list_packages(base_url, token, owner, package_type)
except urllib.error.HTTPError as exc:
errors.append(f"list {package_type}: HTTP {exc.code}")
continue
except Exception as exc: # noqa: BLE001
errors.append(f"list {package_type}: {exc}")
continue
for package in packages:
name = str(package.get("name") or package.get("package_name") or "")
if not name:
continue
try:
versions = list_versions(base_url, token, owner, package_type, name)
except urllib.error.HTTPError as exc:
errors.append(f"versions {package_type}/{name}: HTTP {exc.code}")
continue
except Exception as exc: # noqa: BLE001
errors.append(f"versions {package_type}/{name}: {exc}")
continue
sorted_versions = sorted(
versions,
key=lambda item: _parse_created_at(str(item.get("created_at") or "")),
reverse=True,
)
keep = {
str(item.get("version") or "")
for item in sorted_versions[:max_versions]
if str(item.get("version") or "")
}
for item in sorted_versions[max_versions:]:
version = str(item.get("version") or "")
if not version or version in keep:
continue
key = (package_type, name, version)
is_protected = key in protected
plans.append(
DeletePlan(
package_type=package_type,
name=name,
version=version,
created_at=str(item.get("created_at") or ""),
protected=is_protected,
reason="protected_production_tag" if is_protected else "beyond_retention_depth",
)
)
return plans, errors
def load_token() -> str:
token = os.environ.get("FORGEJO_TOKEN", "").strip()
if token:
return token
token_file = os.environ.get("FORGEJO_TOKEN_FILE", "/tmp/forgejo-tegwick-api-token")
path = Path(token_file).expanduser()
if path.is_file():
return path.read_text(encoding="utf-8").strip()
raise SystemExit(
"ERROR: set FORGEJO_TOKEN or FORGEJO_TOKEN_FILE "
"(Forgejo PAT with read:package and write:package)"
)
def emit_summary(
*,
owner: str,
max_versions: int,
package_types: list[str],
protected: set[tuple[str, str, str]],
plans: list[DeletePlan],
errors: list[str],
apply: bool,
deleted: list[DeletePlan],
) -> dict[str, Any]:
would_delete = [p for p in plans if not p.protected]
skipped_protected = [p for p in plans if p.protected]
return {
"kind": "forgejo_package_prune",
"owner": owner,
"max_versions": max_versions,
"package_types": package_types,
"protected_count": len(protected),
"candidate_count": len(would_delete),
"skipped_protected_count": len(skipped_protected),
"deleted_count": len(deleted),
"apply": apply,
"would_delete": [
{
"type": p.package_type,
"name": p.name,
"version": p.version,
"created_at": p.created_at,
}
for p in would_delete
],
"skipped_protected": [
{
"type": p.package_type,
"name": p.name,
"version": p.version,
"reason": p.reason,
}
for p in skipped_protected
],
"deleted": [
{
"type": p.package_type,
"name": p.name,
"version": p.version,
}
for p in deleted
],
"errors": errors,
}
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Prune old Forgejo package versions")
parser.add_argument("--owner", default=DEFAULT_OWNER)
parser.add_argument("--base-url", default=os.environ.get("FORGEJO_BASE_URL", DEFAULT_BASE))
parser.add_argument("--max-versions", type=int, default=DEFAULT_MAX_VERSIONS)
parser.add_argument(
"--types",
default=",".join(DEFAULT_TYPES),
help="Comma-separated package types",
)
parser.add_argument("--apps-root", type=Path, default=DEFAULT_APPS_ROOT)
parser.add_argument("--apply", action="store_true")
parser.add_argument("--json", action="store_true", help="Emit JSON summary on stdout")
args = parser.parse_args(argv)
apply = bool(args.apply)
dry_run = not apply
package_types = [part.strip() for part in args.types.split(",") if part.strip()]
token = load_token()
protected = collect_protected_versions(args.apps_root.expanduser())
print(f"Forgejo package prune — owner={args.owner} keep={args.max_versions}", file=sys.stderr)
print(f"Protected production tags: {len(protected)}", file=sys.stderr)
plans, errors = build_delete_plans(
base_url=args.base_url,
token=token,
owner=args.owner,
package_types=package_types,
max_versions=max(1, args.max_versions),
protected=protected,
)
deleted: list[DeletePlan] = []
for plan in plans:
if plan.protected:
print(
f" skip protected {plan.package_type}/{plan.name}:{plan.version}",
file=sys.stderr,
)
continue
if dry_run:
print(
f" would delete {plan.package_type}/{plan.name}:{plan.version}",
file=sys.stderr,
)
continue
try:
delete_version(
args.base_url,
token,
args.owner,
plan.package_type,
plan.name,
plan.version,
dry_run=False,
)
deleted.append(plan)
print(
f" deleted {plan.package_type}/{plan.name}:{plan.version}",
file=sys.stderr,
)
except urllib.error.HTTPError as exc:
errors.append(f"delete {plan.package_type}/{plan.name}:{plan.version}: HTTP {exc.code}")
except Exception as exc: # noqa: BLE001
errors.append(f"delete {plan.package_type}/{plan.name}:{plan.version}: {exc}")
summary = emit_summary(
owner=args.owner,
max_versions=args.max_versions,
package_types=package_types,
protected=protected,
plans=plans,
errors=errors,
apply=apply,
deleted=deleted,
)
if args.json or not sys.stdout.isatty():
print(json.dumps(summary, indent=2))
else:
print(json.dumps(summary, indent=2))
return 1 if errors else 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,67 @@
from __future__ import annotations
import importlib.util
import sys
import unittest
from pathlib import Path
from unittest import mock
REPO_DIR = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location(
"forgejo_package_prune", REPO_DIR / "scripts/forgejo_package_prune.py"
)
prune = importlib.util.module_from_spec(SPEC)
assert SPEC.loader is not None
sys.modules[SPEC.name] = prune
SPEC.loader.exec_module(prune)
class ForgejoPackagePruneTests(unittest.TestCase):
def test_collect_protected_versions_from_helm_values(self) -> None:
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
helm = root / "helm"
helm.mkdir()
(helm / "vergabe-teilnahme-values.yaml").write_text(
"""
image:
repository: forgejo.coulomb.social/coulomb/vergabe-teilnahme
tag: "064d295"
""".lstrip(),
encoding="utf-8",
)
protected = prune.collect_protected_versions(root)
self.assertIn(("container", "vergabe-teilnahme", "064d295"), protected)
def test_build_delete_plans_respects_depth_and_protection(self) -> None:
protected = {("container", "vergabe-teilnahme", "old-prod")}
versions = [
{"version": "v4", "created_at": "2026-07-12T10:00:00Z"},
{"version": "v3", "created_at": "2026-07-11T10:00:00Z"},
{"version": "v2", "created_at": "2026-07-10T10:00:00Z"},
{"version": "v1", "created_at": "2026-07-09T10:00:00Z"},
{"version": "old-prod", "created_at": "2026-07-08T10:00:00Z"},
]
with mock.patch.object(prune, "list_packages", return_value=[{"name": "vergabe-teilnahme"}]):
with mock.patch.object(prune, "list_versions", return_value=versions):
plans, errors = prune.build_delete_plans(
base_url="https://forgejo.example",
token="token",
owner="coulomb",
package_types=["container"],
max_versions=3,
protected=protected,
)
self.assertEqual(errors, [])
deletable = {(p.name, p.version) for p in plans if not p.protected}
protected_plans = {(p.name, p.version) for p in plans if p.protected}
self.assertEqual(deletable, {("vergabe-teilnahme", "v1")})
self.assertEqual(protected_plans, {("vergabe-teilnahme", "old-prod")})
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,5 @@
#!/usr/bin/env bash
# tools/cmd/forgejo-package-prune — retention prune for Forgejo packages (ACTIVITY-WP-0020)
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
exec python3 "${ROOT}/scripts/forgejo_package_prune.py" --json "$@"