the-custodian/docs/forgejo-production-decisions.md

# Forgejo Production Decisions (Wave 1)

Date: 2026-07-03  
Workplans: `RAIL-HO-WP-0005-T02`, `CUST-WP-0054-T04`  
Operator input: 2026-07-03

## Decision log

| # | Topic | Decision | Status | Evidence (2026-07-03) |
| --- | --- | --- | --- | --- |
| 1 | **Production hostname** | `forgejo.coulomb.social` | **decided** | DNS A → `92.205.62.239` (railiance01); HTTPS reaches Traefik on railiance01 |
| 2 | Exposure model | Private HTTPS via railiance01 Traefik ingress + cert-manager `letsencrypt-prod` | **decided** | Same pattern as Gitea (`manifests/forgejo-ingress.yaml` in `railiance-apps`) |
| 2b | Deployment pattern | `gitea-charts/gitea` **12.5.0** Helm + Forgejo image; CNPG `forgejo-db` in `railiance-platform`; Makefile in `railiance-apps` | **decided** | Chart 12.6+ requires Gitea 1.26 `config edit-ini` (incompatible with Forgejo 11); see `railiance-apps/docs/forgejo-on-railiance01.md` |
| 2c | Live deploy | Forgejo pod + ingress + TLS on railiance01 | **done** (2026-07-03) | `make forgejo-smoke` → HTTP 200 + OCI `/v2/` 401 challenge; cert `forgejo-tls` Ready |
| 3 | Gitea during transition | `gitea.coulomb.social` on coulombcore remains canonical **until** Forgejo restore/migration drills pass; then read-only mirror | unchanged | Per `RAIL-HO-WP-0005` safety contract |
| 4 | SMTP / password reset | **IONOS** SMTP; sender `forgejo@coulomb.social`; OpenBao/ESO mailer on railiance01 | **done** (2026-07-08) | `smtp.ionos.de:587` + `smtp+starttls`; T06 verified |
| 5 | Package registry scope | **Multi-ecosystem (Option C):** OCI + **npm** + generic; PyPI/Go/Maven/Helm as inventory proves need | **decided** (2026-07-08) | OCI live; npm is launch requirement — see § Package scope |
| 6 | Actions runner model | **In-cluster** on railiance01: `forgejo-runner` Deployment + DinD (`railiance01-build-01`) | **done** (2026-07-03) | Runner 2/2 Ready; coulombcore host runner disabled; `image-build` probe pushed OCI image via org secrets |
| 7 | Backup target + retention | **Option A:** age-encrypted `forgejo dump` + `pg_dump` → **Nextcloud** WebDAV | **decided** (2026-07-09) | See § Backup; T04/T09 implementation |
| 8 | Cutover mode | **Option A:** staged per-repo promotion (no org-wide freeze) | **decided** (2026-07-09) | See § Cutover; tiers 0–2.5 + partial tier-3 proven |

## Hostname decision detail

**Chosen hostname:** `https://forgejo.coulomb.social`

| Field | Value |
| --- | --- |
| DNS | `forgejo.coulomb.social` → `92.205.62.239` (railiance01) |
| Edge | railiance01 k3s Traefik (`kube-system/traefik` LoadBalancer) |
| Target machine | railiance01 (production home per `CUST-WP-0054`) |
| Canonical git remote (post-cutover) | `https://forgejo.coulomb.social/coulomb/<repo>.git` |
| OCI registry (post-cutover) | `forgejo.coulomb.social/coulomb/<image>` |

### Live probe (2026-07-03, post-deploy)

```bash
getent hosts forgejo.coulomb.social   # 92.205.62.239
curl -fsS -o /dev/null -w '%{http_code}\n' https://forgejo.coulomb.social/   # 200
curl -sSI -X GET https://forgejo.coulomb.social/v2/ | grep -i docker-distribution  # registry/2.0
KUBECONFIG=~/.kube/config-hosteurope kubectl get pods,ingress,certificate -n forgejo
```

Forgejo is serving HTTPS with a valid Let's Encrypt cert. Gitea on coulombcore
remains canonical for git remotes until migration drills pass.

### Implications for CUST-WP-0054

- Wave 1 can proceed with a fixed hostname for overlays, ingress manifests, and
  CI `IMAGE_REPOSITORY` variables.
- State Hub / sweep checkouts on railiance01 (T05) should clone from
  `forgejo.coulomb.social` once cutover completes.
- Runners and image-build CI are proven; first repo migration pilot
  (`glas-harness`) documents git/SSH/CI routing in
  `docs/forgejo-repo-migration-pilot-glas-harness.md`.
- Remaining blockers for production cutover: **scheduled backup automation**
  (T04/T09 per Option A), npm/package hardening (T07), and staged per-repo
  migration execution (`RAIL-HO-WP-0005-T10` / `T11`).

## SMTP decision (IONOS) — 2026-07-07

**Provider:** IONOS mail for `coulomb.social` and subdomains (operator owns DNS zone
via IONOS; mailboxes provisioned in IONOS control panel).

**Recommended Forgejo mailer settings** (non-secret; password in
`railiance-apps/helm/forgejo-secrets.sops.yaml`):

| Field | Value |
| --- | --- |
| `MAILER_TYPE` | `smtp` |
| `HOST` | `smtp.ionos.com:587` (EU accounts may use `smtp.ionos.de:587`) |
| `IS_TLS_ENABLED` | `true` (STARTTLS on 587) |
| `FROM` | `forgejo@coulomb.social` (or dedicated `noreply@…` mailbox) |
| `USER` | full mailbox address (same as `FROM` unless using a shared relay mailbox) |
| `PASSWD` | IONOS mailbox password — SOPS only, never Git plaintext |

**Operator setup checklist:**

1. Create mailbox in IONOS (e.g. `forgejo@coulomb.social`).
2. Confirm SPF includes IONOS (`include:_spf-eu.ionos.com` or current IONOS
   guidance for the zone).
3. Enable DKIM in IONOS for `coulomb.social` if offered.
4. Store mailbox password in `helm/forgejo-secrets.sops.yaml` under
   `gitea.config.mailer.PASSWD`; redeploy Forgejo.
5. T06 gate: trigger password reset for a controlled non-admin test account;
   confirm delivery and link works.

**Implementation:** `railiance-apps/docs/forgejo-on-railiance01.md` (SMTP section).
Password in OpenBao `platform/workloads/forgejo/forgejo-mailer` (field `PASSWD`).

## Package scope decision (Option C) — 2026-07-08

**Operator choice:** Multi-ecosystem registry — not OCI-only at launch.

| Phase | Package types | Status |
| --- | --- | --- |
| **Now (proven)** | OCI container images | Live — tier 0–3 CI (`container-build-push` templates) |
| **Launch requirement** | **npm** | Decide → implement in T07 (auth, publish/pull docs, CI template) |
| **In scope** | Generic (release tarballs/binaries) | Enable with retention policy |
| **As needed** | PyPI, Go, Maven, Helm | Add per repo only after T01 inventory shows Gitea package data or an explicit consumer need |

**Rationale:** Railiance needs npm publishing; other ecosystems follow evidence, not
a big-bang enable-everything default.

**Implications:**

- T07 expands beyond OCI smoke: npm publish/pull runbook, org/repo token policy,
  retention/cleanup for non-OCI blobs.
- T09 backups must include **all enabled package types** under `/data/packages`.
- T01 authenticated Gitea inventory should classify existing npm (and other) package
  data before cutover waves.

**Forgejo endpoints (post-cutover):**

| Type | URL pattern |
| --- | --- |
| OCI | `forgejo.coulomb.social/coulomb/<image>` |
| npm | `https://forgejo.coulomb.social/api/packages/coulomb/npm/` (configure scope in T07) |
| Generic | per-repo generic package API |

## Backup decision (Option A) — 2026-07-09

**Operator choice:** Extend the existing **Railiance platform backup lane** —
age-encrypted artifacts uploaded to **Nextcloud WebDAV** (same pattern as
`railiance-platform` `make backup` / coulombcore `railiance-backup`).

| Component | Method | Schedule |
| --- | --- | --- |
| **Forgejo blob state** | `forgejo dump` zip (repos, packages incl. OCI/npm/generic, attachments, LFS, avatars) | Daily |
| **PostgreSQL** | `pg_dump` from CNPG `forgejo-db` (logical; no WAL/PITR in Phase 1) | Daily |
| **Encryption** | age (platform backup public key) before upload | per artifact |
| **Destination** | Nextcloud WebDAV file drop (off-node) | upload after each run |
| **Restore proof** | Re-run `tools/forgejo-restore-drill.sh` from automated backup quarterly | manual gate |

**Retention (balanced profile):**

| Artifact | Keep |
| --- | --- |
| Daily dumps | 14 rotations |
| Weekly dumps | 4 rotations (promote Sunday daily or explicit weekly job) |
| Local cache (if any) | 7 copies per type (match existing `railiance-backup` prune) |

**RPO / RTO targets:**

| Metric | Target |
| --- | --- |
| **RPO** | 24 hours (daily schedule) |
| **RTO** | 4 hours (operator response + restore drill path + validation) |

**Implementation owners:**

- `railiance-platform` — `make forgejo-backup` (or extend `make backup`): cron
  schedule, age encrypt, Nextcloud upload, retention prune.
- `railiance-infra` — restore runbook + quarterly drill evidence (`T09`).
- `railiance-apps` — no backup secrets in Git; dump runs against live pod.

**Explicitly not in scope (Phase 1):** CNPG WAL archiving to S3/MinIO; on-node-only
backups without Nextcloud upload. Revisit if dump size or RPO requirements outgrow
daily logical backup.

**Promotion gate:** No further tier-3 repo cutovers until automated daily backups
have succeeded **7 consecutive days** and one restore drill uses a Nextcloud
artifact (not workstation `/tmp`).

## Cutover decision (Option A) — 2026-07-09

**Operator choice:** **Staged per-repo** promotion — continue the proven migration
ladder; **no** org-wide freeze-all window.

**Per-repo sequence (T11):**

1. Gitea backup snapshot before promotion.
2. Mirror/sync git to Forgejo if not already present.
3. Verify Forgejo Actions and packages (OCI/npm per repo).
4. Switch workstation `origin` → `forgejo-remote`; retain `gitea` legacy remote.
5. Update State Hub `remote_url` and railiance01 sweep checkouts when promoted.
6. Mark **that** Gitea repo read-only; do not delete (safety contract).
7. Stabilization window before the next repo (operator judgment; `state-hub` used 7d).

**Explicitly rejected for primary plan:** org-wide freeze-all cutover (Option B).
Freeze-all remains documented only as an **emergency fallback** if staged drift
becomes unacceptable — not the default path.

**Rollback (per repo):** Re-point `origin` to `gitea-remote`; Forgejo copy becomes
stale mirror. No Gitea repo deletes for **30 days** after promotion (cutover
2026-07-08 → rollback window ends **2026-08-07**). Gitea Helm retained as
read-only fallback per T12 Option A (2026-07-09); revisit retirement after the
window closes.

**Gates before each tier-3 promotion:**

- T09 automated daily backups: 7 consecutive successes (Option A).
- T01 inventory classification for that repo (issues/wiki/LFS/packages).
- Operator explicit approval for Wave-1 production repos (`CUST-WP-0054`).

**Evidence:** tiers 0–2.5 complete; `glas-harness` pilot;
`state-hub` cutover under `CUST-WP-0054-T05`.

## T02 decision set — complete (2026-07-09)

All Wave-1 production design choices for Forgejo are decided:

| # | Topic | Option |
| --- | --- | --- |
| Hostname / exposure / deploy | decided 2026-07-03 |
| SMTP | IONOS + OpenBao/ESO (T06 done) |
| Package scope | Option C (OCI + npm + generic) |
| Actions runner | In-cluster ADR-004 |
| Backup | Option A (Nextcloud + age) |
| Cutover | **Option A (staged per-repo)** |

## T12 — Gitea retirement (decided 2026-07-09)

**Operator choice:** **Option A** — keep coulombcore Gitea Helm as read-only
fallback; do **not** retire the release now.

| Parameter | Value |
| --- | --- |
| Rollback window | **30 days** from cutover (2026-07-08 → **2026-08-07**) |
| Gitea state | 79/79 repos archived; package push disabled |
| Revisit | After **2026-08-07** — reassess Option B (Helm retirement) |
| Evidence | `railiance-infra/docs/forgejo-gitea-retirement-checklist.md` |

## Open decisions (need operator input)

_All T02 and T12 items decided as of 2026-07-09._ Implementation tracks: T04/T09
backup gate (7-day), stale OCI/npm doc references in layer repos.