diff --git a/docs/deploy-stack.md b/docs/deploy-stack.md index 23350e5..2d808ae 100644 --- a/docs/deploy-stack.md +++ b/docs/deploy-stack.md @@ -11,7 +11,8 @@ Before starting, verify you have: - [ ] SSH access to the target server (COULOMBCORE: 92.205.130.254, user: tegwick, key: `~/.ssh/id_ops`) - [ ] SOPS age private key available (`~/.config/sops/age/keys.txt` or `SOPS_AGE_KEY` env) - [ ] ops-bridge running on the workstation (needed for state hub MCP): `make mcp-http` in `~/the-custodian/state-hub/` -- [ ] Gitea accessible (needed for git pull on remote): SSH via `gitea-remote:coulomb/.git` +- [ ] Forgejo accessible (git pull on remote): SSH via `forgejo-remote:coulomb/.git` +- [ ] Gitea on coulombcore remains read-only fallback only (`gitea-remote` for emergency rollback) - [ ] If re-provisioning from scratch: Hetzner/HostEurope API credentials decryptable via SOPS --- @@ -64,26 +65,22 @@ cnpg-system namespace active. # On CoulombCore (kubectl available after S2) cd ~/railiance-platform && git pull -# Create Gitea DB credentials secret (one-time; do NOT commit plaintext) -kubectl create secret generic gitea-db-credentials \ - --namespace databases \ - --from-literal=username=gitea \ - --from-literal=password= +# Deploy Forgejo database cluster (production on railiance01) +# Credentials via SOPS — see railiance-platform docs/forgejo-backup.md +make forgejo-db-deploy +make forgejo-db-status -# Deploy cnpg Gitea database cluster -make db-deploy +# Legacy coulombcore Gitea DB (read-only fallback — retain until T12 retires Gitea) +# make db-deploy && make db-status -# Wait for cluster to be healthy (~60s) -make db-status - -# Deploy Valkey cache (standalone, not as Gitea subchart) +# Deploy Valkey cache if required by workloads # Requires: helm/valkey-values.sops.yaml with encrypted password make valkey-deploy make valkey-status ``` -**Checkpoint:** `kubectl get cluster -n databases` shows `gitea-db` healthy; -Valkey pod Running in platform namespace. +**Checkpoint:** `kubectl get cluster -n databases` shows `forgejo-db` healthy on +railiance01; Valkey pod Running if deployed. --- @@ -100,22 +97,19 @@ violation, tracked in RAIL-HO-WP-0004). No S4-specific steps required at this ti # On CoulombCore cd ~/railiance-apps && git pull -# Deploy Gitea (git hosting) -# Requires: helm/gitea-values.sops.yaml with encrypted values -make gitea-deploy -make gitea-status +# Deploy Forgejo (production source forge on railiance01) +# See railiance-apps/docs/forgejo-on-railiance01.md +make forgejo-deploy +make forgejo-verify -# Deploy state-hub (Custodian cognitive infrastructure) -# See RAIL-HO-WP-0004-T09 for full steps -make state-hub-deploy # (not yet implemented — pending T09) +# Legacy Gitea on coulombcore — read-only fallback; do not use for new repos +# make gitea-deploy && make gitea-status -# Deploy activity-core -# See RAIL-HO-WP-0004-T10 for full steps -make activity-core-deploy # (not yet implemented — pending T10) +# Deploy state-hub, activity-core, and other workloads per railiance-apps Makefile ``` -**Checkpoint:** Gitea accessible and all repos cloneable via SSH; state-hub -`/state/health` returns 200. +**Checkpoint:** `https://forgejo.coulomb.social` serves HTTPS; git clone/push via +`forgejo-remote`; state-hub `/state/health` returns 200. --- @@ -135,10 +129,12 @@ Verify: `bridge status` shows all three connected. ## Recovery pointers +- **Forgejo backup/restore:** `railiance-platform/docs/forgejo-backup.md`, `docs/forgejo-restore-drill-evidence.md` +- **Cutover rollback:** `docs/forgejo-cutover-rollback.md` - **Node overload / SSH unresponsive:** See `the-custodian/ops/runbooks/gitea-coulombcore.md` Issue #3 - **Incident report:** `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md` - **Cluster backup restore:** `railiance-cluster/tools/cmd/railiance-restore-s2` -- **Gitea SSH not working:** Check `gitea-ssh-nodeport` service exists: `kubectl get svc -n default gitea-ssh-nodeport` +- **Forgejo SSH:** NodePort `30022` on railiance01 (`forgejo-ssh-nodeport` in `forgejo` namespace) --- diff --git a/docs/forgejo-gitea-retirement-checklist.md b/docs/forgejo-gitea-retirement-checklist.md new file mode 100644 index 0000000..bbbb7e1 --- /dev/null +++ b/docs/forgejo-gitea-retirement-checklist.md @@ -0,0 +1,104 @@ +# Gitea Retirement Checklist (T12 prep) + +Date: 2026-07-09 +Workplan: `RAIL-HO-WP-0005-T12` +Status: **prep only** — no Gitea retirement executed + +## Prerequisites (not yet met) + +| Gate | Status | ETA | +| --- | --- | --- | +| T04/T09 7-day backup gate | **4/7** (`make forgejo-backup-status`) | **2026-07-12** if daily 02:15 UTC cron keeps succeeding | +| `state-hub` stabilization window | Ends **2026-07-09** | Met | +| Operator approval (`needs_human`) | Pending | Required before any Helm teardown | + +Monitor: + +```bash +cd ~/railiance-platform && make forgejo-backup-status +tail -5 ~/.cache/railiance/backups/forgejo/cron.log +``` + +## Dependency scan (2026-07-09) + +### Clear — no active git remotes on Gitea + +| Check | Result | +| --- | --- | +| State Hub `remote_url` | **82 Forgejo**, **0 Gitea**, **0** without remote | +| Workstation `origin` using `gitea-remote` | **0** (legacy `gitea` remotes retained for rollback only) | +| Forgejo Actions workflows referencing `gitea-remote` | **0** | + +### Gitea still running (intentional fallback) + +| Host | Release | Namespace | Notes | +| --- | --- | --- | --- | +| coulombcore (`~/.kube/config`) | `gitea` Helm 12.5.0 | `default` | Archived read-only repos; package push disabled | +| railiance01 (`config-hosteurope`) | `forgejo` Helm 12.5.0 | `forgejo` | Production canonical | + +Rollback runbook: `docs/forgejo-cutover-rollback.md` (90-day no-delete contract). + +### Stale references — docs/config, not live git remotes + +These do **not** block Forgejo operation but should be cleaned before or during +Gitea retirement: + +| Area | Examples | Owner repo | +| --- | --- | --- | +| OCI image repos in Helm values | `gitea.coulomb.social/coulomb/inter-hub`, `reuse-surface`, `vergabe-teilnahme` | `railiance-apps` | +| npm registry URLs | `gitea.coulomb.social/api/packages/coulomb/npm/` | `secrets-engine`, `activity-core` | +| Ops inventory endpoint | `activity-core/k8s/railiance/20-runtime.yaml` | `activity-core` | +| Deploy runbook | `docs/deploy-stack.md` (this repo) | `railiance-infra` — updated 2026-07-09 | + +### State Hub alias: `railiance-bootstrap` + +| Field | `railiance-bootstrap` | `railiance-cluster` | +| --- | --- | --- | +| `local_path` | `/home/worsch/railiance-bootstrap` | `/home/worsch/railiance-cluster` | +| `remote_url` | `forgejo-remote:coulomb/railiance-bootstrap.git` | `forgejo-remote:coulomb/railiance-cluster.git` | +| `git_fingerprint` | `6b930728…` (same) | `6b930728…` (same) | + +Working assumption confirmed: stale logical alias. T12 action: operator decides +whether to archive the hub entry or repoint `local_path` to `railiance-cluster` +after confirming no checkout depends on `/home/worsch/railiance-bootstrap`. + +## Operator decision + +Choose one before executing T12: + +### Option A — Keep Gitea as documented fallback (recommended until 2026-10-08) + +- Leave coulombcore `gitea` Helm release running. +- Document in runbooks: read-only archive, no new pushes. +- Revisit retirement after 90-day rollback window. +- **T12 done when:** Forgejo is sole *active* forge; Gitea retained intentionally. + +### Option B — Retire Gitea Helm release + +Only after prerequisites + explicit approval: + +1. Re-run dependency scan (State Hub remotes, ArgoCD `repoURL`, CI workflows). +2. Take final Gitea backup (`forgejo dump` equivalent via Gitea admin or volume snapshot). +3. Export archived repo list for audit. +4. Scale down or uninstall `gitea` Helm on coulombcore. +5. Update `docs/forgejo-cutover-rollback.md` — rollback becomes volume-restore only. +6. Prune DNS/ingress if no longer needed. + +## T12 task checklist + +- [ ] T04/T09 7-day gate **7/7** +- [ ] Operator chooses Option A or B +- [ ] Final Gitea backup captured and catalogued +- [ ] No active remotes/webhooks/dashboards on Gitea (re-verify) +- [ ] Runbooks point to Forgejo (`deploy-stack.md`, layer repo docs) +- [ ] `railiance-bootstrap` hub alias resolved +- [ ] Stale OCI/npm references migrated to `forgejo.coulomb.social` (follow-up workplans in `railiance-apps`, `activity-core`) +- [ ] If Option B: Gitea Helm retired; evidence logged to State Hub + +## References + +- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` +- `docs/forgejo-cutover-rollback.md` +- `docs/forgejo-migration-inventory.md` +- `railiance-platform/docs/forgejo-backup.md` +- `the-custodian/tools/gitea-set-readonly.sh` \ No newline at end of file diff --git a/docs/forgejo-migration-inventory.md b/docs/forgejo-migration-inventory.md index 0160e23..3ce905c 100644 --- a/docs/forgejo-migration-inventory.md +++ b/docs/forgejo-migration-inventory.md @@ -366,8 +366,9 @@ classification, and admin snapshots (Forgejo full; Gitea partial without `read:admin` PAT). Residual `read:admin` sweep is **out of scope** for cutover — sample repos showed 0 deploy keys and 0 webhooks; org webhooks 0 on both forges. -**Remaining before T11 signoff:** Gitea repos marked read-only (not deleted); -7-day scheduled backup gate (T04/T09); operator stabilization signoff. +**Remaining before workplan close (2026-07-09):** T04/T09 7-day backup gate +**4/7** (ETA 2026-07-12); T12 operator decision (retire vs keep Gitea fallback). +T11 cutover complete — Gitea repos archived read-only, not deleted. ## Addendum (2026-07-04) — migration ladder and new repos diff --git a/docs/forgejo-restore-drill-evidence.md b/docs/forgejo-restore-drill-evidence.md index 4284d4d..502c1a8 100644 --- a/docs/forgejo-restore-drill-evidence.md +++ b/docs/forgejo-restore-drill-evidence.md @@ -93,20 +93,25 @@ secret, `mailer.ENABLED=false`, Helm `--timeout=20m`. Exit marker: `restore-drill-complete` -## Gaps (not closed by this drill) +## Gaps (remaining) -- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured - (`kubectl cnpg` plugin absent on workstation). -- **Encryption at rest:** dump stored locally on workstation for drill only; no - approved backup target wired. -- **Automation:** `railiance-platform` `make forgejo-backup` — daily cron 02:15 UTC; - 7-day gate still counting. +- **Scheduled backups:** `railiance-platform` `make forgejo-backup` — daily cron + 02:15 UTC; 7-day gate **4/7** as of 2026-07-09 (ETA 2026-07-12). CNPG `Backup` + CRs deferred (logical `pg_dump` via `forgejo-backup` tool). - **Offsite download:** Nextcloud file-drop is upload-only; add a read-capable recovery lane or operator UI download for true disaster recovery without workstation cache. - **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe `forgejo-restore-drill` before import (SQL import is not idempotent). +## Closed since initial drill (2026-07-04) + +- **Encryption + offsite target:** age-encrypted artifacts uploaded to Nextcloud + `forgejo/` (Option A, T02). +- **Automation tooling:** `railiance-platform/tools/cmd/forgejo-backup` + + `docs/forgejo-backup.md`. +- **Nextcloud restore drill:** passed 2026-07-08 using artifact `20260707T180844Z`. + ## Cleanup After evidence capture, delete the drill namespace: diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index ee42696..4c32cd3 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -8,7 +8,7 @@ status: active owner: railiance topic_slug: railiance created: "2026-05-03" -updated: "2026-07-08" +updated: "2026-07-09" state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018" --- @@ -118,7 +118,7 @@ Still to prove before T11 closeout: - ~~SMTP/password reset end-to-end (T06).~~ **done** (2026-07-08). - ~~Backup restore from offsite artifact (T09).~~ **done** (2026-07-08); 7-day - scheduled backup gate **3/7** (cron PATH fixed 2026-07-08). + scheduled backup gate **4/7** (Jul 9 02:15 UTC cron succeeded; ETA **2026-07-12**). - ~~Repo migration ladder (T10).~~ **done** (2026-07-08); 0 Gitea-only repos. - ~~Operator SSH on Forgejo as `tegwick` (T05/T10).~~ **done**. - ~~Gitea read-only marking + operator signoff (T11).~~ **done** (2026-07-08). @@ -257,8 +257,10 @@ decided (Option A, Nextcloud).** **Automation:** `railiance-platform` `pg_dump` → age → Nextcloud `forgejo/`. Daily cron 02:15 UTC; **cron PATH fixed 2026-07-08** (`kubectl` was missing from cron env — Jul 8 02:15 run failed). **Nextcloud restore drill passed 2026-07-08.** Manual backup -`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Remaining: 7-day -success gate (**3/7** as of 2026-07-08). +`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Jul 8 02:15 cron +failed (`kubectl` missing from cron PATH); fixed same day. Jul 9 02:15 cron +succeeded (`20260709T001505Z`, ~967M dump + 20M pg_dump). Remaining: 7-day +success gate (**4/7** as of 2026-07-09; ETA **2026-07-12**). **Done when:** platform dependencies can be deployed and restored without the Forgejo app running. @@ -457,9 +459,9 @@ Makefile targets. **First scheduled backup:** 20260707T180844Z (~686M, 14 repos) Daily cron 02:15 UTC. **Nextcloud restore drill passed 2026-07-08** — decrypt uploaded `.age` artifact, chunked copy, 14 org repos restored in `forgejo-restore-drill` (evidence: `docs/forgejo-restore-drill-evidence.md`). -Remaining: 7 consecutive backup successes (**3/7** as of 2026-07-08); cron PATH -fixed so nightly runs resume. Nextcloud file-drop is upload-only — read-lane TBD -for DR without workstation cache. +Remaining: 7 consecutive backup successes (**4/7** as of 2026-07-09; ETA +**2026-07-12**). Jul 9 02:15 UTC cron run succeeded after PATH fix. Nextcloud +file-drop is upload-only — read-lane TBD for DR without workstation cache. **Done when:** a fresh backup restores to a working isolated Forgejo instance with repository, package, and user recovery checks passing **and** scheduled @@ -583,6 +585,12 @@ state_hub_task_id: "a63147b0-31d5-4705-89ea-40c10faf779f" Retire legacy Gitea only after a stabilization period and explicit approval. +**Prep (2026-07-09):** dependency scan complete — State Hub **0 Gitea remotes**, +workstation **0 `gitea-remote` origins**; coulombcore `gitea` Helm still running +as read-only fallback. Checklist: `docs/forgejo-gitea-retirement-checklist.md`. +Operator must choose: **Option A** keep fallback (recommended until 90-day +rollback window) or **Option B** retire Helm release. + Minimum scope: - Confirm no active remotes, webhooks, packages, or dashboards depend on Gitea. @@ -615,9 +623,10 @@ T05+T08 ──► T10 migration ladder ──► T11 production cutover ── T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots) ``` -**Current focus (2026-07-08):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09 -await 7-day backup gate (**3/7**; cron PATH fixed). T12 next after -stabilization window — Gitea archived + package push disabled; ArgoCD on Forgejo. +**Current focus (2026-07-09):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09 +await 7-day backup gate (**4/7**, ETA **2026-07-12**). T12 prep complete — +operator decision pending (retire vs keep fallback). `state-hub` stabilization +window closed 2026-07-09. **Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation build retirement; staged repo promotion before State Hub primary move (T05). @@ -646,6 +655,7 @@ purpose is identified. - `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md` - `docs/forgejo-migration-inventory.md` - `docs/forgejo-cutover-rollback.md` +- `docs/forgejo-gitea-retirement-checklist.md` - `the-custodian/docs/forgejo-production-decisions.md` - `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` - `railiance-apps/docs/forgejo-on-railiance01.md`