WP-0005: T12 prep checklist, backup gate 4/7, doc refresh

Update workplan with Jul 9 cron success (4/7 gate, ETA Jul 12), add Gitea
retirement dependency scan and operator checklist, and align deploy-stack,
restore-drill evidence, and migration inventory with post-cutover state.

docs/deploy-stack.md

@@ -11,7 +11,8 @@ Before starting, verify you have:
 - [ ] SSH access to the target server (COULOMBCORE: 92.205.130.254, user: tegwick, key: `~/.ssh/id_ops`)
 - [ ] SOPS age private key available (`~/.config/sops/age/keys.txt` or `SOPS_AGE_KEY` env)
 - [ ] ops-bridge running on the workstation (needed for state hub MCP): `make mcp-http` in `~/the-custodian/state-hub/`
-- [ ] Gitea accessible (needed for git pull on remote): SSH via `gitea-remote:coulomb/<repo>.git`
+- [ ] Forgejo accessible (git pull on remote): SSH via `forgejo-remote:coulomb/<repo>.git`
+- [ ] Gitea on coulombcore remains read-only fallback only (`gitea-remote` for emergency rollback)
 - [ ] If re-provisioning from scratch: Hetzner/HostEurope API credentials decryptable via SOPS
 
 ---
@@ -64,26 +65,22 @@ cnpg-system namespace active.
 # On CoulombCore (kubectl available after S2)
 cd ~/railiance-platform && git pull
 
-# Create Gitea DB credentials secret (one-time; do NOT commit plaintext)
+# Deploy Forgejo database cluster (production on railiance01)
-kubectl create secret generic gitea-db-credentials \
+# Credentials via SOPS — see railiance-platform docs/forgejo-backup.md
-  --namespace databases \
+make forgejo-db-deploy
-  --from-literal=username=gitea \
+make forgejo-db-status
-  --from-literal=password=<GITEA_DB_PASSWORD>
 
-# Deploy cnpg Gitea database cluster
+# Legacy coulombcore Gitea DB (read-only fallback — retain until T12 retires Gitea)
-make db-deploy
+# make db-deploy && make db-status
 
-# Wait for cluster to be healthy (~60s)
+# Deploy Valkey cache if required by workloads
-make db-status
-
-# Deploy Valkey cache (standalone, not as Gitea subchart)
 # Requires: helm/valkey-values.sops.yaml with encrypted password
 make valkey-deploy
 make valkey-status
 ```
 
-**Checkpoint:** `kubectl get cluster -n databases` shows `gitea-db` healthy;
+**Checkpoint:** `kubectl get cluster -n databases` shows `forgejo-db` healthy on
-Valkey pod Running in platform namespace.
+railiance01; Valkey pod Running if deployed.
 
 ---
 
@@ -100,22 +97,19 @@ violation, tracked in RAIL-HO-WP-0004). No S4-specific steps required at this ti
 # On CoulombCore
 cd ~/railiance-apps && git pull
 
-# Deploy Gitea (git hosting)
+# Deploy Forgejo (production source forge on railiance01)
-# Requires: helm/gitea-values.sops.yaml with encrypted values
+# See railiance-apps/docs/forgejo-on-railiance01.md
-make gitea-deploy
+make forgejo-deploy
-make gitea-status
+make forgejo-verify
 
-# Deploy state-hub (Custodian cognitive infrastructure)
+# Legacy Gitea on coulombcore — read-only fallback; do not use for new repos
-# See RAIL-HO-WP-0004-T09 for full steps
+# make gitea-deploy && make gitea-status
-make state-hub-deploy   # (not yet implemented — pending T09)
 
-# Deploy activity-core
+# Deploy state-hub, activity-core, and other workloads per railiance-apps Makefile
-# See RAIL-HO-WP-0004-T10 for full steps
-make activity-core-deploy  # (not yet implemented — pending T10)
 ```
 
-**Checkpoint:** Gitea accessible and all repos cloneable via SSH; state-hub
+**Checkpoint:** `https://forgejo.coulomb.social` serves HTTPS; git clone/push via
-`/state/health` returns 200.
+`forgejo-remote`; state-hub `/state/health` returns 200.
 
 ---
 
@@ -135,10 +129,12 @@ Verify: `bridge status` shows all three connected.
 
 ## Recovery pointers
 
+- **Forgejo backup/restore:** `railiance-platform/docs/forgejo-backup.md`, `docs/forgejo-restore-drill-evidence.md`
+- **Cutover rollback:** `docs/forgejo-cutover-rollback.md`
 - **Node overload / SSH unresponsive:** See `the-custodian/ops/runbooks/gitea-coulombcore.md` Issue #3
 - **Incident report:** `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
 - **Cluster backup restore:** `railiance-cluster/tools/cmd/railiance-restore-s2`
-- **Gitea SSH not working:** Check `gitea-ssh-nodeport` service exists: `kubectl get svc -n default gitea-ssh-nodeport`
+- **Forgejo SSH:** NodePort `30022` on railiance01 (`forgejo-ssh-nodeport` in `forgejo` namespace)
 
 ---
 

docs/forgejo-gitea-retirement-checklist.md

@@ -0,0 +1,104 @@
+# Gitea Retirement Checklist (T12 prep)
+
+Date: 2026-07-09  
+Workplan: `RAIL-HO-WP-0005-T12`  
+Status: **prep only** — no Gitea retirement executed
+
+## Prerequisites (not yet met)
+
+| Gate | Status | ETA |
+| --- | --- | --- |
+| T04/T09 7-day backup gate | **4/7** (`make forgejo-backup-status`) | **2026-07-12** if daily 02:15 UTC cron keeps succeeding |
+| `state-hub` stabilization window | Ends **2026-07-09** | Met |
+| Operator approval (`needs_human`) | Pending | Required before any Helm teardown |
+
+Monitor:
+
+```bash
+cd ~/railiance-platform && make forgejo-backup-status
+tail -5 ~/.cache/railiance/backups/forgejo/cron.log
+```
+
+## Dependency scan (2026-07-09)
+
+### Clear — no active git remotes on Gitea
+
+| Check | Result |
+| --- | --- |
+| State Hub `remote_url` | **82 Forgejo**, **0 Gitea**, **0** without remote |
+| Workstation `origin` using `gitea-remote` | **0** (legacy `gitea` remotes retained for rollback only) |
+| Forgejo Actions workflows referencing `gitea-remote` | **0** |
+
+### Gitea still running (intentional fallback)
+
+| Host | Release | Namespace | Notes |
+| --- | --- | --- | --- |
+| coulombcore (`~/.kube/config`) | `gitea` Helm 12.5.0 | `default` | Archived read-only repos; package push disabled |
+| railiance01 (`config-hosteurope`) | `forgejo` Helm 12.5.0 | `forgejo` | Production canonical |
+
+Rollback runbook: `docs/forgejo-cutover-rollback.md` (90-day no-delete contract).
+
+### Stale references — docs/config, not live git remotes
+
+These do **not** block Forgejo operation but should be cleaned before or during
+Gitea retirement:
+
+| Area | Examples | Owner repo |
+| --- | --- | --- |
+| OCI image repos in Helm values | `gitea.coulomb.social/coulomb/inter-hub`, `reuse-surface`, `vergabe-teilnahme` | `railiance-apps` |
+| npm registry URLs | `gitea.coulomb.social/api/packages/coulomb/npm/` | `secrets-engine`, `activity-core` |
+| Ops inventory endpoint | `activity-core/k8s/railiance/20-runtime.yaml` | `activity-core` |
+| Deploy runbook | `docs/deploy-stack.md` (this repo) | `railiance-infra` — updated 2026-07-09 |
+
+### State Hub alias: `railiance-bootstrap`
+
+| Field | `railiance-bootstrap` | `railiance-cluster` |
+| --- | --- | --- |
+| `local_path` | `/home/worsch/railiance-bootstrap` | `/home/worsch/railiance-cluster` |
+| `remote_url` | `forgejo-remote:coulomb/railiance-bootstrap.git` | `forgejo-remote:coulomb/railiance-cluster.git` |
+| `git_fingerprint` | `6b930728…` (same) | `6b930728…` (same) |
+
+Working assumption confirmed: stale logical alias. T12 action: operator decides
+whether to archive the hub entry or repoint `local_path` to `railiance-cluster`
+after confirming no checkout depends on `/home/worsch/railiance-bootstrap`.
+
+## Operator decision
+
+Choose one before executing T12:
+
+### Option A — Keep Gitea as documented fallback (recommended until 2026-10-08)
+
+- Leave coulombcore `gitea` Helm release running.
+- Document in runbooks: read-only archive, no new pushes.
+- Revisit retirement after 90-day rollback window.
+- **T12 done when:** Forgejo is sole *active* forge; Gitea retained intentionally.
+
+### Option B — Retire Gitea Helm release
+
+Only after prerequisites + explicit approval:
+
+1. Re-run dependency scan (State Hub remotes, ArgoCD `repoURL`, CI workflows).
+2. Take final Gitea backup (`forgejo dump` equivalent via Gitea admin or volume snapshot).
+3. Export archived repo list for audit.
+4. Scale down or uninstall `gitea` Helm on coulombcore.
+5. Update `docs/forgejo-cutover-rollback.md` — rollback becomes volume-restore only.
+6. Prune DNS/ingress if no longer needed.
+
+## T12 task checklist
+
+- [ ] T04/T09 7-day gate **7/7**
+- [ ] Operator chooses Option A or B
+- [ ] Final Gitea backup captured and catalogued
+- [ ] No active remotes/webhooks/dashboards on Gitea (re-verify)
+- [ ] Runbooks point to Forgejo (`deploy-stack.md`, layer repo docs)
+- [ ] `railiance-bootstrap` hub alias resolved
+- [ ] Stale OCI/npm references migrated to `forgejo.coulomb.social` (follow-up workplans in `railiance-apps`, `activity-core`)
+- [ ] If Option B: Gitea Helm retired; evidence logged to State Hub
+
+## References
+
+- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md`
+- `docs/forgejo-cutover-rollback.md`
+- `docs/forgejo-migration-inventory.md`
+- `railiance-platform/docs/forgejo-backup.md`
+- `the-custodian/tools/gitea-set-readonly.sh`

docs/forgejo-migration-inventory.md

@@ -366,8 +366,9 @@ classification, and admin snapshots (Forgejo full; Gitea partial without
 `read:admin` PAT). Residual `read:admin` sweep is **out of scope** for cutover
 — sample repos showed 0 deploy keys and 0 webhooks; org webhooks 0 on both forges.
 
-**Remaining before T11 signoff:** Gitea repos marked read-only (not deleted);
+**Remaining before workplan close (2026-07-09):** T04/T09 7-day backup gate
-7-day scheduled backup gate (T04/T09); operator stabilization signoff.
+**4/7** (ETA 2026-07-12); T12 operator decision (retire vs keep Gitea fallback).
+T11 cutover complete — Gitea repos archived read-only, not deleted.
 
 ## Addendum (2026-07-04) — migration ladder and new repos
 

docs/forgejo-restore-drill-evidence.md

@@ -93,20 +93,25 @@ secret, `mailer.ENABLED=false`, Helm `--timeout=20m`.
 
 Exit marker: `restore-drill-complete`
 
-## Gaps (not closed by this drill)
+## Gaps (remaining)
 
-- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured
+- **Scheduled backups:** `railiance-platform` `make forgejo-backup` — daily cron
-  (`kubectl cnpg` plugin absent on workstation).
+  02:15 UTC; 7-day gate **4/7** as of 2026-07-09 (ETA 2026-07-12). CNPG `Backup`
-- **Encryption at rest:** dump stored locally on workstation for drill only; no
+  CRs deferred (logical `pg_dump` via `forgejo-backup` tool).
-  approved backup target wired.
-- **Automation:** `railiance-platform` `make forgejo-backup` — daily cron 02:15 UTC;
-  7-day gate still counting.
 - **Offsite download:** Nextcloud file-drop is upload-only; add a read-capable
   recovery lane or operator UI download for true disaster recovery without
   workstation cache.
 - **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe
   `forgejo-restore-drill` before import (SQL import is not idempotent).
 
+## Closed since initial drill (2026-07-04)
+
+- **Encryption + offsite target:** age-encrypted artifacts uploaded to Nextcloud
+  `forgejo/` (Option A, T02).
+- **Automation tooling:** `railiance-platform/tools/cmd/forgejo-backup` +
+  `docs/forgejo-backup.md`.
+- **Nextcloud restore drill:** passed 2026-07-08 using artifact `20260707T180844Z`.
+
 ## Cleanup
 
 After evidence capture, delete the drill namespace:

workplans/RAIL-HO-WP-0005-forgejo-production-migration.md

@@ -8,7 +8,7 @@ status: active
 owner: railiance
 topic_slug: railiance
 created: "2026-05-03"
-updated: "2026-07-08"
+updated: "2026-07-09"
 state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018"
 ---
 
@@ -118,7 +118,7 @@ Still to prove before T11 closeout:
 
 - ~~SMTP/password reset end-to-end (T06).~~ **done** (2026-07-08).
 - ~~Backup restore from offsite artifact (T09).~~ **done** (2026-07-08); 7-day
-  scheduled backup gate **3/7** (cron PATH fixed 2026-07-08).
+  scheduled backup gate **4/7** (Jul 9 02:15 UTC cron succeeded; ETA **2026-07-12**).
 - ~~Repo migration ladder (T10).~~ **done** (2026-07-08); 0 Gitea-only repos.
 - ~~Operator SSH on Forgejo as `tegwick` (T05/T10).~~ **done**.
 - ~~Gitea read-only marking + operator signoff (T11).~~ **done** (2026-07-08).
@@ -257,8 +257,10 @@ decided (Option A, Nextcloud).** **Automation:** `railiance-platform`
 `pg_dump` → age → Nextcloud `forgejo/`. Daily cron 02:15 UTC; **cron PATH fixed
 2026-07-08** (`kubectl` was missing from cron env — Jul 8 02:15 run failed).
 **Nextcloud restore drill passed 2026-07-08.** Manual backup
-`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Remaining: 7-day
+`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Jul 8 02:15 cron
-success gate (**3/7** as of 2026-07-08).
+failed (`kubectl` missing from cron PATH); fixed same day. Jul 9 02:15 cron
+succeeded (`20260709T001505Z`, ~967M dump + 20M pg_dump). Remaining: 7-day
+success gate (**4/7** as of 2026-07-09; ETA **2026-07-12**).
 
 **Done when:** platform dependencies can be deployed and restored without the
 Forgejo app running.
@@ -457,9 +459,9 @@ Makefile targets. **First scheduled backup:** 20260707T180844Z (~686M, 14 repos)
 Daily cron 02:15 UTC. **Nextcloud restore drill passed 2026-07-08** — decrypt
 uploaded `.age` artifact, chunked copy, 14 org repos restored in
 `forgejo-restore-drill` (evidence: `docs/forgejo-restore-drill-evidence.md`).
-Remaining: 7 consecutive backup successes (**3/7** as of 2026-07-08); cron PATH
+Remaining: 7 consecutive backup successes (**4/7** as of 2026-07-09; ETA
-fixed so nightly runs resume. Nextcloud file-drop is upload-only — read-lane TBD
+**2026-07-12**). Jul 9 02:15 UTC cron run succeeded after PATH fix. Nextcloud
-for DR without workstation cache.
+file-drop is upload-only — read-lane TBD for DR without workstation cache.
 
 **Done when:** a fresh backup restores to a working isolated Forgejo instance
 with repository, package, and user recovery checks passing **and** scheduled
@@ -583,6 +585,12 @@ state_hub_task_id: "a63147b0-31d5-4705-89ea-40c10faf779f"
 
 Retire legacy Gitea only after a stabilization period and explicit approval.
 
+**Prep (2026-07-09):** dependency scan complete — State Hub **0 Gitea remotes**,
+workstation **0 `gitea-remote` origins**; coulombcore `gitea` Helm still running
+as read-only fallback. Checklist: `docs/forgejo-gitea-retirement-checklist.md`.
+Operator must choose: **Option A** keep fallback (recommended until 90-day
+rollback window) or **Option B** retire Helm release.
+
 Minimum scope:
 
 - Confirm no active remotes, webhooks, packages, or dashboards depend on Gitea.
@@ -615,9 +623,10 @@ T05+T08 ──► T10 migration ladder ──► T11 production cutover ──
 T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots)
 ```
 
-**Current focus (2026-07-08):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09
+**Current focus (2026-07-09):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09
-await 7-day backup gate (**3/7**; cron PATH fixed). T12 next after
+await 7-day backup gate (**4/7**, ETA **2026-07-12**). T12 prep complete —
-stabilization window — Gitea archived + package push disabled; ArgoCD on Forgejo.
+operator decision pending (retire vs keep fallback). `state-hub` stabilization
+window closed 2026-07-09.
 
 **Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation
 build retirement; staged repo promotion before State Hub primary move (T05).
@@ -646,6 +655,7 @@ purpose is identified.
 - `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md`
 - `docs/forgejo-migration-inventory.md`
 - `docs/forgejo-cutover-rollback.md`
+- `docs/forgejo-gitea-retirement-checklist.md`
 - `the-custodian/docs/forgejo-production-decisions.md`
 - `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`
 - `railiance-apps/docs/forgejo-on-railiance01.md`