WP-0005: T12 prep checklist, backup gate 4/7, doc refresh

Update workplan with Jul 9 cron success (4/7 gate, ETA Jul 12), add Gitea
retirement dependency scan and operator checklist, and align deploy-stack,
restore-drill evidence, and migration inventory with post-cutover state.
This commit is contained in:
tegwick 2026-07-09 10:34:44 +02:00
parent 0c3968a108
commit 040ccccc8e
5 changed files with 162 additions and 46 deletions

View file

@ -11,7 +11,8 @@ Before starting, verify you have:
- [ ] SSH access to the target server (COULOMBCORE: 92.205.130.254, user: tegwick, key: `~/.ssh/id_ops`)
- [ ] SOPS age private key available (`~/.config/sops/age/keys.txt` or `SOPS_AGE_KEY` env)
- [ ] ops-bridge running on the workstation (needed for state hub MCP): `make mcp-http` in `~/the-custodian/state-hub/`
- [ ] Gitea accessible (needed for git pull on remote): SSH via `gitea-remote:coulomb/<repo>.git`
- [ ] Forgejo accessible (git pull on remote): SSH via `forgejo-remote:coulomb/<repo>.git`
- [ ] Gitea on coulombcore remains read-only fallback only (`gitea-remote` for emergency rollback)
- [ ] If re-provisioning from scratch: Hetzner/HostEurope API credentials decryptable via SOPS
---
@ -64,26 +65,22 @@ cnpg-system namespace active.
# On CoulombCore (kubectl available after S2)
cd ~/railiance-platform && git pull
# Create Gitea DB credentials secret (one-time; do NOT commit plaintext)
kubectl create secret generic gitea-db-credentials \
--namespace databases \
--from-literal=username=gitea \
--from-literal=password=<GITEA_DB_PASSWORD>
# Deploy Forgejo database cluster (production on railiance01)
# Credentials via SOPS — see railiance-platform docs/forgejo-backup.md
make forgejo-db-deploy
make forgejo-db-status
# Deploy cnpg Gitea database cluster
make db-deploy
# Legacy coulombcore Gitea DB (read-only fallback — retain until T12 retires Gitea)
# make db-deploy && make db-status
# Wait for cluster to be healthy (~60s)
make db-status
# Deploy Valkey cache (standalone, not as Gitea subchart)
# Deploy Valkey cache if required by workloads
# Requires: helm/valkey-values.sops.yaml with encrypted password
make valkey-deploy
make valkey-status
```
**Checkpoint:** `kubectl get cluster -n databases` shows `gitea-db` healthy;
Valkey pod Running in platform namespace.
**Checkpoint:** `kubectl get cluster -n databases` shows `forgejo-db` healthy on
railiance01; Valkey pod Running if deployed.
---
@ -100,22 +97,19 @@ violation, tracked in RAIL-HO-WP-0004). No S4-specific steps required at this ti
# On CoulombCore
cd ~/railiance-apps && git pull
# Deploy Gitea (git hosting)
# Requires: helm/gitea-values.sops.yaml with encrypted values
make gitea-deploy
make gitea-status
# Deploy Forgejo (production source forge on railiance01)
# See railiance-apps/docs/forgejo-on-railiance01.md
make forgejo-deploy
make forgejo-verify
# Deploy state-hub (Custodian cognitive infrastructure)
# See RAIL-HO-WP-0004-T09 for full steps
make state-hub-deploy # (not yet implemented — pending T09)
# Legacy Gitea on coulombcore — read-only fallback; do not use for new repos
# make gitea-deploy && make gitea-status
# Deploy activity-core
# See RAIL-HO-WP-0004-T10 for full steps
make activity-core-deploy # (not yet implemented — pending T10)
# Deploy state-hub, activity-core, and other workloads per railiance-apps Makefile
```
**Checkpoint:** Gitea accessible and all repos cloneable via SSH; state-hub
`/state/health` returns 200.
**Checkpoint:** `https://forgejo.coulomb.social` serves HTTPS; git clone/push via
`forgejo-remote`; state-hub `/state/health` returns 200.
---
@ -135,10 +129,12 @@ Verify: `bridge status` shows all three connected.
## Recovery pointers
- **Forgejo backup/restore:** `railiance-platform/docs/forgejo-backup.md`, `docs/forgejo-restore-drill-evidence.md`
- **Cutover rollback:** `docs/forgejo-cutover-rollback.md`
- **Node overload / SSH unresponsive:** See `the-custodian/ops/runbooks/gitea-coulombcore.md` Issue #3
- **Incident report:** `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
- **Cluster backup restore:** `railiance-cluster/tools/cmd/railiance-restore-s2`
- **Gitea SSH not working:** Check `gitea-ssh-nodeport` service exists: `kubectl get svc -n default gitea-ssh-nodeport`
- **Forgejo SSH:** NodePort `30022` on railiance01 (`forgejo-ssh-nodeport` in `forgejo` namespace)
---

View file

@ -0,0 +1,104 @@
# Gitea Retirement Checklist (T12 prep)
Date: 2026-07-09
Workplan: `RAIL-HO-WP-0005-T12`
Status: **prep only** — no Gitea retirement executed
## Prerequisites (not yet met)
| Gate | Status | ETA |
| --- | --- | --- |
| T04/T09 7-day backup gate | **4/7** (`make forgejo-backup-status`) | **2026-07-12** if daily 02:15 UTC cron keeps succeeding |
| `state-hub` stabilization window | Ends **2026-07-09** | Met |
| Operator approval (`needs_human`) | Pending | Required before any Helm teardown |
Monitor:
```bash
cd ~/railiance-platform && make forgejo-backup-status
tail -5 ~/.cache/railiance/backups/forgejo/cron.log
```
## Dependency scan (2026-07-09)
### Clear — no active git remotes on Gitea
| Check | Result |
| --- | --- |
| State Hub `remote_url` | **82 Forgejo**, **0 Gitea**, **0** without remote |
| Workstation `origin` using `gitea-remote` | **0** (legacy `gitea` remotes retained for rollback only) |
| Forgejo Actions workflows referencing `gitea-remote` | **0** |
### Gitea still running (intentional fallback)
| Host | Release | Namespace | Notes |
| --- | --- | --- | --- |
| coulombcore (`~/.kube/config`) | `gitea` Helm 12.5.0 | `default` | Archived read-only repos; package push disabled |
| railiance01 (`config-hosteurope`) | `forgejo` Helm 12.5.0 | `forgejo` | Production canonical |
Rollback runbook: `docs/forgejo-cutover-rollback.md` (90-day no-delete contract).
### Stale references — docs/config, not live git remotes
These do **not** block Forgejo operation but should be cleaned before or during
Gitea retirement:
| Area | Examples | Owner repo |
| --- | --- | --- |
| OCI image repos in Helm values | `gitea.coulomb.social/coulomb/inter-hub`, `reuse-surface`, `vergabe-teilnahme` | `railiance-apps` |
| npm registry URLs | `gitea.coulomb.social/api/packages/coulomb/npm/` | `secrets-engine`, `activity-core` |
| Ops inventory endpoint | `activity-core/k8s/railiance/20-runtime.yaml` | `activity-core` |
| Deploy runbook | `docs/deploy-stack.md` (this repo) | `railiance-infra` — updated 2026-07-09 |
### State Hub alias: `railiance-bootstrap`
| Field | `railiance-bootstrap` | `railiance-cluster` |
| --- | --- | --- |
| `local_path` | `/home/worsch/railiance-bootstrap` | `/home/worsch/railiance-cluster` |
| `remote_url` | `forgejo-remote:coulomb/railiance-bootstrap.git` | `forgejo-remote:coulomb/railiance-cluster.git` |
| `git_fingerprint` | `6b930728…` (same) | `6b930728…` (same) |
Working assumption confirmed: stale logical alias. T12 action: operator decides
whether to archive the hub entry or repoint `local_path` to `railiance-cluster`
after confirming no checkout depends on `/home/worsch/railiance-bootstrap`.
## Operator decision
Choose one before executing T12:
### Option A — Keep Gitea as documented fallback (recommended until 2026-10-08)
- Leave coulombcore `gitea` Helm release running.
- Document in runbooks: read-only archive, no new pushes.
- Revisit retirement after 90-day rollback window.
- **T12 done when:** Forgejo is sole *active* forge; Gitea retained intentionally.
### Option B — Retire Gitea Helm release
Only after prerequisites + explicit approval:
1. Re-run dependency scan (State Hub remotes, ArgoCD `repoURL`, CI workflows).
2. Take final Gitea backup (`forgejo dump` equivalent via Gitea admin or volume snapshot).
3. Export archived repo list for audit.
4. Scale down or uninstall `gitea` Helm on coulombcore.
5. Update `docs/forgejo-cutover-rollback.md` — rollback becomes volume-restore only.
6. Prune DNS/ingress if no longer needed.
## T12 task checklist
- [ ] T04/T09 7-day gate **7/7**
- [ ] Operator chooses Option A or B
- [ ] Final Gitea backup captured and catalogued
- [ ] No active remotes/webhooks/dashboards on Gitea (re-verify)
- [ ] Runbooks point to Forgejo (`deploy-stack.md`, layer repo docs)
- [ ] `railiance-bootstrap` hub alias resolved
- [ ] Stale OCI/npm references migrated to `forgejo.coulomb.social` (follow-up workplans in `railiance-apps`, `activity-core`)
- [ ] If Option B: Gitea Helm retired; evidence logged to State Hub
## References
- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md`
- `docs/forgejo-cutover-rollback.md`
- `docs/forgejo-migration-inventory.md`
- `railiance-platform/docs/forgejo-backup.md`
- `the-custodian/tools/gitea-set-readonly.sh`

View file

@ -366,8 +366,9 @@ classification, and admin snapshots (Forgejo full; Gitea partial without
`read:admin` PAT). Residual `read:admin` sweep is **out of scope** for cutover
— sample repos showed 0 deploy keys and 0 webhooks; org webhooks 0 on both forges.
**Remaining before T11 signoff:** Gitea repos marked read-only (not deleted);
7-day scheduled backup gate (T04/T09); operator stabilization signoff.
**Remaining before workplan close (2026-07-09):** T04/T09 7-day backup gate
**4/7** (ETA 2026-07-12); T12 operator decision (retire vs keep Gitea fallback).
T11 cutover complete — Gitea repos archived read-only, not deleted.
## Addendum (2026-07-04) — migration ladder and new repos

View file

@ -93,20 +93,25 @@ secret, `mailer.ENABLED=false`, Helm `--timeout=20m`.
Exit marker: `restore-drill-complete`
## Gaps (not closed by this drill)
## Gaps (remaining)
- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured
(`kubectl cnpg` plugin absent on workstation).
- **Encryption at rest:** dump stored locally on workstation for drill only; no
approved backup target wired.
- **Automation:** `railiance-platform` `make forgejo-backup` — daily cron 02:15 UTC;
7-day gate still counting.
- **Scheduled backups:** `railiance-platform` `make forgejo-backup` — daily cron
02:15 UTC; 7-day gate **4/7** as of 2026-07-09 (ETA 2026-07-12). CNPG `Backup`
CRs deferred (logical `pg_dump` via `forgejo-backup` tool).
- **Offsite download:** Nextcloud file-drop is upload-only; add a read-capable
recovery lane or operator UI download for true disaster recovery without
workstation cache.
- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe
`forgejo-restore-drill` before import (SQL import is not idempotent).
## Closed since initial drill (2026-07-04)
- **Encryption + offsite target:** age-encrypted artifacts uploaded to Nextcloud
`forgejo/` (Option A, T02).
- **Automation tooling:** `railiance-platform/tools/cmd/forgejo-backup` +
`docs/forgejo-backup.md`.
- **Nextcloud restore drill:** passed 2026-07-08 using artifact `20260707T180844Z`.
## Cleanup
After evidence capture, delete the drill namespace: