WP-0005: T12 prep checklist, backup gate 4/7, doc refresh
Update workplan with Jul 9 cron success (4/7 gate, ETA Jul 12), add Gitea retirement dependency scan and operator checklist, and align deploy-stack, restore-drill evidence, and migration inventory with post-cutover state.
This commit is contained in:
parent
0c3968a108
commit
040ccccc8e
5 changed files with 162 additions and 46 deletions
|
|
@ -11,7 +11,8 @@ Before starting, verify you have:
|
|||
- [ ] SSH access to the target server (COULOMBCORE: 92.205.130.254, user: tegwick, key: `~/.ssh/id_ops`)
|
||||
- [ ] SOPS age private key available (`~/.config/sops/age/keys.txt` or `SOPS_AGE_KEY` env)
|
||||
- [ ] ops-bridge running on the workstation (needed for state hub MCP): `make mcp-http` in `~/the-custodian/state-hub/`
|
||||
- [ ] Gitea accessible (needed for git pull on remote): SSH via `gitea-remote:coulomb/<repo>.git`
|
||||
- [ ] Forgejo accessible (git pull on remote): SSH via `forgejo-remote:coulomb/<repo>.git`
|
||||
- [ ] Gitea on coulombcore remains read-only fallback only (`gitea-remote` for emergency rollback)
|
||||
- [ ] If re-provisioning from scratch: Hetzner/HostEurope API credentials decryptable via SOPS
|
||||
|
||||
---
|
||||
|
|
@ -64,26 +65,22 @@ cnpg-system namespace active.
|
|||
# On CoulombCore (kubectl available after S2)
|
||||
cd ~/railiance-platform && git pull
|
||||
|
||||
# Create Gitea DB credentials secret (one-time; do NOT commit plaintext)
|
||||
kubectl create secret generic gitea-db-credentials \
|
||||
--namespace databases \
|
||||
--from-literal=username=gitea \
|
||||
--from-literal=password=<GITEA_DB_PASSWORD>
|
||||
# Deploy Forgejo database cluster (production on railiance01)
|
||||
# Credentials via SOPS — see railiance-platform docs/forgejo-backup.md
|
||||
make forgejo-db-deploy
|
||||
make forgejo-db-status
|
||||
|
||||
# Deploy cnpg Gitea database cluster
|
||||
make db-deploy
|
||||
# Legacy coulombcore Gitea DB (read-only fallback — retain until T12 retires Gitea)
|
||||
# make db-deploy && make db-status
|
||||
|
||||
# Wait for cluster to be healthy (~60s)
|
||||
make db-status
|
||||
|
||||
# Deploy Valkey cache (standalone, not as Gitea subchart)
|
||||
# Deploy Valkey cache if required by workloads
|
||||
# Requires: helm/valkey-values.sops.yaml with encrypted password
|
||||
make valkey-deploy
|
||||
make valkey-status
|
||||
```
|
||||
|
||||
**Checkpoint:** `kubectl get cluster -n databases` shows `gitea-db` healthy;
|
||||
Valkey pod Running in platform namespace.
|
||||
**Checkpoint:** `kubectl get cluster -n databases` shows `forgejo-db` healthy on
|
||||
railiance01; Valkey pod Running if deployed.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -100,22 +97,19 @@ violation, tracked in RAIL-HO-WP-0004). No S4-specific steps required at this ti
|
|||
# On CoulombCore
|
||||
cd ~/railiance-apps && git pull
|
||||
|
||||
# Deploy Gitea (git hosting)
|
||||
# Requires: helm/gitea-values.sops.yaml with encrypted values
|
||||
make gitea-deploy
|
||||
make gitea-status
|
||||
# Deploy Forgejo (production source forge on railiance01)
|
||||
# See railiance-apps/docs/forgejo-on-railiance01.md
|
||||
make forgejo-deploy
|
||||
make forgejo-verify
|
||||
|
||||
# Deploy state-hub (Custodian cognitive infrastructure)
|
||||
# See RAIL-HO-WP-0004-T09 for full steps
|
||||
make state-hub-deploy # (not yet implemented — pending T09)
|
||||
# Legacy Gitea on coulombcore — read-only fallback; do not use for new repos
|
||||
# make gitea-deploy && make gitea-status
|
||||
|
||||
# Deploy activity-core
|
||||
# See RAIL-HO-WP-0004-T10 for full steps
|
||||
make activity-core-deploy # (not yet implemented — pending T10)
|
||||
# Deploy state-hub, activity-core, and other workloads per railiance-apps Makefile
|
||||
```
|
||||
|
||||
**Checkpoint:** Gitea accessible and all repos cloneable via SSH; state-hub
|
||||
`/state/health` returns 200.
|
||||
**Checkpoint:** `https://forgejo.coulomb.social` serves HTTPS; git clone/push via
|
||||
`forgejo-remote`; state-hub `/state/health` returns 200.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -135,10 +129,12 @@ Verify: `bridge status` shows all three connected.
|
|||
|
||||
## Recovery pointers
|
||||
|
||||
- **Forgejo backup/restore:** `railiance-platform/docs/forgejo-backup.md`, `docs/forgejo-restore-drill-evidence.md`
|
||||
- **Cutover rollback:** `docs/forgejo-cutover-rollback.md`
|
||||
- **Node overload / SSH unresponsive:** See `the-custodian/ops/runbooks/gitea-coulombcore.md` Issue #3
|
||||
- **Incident report:** `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
|
||||
- **Cluster backup restore:** `railiance-cluster/tools/cmd/railiance-restore-s2`
|
||||
- **Gitea SSH not working:** Check `gitea-ssh-nodeport` service exists: `kubectl get svc -n default gitea-ssh-nodeport`
|
||||
- **Forgejo SSH:** NodePort `30022` on railiance01 (`forgejo-ssh-nodeport` in `forgejo` namespace)
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
104
docs/forgejo-gitea-retirement-checklist.md
Normal file
104
docs/forgejo-gitea-retirement-checklist.md
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
# Gitea Retirement Checklist (T12 prep)
|
||||
|
||||
Date: 2026-07-09
|
||||
Workplan: `RAIL-HO-WP-0005-T12`
|
||||
Status: **prep only** — no Gitea retirement executed
|
||||
|
||||
## Prerequisites (not yet met)
|
||||
|
||||
| Gate | Status | ETA |
|
||||
| --- | --- | --- |
|
||||
| T04/T09 7-day backup gate | **4/7** (`make forgejo-backup-status`) | **2026-07-12** if daily 02:15 UTC cron keeps succeeding |
|
||||
| `state-hub` stabilization window | Ends **2026-07-09** | Met |
|
||||
| Operator approval (`needs_human`) | Pending | Required before any Helm teardown |
|
||||
|
||||
Monitor:
|
||||
|
||||
```bash
|
||||
cd ~/railiance-platform && make forgejo-backup-status
|
||||
tail -5 ~/.cache/railiance/backups/forgejo/cron.log
|
||||
```
|
||||
|
||||
## Dependency scan (2026-07-09)
|
||||
|
||||
### Clear — no active git remotes on Gitea
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| State Hub `remote_url` | **82 Forgejo**, **0 Gitea**, **0** without remote |
|
||||
| Workstation `origin` using `gitea-remote` | **0** (legacy `gitea` remotes retained for rollback only) |
|
||||
| Forgejo Actions workflows referencing `gitea-remote` | **0** |
|
||||
|
||||
### Gitea still running (intentional fallback)
|
||||
|
||||
| Host | Release | Namespace | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| coulombcore (`~/.kube/config`) | `gitea` Helm 12.5.0 | `default` | Archived read-only repos; package push disabled |
|
||||
| railiance01 (`config-hosteurope`) | `forgejo` Helm 12.5.0 | `forgejo` | Production canonical |
|
||||
|
||||
Rollback runbook: `docs/forgejo-cutover-rollback.md` (90-day no-delete contract).
|
||||
|
||||
### Stale references — docs/config, not live git remotes
|
||||
|
||||
These do **not** block Forgejo operation but should be cleaned before or during
|
||||
Gitea retirement:
|
||||
|
||||
| Area | Examples | Owner repo |
|
||||
| --- | --- | --- |
|
||||
| OCI image repos in Helm values | `gitea.coulomb.social/coulomb/inter-hub`, `reuse-surface`, `vergabe-teilnahme` | `railiance-apps` |
|
||||
| npm registry URLs | `gitea.coulomb.social/api/packages/coulomb/npm/` | `secrets-engine`, `activity-core` |
|
||||
| Ops inventory endpoint | `activity-core/k8s/railiance/20-runtime.yaml` | `activity-core` |
|
||||
| Deploy runbook | `docs/deploy-stack.md` (this repo) | `railiance-infra` — updated 2026-07-09 |
|
||||
|
||||
### State Hub alias: `railiance-bootstrap`
|
||||
|
||||
| Field | `railiance-bootstrap` | `railiance-cluster` |
|
||||
| --- | --- | --- |
|
||||
| `local_path` | `/home/worsch/railiance-bootstrap` | `/home/worsch/railiance-cluster` |
|
||||
| `remote_url` | `forgejo-remote:coulomb/railiance-bootstrap.git` | `forgejo-remote:coulomb/railiance-cluster.git` |
|
||||
| `git_fingerprint` | `6b930728…` (same) | `6b930728…` (same) |
|
||||
|
||||
Working assumption confirmed: stale logical alias. T12 action: operator decides
|
||||
whether to archive the hub entry or repoint `local_path` to `railiance-cluster`
|
||||
after confirming no checkout depends on `/home/worsch/railiance-bootstrap`.
|
||||
|
||||
## Operator decision
|
||||
|
||||
Choose one before executing T12:
|
||||
|
||||
### Option A — Keep Gitea as documented fallback (recommended until 2026-10-08)
|
||||
|
||||
- Leave coulombcore `gitea` Helm release running.
|
||||
- Document in runbooks: read-only archive, no new pushes.
|
||||
- Revisit retirement after 90-day rollback window.
|
||||
- **T12 done when:** Forgejo is sole *active* forge; Gitea retained intentionally.
|
||||
|
||||
### Option B — Retire Gitea Helm release
|
||||
|
||||
Only after prerequisites + explicit approval:
|
||||
|
||||
1. Re-run dependency scan (State Hub remotes, ArgoCD `repoURL`, CI workflows).
|
||||
2. Take final Gitea backup (`forgejo dump` equivalent via Gitea admin or volume snapshot).
|
||||
3. Export archived repo list for audit.
|
||||
4. Scale down or uninstall `gitea` Helm on coulombcore.
|
||||
5. Update `docs/forgejo-cutover-rollback.md` — rollback becomes volume-restore only.
|
||||
6. Prune DNS/ingress if no longer needed.
|
||||
|
||||
## T12 task checklist
|
||||
|
||||
- [ ] T04/T09 7-day gate **7/7**
|
||||
- [ ] Operator chooses Option A or B
|
||||
- [ ] Final Gitea backup captured and catalogued
|
||||
- [ ] No active remotes/webhooks/dashboards on Gitea (re-verify)
|
||||
- [ ] Runbooks point to Forgejo (`deploy-stack.md`, layer repo docs)
|
||||
- [ ] `railiance-bootstrap` hub alias resolved
|
||||
- [ ] Stale OCI/npm references migrated to `forgejo.coulomb.social` (follow-up workplans in `railiance-apps`, `activity-core`)
|
||||
- [ ] If Option B: Gitea Helm retired; evidence logged to State Hub
|
||||
|
||||
## References
|
||||
|
||||
- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md`
|
||||
- `docs/forgejo-cutover-rollback.md`
|
||||
- `docs/forgejo-migration-inventory.md`
|
||||
- `railiance-platform/docs/forgejo-backup.md`
|
||||
- `the-custodian/tools/gitea-set-readonly.sh`
|
||||
|
|
@ -366,8 +366,9 @@ classification, and admin snapshots (Forgejo full; Gitea partial without
|
|||
`read:admin` PAT). Residual `read:admin` sweep is **out of scope** for cutover
|
||||
— sample repos showed 0 deploy keys and 0 webhooks; org webhooks 0 on both forges.
|
||||
|
||||
**Remaining before T11 signoff:** Gitea repos marked read-only (not deleted);
|
||||
7-day scheduled backup gate (T04/T09); operator stabilization signoff.
|
||||
**Remaining before workplan close (2026-07-09):** T04/T09 7-day backup gate
|
||||
**4/7** (ETA 2026-07-12); T12 operator decision (retire vs keep Gitea fallback).
|
||||
T11 cutover complete — Gitea repos archived read-only, not deleted.
|
||||
|
||||
## Addendum (2026-07-04) — migration ladder and new repos
|
||||
|
||||
|
|
|
|||
|
|
@ -93,20 +93,25 @@ secret, `mailer.ENABLED=false`, Helm `--timeout=20m`.
|
|||
|
||||
Exit marker: `restore-drill-complete`
|
||||
|
||||
## Gaps (not closed by this drill)
|
||||
## Gaps (remaining)
|
||||
|
||||
- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured
|
||||
(`kubectl cnpg` plugin absent on workstation).
|
||||
- **Encryption at rest:** dump stored locally on workstation for drill only; no
|
||||
approved backup target wired.
|
||||
- **Automation:** `railiance-platform` `make forgejo-backup` — daily cron 02:15 UTC;
|
||||
7-day gate still counting.
|
||||
- **Scheduled backups:** `railiance-platform` `make forgejo-backup` — daily cron
|
||||
02:15 UTC; 7-day gate **4/7** as of 2026-07-09 (ETA 2026-07-12). CNPG `Backup`
|
||||
CRs deferred (logical `pg_dump` via `forgejo-backup` tool).
|
||||
- **Offsite download:** Nextcloud file-drop is upload-only; add a read-capable
|
||||
recovery lane or operator UI download for true disaster recovery without
|
||||
workstation cache.
|
||||
- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe
|
||||
`forgejo-restore-drill` before import (SQL import is not idempotent).
|
||||
|
||||
## Closed since initial drill (2026-07-04)
|
||||
|
||||
- **Encryption + offsite target:** age-encrypted artifacts uploaded to Nextcloud
|
||||
`forgejo/` (Option A, T02).
|
||||
- **Automation tooling:** `railiance-platform/tools/cmd/forgejo-backup` +
|
||||
`docs/forgejo-backup.md`.
|
||||
- **Nextcloud restore drill:** passed 2026-07-08 using artifact `20260707T180844Z`.
|
||||
|
||||
## Cleanup
|
||||
|
||||
After evidence capture, delete the drill namespace:
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: railiance
|
||||
topic_slug: railiance
|
||||
created: "2026-05-03"
|
||||
updated: "2026-07-08"
|
||||
updated: "2026-07-09"
|
||||
state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018"
|
||||
---
|
||||
|
||||
|
|
@ -118,7 +118,7 @@ Still to prove before T11 closeout:
|
|||
|
||||
- ~~SMTP/password reset end-to-end (T06).~~ **done** (2026-07-08).
|
||||
- ~~Backup restore from offsite artifact (T09).~~ **done** (2026-07-08); 7-day
|
||||
scheduled backup gate **3/7** (cron PATH fixed 2026-07-08).
|
||||
scheduled backup gate **4/7** (Jul 9 02:15 UTC cron succeeded; ETA **2026-07-12**).
|
||||
- ~~Repo migration ladder (T10).~~ **done** (2026-07-08); 0 Gitea-only repos.
|
||||
- ~~Operator SSH on Forgejo as `tegwick` (T05/T10).~~ **done**.
|
||||
- ~~Gitea read-only marking + operator signoff (T11).~~ **done** (2026-07-08).
|
||||
|
|
@ -257,8 +257,10 @@ decided (Option A, Nextcloud).** **Automation:** `railiance-platform`
|
|||
`pg_dump` → age → Nextcloud `forgejo/`. Daily cron 02:15 UTC; **cron PATH fixed
|
||||
2026-07-08** (`kubectl` was missing from cron env — Jul 8 02:15 run failed).
|
||||
**Nextcloud restore drill passed 2026-07-08.** Manual backup
|
||||
`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Remaining: 7-day
|
||||
success gate (**3/7** as of 2026-07-08).
|
||||
`20260708T131118Z` (~890M dump + 13M pg_dump) succeeded. Jul 8 02:15 cron
|
||||
failed (`kubectl` missing from cron PATH); fixed same day. Jul 9 02:15 cron
|
||||
succeeded (`20260709T001505Z`, ~967M dump + 20M pg_dump). Remaining: 7-day
|
||||
success gate (**4/7** as of 2026-07-09; ETA **2026-07-12**).
|
||||
|
||||
**Done when:** platform dependencies can be deployed and restored without the
|
||||
Forgejo app running.
|
||||
|
|
@ -457,9 +459,9 @@ Makefile targets. **First scheduled backup:** 20260707T180844Z (~686M, 14 repos)
|
|||
Daily cron 02:15 UTC. **Nextcloud restore drill passed 2026-07-08** — decrypt
|
||||
uploaded `.age` artifact, chunked copy, 14 org repos restored in
|
||||
`forgejo-restore-drill` (evidence: `docs/forgejo-restore-drill-evidence.md`).
|
||||
Remaining: 7 consecutive backup successes (**3/7** as of 2026-07-08); cron PATH
|
||||
fixed so nightly runs resume. Nextcloud file-drop is upload-only — read-lane TBD
|
||||
for DR without workstation cache.
|
||||
Remaining: 7 consecutive backup successes (**4/7** as of 2026-07-09; ETA
|
||||
**2026-07-12**). Jul 9 02:15 UTC cron run succeeded after PATH fix. Nextcloud
|
||||
file-drop is upload-only — read-lane TBD for DR without workstation cache.
|
||||
|
||||
**Done when:** a fresh backup restores to a working isolated Forgejo instance
|
||||
with repository, package, and user recovery checks passing **and** scheduled
|
||||
|
|
@ -583,6 +585,12 @@ state_hub_task_id: "a63147b0-31d5-4705-89ea-40c10faf779f"
|
|||
|
||||
Retire legacy Gitea only after a stabilization period and explicit approval.
|
||||
|
||||
**Prep (2026-07-09):** dependency scan complete — State Hub **0 Gitea remotes**,
|
||||
workstation **0 `gitea-remote` origins**; coulombcore `gitea` Helm still running
|
||||
as read-only fallback. Checklist: `docs/forgejo-gitea-retirement-checklist.md`.
|
||||
Operator must choose: **Option A** keep fallback (recommended until 90-day
|
||||
rollback window) or **Option B** retire Helm release.
|
||||
|
||||
Minimum scope:
|
||||
|
||||
- Confirm no active remotes, webhooks, packages, or dashboards depend on Gitea.
|
||||
|
|
@ -615,9 +623,10 @@ T05+T08 ──► T10 migration ladder ──► T11 production cutover ──
|
|||
T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots)
|
||||
```
|
||||
|
||||
**Current focus (2026-07-08):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09
|
||||
await 7-day backup gate (**3/7**; cron PATH fixed). T12 next after
|
||||
stabilization window — Gitea archived + package push disabled; ArgoCD on Forgejo.
|
||||
**Current focus (2026-07-09):** T01/T05/T06/T07/T08/T10/T11 **done**; T04/T09
|
||||
await 7-day backup gate (**4/7**, ETA **2026-07-12**). T12 prep complete —
|
||||
operator decision pending (retire vs keep fallback). `state-hub` stabilization
|
||||
window closed 2026-07-09.
|
||||
|
||||
**Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation
|
||||
build retirement; staged repo promotion before State Hub primary move (T05).
|
||||
|
|
@ -646,6 +655,7 @@ purpose is identified.
|
|||
- `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md`
|
||||
- `docs/forgejo-migration-inventory.md`
|
||||
- `docs/forgejo-cutover-rollback.md`
|
||||
- `docs/forgejo-gitea-retirement-checklist.md`
|
||||
- `the-custodian/docs/forgejo-production-decisions.md`
|
||||
- `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`
|
||||
- `railiance-apps/docs/forgejo-on-railiance01.md`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue