diff --git a/.claude/rules/credential-routing.md b/.claude/rules/credential-routing.md index fe1b70b..c0fa603 100644 --- a/.claude/rules/credential-routing.md +++ b/.claude/rules/credential-routing.md @@ -20,7 +20,7 @@ Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run wa | Agent runtime | How to orient | | --- | --- | | **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending | -| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership | +| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership | | **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` | ### Quick routing table diff --git a/.claude/rules/first-session.md b/.claude/rules/first-session.md index 0d0b183..412c4c4 100644 --- a/.claude/rules/first-session.md +++ b/.claude/rules/first-session.md @@ -1,37 +1,41 @@ ## First Session Protocol -Triggered when `get_domain_summary("railiance")` shows **no workstreams**. +Triggered when `get_domain_summary("financials")` shows **no workplans**. The project is registered but work has not yet been structured. **Step 1 — Read, don't write** -- `~/the-custodian/canon/projects/railiance/project_charter_v0.1.md` — purpose, scope -- `~/the-custodian/canon/projects/railiance/roadmap_v0.1.md` — planned phases +- `~/the-custodian/canon/projects/financials/project_charter_v0.1.md` — purpose, scope +- `~/the-custodian/canon/projects/financials/roadmap_v0.1.md` — planned phases - Scan repo root: README, directory structure, existing code or docs **Step 2 — Survey in-progress work** Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete. -**Step 3 — Propose workstreams to Bernd** -Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a +**Step 3 — Propose workplans to Bernd** +Propose 1–3 workplans — each a coherent strand, weeks to months, anchored to a roadmap phase. **Wait for approval before creating.** -**Step 4 — Create workplan file first, then DB record (ADR-001)** +**Step 4 — Write the workplan file; fix-consistency registers it (ADR-001)** ``` -workplans/railiance-hosts-WP-NNNN-.md ← write this first +workplans/RAIL-HO-WP-NNNN-.md ← write this, commit it ``` -Then register in the hub: -``` -create_workstream(topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", title="...", owner="...", description="...") -create_task(workstream_id="", title="...", priority="high|medium|low") +Then register by running the consistency check — do **not** call +`create_workplan`/`create_task` (or legacy `create_workstream`) yourself; +manual registration duplicates what C-06 creates from the file: +```bash +statehub fix-consistency --repo railiance-infra ``` +C-06 creates the hub workplan + tasks and writes `state_hub_workstream_id` / +`state_hub_task_id` back into the file (legacy field names, kept for +compatibility — they hold workplan/task IDs). **Step 5 — Record the setup** ``` add_progress_event( - summary="First session: structured railiance into N workstreams, M tasks", + summary="First session: structured financials into N workplans, M tasks", event_type="milestone", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", - detail={"workstreams": [...], "tasks_created": M} + detail={"workplans": [...], "tasks_created": M} ) ``` diff --git a/.claude/rules/repo-boundary.md b/.claude/rules/repo-boundary.md index 335c457..8b48c17 100644 --- a/.claude/rules/repo-boundary.md +++ b/.claude/rules/repo-boundary.md @@ -1,6 +1,6 @@ ## Repo boundary -This repo owns **railiance-hosts** only. It does not own: +This repo owns **railiance-infra** only. It does not own: -- **Language:** -- **Key deps:** +- **Language:** Terraform (HCL), cloud-init, Ansible, Bash +- **Key deps:** Hetzner/HostEurope provisioning, SOPS + age for secrets, repo-local git hooks (`.githooks`) ## Dev Commands ```bash -# TODO: Fill in the standard commands for this repo - -# Install dependencies - -# Run tests - -# Lint / type check - -# Build / package (if applicable) +make help # list all targets +make hooks # configure repo-local git hooks (secrets guard) +make check-secrets # fail if anything in secrets/ is not SOPS-encrypted +make sops-edit # edit global secrets with SOPS +make sops-encrypt FILE=secrets/foo.yaml +make tf-fmt # terraform fmt +make tf-init && make tf-plan # plan (needs decrypted HCLOUD_TOKEN) +make tf-apply # provision — production-touching +make tf-destroy # tear down — destructive, operator only ``` + +`tf-apply`/`tf-destroy` change production infrastructure — operator approval +required. No unit test suite; validation is `tf-plan` + the server test suite +workplans. diff --git a/.claude/rules/workplan-convention.md b/.claude/rules/workplan-convention.md index 3c92987..cf5e620 100644 --- a/.claude/rules/workplan-convention.md +++ b/.claude/rules/workplan-convention.md @@ -1,28 +1,45 @@ ## Workplan Convention (ADR-001) -File location: `workplans/railiance-hosts-WP-NNNN-.md` -ID prefix: `RAILIANCE-WP` +File location: `workplans/RAIL-HO-WP-NNNN-.md` +ID prefix: `RAIL-HO-WP-` Work items originate as files in this repo **before** being registered in the hub. -Canonical workplan/workstream frontmatter statuses are: +Canonical workplan frontmatter statuses are: `proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`. Use `proposed` for a newly drafted plan, `ready` after review against current repo state, and `finished` when implementation is complete. `stalled` and `needs_review` are derived health labels, not stored statuses. Closed workplans may be moved to `workplans/archived/` with a completion-date -prefix: `YYMMDD-railiance-hosts-WP-NNNN-.md`. The frontmatter id remains +prefix: `YYMMDD-RAIL-HO-WP-NNNN-.md`. The frontmatter id remains unchanged; the prefix is only for quick visual reference. Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**: -`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids +`workplans/ADHOC-YYYY-MM-DD.md`, workplan slug `adhoc-YYYY-MM-DD`, and task ids `ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed directly. Promote anything requiring analysis, design, approval, dependencies, or multiple planned phases into a normal workplan. -Ecosystem todos from other agents arrive as `[repo:railiance-hosts]` hub tasks — -visible at session start. Pick one up by creating the workplan file, then registering -the workstream. +Ecosystem todos from other agents arrive as `[repo:railiance-infra]` hub tasks — +visible at session start. Pick one up by creating the workplan file, committing, +and running `statehub fix-consistency` — C-06 registers the workplan in the hub. +Never register by hand with `create_workplan`/`create_workstream`. + +Task blocks use this shape: + +```task +id: RAIL-HO-WP-NNNN-T01 +status: wait | todo | progress | done | cancel +priority: high | medium | low +state_hub_task_id: "" # written by fix-consistency — do not edit +``` + +Status progression is `todo` → `progress` → `done`; use `wait` for waiting or +blocked work and `cancel` for stopped work. + +Workplan frontmatter carries `state_hub_workstream_id` — a legacy field name +kept for compatibility ("workstream" is the old term for workplan); it holds +the hub workplan id and is written by fix-consistency. Do not edit or rename it. diff --git a/.custodian-brief.md b/.custodian-brief.md index 879628d..8da959e 100644 --- a/.custodian-brief.md +++ b/.custodian-brief.md @@ -2,23 +2,23 @@ # Custodian Brief — railiance-infra **Domain:** financials -**Last synced:** 2026-06-22 20:26 UTC +**Last synced:** 2026-07-04 09:27 UTC **State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)* ## Active Workstreams ### Forgejo Production Migration on railiance01 -Progress: 0/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018` +Progress: 1/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018` **Open tasks:** - ► T01 — Inventory current Gitea functionality and migration requirements `cf59d171` -- · T02 — Resolve Forgejo production design decisions `f88115bf` -- · T03 — Build forgejo-railiance-probe `b516018a` +- ► T02 — Resolve Forgejo production design decisions `f88115bf` +- ► T05 — Define production Forgejo application deployment `11540ba4` +- ► T08 — Enable Forgejo Actions `f45f98c9` +- ► T09 — Implement Forgejo backup and restore automation `25892007` +- ► T10 — Drill Gitea to Forgejo migration `6befde73` - · T04 — Define Forgejo platform services `28b351fe` -- · T05 — Define production Forgejo application deployment `11540ba4` -- · T06 — Implement usable email recovery cycle `417faa4d` -- · T07 — Enable and harden package registry base `9578f672` -- … and 5 more open tasks +- … and 4 more open tasks --- ## MCP Orientation (when available) diff --git a/.forgejo/workflows/ci-smoke.yaml b/.forgejo/workflows/ci-smoke.yaml new file mode 100644 index 0000000..bd44c56 --- /dev/null +++ b/.forgejo/workflows/ci-smoke.yaml @@ -0,0 +1,29 @@ +# Canonical CI smoke template (tier 1 routing drill). +# Copy to: .forgejo/workflows/ci-smoke.yaml in consumer repos. +name: CI Smoke + +on: + push: + branches: + - main + workflow_dispatch: + +jobs: + host-smoke: + runs-on: self-hosted + steps: + - name: Routing probe (host runner) + run: | + set -eu + echo "repository=${GITHUB_REPOSITORY:-unknown}" + echo "sha=${GITHUB_SHA:-unknown}" + echo "runner=${RUNNER_NAME:-unknown}" + uname -a + + container-smoke: + runs-on: ubuntu-latest + steps: + - name: Routing probe (container label) + run: | + set -eu + echo "container-smoke ok for ${GITHUB_REPOSITORY:-unknown}" \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index e09873b..ed01c8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,13 +1,13 @@ -# railiance-hosts — Agent Instructions +# railiance-infra — Agent Instructions ## Repo Identity -**Purpose:** Host inventory and node-level configuration for railiance infrastructure. Tracks server specs, network topology, and per-host service assignments. +**Purpose:** OAS S1 Infrastructure Substrate — OS provisioning, Ansible roles, Goss spec, Terraform, inventory -**Domain:** railiance -**Repo slug:** railiance-hosts +**Domain:** financials +**Repo slug:** railiance-infra **Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38` -**Workplan prefix:** `RAILIANCE-WP-` +**Workplan prefix:** `RAIL-HO-WP-` --- @@ -20,6 +20,12 @@ there is no MCP server for Codex agents. |---------|-----| | Local workstation | `http://127.0.0.1:8000` | | Remote via tunnel | `http://127.0.0.1:18000` | +| Optional local edge relay | http://127.0.0.1:18080 | + +When an operator has enabled the edge relay, set API_BASE to the relay URL. +Queueable writes return an explicit queued receipt if the central hub is +unreachable. Treat that as pending local evidence, then ask the operator to run +statehub outbox status/replay after connectivity returns. ### Orient at session start @@ -27,12 +33,12 @@ there is no MCP server for Codex agents. # Offline brief — works without hub connection cat .custodian-brief.md -# Active workstreams for this domain -curl -s "http://127.0.0.1:8000/workstreams/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \ +# Active workplans for this domain +curl -s "http://127.0.0.1:8000/workplans/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \ | python3 -m json.tool # Check inbox -curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-hosts&unread_only=true" \ +curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-infra&unread_only=true" \ | python3 -m json.tool ``` @@ -51,20 +57,20 @@ curl -s -X POST http://127.0.0.1:8000/progress/ \ "summary": "what was done", "event_type": "note", "author": "codex", - "workstream_id": "", + "workplan_id": "", "task_id": "" }' ``` -Omit `workstream_id` / `task_id` when not applicable. +Omit `workplan_id` / `task_id` when not applicable. ### Update task status ```bash curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ -H "Content-Type: application/json" \ - -d '{"status": "in_progress"}' -# values: todo | in_progress | done | blocked + -d '{"status": "progress"}' +# values: wait | todo | progress | done | cancel ``` ### Flag a task for human review @@ -80,10 +86,10 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ ## Session Protocol **Start:** -1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe) -2. Check inbox: `GET /messages/?to_agent=railiance-hosts&unread_only=true`; mark read +1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe) +2. Check inbox: `GET /messages/?to_agent=railiance-infra&unread_only=true`; mark read 3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks -4. Check blocked tasks: `GET /tasks/?needs_human=true` +4. Check human-needed tasks: `GET /tasks/?needs_human=true` **During work:** - Update task statuses in workplan files as tasks progress @@ -92,12 +98,12 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ **Close:** 1. Update workplan file task statuses to reflect progress 2. Log: `POST /progress/` with a summary of what changed -3. Note for the custodian operator: after workplan file changes, run from - `~/state-hub`: +3. After workplan file changes, run: ```bash - make fix-consistency REPO=railiance-hosts + statehub fix-consistency ``` - This syncs task status from files into the hub DB. + Coding agents should run this directly; ask the operator only if the CLI or + State Hub API is unavailable. This syncs task status from files into the hub DB. --- @@ -123,7 +129,7 @@ Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run wa | Agent runtime | How to orient | | --- | --- | | **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending | -| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership | +| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership | | **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` | ### Quick routing table @@ -151,6 +157,11 @@ every repo's agent instructions because it is high-frequency, high-risk, and eas get wrong. **Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml` + + + + --- ## Workplan Convention (ADR-001) @@ -176,8 +187,8 @@ anything needing analysis, design, approval, dependencies, or multiple phases. id: RAILIANCE-WP-NNNN type: workplan title: "..." -domain: railiance -repo: railiance-hosts +domain: financials +repo: railiance-infra status: proposed | ready | active | blocked | backlog | finished | archived owner: codex topic_slug: ... @@ -198,7 +209,7 @@ derived health labels, not frontmatter statuses. ` ` `task id: RAILIANCE-WP-NNNN-T01 -status: todo | in_progress | done | blocked +status: wait | todo | progress | done | cancel priority: high | medium | low state_hub_task_id: "" # written by fix-consistency — do not edit ` ` ` @@ -206,9 +217,9 @@ state_hub_task_id: "" # written by fix-consistency — do not edit Task description text. ``` -Status progression: `todo` → `in_progress` → `done` (or `blocked`) +Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work. To create a new workplan: 1. Write the file following the format above -2. Notify the custodian operator to run `make fix-consistency REPO=railiance-hosts` +2. Notify the custodian operator to run `make fix-consistency REPO=railiance-infra` (or send a message to the hub agent via `POST /messages/`) diff --git a/CLAUDE.md b/CLAUDE.md index ef28d7b..871d83d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,4 +1,4 @@ -# railiance-hosts — Claude Code Instructions +# railiance-infra — Claude Code Instructions @SCOPE.md @.claude/rules/repo-identity.md diff --git a/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md b/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md new file mode 100644 index 0000000..1f56d2a --- /dev/null +++ b/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md @@ -0,0 +1,104 @@ +# ADR-004 — Forgejo In-Cluster Actions Runner on railiance01 + +**Status:** Accepted +**Date:** 2026-07-03 +**Deciders:** Bernd Worsch (operator), custodian agents +**Workplans:** `RAIL-HO-WP-0005-T02`, `CUST-WP-0054-T04` + +--- + +## Context + +Forgejo production runs on **railiance01 k3s** (`railiance-apps`, S5). An interim +**host runner** on coulombcore proved Actions scheduling (`coulomb/forgejo-actions-probe`) +but: + +- coulombcore is a legacy machine slated for drain (CUST-WP-0054-T03). +- Host runners require Docker or Podman on the OS — not installed, not desired on + coulombcore long term. +- Forgejo upstream recommends **not** co-locating runners on the same machine as the + forge instance; in-cluster **separate pods** satisfy isolation while staying on the + production fleet node. +- `RAIL-HO-WP-0005-T02` left the runner model undecided among host, in-cluster, and + ephemeral options. + +Goal: a **coherent Kubernetes-from-the-start** CI substrate — Forgejo app, database, +ingress, and Actions runner all lifecycle-managed on railiance01. + +## Decision + +### Runner placement + +Deploy **one long-lived Forgejo Actions runner Deployment** in the `forgejo` namespace +on railiance01: + +| Component | Implementation | +| --- | --- | +| Runner | `data.forgejo.org/forgejo/runner:6.3.1` | +| Container runtime for jobs | `docker:dind` sidecar (privileged) | +| State | PVC `forgejo-runner-data` (`.runner`, `config.yaml`, action cache) | +| Registration scope | `coulomb` organization | +| Runner name | `railiance01-build-01` | +| Deploy surface | `railiance-apps/manifests/forgejo-runner.yaml` | +| Operator targets | `make forgejo-runner-deploy`, `forgejo-runner-status` | + +### Label contract + +Preserve Gitea migration compatibility and semantic capability labels: + +```text +self-hosted:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host,railiance01:host,ubuntu-latest:docker://node:20-bookworm,docker:docker://node:20-bookworm +``` + +### Security boundaries + +- Runner pod receives **no** cluster-admin kubeconfig and **no** OpenBao tokens by default. +- `registry-publish` jobs use **repo/org-scoped Forgejo secrets** only. +- DinD sidecar runs **privileged** — accepted for single-node railiance01 with + dedicated `forgejo` namespace; revisit when a third node or multi-tenant runners appear. +- Registration tokens live in Kubernetes Secret `forgejo-runner-registration` (SOPS + template committed; live value never in Git). + +### Retire interim host runner + +Stop and disable `forgejo-runner.service` on coulombcore after in-cluster runner is +healthy. Do not register new host runners without an explicit ADR amendment. + +## Alternatives considered + +| Option | Outcome | +| --- | --- | +| Host runner + Docker on coulombcore | Rejected — legacy host, contradicts drain plan | +| Host runner + Podman on haskelseed | Viable fallback; not chosen as primary | +| Kaniko/Buildah without DinD | Deferred — higher workflow churn during Gitea migration | +| Multiple ephemeral runner Jobs | Deferred — start with capacity=1 long-lived pod | + +## Consequences + +**Positive** + +- Single-machine production loop: forge + runner on railiance01, workstation not required. +- Container image CI (`docker build` / `docker push`) works without OS-level Docker. +- Runner upgrades roll with Git-managed manifests and `kubectl`/Makefile. + +**Negative / follow-on** + +- Privileged DinD increases blast radius within the node — monitor and restrict namespace RBAC. +- SOPS-encrypted registration secret still requires operator age key. +- `cluster-deploy` / `s5-release-check` labels remain **out of scope** until credential paths reviewed. + +## Ownership (OAS) + +| Concern | Repo | Layer | +| --- | --- | --- | +| ADR + umbrella sequencing | `railiance-infra` | S1 | +| Runner manifests + Makefile | `railiance-apps` | S5 | +| Label contract + runner evidence docs | `railiance-forge` | S5 forge substrate | +| Reusable workflow templates | `railiance-enablement` | S4 | + +## References + +- `railiance-apps/docs/forgejo-on-railiance01.md` +- `railiance-forge/docs/forgejo-actions-runner-substrate.md` +- `the-custodian/docs/forgejo-production-decisions.md` +- [Forgejo runner installation](https://forgejo.org/docs/v11.0/admin/actions/runner-installation/) \ No newline at end of file diff --git a/docs/forgejo-migration-inventory.md b/docs/forgejo-migration-inventory.md index dd047b4..0131e40 100644 --- a/docs/forgejo-migration-inventory.md +++ b/docs/forgejo-migration-inventory.md @@ -204,3 +204,28 @@ lost or left with an untracked remote. This first pass satisfies the public and infrastructure metadata part of T01. T01 should remain open until the authenticated admin inventory and missing repo classification are complete. + +## Addendum (2026-07-04) — migration ladder and new repos + +`RAIL-HO-WP-0005` now uses a **staged per-repo ladder** instead of an isolated +probe namespace (T03 cancelled). Repos to add or re-classify on next inventory +refresh: + +| Repo | On Gitea (2026-06) | On Forgejo (2026-07-04) | Tier | Notes | +| --- | --- | --- | ---: | --- | +| `forgejo-actions-probe` | — | yes | 0 | Disposable runner/OCI probe | +| `glas-harness` | yes (not in table above) | yes (canonical) | 1 | Git+SSH+CI pilot | +| `key-cape` | yes | yes (canonical) | 2 | Image CI + `crictl pull` on railiance01 | + +Evidence: `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` + +**Tier definitions** (for per-repo `migration tier` column in a future refresh): + +| Tier | Criteria | Examples | +| ---: | --- | --- | +| 0 | Disposable integration probes | `forgejo-actions-probe` | +| 1 | Non-production; git+CI only | `glas-harness` | +| 2 | Non-production with container image + registry pull | `key-cape` (done) | +| 3 | Production drain wave / sweep registration | `state-hub`, `issue-core`, … | + +Production repos stay on Gitea until tier 0–2 gates and T09 backup drill pass. diff --git a/docs/forgejo-restore-drill-evidence.md b/docs/forgejo-restore-drill-evidence.md new file mode 100644 index 0000000..adceb3b --- /dev/null +++ b/docs/forgejo-restore-drill-evidence.md @@ -0,0 +1,93 @@ +# Forgejo Backup/Restore Drill Evidence + +Date: 2026-07-04 +Workplan: RAIL-HO-WP-0005 +Task: RAIL-HO-WP-0005-T09 +`no_secret_material_recorded: true` + +## Purpose + +Prove that a production `forgejo dump` can be restored into an isolated +namespace and serve repository metadata without touching production Forgejo or +Gitea. + +## Backup source + +| Field | Value | +| --- | --- | +| Method | `forgejo dump` from production pod | +| Production pod | `forgejo-gitea-64c5b57684-ph9vt` (namespace `forgejo`) | +| Archive path (workstation) | `/tmp/forgejo-drill/forgejo-drill-backup.zip` | +| Archive size | 12,284,847 bytes (~11.7 MiB) | +| Archive timestamp | 2026-07-04 11:20 +0200 | +| Archive contents (top-level) | `repos/`, `data/`, `forgejo-db.sql`, `app.ini` | + +Repos present in dump: `forgejo-actions-probe`, `glas-harness`, `key-cape` +(all under `repos/coulomb/`). + +## Restore target + +| Field | Value | +| --- | --- | +| Namespace | `forgejo-restore-drill` | +| Database | CNPG cluster `forgejo-db-restore` (isolated, 1 instance) | +| App data PVC | `forgejo-restore-data` (`local-path`, 10Gi) | +| Helm release | `forgejo-restore` (`gitea-charts/gitea` 12.5.0) | +| Orchestration | `tools/forgejo-restore-drill.sh` | + +Restore path (Forgejo 11.0.3 has no `forgejo restore` CLI): + +1. Unzip dump into import pod staging area. +2. Copy `repos/` → `/data/git/gitea-repositories/`. +3. Copy `data/` → `/data/` (packages, attachments, avatars). +4. Import `forgejo-db.sql` via `psql` into `forgejo-db-restore`. +5. Deploy isolated Helm release bound to restored PVC + restore DB host. + +## Post-restore checks (2026-07-04) + +Port-forward: `svc/forgejo-restore-gitea-http` → `127.0.0.1:13000` + +| Check | Result | +| --- | --- | +| `GET /` health | HTTP 200 | +| `GET /api/v1/repos/coulomb/glas-harness` | `full_name=coulomb/glas-harness`, `default_branch=main` | +| `GET /api/v1/repos/coulomb/key-cape` | `full_name=coulomb/key-cape`, `default_branch=main` | +| `GET /api/v1/orgs/coulomb/repos` | 3 repos: `forgejo-actions-probe`, `glas-harness`, `key-cape` | + +Script exit marker: `restore-drill-complete` + +## RPO / RTO (drill scope) + +| Metric | Observed / assumed | +| --- | --- | +| RPO (manual dump) | Point-in-time of `forgejo dump` execution; no scheduled backup yet | +| RTO (isolated restore) | ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01 | +| Production impact | None — read-only dump from running pod; separate namespace | + +## Gaps (not closed by this drill) + +- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured + (`kubectl cnpg` plugin absent on workstation). +- **Encryption at rest:** dump stored locally on workstation for drill only; no + approved backup target wired. +- **Automation:** `forgejo dump` is manual; T04/T09 still need cron/operator + schedule and retention policy (T02 decision). +- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe + `forgejo-restore-drill` before import (SQL import is not idempotent). + +## Cleanup + +After evidence capture, delete the drill namespace: + +```bash +kubectl delete namespace forgejo-restore-drill --wait=true +``` + +Production Forgejo (`forgejo` namespace) and Gitea remain unchanged. + +## References + +- `infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml` +- `infra/forgejo-restore-drill/restore-job.yaml` +- `tools/forgejo-restore-drill.sh` +- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` (T09) \ No newline at end of file diff --git a/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml b/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml new file mode 100644 index 0000000..302c396 --- /dev/null +++ b/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml @@ -0,0 +1,21 @@ +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: forgejo-db-restore + namespace: forgejo-restore-drill + labels: + app.kubernetes.io/name: forgejo-db-restore + railiance.io/layer: s3-platform + railiance.io/consumer: forgejo-restore-drill +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:16 + storage: + size: 10Gi + bootstrap: + initdb: + database: forgejo + owner: forgejo + secret: + name: forgejo-db-credentials \ No newline at end of file diff --git a/infra/forgejo-restore-drill/restore-job.yaml b/infra/forgejo-restore-drill/restore-job.yaml new file mode 100644 index 0000000..0c02577 --- /dev/null +++ b/infra/forgejo-restore-drill/restore-job.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: forgejo-restore-data + namespace: forgejo-restore-drill +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: local-path \ No newline at end of file diff --git a/tools/forgejo-restore-drill.sh b/tools/forgejo-restore-drill.sh new file mode 100755 index 0000000..10fb43e --- /dev/null +++ b/tools/forgejo-restore-drill.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Non-production Forgejo backup/restore drill (RAIL-HO-WP-0005-T09). +# Re-run: DRILL_CLEAN=1 ./tools/forgejo-restore-drill.sh (wipes namespace first) +set -euo pipefail + +KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config-hosteurope}" +export KUBECONFIG +NS=forgejo-restore-drill +DRILL_CLEAN="${DRILL_CLEAN:-0}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +BACKUP_LOCAL="${BACKUP_LOCAL:-/tmp/forgejo-drill/forgejo-drill-backup.zip}" +PROD_POD="${PROD_POD:-$(kubectl get pods -n forgejo -l app.kubernetes.io/instance=forgejo -o jsonpath='{.items[0].metadata.name}')}" + +step() { echo "==> $*"; } + +if [[ "${DRILL_CLEAN}" == "1" ]]; then + step "Clean prior drill namespace ${NS}" + kubectl delete namespace "${NS}" --wait=true --timeout=5m || true +fi + +step "Create namespace ${NS}" +kubectl create namespace "${NS}" --dry-run=client -o yaml | kubectl apply -f - + +step "Copy forgejo-db-credentials into ${NS}" +kubectl get secret forgejo-db-credentials -n databases -o json \ + | python3 -c "import json,sys; s=json.load(sys.stdin); s['metadata']={k:v for k,v in s['metadata'].items() if k in ('name','labels','annotations')}; s['metadata']['namespace']='${NS}'; print(json.dumps(s))" \ + | kubectl apply -f - + +step "Deploy restore CNPG cluster" +kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml" +kubectl wait --for=condition=Ready cluster/forgejo-db-restore -n "${NS}" --timeout=10m + +step "Ensure local backup exists" +if [[ ! -f "${BACKUP_LOCAL}" ]]; then + kubectl exec -n forgejo "${PROD_POD}" -c gitea -- forgejo dump -f /tmp/forgejo-drill-backup.zip + mkdir -p "$(dirname "${BACKUP_LOCAL}")" + kubectl cp "forgejo/${PROD_POD}:/tmp/forgejo-drill-backup.zip" "${BACKUP_LOCAL}" -c gitea +fi +ls -lh "${BACKUP_LOCAL}" + +step "Apply restore PVC" +kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/restore-job.yaml" + +step "Run restore pod (stage backup, import files + SQL)" +kubectl delete pod forgejo-restore-import -n "${NS}" --ignore-not-found --wait=true +cat </dev/null +rm -rf /data/* +mkdir -p /data/git/gitea-repositories +unzip -q /backup/forgejo-drill-backup.zip -d /tmp/dump +cp -a /tmp/dump/repos/. /data/git/gitea-repositories/ +cp -a /tmp/dump/data/. /data/ +chown -R git:git /data +PGPASSWORD="${POSTGRES_PASSWORD}" psql -h forgejo-db-restore-rw.forgejo-restore-drill.svc.cluster.local -U forgejo -d forgejo -v ON_ERROR_STOP=1 -f /tmp/dump/forgejo-db.sql +echo restore-import-ok +' +unset DB_PASS +kubectl delete pod forgejo-restore-import -n "${NS}" --wait=true + +step "Deploy isolated Forgejo release" +cd "${HOME}/railiance-apps" +DB_PASS="$(kubectl get secret forgejo-db-credentials -n "${NS}" -o jsonpath='{.data.password}' | base64 -d)" +helm upgrade --install forgejo-restore gitea-charts/gitea --version 12.5.0 \ + --namespace "${NS}" --create-namespace \ + -f helm/forgejo-values.yaml \ + -f helm/forgejo-registry-values.yaml \ + --set strategy.type=Recreate \ + --set persistence.existingClaim=forgejo-restore-data \ + --set gitea.config.database.HOST=forgejo-db-restore-rw.${NS}.svc.cluster.local:5432 \ + --set gitea.config.database.PASSWD="${DB_PASS}" \ + --set gitea.config.server.DOMAIN=forgejo-restore.local \ + --set gitea.config.server.ROOT_URL=http://forgejo-restore.local:3000/ \ + --set gitea.admin.password=restore-drill-local-only \ + --set ingress.enabled=false \ + --wait --timeout=10m +unset DB_PASS + +step "Post-restore checks via port-forward" +kubectl port-forward -n "${NS}" svc/forgejo-restore-gitea-http 13000:3000 >/tmp/forgejo-restore-pf.log 2>&1 & +PF_PID=$! +sleep 5 +curl -fsS -o /dev/null -w 'health:%{http_code}\n' http://127.0.0.1:13000/ +curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/glas-harness | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))" +curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/key-cape | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))" +kill "${PF_PID}" 2>/dev/null || true +echo "restore-drill-complete" \ No newline at end of file diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index f44f550..4eaef57 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -2,13 +2,13 @@ id: RAIL-HO-WP-0005 type: workplan title: "Forgejo Production Migration on railiance01" -domain: railiance +domain: financials repo: railiance-infra status: active owner: railiance topic_slug: railiance created: "2026-05-03" -updated: "2026-06-04" +updated: "2026-07-04" state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018" --- @@ -24,6 +24,13 @@ Forgejo will become the heart of Railiance infrastructure. The work must be fully automated, backup-backed, recovery-drilled, and suitable for long-lived operation on railiance01 before any production cutover happens. +**Sequencing update (2026-07-04):** Production Forgejo is live on railiance01 +with Gitea still canonical per the safety contract. Repo cutover proceeds +**staged per-repo** using a migration ladder (disposable probes → non-production +pilots → image-capable pilots → production repos). `state-hub` is last. See +`CUST-WP-0054-T04` and +`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`. + ## Placement in the Railiance Tooling Set This workplan lives in `railiance-infra` because it is the cross-layer @@ -46,18 +53,23 @@ change is made there. ## Key Decisions to Confirm -1. Public/private hostname for Forgejo and whether Gitea remains reachable - during the transition. +1. ~~Public/private hostname for Forgejo~~ **DECIDED 2026-07-03:** + `forgejo.coulomb.social` → railiance01 (`92.205.62.239`). DNS active; + Traefik edge live; Forgejo workload deployed and serving HTTPS. Gitea remains + canonical until migration drills pass. Record: + `the-custodian/docs/forgejo-production-decisions.md`. 2. Mail delivery path for password reset and account recovery (SMTP relay, sender domain, SPF/DKIM/DMARC expectations). 3. Package registry scope: container images only at first, or also generic, npm, PyPI, Go, Maven, and Helm packages. -4. Actions runner model: in-cluster ephemeral runners, long-lived runner pod, - or isolated host runner. +4. ~~Actions runner model~~ **DECIDED 2026-07-03:** in-cluster long-lived runner + Deployment with DinD sidecar on railiance01 (`ADR-004`). Interim coulombcore + host runner retired after cutover. 5. Backup destination and retention target for database, repositories, attachments, LFS, Actions artifacts/logs, and package data. -6. Cutover mode: freeze-and-migrate all repos in one window, or staged - project-by-project transition. +6. Cutover mode: ~~freeze-all vs staged~~ **LEANING staged per-repo (2026-07-04)** + based on `glas-harness` pilot; operator confirmation still needed. Freeze-all + remains fallback for final production wave if drift risk is unacceptable. ## Safety Contract @@ -76,30 +88,37 @@ change is made there. repo. No plaintext SMTP passwords, admin tokens, runner tokens, or registry credentials in Git. -## Probe Strategy +## Probe and pilot strategy (revised 2026-07-04) -A `forgejo-railiance-probe` is reasonable and should be treated as a disposable -S5/S4 integration probe, not as the production install. +Original T03 planned a **disposable isolated-namespace probe** before any +production install. That path was **superseded**: production Forgejo deployed on +railiance01 under the safety contract (Gitea remains canonical; no Gitea deletes). -The probe should prove: +Integration evidence now comes from **in-production probes and repo pilots**: -- Helm values and cnpg database wiring converge cleanly. -- Initial admin bootstrap is automated and repeatable. -- SMTP/password reset works end-to-end. -- Package registry endpoints work for the package types Railiance needs first. -- Forgejo Actions can run a minimal workflow and publish a test package. -- Backup and restore works in an isolated namespace. -- Migration from a sample Gitea repo preserves git history, issues, releases, - wiki, LFS or attachments where applicable. +| Tier | Repo | Purpose | Status | +| --- | --- | --- | --- | +| 0 | `coulomb/forgejo-actions-probe` | Runner scheduling, DinD, OCI image-build | **done** | +| 1 | `coulomb/glas-harness` | Non-production git+SSH+CI routing drill | **done** | +| 2 | `coulomb/key-cape` | Image-build workflow + registry pull on railiance01 | **done** | +| 3 | Production set (`state-hub`, `issue-core`, …) | Canonical remotes, sweep paths, deploy loops | **gated** | -The probe is destroyed or explicitly archived after production Forgejo is live. +Each tier must pass before the next. T03 (isolated probe namespace) is cancelled; +acceptance criteria below are tracked across T05, T07, T08, and T10 instead. + +Still to prove before T11: + +- SMTP/password reset end-to-end (T06). +- Backup and restore in isolated namespace (T09) — **drill passed 2026-07-04**; + scheduled automation pending. +- Issues/releases/wiki/LFS per inventory classification (T10 matrix). +- Operator SSH identity on Forgejo beyond interim `forgejo_admin` keys (T02/T10). ## Target Architecture ``` operator / agents / developers - -> private HTTPS endpoint - -> railiance01 ingress + -> https://forgejo.coulomb.social (railiance01 Traefik ingress) -> forgejo Service in forgejo namespace -> Forgejo Deployment/StatefulSet -> forgejo-db CloudNative PG Cluster in databases namespace @@ -115,7 +134,7 @@ operator / agents / developers ```task id: RAIL-HO-WP-0005-T01 -status: in_progress +status: progress priority: high state_hub_task_id: "cf59d171-5629-45c9-9d44-8d6499827ffc" ``` @@ -138,13 +157,17 @@ Minimum inventory: Forgejo before cutover and classifies each migration item as automatic, manual, unsupported, or explicitly out of scope. +**Gap (2026-07-04):** first-pass inventory predates repos created after +2026-06-04 (e.g. `glas-harness`, `forgejo-actions-probe`). Refresh org repo +list and add a **migration tier** column (0–3) per repo before T11. + --- ### T02 — Resolve Forgejo production design decisions ```task id: RAIL-HO-WP-0005-T02 -status: todo +status: progress priority: high needs_human: true state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3" @@ -152,10 +175,16 @@ state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3" Decide the production choices listed in "Key Decisions to Confirm". +**Partial (2026-07-04):** hostname, exposure, deployment pattern, live deploy, +and in-cluster runner model decided (`ADR-004`). Cutover mode **leaning** staged +per-repo (glas-harness pilot). Remaining operator decisions: SMTP, package scope +beyond OCI, backup target, final cutover confirmation. See +`the-custodian/docs/forgejo-production-decisions.md`. + Expected output: - A short decision record in this workplan or a dedicated ADR. -- Hostname and exposure model. +- Hostname and exposure model. ✓ hostname; exposure follows railiance01 Traefik - SMTP provider and sender identity. - Package registry scope. - Actions runner isolation model. @@ -167,36 +196,21 @@ choices. --- -### T03 — Build forgejo-railiance-probe +### T03 — Build forgejo-railiance-probe (isolated namespace) ```task id: RAIL-HO-WP-0005-T03 -status: todo +status: cancel priority: high state_hub_task_id: "b516018a-415e-4a58-8c62-07c14ece9353" ``` -Create a disposable probe environment for Forgejo before touching production. - -Expected repo ownership: - -- `railiance-platform`: probe cnpg database and storage dependencies. -- `railiance-apps`: probe Forgejo Helm values and namespace. -- `railiance-enablement`: probe Actions runner template and workflows. - -Probe acceptance: - -- `make forgejo-probe-deploy` or equivalent converges from a clean cluster - state. -- Admin bootstrap is automated. -- A test user can reset a password via email. -- A test repository can be created, cloned, pushed, and protected. -- A test package can be published and pulled. -- A test Forgejo Actions workflow runs successfully. -- A probe backup restores into an isolated namespace. - -**Done when:** the probe demonstrates the whole lifecycle without manual -cluster surgery. +**Cancelled 2026-07-04:** superseded by production Forgejo on railiance01 (T05) +plus in-production integration probes (`forgejo-actions-probe`, `glas-harness`). +Isolated-namespace probe added latency without reducing risk given the safety +contract (Gitea canonical, no deletes). Remaining T03 acceptance items map to: +T05 (deploy), T06 (mail), T07 (packages), T08 (Actions), T09 (backup restore), +T10 (repo migration drill). --- @@ -220,6 +234,11 @@ Minimum scope: packages, Actions artifacts, and logs. - Restore runbook for database and blob/package data. +**Partial (2026-07-04):** `forgejo-db` CNPG cluster healthy on railiance01 +(`make forgejo-db-status` → Cluster in healthy state). SOPS secret path and +network policies in `railiance-platform`. Remaining: backup/WAL archiving to +approved target, blob/package storage restore drill (feeds T09). + **Done when:** platform dependencies can be deployed and restored without the Forgejo app running. @@ -229,7 +248,7 @@ Forgejo app running. ```task id: RAIL-HO-WP-0005-T05 -status: todo +status: progress priority: high state_hub_task_id: "11540ba4-d31c-4f64-836b-c6de69107aa4" ``` @@ -245,6 +264,12 @@ Minimum scope: - Health/status targets in the Makefile. - Migration-safe configuration for coexistence with Gitea during the cutover. +**Partial (2026-07-04):** `railiance-apps` deploy live — HTTPS smoke pass, +ingress + TLS, SSH NodePort `30022`, Actions enabled, `coulomb` org, +`railiance01-build-01` runner (ADR-004). Git push/pull via HTTPS and +`forgejo-remote` SSH proven. Remaining: SOPS hardening for all secrets, +SMTP (T06), operator user accounts beyond `forgejo_admin`. + **Done when:** Forgejo runs on railiance01 against production platform services and can serve login, git clone/push, package registry, and admin operations. @@ -301,8 +326,14 @@ Acceptance: - Retention and cleanup expectations are documented. - Package data is included in backup and restore drills. -**Done when:** `state-hub` or a probe image can be published to Forgejo and -pulled by railiance01. +**Partial (2026-07-04):** OCI registry live (`/v2/` auth challenge). Tier-0/2 +images built and pulled on railiance01: `forgejo-actions-probe`, `key-cape` +(`crictl pull forgejo.coulomb.social/coulomb/key-cape:latest` succeeded). +Remaining: `state-hub` image after tier-3 approval; document retention; include +packages in backup drill (T09). + +**Done when:** tier-2 gate is fully satisfied (✓) and tier-3 production images +follow the same pattern after explicit approval. --- @@ -310,7 +341,7 @@ pulled by railiance01. ```task id: RAIL-HO-WP-0005-T08 -status: todo +status: progress priority: high state_hub_task_id: "f45f98c9-2f02-4224-bbfd-c2e1ec38581e" ``` @@ -326,8 +357,20 @@ Minimum scope: - Secret handling policy for Actions. - Resource limits to avoid repeating previous single-node overload patterns. -**Done when:** a representative repository can run Forgejo Actions and publish -a test artifact without privileged cluster-wide credentials. +**Partial (2026-07-04):** in-cluster runner live (`railiance-apps/manifests/ +forgejo-runner.yaml`, ADR-004). Proven workflows: `forgejo-actions-probe` +(image-build), `glas-harness` (host+container CI smoke). Org secrets +`REGISTRY_USER`/`REGISTRY_TOKEN` set. Documented constraints: host runner is +non-root (static docker-cli, no `apk add`); `actions/checkout@v4` fails — use +`git clone` in job. Remaining: reusable workflow templates in +`railiance-enablement` (S4); resource limits review; no cluster-admin on runner. + +**Partial (2026-07-04):** tier-2 satisfied by `key-cape` (`container-build`, +archive checkout, static docker-cli). Remaining: publish reusable workflow +template in `railiance-enablement` (S4). + +**Done when:** tier-2 pilot repo runs Forgejo Actions end-to-end and publishes +a pullable image without privileged cluster-wide credentials. **Tier 2: done.** --- @@ -335,7 +378,7 @@ a test artifact without privileged cluster-wide credentials. ```task id: RAIL-HO-WP-0005-T09 -status: todo +status: progress priority: high state_hub_task_id: "25892007-36ca-4bd9-8adf-84d505465d7d" ``` @@ -360,34 +403,56 @@ Acceptance: - Restore into an isolated namespace is drilled and documented. - RPO/RTO expectations are recorded. +**Partial (2026-07-04):** isolated restore drill **passed**. Production +`forgejo dump` (~11.7 MiB) restored into `forgejo-restore-drill` namespace; +post-restore API checks: health 200, `coulomb/glas-harness` and +`coulomb/key-cape` on `main`, 3 org repos visible. Evidence: +`docs/forgejo-restore-drill-evidence.md`. Assets: `infra/forgejo-restore-drill/`, +`tools/forgejo-restore-drill.sh`. Remaining: scheduled CNPG/off-cluster backups, +encryption/approved target (T02/T04), automated dump schedule. + **Done when:** a fresh backup restores to a working isolated Forgejo instance -with repository, package, and user recovery checks passing. +with repository, package, and user recovery checks passing **and** scheduled +backups run without manual intervention. --- -### T10 — Drill Gitea to Forgejo migration +### T10 — Drill Gitea to Forgejo migration (staged ladder) ```task id: RAIL-HO-WP-0005-T10 -status: todo +status: progress priority: high state_hub_task_id: "6befde73-00bc-4643-be0b-a7ce7944e75f" ``` -Run a non-production migration drill from Gitea to Forgejo. +Run staged migration drills from Gitea to Forgejo before production repos move. -Minimum checks: +**Tier 1 complete (2026-07-04):** `glas-harness` — git history preserved, +`origin` on Forgejo, `gitea` legacy remote retained, SSH+HTTPS push, CI smoke +green. Result matrix: +`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`. + +Minimum checks (per tier): - Git history and default branches preserved. - Issues, labels, milestones, releases, wiki, and attachments handled per - inventory classification. -- SSH/HTTPS clone and push paths work. -- Existing local remotes can be transformed predictably. -- State Hub registered repo remotes can be updated safely. -- Rollback plan is rehearsed. + inventory classification (N/A for tier-1 git-only repos). +- SSH/HTTPS clone and push paths work (`forgejo-remote` in `~/.ssh/config`). +- Existing local remotes can be transformed predictably (`origin`/`gitea` split). +- State Hub registered repo remotes can be updated safely (deferred for tier-1). +- Rollback plan is rehearsed (Gitea copy unchanged). -**Done when:** a sample migration has a written result matrix and no unknown -critical migration gaps remain. +**Tier 2 complete (2026-07-04):** `key-cape` — multi-stage Dockerfile built and +pushed via archive-checkout workflow; `crictl pull` on railiance01 succeeded. +Evidence in `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` +(tier 2 section). + +**Not ready:** `state-hub` (tier 3) until hub-core build context template and +sweep `remote_url` playbook exist. + +**Done when:** tiers 0–2 pass with written result matrices and no unknown +critical migration gaps remain for production repos. --- @@ -401,19 +466,21 @@ needs_human: true state_hub_task_id: "b1b66687-ca33-4971-b312-743c8e059c5e" ``` -Execute the production migration only after the probe, backup restore, package -registry, email recovery, and Actions gates pass. +Execute production migration only after T06, T07, T08, T09, and T10 tier 0–2 +gates pass. `state-hub` and other Wave-1 production repos require explicit +operator approval per `CUST-WP-0054` drain sequence. -Cutover sequence: +**Preferred cutover (staged per-repo):** -1. Announce freeze window. -2. Take final Gitea backup and verify it exists. -3. Freeze Gitea writes. -4. Migrate repositories and metadata to Forgejo. -5. Validate critical repositories and package pulls. -6. Update State Hub repo remotes and host paths as needed. -7. Update local and railiance01 remotes. -8. Keep Gitea read-only as rollback until the stabilization window passes. +1. Per repo: Gitea backup snapshot (or org-wide before each wave). +2. Mirror git to Forgejo; switch workstation `origin` to `forgejo-remote`. +3. Port/verify Actions workflows on Forgejo runner. +4. Update State Hub `remote_url` and railiance01 sweep checkouts when promoted. +5. Mark Gitea repo read-only (org policy); do not delete. +6. Repeat until production set complete. + +**Freeze-all fallback:** single window if staged drift is unacceptable — same +steps but all repos in one maintenance period. **Done when:** all Railiance/Custodian repos use Forgejo as primary, Gitea is read-only fallback, and rollback instructions are documented. @@ -447,19 +514,30 @@ legacy Gitea either archived or intentionally retained as documented fallback. ## Phasing and Dependencies ``` -T01 inventory ─┬─► T02 decisions ─┬─► T03 probe ─┬─► T04 platform - │ │ ├─► T05 app - │ │ ├─► T06 mail recovery - │ │ ├─► T07 packages - │ │ ├─► T08 actions - │ │ └─► T09 backups - └────────────────────────────────────► T10 migration drill +T01 inventory ──► T02 decisions ──┬──► T04 platform (forgejo-db ✓ partial) + ├──► T05 app (live ✓ partial) + ├──► T06 mail recovery + ├──► T07 packages (OCI probe ✓ partial) + ├──► T08 actions (runner ✓ partial) + └──► T09 backups -T03-T10 all pass ─► T11 production cutover ─► T12 legacy Gitea retirement +T05+T08 ──► T10 migration ladder ──► T11 production cutover ──► T12 Gitea retire + tier0 probe ✓ + tier1 glas-harness ✓ + tier2 key-cape ✓ + tier3 production (gated) + +T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots) ``` -Recommended first slice: T01, T02, T03. Do not start T11 until T06, T07, T08, -T09, and T10 are complete. +**Current focus (2026-07-04):** T10 tiers 0–2 **complete**; T09 restore drill +**passed** (scheduled backups + backup target still open); T02 decisions (SMTP, +backup target) before tier-3 production repos. +Do not start T11 `state-hub` until T09 complete and `CUST-WP-0054` Wave-1 +gates satisfied. + +**Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation +build retirement; staged repo promotion before State Hub primary move (T05). ## railiance-bootstrap Note @@ -479,7 +557,14 @@ purpose is identified. - `RAIL-HO-WP-0004-production-readiness.md` - `RAIL-HO-WP-0003-5repo-stack-restructure.md` +- `CUST-WP-0054-workstation-independence-and-fleet-realignment.md` (T04 forge+CI) - `CUST-WP-0014-repo-sync-automation.md` - `CUST-WP-0021-multi-host-repo-paths.md` +- `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md` +- `docs/forgejo-migration-inventory.md` +- `the-custodian/docs/forgejo-production-decisions.md` +- `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` +- `railiance-apps/docs/forgejo-on-railiance01.md` +- `railiance-forge/docs/forgejo-actions-runner-substrate.md` - `ops/incidents/2026-03-25-gitea-pgpool-crashloop.md` - `ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md` diff --git a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md b/workplans/archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md similarity index 98% rename from workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md rename to workplans/archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md index 9216e42..c2309f8 100644 --- a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md +++ b/workplans/archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md @@ -2,9 +2,9 @@ id: RAIL-HO-WP-0001 type: workplan title: "Secure Single-Server Bootstrap at HostEurope" -domain: railiance +domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance repo_goal_id: 9e835b82-acca-493a-943f-2553ffe0bf54 @@ -18,8 +18,7 @@ handoff_note: > done in railiance-bootstrap (harden.yml, bootstrap.yml) is superseded by the roles/base and roles/sops_agent structure in this repo. The HostEurope server IP is 92.205.62.239 (hosts.ini is gitignored — recreate from - inventory/servers.yaml when working on the host). ---- + inventory/servers.yaml when working on the host).--- # Secure Single-Server Bootstrap at HostEurope @@ -56,7 +55,6 @@ ssh -R 8000:127.0.0.1:8000 @92.205.62.239 Then Claude on the remote host can reach `http://127.0.0.1:8000` as normal. See also: the tunnel setup note in `CLAUDE.md`. - --- ## Tasks diff --git a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md b/workplans/archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md similarity index 99% rename from workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md rename to workplans/archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md index 1fd1eb2..29ef533 100644 --- a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md +++ b/workplans/archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md @@ -2,9 +2,9 @@ id: RAIL-HO-WP-0002 type: workplan title: "Server Specification and Automated Test Suite" -domain: railiance +domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance state_hub_workstream_id: "8fed53c2-4c39-4471-8bb9-61f58771fe0c" diff --git a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md b/workplans/archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md similarity index 99% rename from workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md rename to workplans/archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md index 19ba457..748fed5 100644 --- a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md +++ b/workplans/archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md @@ -2,9 +2,9 @@ id: RAIL-HO-WP-0003 type: workplan title: "Railiance 5-Repo Stack Restructure" -domain: railiance +domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance state_hub_workstream_id: "3ae0afc5-13f2-4e6c-aea7-1c1fb9f1ab81" diff --git a/workplans/RAIL-HO-WP-0004-production-readiness.md b/workplans/archived/260702-RAIL-HO-WP-0004-production-readiness.md similarity index 99% rename from workplans/RAIL-HO-WP-0004-production-readiness.md rename to workplans/archived/260702-RAIL-HO-WP-0004-production-readiness.md index 3b5a005..3a1a931 100644 --- a/workplans/RAIL-HO-WP-0004-production-readiness.md +++ b/workplans/archived/260702-RAIL-HO-WP-0004-production-readiness.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0004 type: workplan title: "Railiance Production Readiness — Automated, Reproducible Stack" -domain: railiance +domain: financials repo: railiance-infra status: finished owner: worsch @@ -437,7 +437,7 @@ context. ```task id: RAIL-HO-WP-0004-T09 -status: cancelled +status: cancel priority: medium state_hub_task_id: "d2afe78a-eb51-4ce9-b332-f181323d2370" needs_human: false