From 723f23d05dc946b04181995b8682cba97db137a3 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 22 Jun 2026 23:16:28 +0200 Subject: [PATCH 01/10] Normalize agent instructions and workplan frontmatter (STATE-WP-0067) - Align agent files with on-disk workplan prefixes (infer from workplan ids) - Set workplan domain to registered domain_slug; add topic_slug where applicable - Repair frontmatter delimiter formatting; migrate legacy task status literals - Regenerate AGENTS.md, CLAUDE.md, and .claude/rules from State Hub templates --- .claude/rules/first-session.md | 10 ++--- .claude/rules/repo-boundary.md | 2 +- .claude/rules/repo-identity.md | 6 +-- .claude/rules/session-protocol.md | 19 +++++----- .claude/rules/workplan-convention.md | 20 ++++++++-- AGENTS.md | 37 +++++++++++-------- CLAUDE.md | 2 +- .../RAIL-HO-WP-0001-hosteurope-bootstrap.md | 6 +-- ...L-HO-WP-0002-server-spec-and-test-suite.md | 2 +- ...RAIL-HO-WP-0003-5repo-stack-restructure.md | 2 +- .../RAIL-HO-WP-0004-production-readiness.md | 4 +- ...HO-WP-0005-forgejo-production-migration.md | 4 +- 12 files changed, 65 insertions(+), 49 deletions(-) diff --git a/.claude/rules/first-session.md b/.claude/rules/first-session.md index 0d0b183..9a4f826 100644 --- a/.claude/rules/first-session.md +++ b/.claude/rules/first-session.md @@ -1,11 +1,11 @@ ## First Session Protocol -Triggered when `get_domain_summary("railiance")` shows **no workstreams**. +Triggered when `get_domain_summary("financials")` shows **no workstreams**. The project is registered but work has not yet been structured. **Step 1 — Read, don't write** -- `~/the-custodian/canon/projects/railiance/project_charter_v0.1.md` — purpose, scope -- `~/the-custodian/canon/projects/railiance/roadmap_v0.1.md` — planned phases +- `~/the-custodian/canon/projects/financials/project_charter_v0.1.md` — purpose, scope +- `~/the-custodian/canon/projects/financials/roadmap_v0.1.md` — planned phases - Scan repo root: README, directory structure, existing code or docs **Step 2 — Survey in-progress work** @@ -17,7 +17,7 @@ roadmap phase. **Wait for approval before creating.** **Step 4 — Create workplan file first, then DB record (ADR-001)** ``` -workplans/railiance-hosts-WP-NNNN-.md ← write this first +workplans/RAIL-HO-WP-NNNN-.md ← write this first ``` Then register in the hub: ``` @@ -28,7 +28,7 @@ create_task(workstream_id="", title="...", priority="high|medium|low") **Step 5 — Record the setup** ``` add_progress_event( - summary="First session: structured railiance into N workstreams, M tasks", + summary="First session: structured financials into N workstreams, M tasks", event_type="milestone", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", detail={"workstreams": [...], "tasks_created": M} diff --git a/.claude/rules/repo-boundary.md b/.claude/rules/repo-boundary.md index 335c457..8b48c17 100644 --- a/.claude/rules/repo-boundary.md +++ b/.claude/rules/repo-boundary.md @@ -1,6 +1,6 @@ ## Repo boundary -This repo owns **railiance-hosts** only. It does not own: +This repo owns **railiance-infra** only. It does not own: diff --git a/AGENTS.md b/AGENTS.md index e09873b..3d3f41a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,13 +1,13 @@ -# railiance-hosts — Agent Instructions +# railiance-infra — Agent Instructions ## Repo Identity -**Purpose:** Host inventory and node-level configuration for railiance infrastructure. Tracks server specs, network topology, and per-host service assignments. +**Purpose:** OAS S1 Infrastructure Substrate — OS provisioning, Ansible roles, Goss spec, Terraform, inventory -**Domain:** railiance -**Repo slug:** railiance-hosts +**Domain:** financials +**Repo slug:** railiance-infra **Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38` -**Workplan prefix:** `RAILIANCE-WP-` +**Workplan prefix:** `RAIL-HO-WP-` --- @@ -32,7 +32,7 @@ curl -s "http://127.0.0.1:8000/workstreams/?topic_id=ca369340-a64e-442e-98f1-a4f | python3 -m json.tool # Check inbox -curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-hosts&unread_only=true" \ +curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-infra&unread_only=true" \ | python3 -m json.tool ``` @@ -63,8 +63,8 @@ Omit `workstream_id` / `task_id` when not applicable. ```bash curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ -H "Content-Type: application/json" \ - -d '{"status": "in_progress"}' -# values: todo | in_progress | done | blocked + -d '{"status": "progress"}' +# values: wait | todo | progress | done | cancel ``` ### Flag a task for human review @@ -81,9 +81,9 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ **Start:** 1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe) -2. Check inbox: `GET /messages/?to_agent=railiance-hosts&unread_only=true`; mark read +2. Check inbox: `GET /messages/?to_agent=railiance-infra&unread_only=true`; mark read 3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks -4. Check blocked tasks: `GET /tasks/?needs_human=true` +4. Check human-needed tasks: `GET /tasks/?needs_human=true` **During work:** - Update task statuses in workplan files as tasks progress @@ -95,7 +95,7 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ 3. Note for the custodian operator: after workplan file changes, run from `~/state-hub`: ```bash - make fix-consistency REPO=railiance-hosts + make fix-consistency REPO=railiance-infra ``` This syncs task status from files into the hub DB. @@ -151,6 +151,11 @@ every repo's agent instructions because it is high-frequency, high-risk, and eas get wrong. **Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml` + + + + --- ## Workplan Convention (ADR-001) @@ -176,8 +181,8 @@ anything needing analysis, design, approval, dependencies, or multiple phases. id: RAILIANCE-WP-NNNN type: workplan title: "..." -domain: railiance -repo: railiance-hosts +domain: financials +repo: railiance-infra status: proposed | ready | active | blocked | backlog | finished | archived owner: codex topic_slug: ... @@ -198,7 +203,7 @@ derived health labels, not frontmatter statuses. ` ` `task id: RAILIANCE-WP-NNNN-T01 -status: todo | in_progress | done | blocked +status: wait | todo | progress | done | cancel priority: high | medium | low state_hub_task_id: "" # written by fix-consistency — do not edit ` ` ` @@ -206,9 +211,9 @@ state_hub_task_id: "" # written by fix-consistency — do not edit Task description text. ``` -Status progression: `todo` → `in_progress` → `done` (or `blocked`) +Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work. To create a new workplan: 1. Write the file following the format above -2. Notify the custodian operator to run `make fix-consistency REPO=railiance-hosts` +2. Notify the custodian operator to run `make fix-consistency REPO=railiance-infra` (or send a message to the hub agent via `POST /messages/`) diff --git a/CLAUDE.md b/CLAUDE.md index ef28d7b..871d83d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,4 +1,4 @@ -# railiance-hosts — Claude Code Instructions +# railiance-infra — Claude Code Instructions @SCOPE.md @.claude/rules/repo-identity.md diff --git a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md b/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md index 9216e42..6463257 100644 --- a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md +++ b/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0001 type: workplan title: "Secure Single-Server Bootstrap at HostEurope" -domain: railiance +domain: financials repo: railiance-infra status: completed owner: railiance @@ -18,8 +18,7 @@ handoff_note: > done in railiance-bootstrap (harden.yml, bootstrap.yml) is superseded by the roles/base and roles/sops_agent structure in this repo. The HostEurope server IP is 92.205.62.239 (hosts.ini is gitignored — recreate from - inventory/servers.yaml when working on the host). ---- + inventory/servers.yaml when working on the host).--- # Secure Single-Server Bootstrap at HostEurope @@ -56,7 +55,6 @@ ssh -R 8000:127.0.0.1:8000 @92.205.62.239 Then Claude on the remote host can reach `http://127.0.0.1:8000` as normal. See also: the tunnel setup note in `CLAUDE.md`. - --- ## Tasks diff --git a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md b/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md index 1fd1eb2..41a6761 100644 --- a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md +++ b/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0002 type: workplan title: "Server Specification and Automated Test Suite" -domain: railiance +domain: financials repo: railiance-infra status: completed owner: railiance diff --git a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md b/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md index 19ba457..7a9d7bd 100644 --- a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md +++ b/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0003 type: workplan title: "Railiance 5-Repo Stack Restructure" -domain: railiance +domain: financials repo: railiance-infra status: completed owner: railiance diff --git a/workplans/RAIL-HO-WP-0004-production-readiness.md b/workplans/RAIL-HO-WP-0004-production-readiness.md index 3b5a005..3a1a931 100644 --- a/workplans/RAIL-HO-WP-0004-production-readiness.md +++ b/workplans/RAIL-HO-WP-0004-production-readiness.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0004 type: workplan title: "Railiance Production Readiness — Automated, Reproducible Stack" -domain: railiance +domain: financials repo: railiance-infra status: finished owner: worsch @@ -437,7 +437,7 @@ context. ```task id: RAIL-HO-WP-0004-T09 -status: cancelled +status: cancel priority: medium state_hub_task_id: "d2afe78a-eb51-4ce9-b332-f181323d2370" needs_human: false diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index f44f550..4614e83 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -2,7 +2,7 @@ id: RAIL-HO-WP-0005 type: workplan title: "Forgejo Production Migration on railiance01" -domain: railiance +domain: financials repo: railiance-infra status: active owner: railiance @@ -115,7 +115,7 @@ operator / agents / developers ```task id: RAIL-HO-WP-0005-T01 -status: in_progress +status: progress priority: high state_hub_task_id: "cf59d171-5629-45c9-9d44-8d6499827ffc" ``` From 73a30f0a44610fb64f065d7c45afd05bddf10e18 Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 2 Jul 2026 00:21:48 +0200 Subject: [PATCH 02/10] Repo hygiene: fill stack-and-commands, normalize workplan statuses - Fill .claude/rules/stack-and-commands.md (was an empty TODO template) - Normalize workplan frontmatter statuses to canonical vocabulary (completed/done -> finished) per ADR-001 Co-Authored-By: Claude Fable 5 --- .claude/rules/stack-and-commands.md | 27 ++++++++++--------- .../RAIL-HO-WP-0001-hosteurope-bootstrap.md | 2 +- ...L-HO-WP-0002-server-spec-and-test-suite.md | 2 +- ...RAIL-HO-WP-0003-5repo-stack-restructure.md | 2 +- 4 files changed, 18 insertions(+), 15 deletions(-) diff --git a/.claude/rules/stack-and-commands.md b/.claude/rules/stack-and-commands.md index dc53ac6..dc279b4 100644 --- a/.claude/rules/stack-and-commands.md +++ b/.claude/rules/stack-and-commands.md @@ -1,19 +1,22 @@ ## Stack - -- **Language:** -- **Key deps:** +- **Language:** Terraform (HCL), cloud-init, Ansible, Bash +- **Key deps:** Hetzner/HostEurope provisioning, SOPS + age for secrets, repo-local git hooks (`.githooks`) ## Dev Commands ```bash -# TODO: Fill in the standard commands for this repo - -# Install dependencies - -# Run tests - -# Lint / type check - -# Build / package (if applicable) +make help # list all targets +make hooks # configure repo-local git hooks (secrets guard) +make check-secrets # fail if anything in secrets/ is not SOPS-encrypted +make sops-edit # edit global secrets with SOPS +make sops-encrypt FILE=secrets/foo.yaml +make tf-fmt # terraform fmt +make tf-init && make tf-plan # plan (needs decrypted HCLOUD_TOKEN) +make tf-apply # provision — production-touching +make tf-destroy # tear down — destructive, operator only ``` + +`tf-apply`/`tf-destroy` change production infrastructure — operator approval +required. No unit test suite; validation is `tf-plan` + the server test suite +workplans. diff --git a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md b/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md index 6463257..c2309f8 100644 --- a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md +++ b/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md @@ -4,7 +4,7 @@ type: workplan title: "Secure Single-Server Bootstrap at HostEurope" domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance repo_goal_id: 9e835b82-acca-493a-943f-2553ffe0bf54 diff --git a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md b/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md index 41a6761..29ef533 100644 --- a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md +++ b/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md @@ -4,7 +4,7 @@ type: workplan title: "Server Specification and Automated Test Suite" domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance state_hub_workstream_id: "8fed53c2-4c39-4471-8bb9-61f58771fe0c" diff --git a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md b/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md index 7a9d7bd..748fed5 100644 --- a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md +++ b/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md @@ -4,7 +4,7 @@ type: workplan title: "Railiance 5-Repo Stack Restructure" domain: financials repo: railiance-infra -status: completed +status: finished owner: railiance topic_slug: railiance state_hub_workstream_id: "3ae0afc5-13f2-4e6c-aea7-1c1fb9f1ab81" From 3f2564748c421db03c304b4373cc63c5beaec47a Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 2 Jul 2026 00:25:41 +0200 Subject: [PATCH 03/10] Archive closed workplans to workplans/archived/ (ADR-001) Co-Authored-By: Claude Fable 5 --- .../260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md} | 0 .../260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md} | 0 .../260702-RAIL-HO-WP-0003-5repo-stack-restructure.md} | 0 .../260702-RAIL-HO-WP-0004-production-readiness.md} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename workplans/{RAIL-HO-WP-0001-hosteurope-bootstrap.md => archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md} (100%) rename workplans/{RAIL-HO-WP-0002-server-spec-and-test-suite.md => archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md} (100%) rename workplans/{RAIL-HO-WP-0003-5repo-stack-restructure.md => archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md} (100%) rename workplans/{RAIL-HO-WP-0004-production-readiness.md => archived/260702-RAIL-HO-WP-0004-production-readiness.md} (100%) diff --git a/workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md b/workplans/archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md similarity index 100% rename from workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md rename to workplans/archived/260702-RAIL-HO-WP-0001-hosteurope-bootstrap.md diff --git a/workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md b/workplans/archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md similarity index 100% rename from workplans/RAIL-HO-WP-0002-server-spec-and-test-suite.md rename to workplans/archived/260702-RAIL-HO-WP-0002-server-spec-and-test-suite.md diff --git a/workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md b/workplans/archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md similarity index 100% rename from workplans/RAIL-HO-WP-0003-5repo-stack-restructure.md rename to workplans/archived/260702-RAIL-HO-WP-0003-5repo-stack-restructure.md diff --git a/workplans/RAIL-HO-WP-0004-production-readiness.md b/workplans/archived/260702-RAIL-HO-WP-0004-production-readiness.md similarity index 100% rename from workplans/RAIL-HO-WP-0004-production-readiness.md rename to workplans/archived/260702-RAIL-HO-WP-0004-production-readiness.md From b32c56db4feeca55947117085b878b0717e68a00 Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 2 Jul 2026 01:47:45 +0200 Subject: [PATCH 04/10] Regenerate agent instructions: workstream -> workplan terminology Registration guidance now prescribes file-first + fix-consistency (C-06) instead of manual create_workplan/create_workstream calls; progress-event examples use workplan_id; legacy field names annotated. Co-Authored-By: Claude Fable 5 --- .claude/rules/credential-routing.md | 2 +- .claude/rules/first-session.md | 26 +++++++++++++---------- .claude/rules/session-protocol.md | 31 ++++++++++++++++++---------- .claude/rules/workplan-convention.md | 13 ++++++++---- AGENTS.md | 26 ++++++++++++++--------- 5 files changed, 61 insertions(+), 37 deletions(-) diff --git a/.claude/rules/credential-routing.md b/.claude/rules/credential-routing.md index fe1b70b..c0fa603 100644 --- a/.claude/rules/credential-routing.md +++ b/.claude/rules/credential-routing.md @@ -20,7 +20,7 @@ Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run wa | Agent runtime | How to orient | | --- | --- | | **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending | -| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership | +| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership | | **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` | ### Quick routing table diff --git a/.claude/rules/first-session.md b/.claude/rules/first-session.md index 9a4f826..412c4c4 100644 --- a/.claude/rules/first-session.md +++ b/.claude/rules/first-session.md @@ -1,6 +1,6 @@ ## First Session Protocol -Triggered when `get_domain_summary("financials")` shows **no workstreams**. +Triggered when `get_domain_summary("financials")` shows **no workplans**. The project is registered but work has not yet been structured. **Step 1 — Read, don't write** @@ -11,27 +11,31 @@ The project is registered but work has not yet been structured. **Step 2 — Survey in-progress work** Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete. -**Step 3 — Propose workstreams to Bernd** -Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a +**Step 3 — Propose workplans to Bernd** +Propose 1–3 workplans — each a coherent strand, weeks to months, anchored to a roadmap phase. **Wait for approval before creating.** -**Step 4 — Create workplan file first, then DB record (ADR-001)** +**Step 4 — Write the workplan file; fix-consistency registers it (ADR-001)** ``` -workplans/RAIL-HO-WP-NNNN-.md ← write this first +workplans/RAIL-HO-WP-NNNN-.md ← write this, commit it ``` -Then register in the hub: -``` -create_workstream(topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", title="...", owner="...", description="...") -create_task(workstream_id="", title="...", priority="high|medium|low") +Then register by running the consistency check — do **not** call +`create_workplan`/`create_task` (or legacy `create_workstream`) yourself; +manual registration duplicates what C-06 creates from the file: +```bash +statehub fix-consistency --repo railiance-infra ``` +C-06 creates the hub workplan + tasks and writes `state_hub_workstream_id` / +`state_hub_task_id` back into the file (legacy field names, kept for +compatibility — they hold workplan/task IDs). **Step 5 — Record the setup** ``` add_progress_event( - summary="First session: structured financials into N workstreams, M tasks", + summary="First session: structured financials into N workplans, M tasks", event_type="milestone", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", - detail={"workstreams": [...], "tasks_created": M} + detail={"workplans": [...], "tasks_created": M} ) ``` diff --git a/.claude/rules/session-protocol.md b/.claude/rules/session-protocol.md index 6d31c8e..09d197f 100644 --- a/.claude/rules/session-protocol.md +++ b/.claude/rules/session-protocol.md @@ -44,7 +44,7 @@ For each file with `status: ready`, `active`, or `blocked`, note pending **Step 4 — Present brief** -1. **Active workstreams** for `financials` — title, task counts, blocking decisions +1. **Active workplans** for `financials` — title, task counts, blocking decisions 2. **Pending tasks** from `workplans/` + any `[repo:railiance-infra]` hub tasks 3. **Goal guidance** — if `goal_guidance` in summary: - `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"* @@ -52,33 +52,42 @@ For each file with `status: ready`, `active`, or `blocked`, note pending 4. **Suggested next action** — highest-priority open item 5. **SBOM status** — flag if `last_sbom_at` is unset for this repo -If no workstreams: follow First Session Protocol (`first-session.md`). +If no workplans: follow First Session Protocol (`first-session.md`). **During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()` -> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`) -> are First Session Protocol only. Work structure belongs in repo files (ADR-001). +> State Hub is a *read model*. **Never register workplans or tasks by hand** +> (`create_workplan`, `create_task`, or the legacy `create_workstream`) — write +> the workplan file in `workplans/` and run `fix-consistency`; its C-06 check +> registers the workplan and its tasks in the hub and writes the IDs back into +> the file. Manual registration creates duplicates the moment fix-consistency +> runs. Work structure belongs in repo files (ADR-001). +> +> Terminology: "workstream" is the legacy name for workplan. Some API/frontmatter +> field names keep it for compatibility (`state_hub_workstream_id`, +> `workstream_id` params) — treat them as workplan IDs. **Session close:** With MCP tools: ``` -add_progress_event(summary="...", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", workstream_id="") +add_progress_event(summary="...", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", workplan_id="") ``` Without MCP tools: ```bash curl -s -X POST http://127.0.0.1:8000/progress/ \ -H "Content-Type: application/json" \ - -d '{"topic_id":"ca369340-a64e-442e-98f1-a4fa7dc74a38","workstream_id":"","event_type":"note","summary":"what changed","author":"codex"}' + -d '{"topic_id":"ca369340-a64e-442e-98f1-a4fa7dc74a38","workplan_id":"","event_type":"note","summary":"what changed","author":"codex"}' ``` -If workplan files were modified, ensure the local copy is up to date first: +If workplan files were modified, ensure the local copy is up to date first, +then sync from the repo checkout: ```bash -git -C pull --ff-only -cd ~/state-hub && make fix-consistency REPO=railiance-infra +git pull --ff-only +statehub fix-consistency ``` For repos where implementation runs on a remote machine (e.g. CoulombCore), -use the combined target which pulls before fixing: +use the pull-before-fix mode from any shell with the State Hub CLI: ```bash -cd ~/state-hub && make fix-consistency-remote REPO=railiance-infra +statehub fix-consistency --repo railiance-infra --remote ``` **C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback will sync the file to match DB. **C-16** (repo behind remote) blocks all writes diff --git a/.claude/rules/workplan-convention.md b/.claude/rules/workplan-convention.md index 7f9e5cc..cf5e620 100644 --- a/.claude/rules/workplan-convention.md +++ b/.claude/rules/workplan-convention.md @@ -5,7 +5,7 @@ ID prefix: `RAIL-HO-WP-` Work items originate as files in this repo **before** being registered in the hub. -Canonical workplan/workstream frontmatter statuses are: +Canonical workplan frontmatter statuses are: `proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`. Use `proposed` for a newly drafted plan, `ready` after review against current repo state, and `finished` when implementation is complete. `stalled` and @@ -16,14 +16,15 @@ prefix: `YYMMDD-RAIL-HO-WP-NNNN-.md`. The frontmatter id remains unchanged; the prefix is only for quick visual reference. Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**: -`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids +`workplans/ADHOC-YYYY-MM-DD.md`, workplan slug `adhoc-YYYY-MM-DD`, and task ids `ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed directly. Promote anything requiring analysis, design, approval, dependencies, or multiple planned phases into a normal workplan. Ecosystem todos from other agents arrive as `[repo:railiance-infra]` hub tasks — -visible at session start. Pick one up by creating the workplan file, then registering -the workstream. +visible at session start. Pick one up by creating the workplan file, committing, +and running `statehub fix-consistency` — C-06 registers the workplan in the hub. +Never register by hand with `create_workplan`/`create_workstream`. Task blocks use this shape: @@ -37,4 +38,8 @@ state_hub_task_id: "" # written by fix-consistency — do not edit Status progression is `todo` → `progress` → `done`; use `wait` for waiting or blocked work and `cancel` for stopped work. +Workplan frontmatter carries `state_hub_workstream_id` — a legacy field name +kept for compatibility ("workstream" is the old term for workplan); it holds +the hub workplan id and is written by fix-consistency. Do not edit or rename it. + diff --git a/AGENTS.md b/AGENTS.md index 3d3f41a..ed01c8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,12 @@ there is no MCP server for Codex agents. |---------|-----| | Local workstation | `http://127.0.0.1:8000` | | Remote via tunnel | `http://127.0.0.1:18000` | +| Optional local edge relay | http://127.0.0.1:18080 | + +When an operator has enabled the edge relay, set API_BASE to the relay URL. +Queueable writes return an explicit queued receipt if the central hub is +unreachable. Treat that as pending local evidence, then ask the operator to run +statehub outbox status/replay after connectivity returns. ### Orient at session start @@ -27,8 +33,8 @@ there is no MCP server for Codex agents. # Offline brief — works without hub connection cat .custodian-brief.md -# Active workstreams for this domain -curl -s "http://127.0.0.1:8000/workstreams/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \ +# Active workplans for this domain +curl -s "http://127.0.0.1:8000/workplans/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \ | python3 -m json.tool # Check inbox @@ -51,12 +57,12 @@ curl -s -X POST http://127.0.0.1:8000/progress/ \ "summary": "what was done", "event_type": "note", "author": "codex", - "workstream_id": "", + "workplan_id": "", "task_id": "" }' ``` -Omit `workstream_id` / `task_id` when not applicable. +Omit `workplan_id` / `task_id` when not applicable. ### Update task status @@ -80,7 +86,7 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ ## Session Protocol **Start:** -1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe) +1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe) 2. Check inbox: `GET /messages/?to_agent=railiance-infra&unread_only=true`; mark read 3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks 4. Check human-needed tasks: `GET /tasks/?needs_human=true` @@ -92,12 +98,12 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ **Close:** 1. Update workplan file task statuses to reflect progress 2. Log: `POST /progress/` with a summary of what changed -3. Note for the custodian operator: after workplan file changes, run from - `~/state-hub`: +3. After workplan file changes, run: ```bash - make fix-consistency REPO=railiance-infra + statehub fix-consistency ``` - This syncs task status from files into the hub DB. + Coding agents should run this directly; ask the operator only if the CLI or + State Hub API is unavailable. This syncs task status from files into the hub DB. --- @@ -123,7 +129,7 @@ Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run wa | Agent runtime | How to orient | | --- | --- | | **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending | -| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership | +| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership | | **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` | ### Quick routing table From 6b0ededee2d29b64db39ad3dd470f3df8ff2dfa2 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 3 Jul 2026 22:29:28 +0200 Subject: [PATCH 05/10] ADR-004: Forgejo in-cluster Actions runner on railiance01 Decides long-lived runner Deployment with DinD sidecar; updates RAIL-HO-WP-0005 runner model decision. --- ...R-004-forgejo-in-cluster-actions-runner.md | 104 ++++++++++++++++++ ...HO-WP-0005-forgejo-production-migration.md | 29 +++-- 2 files changed, 124 insertions(+), 9 deletions(-) create mode 100644 docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md diff --git a/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md b/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md new file mode 100644 index 0000000..1f56d2a --- /dev/null +++ b/docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md @@ -0,0 +1,104 @@ +# ADR-004 — Forgejo In-Cluster Actions Runner on railiance01 + +**Status:** Accepted +**Date:** 2026-07-03 +**Deciders:** Bernd Worsch (operator), custodian agents +**Workplans:** `RAIL-HO-WP-0005-T02`, `CUST-WP-0054-T04` + +--- + +## Context + +Forgejo production runs on **railiance01 k3s** (`railiance-apps`, S5). An interim +**host runner** on coulombcore proved Actions scheduling (`coulomb/forgejo-actions-probe`) +but: + +- coulombcore is a legacy machine slated for drain (CUST-WP-0054-T03). +- Host runners require Docker or Podman on the OS — not installed, not desired on + coulombcore long term. +- Forgejo upstream recommends **not** co-locating runners on the same machine as the + forge instance; in-cluster **separate pods** satisfy isolation while staying on the + production fleet node. +- `RAIL-HO-WP-0005-T02` left the runner model undecided among host, in-cluster, and + ephemeral options. + +Goal: a **coherent Kubernetes-from-the-start** CI substrate — Forgejo app, database, +ingress, and Actions runner all lifecycle-managed on railiance01. + +## Decision + +### Runner placement + +Deploy **one long-lived Forgejo Actions runner Deployment** in the `forgejo` namespace +on railiance01: + +| Component | Implementation | +| --- | --- | +| Runner | `data.forgejo.org/forgejo/runner:6.3.1` | +| Container runtime for jobs | `docker:dind` sidecar (privileged) | +| State | PVC `forgejo-runner-data` (`.runner`, `config.yaml`, action cache) | +| Registration scope | `coulomb` organization | +| Runner name | `railiance01-build-01` | +| Deploy surface | `railiance-apps/manifests/forgejo-runner.yaml` | +| Operator targets | `make forgejo-runner-deploy`, `forgejo-runner-status` | + +### Label contract + +Preserve Gitea migration compatibility and semantic capability labels: + +```text +self-hosted:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host,railiance01:host,ubuntu-latest:docker://node:20-bookworm,docker:docker://node:20-bookworm +``` + +### Security boundaries + +- Runner pod receives **no** cluster-admin kubeconfig and **no** OpenBao tokens by default. +- `registry-publish` jobs use **repo/org-scoped Forgejo secrets** only. +- DinD sidecar runs **privileged** — accepted for single-node railiance01 with + dedicated `forgejo` namespace; revisit when a third node or multi-tenant runners appear. +- Registration tokens live in Kubernetes Secret `forgejo-runner-registration` (SOPS + template committed; live value never in Git). + +### Retire interim host runner + +Stop and disable `forgejo-runner.service` on coulombcore after in-cluster runner is +healthy. Do not register new host runners without an explicit ADR amendment. + +## Alternatives considered + +| Option | Outcome | +| --- | --- | +| Host runner + Docker on coulombcore | Rejected — legacy host, contradicts drain plan | +| Host runner + Podman on haskelseed | Viable fallback; not chosen as primary | +| Kaniko/Buildah without DinD | Deferred — higher workflow churn during Gitea migration | +| Multiple ephemeral runner Jobs | Deferred — start with capacity=1 long-lived pod | + +## Consequences + +**Positive** + +- Single-machine production loop: forge + runner on railiance01, workstation not required. +- Container image CI (`docker build` / `docker push`) works without OS-level Docker. +- Runner upgrades roll with Git-managed manifests and `kubectl`/Makefile. + +**Negative / follow-on** + +- Privileged DinD increases blast radius within the node — monitor and restrict namespace RBAC. +- SOPS-encrypted registration secret still requires operator age key. +- `cluster-deploy` / `s5-release-check` labels remain **out of scope** until credential paths reviewed. + +## Ownership (OAS) + +| Concern | Repo | Layer | +| --- | --- | --- | +| ADR + umbrella sequencing | `railiance-infra` | S1 | +| Runner manifests + Makefile | `railiance-apps` | S5 | +| Label contract + runner evidence docs | `railiance-forge` | S5 forge substrate | +| Reusable workflow templates | `railiance-enablement` | S4 | + +## References + +- `railiance-apps/docs/forgejo-on-railiance01.md` +- `railiance-forge/docs/forgejo-actions-runner-substrate.md` +- `the-custodian/docs/forgejo-production-decisions.md` +- [Forgejo runner installation](https://forgejo.org/docs/v11.0/admin/actions/runner-installation/) \ No newline at end of file diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index 4614e83..be28c70 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -46,14 +46,18 @@ change is made there. ## Key Decisions to Confirm -1. Public/private hostname for Forgejo and whether Gitea remains reachable - during the transition. +1. ~~Public/private hostname for Forgejo~~ **DECIDED 2026-07-03:** + `forgejo.coulomb.social` → railiance01 (`92.205.62.239`). DNS active; + Traefik edge live; Forgejo workload not deployed yet (404). Gitea remains + canonical until migration drills pass. Record: + `the-custodian/docs/forgejo-production-decisions.md`. 2. Mail delivery path for password reset and account recovery (SMTP relay, sender domain, SPF/DKIM/DMARC expectations). 3. Package registry scope: container images only at first, or also generic, npm, PyPI, Go, Maven, and Helm packages. -4. Actions runner model: in-cluster ephemeral runners, long-lived runner pod, - or isolated host runner. +4. ~~Actions runner model~~ **DECIDED 2026-07-03:** in-cluster long-lived runner + Deployment with DinD sidecar on railiance01 (`ADR-004`). Interim coulombcore + host runner retired after cutover. 5. Backup destination and retention target for database, repositories, attachments, LFS, Actions artifacts/logs, and package data. 6. Cutover mode: freeze-and-migrate all repos in one window, or staged @@ -98,8 +102,7 @@ The probe is destroyed or explicitly archived after production Forgejo is live. ``` operator / agents / developers - -> private HTTPS endpoint - -> railiance01 ingress + -> https://forgejo.coulomb.social (railiance01 Traefik ingress) -> forgejo Service in forgejo namespace -> Forgejo Deployment/StatefulSet -> forgejo-db CloudNative PG Cluster in databases namespace @@ -144,7 +147,7 @@ manual, unsupported, or explicitly out of scope. ```task id: RAIL-HO-WP-0005-T02 -status: todo +status: progress priority: high needs_human: true state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3" @@ -152,10 +155,14 @@ state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3" Decide the production choices listed in "Key Decisions to Confirm". +**Partial (2026-07-03):** hostname and in-cluster runner model decided (`ADR-004`). +Remaining: SMTP, package scope, backup, cutover mode. See +`the-custodian/docs/forgejo-production-decisions.md`. + Expected output: - A short decision record in this workplan or a dedicated ADR. -- Hostname and exposure model. +- Hostname and exposure model. ✓ hostname; exposure follows railiance01 Traefik - SMTP provider and sender identity. - Package registry scope. - Actions runner isolation model. @@ -229,7 +236,7 @@ Forgejo app running. ```task id: RAIL-HO-WP-0005-T05 -status: todo +status: progress priority: high state_hub_task_id: "11540ba4-d31c-4f64-836b-c6de69107aa4" ``` @@ -245,6 +252,10 @@ Minimum scope: - Health/status targets in the Makefile. - Migration-safe configuration for coexistence with Gitea during the cutover. +**Partial (2026-07-03):** `railiance-apps` deploy live — HTTPS smoke pass, Actions +enabled, `coulomb` org + probe workflow success. Remaining: SOPS secrets, +SMTP, Docker on runner host for image builds, migration drills. + **Done when:** Forgejo runs on railiance01 against production platform services and can serve login, git clone/push, package registry, and admin operations. From 67b259f6dcbcafe8b5fed8801068bb64dae5425e Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 4 Jul 2026 01:02:42 +0200 Subject: [PATCH 06/10] Adapt RAIL-HO-WP-0005 for production Forgejo and staged repo ladder Reflects live railiance01 deploy, cancels isolated probe T03 in favor of in-production pilots, marks T08/T10 progress (forgejo-actions-probe, glas-harness), and documents tier 0-3 migration sequencing before state-hub. --- docs/forgejo-migration-inventory.md | 22 ++ ...HO-WP-0005-forgejo-production-migration.md | 217 +++++++++++------- 2 files changed, 157 insertions(+), 82 deletions(-) diff --git a/docs/forgejo-migration-inventory.md b/docs/forgejo-migration-inventory.md index dd047b4..860a127 100644 --- a/docs/forgejo-migration-inventory.md +++ b/docs/forgejo-migration-inventory.md @@ -204,3 +204,25 @@ lost or left with an untracked remote. This first pass satisfies the public and infrastructure metadata part of T01. T01 should remain open until the authenticated admin inventory and missing repo classification are complete. + +## Addendum (2026-07-04) — migration ladder and new repos + +`RAIL-HO-WP-0005` now uses a **staged per-repo ladder** instead of an isolated +probe namespace (T03 cancelled). Repos to add or re-classify on next inventory +refresh: + +| Repo | On Gitea (2026-06) | On Forgejo (2026-07-04) | Tier | Notes | +| --- | --- | --- | ---: | --- | +| `forgejo-actions-probe` | — | yes | 0 | Disposable runner/OCI probe | +| `glas-harness` | yes (not in table above) | yes (canonical) | 1 | Git+SSH+CI pilot; see `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` | + +**Tier definitions** (for per-repo `migration tier` column in a future refresh): + +| Tier | Criteria | Examples | +| ---: | --- | --- | +| 0 | Disposable integration probes | `forgejo-actions-probe` | +| 1 | Non-production; git+CI only | `glas-harness` | +| 2 | Non-production with container image + registry pull | TBD (`key-cape` candidate) | +| 3 | Production drain wave / sweep registration | `state-hub`, `issue-core`, … | + +Production repos stay on Gitea until tier 0–2 gates and T09 backup drill pass. diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index be28c70..a341b4d 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -8,7 +8,7 @@ status: active owner: railiance topic_slug: railiance created: "2026-05-03" -updated: "2026-06-04" +updated: "2026-07-04" state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018" --- @@ -24,6 +24,13 @@ Forgejo will become the heart of Railiance infrastructure. The work must be fully automated, backup-backed, recovery-drilled, and suitable for long-lived operation on railiance01 before any production cutover happens. +**Sequencing update (2026-07-04):** Production Forgejo is live on railiance01 +with Gitea still canonical per the safety contract. Repo cutover proceeds +**staged per-repo** using a migration ladder (disposable probes → non-production +pilots → image-capable pilots → production repos). `state-hub` is last. See +`CUST-WP-0054-T04` and +`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`. + ## Placement in the Railiance Tooling Set This workplan lives in `railiance-infra` because it is the cross-layer @@ -48,7 +55,7 @@ change is made there. 1. ~~Public/private hostname for Forgejo~~ **DECIDED 2026-07-03:** `forgejo.coulomb.social` → railiance01 (`92.205.62.239`). DNS active; - Traefik edge live; Forgejo workload not deployed yet (404). Gitea remains + Traefik edge live; Forgejo workload deployed and serving HTTPS. Gitea remains canonical until migration drills pass. Record: `the-custodian/docs/forgejo-production-decisions.md`. 2. Mail delivery path for password reset and account recovery @@ -60,8 +67,9 @@ change is made there. host runner retired after cutover. 5. Backup destination and retention target for database, repositories, attachments, LFS, Actions artifacts/logs, and package data. -6. Cutover mode: freeze-and-migrate all repos in one window, or staged - project-by-project transition. +6. Cutover mode: ~~freeze-all vs staged~~ **LEANING staged per-repo (2026-07-04)** + based on `glas-harness` pilot; operator confirmation still needed. Freeze-all + remains fallback for final production wave if drift risk is unacceptable. ## Safety Contract @@ -80,23 +88,30 @@ change is made there. repo. No plaintext SMTP passwords, admin tokens, runner tokens, or registry credentials in Git. -## Probe Strategy +## Probe and pilot strategy (revised 2026-07-04) -A `forgejo-railiance-probe` is reasonable and should be treated as a disposable -S5/S4 integration probe, not as the production install. +Original T03 planned a **disposable isolated-namespace probe** before any +production install. That path was **superseded**: production Forgejo deployed on +railiance01 under the safety contract (Gitea remains canonical; no Gitea deletes). -The probe should prove: +Integration evidence now comes from **in-production probes and repo pilots**: -- Helm values and cnpg database wiring converge cleanly. -- Initial admin bootstrap is automated and repeatable. -- SMTP/password reset works end-to-end. -- Package registry endpoints work for the package types Railiance needs first. -- Forgejo Actions can run a minimal workflow and publish a test package. -- Backup and restore works in an isolated namespace. -- Migration from a sample Gitea repo preserves git history, issues, releases, - wiki, LFS or attachments where applicable. +| Tier | Repo | Purpose | Status | +| --- | --- | --- | --- | +| 0 | `coulomb/forgejo-actions-probe` | Runner scheduling, DinD, OCI image-build | **done** | +| 1 | `coulomb/glas-harness` | Non-production git+SSH+CI routing drill | **done** | +| 2 | TBD (small lib with image, e.g. `key-cape`) | Image-build workflow + registry pull on railiance01 | **next** | +| 3 | Production set (`state-hub`, `issue-core`, …) | Canonical remotes, sweep paths, deploy loops | **gated** | -The probe is destroyed or explicitly archived after production Forgejo is live. +Each tier must pass before the next. T03 (isolated probe namespace) is cancelled; +acceptance criteria below are tracked across T05, T07, T08, and T10 instead. + +Still to prove before T11: + +- SMTP/password reset end-to-end (T06). +- Backup and restore in isolated namespace (T09). +- Issues/releases/wiki/LFS per inventory classification (T10 matrix). +- Operator SSH identity on Forgejo beyond interim `forgejo_admin` keys (T02/T10). ## Target Architecture @@ -141,6 +156,10 @@ Minimum inventory: Forgejo before cutover and classifies each migration item as automatic, manual, unsupported, or explicitly out of scope. +**Gap (2026-07-04):** first-pass inventory predates repos created after +2026-06-04 (e.g. `glas-harness`, `forgejo-actions-probe`). Refresh org repo +list and add a **migration tier** column (0–3) per repo before T11. + --- ### T02 — Resolve Forgejo production design decisions @@ -155,8 +174,10 @@ state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3" Decide the production choices listed in "Key Decisions to Confirm". -**Partial (2026-07-03):** hostname and in-cluster runner model decided (`ADR-004`). -Remaining: SMTP, package scope, backup, cutover mode. See +**Partial (2026-07-04):** hostname, exposure, deployment pattern, live deploy, +and in-cluster runner model decided (`ADR-004`). Cutover mode **leaning** staged +per-repo (glas-harness pilot). Remaining operator decisions: SMTP, package scope +beyond OCI, backup target, final cutover confirmation. See `the-custodian/docs/forgejo-production-decisions.md`. Expected output: @@ -174,36 +195,21 @@ choices. --- -### T03 — Build forgejo-railiance-probe +### T03 — Build forgejo-railiance-probe (isolated namespace) ```task id: RAIL-HO-WP-0005-T03 -status: todo +status: cancel priority: high state_hub_task_id: "b516018a-415e-4a58-8c62-07c14ece9353" ``` -Create a disposable probe environment for Forgejo before touching production. - -Expected repo ownership: - -- `railiance-platform`: probe cnpg database and storage dependencies. -- `railiance-apps`: probe Forgejo Helm values and namespace. -- `railiance-enablement`: probe Actions runner template and workflows. - -Probe acceptance: - -- `make forgejo-probe-deploy` or equivalent converges from a clean cluster - state. -- Admin bootstrap is automated. -- A test user can reset a password via email. -- A test repository can be created, cloned, pushed, and protected. -- A test package can be published and pulled. -- A test Forgejo Actions workflow runs successfully. -- A probe backup restores into an isolated namespace. - -**Done when:** the probe demonstrates the whole lifecycle without manual -cluster surgery. +**Cancelled 2026-07-04:** superseded by production Forgejo on railiance01 (T05) +plus in-production integration probes (`forgejo-actions-probe`, `glas-harness`). +Isolated-namespace probe added latency without reducing risk given the safety +contract (Gitea canonical, no deletes). Remaining T03 acceptance items map to: +T05 (deploy), T06 (mail), T07 (packages), T08 (Actions), T09 (backup restore), +T10 (repo migration drill). --- @@ -227,6 +233,11 @@ Minimum scope: packages, Actions artifacts, and logs. - Restore runbook for database and blob/package data. +**Partial (2026-07-04):** `forgejo-db` CNPG cluster healthy on railiance01 +(`make forgejo-db-status` → Cluster in healthy state). SOPS secret path and +network policies in `railiance-platform`. Remaining: backup/WAL archiving to +approved target, blob/package storage restore drill (feeds T09). + **Done when:** platform dependencies can be deployed and restored without the Forgejo app running. @@ -252,9 +263,11 @@ Minimum scope: - Health/status targets in the Makefile. - Migration-safe configuration for coexistence with Gitea during the cutover. -**Partial (2026-07-03):** `railiance-apps` deploy live — HTTPS smoke pass, Actions -enabled, `coulomb` org + probe workflow success. Remaining: SOPS secrets, -SMTP, Docker on runner host for image builds, migration drills. +**Partial (2026-07-04):** `railiance-apps` deploy live — HTTPS smoke pass, +ingress + TLS, SSH NodePort `30022`, Actions enabled, `coulomb` org, +`railiance01-build-01` runner (ADR-004). Git push/pull via HTTPS and +`forgejo-remote` SSH proven. Remaining: SOPS hardening for all secrets, +SMTP (T06), operator user accounts beyond `forgejo_admin`. **Done when:** Forgejo runs on railiance01 against production platform services and can serve login, git clone/push, package registry, and admin @@ -312,8 +325,13 @@ Acceptance: - Retention and cleanup expectations are documented. - Package data is included in backup and restore drills. -**Done when:** `state-hub` or a probe image can be published to Forgejo and -pulled by railiance01. +**Partial (2026-07-04):** OCI registry live (`/v2/` auth challenge). Probe image +`forgejo.coulomb.social/coulomb/forgejo-actions-probe` built and pushed via +Actions. Remaining: publish and pull a **tier-2 pilot** app image (not yet +`state-hub`); document retention; include packages in backup drill (T09). + +**Done when:** a tier-2 pilot image (or `state-hub` after explicit approval) can +be published to Forgejo and pulled by railiance01 k3s. --- @@ -321,7 +339,7 @@ pulled by railiance01. ```task id: RAIL-HO-WP-0005-T08 -status: todo +status: progress priority: high state_hub_task_id: "f45f98c9-2f02-4224-bbfd-c2e1ec38581e" ``` @@ -337,8 +355,16 @@ Minimum scope: - Secret handling policy for Actions. - Resource limits to avoid repeating previous single-node overload patterns. -**Done when:** a representative repository can run Forgejo Actions and publish -a test artifact without privileged cluster-wide credentials. +**Partial (2026-07-04):** in-cluster runner live (`railiance-apps/manifests/ +forgejo-runner.yaml`, ADR-004). Proven workflows: `forgejo-actions-probe` +(image-build), `glas-harness` (host+container CI smoke). Org secrets +`REGISTRY_USER`/`REGISTRY_TOKEN` set. Documented constraints: host runner is +non-root (static docker-cli, no `apk add`); `actions/checkout@v4` fails — use +`git clone` in job. Remaining: reusable workflow templates in +`railiance-enablement` (S4); resource limits review; no cluster-admin on runner. + +**Done when:** tier-2 pilot repo runs Forgejo Actions end-to-end and publishes +a pullable image without privileged cluster-wide credentials. --- @@ -376,29 +402,38 @@ with repository, package, and user recovery checks passing. --- -### T10 — Drill Gitea to Forgejo migration +### T10 — Drill Gitea to Forgejo migration (staged ladder) ```task id: RAIL-HO-WP-0005-T10 -status: todo +status: progress priority: high state_hub_task_id: "6befde73-00bc-4643-be0b-a7ce7944e75f" ``` -Run a non-production migration drill from Gitea to Forgejo. +Run staged migration drills from Gitea to Forgejo before production repos move. -Minimum checks: +**Tier 1 complete (2026-07-04):** `glas-harness` — git history preserved, +`origin` on Forgejo, `gitea` legacy remote retained, SSH+HTTPS push, CI smoke +green. Result matrix: +`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`. + +Minimum checks (per tier): - Git history and default branches preserved. - Issues, labels, milestones, releases, wiki, and attachments handled per - inventory classification. -- SSH/HTTPS clone and push paths work. -- Existing local remotes can be transformed predictably. -- State Hub registered repo remotes can be updated safely. -- Rollback plan is rehearsed. + inventory classification (N/A for tier-1 git-only repos). +- SSH/HTTPS clone and push paths work (`forgejo-remote` in `~/.ssh/config`). +- Existing local remotes can be transformed predictably (`origin`/`gitea` split). +- State Hub registered repo remotes can be updated safely (deferred for tier-1). +- Rollback plan is rehearsed (Gitea copy unchanged). -**Done when:** a sample migration has a written result matrix and no unknown -critical migration gaps remain. +**Next:** tier-2 repo with container image + `.gitea/workflows` port to +`.forgejo/workflows`. **Not ready:** `state-hub` until hub-core build context +template and sweep `remote_url` playbook exist. + +**Done when:** tiers 0–2 pass with written result matrices and no unknown +critical migration gaps remain for production repos. --- @@ -412,19 +447,21 @@ needs_human: true state_hub_task_id: "b1b66687-ca33-4971-b312-743c8e059c5e" ``` -Execute the production migration only after the probe, backup restore, package -registry, email recovery, and Actions gates pass. +Execute production migration only after T06, T07, T08, T09, and T10 tier 0–2 +gates pass. `state-hub` and other Wave-1 production repos require explicit +operator approval per `CUST-WP-0054` drain sequence. -Cutover sequence: +**Preferred cutover (staged per-repo):** -1. Announce freeze window. -2. Take final Gitea backup and verify it exists. -3. Freeze Gitea writes. -4. Migrate repositories and metadata to Forgejo. -5. Validate critical repositories and package pulls. -6. Update State Hub repo remotes and host paths as needed. -7. Update local and railiance01 remotes. -8. Keep Gitea read-only as rollback until the stabilization window passes. +1. Per repo: Gitea backup snapshot (or org-wide before each wave). +2. Mirror git to Forgejo; switch workstation `origin` to `forgejo-remote`. +3. Port/verify Actions workflows on Forgejo runner. +4. Update State Hub `remote_url` and railiance01 sweep checkouts when promoted. +5. Mark Gitea repo read-only (org policy); do not delete. +6. Repeat until production set complete. + +**Freeze-all fallback:** single window if staged drift is unacceptable — same +steps but all repos in one maintenance period. **Done when:** all Railiance/Custodian repos use Forgejo as primary, Gitea is read-only fallback, and rollback instructions are documented. @@ -458,19 +495,28 @@ legacy Gitea either archived or intentionally retained as documented fallback. ## Phasing and Dependencies ``` -T01 inventory ─┬─► T02 decisions ─┬─► T03 probe ─┬─► T04 platform - │ │ ├─► T05 app - │ │ ├─► T06 mail recovery - │ │ ├─► T07 packages - │ │ ├─► T08 actions - │ │ └─► T09 backups - └────────────────────────────────────► T10 migration drill +T01 inventory ──► T02 decisions ──┬──► T04 platform (forgejo-db ✓ partial) + ├──► T05 app (live ✓ partial) + ├──► T06 mail recovery + ├──► T07 packages (OCI probe ✓ partial) + ├──► T08 actions (runner ✓ partial) + └──► T09 backups -T03-T10 all pass ─► T11 production cutover ─► T12 legacy Gitea retirement +T05+T08 ──► T10 migration ladder ──► T11 production cutover ──► T12 Gitea retire + tier0 probe ✓ + tier1 glas-harness ✓ + tier2 image repo (next) + tier3 production (gated) + +T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots) ``` -Recommended first slice: T01, T02, T03. Do not start T11 until T06, T07, T08, -T09, and T10 are complete. +**Current focus (2026-07-04):** T10 tier-2 image pilot; parallel T09 backup +drill and T02 open decisions (SMTP, backup target). Do not start T11 +`state-hub` until T09 complete and `CUST-WP-0054` Wave-1 gates satisfied. + +**Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation +build retirement; staged repo promotion before State Hub primary move (T05). ## railiance-bootstrap Note @@ -490,7 +536,14 @@ purpose is identified. - `RAIL-HO-WP-0004-production-readiness.md` - `RAIL-HO-WP-0003-5repo-stack-restructure.md` +- `CUST-WP-0054-workstation-independence-and-fleet-realignment.md` (T04 forge+CI) - `CUST-WP-0014-repo-sync-automation.md` - `CUST-WP-0021-multi-host-repo-paths.md` +- `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md` +- `docs/forgejo-migration-inventory.md` +- `the-custodian/docs/forgejo-production-decisions.md` +- `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` +- `railiance-apps/docs/forgejo-on-railiance01.md` +- `railiance-forge/docs/forgejo-actions-runner-substrate.md` - `ops/incidents/2026-03-25-gitea-pgpool-crashloop.md` - `ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md` From 2d62317adaf5f61a0b2dc35e0b7453cdf1df42a7 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 4 Jul 2026 10:26:28 +0200 Subject: [PATCH 07/10] RAIL-HO-WP-0005-T10: tier-2 key-cape image pilot complete Records successful key-cape Forgejo migration, image CI workflow, and railiance01 registry pull; tiers 0-2 of migration ladder now satisfied. --- docs/forgejo-migration-inventory.md | 7 +++- ...HO-WP-0005-forgejo-production-migration.md | 40 ++++++++++++------- 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/docs/forgejo-migration-inventory.md b/docs/forgejo-migration-inventory.md index 860a127..0131e40 100644 --- a/docs/forgejo-migration-inventory.md +++ b/docs/forgejo-migration-inventory.md @@ -214,7 +214,10 @@ refresh: | Repo | On Gitea (2026-06) | On Forgejo (2026-07-04) | Tier | Notes | | --- | --- | --- | ---: | --- | | `forgejo-actions-probe` | — | yes | 0 | Disposable runner/OCI probe | -| `glas-harness` | yes (not in table above) | yes (canonical) | 1 | Git+SSH+CI pilot; see `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` | +| `glas-harness` | yes (not in table above) | yes (canonical) | 1 | Git+SSH+CI pilot | +| `key-cape` | yes | yes (canonical) | 2 | Image CI + `crictl pull` on railiance01 | + +Evidence: `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` **Tier definitions** (for per-repo `migration tier` column in a future refresh): @@ -222,7 +225,7 @@ refresh: | ---: | --- | --- | | 0 | Disposable integration probes | `forgejo-actions-probe` | | 1 | Non-production; git+CI only | `glas-harness` | -| 2 | Non-production with container image + registry pull | TBD (`key-cape` candidate) | +| 2 | Non-production with container image + registry pull | `key-cape` (done) | | 3 | Production drain wave / sweep registration | `state-hub`, `issue-core`, … | Production repos stay on Gitea until tier 0–2 gates and T09 backup drill pass. diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index a341b4d..f762088 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -100,7 +100,7 @@ Integration evidence now comes from **in-production probes and repo pilots**: | --- | --- | --- | --- | | 0 | `coulomb/forgejo-actions-probe` | Runner scheduling, DinD, OCI image-build | **done** | | 1 | `coulomb/glas-harness` | Non-production git+SSH+CI routing drill | **done** | -| 2 | TBD (small lib with image, e.g. `key-cape`) | Image-build workflow + registry pull on railiance01 | **next** | +| 2 | `coulomb/key-cape` | Image-build workflow + registry pull on railiance01 | **done** | | 3 | Production set (`state-hub`, `issue-core`, …) | Canonical remotes, sweep paths, deploy loops | **gated** | Each tier must pass before the next. T03 (isolated probe namespace) is cancelled; @@ -325,13 +325,14 @@ Acceptance: - Retention and cleanup expectations are documented. - Package data is included in backup and restore drills. -**Partial (2026-07-04):** OCI registry live (`/v2/` auth challenge). Probe image -`forgejo.coulomb.social/coulomb/forgejo-actions-probe` built and pushed via -Actions. Remaining: publish and pull a **tier-2 pilot** app image (not yet -`state-hub`); document retention; include packages in backup drill (T09). +**Partial (2026-07-04):** OCI registry live (`/v2/` auth challenge). Tier-0/2 +images built and pulled on railiance01: `forgejo-actions-probe`, `key-cape` +(`crictl pull forgejo.coulomb.social/coulomb/key-cape:latest` succeeded). +Remaining: `state-hub` image after tier-3 approval; document retention; include +packages in backup drill (T09). -**Done when:** a tier-2 pilot image (or `state-hub` after explicit approval) can -be published to Forgejo and pulled by railiance01 k3s. +**Done when:** tier-2 gate is fully satisfied (✓) and tier-3 production images +follow the same pattern after explicit approval. --- @@ -363,8 +364,12 @@ non-root (static docker-cli, no `apk add`); `actions/checkout@v4` fails — use `git clone` in job. Remaining: reusable workflow templates in `railiance-enablement` (S4); resource limits review; no cluster-admin on runner. +**Partial (2026-07-04):** tier-2 satisfied by `key-cape` (`container-build`, +archive checkout, static docker-cli). Remaining: publish reusable workflow +template in `railiance-enablement` (S4). + **Done when:** tier-2 pilot repo runs Forgejo Actions end-to-end and publishes -a pullable image without privileged cluster-wide credentials. +a pullable image without privileged cluster-wide credentials. **Tier 2: done.** --- @@ -428,9 +433,13 @@ Minimum checks (per tier): - State Hub registered repo remotes can be updated safely (deferred for tier-1). - Rollback plan is rehearsed (Gitea copy unchanged). -**Next:** tier-2 repo with container image + `.gitea/workflows` port to -`.forgejo/workflows`. **Not ready:** `state-hub` until hub-core build context -template and sweep `remote_url` playbook exist. +**Tier 2 complete (2026-07-04):** `key-cape` — multi-stage Dockerfile built and +pushed via archive-checkout workflow; `crictl pull` on railiance01 succeeded. +Evidence in `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md` +(tier 2 section). + +**Not ready:** `state-hub` (tier 3) until hub-core build context template and +sweep `remote_url` playbook exist. **Done when:** tiers 0–2 pass with written result matrices and no unknown critical migration gaps remain for production repos. @@ -505,15 +514,16 @@ T01 inventory ──► T02 decisions ──┬──► T04 platform (forgejo-d T05+T08 ──► T10 migration ladder ──► T11 production cutover ──► T12 Gitea retire tier0 probe ✓ tier1 glas-harness ✓ - tier2 image repo (next) + tier2 key-cape ✓ tier3 production (gated) T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots) ``` -**Current focus (2026-07-04):** T10 tier-2 image pilot; parallel T09 backup -drill and T02 open decisions (SMTP, backup target). Do not start T11 -`state-hub` until T09 complete and `CUST-WP-0054` Wave-1 gates satisfied. +**Current focus (2026-07-04):** T10 tiers 0–2 **complete**; T09 backup drill +and T02 open decisions (SMTP, backup target) before tier-3 production repos. +Do not start T11 `state-hub` until T09 complete and `CUST-WP-0054` Wave-1 +gates satisfied. **Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation build retirement; staged repo promotion before State Hub primary move (T05). From 092315895fc801825fb7b16411255d896769149f Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 4 Jul 2026 11:26:50 +0200 Subject: [PATCH 08/10] RAIL-HO-WP-0005-T09: Forgejo backup/restore drill assets and evidence Add isolated-namespace restore drill (CNPG cluster, PVC, orchestration script) and document successful 2026-07-04 run: production forgejo dump restored with health 200 and pilot repos visible via API. Scheduled backups remain open. --- docs/forgejo-restore-drill-evidence.md | 93 ++++++++++++++ .../forgejo-db-restore-cluster.yaml | 21 ++++ infra/forgejo-restore-drill/restore-job.yaml | 12 ++ tools/forgejo-restore-drill.sh | 115 ++++++++++++++++++ ...HO-WP-0005-forgejo-production-migration.md | 21 +++- 5 files changed, 257 insertions(+), 5 deletions(-) create mode 100644 docs/forgejo-restore-drill-evidence.md create mode 100644 infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml create mode 100644 infra/forgejo-restore-drill/restore-job.yaml create mode 100755 tools/forgejo-restore-drill.sh diff --git a/docs/forgejo-restore-drill-evidence.md b/docs/forgejo-restore-drill-evidence.md new file mode 100644 index 0000000..adceb3b --- /dev/null +++ b/docs/forgejo-restore-drill-evidence.md @@ -0,0 +1,93 @@ +# Forgejo Backup/Restore Drill Evidence + +Date: 2026-07-04 +Workplan: RAIL-HO-WP-0005 +Task: RAIL-HO-WP-0005-T09 +`no_secret_material_recorded: true` + +## Purpose + +Prove that a production `forgejo dump` can be restored into an isolated +namespace and serve repository metadata without touching production Forgejo or +Gitea. + +## Backup source + +| Field | Value | +| --- | --- | +| Method | `forgejo dump` from production pod | +| Production pod | `forgejo-gitea-64c5b57684-ph9vt` (namespace `forgejo`) | +| Archive path (workstation) | `/tmp/forgejo-drill/forgejo-drill-backup.zip` | +| Archive size | 12,284,847 bytes (~11.7 MiB) | +| Archive timestamp | 2026-07-04 11:20 +0200 | +| Archive contents (top-level) | `repos/`, `data/`, `forgejo-db.sql`, `app.ini` | + +Repos present in dump: `forgejo-actions-probe`, `glas-harness`, `key-cape` +(all under `repos/coulomb/`). + +## Restore target + +| Field | Value | +| --- | --- | +| Namespace | `forgejo-restore-drill` | +| Database | CNPG cluster `forgejo-db-restore` (isolated, 1 instance) | +| App data PVC | `forgejo-restore-data` (`local-path`, 10Gi) | +| Helm release | `forgejo-restore` (`gitea-charts/gitea` 12.5.0) | +| Orchestration | `tools/forgejo-restore-drill.sh` | + +Restore path (Forgejo 11.0.3 has no `forgejo restore` CLI): + +1. Unzip dump into import pod staging area. +2. Copy `repos/` → `/data/git/gitea-repositories/`. +3. Copy `data/` → `/data/` (packages, attachments, avatars). +4. Import `forgejo-db.sql` via `psql` into `forgejo-db-restore`. +5. Deploy isolated Helm release bound to restored PVC + restore DB host. + +## Post-restore checks (2026-07-04) + +Port-forward: `svc/forgejo-restore-gitea-http` → `127.0.0.1:13000` + +| Check | Result | +| --- | --- | +| `GET /` health | HTTP 200 | +| `GET /api/v1/repos/coulomb/glas-harness` | `full_name=coulomb/glas-harness`, `default_branch=main` | +| `GET /api/v1/repos/coulomb/key-cape` | `full_name=coulomb/key-cape`, `default_branch=main` | +| `GET /api/v1/orgs/coulomb/repos` | 3 repos: `forgejo-actions-probe`, `glas-harness`, `key-cape` | + +Script exit marker: `restore-drill-complete` + +## RPO / RTO (drill scope) + +| Metric | Observed / assumed | +| --- | --- | +| RPO (manual dump) | Point-in-time of `forgejo dump` execution; no scheduled backup yet | +| RTO (isolated restore) | ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01 | +| Production impact | None — read-only dump from running pod; separate namespace | + +## Gaps (not closed by this drill) + +- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured + (`kubectl cnpg` plugin absent on workstation). +- **Encryption at rest:** dump stored locally on workstation for drill only; no + approved backup target wired. +- **Automation:** `forgejo dump` is manual; T04/T09 still need cron/operator + schedule and retention policy (T02 decision). +- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe + `forgejo-restore-drill` before import (SQL import is not idempotent). + +## Cleanup + +After evidence capture, delete the drill namespace: + +```bash +kubectl delete namespace forgejo-restore-drill --wait=true +``` + +Production Forgejo (`forgejo` namespace) and Gitea remain unchanged. + +## References + +- `infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml` +- `infra/forgejo-restore-drill/restore-job.yaml` +- `tools/forgejo-restore-drill.sh` +- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` (T09) \ No newline at end of file diff --git a/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml b/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml new file mode 100644 index 0000000..302c396 --- /dev/null +++ b/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml @@ -0,0 +1,21 @@ +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: forgejo-db-restore + namespace: forgejo-restore-drill + labels: + app.kubernetes.io/name: forgejo-db-restore + railiance.io/layer: s3-platform + railiance.io/consumer: forgejo-restore-drill +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:16 + storage: + size: 10Gi + bootstrap: + initdb: + database: forgejo + owner: forgejo + secret: + name: forgejo-db-credentials \ No newline at end of file diff --git a/infra/forgejo-restore-drill/restore-job.yaml b/infra/forgejo-restore-drill/restore-job.yaml new file mode 100644 index 0000000..0c02577 --- /dev/null +++ b/infra/forgejo-restore-drill/restore-job.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: forgejo-restore-data + namespace: forgejo-restore-drill +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: local-path \ No newline at end of file diff --git a/tools/forgejo-restore-drill.sh b/tools/forgejo-restore-drill.sh new file mode 100755 index 0000000..10fb43e --- /dev/null +++ b/tools/forgejo-restore-drill.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Non-production Forgejo backup/restore drill (RAIL-HO-WP-0005-T09). +# Re-run: DRILL_CLEAN=1 ./tools/forgejo-restore-drill.sh (wipes namespace first) +set -euo pipefail + +KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config-hosteurope}" +export KUBECONFIG +NS=forgejo-restore-drill +DRILL_CLEAN="${DRILL_CLEAN:-0}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +BACKUP_LOCAL="${BACKUP_LOCAL:-/tmp/forgejo-drill/forgejo-drill-backup.zip}" +PROD_POD="${PROD_POD:-$(kubectl get pods -n forgejo -l app.kubernetes.io/instance=forgejo -o jsonpath='{.items[0].metadata.name}')}" + +step() { echo "==> $*"; } + +if [[ "${DRILL_CLEAN}" == "1" ]]; then + step "Clean prior drill namespace ${NS}" + kubectl delete namespace "${NS}" --wait=true --timeout=5m || true +fi + +step "Create namespace ${NS}" +kubectl create namespace "${NS}" --dry-run=client -o yaml | kubectl apply -f - + +step "Copy forgejo-db-credentials into ${NS}" +kubectl get secret forgejo-db-credentials -n databases -o json \ + | python3 -c "import json,sys; s=json.load(sys.stdin); s['metadata']={k:v for k,v in s['metadata'].items() if k in ('name','labels','annotations')}; s['metadata']['namespace']='${NS}'; print(json.dumps(s))" \ + | kubectl apply -f - + +step "Deploy restore CNPG cluster" +kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml" +kubectl wait --for=condition=Ready cluster/forgejo-db-restore -n "${NS}" --timeout=10m + +step "Ensure local backup exists" +if [[ ! -f "${BACKUP_LOCAL}" ]]; then + kubectl exec -n forgejo "${PROD_POD}" -c gitea -- forgejo dump -f /tmp/forgejo-drill-backup.zip + mkdir -p "$(dirname "${BACKUP_LOCAL}")" + kubectl cp "forgejo/${PROD_POD}:/tmp/forgejo-drill-backup.zip" "${BACKUP_LOCAL}" -c gitea +fi +ls -lh "${BACKUP_LOCAL}" + +step "Apply restore PVC" +kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/restore-job.yaml" + +step "Run restore pod (stage backup, import files + SQL)" +kubectl delete pod forgejo-restore-import -n "${NS}" --ignore-not-found --wait=true +cat </dev/null +rm -rf /data/* +mkdir -p /data/git/gitea-repositories +unzip -q /backup/forgejo-drill-backup.zip -d /tmp/dump +cp -a /tmp/dump/repos/. /data/git/gitea-repositories/ +cp -a /tmp/dump/data/. /data/ +chown -R git:git /data +PGPASSWORD="${POSTGRES_PASSWORD}" psql -h forgejo-db-restore-rw.forgejo-restore-drill.svc.cluster.local -U forgejo -d forgejo -v ON_ERROR_STOP=1 -f /tmp/dump/forgejo-db.sql +echo restore-import-ok +' +unset DB_PASS +kubectl delete pod forgejo-restore-import -n "${NS}" --wait=true + +step "Deploy isolated Forgejo release" +cd "${HOME}/railiance-apps" +DB_PASS="$(kubectl get secret forgejo-db-credentials -n "${NS}" -o jsonpath='{.data.password}' | base64 -d)" +helm upgrade --install forgejo-restore gitea-charts/gitea --version 12.5.0 \ + --namespace "${NS}" --create-namespace \ + -f helm/forgejo-values.yaml \ + -f helm/forgejo-registry-values.yaml \ + --set strategy.type=Recreate \ + --set persistence.existingClaim=forgejo-restore-data \ + --set gitea.config.database.HOST=forgejo-db-restore-rw.${NS}.svc.cluster.local:5432 \ + --set gitea.config.database.PASSWD="${DB_PASS}" \ + --set gitea.config.server.DOMAIN=forgejo-restore.local \ + --set gitea.config.server.ROOT_URL=http://forgejo-restore.local:3000/ \ + --set gitea.admin.password=restore-drill-local-only \ + --set ingress.enabled=false \ + --wait --timeout=10m +unset DB_PASS + +step "Post-restore checks via port-forward" +kubectl port-forward -n "${NS}" svc/forgejo-restore-gitea-http 13000:3000 >/tmp/forgejo-restore-pf.log 2>&1 & +PF_PID=$! +sleep 5 +curl -fsS -o /dev/null -w 'health:%{http_code}\n' http://127.0.0.1:13000/ +curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/glas-harness | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))" +curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/key-cape | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))" +kill "${PF_PID}" 2>/dev/null || true +echo "restore-drill-complete" \ No newline at end of file diff --git a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md index f762088..4eaef57 100644 --- a/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md +++ b/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md @@ -109,7 +109,8 @@ acceptance criteria below are tracked across T05, T07, T08, and T10 instead. Still to prove before T11: - SMTP/password reset end-to-end (T06). -- Backup and restore in isolated namespace (T09). +- Backup and restore in isolated namespace (T09) — **drill passed 2026-07-04**; + scheduled automation pending. - Issues/releases/wiki/LFS per inventory classification (T10 matrix). - Operator SSH identity on Forgejo beyond interim `forgejo_admin` keys (T02/T10). @@ -377,7 +378,7 @@ a pullable image without privileged cluster-wide credentials. **Tier 2: done.** ```task id: RAIL-HO-WP-0005-T09 -status: todo +status: progress priority: high state_hub_task_id: "25892007-36ca-4bd9-8adf-84d505465d7d" ``` @@ -402,8 +403,17 @@ Acceptance: - Restore into an isolated namespace is drilled and documented. - RPO/RTO expectations are recorded. +**Partial (2026-07-04):** isolated restore drill **passed**. Production +`forgejo dump` (~11.7 MiB) restored into `forgejo-restore-drill` namespace; +post-restore API checks: health 200, `coulomb/glas-harness` and +`coulomb/key-cape` on `main`, 3 org repos visible. Evidence: +`docs/forgejo-restore-drill-evidence.md`. Assets: `infra/forgejo-restore-drill/`, +`tools/forgejo-restore-drill.sh`. Remaining: scheduled CNPG/off-cluster backups, +encryption/approved target (T02/T04), automated dump schedule. + **Done when:** a fresh backup restores to a working isolated Forgejo instance -with repository, package, and user recovery checks passing. +with repository, package, and user recovery checks passing **and** scheduled +backups run without manual intervention. --- @@ -520,8 +530,9 @@ T05+T08 ──► T10 migration ladder ──► T11 production cutover ── T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots) ``` -**Current focus (2026-07-04):** T10 tiers 0–2 **complete**; T09 backup drill -and T02 open decisions (SMTP, backup target) before tier-3 production repos. +**Current focus (2026-07-04):** T10 tiers 0–2 **complete**; T09 restore drill +**passed** (scheduled backups + backup target still open); T02 decisions (SMTP, +backup target) before tier-3 production repos. Do not start T11 `state-hub` until T09 complete and `CUST-WP-0054` Wave-1 gates satisfied. From e7eae7f7f5a2b8d7517a2395a5d94cde9250195e Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 4 Jul 2026 11:27:06 +0200 Subject: [PATCH 09/10] chore(consistency): sync task status from DB [auto] Updated by fix-consistency on 2026-07-04: - update .custodian-brief.md for railiance-infra --- .custodian-brief.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.custodian-brief.md b/.custodian-brief.md index 879628d..8da959e 100644 --- a/.custodian-brief.md +++ b/.custodian-brief.md @@ -2,23 +2,23 @@ # Custodian Brief — railiance-infra **Domain:** financials -**Last synced:** 2026-06-22 20:26 UTC +**Last synced:** 2026-07-04 09:27 UTC **State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)* ## Active Workstreams ### Forgejo Production Migration on railiance01 -Progress: 0/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018` +Progress: 1/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018` **Open tasks:** - ► T01 — Inventory current Gitea functionality and migration requirements `cf59d171` -- · T02 — Resolve Forgejo production design decisions `f88115bf` -- · T03 — Build forgejo-railiance-probe `b516018a` +- ► T02 — Resolve Forgejo production design decisions `f88115bf` +- ► T05 — Define production Forgejo application deployment `11540ba4` +- ► T08 — Enable Forgejo Actions `f45f98c9` +- ► T09 — Implement Forgejo backup and restore automation `25892007` +- ► T10 — Drill Gitea to Forgejo migration `6befde73` - · T04 — Define Forgejo platform services `28b351fe` -- · T05 — Define production Forgejo application deployment `11540ba4` -- · T06 — Implement usable email recovery cycle `417faa4d` -- · T07 — Enable and harden package registry base `9578f672` -- … and 5 more open tasks +- … and 4 more open tasks --- ## MCP Orientation (when available) From 67beab1525b1b1eeddf6adedfb22346394e84aab Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 4 Jul 2026 12:49:52 +0200 Subject: [PATCH 10/10] Add Forgejo CI smoke workflow (enablement template) --- .forgejo/workflows/ci-smoke.yaml | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 .forgejo/workflows/ci-smoke.yaml diff --git a/.forgejo/workflows/ci-smoke.yaml b/.forgejo/workflows/ci-smoke.yaml new file mode 100644 index 0000000..bd44c56 --- /dev/null +++ b/.forgejo/workflows/ci-smoke.yaml @@ -0,0 +1,29 @@ +# Canonical CI smoke template (tier 1 routing drill). +# Copy to: .forgejo/workflows/ci-smoke.yaml in consumer repos. +name: CI Smoke + +on: + push: + branches: + - main + workflow_dispatch: + +jobs: + host-smoke: + runs-on: self-hosted + steps: + - name: Routing probe (host runner) + run: | + set -eu + echo "repository=${GITHUB_REPOSITORY:-unknown}" + echo "sha=${GITHUB_SHA:-unknown}" + echo "runner=${RUNNER_NAME:-unknown}" + uname -a + + container-smoke: + runs-on: ubuntu-latest + steps: + - name: Routing probe (container label) + run: | + set -eu + echo "container-smoke ok for ${GITHUB_REPOSITORY:-unknown}" \ No newline at end of file