Archive closed workplans to workplans/archived/ (ADR-001)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
73a30f0a44
commit
3f2564748c
4 changed files with 0 additions and 0 deletions
|
|
@ -0,0 +1,219 @@
|
|||
---
|
||||
id: RAIL-HO-WP-0001
|
||||
type: workplan
|
||||
title: "Secure Single-Server Bootstrap at HostEurope"
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
repo_goal_id: 9e835b82-acca-493a-943f-2553ffe0bf54
|
||||
state_hub_workstream_id: "bf40b47e-be5b-4930-a7d2-362e76b943bb"
|
||||
created: "2026-03-08"
|
||||
updated: "2026-03-09"
|
||||
completed: "2026-03-09"
|
||||
handoff_note: >
|
||||
T01 and T02 (inventory entry and SSH tunnel setup) are prerequisites to run
|
||||
T03 onwards from the HostEurope server itself. The ansible work previously
|
||||
done in railiance-bootstrap (harden.yml, bootstrap.yml) is superseded by
|
||||
the roles/base and roles/sops_agent structure in this repo. The HostEurope
|
||||
server IP is 92.205.62.239 (hosts.ini is gitignored — recreate from
|
||||
inventory/servers.yaml when working on the host).---
|
||||
|
||||
# Secure Single-Server Bootstrap at HostEurope
|
||||
|
||||
## Goal
|
||||
|
||||
Provision and converge the HostEurope server so that it becomes a secure,
|
||||
hardened node ready to join the ThreePhoenix Kubernetes cluster. This workplan
|
||||
covers everything from establishing SSH access and tunnel connectivity through a
|
||||
fully converged, verified server — all secrets managed via SOPS/age, all
|
||||
services secured from the start.
|
||||
|
||||
Scope is deliberately narrow: one server, secure from day one. Automated
|
||||
provisioning of additional server resources and the full three-node setup
|
||||
are deferred.
|
||||
|
||||
## Boundary conditions
|
||||
|
||||
- Ubuntu 24.04 LTS at HostEurope (92.205.62.239), manually provisioned
|
||||
- All remote access key-based only (no password auth)
|
||||
- Firewall active before k3s is installed
|
||||
- Secrets managed via SOPS/age — nothing committed in plaintext
|
||||
- This workplan is **run from the HostEurope server** via SSH, not from WSL
|
||||
|
||||
## Remote execution setup
|
||||
|
||||
This repo is intended to run from the HostEurope server itself. To retain
|
||||
State Hub connectivity during Claude sessions on that server, set up an SSH
|
||||
reverse tunnel from your local machine before connecting:
|
||||
|
||||
```bash
|
||||
# On your local machine — forward local state-hub to the remote
|
||||
ssh -R 8000:127.0.0.1:8000 <user>@92.205.62.239
|
||||
```
|
||||
|
||||
Then Claude on the remote host can reach `http://127.0.0.1:8000` as normal.
|
||||
See also: the tunnel setup note in `CLAUDE.md`.
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### T01 — Add HostEurope host to inventory
|
||||
|
||||
```task
|
||||
id: T01
|
||||
status: done
|
||||
completed: "2026-03-08"
|
||||
priority: high
|
||||
state_hub_task_id: "9b2222a3-0f9f-4543-9321-e4cd5f87a457"
|
||||
```
|
||||
|
||||
Add the HostEurope host to `inventory/servers.yaml` under a `hosteurope` group.
|
||||
Create `ansible/hosts.ini` from the inventory (gitignored — recreate on each
|
||||
new working machine):
|
||||
|
||||
```bash
|
||||
python3 ansible/inventory_from_yaml.py > ansible/hosts.ini
|
||||
# or add manually: [hosteurope]\n92.205.62.239 ansible_user=...
|
||||
```
|
||||
|
||||
Verify connectivity:
|
||||
|
||||
```bash
|
||||
ansible -i ansible/hosts.ini hosteurope -m ping
|
||||
```
|
||||
|
||||
**Done when:** ping succeeds from a control node with network access to the host.
|
||||
|
||||
---
|
||||
|
||||
### T02 — Set up SSH tunnel for State Hub access
|
||||
|
||||
```task
|
||||
id: T02
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: high
|
||||
state_hub_task_id: "e4dda416-19bc-4672-b9ba-8ddb1b9e9659"
|
||||
```
|
||||
|
||||
On your local machine, establish a reverse tunnel before SSH-ing to the
|
||||
HostEurope server so the State Hub MCP server is reachable from Claude
|
||||
sessions on that host:
|
||||
|
||||
```bash
|
||||
make tunnel # reads host from inventory/servers.yaml
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
ssh -R 8000:127.0.0.1:8000 tegwick@92.205.62.239
|
||||
```
|
||||
|
||||
Verify the tunnel is working from the remote:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8000/state/health
|
||||
```
|
||||
|
||||
**Done when:** `make tunnel` target implemented; procedure documented.
|
||||
|
||||
---
|
||||
|
||||
### T03 — Extend base role with security hardening
|
||||
|
||||
```task
|
||||
id: T03
|
||||
status: done
|
||||
completed: "2026-03-08"
|
||||
priority: high
|
||||
state_hub_task_id: "6eda6875-1301-4794-a07e-3e13ff1d92bf"
|
||||
```
|
||||
|
||||
Extend `ansible/roles/base/tasks/main.yml` to cover security hardening that
|
||||
must run before any service installation:
|
||||
|
||||
- Disable root SSH login (`PermitRootLogin no`)
|
||||
- Disable password authentication (`PasswordAuthentication no`)
|
||||
- Enable and configure UFW: deny all inbound, allow SSH (22), k3s API (6443),
|
||||
Flannel VXLAN (8472/UDP)
|
||||
- Install and enable `fail2ban` with SSH jail
|
||||
- Set `HISTCONTROL=ignorespace` in `/etc/profile.d/`
|
||||
|
||||
Verify with:
|
||||
```bash
|
||||
ansible-playbook -i ansible/hosts.ini -l hosteurope ansible/playbooks/bootstrap.yaml --check
|
||||
```
|
||||
|
||||
**Done when:** dry-run produces no errors and hardening tasks are visible in
|
||||
the play recap.
|
||||
|
||||
---
|
||||
|
||||
### T04 — Run bootstrap on the HostEurope host
|
||||
|
||||
```task
|
||||
id: T04
|
||||
status: done
|
||||
completed: "2026-03-08"
|
||||
priority: high
|
||||
state_hub_task_id: "77921431-3a45-45b2-a0b0-cf0c43262205"
|
||||
```
|
||||
|
||||
Execute the full bootstrap playbook from the HostEurope server (tunnel must
|
||||
be active per T02):
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/hosts.ini -l hosteurope ansible/playbooks/bootstrap.yaml
|
||||
```
|
||||
|
||||
**Done when:**
|
||||
- Playbook completes with no failed tasks
|
||||
- UFW is active and SSH still works after the run
|
||||
- SOPS agent role applied successfully
|
||||
|
||||
---
|
||||
|
||||
### T05 — Smoke test and record
|
||||
|
||||
```task
|
||||
id: T05
|
||||
status: done
|
||||
completed: "2026-03-08"
|
||||
priority: medium
|
||||
state_hub_task_id: "c573c200-bf22-49d1-86f9-dca1fc71743c"
|
||||
```
|
||||
|
||||
Verify the converged state:
|
||||
|
||||
```bash
|
||||
# Confirm UFW active
|
||||
ansible -i ansible/hosts.ini hosteurope -m shell -a "ufw status"
|
||||
# Confirm fail2ban running
|
||||
ansible -i ansible/hosts.ini hosteurope -m shell -a "systemctl is-active fail2ban"
|
||||
# Confirm SSH hardening applied
|
||||
ansible -i ansible/hosts.ini hosteurope -m shell -a "sshd -T | grep -E 'permitrootlogin|passwordauthentication'"
|
||||
```
|
||||
|
||||
Add `docs/hosteurope-bootstrap.md` recording:
|
||||
- Server specs (vCPU, RAM, disk)
|
||||
- Public IP (no credentials)
|
||||
- Date bootstrapped
|
||||
- Roles applied
|
||||
|
||||
Log completion to the State Hub:
|
||||
```
|
||||
add_progress_event(summary="HostEurope server bootstrapped and hardened",
|
||||
event_type="milestone", workstream_id="bf40b47e-be5b-4930-a7d2-362e76b943bb")
|
||||
```
|
||||
|
||||
**Done when:** all checks pass and the progress event is logged.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Repo goal: `9e835b82-acca-493a-943f-2553ffe0bf54`
|
||||
- Domain goal: `6f96c712-60e6-4ea9-ab06-168878eafbce` (Three-Phoenix Secure Kubernetes Infrastructure)
|
||||
- Previous ansible work: `railiance-bootstrap/ansible/` (harden.yml, bootstrap.yml — superseded by this repo's role structure)
|
||||
|
|
@ -0,0 +1,273 @@
|
|||
---
|
||||
id: RAIL-HO-WP-0002
|
||||
type: workplan
|
||||
title: "Server Specification and Automated Test Suite"
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
state_hub_workstream_id: "8fed53c2-4c39-4471-8bb9-61f58771fe0c"
|
||||
created: "2026-03-09"
|
||||
updated: "2026-03-09"
|
||||
completed: "2026-03-09"
|
||||
---
|
||||
|
||||
# Server Specification and Automated Test Suite
|
||||
|
||||
## Motivation
|
||||
|
||||
`make status` produces raw shell output that requires manual interpretation.
|
||||
There is no machine-readable specification of what a converged Railiance node
|
||||
should look like, and therefore no way to assert automatically whether a server
|
||||
is in the correct state.
|
||||
|
||||
This workplan closes that gap by introducing:
|
||||
|
||||
1. **A declarative server specification** (`spec/server-baseline.yaml`) — the
|
||||
single source of truth for the target state of every managed node.
|
||||
2. **A Goss test suite** derived from that spec — YAML assertions that map
|
||||
one-to-one to spec items and produce a structured pass/fail report.
|
||||
3. **`make verify`** — runs the test suite against all hosts and exits non-zero
|
||||
on failure, suitable for CI.
|
||||
4. **An ADR** that formally defines the boundary between `railiance-hosts` and
|
||||
`railiance-bootstrap`.
|
||||
|
||||
## Concept
|
||||
|
||||
### Separation of concerns
|
||||
|
||||
| Repo | Responsibility |
|
||||
|------|----------------|
|
||||
| `railiance-hosts` | **What** a managed node should look like (spec), **how** to get it there (Ansible roles), **how to verify** it got there (Goss tests), inventory, secrets |
|
||||
| `railiance-bootstrap` | Upstream Kubernetes/app-layer provisioning that builds on an already-converged base node; does NOT own security baseline |
|
||||
|
||||
The `railiance-bootstrap` ansible work (harden.yml, bootstrap.yml) is
|
||||
superseded by `roles/base` and `roles/sops_agent` in this repo. Going forward,
|
||||
any security or OS-level configuration belongs here. `railiance-bootstrap` may
|
||||
consume a node that has already been converged by this repo, but must not
|
||||
re-configure items owned here.
|
||||
|
||||
### Test framework: Goss
|
||||
|
||||
[Goss](https://github.com/goss-org/goss) is a Go binary that evaluates YAML
|
||||
test files against the live node. It was chosen because:
|
||||
|
||||
- Tests and spec map one-to-one (Goss YAML IS the assertion)
|
||||
- Single binary, no Python/Ruby runtime on target host
|
||||
- Fast (runs in-process, no SSH per test)
|
||||
- Output can be TAP, JSON, or human-readable
|
||||
- Deployable via Ansible in a single task
|
||||
|
||||
### Directory layout
|
||||
|
||||
```
|
||||
spec/
|
||||
server-baseline.yaml ← authoritative target-state spec (already created)
|
||||
|
||||
goss/
|
||||
baseline.yaml ← Goss assertions (derived from spec)
|
||||
vars/
|
||||
baseline-vars.yaml ← parameterised values (ports, users, etc.)
|
||||
|
||||
ansible/
|
||||
playbooks/
|
||||
verify.yaml ← deploy Goss + run tests + fetch results
|
||||
roles/
|
||||
goss/ ← role: install binary, copy tests, run, report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### T01 — Resolve duplicate `converge` target and fix SSH check
|
||||
|
||||
```task
|
||||
id: T01
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: high
|
||||
state_hub_task_id: "892f8bb8-beff-463a-b47c-ffd9a672d065"
|
||||
```
|
||||
|
||||
- Remove redundant `converge: ansible-bootstrap` alias (caused Makefile warning)
|
||||
- Fix `sshd -T` command (requires hostkeys) → replaced with
|
||||
`grep -iE '^(PermitRootLogin|PasswordAuthentication)' /etc/ssh/sshd_config`
|
||||
|
||||
**Done when:** `make status` completes without warnings and SSH section returns
|
||||
`PermitRootLogin no` / `PasswordAuthentication no`.
|
||||
|
||||
---
|
||||
|
||||
### T02 — Finalise server baseline spec
|
||||
|
||||
```task
|
||||
id: T02
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: high
|
||||
state_hub_task_id: "293d950e-c0b3-4ae2-ac08-dcbf3fe5b114"
|
||||
```
|
||||
|
||||
Created `spec/server-baseline.yaml` covering:
|
||||
- Firewall rules (UFW, default deny, allowed ports)
|
||||
- SSH daemon settings
|
||||
- Required services and packages
|
||||
- Admin user constraints
|
||||
- Security settings (fail2ban jails, HISTCONTROL)
|
||||
|
||||
**Done when:** spec reviewed and agreed — it becomes the contract that roles
|
||||
and tests must satisfy.
|
||||
|
||||
---
|
||||
|
||||
### T03 — Implement Goss test suite
|
||||
|
||||
```task
|
||||
id: T03
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: high
|
||||
state_hub_task_id: "a34a1626-ff38-4925-a957-d94036fbded6"
|
||||
```
|
||||
|
||||
Create `goss/baseline.yaml` with Goss assertions that implement every item in
|
||||
`spec/server-baseline.yaml`. Each spec section maps to a Goss resource type:
|
||||
|
||||
| spec section | Goss resource |
|
||||
|---|---|
|
||||
| `firewall.status` | `command: ufw status` |
|
||||
| `firewall.rules` | `command: ufw status` stdout contains |
|
||||
| `ssh.*` | `file: /etc/ssh/sshd_config` contains |
|
||||
| `services` | `service:` blocks |
|
||||
| `packages` | `package:` blocks |
|
||||
| `users` | `user:` + `file: /etc/sudoers.d/admin` |
|
||||
|
||||
Example structure:
|
||||
|
||||
```yaml
|
||||
# goss/baseline.yaml
|
||||
package:
|
||||
ufw:
|
||||
installed: true
|
||||
fail2ban:
|
||||
installed: true
|
||||
|
||||
service:
|
||||
ufw:
|
||||
enabled: true
|
||||
running: true
|
||||
fail2ban:
|
||||
enabled: true
|
||||
running: true
|
||||
|
||||
file:
|
||||
/etc/ssh/sshd_config:
|
||||
exists: true
|
||||
contains:
|
||||
- /^PermitRootLogin no/
|
||||
- /^PasswordAuthentication no/
|
||||
|
||||
command:
|
||||
ufw status:
|
||||
exit-status: 0
|
||||
stdout:
|
||||
- "Status: active"
|
||||
- "22/tcp.*ALLOW"
|
||||
- "6443/tcp.*ALLOW"
|
||||
- "8472/udp.*ALLOW"
|
||||
|
||||
user:
|
||||
admin:
|
||||
exists: true
|
||||
groups:
|
||||
- sudo
|
||||
shell: /bin/bash
|
||||
```
|
||||
|
||||
**Done when:** `goss validate` passes on a freshly converged node.
|
||||
|
||||
---
|
||||
|
||||
### T04 — Ansible role and playbook for Goss
|
||||
|
||||
```task
|
||||
id: T04
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: high
|
||||
state_hub_task_id: "c072c45b-f18d-45be-b747-6d219c3f1439"
|
||||
```
|
||||
|
||||
Create `ansible/roles/goss/` with tasks that:
|
||||
1. Download the Goss binary (pinned version) to `/usr/local/bin/goss`
|
||||
2. Copy `goss/baseline.yaml` to `/etc/goss/baseline.yaml`
|
||||
3. Run `goss -g /etc/goss/baseline.yaml validate --format tap`
|
||||
4. Fetch the TAP output back to the control node as `reports/goss-<host>-<date>.tap`
|
||||
5. Fail the play if any test fails (`rc != 0`)
|
||||
|
||||
Create `ansible/playbooks/verify.yaml`:
|
||||
```yaml
|
||||
- hosts: all
|
||||
become: true
|
||||
roles:
|
||||
- role: goss
|
||||
```
|
||||
|
||||
**Done when:** `ansible-playbook ansible/playbooks/verify.yaml` exits 0 on a
|
||||
clean node, non-zero on a deliberately broken one (test with a manual config change).
|
||||
|
||||
---
|
||||
|
||||
### T05 — Add `make verify` target
|
||||
|
||||
```task
|
||||
id: T05
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: medium
|
||||
state_hub_task_id: "a8100b8e-aed0-4bb4-a0dc-a6bdf3938b8d"
|
||||
```
|
||||
|
||||
Add to Makefile:
|
||||
|
||||
```makefile
|
||||
verify: ## Run Goss test suite against all hosts — exits non-zero on failure
|
||||
cd $(ANS_DIR) && ansible-playbook playbooks/verify.yaml -u $(SSH_USER)
|
||||
```
|
||||
|
||||
Also update `make status` to print a summary line ("All assertions passed" /
|
||||
"N assertions FAILED") rather than raw shell output.
|
||||
|
||||
**Done when:** `make verify` exits 0 on a good node, non-zero on a bad one.
|
||||
|
||||
---
|
||||
|
||||
### T06 — Write ADR: railiance-hosts vs railiance-bootstrap boundary
|
||||
|
||||
```task
|
||||
id: T06
|
||||
status: done
|
||||
completed: "2026-03-09"
|
||||
priority: medium
|
||||
state_hub_task_id: "c3d98022-638d-4dcb-bdc7-a9501e1b6cd9"
|
||||
```
|
||||
|
||||
Create `docs/adr/ADR-002-repo-boundary-hosts-vs-bootstrap.md` documenting:
|
||||
|
||||
- What `railiance-hosts` owns (OS baseline, security, spec, tests)
|
||||
- What `railiance-bootstrap` owns (Kubernetes/app layer, consumes a converged node)
|
||||
- Decision: any item present in `spec/server-baseline.yaml` must NOT be
|
||||
managed by `railiance-bootstrap`
|
||||
- Migration note: superseded bootstrap.yml / harden.yml in that repo
|
||||
|
||||
**Done when:** ADR written and merged.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Goss documentation: https://github.com/goss-org/goss
|
||||
- Server spec: `spec/server-baseline.yaml`
|
||||
- Bootstrap workplan: `workplans/RAIL-HO-WP-0001-hosteurope-bootstrap.md`
|
||||
|
|
@ -0,0 +1,276 @@
|
|||
---
|
||||
id: RAIL-HO-WP-0003
|
||||
type: workplan
|
||||
title: "Railiance 5-Repo Stack Restructure"
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
state_hub_workstream_id: "3ae0afc5-13f2-4e6c-aea7-1c1fb9f1ab81"
|
||||
created: "2026-03-09"
|
||||
updated: "2026-03-10"
|
||||
completed: "2026-03-10"
|
||||
---
|
||||
|
||||
# Railiance 5-Repo Stack Restructure
|
||||
|
||||
## Goal
|
||||
|
||||
Restructure the railiance tooling to align one-to-one with the OAS Stack
|
||||
dimensions (S1–S5). Each repo owns exactly one stack layer.
|
||||
|
||||
| OAS Level | Repo | Stack Responsibility |
|
||||
|-----------|------|---------------------|
|
||||
| S1 Infrastructure Substrate | `railiance-infra` | OS, Ansible roles, Goss spec, Terraform, inventory |
|
||||
| S2 Cluster Runtime | `railiance-cluster` | k3s, Helm, ingress, CNI, operators |
|
||||
| S3 Platform Services | `railiance-platform` | PostgreSQL HA, object storage, secret mgmt, identity |
|
||||
| S4 Developer Enablement | `railiance-enablement` | CI/CD, developer portal, platform templates |
|
||||
| S5 Workloads & Experience | `railiance-apps` | App Helm releases, Gitea, coulomb services |
|
||||
|
||||
## Motivation
|
||||
|
||||
The original names (`railiance-hosts`, `railiance-bootstrap`) do not signal
|
||||
their stack layer. The new names make the architecture self-documenting and
|
||||
enforce the OAS P1 principle (Separation of Dimensions).
|
||||
|
||||
## Boundary rule (new)
|
||||
|
||||
> **Every file and playbook belongs in exactly one repo, determined by the
|
||||
> OAS Stack level of the concern it addresses.**
|
||||
|
||||
ADR-003 (written in T01) supersedes ADR-002.
|
||||
|
||||
## Content relocations from railiance-cluster
|
||||
|
||||
The following items currently in railiance-bootstrap (→ railiance-cluster)
|
||||
belong at other layers and must move:
|
||||
|
||||
| Item | Current location | Destination | Layer |
|
||||
|------|-----------------|-------------|-------|
|
||||
| `cloudinit/user-data.yaml` | railiance-cluster | railiance-infra | S1 |
|
||||
| `tools/cmd/railiance-plan-host` | railiance-cluster | railiance-infra | S1 |
|
||||
| `tools/cmd/railiance-backup` | railiance-cluster | railiance-platform | S3 |
|
||||
|
||||
Everything else in railiance-cluster (k3s playbook, Helm, smoke tests,
|
||||
preflight, doctor, seed_node) is correctly placed at S2.
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### T01 — Write ADR-003: Railiance 5-Repo Stack Architecture
|
||||
|
||||
```task
|
||||
id: T01
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "4ff2c35f-7cf8-46a3-a5e3-366bb0e193c7"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
Create `docs/adr/ADR-003-railiance-5repo-stack-architecture.md` in
|
||||
railiance-infra (formerly railiance-hosts) covering:
|
||||
|
||||
- The five-repo layout and OAS S1–S5 mapping
|
||||
- Naming rationale (`railiance-{infra,cluster,platform,enablement,apps}`)
|
||||
- Boundary rule: one repo per stack layer
|
||||
- Content relocation table (items moving from railiance-cluster)
|
||||
- Migration note: supersedes ADR-002
|
||||
- Invalidation note in ADR-002 header pointing to ADR-003
|
||||
|
||||
**Done when:** ADR-003 written; ADR-002 updated with superseded status and
|
||||
forward reference.
|
||||
|
||||
---
|
||||
|
||||
### T02 — Rename railiance-hosts → railiance-infra
|
||||
|
||||
```task
|
||||
id: T02
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "5b29d8f8-ab9b-4d8c-a2a0-87918eac5cc4"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
**Local:**
|
||||
```bash
|
||||
mv ~/railiance-hosts ~/railiance-infra
|
||||
cd ~/railiance-infra
|
||||
# Update any self-referencing paths (CLAUDE.md, Makefile REPO var, etc.)
|
||||
grep -r "railiance-hosts" . --include="*.md" --include="*.sh" --include="Makefile"
|
||||
```
|
||||
|
||||
**Gitea remote:**
|
||||
Rename the repository in Gitea UI or via API, then update the local remote:
|
||||
```bash
|
||||
git remote set-url origin https://gitea.example.com/coulomb/railiance-infra.git
|
||||
```
|
||||
|
||||
**State Hub:**
|
||||
```
|
||||
rename_domain is not the right tool — use the API directly or update
|
||||
the ManagedRepo slug via the admin API once rename_repo is available.
|
||||
```
|
||||
Current slug: `railiance-hosts` → new slug: `railiance-infra`
|
||||
|
||||
**CLAUDE.md:** Update repo name references.
|
||||
|
||||
**Workplan files:** Update `repo:` frontmatter field in all workplans.
|
||||
|
||||
**Done when:** `ls ~/railiance-infra` works; git remote points to renamed
|
||||
Gitea repo; state hub slug updated.
|
||||
|
||||
---
|
||||
|
||||
### T03 — Rename railiance-bootstrap → railiance-cluster
|
||||
|
||||
```task
|
||||
id: T03
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "caf3c22e-3d83-46a3-9bd8-23a5d9021f72"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
Same procedure as T02 for railiance-bootstrap → railiance-cluster:
|
||||
|
||||
```bash
|
||||
mv ~/railiance-bootstrap ~/railiance-cluster
|
||||
```
|
||||
|
||||
Update:
|
||||
- Gitea remote
|
||||
- State hub slug (`railiance-bootstrap` → `railiance-cluster`)
|
||||
- CLAUDE.md repo name references
|
||||
- Workplan `repo:` frontmatter fields
|
||||
|
||||
**Done when:** rename complete; all cross-references updated.
|
||||
|
||||
---
|
||||
|
||||
### T04 — Relocate misplaced content to correct repos
|
||||
|
||||
```task
|
||||
id: T04
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "35d3b599-8325-472c-abd0-2c4939f1f8cd"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
After T02 and T03 are complete, move items to their correct layer repos:
|
||||
|
||||
**→ railiance-infra (S1):**
|
||||
```bash
|
||||
# From railiance-cluster:
|
||||
git mv cloudinit/user-data.yaml /path/to/railiance-infra/cloudinit/
|
||||
git mv tools/cmd/railiance-plan-host /path/to/railiance-infra/tools/cmd/
|
||||
```
|
||||
|
||||
**→ railiance-platform (S3):**
|
||||
```bash
|
||||
# From railiance-cluster (after T05 creates the repo):
|
||||
git mv tools/cmd/railiance-backup /path/to/railiance-platform/tools/cmd/
|
||||
```
|
||||
|
||||
Commit in both source and destination repos. Add a tombstone stub in the
|
||||
source location pointing to the new home.
|
||||
|
||||
**Done when:** all three items committed in their destination repos; stubs
|
||||
in place in railiance-cluster.
|
||||
|
||||
---
|
||||
|
||||
### T05 — Create and scaffold three new repos
|
||||
|
||||
```task
|
||||
id: T05
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "ba48bdca-a3e5-4efa-8bd1-55077e92a787"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
Create `railiance-platform`, `railiance-enablement`, `railiance-apps` as
|
||||
git repos with minimal scaffolding:
|
||||
|
||||
For each repo:
|
||||
1. `git init ~/railiance-{platform,enablement,apps}`
|
||||
2. Create `CLAUDE.md` using the Custodian project template
|
||||
3. Create `workplans/` directory with a `.gitkeep`
|
||||
4. Create a `Makefile` stub with `help` target
|
||||
5. Create `README.md` describing the layer's scope
|
||||
6. Initial commit
|
||||
|
||||
**Done when:** all three repos exist locally with at least CLAUDE.md,
|
||||
workplans/, and a Makefile stub.
|
||||
|
||||
---
|
||||
|
||||
### T06 — Register all five repos in the state hub
|
||||
|
||||
```task
|
||||
id: T06
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "a5310b9a-665c-4341-92bf-e51db65c2388"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
After T02, T03, T05:
|
||||
|
||||
```
|
||||
register_repo(domain_slug="railiance", name="railiance-infra",
|
||||
slug="railiance-infra", local_path="/home/worsch/railiance-infra")
|
||||
|
||||
register_repo(domain_slug="railiance", name="railiance-cluster",
|
||||
slug="railiance-cluster", local_path="/home/worsch/railiance-cluster")
|
||||
|
||||
register_repo(domain_slug="railiance", name="railiance-platform",
|
||||
slug="railiance-platform", local_path="/home/worsch/railiance-platform")
|
||||
|
||||
register_repo(domain_slug="railiance", name="railiance-enablement",
|
||||
slug="railiance-enablement", local_path="/home/worsch/railiance-enablement")
|
||||
|
||||
register_repo(domain_slug="railiance", name="railiance-apps",
|
||||
slug="railiance-apps", local_path="/home/worsch/railiance-apps")
|
||||
```
|
||||
|
||||
The old slugs (`railiance-hosts`, `railiance-bootstrap`) can be archived
|
||||
or left as stale records until a rename-repo API endpoint exists.
|
||||
|
||||
**Done when:** all five repos appear in `list_domain_repos("railiance")`.
|
||||
|
||||
---
|
||||
|
||||
### T07 — Resolve pending railiance-apps decision
|
||||
|
||||
```task
|
||||
id: T07
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "2aaebc21-da59-456e-abdf-8cf03fd82e49"
|
||||
completed: "2026-03-10"
|
||||
```
|
||||
|
||||
Decision `7cddead6` ("Create railiance-apps repo for application-layer
|
||||
deployments") is now resolved by this workplan. Resolve it in the state hub:
|
||||
|
||||
```
|
||||
resolve_decision(decision_id="7cddead6-79dd-4cca-a6d5-6512369603bb",
|
||||
rationale="railiance-apps created as S5 layer of the 5-repo OAS stack.",
|
||||
decided_by="Bernd Worsch")
|
||||
```
|
||||
|
||||
**Done when:** decision marked resolved.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- OAS Standard: `canon/standards/orthogonal-architecture_v1.0.md`
|
||||
- ADR-002 (superseded): `docs/adr/ADR-002-repo-boundary-hosts-vs-bootstrap.md`
|
||||
- State Hub workstream: `3ae0afc5-13f2-4e6c-aea7-1c1fb9f1ab81`
|
||||
- Pending decision (railiance-apps): `7cddead6-79dd-4cca-a6d5-6512369603bb`
|
||||
|
|
@ -0,0 +1,546 @@
|
|||
---
|
||||
id: RAIL-HO-WP-0004
|
||||
type: workplan
|
||||
title: "Railiance Production Readiness — Automated, Reproducible Stack"
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: finished
|
||||
owner: worsch
|
||||
topic_slug: railiance
|
||||
created: "2026-03-26"
|
||||
updated: "2026-05-22"
|
||||
completed: "2026-05-22"
|
||||
supersedes: RAIL-PL-WP-0001
|
||||
state_hub_workstream_id: "cee078e9-b18c-4f84-8a8a-6f27c2f9f407"
|
||||
---
|
||||
|
||||
# Railiance Production Readiness — Automated, Reproducible Stack
|
||||
|
||||
## Goal
|
||||
|
||||
Make the Railiance cluster fully reproducible from a clean server — zero manual
|
||||
interventions required. Then migrate operational workloads (state-hub,
|
||||
activity-core) from the workstation and ad-hoc CoulombCore setup onto the
|
||||
cluster, with confidence that a rebuild is always a `make deploy-stack` away.
|
||||
|
||||
This workplan supersedes `RAIL-PL-WP-0001` (which targeted Bitnami
|
||||
postgresql-ha; cnpg is now the deployed and active operator).
|
||||
|
||||
## Why now
|
||||
|
||||
Three forcing functions are converging:
|
||||
1. **CoulombCore hardening applied manually** (swapfile, nproc limits, systemd
|
||||
slice) after INC-002. These must be in Ansible before the next node
|
||||
rebuild — or the next operator overstep will repeat the incident.
|
||||
2. **cnpg is deployed** (cnpg-system namespace, databases namespace active).
|
||||
WP-0001 targeted Bitnami postgresql-ha which is now stale. A clean platform
|
||||
baseline must match reality.
|
||||
3. **State-hub and activity-core live on the workstation** — fragile,
|
||||
non-redundant, not self-documenting. Moving them to the cluster is the last
|
||||
step to making Railiance the durable operational home it was designed to be.
|
||||
|
||||
## Current deployed state (reference snapshot 2026-03-26)
|
||||
|
||||
| Component | Namespace | Manager | Boundary status |
|
||||
|-----------|-----------|---------|-----------------|
|
||||
| cert-manager | cert-manager | Helm S2 | ✓ correct |
|
||||
| CloudNative PG operator | cnpg-system | Helm S2 | boundary violation: operator is S3 concern |
|
||||
| nginx ingress | ingress-nginx | Helm S2 | ✓ correct |
|
||||
| Gitea 12.5.0 | gitea | Helm S2 | boundary violation: should be S5 |
|
||||
| ArgoCD | argocd | kubectl S2 | boundary violation: S4 concern |
|
||||
| SSO/MFA stack | mfa + sso | ? | per net-kingdom |
|
||||
| cnpg databases | databases | kubectl S3 | ✓ correct layer, no cluster defined yet |
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1 — Ansible-codify server hardening (S1)
|
||||
|
||||
All manual CoulombCore interventions from INC-002 must become Ansible roles
|
||||
so they survive node rebuild. No more drift between code and reality.
|
||||
|
||||
### Phase 2 — S3 platform baseline with cnpg (supersedes WP-0001)
|
||||
|
||||
Define a cnpg `Cluster` resource for the Gitea database in `railiance-platform`.
|
||||
Migrate Gitea from its built-in postgresql-ha subchart to this cluster.
|
||||
Codify Valkey as a standalone S3 Helm release.
|
||||
|
||||
### Phase 3 — S2 boundary cleanup
|
||||
|
||||
Move `gitea-values.sops.yaml` from `railiance-cluster` to `railiance-apps`.
|
||||
Document remaining boundary violations (cnpg operator in S2, ArgoCD in S2)
|
||||
and create forward-dated migration stubs.
|
||||
|
||||
### Phase 4 — Git operations from CoulombCore
|
||||
|
||||
CoulombCore cannot push to Gitea via HTTP (NodePort hairpin). Configure SSH
|
||||
remote so all on-cluster git operations use SSH.
|
||||
|
||||
### Phase 5 — Automated stack deploy
|
||||
|
||||
Write a `deploy-stack` target (or script) that converges S1→S5 in dependency
|
||||
order. The goal: a new operator can onboard a server and reach a working
|
||||
cluster with one command sequence.
|
||||
|
||||
### Phase 6 — Migrate operational workloads (S5)
|
||||
|
||||
Deploy state-hub and activity-core to the cluster. This is the payoff phase —
|
||||
the cluster becomes the operational home, not the workstation.
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- railiance-cluster converged and all S2 workplans done (they are: ✓)
|
||||
- Gitea operational (it is: ✓, gitea namespace running)
|
||||
- ops-bridge state-hub tunnel active (bridge up state-hub-coulombcore)
|
||||
- Active backup before any phase touching live data (make backup in railiance-cluster)
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### T01 — Ansible: swapfile role for CoulombCore
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T01
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7c586940-f7b8-4e55-b1d6-72eba6a675b7"
|
||||
```
|
||||
|
||||
Create an Ansible role `swapfile` (or extend `roles/base`) that provisions the
|
||||
4 GB swapfile applied manually after INC-002.
|
||||
|
||||
Desired state:
|
||||
```yaml
|
||||
# inventory/host_vars/coulombcore.yml (or group_vars)
|
||||
swap_size_gb: 4
|
||||
swap_swappiness: 10
|
||||
```
|
||||
|
||||
Role tasks:
|
||||
1. Check `/swapfile` existence + correct size (fallocate idempotent)
|
||||
2. `chmod 600 /swapfile`, `mkswap`, `swapon` if not already active
|
||||
3. Ensure `/etc/fstab` entry present
|
||||
4. Set `vm.swappiness=10` via `sysctl` module (persist in `/etc/sysctl.d/`)
|
||||
|
||||
**Convergence pattern:** Ansible is not installed on the workstation. Run convergence
|
||||
directly on CoulombCore (local Ansible, connection=local):
|
||||
```bash
|
||||
ssh -i ~/.ssh/id_ops tegwick@92.205.130.254 \
|
||||
'cd ~/railiance-infra && git pull && ansible-playbook ansible/playbooks/bootstrap.yaml -c local -u tegwick --become -l CoulombCore'
|
||||
```
|
||||
|
||||
**Done when:** Convergence runs without errors; `free -h` on CoulombCore shows
|
||||
4 GB swap; Goss verify passes.
|
||||
|
||||
---
|
||||
|
||||
### T02 — Ansible: nproc limits and systemd user slice
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T02
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "42f1f02b-0d8b-432c-8bc8-4930417e15dd"
|
||||
```
|
||||
|
||||
Codify the PAM nproc limits and systemd user slice hardening applied after
|
||||
INC-002 into Ansible (role `security` or a new `resource-limits` role).
|
||||
|
||||
Desired state:
|
||||
```yaml
|
||||
nproc_soft: 512
|
||||
nproc_hard: 1024
|
||||
user_memory_max: "1500M"
|
||||
user_memory_swap_max: "512M"
|
||||
```
|
||||
|
||||
Tasks:
|
||||
1. Template `/etc/security/limits.conf` entry for tegwick (nproc soft/hard)
|
||||
2. Create `/etc/systemd/system/user-1000.slice.d/limits.conf` via template
|
||||
3. `systemctl daemon-reload` handler
|
||||
|
||||
**Done when:** `make converge` idempotent; `cat /proc/<tegwick-pid>/limits`
|
||||
reflects caps; `make verify` passes; Goss test for nproc limit added.
|
||||
|
||||
---
|
||||
|
||||
### T03 — Define Gitea cnpg database cluster in railiance-platform
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T03
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "8e8cff04-96c6-4386-8caa-b0586114a49d"
|
||||
```
|
||||
|
||||
Mark `RAIL-PL-WP-0001` as superseded (update its status field). Then define
|
||||
the Gitea database cluster using CloudNative PG in `railiance-platform`.
|
||||
|
||||
Files to create in `railiance-platform/`:
|
||||
```
|
||||
helm/gitea-db-cluster.yaml # cnpg Cluster manifest (SOPS-encrypted secrets inline or ref)
|
||||
Makefile targets: db-deploy, db-status, db-shell
|
||||
```
|
||||
|
||||
Cluster manifest skeleton:
|
||||
```yaml
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: gitea-db
|
||||
namespace: databases
|
||||
spec:
|
||||
instances: 1 # single-node to start; bump to 3 when RAM allows
|
||||
postgresql:
|
||||
version: "16"
|
||||
storage:
|
||||
size: 10Gi
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: gitea
|
||||
owner: gitea
|
||||
secret:
|
||||
name: gitea-db-credentials # k8s Secret (SOPS-managed)
|
||||
```
|
||||
|
||||
Add `make db-deploy` target that applies the manifest to the `databases`
|
||||
namespace. Add `make db-status` that shows cluster health via `kubectl cnpg
|
||||
status`.
|
||||
|
||||
**Done when:** `make db-deploy` succeeds; `kubectl get cluster -n databases`
|
||||
shows `gitea-db` in `Cluster in healthy state`; credentials secret present.
|
||||
|
||||
---
|
||||
|
||||
### T04 — Migrate Gitea to external cnpg database
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T04
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "4f4196b5-4d84-4648-b470-e6941444ea46"
|
||||
```
|
||||
|
||||
**Pre-condition:** T03 done and gitea-db cluster healthy.
|
||||
|
||||
Migration steps (execute from CoulombCore with kubectl access):
|
||||
|
||||
1. Backup: `make backup` in railiance-cluster — verify success.
|
||||
2. Dump current Gitea DB:
|
||||
```bash
|
||||
kubectl exec -n gitea deploy/gitea -- \
|
||||
pg_dump -h localhost -U gitea gitea > /tmp/gitea-dump.sql
|
||||
```
|
||||
(Gitea's built-in postgresql-ha is at localhost within the pod)
|
||||
3. Restore into cnpg cluster:
|
||||
```bash
|
||||
kubectl exec -n databases gitea-db-1 -- \
|
||||
psql -U gitea gitea < /tmp/gitea-dump.sql
|
||||
```
|
||||
4. Update Gitea Helm values to disable subchart and point to cnpg:
|
||||
```yaml
|
||||
postgresql-ha:
|
||||
enabled: false
|
||||
redis-cluster:
|
||||
enabled: false # Valkey handled in T06
|
||||
gitea:
|
||||
config:
|
||||
database:
|
||||
DB_TYPE: postgres
|
||||
HOST: gitea-db-rw.databases.svc.cluster.local:5432
|
||||
NAME: gitea
|
||||
USER: gitea
|
||||
PASSWD: <from cnpg secret>
|
||||
```
|
||||
5. `helm upgrade gitea` — verify login and all repos intact.
|
||||
6. Confirm old postgresql-ha pods are terminated.
|
||||
|
||||
**Done when:** Gitea login works; all repos accessible; no postgresql-ha pods
|
||||
running; `kubectl cnpg status gitea-db -n databases` healthy.
|
||||
|
||||
---
|
||||
|
||||
### T05 — Codify Valkey as standalone S3 asset
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T05
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "36c66ceb-ebc9-425a-a329-c37496278c6b"
|
||||
```
|
||||
|
||||
Create `railiance-platform/helm/valkey-values.sops.yaml` and deploy Valkey
|
||||
as a standalone Helm release in the `platform` namespace (independent of Gitea
|
||||
subchart).
|
||||
|
||||
```yaml
|
||||
# helm/valkey-values.sops.yaml
|
||||
auth:
|
||||
enabled: true
|
||||
password: ENC[age,...]
|
||||
replica:
|
||||
replicaCount: 1
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 2Gi
|
||||
```
|
||||
|
||||
Add `make valkey-deploy` and `make valkey-status` to `railiance-platform/Makefile`.
|
||||
|
||||
Update Gitea Helm values to point to standalone Valkey:
|
||||
```yaml
|
||||
redis-cluster:
|
||||
enabled: false
|
||||
gitea:
|
||||
config:
|
||||
cache:
|
||||
ADAPTER: redis
|
||||
HOST: redis://:password@valkey.platform.svc.cluster.local:6379/0
|
||||
session:
|
||||
PROVIDER: redis
|
||||
PROVIDER_CONFIG: redis://:password@valkey.platform.svc.cluster.local:6379/1
|
||||
queue:
|
||||
TYPE: redis
|
||||
CONN_STR: redis://:password@valkey.platform.svc.cluster.local:6379/2
|
||||
```
|
||||
|
||||
**Done when:** `make valkey-deploy` succeeds; Gitea session/cache operational
|
||||
on standalone Valkey; no redis subchart pods running.
|
||||
|
||||
---
|
||||
|
||||
### T06 — Move Gitea Helm values to railiance-apps (boundary fix)
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T06
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "6d8323b3-e842-4dc1-9a12-2b153b2afcce"
|
||||
```
|
||||
|
||||
**Pre-condition:** T04 done (Gitea on external DB; Helm values updated).
|
||||
|
||||
```bash
|
||||
# In railiance-cluster:
|
||||
git mv helm/gitea-values.sops.yaml ../railiance-apps/helm/gitea-values.sops.yaml
|
||||
```
|
||||
|
||||
Add to `railiance-apps/Makefile`:
|
||||
```makefile
|
||||
gitea-deploy: ## Deploy / upgrade Gitea (S5 workload)
|
||||
helm upgrade --install gitea gitea-charts/gitea \
|
||||
-f <(sops -d helm/gitea-values.sops.yaml) \
|
||||
--namespace gitea --create-namespace
|
||||
|
||||
gitea-status: ## Check Gitea health
|
||||
kubectl get pods -n gitea
|
||||
kubectl cnpg status gitea-db -n databases
|
||||
```
|
||||
|
||||
Add tombstone in `railiance-cluster/helm/MOVED.md`:
|
||||
```
|
||||
gitea-values.sops.yaml → railiance-apps/helm/ (2026-03-xx, RAIL-HO-WP-0004-T06)
|
||||
```
|
||||
|
||||
Update `railiance-cluster/SCOPE.md` to remove Gitea boundary violation note.
|
||||
Update `railiance-apps/SCOPE.md` Current State to reflect resolved violation.
|
||||
|
||||
**Done when:** `make gitea-deploy` from railiance-apps converges correctly;
|
||||
Gitea operational; tombstone in place in railiance-cluster.
|
||||
|
||||
---
|
||||
|
||||
### T07 — SSH remote for git operations from CoulombCore
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T07
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "3d76754d-2dc0-4fe5-8bf2-c74d77cebe36"
|
||||
```
|
||||
|
||||
CoulombCore cannot push to Gitea via HTTP (NodePort hairpin, no stored
|
||||
credentials). Fix by configuring SSH-based remotes for all repos on CoulombCore.
|
||||
|
||||
Steps:
|
||||
1. Generate an SSH key for the `tegwick` user on CoulombCore if not present:
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -C "tegwick@coulombcore" -f ~/.ssh/id_ed25519_gitea
|
||||
```
|
||||
2. Add the public key to Gitea (coulomb user or a dedicated `coulombcore` bot
|
||||
account via Gitea admin UI or API).
|
||||
3. Add SSH config on CoulombCore:
|
||||
```
|
||||
# ~/.ssh/config
|
||||
Host gitea-local
|
||||
HostName localhost
|
||||
Port <Gitea SSH NodePort>
|
||||
User git
|
||||
IdentityFile ~/.ssh/id_ed25519_gitea
|
||||
```
|
||||
Note: Gitea exposes SSH on a NodePort (check current value: `kubectl get svc -n gitea`).
|
||||
4. Update remotes for all repos on CoulombCore:
|
||||
```bash
|
||||
git remote set-url origin ssh://git@gitea-local/coulomb/<repo>.git
|
||||
```
|
||||
5. Test: `git push origin main` from a repo on CoulombCore.
|
||||
|
||||
Codify the SSH key deployment step into Ansible
|
||||
(`roles/base` or `roles/git-access`): ensure the key is present and the SSH
|
||||
config block is templated.
|
||||
|
||||
**Done when:** `git push` from CoulombCore to Gitea succeeds over SSH without
|
||||
prompts; Ansible role deploys the key idempotently.
|
||||
|
||||
---
|
||||
|
||||
### T08 — Automated stack deploy documentation + Makefile
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T08
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "b076e540-2d81-4be8-a454-61cfd329bb05"
|
||||
```
|
||||
|
||||
Write `railiance-infra/docs/deploy-stack.md` — the operator runbook for
|
||||
standing up the full Railiance stack from scratch. This is the canonical
|
||||
"I have a clean server, now what?" reference.
|
||||
|
||||
Standard sequence:
|
||||
```
|
||||
S1: make tf-apply && make converge && make verify (railiance-infra)
|
||||
S2: make converge && make smoke (railiance-cluster)
|
||||
S3: make db-deploy && make valkey-deploy (railiance-platform)
|
||||
S4: (ArgoCD already at cluster level; no S4 workplan yet)
|
||||
S5: make gitea-deploy (railiance-apps)
|
||||
make state-hub-deploy (railiance-apps, T09)
|
||||
make activity-core-deploy (railiance-apps, T10)
|
||||
```
|
||||
|
||||
Add a `make deploy-stack` target in `railiance-infra/Makefile` that prints
|
||||
the ordered sequence with per-step instructions (not a single runaway script —
|
||||
operator confirms each layer before proceeding).
|
||||
|
||||
Document:
|
||||
- Pre-conditions checklist (Hetzner/HostEurope creds, age key, SOPS key)
|
||||
- State Hub tunnel bring-up (ops-bridge)
|
||||
- Recovery runbook pointer (INC-002 pattern)
|
||||
|
||||
**Done when:** `docs/deploy-stack.md` accurate and reviewed; `make deploy-stack`
|
||||
prints the sequence; a new operator could follow it end-to-end without prior
|
||||
context.
|
||||
|
||||
---
|
||||
|
||||
### T09 — Deploy state-hub to railiance01 as cluster primary (S5)
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T09
|
||||
status: cancel
|
||||
priority: medium
|
||||
state_hub_task_id: "d2afe78a-eb51-4ce9-b332-f181323d2370"
|
||||
needs_human: false
|
||||
intervention_note: "Superseded by CUST-WP-0038 State Hub Full ThreePhoenix HA Migration, which owns the hostname, registry, private exposure model, HA database/storage strategy, restore drills, and production data migration approval."
|
||||
```
|
||||
|
||||
**Superseded 2026-05-22:** State Hub cluster migration is now owned by
|
||||
`CUST-WP-0038` (`State Hub Full ThreePhoenix HA Migration`). This Railiance
|
||||
task is closed as superseded so `RAIL-HO-WP-0004` can finish without splitting
|
||||
State Hub HA/data-migration governance across two workplans.
|
||||
|
||||
**Pre-condition:** T04 done (cnpg Gitea DB working); T08 done (deploy sequence
|
||||
documented). Custodian-side safety gate `CUST-WP-0011-T01` must have passed:
|
||||
a fresh WSL2 State Hub backup restore drill with matching row counts.
|
||||
|
||||
State-hub needs a PostgreSQL database — use a cnpg cluster in `databases`
|
||||
namespace. This is the pragmatic railiance01 migration path; full multi-node
|
||||
ThreePhoenix HA remains a separate Custodian follow-up (`CUST-WP-0038`).
|
||||
|
||||
Steps:
|
||||
1. Define `state-hub-db` cnpg Cluster in `railiance-platform` (same pattern as T03).
|
||||
2. Create a container image for state-hub (Dockerfile in `the-custodian/state-hub/`).
|
||||
3. Push image to Gitea's container registry (or ghcr.io as interim).
|
||||
4. Write Helm chart or plain manifests in `railiance-apps/apps/state-hub/`:
|
||||
- Deployment (state-hub API, port 8000)
|
||||
- Service + Ingress (https://state-hub.<domain>)
|
||||
- ConfigMap for environment (DB URL, etc.)
|
||||
- Secret for DB credentials (SOPS-managed)
|
||||
5. Deploy empty State Hub and run Alembic migrations in-cluster.
|
||||
6. Restore a copy of WSL2 data into the cnpg cluster and compare table counts
|
||||
while the workstation remains the source of truth.
|
||||
7. With explicit human approval, freeze workstation writes, take a final dump,
|
||||
restore it to the cluster, and make railiance01 the primary endpoint.
|
||||
8. Update ops-bridge tunnel targets or MCP `API_BASE` if the State Hub URL changes.
|
||||
9. Update operator instructions to describe cluster primary plus WSL2 fallback.
|
||||
|
||||
**Done when:** the private State Hub endpoint returns healthy, MCP tools work
|
||||
against the cluster-backed API, and WSL2 is retained as documented fallback.
|
||||
Permanent WSL2 retirement is out of scope here and requires a later explicit
|
||||
approval after stabilisation.
|
||||
|
||||
---
|
||||
|
||||
### T10 — Deploy activity-core to cluster (S5)
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T10
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "34d73215-f016-4750-8da5-69f82d63d619"
|
||||
needs_human: false
|
||||
intervention_note: "Deployed activity-core as a K3s production service on railiance01 on 2026-05-22. API/worker/event-router running; health endpoint OK; migrations and sync jobs completed."
|
||||
```
|
||||
|
||||
**Completion note 2026-05-22:** activity-core was deployed to the cluster
|
||||
independently of the State Hub HA migration. The original T09 dependency is
|
||||
superseded along with T09; ongoing State Hub migration work is tracked in
|
||||
`CUST-WP-0038`.
|
||||
|
||||
Activity-core is the Rails/Go/other application running on CoulombCore ad-hoc.
|
||||
This task packages and deploys it as a proper S5 workload.
|
||||
|
||||
Steps:
|
||||
1. Verify activity-core has a working Dockerfile (check repo).
|
||||
2. Define a cnpg database cluster for activity-core in `railiance-platform`
|
||||
(if it uses postgres).
|
||||
3. Write Helm chart / manifests in `railiance-apps/apps/activity-core/`.
|
||||
4. Migrate any existing data from the ad-hoc CoulombCore deployment.
|
||||
5. Add to `railiance-apps/Makefile`:
|
||||
```makefile
|
||||
activity-core-deploy: ## Deploy activity-core to cluster
|
||||
activity-core-status: ## Check activity-core health
|
||||
```
|
||||
6. Remove or archive the ad-hoc CoulombCore deployment.
|
||||
|
||||
**Done when:** Activity-core accessible at its cluster URL; no ad-hoc process
|
||||
remaining on CoulombCore; all prior functionality intact.
|
||||
|
||||
---
|
||||
|
||||
## Phasing and dependencies
|
||||
|
||||
```
|
||||
T01 (swap) ─┐
|
||||
T02 (nproc) ─┴─ independent, can parallelize
|
||||
|
||||
T03 (cnpg cluster def) ──► T04 (migrate Gitea DB) ──► T05 (Valkey standalone) ──► T06 (move Gitea to S5)
|
||||
|
||||
T07 (SSH remotes) ─ independent, unblock early
|
||||
|
||||
T08 (deploy docs) ─ can be written in parallel with T03-T06
|
||||
|
||||
T09 (state-hub on cluster) ─ superseded by CUST-WP-0038
|
||||
T10 (activity-core) ─ completed independently on 2026-05-22
|
||||
```
|
||||
|
||||
Completed order: T07 → T01+T02 → T03 → T04 → T05 → T06 → T08 → T10;
|
||||
T09 superseded by `CUST-WP-0038`.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-003 (OAS boundary rules): `railiance-infra/docs/adr/ADR-003-railiance-5repo-stack-architecture.md`
|
||||
- ADR-004 (connectivity-first): `the-custodian/canon/architecture/adr-004-connectivity-first-network-posture.md`
|
||||
- INC-002 (overload incident): `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
|
||||
- Superseded: `railiance-platform/workplans/RAIL-PL-WP-0001-platform-baseline.md`
|
||||
- ops-bridge runbook: `ops-bridge/docs/`
|
||||
Loading…
Add table
Add a link
Reference in a new issue