railiance-infra/docs/deploy-stack.md

# Railiance Stack — Full Deploy Runbook

> **When to use this:** Starting from a bare server (post-OS install) or rebuilding
> after a catastrophic failure. For day-to-day operations use the individual layer
> repos. See ADR-003 for layer boundaries and ADR-004 for connectivity posture.

## Pre-conditions checklist

Before starting, verify you have:

- [ ] SSH access to the target server (COULOMBCORE: 92.205.130.254, user: tegwick, key: `~/.ssh/id_ops`)
- [ ] SOPS age private key available (`~/.config/sops/age/keys.txt` or `SOPS_AGE_KEY` env)
- [ ] ops-bridge running on the workstation (needed for state hub MCP): `make mcp-http` in `~/the-custodian/state-hub/`
- [ ] Forgejo accessible (git pull on remote): SSH via `forgejo-remote:coulomb/<repo>.git`
- [ ] Gitea on coulombcore remains read-only fallback only (`gitea-remote` for emergency rollback)
- [ ] If re-provisioning from scratch: Hetzner/HostEurope API credentials decryptable via SOPS

---

## S1 — Infrastructure Substrate (`railiance-infra`)

```bash
# On workstation
cd ~/railiance-infra

# Provision server (skip if server already exists)
make tf-plan        # review Terraform plan
make tf-apply       # create/update server

# Converge OS baseline
# NOTE: Ansible runs locally on CoulombCore (workstation has no Ansible installed)
ssh -i ~/.ssh/id_ops tegwick@92.205.130.254 \
  'cd ~/railiance-infra && git pull && \
   cd ansible && ansible-playbook playbooks/bootstrap.yaml \
     -c local --become -l CoulombCore'

# Verify OS baseline
make verify
```

**Checkpoint:** UFW active, fail2ban running, swap enabled, nproc limits in place,
SOPS/age installed.

---

## S2 — Cluster Runtime (`railiance-cluster`)

```bash
# On CoulombCore (SSH in first)
ssh -i ~/.ssh/id_ops tegwick@92.205.130.254

cd ~/railiance-cluster
make converge       # installs k3s, Helm, cert-manager, nginx ingress, cnpg operator
make smoke          # runs cluster health assertions
```

**Checkpoint:** k3s running, Helm available, cert-manager and nginx-ingress pods Running,
cnpg-system namespace active.

---

## S3 — Platform Services (`railiance-platform`)

```bash
# On CoulombCore (kubectl available after S2)
cd ~/railiance-platform && git pull

# Deploy Forgejo database cluster (production on railiance01)
# Credentials via SOPS — see railiance-platform docs/forgejo-backup.md
make forgejo-db-deploy
make forgejo-db-status

# Legacy coulombcore Gitea DB (read-only fallback — retain until T12 retires Gitea)
# make db-deploy && make db-status

# Deploy Valkey cache if required by workloads
# Requires: helm/valkey-values.sops.yaml with encrypted password
make valkey-deploy
make valkey-status
```

**Checkpoint:** `kubectl get cluster -n databases` shows `forgejo-db` healthy on
railiance01; Valkey pod Running if deployed.

---

## S4 — Developer Enablement (`railiance-enablement`)

No formal workplan yet. ArgoCD is currently deployed at cluster level (S2 boundary
violation, tracked in RAIL-HO-WP-0004). No S4-specific steps required at this time.

---

## S5 — Workloads & Experience (`railiance-apps`)

```bash
# On CoulombCore
cd ~/railiance-apps && git pull

# Deploy Forgejo (production source forge on railiance01)
# See railiance-apps/docs/forgejo-on-railiance01.md
make forgejo-deploy
make forgejo-verify

# Legacy Gitea on coulombcore — read-only fallback; do not use for new repos
# make gitea-deploy && make gitea-status

# Deploy state-hub, activity-core, and other workloads per railiance-apps Makefile
```

**Checkpoint:** `https://forgejo.coulomb.social` serves HTTPS; git clone/push via
`forgejo-remote`; state-hub `/state/health` returns 200.

---

## ops-bridge tunnel setup (workstation)

After S2 is up, establish the persistent tunnels from the workstation:

```bash
bridge up state-hub-coulombcore       # state-hub HTTP (port 18000 remote)
bridge up state-hub-mcp-coulombcore   # state-hub MCP (port 18001 remote)
bridge up k3s-api-coulombcore         # k3s API (port 16443 local)
```

Verify: `bridge status` shows all three connected.

---

## Recovery pointers

- **Forgejo backup/restore:** `railiance-platform/docs/forgejo-backup.md`, `docs/forgejo-restore-drill-evidence.md`
- **Cutover rollback:** `docs/forgejo-cutover-rollback.md`
- **Node overload / SSH unresponsive:** See `the-custodian/ops/runbooks/gitea-coulombcore.md` Issue #3
- **Incident report:** `the-custodian/ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
- **Cluster backup restore:** `railiance-cluster/tools/cmd/railiance-restore-s2`
- **Forgejo SSH:** NodePort `30022` on railiance01 (`forgejo-ssh-nodeport` in `forgejo` namespace)

---

## Layer dependency chain

```
S1 (infra) → S2 (cluster) → S3 (platform) → S4 (enablement) → S5 (workloads)
```

Each layer must be fully converged and verified before starting the next.
Never configure S2 concerns from S3+ repos (ADR-003 boundary rule).