Close RAIL-HO-WP-0009 declared-state gaps; leave live 6443 prune gated

Make the k3s API tunnel-only (ADR-005), stop declaring Flannel VXLAN
open to Anywhere, tag the base role so firewall can be scoped, and
schedule the Goss declared-vs-live check. CoulombCore sets ufw_manage
false so a converge cannot enable UFW there. T02 still needs operator
approval for make converge-firewall HOST=Railiance01.

docs/adr/ADR-005-k3s-api-tunnel-only.md

@@ -0,0 +1,58 @@
+# ADR-005 — k3s API is tunnel-only
+
+**Status:** Accepted
+**Date:** 2026-08-15
+**Deciders:** implementation of RAIL-HO-WP-0009-T04
+**Workplans:** `RAIL-HO-WP-0009`
+
+---
+
+## Context
+
+Operator addresses on this network rotate with the ISP lease. A public UFW
+allowlist for `6443/tcp` is therefore a treadmill:
+
+- miss a rotation and `kubectl` breaks mid-session
+- leave the old grant standing and it becomes a grant to a stranger
+- hand-add the new address and the declaration drifts again
+
+That cycle produced this workplan. The live allowlist drifted by hand *during
+the session that was fixing allowlist drift*, and again before the next
+session (`89.244.90.248` appeared undeclared). On 2026-08-15 the workstation
+egress address was `85.132.220.102` — already on the revoked list as a
+"historic" operator address.
+
+`docs/deploy-stack.md` already documents API access over the ops-bridge SSH
+tunnel for CoulombCore (`k3s-api-coulombcore`, local port 16443). The same
+tunnel already exists for Railiance01 (`k3s-api-railiance01`, local port
+16444). SSH itself stays public, so the host remains recoverable.
+
+## Decision
+
+The public k3s API allowlist is **empty**. Operator and agent `kubectl` access
+uses the ops-bridge tunnels:
+
+| Cluster     | Tunnel                  | Local port | Remote |
+|-------------|-------------------------|------------|--------|
+| CoulombCore | `k3s-api-coulombcore`   | 16443      | 6443   |
+| Railiance01 | `k3s-api-railiance01`   | 16444      | 6443   |
+
+```bash
+bridge up k3s-api-railiance01
+# kubeconfig server: https://127.0.0.1:16444
+```
+
+Trade: every operator `kubectl` action depends on ops-bridge. That is accepted.
+A rotating public allowlist is the worse dependency.
+
+Emergency break-glass remains SSH: `ssh railiance01 -- sudo k3s kubectl …`.
+Do not re-open `6443/tcp` to Anywhere.
+
+## Consequences
+
+- `k3s_api_allowed_sources` stays `[]`.
+- Former public grants live in `k3s_api_revoked_sources` so a firewall-tagged
+  converge deletes them.
+- Goss asserts the 6443 allowlist size is exactly the declared length (zero)
+  and that no revoked address remains.
+- Amending this ADR is required before adding any new public 6443 source.

docs/conformance-loop.md

@@ -0,0 +1,61 @@
+# Declared-vs-live conformance loop
+
+This is the S1 slice of the unowned **Q7 Governance and Change Management**
+gap recorded in `railiance-platform/ArchitectureBlueprint.md` §5.3. The
+family owner of the loop is **`railiance-master`**. This repo owns the host
+check and the first reaction path; it does not own rapp/rail/reef schema
+conformance (that is 10c, blocked on the family declaration validator).
+
+## What runs
+
+| Piece | Where | Cadence |
+| --- | --- | --- |
+| Rendered Goss baseline | `/etc/goss/baseline.yaml` from `goss/baseline.yaml.j2` | every `make verify` |
+| On-host timer | `railiance-goss-baseline.timer` | hourly, plus 5 minutes after boot |
+| Wrapper | `/usr/local/sbin/goss-baseline-check` | writes `/var/lib/railiance/goss/` |
+| Operator collect | `make goss-status` | on demand |
+| Operator run | `make verify` / `make verify-host HOST=…` | on demand; commits TAP to `reports/` |
+
+The firewall assertions are generated from the same inventory lists that
+converge UFW (`k3s_api_allowed_sources`, `k3s_api_revoked_sources`,
+`flannel_vxlan_allowed_sources`, `ufw_extra_allowed`). They assert:
+
+- each declared source is present
+- the **count** of non-Anywhere 6443/8472 allows equals the declared length
+  (extra hand grants fail)
+- no `ALLOW Anywhere` on 6443 or 8472
+- every revoked 6443 source is gone
+
+That is the check that would have caught `89.244.90.248` appearing on
+Railiance01 after T01.
+
+## Where failures go
+
+1. **journald** — `logger -t railiance-goss`
+2. **on-host flag** — `/var/lib/railiance/goss/FAILED` and `last.status`
+3. **optional State Hub POST** — set `RAILIANCE_GOSS_NOTIFY_URL` on the host
+   to a State Hub `/progress/` endpoint. The wrapper notifies only on a
+   pass↔fail transition, so a known-fail host (CoulombCore, UFW inactive)
+   does not spam.
+4. **git TAP** — `make verify` still commits `reports/goss-<host>-<ts>.tap`
+
+`make goss-status` fails the play if any host still has the `FAILED` flag.
+
+## Routing to Q7
+
+`railiance-master` owns the conformance loop as a family capability
+(ArchitectureBlueprint §5.3; RAILIANCE-WP-0016 item 10b). This file is the
+handoff:
+
+- S1 check: **exists and is scheduled** in this repo
+- S1 reaction: journald + flag + optional progress event
+- Family reaction (page a human, open a task, block a rollout): **not here**
+
+Do not build a second alert path in `railiance-telemetry` for the same
+"check failed → someone sees it" plumbing. Item 9b should reuse this.
+
+## Known expected fail
+
+`CoulombCore` has UFW inactive. The baseline asserts `Status: active`. The
+timer will fail there until an explicit decision enables UFW or the host is
+removed from the verify inventory. That failure is evidence, not noise.

docs/convergence.md

@@ -9,7 +9,9 @@ This process is called **convergence**.
 When you run `make converge`, Ansible connects to all declared hosts and applies baseline roles:
 
 - **User setup** → ensures the `admin` user exists with your SSH key and passwordless sudo
-- **Firewall** → configures `ufw` with sensible defaults (deny incoming, allow SSH)
+- **Firewall** → configures `ufw` with deny incoming, SSH, declared extras
+  (Nydus), and a tunnel-only k3s API (ADR-005). Use
+  `make converge-firewall HOST=Railiance01` to apply only those tasks.
 - **Hardening** → basic SSH daemon hardening, disable root login, disable password auth
 - **Tooling** → installs essential packages (htop, vim, git, curl, fail2ban, etc.)
 - **SOPS agent** → ensures decryption tooling (`age`, `sops`) is available on the host

docs/deploy-stack.md

@@ -120,7 +120,8 @@ After S2 is up, establish the persistent tunnels from the workstation:
 ```bash
 bridge up state-hub-coulombcore       # state-hub HTTP (port 18000 remote)
 bridge up state-hub-mcp-coulombcore   # state-hub MCP (port 18001 remote)
-bridge up k3s-api-coulombcore         # k3s API (port 16443 local)
+bridge up k3s-api-coulombcore         # CoulombCore k3s API (port 16443 local)
+bridge up k3s-api-railiance01         # Railiance01 k3s API (port 16444 local)
 ```
 
 Verify: `bridge status` shows all three connected.

docs/evidence/RAIL-HO-WP-0009-T03-declared-vs-live-audit.md

@@ -0,0 +1,141 @@
+# RAIL-HO-WP-0009-T03 — declared-vs-live audit
+
+Audited 2026-08-15 from the workstation against `Railiance01`
+(`92.205.62.239`) and `CoulombCore` (`92.205.130.254`). SSH as `tegwick`.
+This is a record of what was found; absence of drift is recorded too.
+
+## Method
+
+Compared `ansible/roles/base/tasks/main.yml`, sibling bootstrap roles, and
+`spec/server-baseline.yaml` to live `ufw status`, `sshd -T`, fail2ban,
+sudoers, listening sockets, systemd user-slice, nproc, swappiness, and
+timezone.
+
+## Matches declaration
+
+| Surface | Railiance01 | CoulombCore |
+| --- | --- | --- |
+| sshd drop-in `10-hardening.conf` | present; root/password off, pubkey on | same |
+| `sshd -T` effective | matches drop-in | matches drop-in |
+| fail2ban `sshd` jail | active | active |
+| `/etc/profile.d/histcontrol.sh` | `ignorespace` | `ignorespace` |
+| sudoers `tegwick` NOPASSWD | present | present |
+| OpenSSH UFW | ALLOW Anywhere | UFW inactive (see below) |
+
+## Divergences
+
+### 1. k3s API allowlist (Railiance01) — same defect class as T01
+
+Live 6443 grants on 2026-08-15:
+
+| From | Declared? |
+| --- | --- |
+| `89.244.90.236` | was declared; now revoked by ADR-005 |
+| `89.244.90.255` | was declared; now revoked by ADR-005 |
+| `89.244.90.248` | **never declared** — third hand grant |
+
+No blanket `Anywhere` rule. Stale grants from T02 (`.246`, `.102`) are gone.
+
+`.248` appeared after the 2026-08-12 session that was already fixing
+allowlist drift. Goss as of T01 would not have failed: it asserted declared
+sources present, not that the live set equalled the declared set.
+
+Workstation egress on this audit was `85.132.220.102` — already on the
+revoked list. That is the allowlist treadmill in one line.
+
+### 2. Flannel VXLAN declaration weaker than needed (T06)
+
+`8472/udp` listens on `0.0.0.0` on both hosts. Railiance01 has **no** UFW
+rule for it (default deny incoming covers it). The role used to declare
+`8472/udp` ALLOW with no source restriction, so a converge would have
+*opened* VXLAN to the internet. Fixed: empty `flannel_vxlan_allowed_sources`,
+blanket rule deleted if present.
+
+### 3. Port 2224/tcp Anywhere — intended, now declared
+
+Both hosts run `nydus-ex-api` on `0.0.0.0:2224`. Railiance01 UFW allows it
+Anywhere with comment `nydus-ex-api dashboard agent`. This is the HostEurope
+Nydus agent (dashboard, root-password reset, backups); see
+`hosteurope/260308-dependency-nydus.md`. Removing it would break the
+provider. Declared as `ufw_extra_allowed`. Source-restricting it is not
+possible without HostEurope's egress list.
+
+### 4. CoulombCore: UFW inactive, but not "no firewall"
+
+`ufw status` → `Status: inactive`. Live filter is `iptables` `INPUT DROP`
+with a Plesk-era accept list: 22, 80, 443, 2224, mail, 3306, 5432, 53,
+445/139, and **tcp 49152-65535**. 6443 and 8472 are **not** in that accept
+list, so they are dropped even with UFW off.
+
+Converging the base role would enable UFW, allow only SSH + (now empty) k3s
++ nydus, and take down 80/443 and the rest of the accepted surface.
+`ufw_manage: false` on CoulombCore prevents that. Enabling UFW there is its
+own decision and needs a declared accept list first.
+
+### 5. Resource limits and swap — applied on CoulombCore, never on Railiance01
+
+| | Railiance01 | CoulombCore |
+| --- | --- | --- |
+| `user-1000.slice` MemoryMax | absent | `1500M` / swap `512M` (Ansible) |
+| PAM nproc | no `limits.d` files | `60-nproc-tegwick.conf` |
+| swappiness | 60 (kernel default) | 10 |
+| timezone | `Etc/UTC` | `Etc/UTC` |
+
+The user-slice cap can OOM agent workloads (T02). Do **not** apply it to
+Railiance01 without a separate decision. Role tags now isolate it
+(`resource_limits`, `swap`, `timezone`).
+
+`inventory/group_vars/all.yaml` sets `timezone: Europe/Berlin` but that file
+is not loaded by the Ansible inventory script. The role default is UTC, which
+matches live. Leave it; do not start loading that file as a side effect.
+
+### 6. Inventory host_vars path bug
+
+`ansible/inventory_from_yaml.py` loaded `inventory/host_vars/<name>.yml`
+(does not exist). The real file is `ansible/inventory/host_vars/CoulombCore.yml`.
+Ansible does not auto-load host_vars next to a script inventory, so
+`ufw_manage`, swap, and resource-limit overrides would not reach the play.
+Fixed: the script now reads `ansible/inventory/host_vars/` first.
+
+### 7. sshd restart was an unconditioned task
+
+`Restart sshd` ran on every converge, which is why `--check` reported an
+sshd restart among the 11 changes. It is now a handler notified only when
+the hardening drop-in changes.
+
+### 8. Undeclared listening sockets (not opened in UFW)
+
+On Railiance01, not in this role, UFW default-deny so not currently public:
+
+- `*:6443` k3s-server (the API this workplan closes)
+- `92.205.62.239:2379` / `:2380` etcd on the public address — S2 concern
+- `*:10250` kubelet — S2 concern
+- `0.0.0.0:18081` Python reverse proxy to `127.0.0.1:18765` (issue-core),
+  started 2026-07-26. No UFW rule. Out of scope here; record only.
+
+On CoulombCore, `*:6443` and `0.0.0.0:8472` listen; iptables INPUT DROP
+without an accept for those ports is what currently protects them, not UFW.
+
+### 9. Routed policy
+
+Railiance01 UFW default routed is **deny**. The role used to set routed
+**allow** for Flannel. Routed allow is now applied only when
+`flannel_vxlan_allowed_sources` is non-empty (multi-node). Single-node
+converge will not flip it.
+
+## Decisions taken in this task
+
+- Declare Nydus 2224; do not remove it.
+- Do not enable UFW on CoulombCore (`ufw_manage: false`).
+- Do not apply Railiance01 resource limits / swappiness / timezone as part
+  of this workplan.
+- Tag the base role and sibling bootstrap roles so firewall can be scoped.
+- Fix the host_vars loader.
+
+## Still operator decisions (not this workplan)
+
+- Enable UFW on CoulombCore, with a declared accept list that includes 80/443.
+- Apply or drop the declared user-slice memory cap on Railiance01.
+- Whether etcd/kubelet listening on the public address should be rebound
+  (`railiance-cluster`).
+- What the `0.0.0.0:18081` issue-core proxy is for, and whether it stays.

docs/k3s-api-access.md

@@ -0,0 +1,48 @@
+# k3s API access
+
+The Kubernetes API is not on the public internet. See
+[ADR-005](adr/ADR-005-k3s-api-tunnel-only.md).
+
+## Day-to-day (workstation)
+
+```bash
+bridge up k3s-api-railiance01     # https://127.0.0.1:16444  →  Railiance01 :6443
+bridge up k3s-api-coulombcore     # https://127.0.0.1:16443  →  CoulombCore :6443
+bridge status
+```
+
+Point the kubeconfig `server` at the matching local port. The default
+`~/.kube/config` and `~/.kube/config-hosteurope` currently both target
+`https://127.0.0.1:16443` (CoulombCore). For Railiance01 use 16444.
+
+```bash
+# Railiance01
+KUBECONFIG=~/.kube/config-railiance01 kubectl get nodes
+# or, once:
+#   kubectl --kubeconfig ~/.kube/config-railiance01 config set-cluster default --server=https://127.0.0.1:16444
+```
+
+SSH is unaffected. If the tunnel is down, recover over SSH:
+
+```bash
+ssh tegwick@92.205.62.239 -- sudo k3s kubectl get nodes
+```
+
+## What not to do
+
+- Do not `ufw allow 6443/tcp` from the current public IP.
+- Do not add addresses to `k3s_api_allowed_sources` without amending ADR-005.
+- When an old grant must die, put it in `k3s_api_revoked_sources` and run
+  `make converge-firewall HOST=Railiance01`.
+
+## Converge the firewall only
+
+The base role is tagged. Firewall changes no longer drag sshd restarts or
+user-slice memory caps with them:
+
+```bash
+make converge-firewall HOST=Railiance01
+```
+
+CoulombCore sets `ufw_manage: false`. That host's live filter is iptables, not
+UFW; this target will skip UFW tasks there.

docs/verification.md

@@ -10,11 +10,13 @@ a reproducible, CI-friendly pass/fail verdict.
 of every managed node. It covers:
 
 - **Firewall** — UFW active, default deny inbound, required ports allowed
-  (SSH 22/tcp, Flannel VXLAN 8472/udp). The k3s API (6443/tcp) is
-  **source-restricted**, not world-open: only addresses in
-  `k3s_api_allowed_sources` may reach it, and `k3s_api_revoked_sources` is
-  pruned on convergence. A host with an empty allowlist has 6443 closed to all
-  external sources — the safe failure, recoverable over SSH.
+  (SSH 22/tcp; HostEurope Nydus 2224/tcp). The k3s API (6443/tcp) is
+  **tunnel-only** (ADR-005): `k3s_api_allowed_sources` is empty, and
+  `k3s_api_revoked_sources` is pruned on a firewall-tagged converge. Flannel
+  VXLAN (8472/udp) is omitted while the cluster is single-node; peer
+  addresses go in `flannel_vxlan_allowed_sources` when a second node appears.
+  Hosts with `ufw_manage: false` (CoulombCore) are not rewritten. A host with
+  an empty 6443 allowlist is recoverable over SSH.
 - **SSH daemon** — root login disabled, password auth disabled, pubkey auth enabled
 - **Services** — ufw, fail2ban, ssh.socket enabled and running
 - **Packages** — ufw, fail2ban, git, curl, vim, htop (age and sops installed as binaries)
@@ -56,7 +58,8 @@ you need a structured, automatable check.
 
 ## Goss test file
 
-`goss/baseline.yaml` contains one Goss assertion per spec item. The mapping is:
+`goss/baseline.yaml.j2` is rendered per host from the same inventory lists
+that converge UFW. The mapping is:
 
 | spec section | Goss resource |
 |---|---|
@@ -73,8 +76,11 @@ you need a structured, automatable check.
 
 1. Add the desired state to `spec/server-baseline.yaml`
 2. Add the Ansible task to `ansible/roles/base/tasks/main.yml`
-3. Add the Goss assertion to `goss/baseline.yaml`
-4. Run `make converge && make verify` to confirm
+3. Add the Goss assertion to `goss/baseline.yaml.j2`
+4. Run `make converge-firewall` and `make verify-host HOST=…` to confirm
+
+An hourly on-host timer (`railiance-goss-baseline.timer`) reruns the last
+rendered baseline. See `docs/conformance-loop.md`.
 
 ## Reports