--- id: RAIL-HO-WP-0009 type: workplan title: "Firewall declared-state integrity and k3s API exposure" domain: financials repo: railiance-infra status: finished owner: codex topic_slug: railiance created: "2026-08-12" updated: "2026-08-15" related_repos: - railiance-cluster - railiance-platform - railiance-master state_hub_workstream_id: "91ac6340-c9e4-591b-b04d-a53db463e308" --- # RAIL-HO-WP-0009 - Firewall declared-state integrity and k3s API exposure ## Goal Close a security defect in which this repo's declared firewall configuration was **weaker** than the live host, and remove the recurring exposure that produced it. ## What was found Discovered 2026-08-11 while diagnosing lost cluster access. The lost access was incidental; the defect underneath was not. **The defect.** `ansible/roles/base/tasks/main.yml` declared `6443/tcp` allowed with no source restriction: ```yaml - name: Allow k3s API in UFW ansible.builtin.ufw: rule: allow port: '6443' proto: tcp ``` The live host had **source-restricted** rules, added by hand with operator comments (`k3s-api-operator-current`, `k3s-api-operator-hist`). Security was tightened on the host and never fed back into the source of truth. Consequence: **running this role would have removed the restriction and exposed the Kubernetes API to the internet.** A convergence run intended to harden the host would have de-hardened it, silently, with no failure to notice. **Why it went undetected.** Nothing compares declared UFW state to live UFW state. The divergence was security-relevant, had existed for some time, and surfaced only because an ISP lease rotation (`89.244.90.246` → `.236`) happened to break operator access mid-session. **Contradiction with our own INTENT.** This repo declares "Declarative and Reproducible — no irreproducible, hand-tuned hosts" and "Hardened by Default". The hardening that actually protected the API was precisely the part that was not declarative. **Stale grants.** Two standing allowlist entries pointed at addresses no longer ours: `89.244.90.246` (rotated) and `85.132.220.102` (historic). Dynamic ISP addresses get reassigned, so an abandoned grant becomes a grant to a stranger. ## Boundaries This workplan may: - change firewall declarations and their inventory variables in this repo - converge the base role against managed hosts, with operator approval - propose the tunnel-based API access pattern and implement it here It must not: - change k3s or cluster runtime configuration (that is `railiance-cluster`, S2) - silently widen any listening surface - leave a host unreachable over SSH ## Tasks ```task id: RAIL-HO-WP-0009-T01 status: done priority: high state_hub_task_id: "16010958-40b2-5463-b64b-c86800a5c08a" ``` Make the allowlist declarative. Add `k3s_api_allowed_sources` (empty default — 6443 closed to all external sources, the safe failure, SSH unaffected so hosts stay recoverable) and `k3s_api_revoked_sources` so rotated addresses are pruned rather than left standing. Grant approved sources **before** deleting the blanket rule so convergence never opens a window without API access. Declare the current operator address and the two stale grants in `group_vars/all.yaml`; correct `docs/verification.md` to state that 6443 is source-restricted. Committed 2026-08-11. Access restored on the live host by hand in the same session (`89.244.90.236` granted; `kubectl` verified, node Ready v1.35.1+k3s1). ```task id: RAIL-HO-WP-0009-T02 status: done priority: high state_hub_task_id: "bf66bb87-1601-587b-a91d-1908b8c1142d" ``` Converge the base role against `railiance01` and confirm the resulting UFW state matches the declaration exactly: the current operator source allowed, no blanket `Anywhere` rule on 6443, and both revoked addresses gone. **Production action — operator approval required before running.** Until this runs, the live host still carries the two stale grants. Verify SSH remains available throughout, and re-check `kubectl get nodes` after. **Progress 2026-08-12.** The security goal is met surgically: both stale grants (`89.244.90.246`, `85.132.220.102`) are deleted from the live host, and the live 6443 allowlist now matches the declaration exactly. Full convergence was **deliberately not run**. A `--check` against `Railiance01` reported **11 changes**, most unrelated to the firewall: an sshd restart, `MemoryMax=1500M` and `MemorySwapMax=512M` on `user-1000.slice`, PAM `nproc` caps for `tegwick`, swappiness, timezone, and the ops-bridge key injection. That is a substantial and never-applied behavioural change to a production host — notably the user-slice memory cap, which could OOM running agent workloads. It is a separate decision from pruning two firewall grants, and the base role has **no tags**, so convergence cannot currently be scoped to UFW alone. Two follow-ons fall out of this: - adding `tags:` to the base role, so firewall changes can be converged without dragging unrelated drift with them - deciding whether the resource-limit and sshd changes should be applied; they are the declared baseline and have simply never been run **A third operator address was found mid-session.** `89.244.90.255` appeared in the live allowlist between two checks. It is legitimate — SSH pubkey auth as `tegwick` from that address on 2026-08-02, and `[UFW BLOCK]` entries on 6443 at 23:41 on 2026-08-12 immediately before it was granted. It is now declared in `group_vars/all.yaml`. That is worth recording as evidence rather than as a footnote: the allowlist drifted again, by hand, *during the very session that was fixing allowlist drift*. It is the strongest available argument for T04. **Progress 2026-08-15.** Tags now exist (`make converge-firewall HOST=Railiance01`). Live 6443 grants on this date: `.236`, `.255`, and an undeclared `.248`. T04 emptied the declared allowlist, so this task's remaining work is the operator-approved prune of those three grants. SSH stays; kubectl goes via `k3s-api-railiance01` (`:16444`). CoulombCore is not touched (`ufw_manage: false`). **Done 2026-08-15.** Operator approved. `ansible-playbook playbooks/firewall.yaml -l Railiance01 --tags firewall` deleted the three live 6443 grants (`.248`, `.236`, `.255`). Live UFW is now OpenSSH + Nydus 2224 only. SSH verified. `sudo k3s kubectl get nodes` → Ready v1.35.1+k3s1. Public `:6443` times out. Tunnel `k3s-api-railiance01` still answers on `:16444`. ```task id: RAIL-HO-WP-0009-T03 status: done priority: high state_hub_task_id: "847837f8-e4b4-5b9d-b625-e69d4c340f92" ``` Audit the rest of the base role for the same class of defect: any place where the declared state is weaker than, or simply divergent from, what was applied by hand. UFW was found by accident; sshd config, fail2ban jails, sudoers, and listening services deserve a deliberate pass. Record what is found even where no change is needed — the absence of drift is itself evidence worth having. Note during the same session: port `2224/tcp` is open to Anywhere on `railiance01` (comment `nydus-ex-api dashboard agent`) and does not appear in this role at all. Establish whether it is intended, and either declare it or remove it. **Second finding, 2026-08-12: UFW is entirely inactive on `CoulombCore`.** `ufw status` returns `Status: inactive` — no firewall at all, on a host running ArgoCD, the container registry and databases. The declared baseline (`docs/verification.md`) says "UFW active, default deny inbound". Not an emergency: 6443 is not reachable from outside (k3s appears bound locally or a provider firewall is in front), and 22/443/80 are the expected surface. But it is the same defect class as the k3s finding, in the opposite direction — the declaration is *stronger* than reality here, and equally undetected. Converging `CoulombCore` would **enable UFW on a frozen production host**, which is a real availability risk and must not be done casually. Treat as its own decision. **Third finding: full convergence carries unrelated drift.** `--check` against `Railiance01` reports 11 changes, most of them not firewall-related and none ever applied — an sshd restart, `MemoryMax=1500M` / `MemorySwapMax=512M` on `user-1000.slice`, PAM `nproc` caps, swappiness, timezone. The user-slice memory cap in particular could OOM running agent workloads. The base role has **no tags**, so convergence cannot be scoped. Add tags as part of this task. **Done 2026-08-15.** Full audit: `docs/evidence/RAIL-HO-WP-0009-T03-declared-vs-live-audit.md`. - sshd / fail2ban / HISTCONTROL / sudoers match on both hosts. - 2224/tcp is HostEurope Nydus; declared as `ufw_extra_allowed`. - CoulombCore UFW is inactive; live filter is iptables INPUT DROP (6443/8472 not accepted). Set `ufw_manage: false` so a converge cannot enable UFW there. - Base role and sibling bootstrap roles now have tags (`firewall`, `ssh`, `packages`, `fail2ban`, `swap`, `resource_limits`, `sops`). sshd restart is a handler. Routed-allow applies only when VXLAN peers are declared. - Inventory script now loads `ansible/inventory/host_vars/` (it was looking at a path that does not exist). - Not in this workplan: Railiance01 resource limits, CoulombCore UFW enable, etcd/kubelet bind addresses, the `0.0.0.0:18081` issue-core proxy. ```task id: RAIL-HO-WP-0009-T04 status: done priority: medium state_hub_task_id: "ad315a7c-6a33-5716-9ab1-5b05f547e208" ``` Remove the k3s API from the public internet. Operator addresses rotate, so an allowlist is a treadmill: it will drift again, and each drift is either an outage or a stale grant. `docs/deploy-stack.md` already documents API access over the ops-bridge SSH tunnel (`k3s-api-coulombcore`, local port 16443) — apply the same pattern to `railiance01` and reduce the public allowlist to nothing. Decide explicitly rather than by default: this trades convenience for exposure, and the tunnel becomes a dependency of every operator action. **Done 2026-08-15.** Decision recorded in `docs/adr/ADR-005-k3s-api-tunnel-only.md`. `k3s_api_allowed_sources` is empty; former public grants (including the undeclared `.248` and the 2026-08-15 workstation lease `.102`) are in `k3s_api_revoked_sources`. Operator access is `bridge up k3s-api-railiance01` (local 16444). Live prune of the three remaining grants is T02. ```task id: RAIL-HO-WP-0009-T06 status: done priority: high state_hub_task_id: "e498c6d5-3e29-5dea-80d2-08c712b15466" ``` **Flannel VXLAN is declared open to Anywhere.** Found 2026-08-12 by the newly tightened Goss assertion — the first defect the conformance check caught on its own. `- name: Allow Flannel VXLAN in UFW` declares `8472/udp` with **no source restriction**, the same defect pattern as the k3s API rule. The rule is currently **absent** from `Railiance01`, so there is no live exposure — but converging the base role would *add* it, opening VXLAN to the internet and introducing the exact class of defect this workplan exists to remove. VXLAN on 8472/udp accepts encapsulated frames; exposing it publicly invites injection into the pod network. Restrict it to cluster node addresses, or omit it entirely while the cluster is single-node. Note the sequencing risk: `RAIL-BS-WP-0007` (ThreePhoenix HA, multi-node) will need working VXLAN between nodes. Resolve this before that lands, or the fix will be made under delivery pressure. **Done 2026-08-15.** `flannel_vxlan_allowed_sources` defaults to empty; the role no longer adds a blanket 8472/udp allow and deletes one if present. Goss asserts 8472 is never `ALLOW Anywhere` and that the non-Anywhere count equals the declared peer list. When ThreePhoenix adds a second node, put its address in that list before converging firewall. ```task id: RAIL-HO-WP-0009-T05 status: done priority: medium state_hub_task_id: "ac0c3e8e-3384-5765-8ad1-98ffa68f19c1" ``` Propose a declared-vs-live conformance check for firewall state, and route it to whoever owns the conformance loop. This defect is a concrete instance of the unowned **Q7 Governance and Change Management** gap recorded in `railiance-platform/ArchitectureBlueprint.md` §5.3 — a UFW declared-vs-live comparison is a small, sharp first thing for that loop to do. The existing Goss verification suite is the natural home for the check itself; what is missing is the loop that runs it and reacts. **Progress 2026-08-12 — the check now exists.** `goss/baseline.yaml` became `goss/baseline.yaml.j2`, rendered per host from the *same* inventory declaration that converges it (`k3s_api_allowed_sources`, `k3s_api_revoked_sources`), so the assertion cannot drift from the declaration it checks. The old assertion matched `/6443\/tcp.*ALLOW/` — the port is allowed, but not from whom — and passed identically whether the API was restricted to one address or open to the world. It now asserts each declared source is present, that no revoked source remains, and that 6443 is never `ALLOW Anywhere`. `verify.yaml` also gained the `vars_files` the bootstrap play already had; without it the firewall assertions rendered empty and silently asserted nothing. Result on `Railiance01`: 32 assertions, 31 pass, one genuine failure (T06). What remains for this task is the *loop* — scheduling it and routing failures somewhere they are seen. **Done 2026-08-15.** The check now also asserts exact allowlist *size*, so an extra hand grant fails. The loop: - hourly `railiance-goss-baseline.timer` on each host after `make verify` - TAP + `/var/lib/railiance/goss/last.status` + `FAILED` flag - optional State Hub POST on pass↔fail via `RAILIANCE_GOSS_NOTIFY_URL` - `make goss-status` / `make verify-host` Routed to `railiance-master` (ArchitectureBlueprint §5.3, Q7 owner) in `docs/conformance-loop.md`. Family-level reaction (page a human, block a rollout) is not this repo. ## Outcome Finished 2026-08-15. Declaration and live Railiance01 UFW now match ADR-005: no public 6443 grants, no blanket VXLAN allow, Nydus 2224 declared. CoulombCore UFW remains unmanaged (`ufw_manage: false`).