railiance-infra/workplans/RAIL-HO-WP-0009-firewall-declared-state-and-api-exposure.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related_repos	state_hub_workstream_id
RAIL-HO-WP-0009	workplan	Firewall declared-state integrity and k3s API exposure	financials	railiance-infra	finished	codex	railiance	2026-08-12	2026-08-15	railiance-cluster railiance-platform railiance-master	91ac6340-c9e4-591b-b04d-a53db463e308

RAIL-HO-WP-0009 - Firewall declared-state integrity and k3s API exposure

Goal

Close a security defect in which this repo's declared firewall configuration was weaker than the live host, and remove the recurring exposure that produced it.

What was found

Discovered 2026-08-11 while diagnosing lost cluster access. The lost access was incidental; the defect underneath was not.

The defect. ansible/roles/base/tasks/main.yml declared 6443/tcp allowed with no source restriction:

- name: Allow k3s API in UFW
  ansible.builtin.ufw:
    rule: allow
    port: '6443'
    proto: tcp

The live host had source-restricted rules, added by hand with operator comments (k3s-api-operator-current, k3s-api-operator-hist). Security was tightened on the host and never fed back into the source of truth.

Consequence: running this role would have removed the restriction and exposed the Kubernetes API to the internet. A convergence run intended to harden the host would have de-hardened it, silently, with no failure to notice.

Why it went undetected. Nothing compares declared UFW state to live UFW state. The divergence was security-relevant, had existed for some time, and surfaced only because an ISP lease rotation (89.244.90.246 → .236) happened to break operator access mid-session.

Contradiction with our own INTENT. This repo declares "Declarative and Reproducible — no irreproducible, hand-tuned hosts" and "Hardened by Default". The hardening that actually protected the API was precisely the part that was not declarative.

Stale grants. Two standing allowlist entries pointed at addresses no longer ours: 89.244.90.246 (rotated) and 85.132.220.102 (historic). Dynamic ISP addresses get reassigned, so an abandoned grant becomes a grant to a stranger.

Boundaries

This workplan may:

• change firewall declarations and their inventory variables in this repo
• converge the base role against managed hosts, with operator approval
• propose the tunnel-based API access pattern and implement it here

It must not:

• change k3s or cluster runtime configuration (that is railiance-cluster, S2)
• silently widen any listening surface
• leave a host unreachable over SSH

Tasks

id: RAIL-HO-WP-0009-T01
status: done
priority: high
state_hub_task_id: "16010958-40b2-5463-b64b-c86800a5c08a"

Make the allowlist declarative. Add k3s_api_allowed_sources (empty default — 6443 closed to all external sources, the safe failure, SSH unaffected so hosts stay recoverable) and k3s_api_revoked_sources so rotated addresses are pruned rather than left standing. Grant approved sources before deleting the blanket rule so convergence never opens a window without API access. Declare the current operator address and the two stale grants in group_vars/all.yaml; correct docs/verification.md to state that 6443 is source-restricted.

Committed 2026-08-11. Access restored on the live host by hand in the same session (89.244.90.236 granted; kubectl verified, node Ready v1.35.1+k3s1).

id: RAIL-HO-WP-0009-T02
status: done
priority: high
state_hub_task_id: "bf66bb87-1601-587b-a91d-1908b8c1142d"

Converge the base role against railiance01 and confirm the resulting UFW state matches the declaration exactly: the current operator source allowed, no blanket Anywhere rule on 6443, and both revoked addresses gone. Production action — operator approval required before running. Until this runs, the live host still carries the two stale grants.

Verify SSH remains available throughout, and re-check kubectl get nodes after.

Progress 2026-08-12. The security goal is met surgically: both stale grants (89.244.90.246, 85.132.220.102) are deleted from the live host, and the live 6443 allowlist now matches the declaration exactly.

Full convergence was deliberately not run. A --check against Railiance01 reported 11 changes, most unrelated to the firewall: an sshd restart, MemoryMax=1500M and MemorySwapMax=512M on user-1000.slice, PAM nproc caps for tegwick, swappiness, timezone, and the ops-bridge key injection. That is a substantial and never-applied behavioural change to a production host — notably the user-slice memory cap, which could OOM running agent workloads. It is a separate decision from pruning two firewall grants, and the base role has no tags, so convergence cannot currently be scoped to UFW alone.

Two follow-ons fall out of this:

• adding tags: to the base role, so firewall changes can be converged without dragging unrelated drift with them
• deciding whether the resource-limit and sshd changes should be applied; they are the declared baseline and have simply never been run

A third operator address was found mid-session. 89.244.90.255 appeared in the live allowlist between two checks. It is legitimate — SSH pubkey auth as tegwick from that address on 2026-08-02, and [UFW BLOCK] entries on 6443 at 23:41 on 2026-08-12 immediately before it was granted. It is now declared in group_vars/all.yaml.

That is worth recording as evidence rather than as a footnote: the allowlist drifted again, by hand, during the very session that was fixing allowlist drift. It is the strongest available argument for T04.

Progress 2026-08-15. Tags now exist (make converge-firewall HOST=Railiance01). Live 6443 grants on this date: .236, .255, and an undeclared .248. T04 emptied the declared allowlist, so this task's remaining work is the operator-approved prune of those three grants. SSH stays; kubectl goes via k3s-api-railiance01 (:16444). CoulombCore is not touched (ufw_manage: false).

Done 2026-08-15. Operator approved. ansible-playbook playbooks/firewall.yaml -l Railiance01 --tags firewall deleted the three live 6443 grants (.248, .236, .255). Live UFW is now OpenSSH + Nydus 2224 only. SSH verified. sudo k3s kubectl get nodes → Ready v1.35.1+k3s1. Public :6443 times out. Tunnel k3s-api-railiance01 still answers on :16444.

id: RAIL-HO-WP-0009-T03
status: done
priority: high
state_hub_task_id: "847837f8-e4b4-5b9d-b625-e69d4c340f92"

Audit the rest of the base role for the same class of defect: any place where the declared state is weaker than, or simply divergent from, what was applied by hand. UFW was found by accident; sshd config, fail2ban jails, sudoers, and listening services deserve a deliberate pass. Record what is found even where no change is needed — the absence of drift is itself evidence worth having.

Note during the same session: port 2224/tcp is open to Anywhere on railiance01 (comment nydus-ex-api dashboard agent) and does not appear in this role at all. Establish whether it is intended, and either declare it or remove it.

Second finding, 2026-08-12: UFW is entirely inactive on CoulombCore. ufw status returns Status: inactive — no firewall at all, on a host running ArgoCD, the container registry and databases. The declared baseline (docs/verification.md) says "UFW active, default deny inbound". Not an emergency: 6443 is not reachable from outside (k3s appears bound locally or a provider firewall is in front), and 22/443/80 are the expected surface. But it is the same defect class as the k3s finding, in the opposite direction — the declaration is stronger than reality here, and equally undetected.

Converging CoulombCore would enable UFW on a frozen production host, which is a real availability risk and must not be done casually. Treat as its own decision.

Third finding: full convergence carries unrelated drift. --check against Railiance01 reports 11 changes, most of them not firewall-related and none ever applied — an sshd restart, MemoryMax=1500M / MemorySwapMax=512M on user-1000.slice, PAM nproc caps, swappiness, timezone. The user-slice memory cap in particular could OOM running agent workloads. The base role has no tags, so convergence cannot be scoped. Add tags as part of this task.

Done 2026-08-15. Full audit: docs/evidence/RAIL-HO-WP-0009-T03-declared-vs-live-audit.md.

• sshd / fail2ban / HISTCONTROL / sudoers match on both hosts.
• 2224/tcp is HostEurope Nydus; declared as ufw_extra_allowed.
• CoulombCore UFW is inactive; live filter is iptables INPUT DROP (6443/8472 not accepted). Set ufw_manage: false so a converge cannot enable UFW there.
• Base role and sibling bootstrap roles now have tags (firewall, ssh, packages, fail2ban, swap, resource_limits, sops). sshd restart is a handler. Routed-allow applies only when VXLAN peers are declared.
• Inventory script now loads ansible/inventory/host_vars/ (it was looking at a path that does not exist).
• Not in this workplan: Railiance01 resource limits, CoulombCore UFW enable, etcd/kubelet bind addresses, the 0.0.0.0:18081 issue-core proxy.

id: RAIL-HO-WP-0009-T04
status: done
priority: medium
state_hub_task_id: "ad315a7c-6a33-5716-9ab1-5b05f547e208"

Remove the k3s API from the public internet. Operator addresses rotate, so an allowlist is a treadmill: it will drift again, and each drift is either an outage or a stale grant. docs/deploy-stack.md already documents API access over the ops-bridge SSH tunnel (k3s-api-coulombcore, local port 16443) — apply the same pattern to railiance01 and reduce the public allowlist to nothing.

Decide explicitly rather than by default: this trades convenience for exposure, and the tunnel becomes a dependency of every operator action.

Done 2026-08-15. Decision recorded in docs/adr/ADR-005-k3s-api-tunnel-only.md. k3s_api_allowed_sources is empty; former public grants (including the undeclared .248 and the 2026-08-15 workstation lease .102) are in k3s_api_revoked_sources. Operator access is bridge up k3s-api-railiance01 (local 16444). Live prune of the three remaining grants is T02.

id: RAIL-HO-WP-0009-T06
status: done
priority: high
state_hub_task_id: "e498c6d5-3e29-5dea-80d2-08c712b15466"

Flannel VXLAN is declared open to Anywhere. Found 2026-08-12 by the newly tightened Goss assertion — the first defect the conformance check caught on its own.

- name: Allow Flannel VXLAN in UFW declares 8472/udp with no source restriction, the same defect pattern as the k3s API rule. The rule is currently absent from Railiance01, so there is no live exposure — but converging the base role would add it, opening VXLAN to the internet and introducing the exact class of defect this workplan exists to remove.

VXLAN on 8472/udp accepts encapsulated frames; exposing it publicly invites injection into the pod network. Restrict it to cluster node addresses, or omit it entirely while the cluster is single-node.

Note the sequencing risk: RAIL-BS-WP-0007 (ThreePhoenix HA, multi-node) will need working VXLAN between nodes. Resolve this before that lands, or the fix will be made under delivery pressure.

Done 2026-08-15. flannel_vxlan_allowed_sources defaults to empty; the role no longer adds a blanket 8472/udp allow and deletes one if present. Goss asserts 8472 is never ALLOW Anywhere and that the non-Anywhere count equals the declared peer list. When ThreePhoenix adds a second node, put its address in that list before converging firewall.

id: RAIL-HO-WP-0009-T05
status: done
priority: medium
state_hub_task_id: "ac0c3e8e-3384-5765-8ad1-98ffa68f19c1"

Propose a declared-vs-live conformance check for firewall state, and route it to whoever owns the conformance loop. This defect is a concrete instance of the unowned Q7 Governance and Change Management gap recorded in railiance-platform/ArchitectureBlueprint.md §5.3 — a UFW declared-vs-live comparison is a small, sharp first thing for that loop to do.

The existing Goss verification suite is the natural home for the check itself; what is missing is the loop that runs it and reacts.

Progress 2026-08-12 — the check now exists. goss/baseline.yaml became goss/baseline.yaml.j2, rendered per host from the same inventory declaration that converges it (k3s_api_allowed_sources, k3s_api_revoked_sources), so the assertion cannot drift from the declaration it checks. The old assertion matched /6443\/tcp.*ALLOW/ — the port is allowed, but not from whom — and passed identically whether the API was restricted to one address or open to the world. It now asserts each declared source is present, that no revoked source remains, and that 6443 is never ALLOW Anywhere.

verify.yaml also gained the vars_files the bootstrap play already had; without it the firewall assertions rendered empty and silently asserted nothing.

Result on Railiance01: 32 assertions, 31 pass, one genuine failure (T06). What remains for this task is the loop — scheduling it and routing failures somewhere they are seen.

Done 2026-08-15. The check now also asserts exact allowlist size, so an extra hand grant fails. The loop:

• hourly railiance-goss-baseline.timer on each host after make verify
• TAP + /var/lib/railiance/goss/last.status + FAILED flag
• optional State Hub POST on pass↔fail via RAILIANCE_GOSS_NOTIFY_URL
• make goss-status / make verify-host

Routed to railiance-master (ArchitectureBlueprint §5.3, Q7 owner) in docs/conformance-loop.md. Family-level reaction (page a human, block a rollout) is not this repo.

Outcome

Finished 2026-08-15. Declaration and live Railiance01 UFW now match ADR-005: no public 6443 grants, no blanket VXLAN allow, Nydus 2224 declared. CoulombCore UFW remains unmanaged (ufw_manage: false).