railiance-infra/workplans/RAIL-HO-WP-0009-firewall-declared-state-and-api-exposure.md

---
id: RAIL-HO-WP-0009
type: workplan
title: "Firewall declared-state integrity and k3s API exposure"
domain: financials
repo: railiance-infra
status: finished
owner: codex
topic_slug: railiance
created: "2026-08-12"
updated: "2026-08-15"
related_repos:
  - railiance-cluster
  - railiance-platform
  - railiance-master
state_hub_workstream_id: "91ac6340-c9e4-591b-b04d-a53db463e308"
---

# RAIL-HO-WP-0009 - Firewall declared-state integrity and k3s API exposure

## Goal

Close a security defect in which this repo's declared firewall configuration was
**weaker** than the live host, and remove the recurring exposure that produced
it.

## What was found

Discovered 2026-08-11 while diagnosing lost cluster access. The lost access was
incidental; the defect underneath was not.

**The defect.** `ansible/roles/base/tasks/main.yml` declared `6443/tcp` allowed
with no source restriction:

```yaml
- name: Allow k3s API in UFW
  ansible.builtin.ufw:
    rule: allow
    port: '6443'
    proto: tcp
```

The live host had **source-restricted** rules, added by hand with operator
comments (`k3s-api-operator-current`, `k3s-api-operator-hist`). Security was
tightened on the host and never fed back into the source of truth.

Consequence: **running this role would have removed the restriction and exposed
the Kubernetes API to the internet.** A convergence run intended to harden the
host would have de-hardened it, silently, with no failure to notice.

**Why it went undetected.** Nothing compares declared UFW state to live UFW
state. The divergence was security-relevant, had existed for some time, and
surfaced only because an ISP lease rotation (`89.244.90.246` → `.236`) happened
to break operator access mid-session.

**Contradiction with our own INTENT.** This repo declares "Declarative and
Reproducible — no irreproducible, hand-tuned hosts" and "Hardened by Default".
The hardening that actually protected the API was precisely the part that was
not declarative.

**Stale grants.** Two standing allowlist entries pointed at addresses no longer
ours: `89.244.90.246` (rotated) and `85.132.220.102` (historic). Dynamic ISP
addresses get reassigned, so an abandoned grant becomes a grant to a stranger.

## Boundaries

This workplan may:

- change firewall declarations and their inventory variables in this repo
- converge the base role against managed hosts, with operator approval
- propose the tunnel-based API access pattern and implement it here

It must not:

- change k3s or cluster runtime configuration (that is `railiance-cluster`, S2)
- silently widen any listening surface
- leave a host unreachable over SSH

## Tasks

```task
id: RAIL-HO-WP-0009-T01
status: done
priority: high
state_hub_task_id: "16010958-40b2-5463-b64b-c86800a5c08a"
```

Make the allowlist declarative. Add `k3s_api_allowed_sources` (empty default —
6443 closed to all external sources, the safe failure, SSH unaffected so hosts
stay recoverable) and `k3s_api_revoked_sources` so rotated addresses are pruned
rather than left standing. Grant approved sources **before** deleting the
blanket rule so convergence never opens a window without API access. Declare the
current operator address and the two stale grants in `group_vars/all.yaml`;
correct `docs/verification.md` to state that 6443 is source-restricted.

Committed 2026-08-11. Access restored on the live host by hand in the same
session (`89.244.90.236` granted; `kubectl` verified, node Ready v1.35.1+k3s1).

```task
id: RAIL-HO-WP-0009-T02
status: done
priority: high
state_hub_task_id: "bf66bb87-1601-587b-a91d-1908b8c1142d"
```

Converge the base role against `railiance01` and confirm the resulting UFW state
matches the declaration exactly: the current operator source allowed, no blanket
`Anywhere` rule on 6443, and both revoked addresses gone. **Production action —
operator approval required before running.** Until this runs, the live host
still carries the two stale grants.

Verify SSH remains available throughout, and re-check `kubectl get nodes` after.

**Progress 2026-08-12.** The security goal is met surgically: both stale grants
(`89.244.90.246`, `85.132.220.102`) are deleted from the live host, and the live
6443 allowlist now matches the declaration exactly.

Full convergence was **deliberately not run**. A `--check` against `Railiance01`
reported **11 changes**, most unrelated to the firewall: an sshd restart,
`MemoryMax=1500M` and `MemorySwapMax=512M` on `user-1000.slice`, PAM `nproc`
caps for `tegwick`, swappiness, timezone, and the ops-bridge key injection. That
is a substantial and never-applied behavioural change to a production host —
notably the user-slice memory cap, which could OOM running agent workloads. It
is a separate decision from pruning two firewall grants, and the base role has
**no tags**, so convergence cannot currently be scoped to UFW alone.

Two follow-ons fall out of this:

- adding `tags:` to the base role, so firewall changes can be converged without
  dragging unrelated drift with them
- deciding whether the resource-limit and sshd changes should be applied; they
  are the declared baseline and have simply never been run

**A third operator address was found mid-session.** `89.244.90.255` appeared in
the live allowlist between two checks. It is legitimate — SSH pubkey auth as
`tegwick` from that address on 2026-08-02, and `[UFW BLOCK]` entries on 6443 at
23:41 on 2026-08-12 immediately before it was granted. It is now declared in
`group_vars/all.yaml`.

That is worth recording as evidence rather than as a footnote: the allowlist
drifted again, by hand, *during the very session that was fixing allowlist
drift*. It is the strongest available argument for T04.

**Progress 2026-08-15.** Tags now exist (`make converge-firewall HOST=Railiance01`).
Live 6443 grants on this date: `.236`, `.255`, and an undeclared `.248`. T04
emptied the declared allowlist, so this task's remaining work is the
operator-approved prune of those three grants. SSH stays; kubectl goes via
`k3s-api-railiance01` (`:16444`). CoulombCore is not touched
(`ufw_manage: false`).

**Done 2026-08-15.** Operator approved. `ansible-playbook playbooks/firewall.yaml -l Railiance01 --tags firewall` deleted the three live 6443 grants (`.248`, `.236`, `.255`). Live UFW is now OpenSSH + Nydus 2224 only. SSH verified. `sudo k3s kubectl get nodes` → Ready v1.35.1+k3s1. Public `:6443` times out. Tunnel `k3s-api-railiance01` still answers on `:16444`.

```task
id: RAIL-HO-WP-0009-T03
status: done
priority: high
state_hub_task_id: "847837f8-e4b4-5b9d-b625-e69d4c340f92"
```

Audit the rest of the base role for the same class of defect: any place where
the declared state is weaker than, or simply divergent from, what was applied by
hand. UFW was found by accident; sshd config, fail2ban jails, sudoers, and
listening services deserve a deliberate pass. Record what is found even where no
change is needed — the absence of drift is itself evidence worth having.

Note during the same session: port `2224/tcp` is open to Anywhere on
`railiance01` (comment `nydus-ex-api dashboard agent`) and does not appear in
this role at all. Establish whether it is intended, and either declare it or
remove it.

**Second finding, 2026-08-12: UFW is entirely inactive on `CoulombCore`.**
`ufw status` returns `Status: inactive` — no firewall at all, on a host running
ArgoCD, the container registry and databases. The declared baseline
(`docs/verification.md`) says "UFW active, default deny inbound". Not an
emergency: 6443 is not reachable from outside (k3s appears bound locally or a
provider firewall is in front), and 22/443/80 are the expected surface. But it
is the same defect class as the k3s finding, in the opposite direction — the
declaration is *stronger* than reality here, and equally undetected.

Converging `CoulombCore` would **enable UFW on a frozen production host**, which
is a real availability risk and must not be done casually. Treat as its own
decision.

**Third finding: full convergence carries unrelated drift.** `--check` against
`Railiance01` reports 11 changes, most of them not firewall-related and none
ever applied — an sshd restart, `MemoryMax=1500M` / `MemorySwapMax=512M` on
`user-1000.slice`, PAM `nproc` caps, swappiness, timezone. The user-slice memory
cap in particular could OOM running agent workloads. The base role has **no
tags**, so convergence cannot be scoped. Add tags as part of this task.

**Done 2026-08-15.** Full audit:
`docs/evidence/RAIL-HO-WP-0009-T03-declared-vs-live-audit.md`.

- sshd / fail2ban / HISTCONTROL / sudoers match on both hosts.
- 2224/tcp is HostEurope Nydus; declared as `ufw_extra_allowed`.
- CoulombCore UFW is inactive; live filter is iptables INPUT DROP (6443/8472
  not accepted). Set `ufw_manage: false` so a converge cannot enable UFW there.
- Base role and sibling bootstrap roles now have tags (`firewall`, `ssh`,
  `packages`, `fail2ban`, `swap`, `resource_limits`, `sops`). sshd restart is
  a handler. Routed-allow applies only when VXLAN peers are declared.
- Inventory script now loads `ansible/inventory/host_vars/` (it was looking at
  a path that does not exist).
- Not in this workplan: Railiance01 resource limits, CoulombCore UFW enable,
  etcd/kubelet bind addresses, the `0.0.0.0:18081` issue-core proxy.

```task
id: RAIL-HO-WP-0009-T04
status: done
priority: medium
state_hub_task_id: "ad315a7c-6a33-5716-9ab1-5b05f547e208"
```

Remove the k3s API from the public internet. Operator addresses rotate, so an
allowlist is a treadmill: it will drift again, and each drift is either an
outage or a stale grant. `docs/deploy-stack.md` already documents API access
over the ops-bridge SSH tunnel (`k3s-api-coulombcore`, local port 16443) — apply
the same pattern to `railiance01` and reduce the public allowlist to nothing.

Decide explicitly rather than by default: this trades convenience for exposure,
and the tunnel becomes a dependency of every operator action.

**Done 2026-08-15.** Decision recorded in
`docs/adr/ADR-005-k3s-api-tunnel-only.md`. `k3s_api_allowed_sources` is empty;
former public grants (including the undeclared `.248` and the 2026-08-15
workstation lease `.102`) are in `k3s_api_revoked_sources`. Operator access
is `bridge up k3s-api-railiance01` (local 16444). Live prune of the three
remaining grants is T02.

```task
id: RAIL-HO-WP-0009-T06
status: done
priority: high
state_hub_task_id: "e498c6d5-3e29-5dea-80d2-08c712b15466"
```

**Flannel VXLAN is declared open to Anywhere.** Found 2026-08-12 by the newly
tightened Goss assertion — the first defect the conformance check caught on its
own.

`- name: Allow Flannel VXLAN in UFW` declares `8472/udp` with **no source
restriction**, the same defect pattern as the k3s API rule. The rule is
currently **absent** from `Railiance01`, so there is no live exposure — but
converging the base role would *add* it, opening VXLAN to the internet and
introducing the exact class of defect this workplan exists to remove.

VXLAN on 8472/udp accepts encapsulated frames; exposing it publicly invites
injection into the pod network. Restrict it to cluster node addresses, or omit
it entirely while the cluster is single-node.

Note the sequencing risk: `RAIL-BS-WP-0007` (ThreePhoenix HA, multi-node) will
need working VXLAN between nodes. Resolve this before that lands, or the fix
will be made under delivery pressure.

**Done 2026-08-15.** `flannel_vxlan_allowed_sources` defaults to empty; the
role no longer adds a blanket 8472/udp allow and deletes one if present.
Goss asserts 8472 is never `ALLOW Anywhere` and that the non-Anywhere count
equals the declared peer list. When ThreePhoenix adds a second node, put its
address in that list before converging firewall.

```task
id: RAIL-HO-WP-0009-T05
status: done
priority: medium
state_hub_task_id: "ac0c3e8e-3384-5765-8ad1-98ffa68f19c1"
```

Propose a declared-vs-live conformance check for firewall state, and route it to
whoever owns the conformance loop. This defect is a concrete instance of the
unowned **Q7 Governance and Change Management** gap recorded in
`railiance-platform/ArchitectureBlueprint.md` §5.3 — a UFW declared-vs-live
comparison is a small, sharp first thing for that loop to do.

The existing Goss verification suite is the natural home for the check itself;
what is missing is the loop that runs it and reacts.

**Progress 2026-08-12 — the check now exists.** `goss/baseline.yaml` became
`goss/baseline.yaml.j2`, rendered per host from the *same* inventory declaration
that converges it (`k3s_api_allowed_sources`, `k3s_api_revoked_sources`), so the
assertion cannot drift from the declaration it checks. The old assertion matched
`/6443\/tcp.*ALLOW/` — the port is allowed, but not from whom — and passed
identically whether the API was restricted to one address or open to the world.
It now asserts each declared source is present, that no revoked source remains,
and that 6443 is never `ALLOW Anywhere`.

`verify.yaml` also gained the `vars_files` the bootstrap play already had;
without it the firewall assertions rendered empty and silently asserted nothing.

Result on `Railiance01`: 32 assertions, 31 pass, one genuine failure (T06). What
remains for this task is the *loop* — scheduling it and routing failures
somewhere they are seen.

**Done 2026-08-15.** The check now also asserts exact allowlist *size*, so an
extra hand grant fails. The loop:

- hourly `railiance-goss-baseline.timer` on each host after `make verify`
- TAP + `/var/lib/railiance/goss/last.status` + `FAILED` flag
- optional State Hub POST on pass↔fail via `RAILIANCE_GOSS_NOTIFY_URL`
- `make goss-status` / `make verify-host`

Routed to `railiance-master` (ArchitectureBlueprint §5.3, Q7 owner) in
`docs/conformance-loop.md`. Family-level reaction (page a human, block a
rollout) is not this repo.

## Outcome

Finished 2026-08-15. Declaration and live Railiance01 UFW now match ADR-005:
no public 6443 grants, no blanket VXLAN allow, Nydus 2224 declared. CoulombCore
UFW remains unmanaged (`ufw_manage: false`).