railiance-infra/docs/verification.md

Server Verification

railiance-infra ships a declarative baseline spec and a Goss test suite that asserts every managed node matches it. This replaces manual spot-checks with a reproducible, CI-friendly pass/fail verdict.

The spec

spec/server-baseline.yaml is executable through scripts/baseline_contract.py. The dynamic inventory resolves its selected profile into the variables consumed by both Ansible and Goss. It covers:

• Firewall — UFW active, default deny inbound, required ports allowed (SSH 22/tcp; HostEurope Nydus 2224/tcp). The k3s API (6443/tcp) is tunnel-only (ADR-005): k3s_api_allowed_sources is empty, and k3s_api_revoked_sources is pruned on a firewall-tagged converge. Flannel VXLAN (8472/udp) is omitted while the cluster is single-node; peer addresses go in flannel_vxlan_allowed_sources when a second node appears. CoulombCore selects an explicit external-firewall profile and verifies its replacement INPUT default-drop control without rewriting it. A host with an empty 6443 allowlist is recoverable over SSH.
• SSH daemon — root login disabled, password auth disabled, pubkey auth enabled
• Services — ufw, fail2ban, ssh.socket enabled and running
• Packages — ufw, fail2ban, git, curl, vim, htop (age and sops installed as binaries)
• Users — admin user with bash shell and passwordless sudo
• Security — fail2ban sshd jail active, HISTCONTROL=ignorespace in /etc/profile.d/

When you change governed desired state, update this file. The consumers use the same resolved variables; make validate-baseline fails if either consumer is disconnected from the contract.

Running verification

make verify

This runs ansible/playbooks/verify.yaml against all hosts. The playbook:

1. Reads the installed Goss binary and /etc/goss/baseline.yaml metadata.
2. Renders the selected profile on the controller and requires its SHA-256 digest to match the installed baseline exactly.
3. Runs the fixed goss validate --format tap argument vector.
4. Fails the play if the surface is missing/stale or any assertion fails.
5. Writes TAP evidence only on the controller under reports/.

scripts/handoff_contract.py statically rejects remote modules other than stat, assert, and the exact Goss command. Controller writes must be delegated to localhost and remain under reports/. It also pins every Goss command assertion and both profile firewall probes to their reviewed read-only set. make verify therefore does not install packages, rewrite files, reload systemd, or change services on a managed host.

All assertions passed → exit 0 One or more assertions FAILED → exit non-zero, TAP report in reports/

After convergence

The standard workflow after converging a new or updated node:

make converge   # bring the node to the desired state
make verify     # assert it got there

Run make status for a quick human-readable summary; run make verify when you need a structured, automatable check.

If verification fails because the installed surface is stale, review the rendered changes and use the explicit mutating interface for only the intended host:

make verify-refresh-host \
  HOST=Railiance01 \
  APPROVE_VERIFY_REFRESH=REFRESH-GOSS-Railiance01

Refreshing every host requires the distinct approval value APPROVE_VERIFY_REFRESH=REFRESH-GOSS-ALL. Both refresh targets install or update Goss, its baseline, wrapper, service, and timer before running checks.

Goss test file

goss/baseline.yaml.j2 is rendered per host from the same inventory lists that converge UFW. The mapping is:

spec section	Goss resource
firewall	command: ufw status stdout patterns
ssh	file: /etc/ssh/sshd_config.d/10-hardening.conf contains
services	service: blocks
packages	package: blocks
users	user: block + command: grep NOPASSWD
security.histcontrol	command: grep -r HISTCONTROL /etc/profile.d/
security.fail2ban_jails	command: fail2ban-client status sshd
age, sops (binary installs)	command: test -x /usr/local/bin/{age,sops}

Adding new assertions

1. Add the desired state to spec/server-baseline.yaml.
2. If it introduces a new control kind, teach both consumers that kind.
3. Run make validate-baseline and the unit tests.
4. Run the applicable reviewed convergence/refresh command, then use make verify-host HOST=… as the read-only acceptance check.

An hourly on-host timer (railiance-goss-baseline.timer) reruns the last rendered baseline. See docs/conformance-loop.md.

Reports

TAP reports are transient under ignored reports/. They are machine-readable and suitable for CI pipelines. Promote deliberately retained evidence under docs/evidence/ after reviewing it for safe metadata. A cleanup policy for old reports is tracked as extension point EP 78ef4879.