Implement reproducible S1 handoff contracts
Some checks failed
CI Smoke / source-contract (push) Failing after 2s
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02994-7685-7940-bf34-3555b8256018
This commit is contained in:
codex 2026-08-23 12:02:23 +02:00
parent c8cb1c8edf
commit b93af8cc78
44 changed files with 2035 additions and 342 deletions

View file

@ -54,8 +54,9 @@ handoff:
Do not build a second alert path in `railiance-telemetry` for the same
"check failed → someone sees it" plumbing. Item 9b should reuse this.
## Known expected fail
## Host profiles
`CoulombCore` has UFW inactive. The baseline asserts `Status: active`. The
timer will fail there until an explicit decision enables UFW or the host is
removed from the verify inventory. That failure is evidence, not noise.
`Railiance01` selects `ufw-managed`; `CoulombCore` selects
`external-firewall`. The latter does not turn an unmanaged control into a
pass: it asserts the documented replacement control, an iptables INPUT
default-drop policy. An absent replacement control is a failure.

View file

@ -23,8 +23,8 @@ make converge
```
This will:
1. Decrypt secrets locally (with your age key)
2. Run the Ansible playbooks against all hosts in your `inventory/servers.yaml`
1. Validate and resolve each host's baseline profile from source
2. Run the Ansible playbooks against all hosts in `inventory/servers.yaml`
3. Apply the baseline security and tooling configuration
## Verifying
@ -48,5 +48,6 @@ make status
## Notes
- Convergence is **idempotent**: re-running it will not break your server.
- Only your workstation (control node) needs the age private key; hosts never see it.
- Convergence does not load provider credentials. The `sops_agent` role installs
SOPS/age clients but does not place an age private key on a host.
- Additional roles (e.g. WireGuard, Kubernetes, apps) can be layered later.

View file

@ -0,0 +1,41 @@
{
"created_at": "2026-08-23T09:30:00Z",
"event_type": "provisioning-chain",
"inventory_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"phases": [
{
"evidence_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"phase": "plan",
"status": "pass"
},
{
"evidence_sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"phase": "apply",
"provider_resource_ids": ["synthetic-provider-resource"],
"status": "pass"
},
{
"evidence_sha256": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"phase": "cloud-init",
"status": "pass"
},
{
"evidence_sha256": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
"hosts": ["synthetic-host"],
"phase": "convergence",
"status": "pass"
},
{
"evidence_sha256": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
"fresh_until": "2026-08-24T09:30:00Z",
"hosts": ["synthetic-host"],
"phase": "verification",
"status": "pass"
}
],
"receipt_id": "6f974742-7a0f-4fc4-a931-7ff68dc8eb84",
"schema_version": "1.0",
"source_revision": "3734a1c",
"status": "pass",
"synthetic": true
}

View file

@ -1,6 +1,7 @@
# 🚀 Provisioning Servers with railiance-infra
This guide explains **where you declare servers**, **how Terraform uses that declaration**, and **how to provision** (and later destroy) machines on Hetzner.
This guide explains how adopted hosts and provider-managed Hetzner hosts share
an inventory without sharing lifecycle behavior.
---
@ -11,13 +12,16 @@ This script will:
1. Add the new host entry to `inventory/servers.yaml`
2. Decrypt your Hetzner API token with SOPS
3. Run Terraform (`init/plan/apply`) to provision the server
4. Print the IPv4 address and a ready-to-use SSH command
3. Run Terraform init and plan
4. Apply only when the operator supplies the explicit `--apply` flag after
reviewing the plan
**Example:**
```bash
scripts/hcloud_new_server.sh core-01 --type cpx11 --region nbg1 --role core
# after review and the required approval:
scripts/hcloud_new_server.sh core-01 --type cpx11 --region nbg1 --role core --apply
```
This will create a small cpx11 instance in the Nuremberg (nbg1) region, tagged with the role core.
@ -33,17 +37,24 @@ ssh admin@<printed-ip>
## 1) Where you define servers
All desired hosts live in **`inventory/servers.yaml`**. Each entry is a simple YAML object with the required attributes:
All host identities live in **`inventory/servers.yaml`**. Run
`make validate-inventory` before Terraform or Ansible. Adopted hosts have a
provider address but no `provisioning` block. Provider-managed Hetzner hosts
have no committed IP and use this shape:
```yaml
servers:
- name: core-01
labels: [core, wireguard, git]
role: "core"
region: "nbg1" # Hetzner location (e.g., nbg1, fsn1, hel1)
type: "cpx21" # Hetzner server type/flavor
image: "ubuntu-24.04" # OS image slug
ssh_user: "admin" # bootstrap user (cloud-init creates this)
provider: hetzner
lifecycle_mode: provider-managed
ssh_user: admin
baseline_profile: ufw-managed
provisioning:
server_type: cpx21
region: nbg1
image: ubuntu-24.04
role: core
labels: [core, wireguard, git]
```
> Tip: Keep **names stable**. Renaming a server in this file makes Terraform think the old one was destroyed and a new one should be created.
@ -73,7 +84,8 @@ scripts/hcloud_new_server.sh web-01 --type cpx21 --region nbg1 --role web
## 3) How Terraform uses your declaration
The module at `terraform/hetzner/`:
- Reads `inventory/servers.yaml` (`for_each` over `servers`)
- Selects only records with `provider: hetzner` and
`lifecycle_mode: provider-managed`
- Registers your SSH key from `keys/admin_ssh.pub`
- Injects **cloud-init** that sets up the `admin` user and basic hardening
- Creates/updates/destroys servers to match the YAML
@ -84,17 +96,19 @@ Outputs include a map of server names → IPv4 addresses.
## 4) Provision (create/update)
Make sure your Hetzner API token is present and **SOPS-decryptable** in `inventory/group_vars/secrets.sops.yaml` under `ops.hcloud_token`.
The Hetzner API token is **SOPS-decryptable** at
`secrets/hetzner-token.yaml`, field `hetzner.token`. It is decrypted only into
the invoking process environment.
Then run either:
```bash
# plan and apply in separate steps
make tf-plan
make tf-apply
APPROVE_TF_APPLY=YES make tf-apply
```
or the end-to-end convenience:
```bash
make apply # terraform apply + ansible bootstrap
APPROVE_TF_APPLY=YES make apply # terraform apply + Ansible bootstrap
```
If you used the one-shot script:
@ -124,14 +138,16 @@ make ansible-bootstrap
To remove all servers managed by this repo:
```bash
make tf-destroy
APPROVE_TF_DESTROY=DESTROY-MANAGED-HETZNER make tf-destroy
```
To remove just one server, delete its entry from `inventory/servers.yaml`, commit, then:
To remove one **provider-managed Hetzner** server, delete its entry from
`inventory/servers.yaml`, review the plan, obtain the required approval, then:
```bash
make tf-apply
APPROVE_TF_APPLY=YES make tf-apply
```
Terraform will destroy the missing server and leave others intact.
Terraform will destroy the missing managed Hetzner server and leave others
intact. Adopted Host Europe records are never Terraform resources.
---
@ -139,6 +155,7 @@ Terraform will destroy the missing server and leave others intact.
- **Idempotent:** You can run `make apply` repeatedly; Terraform converges infra, Ansible converges config.
- **SSH keys:** Ensure `keys/admin_ssh.pub` exists before provisioning.
- **Secret token:** The Hetzner API token must be in `inventory/group_vars/secrets.sops.yaml` (encrypted with SOPS).
- **Secret token:** The Hetzner API token is `secrets/hetzner-token.yaml`, field
`hetzner.token`, and must remain SOPS-encrypted.
- **Cloud-init delay:** Allow ~3060s after creation for first-boot tasks before first SSH.
- **Labels & role:** `labels` are freeform tags; `role` can drive Ansible plays as you grow.

39
docs/s1-handoff.md Normal file
View file

@ -0,0 +1,39 @@
# S1 Handoff Gate
`make s1-handoff` is the operator-facing gate from host substrate (S1) to the
cluster layer (S2). It validates inventory and baseline contracts, requires a
clean checkout, runs the applicable Goss profile for every selected host, and
emits a metadata-only JSON receipt under `reports/`.
A passing receipt pins:
- source revision and inventory digest
- every selected host and its baseline profile
- observed time and 24-hour default freshness boundary
- per-host exit status
- SHA-256 digests of the resulting TAP evidence
Any failed host fails the aggregate. A passing receipt without host evidence is
invalid. `make s1-handoff-dry-run` validates local inputs but records
`status: not-run`; it cannot authorize S2 handoff.
The current profiles are:
- `Railiance01`: `ufw-managed`
- `CoulombCore`: `external-firewall`, which verifies the declared iptables
INPUT default-drop replacement control
The verification play installs or refreshes the Goss check surface before it
runs. Treat the first run after source changes as an attended host change:
review the rendered contract and access path first. The command does not run
Terraform, change provider resources, rotate credentials, or converge the base
role.
Receipts validate with:
```bash
python3 scripts/s1_receipt.py reports/s1-handoff-*.json
```
S2 should accept only `status: pass` receipts whose source and inventory match
the intended handoff and whose `fresh_until` has not elapsed.

View file

@ -0,0 +1,10 @@
# Metadata-only example. Copy outside Git for an attended approved rotation.
approved: false
approved_by: "operator-name"
approved_at: "2026-08-23T00:00:00Z"
changes:
- path: secrets/hetzner-token.yaml
before_recipients:
- age1old-example-not-valid
after_recipients:
- age1new-example-not-valid

31
docs/sops-rotation.md Normal file
View file

@ -0,0 +1,31 @@
# Bounded SOPS Recipient Rotation
The default operation is metadata-only and does not decrypt values:
```bash
python3 scripts/sops_rotation.py --check
```
It compares each protected file's public age-recipient metadata with the first
matching rule in `.sops.yaml`. CI runs this check to detect recipient drift.
An attended non-printing decryption check may emit a receipt:
```bash
python3 scripts/sops_rotation.py --check --verify-decryption \
--receipt reports/sops-rotation-check.json
```
Decrypted bytes go directly to the null device. They are not retained in the
receipt or command output.
Actual key updates require `--apply` and an approval YAML containing
`approved: true`, `approved_by`, `approved_at`, and an exact `changes` list from
the current plan. The command fails if that list differs from current metadata.
Review and preserve recovery-key custody before approving recipient removal.
Start from `docs/sops-rotation-approval.example.yaml`; the committed example is
deliberately unapproved and contains no usable recipient.
Rollback is a reviewed restoration of the prior `.sops.yaml` recipient set
followed by the same exact-plan approval, `sops updatekeys`, and non-printing
decryption verification. Git history alone is not recovery-key custody.

View file

@ -6,8 +6,9 @@ a reproducible, CI-friendly pass/fail verdict.
## The spec
`spec/server-baseline.yaml` is the single source of truth for the target state
of every managed node. It covers:
`spec/server-baseline.yaml` is executable through
`scripts/baseline_contract.py`. The dynamic inventory resolves its selected
profile into the variables consumed by both Ansible and Goss. It covers:
- **Firewall** — UFW active, default deny inbound, required ports allowed
(SSH 22/tcp; HostEurope Nydus 2224/tcp). The k3s API (6443/tcp) is
@ -15,16 +16,18 @@ of every managed node. It covers:
`k3s_api_revoked_sources` is pruned on a firewall-tagged converge. Flannel
VXLAN (8472/udp) is omitted while the cluster is single-node; peer
addresses go in `flannel_vxlan_allowed_sources` when a second node appears.
Hosts with `ufw_manage: false` (CoulombCore) are not rewritten. A host with
an empty 6443 allowlist is recoverable over SSH.
`CoulombCore` selects an explicit external-firewall profile and verifies its
replacement INPUT default-drop control without rewriting it. A host with an
empty 6443 allowlist is recoverable over SSH.
- **SSH daemon** — root login disabled, password auth disabled, pubkey auth enabled
- **Services** — ufw, fail2ban, ssh.socket enabled and running
- **Packages** — ufw, fail2ban, git, curl, vim, htop (age and sops installed as binaries)
- **Users** — admin user with bash shell and passwordless sudo
- **Security** — fail2ban sshd jail active, HISTCONTROL=ignorespace in /etc/profile.d/
When you change the desired state of a node, update this file first. Then
update the Ansible role **and** the Goss tests to match.
When you change governed desired state, update this file. The consumers use the
same resolved variables; `make validate-baseline` fails if either consumer is
disconnected from the contract.
## Running verification
@ -74,10 +77,11 @@ that converge UFW. The mapping is:
## Adding new assertions
1. Add the desired state to `spec/server-baseline.yaml`
2. Add the Ansible task to `ansible/roles/base/tasks/main.yml`
3. Add the Goss assertion to `goss/baseline.yaml.j2`
4. Run `make converge-firewall` and `make verify-host HOST=…` to confirm
1. Add the desired state to `spec/server-baseline.yaml`.
2. If it introduces a new control kind, teach both consumers that kind.
3. Run `make validate-baseline` and the unit tests.
4. Run `make converge-firewall` and `make verify-host HOST=…` only in the
appropriate reviewed live-change sequence.
An hourly on-host timer (`railiance-goss-baseline.timer`) reruns the last
rendered baseline. See `docs/conformance-loop.md`.