Implement reproducible S1 handoff contracts
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02994-7685-7940-bf34-3555b8256018
This commit is contained in:
parent
c8cb1c8edf
commit
b93af8cc78
44 changed files with 2035 additions and 342 deletions
|
|
@ -54,8 +54,9 @@ handoff:
|
|||
Do not build a second alert path in `railiance-telemetry` for the same
|
||||
"check failed → someone sees it" plumbing. Item 9b should reuse this.
|
||||
|
||||
## Known expected fail
|
||||
## Host profiles
|
||||
|
||||
`CoulombCore` has UFW inactive. The baseline asserts `Status: active`. The
|
||||
timer will fail there until an explicit decision enables UFW or the host is
|
||||
removed from the verify inventory. That failure is evidence, not noise.
|
||||
`Railiance01` selects `ufw-managed`; `CoulombCore` selects
|
||||
`external-firewall`. The latter does not turn an unmanaged control into a
|
||||
pass: it asserts the documented replacement control, an iptables INPUT
|
||||
default-drop policy. An absent replacement control is a failure.
|
||||
|
|
|
|||
|
|
@ -23,8 +23,8 @@ make converge
|
|||
```
|
||||
|
||||
This will:
|
||||
1. Decrypt secrets locally (with your age key)
|
||||
2. Run the Ansible playbooks against all hosts in your `inventory/servers.yaml`
|
||||
1. Validate and resolve each host's baseline profile from source
|
||||
2. Run the Ansible playbooks against all hosts in `inventory/servers.yaml`
|
||||
3. Apply the baseline security and tooling configuration
|
||||
|
||||
## Verifying
|
||||
|
|
@ -48,5 +48,6 @@ make status
|
|||
## Notes
|
||||
|
||||
- Convergence is **idempotent**: re-running it will not break your server.
|
||||
- Only your workstation (control node) needs the age private key; hosts never see it.
|
||||
- Convergence does not load provider credentials. The `sops_agent` role installs
|
||||
SOPS/age clients but does not place an age private key on a host.
|
||||
- Additional roles (e.g. WireGuard, Kubernetes, apps) can be layered later.
|
||||
|
|
|
|||
41
docs/evidence/s1-receipts/synthetic-provisioning-chain.json
Normal file
41
docs/evidence/s1-receipts/synthetic-provisioning-chain.json
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
{
|
||||
"created_at": "2026-08-23T09:30:00Z",
|
||||
"event_type": "provisioning-chain",
|
||||
"inventory_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"phases": [
|
||||
{
|
||||
"evidence_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"phase": "plan",
|
||||
"status": "pass"
|
||||
},
|
||||
{
|
||||
"evidence_sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"phase": "apply",
|
||||
"provider_resource_ids": ["synthetic-provider-resource"],
|
||||
"status": "pass"
|
||||
},
|
||||
{
|
||||
"evidence_sha256": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
|
||||
"phase": "cloud-init",
|
||||
"status": "pass"
|
||||
},
|
||||
{
|
||||
"evidence_sha256": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||
"hosts": ["synthetic-host"],
|
||||
"phase": "convergence",
|
||||
"status": "pass"
|
||||
},
|
||||
{
|
||||
"evidence_sha256": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
|
||||
"fresh_until": "2026-08-24T09:30:00Z",
|
||||
"hosts": ["synthetic-host"],
|
||||
"phase": "verification",
|
||||
"status": "pass"
|
||||
}
|
||||
],
|
||||
"receipt_id": "6f974742-7a0f-4fc4-a931-7ff68dc8eb84",
|
||||
"schema_version": "1.0",
|
||||
"source_revision": "3734a1c",
|
||||
"status": "pass",
|
||||
"synthetic": true
|
||||
}
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
# 🚀 Provisioning Servers with railiance-infra
|
||||
|
||||
This guide explains **where you declare servers**, **how Terraform uses that declaration**, and **how to provision** (and later destroy) machines on Hetzner.
|
||||
This guide explains how adopted hosts and provider-managed Hetzner hosts share
|
||||
an inventory without sharing lifecycle behavior.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -11,13 +12,16 @@ This script will:
|
|||
|
||||
1. Add the new host entry to `inventory/servers.yaml`
|
||||
2. Decrypt your Hetzner API token with SOPS
|
||||
3. Run Terraform (`init/plan/apply`) to provision the server
|
||||
4. Print the IPv4 address and a ready-to-use SSH command
|
||||
3. Run Terraform init and plan
|
||||
4. Apply only when the operator supplies the explicit `--apply` flag after
|
||||
reviewing the plan
|
||||
|
||||
**Example:**
|
||||
|
||||
```bash
|
||||
scripts/hcloud_new_server.sh core-01 --type cpx11 --region nbg1 --role core
|
||||
# after review and the required approval:
|
||||
scripts/hcloud_new_server.sh core-01 --type cpx11 --region nbg1 --role core --apply
|
||||
```
|
||||
|
||||
This will create a small cpx11 instance in the Nuremberg (nbg1) region, tagged with the role core.
|
||||
|
|
@ -33,17 +37,24 @@ ssh admin@<printed-ip>
|
|||
|
||||
## 1) Where you define servers
|
||||
|
||||
All desired hosts live in **`inventory/servers.yaml`**. Each entry is a simple YAML object with the required attributes:
|
||||
All host identities live in **`inventory/servers.yaml`**. Run
|
||||
`make validate-inventory` before Terraform or Ansible. Adopted hosts have a
|
||||
provider address but no `provisioning` block. Provider-managed Hetzner hosts
|
||||
have no committed IP and use this shape:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- name: core-01
|
||||
labels: [core, wireguard, git]
|
||||
role: "core"
|
||||
region: "nbg1" # Hetzner location (e.g., nbg1, fsn1, hel1)
|
||||
type: "cpx21" # Hetzner server type/flavor
|
||||
image: "ubuntu-24.04" # OS image slug
|
||||
ssh_user: "admin" # bootstrap user (cloud-init creates this)
|
||||
provider: hetzner
|
||||
lifecycle_mode: provider-managed
|
||||
ssh_user: admin
|
||||
baseline_profile: ufw-managed
|
||||
provisioning:
|
||||
server_type: cpx21
|
||||
region: nbg1
|
||||
image: ubuntu-24.04
|
||||
role: core
|
||||
labels: [core, wireguard, git]
|
||||
```
|
||||
|
||||
> Tip: Keep **names stable**. Renaming a server in this file makes Terraform think the old one was destroyed and a new one should be created.
|
||||
|
|
@ -73,7 +84,8 @@ scripts/hcloud_new_server.sh web-01 --type cpx21 --region nbg1 --role web
|
|||
## 3) How Terraform uses your declaration
|
||||
|
||||
The module at `terraform/hetzner/`:
|
||||
- Reads `inventory/servers.yaml` (`for_each` over `servers`)
|
||||
- Selects only records with `provider: hetzner` and
|
||||
`lifecycle_mode: provider-managed`
|
||||
- Registers your SSH key from `keys/admin_ssh.pub`
|
||||
- Injects **cloud-init** that sets up the `admin` user and basic hardening
|
||||
- Creates/updates/destroys servers to match the YAML
|
||||
|
|
@ -84,17 +96,19 @@ Outputs include a map of server names → IPv4 addresses.
|
|||
|
||||
## 4) Provision (create/update)
|
||||
|
||||
Make sure your Hetzner API token is present and **SOPS-decryptable** in `inventory/group_vars/secrets.sops.yaml` under `ops.hcloud_token`.
|
||||
The Hetzner API token is **SOPS-decryptable** at
|
||||
`secrets/hetzner-token.yaml`, field `hetzner.token`. It is decrypted only into
|
||||
the invoking process environment.
|
||||
|
||||
Then run either:
|
||||
```bash
|
||||
# plan and apply in separate steps
|
||||
make tf-plan
|
||||
make tf-apply
|
||||
APPROVE_TF_APPLY=YES make tf-apply
|
||||
```
|
||||
or the end-to-end convenience:
|
||||
```bash
|
||||
make apply # terraform apply + ansible bootstrap
|
||||
APPROVE_TF_APPLY=YES make apply # terraform apply + Ansible bootstrap
|
||||
```
|
||||
|
||||
If you used the one-shot script:
|
||||
|
|
@ -124,14 +138,16 @@ make ansible-bootstrap
|
|||
|
||||
To remove all servers managed by this repo:
|
||||
```bash
|
||||
make tf-destroy
|
||||
APPROVE_TF_DESTROY=DESTROY-MANAGED-HETZNER make tf-destroy
|
||||
```
|
||||
|
||||
To remove just one server, delete its entry from `inventory/servers.yaml`, commit, then:
|
||||
To remove one **provider-managed Hetzner** server, delete its entry from
|
||||
`inventory/servers.yaml`, review the plan, obtain the required approval, then:
|
||||
```bash
|
||||
make tf-apply
|
||||
APPROVE_TF_APPLY=YES make tf-apply
|
||||
```
|
||||
Terraform will destroy the missing server and leave others intact.
|
||||
Terraform will destroy the missing managed Hetzner server and leave others
|
||||
intact. Adopted Host Europe records are never Terraform resources.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -139,6 +155,7 @@ Terraform will destroy the missing server and leave others intact.
|
|||
|
||||
- **Idempotent:** You can run `make apply` repeatedly; Terraform converges infra, Ansible converges config.
|
||||
- **SSH keys:** Ensure `keys/admin_ssh.pub` exists before provisioning.
|
||||
- **Secret token:** The Hetzner API token must be in `inventory/group_vars/secrets.sops.yaml` (encrypted with SOPS).
|
||||
- **Secret token:** The Hetzner API token is `secrets/hetzner-token.yaml`, field
|
||||
`hetzner.token`, and must remain SOPS-encrypted.
|
||||
- **Cloud-init delay:** Allow ~30–60s after creation for first-boot tasks before first SSH.
|
||||
- **Labels & role:** `labels` are freeform tags; `role` can drive Ansible plays as you grow.
|
||||
|
|
|
|||
39
docs/s1-handoff.md
Normal file
39
docs/s1-handoff.md
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# S1 Handoff Gate
|
||||
|
||||
`make s1-handoff` is the operator-facing gate from host substrate (S1) to the
|
||||
cluster layer (S2). It validates inventory and baseline contracts, requires a
|
||||
clean checkout, runs the applicable Goss profile for every selected host, and
|
||||
emits a metadata-only JSON receipt under `reports/`.
|
||||
|
||||
A passing receipt pins:
|
||||
|
||||
- source revision and inventory digest
|
||||
- every selected host and its baseline profile
|
||||
- observed time and 24-hour default freshness boundary
|
||||
- per-host exit status
|
||||
- SHA-256 digests of the resulting TAP evidence
|
||||
|
||||
Any failed host fails the aggregate. A passing receipt without host evidence is
|
||||
invalid. `make s1-handoff-dry-run` validates local inputs but records
|
||||
`status: not-run`; it cannot authorize S2 handoff.
|
||||
|
||||
The current profiles are:
|
||||
|
||||
- `Railiance01`: `ufw-managed`
|
||||
- `CoulombCore`: `external-firewall`, which verifies the declared iptables
|
||||
INPUT default-drop replacement control
|
||||
|
||||
The verification play installs or refreshes the Goss check surface before it
|
||||
runs. Treat the first run after source changes as an attended host change:
|
||||
review the rendered contract and access path first. The command does not run
|
||||
Terraform, change provider resources, rotate credentials, or converge the base
|
||||
role.
|
||||
|
||||
Receipts validate with:
|
||||
|
||||
```bash
|
||||
python3 scripts/s1_receipt.py reports/s1-handoff-*.json
|
||||
```
|
||||
|
||||
S2 should accept only `status: pass` receipts whose source and inventory match
|
||||
the intended handoff and whose `fresh_until` has not elapsed.
|
||||
10
docs/sops-rotation-approval.example.yaml
Normal file
10
docs/sops-rotation-approval.example.yaml
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# Metadata-only example. Copy outside Git for an attended approved rotation.
|
||||
approved: false
|
||||
approved_by: "operator-name"
|
||||
approved_at: "2026-08-23T00:00:00Z"
|
||||
changes:
|
||||
- path: secrets/hetzner-token.yaml
|
||||
before_recipients:
|
||||
- age1old-example-not-valid
|
||||
after_recipients:
|
||||
- age1new-example-not-valid
|
||||
31
docs/sops-rotation.md
Normal file
31
docs/sops-rotation.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Bounded SOPS Recipient Rotation
|
||||
|
||||
The default operation is metadata-only and does not decrypt values:
|
||||
|
||||
```bash
|
||||
python3 scripts/sops_rotation.py --check
|
||||
```
|
||||
|
||||
It compares each protected file's public age-recipient metadata with the first
|
||||
matching rule in `.sops.yaml`. CI runs this check to detect recipient drift.
|
||||
|
||||
An attended non-printing decryption check may emit a receipt:
|
||||
|
||||
```bash
|
||||
python3 scripts/sops_rotation.py --check --verify-decryption \
|
||||
--receipt reports/sops-rotation-check.json
|
||||
```
|
||||
|
||||
Decrypted bytes go directly to the null device. They are not retained in the
|
||||
receipt or command output.
|
||||
|
||||
Actual key updates require `--apply` and an approval YAML containing
|
||||
`approved: true`, `approved_by`, `approved_at`, and an exact `changes` list from
|
||||
the current plan. The command fails if that list differs from current metadata.
|
||||
Review and preserve recovery-key custody before approving recipient removal.
|
||||
Start from `docs/sops-rotation-approval.example.yaml`; the committed example is
|
||||
deliberately unapproved and contains no usable recipient.
|
||||
|
||||
Rollback is a reviewed restoration of the prior `.sops.yaml` recipient set
|
||||
followed by the same exact-plan approval, `sops updatekeys`, and non-printing
|
||||
decryption verification. Git history alone is not recovery-key custody.
|
||||
|
|
@ -6,8 +6,9 @@ a reproducible, CI-friendly pass/fail verdict.
|
|||
|
||||
## The spec
|
||||
|
||||
`spec/server-baseline.yaml` is the single source of truth for the target state
|
||||
of every managed node. It covers:
|
||||
`spec/server-baseline.yaml` is executable through
|
||||
`scripts/baseline_contract.py`. The dynamic inventory resolves its selected
|
||||
profile into the variables consumed by both Ansible and Goss. It covers:
|
||||
|
||||
- **Firewall** — UFW active, default deny inbound, required ports allowed
|
||||
(SSH 22/tcp; HostEurope Nydus 2224/tcp). The k3s API (6443/tcp) is
|
||||
|
|
@ -15,16 +16,18 @@ of every managed node. It covers:
|
|||
`k3s_api_revoked_sources` is pruned on a firewall-tagged converge. Flannel
|
||||
VXLAN (8472/udp) is omitted while the cluster is single-node; peer
|
||||
addresses go in `flannel_vxlan_allowed_sources` when a second node appears.
|
||||
Hosts with `ufw_manage: false` (CoulombCore) are not rewritten. A host with
|
||||
an empty 6443 allowlist is recoverable over SSH.
|
||||
`CoulombCore` selects an explicit external-firewall profile and verifies its
|
||||
replacement INPUT default-drop control without rewriting it. A host with an
|
||||
empty 6443 allowlist is recoverable over SSH.
|
||||
- **SSH daemon** — root login disabled, password auth disabled, pubkey auth enabled
|
||||
- **Services** — ufw, fail2ban, ssh.socket enabled and running
|
||||
- **Packages** — ufw, fail2ban, git, curl, vim, htop (age and sops installed as binaries)
|
||||
- **Users** — admin user with bash shell and passwordless sudo
|
||||
- **Security** — fail2ban sshd jail active, HISTCONTROL=ignorespace in /etc/profile.d/
|
||||
|
||||
When you change the desired state of a node, update this file first. Then
|
||||
update the Ansible role **and** the Goss tests to match.
|
||||
When you change governed desired state, update this file. The consumers use the
|
||||
same resolved variables; `make validate-baseline` fails if either consumer is
|
||||
disconnected from the contract.
|
||||
|
||||
## Running verification
|
||||
|
||||
|
|
@ -74,10 +77,11 @@ that converge UFW. The mapping is:
|
|||
|
||||
## Adding new assertions
|
||||
|
||||
1. Add the desired state to `spec/server-baseline.yaml`
|
||||
2. Add the Ansible task to `ansible/roles/base/tasks/main.yml`
|
||||
3. Add the Goss assertion to `goss/baseline.yaml.j2`
|
||||
4. Run `make converge-firewall` and `make verify-host HOST=…` to confirm
|
||||
1. Add the desired state to `spec/server-baseline.yaml`.
|
||||
2. If it introduces a new control kind, teach both consumers that kind.
|
||||
3. Run `make validate-baseline` and the unit tests.
|
||||
4. Run `make converge-firewall` and `make verify-host HOST=…` only in the
|
||||
appropriate reviewed live-change sequence.
|
||||
|
||||
An hourly on-host timer (`railiance-goss-baseline.timer`) reruns the last
|
||||
rendered baseline. See `docs/conformance-loop.md`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue