Document actual infrastructure scope and intent gaps
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02994-7685-7940-bf34-3555b8256018
This commit is contained in:
codex 2026-08-23 10:45:02 +02:00
parent 186b030bee
commit 3734a1c3bc
2 changed files with 273 additions and 92 deletions

241
SCOPE.md
View file

@ -1,134 +1,179 @@
# SCOPE # SCOPE
> This file helps you quickly understand what this repository is about, > This file describes the repository as it works today. Aspirational direction
> when it is relevant, and when it is not. > belongs in `INTENT.md`; known differences are recorded under `history/`.
> It is intentionally lightweight and may be incomplete.
--- ---
## One-liner ## One-liner
S1 Infrastructure Substrate of the Railiance OAS Stack — Git-driven OS provisioning, security hardening, and server baseline using Terraform, cloud-init, and Ansible. The S1 host substrate for Railiance: source-backed inventory, Ansible OS
convergence, host firewall policy, SSH bootstrap, and recurring baseline
verification for the existing Railiance servers.
--- ---
## Core Idea ## Core Idea
Railiance is structured as five independent repos per OAS Stack layer. This repo `railiance-infra` owns host-level facts and controls below Kubernetes. It keeps
is S1 — the foundation and the canonical ownership home for infrastructure the current server identities in source, converges supported Linux hosts toward
substrate facts. It provisions bare-metal or cloud servers (Hetzner, a security baseline, and checks the resulting live state. A higher layer should
HostEurope), hardens the OS, manages secrets (SOPS/age), and validates the receive a reachable, hardened host rather than needing to configure the host
resulting baseline with Goss tests. Future `reef-*` repos will model itself.
purpose-bound substrates, but the source-backed OS baseline, inventory, and
server identity facts start here. S1 must be converged and verified before any The two current managed hosts are existing Host Europe virtual servers. The
higher layer (Kubernetes, platform, etc.) can run. repository also contains a Hetzner Terraform and cloud-init path, but that path
is not an operable representation of the current inventory: it supports only
Hetzner and expects provisioning fields that the two adopted Host Europe
entries do not contain.
--- ---
## In Scope ## In Scope
- OS provisioning via Terraform (Hetzner, HostEurope providers) - Canonical connection inventory for `Railiance01` and `CoulombCore` in
- First-boot configuration via cloud-init `inventory/servers.yaml`
- OS convergence via Ansible (base, security, sops_agent roles) - Dynamic Ansible inventory, including per-host exceptions
- Security hardening and firewall rules - Debian/Ubuntu host convergence for base packages, SSH hardening, fail2ban,
- Secret management: SOPS/age encryption at rest in Git timezone, swap, user resource limits, and host access keys
- Goss specification and test suite for OS baseline validation - Declared UFW management where `ufw_manage` is enabled
- Server inventory management (`inventory/servers.yaml` — source of truth) - Tunnel-only k3s API policy, source-restricted Flannel grants, and removal of
- SSH access management retired public grants
- Canonical S1 identity and substrate-fact base for future `reef-*` repos - Grant validation before opening public web ports 80/443 for a reef
- Source-backed server and workstation substrate facts needed by higher-layer - Installation of OpenBao SSH user-CA trust and per-user principals
architecture work - Installation of SOPS/age tooling and SOPS-encrypted storage of the Hetzner
provider token
- Goss host checks, TAP evidence, and an hourly on-host pass/fail signal
- Non-secret `Railiance01` identity, capacity, utilization, and host-operations
evidence for downstream resource accounting
- Local age-encrypted snapshots of selected S1 OS configuration and installed
package selections
- A reference NetKingdom host-bootstrap capability declaration
--- ---
## Out of Scope ## Out of Scope
- Kubernetes runtime → railiance-cluster (S2) - Kubernetes installation and runtime management (`railiance-cluster`, S2)
- Platform services → railiance-platform (S3) - OpenBao, databases, ingress, Forgejo, and other platform services
- Developer tooling → railiance-enablement (S4) (`railiance-platform`, S3)
- Application deployments → railiance-apps (S5) - Developer tooling and application deployment (S4/S5)
- Workload execution semantics or rail contracts → `rail-*` repos - Workload execution contracts (`rail-*`) and workload packaging (`rapp-*`)
- Purpose-specific reef repo ownership and workload-placement policy beyond the - Reef topology, workload placement, and exposure decisions; this repo only
S1 source-backed substrate facts enforces a supplied substrate grant
- No cross-layer re-configuration from higher layers - Provider billing, contracts, lifecycle dates, and cost records
- Secret issuance or live credential custody; this repo encrypts selected
material at rest and installs client tooling
- Host Europe provisioning or lifecycle automation; the current Host Europe
servers are adopted resources
- Cluster, database, or application backup and restore
Historical Forgejo migration and restore-drill material remains in this repo,
but it is not an S1 capability or an ownership precedent.
--- ---
## Relevant When ## Relevant When
- Provisioning new servers for the Railiance stack - Adding or changing a managed host identity
- OS hardening, Ansible convergence, or Goss verification - Converging or auditing host packages, SSH, fail2ban, UFW, swap, or resource
- Managing server inventory or SSH access limits
- Rotating SOPS/age keys or updating secrets - Changing host-level network grants or proving k3s API exposure remains
- Preparing or validating first-wave `reef-*` rollout identity facts tunnel-only
- Installing SSH CA trust or automation public keys on a host
- Running or collecting host baseline checks
- Producing non-secret host-capacity evidence
- Developing or repairing the Hetzner provisioning path
--- ---
## Not Relevant When ## Not Relevant When
- Kubernetes, platform services, or application work (wrong layer) - The change is to Kubernetes objects, cluster components, platform services,
- Server is already provisioned and converged (use cluster/platform repos) or applications
- The request is for a secret value, login token, provider contract, or invoice
- The decision concerns which workloads or rails belong in a reef
- A backup or restore concerns persistent application or cluster data
An already-provisioned host can still be in scope: convergence, firewall
maintenance, evidence collection, and drift checks are ongoing S1 work.
--- ---
## Current State ## Current State
- Status: maintained / productive - Status: maintained and operational for adopted-host convergence and
- Implementation: HostEurope substrate baseline active for `Railiance01` and verification
`CoulombCore`; server spec + test suite active; first reef rollout source map - Managed inventory: `Railiance01` (`92.205.62.239`) and `CoulombCore`
defined. Railiance is classified along four repo-family axes (`railiance-*`, (`92.205.130.254`), both reached as user `tegwick`
`rail-*`, `rapp-*`, `reef-*`), of which five `railiance-*` repos cover S1S5; - `Railiance01`: UFW is managed from this repo; the k3s API is tunnel-only and
this file previously said "5-repo stack architecture", which predates that the Host Europe Nydus exception is declared
model - `CoulombCore`: UFW is deliberately unmanaged because its live packet filter
- Stability: high for the current single-server and transitional two-server has not been migrated safely to this repo's UFW model
substrate reality; proven in production on `92.205.62.239` - Verification: Goss can run on demand and hourly on-host. `CoulombCore` is a
- Usage: foundation for all Railiance deployments; canonical S1 source for documented expected failure for the uniform UFW-active assertion, so the
higher-layer and future reef planning. `railiance-hosts` is **superseded** by repository does not currently provide an all-host green handoff gate
this repo and carries a banner saying so; its retirement is pending in - Provisioning: a Hetzner-only Terraform template and helper scripts exist.
`railiance-master` They do not currently plan against the mixed/adopted inventory and do not
- **Firewall posture**: `RAIL-HO-WP-0009` finished. k3s API is tunnel-only provision either live Host Europe server
(ADR-005); live `Railiance01` UFW matches the declaration. CoulombCore - Secrets: `secrets/hetzner-token.yaml` is SOPS-encrypted. The Ansible
UFW stays unmanaged (`ufw_manage: false`) `sops_agent` role installs tools but intentionally does not place a private
- **Resource evidence**: `RAIL-HO-WP-0008` finished. Non-secret identity, age key on a host
capacity observations, and host-ops labor live under - Evidence: the latest committed `Railiance01` capacity observation is a
`docs/evidence/resource-hosteurope-railiance01/` point-in-time record, not continuous resource telemetry
--- ---
## How It Fits ## How It Fits
- Upstream dependencies: Terraform, Ansible, SOPS/age (external tools); cloud provider APIs - Upstream tools: Ansible, Goss, SOPS/age, Terraform, SSH, and provider APIs
- Downstream consumers: railiance-cluster (S2) depends on a converged, verified OS from this layer; all higher layers transitively depend on S1 - Downstream: `railiance-cluster` depends on host reachability and an acceptable
- Future substrate-boundary consumers: first-wave `reef-*` repos should project S1 posture; higher layers depend on it transitively
from the source-backed facts here rather than creating a second S1 inventory - Access path: `ops-bridge` supplies the SSH tunnels used for private cluster
- Often used with: railiance-cluster (next layer), ops-bridge (SSH tunnel for remote State Hub access) and State Hub access; this repo declares host-side SSH access and firewall
posture but does not own tunnel orchestration
- Resource accounting: `resource-control` consumes the non-secret evidence
interface under `docs/evidence/resource-hosteurope-railiance01/`; `fin-hub`
owns booked cost
--- ---
## Terminology ## Terminology
- Preferred terms: OAS Stack Level S1, convergence, verification, SOPS/age, Goss specification, boundary rule - **Adopted host**: an existing provider resource represented and managed here,
- Potentially confusing terms: "convergence" = applying Ansible to reach desired state; "verification" = running Goss tests to validate it but not created by this repo
- **Convergence**: applying Ansible roles to move a host toward declared state
- **Verification**: evaluating live host state with Goss
- **Substrate grant**: an approved host-level network opening supplied by the
owning reef declaration
- **S1**: the infrastructure-substrate layer below the cluster runtime
--- ---
## Related / Overlapping ## Related / Overlapping
- `railiance-cluster` (S2) — consumes the OS baseline provided by S1 - `railiance-cluster` — owns the S2 Kubernetes runtime
- `railiance-hosts` — predecessor or migration-duplicate S1 line; not the - `railiance-platform` — owns S3 shared services and service data protection
canonical repo for new architecture work - `ops-bridge` — owns workstation-to-host tunnel orchestration
- `ops-bridge` — used to reach local State Hub from remote HostEurope server - `resource-control` — owns the resource portfolio assembled from evidence
produced here
- `reef-*` repos — own substrate grouping and exposure intent
- `railiance-hosts` — superseded predecessor; not a second source of truth
--- ---
## Getting Oriented ## Getting Oriented
- Start with: `CLAUDE.md` (session protocol, remote execution), `README.md` (provisioning workflow) - Session and work routing: `AGENTS.md` and `.custodian-brief.md`
- Key files / directories: `inventory/servers.yaml` (authoritative server - Current host identities: `inventory/servers.yaml`
list), `ansible/` (playbooks/roles), `terraform/` (provider configs), `goss/` - Host convergence: `ansible/playbooks/bootstrap.yaml` and `ansible/roles/`
(spec + tests), `docs/reef-first-wave-source-map.md`, - Host-specific posture: `ansible/inventory/host_vars/`
`docs/adr/ADR-003-railiance-5repo-stack-architecture.md` - Firewall defaults: `ansible/roles/base/defaults/main.yml`
- Entry points: `make tf-plan`, `make tf-apply`, `make converge`, `make verify` - Verification: `goss/baseline.yaml.j2`, `docs/verification.md`, and
`docs/conformance-loop.md`
- Hetzner prototype path: `terraform/hetzner/` and `docs/provisioning.md`
- Operator entry points: `make converge`, `make converge-check`,
`make converge-firewall`, `make verify`, and `make goss-status`
--- ---
@ -136,31 +181,43 @@ higher layer (Kubernetes, platform, etc.) can run.
```capability ```capability
type: infrastructure type: infrastructure
title: Server provisioning (Terraform) title: Adopted-host inventory and convergence
description: Provision bare-metal and cloud servers on Hetzner and HostEurope via Terraform with cloud-init first-boot configuration. description: Resolve the two current server identities into Ansible inventory and converge supported Debian/Ubuntu host packages, SSH posture, fail2ban, swap, resource limits, and access keys.
keywords: [terraform, server, provisioning, hetzner, hosteurope, cloud-init, infrastructure] keywords: [ansible, inventory, host, convergence, hardening, linux]
```
```capability
type: infrastructure
title: OS hardening and convergence (Ansible)
description: Harden and converge server OS via Ansible (base, security, sops_agent roles) with Goss test suite for baseline validation.
keywords: [ansible, os, hardening, convergence, goss, security, baseline, validation]
``` ```
```capability ```capability
type: security type: security
title: Secret management (SOPS/age) title: Host firewall grant enforcement
description: Manage encrypted secrets at rest in Git using SOPS/age — encrypt, rotate, and distribute secrets for Railiance infrastructure components. description: Converge declared UFW rules, keep the k3s API tunnel-only, restrict Flannel peers, revoke retired sources, and require a reef exposure grant before opening public web ports.
keywords: [sops, age, secrets, encryption, gitops, key-rotation, credential] keywords: [ufw, firewall, k3s, tunnel, flannel, reef, exposure]
```
```capability
type: infrastructure
title: Recurring host baseline verification
description: Render inventory-aware Goss checks, run them on demand or hourly on-host, retain local failure state, and collect TAP evidence; the current all-host gate has a documented CoulombCore exception.
keywords: [goss, verification, drift, systemd-timer, tap, evidence]
```
```capability
type: security
title: SSH trust bootstrap
description: Install automation public keys and OpenBao SSH user-CA trust with host-specific authorized principals; certificate issuance remains with ops-warden.
keywords: [ssh, certificate-authority, openbao, principals, access]
```
```capability
type: infrastructure
title: Host capacity evidence interface
description: Collect a bounded, non-secret Railiance01 observation across provider metadata, operating-system capacity, and k3s allocatable state for resource-control.
keywords: [capacity, evidence, hosteurope, resource-control, inventory]
``` ```
--- ---
## Notes ## Notes
Targets two current server substrates: `CoulombCore` (`92.205.130.254`) and The exact differences between this operational scope and the aspirational
`Railiance01` (`92.205.62.239`). The first-wave reef rollout also recognizes a mission in `INTENT.md` are assessed in
grouped operator workstation substrate. State Hub access uses ops-bridge — `history/2026-08-23-scope-against-intent.md`.
`bridge up state-hub-coulombcore` or `bridge up state-hub-railiance01` from the
workstation (see ADR-004).

View file

@ -0,0 +1,124 @@
# Scope Against Intent Assessment — 2026-08-23
## Purpose
This assessment compares the repository's executable state on 2026-08-23 with
the stable direction in `INTENT.md`. It describes gaps; it does not create or
authorize implementation work.
The assessment was made after confirming that the repository had no actionable
local work: no ready, active, or blocked workplan; no task assigned to
`railiance-infra` in `todo`, `progress`, or `wait`; no unread inbox message; no
active repo dispatch; and no open Forgejo issue or pull request.
## Overall Assessment
The repository is a useful host-convergence and verification substrate, but it
does not yet meet the full intent of reproducibly turning bare machines into a
verified handoff. Its strongest implemented area is post-provisioning host
control. Its largest gap is the seam before that: neither current Host Europe
server is provisioned from source, and the only Terraform module cannot consume
the current inventory.
The intent is therefore **partially realized**. The repo can maintain important
parts of S1 for existing hosts; it cannot currently rebuild the live S1 estate
from its declared source alone.
## Principle-by-Principle Assessment
| Intent principle | Assessment | Current evidence | Gap |
| --- | --- | --- | --- |
| Declarative and reproducible | Partial | Ansible roles, firewall variables, a baseline spec, dynamic inventory, and a Hetzner Terraform module are source-controlled. | The live hosts are adopted Host Europe resources. The Terraform module iterates every inventory entry but requires `type`, `region`, `image`, and `role`, which the current entries lack. Host Europe lifecycle is manual, and no source path can recreate the current estate end to end. |
| Hardened by default | Partial | SSH password/root login are disabled by Ansible; fail2ban and UFW policy exist; k3s API access is tunnel-only; public web ports require a reef grant. | Hardening is applied after access exists rather than guaranteed as one provisioning transaction. `CoulombCore` deliberately bypasses UFW management. The generic cloud-init template still contains a placeholder SSH key and is not the Terraform template. |
| Verified before handoff | Partial | Goss renders inventory-aware checks, emits TAP, and runs hourly plus five minutes after boot. | The uniform baseline intentionally fails on `CoulombCore` because UFW is inactive. Verification is not a mandatory provisioning/handoff gate, and no repository CI exercises the Ansible/Goss contract. |
| Recorded source of truth | Partial | `inventory/servers.yaml` records the two names, IPs, and SSH users; host variables record the CoulombCore exception; Railiance01 has a non-secret evidence interface. | The inventory is authoritative for connection identity but not full resource shape or lifecycle. Current-host provider, location, flavor, image, and role are not modeled there. Evidence coverage is asymmetric and point-in-time. |
| Secure at rest | Partial | The Hetzner token is SOPS/age encrypted and repo hooks check files under `secrets/`. The host role installs SOPS/age without placing the private age key. | Rotation is manual. `inventory/group_vars/secrets.sops.yaml` is a plaintext placeholder outside the hook's checked path, while the docs describe it as encrypted input. The implementation and documentation disagree about the provider-token source. |
| Foundation, not tenant | Mostly met | Active convergence, firewall, SSH, inventory, and verification code stay at host level; cluster and platform ownership are documented elsewhere. | Historical Forgejo migration, runner, and restore-drill artifacts remain in the repo and blur discovery, although they do not form the current S1 execution path. |
## Direction-of-Evolution Assessment
### Stronger reproducibility and drift detection — partial
The hourly Goss timer is a real drift detector and its firewall assertions are
derived from the same inventory variables used for convergence. It records a
host flag and journald event, with an optional State Hub progress post.
Gaps:
- No central reaction is guaranteed; paging, task creation, and rollout
blocking belong elsewhere and are not wired as a dependable end-to-end loop.
- `spec/server-baseline.yaml` is described as authoritative but is not consumed
by Ansible or Goss. Maintaining the spec, roles, and template remains a manual
three-way synchronization task.
- The declared spec and convergence are already imperfectly aligned: for
example, the spec requires `htop` and a `tegwick` passwordless-sudo posture,
while the base role does not establish both of those properties.
- A known expected failure means the all-host signal cannot distinguish only
new regressions without host-profile semantics.
### Broader provider support — gap
Only Hetzner has Terraform resources. Host Europe is represented by adopted
identity/evidence and manual lifecycle notes, not a provider implementation.
The current single inventory cannot safely mix adopted Host Europe records with
the Hetzner `for_each` model.
### Continuous baseline verification — partial
The on-host hourly timer satisfies the cadence part of the direction. It is not
yet a complete continuous assurance capability because deployment does not
require a fresh green result, the failure route is optional, and one managed
host is expected to remain red.
### Automated rotation of at-rest secret material — gap
The repository provides `sops --rotate` as an operator command. It has no
scheduled or policy-driven rotation, recipient-age enforcement, rotation
receipt, or consumer rollout verification.
### Self-evidencing, auditable provisioning — partial
Goss TAP reports, capacity observations, ADRs, and workplan evidence are useful
records. The actual create/apply/converge sequence has no unified immutable
receipt tying together source revision, provider plan, created resource,
cloud-init completion, Ansible result, and a fresh verification result.
The local S1 backup command encrypts selected `/etc` files and package
selections, but it is not scheduled, does not copy off-host, and has no restore
procedure or drill in this repo. It should not be treated as proof that the
substrate is recoverable.
## Concrete Gaps, Ordered by Leverage
1. **Separate adopted and provisionable inventory semantics.** Add an explicit
provider/lifecycle mode and validate the schema, or give each provider a
filtered declaration. A Terraform plan must never interpret an adopted Host
Europe entry as a Hetzner resource.
2. **Make one baseline contract executable.** Generate convergence and checks
from a shared model, or add automated tests that fail when the human spec,
Ansible roles, and Goss assertions diverge.
3. **Define host profiles and a real handoff gate.** Express the CoulombCore
firewall exception in verification rather than accepting a permanently red
host, and require a fresh result before S2 handoff.
4. **Repair the secret-source contract.** Remove or encrypt the plaintext
placeholder file, extend plaintext checks to every declared secret path, and
make the documentation name the provider-token source actually used by the
Makefile.
5. **Capture provisioning receipts.** Record source revision, plan/apply
identity, provider resource identifiers, convergence result, and verification
result without recording secrets.
6. **Close recovery and rotation loops.** Automate age-recipient rotation with
evidence, and give the local S1 backup a scheduled off-host copy plus a
tested restore procedure if live-state backup remains part of this repo.
7. **Reduce scope noise.** Archive or relocate historical Forgejo/platform
artifacts so current S1 ownership is evident from the file tree as well as
from `SCOPE.md`.
## Bottom Line
Today the repo can answer: **"How do we keep these existing Linux hosts closer
to a hardened, observable S1 baseline?"**
It cannot yet answer the full guiding question in `INTENT.md`: **"Can we rebuild
the live substrate from source and prove, through a mandatory auditable gate,
that it is ready before higher layers use it?"**