Document actual infrastructure scope and intent gaps
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02994-7685-7940-bf34-3555b8256018
This commit is contained in:
codex 2026-08-23 10:45:02 +02:00
parent 186b030bee
commit 3734a1c3bc
2 changed files with 273 additions and 92 deletions

241
SCOPE.md
View file

@ -1,134 +1,179 @@
# SCOPE
> This file helps you quickly understand what this repository is about,
> when it is relevant, and when it is not.
> It is intentionally lightweight and may be incomplete.
> This file describes the repository as it works today. Aspirational direction
> belongs in `INTENT.md`; known differences are recorded under `history/`.
---
## One-liner
S1 Infrastructure Substrate of the Railiance OAS Stack — Git-driven OS provisioning, security hardening, and server baseline using Terraform, cloud-init, and Ansible.
The S1 host substrate for Railiance: source-backed inventory, Ansible OS
convergence, host firewall policy, SSH bootstrap, and recurring baseline
verification for the existing Railiance servers.
---
## Core Idea
Railiance is structured as five independent repos per OAS Stack layer. This repo
is S1 — the foundation and the canonical ownership home for infrastructure
substrate facts. It provisions bare-metal or cloud servers (Hetzner,
HostEurope), hardens the OS, manages secrets (SOPS/age), and validates the
resulting baseline with Goss tests. Future `reef-*` repos will model
purpose-bound substrates, but the source-backed OS baseline, inventory, and
server identity facts start here. S1 must be converged and verified before any
higher layer (Kubernetes, platform, etc.) can run.
`railiance-infra` owns host-level facts and controls below Kubernetes. It keeps
the current server identities in source, converges supported Linux hosts toward
a security baseline, and checks the resulting live state. A higher layer should
receive a reachable, hardened host rather than needing to configure the host
itself.
The two current managed hosts are existing Host Europe virtual servers. The
repository also contains a Hetzner Terraform and cloud-init path, but that path
is not an operable representation of the current inventory: it supports only
Hetzner and expects provisioning fields that the two adopted Host Europe
entries do not contain.
---
## In Scope
- OS provisioning via Terraform (Hetzner, HostEurope providers)
- First-boot configuration via cloud-init
- OS convergence via Ansible (base, security, sops_agent roles)
- Security hardening and firewall rules
- Secret management: SOPS/age encryption at rest in Git
- Goss specification and test suite for OS baseline validation
- Server inventory management (`inventory/servers.yaml` — source of truth)
- SSH access management
- Canonical S1 identity and substrate-fact base for future `reef-*` repos
- Source-backed server and workstation substrate facts needed by higher-layer
architecture work
- Canonical connection inventory for `Railiance01` and `CoulombCore` in
`inventory/servers.yaml`
- Dynamic Ansible inventory, including per-host exceptions
- Debian/Ubuntu host convergence for base packages, SSH hardening, fail2ban,
timezone, swap, user resource limits, and host access keys
- Declared UFW management where `ufw_manage` is enabled
- Tunnel-only k3s API policy, source-restricted Flannel grants, and removal of
retired public grants
- Grant validation before opening public web ports 80/443 for a reef
- Installation of OpenBao SSH user-CA trust and per-user principals
- Installation of SOPS/age tooling and SOPS-encrypted storage of the Hetzner
provider token
- Goss host checks, TAP evidence, and an hourly on-host pass/fail signal
- Non-secret `Railiance01` identity, capacity, utilization, and host-operations
evidence for downstream resource accounting
- Local age-encrypted snapshots of selected S1 OS configuration and installed
package selections
- A reference NetKingdom host-bootstrap capability declaration
---
## Out of Scope
- Kubernetes runtime → railiance-cluster (S2)
- Platform services → railiance-platform (S3)
- Developer tooling → railiance-enablement (S4)
- Application deployments → railiance-apps (S5)
- Workload execution semantics or rail contracts → `rail-*` repos
- Purpose-specific reef repo ownership and workload-placement policy beyond the
S1 source-backed substrate facts
- No cross-layer re-configuration from higher layers
- Kubernetes installation and runtime management (`railiance-cluster`, S2)
- OpenBao, databases, ingress, Forgejo, and other platform services
(`railiance-platform`, S3)
- Developer tooling and application deployment (S4/S5)
- Workload execution contracts (`rail-*`) and workload packaging (`rapp-*`)
- Reef topology, workload placement, and exposure decisions; this repo only
enforces a supplied substrate grant
- Provider billing, contracts, lifecycle dates, and cost records
- Secret issuance or live credential custody; this repo encrypts selected
material at rest and installs client tooling
- Host Europe provisioning or lifecycle automation; the current Host Europe
servers are adopted resources
- Cluster, database, or application backup and restore
Historical Forgejo migration and restore-drill material remains in this repo,
but it is not an S1 capability or an ownership precedent.
---
## Relevant When
- Provisioning new servers for the Railiance stack
- OS hardening, Ansible convergence, or Goss verification
- Managing server inventory or SSH access
- Rotating SOPS/age keys or updating secrets
- Preparing or validating first-wave `reef-*` rollout identity facts
- Adding or changing a managed host identity
- Converging or auditing host packages, SSH, fail2ban, UFW, swap, or resource
limits
- Changing host-level network grants or proving k3s API exposure remains
tunnel-only
- Installing SSH CA trust or automation public keys on a host
- Running or collecting host baseline checks
- Producing non-secret host-capacity evidence
- Developing or repairing the Hetzner provisioning path
---
## Not Relevant When
- Kubernetes, platform services, or application work (wrong layer)
- Server is already provisioned and converged (use cluster/platform repos)
- The change is to Kubernetes objects, cluster components, platform services,
or applications
- The request is for a secret value, login token, provider contract, or invoice
- The decision concerns which workloads or rails belong in a reef
- A backup or restore concerns persistent application or cluster data
An already-provisioned host can still be in scope: convergence, firewall
maintenance, evidence collection, and drift checks are ongoing S1 work.
---
## Current State
- Status: maintained / productive
- Implementation: HostEurope substrate baseline active for `Railiance01` and
`CoulombCore`; server spec + test suite active; first reef rollout source map
defined. Railiance is classified along four repo-family axes (`railiance-*`,
`rail-*`, `rapp-*`, `reef-*`), of which five `railiance-*` repos cover S1S5;
this file previously said "5-repo stack architecture", which predates that
model
- Stability: high for the current single-server and transitional two-server
substrate reality; proven in production on `92.205.62.239`
- Usage: foundation for all Railiance deployments; canonical S1 source for
higher-layer and future reef planning. `railiance-hosts` is **superseded** by
this repo and carries a banner saying so; its retirement is pending in
`railiance-master`
- **Firewall posture**: `RAIL-HO-WP-0009` finished. k3s API is tunnel-only
(ADR-005); live `Railiance01` UFW matches the declaration. CoulombCore
UFW stays unmanaged (`ufw_manage: false`)
- **Resource evidence**: `RAIL-HO-WP-0008` finished. Non-secret identity,
capacity observations, and host-ops labor live under
`docs/evidence/resource-hosteurope-railiance01/`
- Status: maintained and operational for adopted-host convergence and
verification
- Managed inventory: `Railiance01` (`92.205.62.239`) and `CoulombCore`
(`92.205.130.254`), both reached as user `tegwick`
- `Railiance01`: UFW is managed from this repo; the k3s API is tunnel-only and
the Host Europe Nydus exception is declared
- `CoulombCore`: UFW is deliberately unmanaged because its live packet filter
has not been migrated safely to this repo's UFW model
- Verification: Goss can run on demand and hourly on-host. `CoulombCore` is a
documented expected failure for the uniform UFW-active assertion, so the
repository does not currently provide an all-host green handoff gate
- Provisioning: a Hetzner-only Terraform template and helper scripts exist.
They do not currently plan against the mixed/adopted inventory and do not
provision either live Host Europe server
- Secrets: `secrets/hetzner-token.yaml` is SOPS-encrypted. The Ansible
`sops_agent` role installs tools but intentionally does not place a private
age key on a host
- Evidence: the latest committed `Railiance01` capacity observation is a
point-in-time record, not continuous resource telemetry
---
## How It Fits
- Upstream dependencies: Terraform, Ansible, SOPS/age (external tools); cloud provider APIs
- Downstream consumers: railiance-cluster (S2) depends on a converged, verified OS from this layer; all higher layers transitively depend on S1
- Future substrate-boundary consumers: first-wave `reef-*` repos should project
from the source-backed facts here rather than creating a second S1 inventory
- Often used with: railiance-cluster (next layer), ops-bridge (SSH tunnel for remote State Hub access)
- Upstream tools: Ansible, Goss, SOPS/age, Terraform, SSH, and provider APIs
- Downstream: `railiance-cluster` depends on host reachability and an acceptable
S1 posture; higher layers depend on it transitively
- Access path: `ops-bridge` supplies the SSH tunnels used for private cluster
and State Hub access; this repo declares host-side SSH access and firewall
posture but does not own tunnel orchestration
- Resource accounting: `resource-control` consumes the non-secret evidence
interface under `docs/evidence/resource-hosteurope-railiance01/`; `fin-hub`
owns booked cost
---
## Terminology
- Preferred terms: OAS Stack Level S1, convergence, verification, SOPS/age, Goss specification, boundary rule
- Potentially confusing terms: "convergence" = applying Ansible to reach desired state; "verification" = running Goss tests to validate it
- **Adopted host**: an existing provider resource represented and managed here,
but not created by this repo
- **Convergence**: applying Ansible roles to move a host toward declared state
- **Verification**: evaluating live host state with Goss
- **Substrate grant**: an approved host-level network opening supplied by the
owning reef declaration
- **S1**: the infrastructure-substrate layer below the cluster runtime
---
## Related / Overlapping
- `railiance-cluster` (S2) — consumes the OS baseline provided by S1
- `railiance-hosts` — predecessor or migration-duplicate S1 line; not the
canonical repo for new architecture work
- `ops-bridge` — used to reach local State Hub from remote HostEurope server
- `railiance-cluster` — owns the S2 Kubernetes runtime
- `railiance-platform` — owns S3 shared services and service data protection
- `ops-bridge` — owns workstation-to-host tunnel orchestration
- `resource-control` — owns the resource portfolio assembled from evidence
produced here
- `reef-*` repos — own substrate grouping and exposure intent
- `railiance-hosts` — superseded predecessor; not a second source of truth
---
## Getting Oriented
- Start with: `CLAUDE.md` (session protocol, remote execution), `README.md` (provisioning workflow)
- Key files / directories: `inventory/servers.yaml` (authoritative server
list), `ansible/` (playbooks/roles), `terraform/` (provider configs), `goss/`
(spec + tests), `docs/reef-first-wave-source-map.md`,
`docs/adr/ADR-003-railiance-5repo-stack-architecture.md`
- Entry points: `make tf-plan`, `make tf-apply`, `make converge`, `make verify`
- Session and work routing: `AGENTS.md` and `.custodian-brief.md`
- Current host identities: `inventory/servers.yaml`
- Host convergence: `ansible/playbooks/bootstrap.yaml` and `ansible/roles/`
- Host-specific posture: `ansible/inventory/host_vars/`
- Firewall defaults: `ansible/roles/base/defaults/main.yml`
- Verification: `goss/baseline.yaml.j2`, `docs/verification.md`, and
`docs/conformance-loop.md`
- Hetzner prototype path: `terraform/hetzner/` and `docs/provisioning.md`
- Operator entry points: `make converge`, `make converge-check`,
`make converge-firewall`, `make verify`, and `make goss-status`
---
@ -136,31 +181,43 @@ higher layer (Kubernetes, platform, etc.) can run.
```capability
type: infrastructure
title: Server provisioning (Terraform)
description: Provision bare-metal and cloud servers on Hetzner and HostEurope via Terraform with cloud-init first-boot configuration.
keywords: [terraform, server, provisioning, hetzner, hosteurope, cloud-init, infrastructure]
```
```capability
type: infrastructure
title: OS hardening and convergence (Ansible)
description: Harden and converge server OS via Ansible (base, security, sops_agent roles) with Goss test suite for baseline validation.
keywords: [ansible, os, hardening, convergence, goss, security, baseline, validation]
title: Adopted-host inventory and convergence
description: Resolve the two current server identities into Ansible inventory and converge supported Debian/Ubuntu host packages, SSH posture, fail2ban, swap, resource limits, and access keys.
keywords: [ansible, inventory, host, convergence, hardening, linux]
```
```capability
type: security
title: Secret management (SOPS/age)
description: Manage encrypted secrets at rest in Git using SOPS/age — encrypt, rotate, and distribute secrets for Railiance infrastructure components.
keywords: [sops, age, secrets, encryption, gitops, key-rotation, credential]
title: Host firewall grant enforcement
description: Converge declared UFW rules, keep the k3s API tunnel-only, restrict Flannel peers, revoke retired sources, and require a reef exposure grant before opening public web ports.
keywords: [ufw, firewall, k3s, tunnel, flannel, reef, exposure]
```
```capability
type: infrastructure
title: Recurring host baseline verification
description: Render inventory-aware Goss checks, run them on demand or hourly on-host, retain local failure state, and collect TAP evidence; the current all-host gate has a documented CoulombCore exception.
keywords: [goss, verification, drift, systemd-timer, tap, evidence]
```
```capability
type: security
title: SSH trust bootstrap
description: Install automation public keys and OpenBao SSH user-CA trust with host-specific authorized principals; certificate issuance remains with ops-warden.
keywords: [ssh, certificate-authority, openbao, principals, access]
```
```capability
type: infrastructure
title: Host capacity evidence interface
description: Collect a bounded, non-secret Railiance01 observation across provider metadata, operating-system capacity, and k3s allocatable state for resource-control.
keywords: [capacity, evidence, hosteurope, resource-control, inventory]
```
---
## Notes
Targets two current server substrates: `CoulombCore` (`92.205.130.254`) and
`Railiance01` (`92.205.62.239`). The first-wave reef rollout also recognizes a
grouped operator workstation substrate. State Hub access uses ops-bridge —
`bridge up state-hub-coulombcore` or `bridge up state-hub-railiance01` from the
workstation (see ADR-004).
The exact differences between this operational scope and the aspirational
mission in `INTENT.md` are assessed in
`history/2026-08-23-scope-against-intent.md`.

View file

@ -0,0 +1,124 @@
# Scope Against Intent Assessment — 2026-08-23
## Purpose
This assessment compares the repository's executable state on 2026-08-23 with
the stable direction in `INTENT.md`. It describes gaps; it does not create or
authorize implementation work.
The assessment was made after confirming that the repository had no actionable
local work: no ready, active, or blocked workplan; no task assigned to
`railiance-infra` in `todo`, `progress`, or `wait`; no unread inbox message; no
active repo dispatch; and no open Forgejo issue or pull request.
## Overall Assessment
The repository is a useful host-convergence and verification substrate, but it
does not yet meet the full intent of reproducibly turning bare machines into a
verified handoff. Its strongest implemented area is post-provisioning host
control. Its largest gap is the seam before that: neither current Host Europe
server is provisioned from source, and the only Terraform module cannot consume
the current inventory.
The intent is therefore **partially realized**. The repo can maintain important
parts of S1 for existing hosts; it cannot currently rebuild the live S1 estate
from its declared source alone.
## Principle-by-Principle Assessment
| Intent principle | Assessment | Current evidence | Gap |
| --- | --- | --- | --- |
| Declarative and reproducible | Partial | Ansible roles, firewall variables, a baseline spec, dynamic inventory, and a Hetzner Terraform module are source-controlled. | The live hosts are adopted Host Europe resources. The Terraform module iterates every inventory entry but requires `type`, `region`, `image`, and `role`, which the current entries lack. Host Europe lifecycle is manual, and no source path can recreate the current estate end to end. |
| Hardened by default | Partial | SSH password/root login are disabled by Ansible; fail2ban and UFW policy exist; k3s API access is tunnel-only; public web ports require a reef grant. | Hardening is applied after access exists rather than guaranteed as one provisioning transaction. `CoulombCore` deliberately bypasses UFW management. The generic cloud-init template still contains a placeholder SSH key and is not the Terraform template. |
| Verified before handoff | Partial | Goss renders inventory-aware checks, emits TAP, and runs hourly plus five minutes after boot. | The uniform baseline intentionally fails on `CoulombCore` because UFW is inactive. Verification is not a mandatory provisioning/handoff gate, and no repository CI exercises the Ansible/Goss contract. |
| Recorded source of truth | Partial | `inventory/servers.yaml` records the two names, IPs, and SSH users; host variables record the CoulombCore exception; Railiance01 has a non-secret evidence interface. | The inventory is authoritative for connection identity but not full resource shape or lifecycle. Current-host provider, location, flavor, image, and role are not modeled there. Evidence coverage is asymmetric and point-in-time. |
| Secure at rest | Partial | The Hetzner token is SOPS/age encrypted and repo hooks check files under `secrets/`. The host role installs SOPS/age without placing the private age key. | Rotation is manual. `inventory/group_vars/secrets.sops.yaml` is a plaintext placeholder outside the hook's checked path, while the docs describe it as encrypted input. The implementation and documentation disagree about the provider-token source. |
| Foundation, not tenant | Mostly met | Active convergence, firewall, SSH, inventory, and verification code stay at host level; cluster and platform ownership are documented elsewhere. | Historical Forgejo migration, runner, and restore-drill artifacts remain in the repo and blur discovery, although they do not form the current S1 execution path. |
## Direction-of-Evolution Assessment
### Stronger reproducibility and drift detection — partial
The hourly Goss timer is a real drift detector and its firewall assertions are
derived from the same inventory variables used for convergence. It records a
host flag and journald event, with an optional State Hub progress post.
Gaps:
- No central reaction is guaranteed; paging, task creation, and rollout
blocking belong elsewhere and are not wired as a dependable end-to-end loop.
- `spec/server-baseline.yaml` is described as authoritative but is not consumed
by Ansible or Goss. Maintaining the spec, roles, and template remains a manual
three-way synchronization task.
- The declared spec and convergence are already imperfectly aligned: for
example, the spec requires `htop` and a `tegwick` passwordless-sudo posture,
while the base role does not establish both of those properties.
- A known expected failure means the all-host signal cannot distinguish only
new regressions without host-profile semantics.
### Broader provider support — gap
Only Hetzner has Terraform resources. Host Europe is represented by adopted
identity/evidence and manual lifecycle notes, not a provider implementation.
The current single inventory cannot safely mix adopted Host Europe records with
the Hetzner `for_each` model.
### Continuous baseline verification — partial
The on-host hourly timer satisfies the cadence part of the direction. It is not
yet a complete continuous assurance capability because deployment does not
require a fresh green result, the failure route is optional, and one managed
host is expected to remain red.
### Automated rotation of at-rest secret material — gap
The repository provides `sops --rotate` as an operator command. It has no
scheduled or policy-driven rotation, recipient-age enforcement, rotation
receipt, or consumer rollout verification.
### Self-evidencing, auditable provisioning — partial
Goss TAP reports, capacity observations, ADRs, and workplan evidence are useful
records. The actual create/apply/converge sequence has no unified immutable
receipt tying together source revision, provider plan, created resource,
cloud-init completion, Ansible result, and a fresh verification result.
The local S1 backup command encrypts selected `/etc` files and package
selections, but it is not scheduled, does not copy off-host, and has no restore
procedure or drill in this repo. It should not be treated as proof that the
substrate is recoverable.
## Concrete Gaps, Ordered by Leverage
1. **Separate adopted and provisionable inventory semantics.** Add an explicit
provider/lifecycle mode and validate the schema, or give each provider a
filtered declaration. A Terraform plan must never interpret an adopted Host
Europe entry as a Hetzner resource.
2. **Make one baseline contract executable.** Generate convergence and checks
from a shared model, or add automated tests that fail when the human spec,
Ansible roles, and Goss assertions diverge.
3. **Define host profiles and a real handoff gate.** Express the CoulombCore
firewall exception in verification rather than accepting a permanently red
host, and require a fresh result before S2 handoff.
4. **Repair the secret-source contract.** Remove or encrypt the plaintext
placeholder file, extend plaintext checks to every declared secret path, and
make the documentation name the provider-token source actually used by the
Makefile.
5. **Capture provisioning receipts.** Record source revision, plan/apply
identity, provider resource identifiers, convergence result, and verification
result without recording secrets.
6. **Close recovery and rotation loops.** Automate age-recipient rotation with
evidence, and give the local S1 backup a scheduled off-host copy plus a
tested restore procedure if live-state backup remains part of this repo.
7. **Reduce scope noise.** Archive or relocate historical Forgejo/platform
artifacts so current S1 ownership is evident from the file tree as well as
from `SCOPE.md`.
## Bottom Line
Today the repo can answer: **"How do we keep these existing Linux hosts closer
to a hardened, observable S1 baseline?"**
It cannot yet answer the full guiding question in `INTENT.md`: **"Can we rebuild
the live substrate from source and prove, through a mandatory auditable gate,
that it is ready before higher layers use it?"**