From 3734a1c3bc17a3df44c01f35c47c6a0c2c2aec55 Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 23 Aug 2026 10:45:02 +0200 Subject: [PATCH] Document actual infrastructure scope and intent gaps Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02994-7685-7940-bf34-3555b8256018 --- SCOPE.md | 241 +++++++++++++-------- history/2026-08-23-scope-against-intent.md | 124 +++++++++++ 2 files changed, 273 insertions(+), 92 deletions(-) create mode 100644 history/2026-08-23-scope-against-intent.md diff --git a/SCOPE.md b/SCOPE.md index 5282db9..f6a7be5 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -1,134 +1,179 @@ # SCOPE -> This file helps you quickly understand what this repository is about, -> when it is relevant, and when it is not. -> It is intentionally lightweight and may be incomplete. +> This file describes the repository as it works today. Aspirational direction +> belongs in `INTENT.md`; known differences are recorded under `history/`. --- ## One-liner -S1 Infrastructure Substrate of the Railiance OAS Stack — Git-driven OS provisioning, security hardening, and server baseline using Terraform, cloud-init, and Ansible. +The S1 host substrate for Railiance: source-backed inventory, Ansible OS +convergence, host firewall policy, SSH bootstrap, and recurring baseline +verification for the existing Railiance servers. --- ## Core Idea -Railiance is structured as five independent repos per OAS Stack layer. This repo -is S1 — the foundation and the canonical ownership home for infrastructure -substrate facts. It provisions bare-metal or cloud servers (Hetzner, -HostEurope), hardens the OS, manages secrets (SOPS/age), and validates the -resulting baseline with Goss tests. Future `reef-*` repos will model -purpose-bound substrates, but the source-backed OS baseline, inventory, and -server identity facts start here. S1 must be converged and verified before any -higher layer (Kubernetes, platform, etc.) can run. +`railiance-infra` owns host-level facts and controls below Kubernetes. It keeps +the current server identities in source, converges supported Linux hosts toward +a security baseline, and checks the resulting live state. A higher layer should +receive a reachable, hardened host rather than needing to configure the host +itself. + +The two current managed hosts are existing Host Europe virtual servers. The +repository also contains a Hetzner Terraform and cloud-init path, but that path +is not an operable representation of the current inventory: it supports only +Hetzner and expects provisioning fields that the two adopted Host Europe +entries do not contain. --- ## In Scope -- OS provisioning via Terraform (Hetzner, HostEurope providers) -- First-boot configuration via cloud-init -- OS convergence via Ansible (base, security, sops_agent roles) -- Security hardening and firewall rules -- Secret management: SOPS/age encryption at rest in Git -- Goss specification and test suite for OS baseline validation -- Server inventory management (`inventory/servers.yaml` — source of truth) -- SSH access management -- Canonical S1 identity and substrate-fact base for future `reef-*` repos -- Source-backed server and workstation substrate facts needed by higher-layer - architecture work +- Canonical connection inventory for `Railiance01` and `CoulombCore` in + `inventory/servers.yaml` +- Dynamic Ansible inventory, including per-host exceptions +- Debian/Ubuntu host convergence for base packages, SSH hardening, fail2ban, + timezone, swap, user resource limits, and host access keys +- Declared UFW management where `ufw_manage` is enabled +- Tunnel-only k3s API policy, source-restricted Flannel grants, and removal of + retired public grants +- Grant validation before opening public web ports 80/443 for a reef +- Installation of OpenBao SSH user-CA trust and per-user principals +- Installation of SOPS/age tooling and SOPS-encrypted storage of the Hetzner + provider token +- Goss host checks, TAP evidence, and an hourly on-host pass/fail signal +- Non-secret `Railiance01` identity, capacity, utilization, and host-operations + evidence for downstream resource accounting +- Local age-encrypted snapshots of selected S1 OS configuration and installed + package selections +- A reference NetKingdom host-bootstrap capability declaration --- ## Out of Scope -- Kubernetes runtime → railiance-cluster (S2) -- Platform services → railiance-platform (S3) -- Developer tooling → railiance-enablement (S4) -- Application deployments → railiance-apps (S5) -- Workload execution semantics or rail contracts → `rail-*` repos -- Purpose-specific reef repo ownership and workload-placement policy beyond the - S1 source-backed substrate facts -- No cross-layer re-configuration from higher layers +- Kubernetes installation and runtime management (`railiance-cluster`, S2) +- OpenBao, databases, ingress, Forgejo, and other platform services + (`railiance-platform`, S3) +- Developer tooling and application deployment (S4/S5) +- Workload execution contracts (`rail-*`) and workload packaging (`rapp-*`) +- Reef topology, workload placement, and exposure decisions; this repo only + enforces a supplied substrate grant +- Provider billing, contracts, lifecycle dates, and cost records +- Secret issuance or live credential custody; this repo encrypts selected + material at rest and installs client tooling +- Host Europe provisioning or lifecycle automation; the current Host Europe + servers are adopted resources +- Cluster, database, or application backup and restore + +Historical Forgejo migration and restore-drill material remains in this repo, +but it is not an S1 capability or an ownership precedent. --- ## Relevant When -- Provisioning new servers for the Railiance stack -- OS hardening, Ansible convergence, or Goss verification -- Managing server inventory or SSH access -- Rotating SOPS/age keys or updating secrets -- Preparing or validating first-wave `reef-*` rollout identity facts +- Adding or changing a managed host identity +- Converging or auditing host packages, SSH, fail2ban, UFW, swap, or resource + limits +- Changing host-level network grants or proving k3s API exposure remains + tunnel-only +- Installing SSH CA trust or automation public keys on a host +- Running or collecting host baseline checks +- Producing non-secret host-capacity evidence +- Developing or repairing the Hetzner provisioning path --- ## Not Relevant When -- Kubernetes, platform services, or application work (wrong layer) -- Server is already provisioned and converged (use cluster/platform repos) +- The change is to Kubernetes objects, cluster components, platform services, + or applications +- The request is for a secret value, login token, provider contract, or invoice +- The decision concerns which workloads or rails belong in a reef +- A backup or restore concerns persistent application or cluster data + +An already-provisioned host can still be in scope: convergence, firewall +maintenance, evidence collection, and drift checks are ongoing S1 work. --- ## Current State -- Status: maintained / productive -- Implementation: HostEurope substrate baseline active for `Railiance01` and - `CoulombCore`; server spec + test suite active; first reef rollout source map - defined. Railiance is classified along four repo-family axes (`railiance-*`, - `rail-*`, `rapp-*`, `reef-*`), of which five `railiance-*` repos cover S1–S5; - this file previously said "5-repo stack architecture", which predates that - model -- Stability: high for the current single-server and transitional two-server - substrate reality; proven in production on `92.205.62.239` -- Usage: foundation for all Railiance deployments; canonical S1 source for - higher-layer and future reef planning. `railiance-hosts` is **superseded** by - this repo and carries a banner saying so; its retirement is pending in - `railiance-master` -- **Firewall posture**: `RAIL-HO-WP-0009` finished. k3s API is tunnel-only - (ADR-005); live `Railiance01` UFW matches the declaration. CoulombCore - UFW stays unmanaged (`ufw_manage: false`) -- **Resource evidence**: `RAIL-HO-WP-0008` finished. Non-secret identity, - capacity observations, and host-ops labor live under - `docs/evidence/resource-hosteurope-railiance01/` +- Status: maintained and operational for adopted-host convergence and + verification +- Managed inventory: `Railiance01` (`92.205.62.239`) and `CoulombCore` + (`92.205.130.254`), both reached as user `tegwick` +- `Railiance01`: UFW is managed from this repo; the k3s API is tunnel-only and + the Host Europe Nydus exception is declared +- `CoulombCore`: UFW is deliberately unmanaged because its live packet filter + has not been migrated safely to this repo's UFW model +- Verification: Goss can run on demand and hourly on-host. `CoulombCore` is a + documented expected failure for the uniform UFW-active assertion, so the + repository does not currently provide an all-host green handoff gate +- Provisioning: a Hetzner-only Terraform template and helper scripts exist. + They do not currently plan against the mixed/adopted inventory and do not + provision either live Host Europe server +- Secrets: `secrets/hetzner-token.yaml` is SOPS-encrypted. The Ansible + `sops_agent` role installs tools but intentionally does not place a private + age key on a host +- Evidence: the latest committed `Railiance01` capacity observation is a + point-in-time record, not continuous resource telemetry --- ## How It Fits -- Upstream dependencies: Terraform, Ansible, SOPS/age (external tools); cloud provider APIs -- Downstream consumers: railiance-cluster (S2) depends on a converged, verified OS from this layer; all higher layers transitively depend on S1 -- Future substrate-boundary consumers: first-wave `reef-*` repos should project - from the source-backed facts here rather than creating a second S1 inventory -- Often used with: railiance-cluster (next layer), ops-bridge (SSH tunnel for remote State Hub access) +- Upstream tools: Ansible, Goss, SOPS/age, Terraform, SSH, and provider APIs +- Downstream: `railiance-cluster` depends on host reachability and an acceptable + S1 posture; higher layers depend on it transitively +- Access path: `ops-bridge` supplies the SSH tunnels used for private cluster + and State Hub access; this repo declares host-side SSH access and firewall + posture but does not own tunnel orchestration +- Resource accounting: `resource-control` consumes the non-secret evidence + interface under `docs/evidence/resource-hosteurope-railiance01/`; `fin-hub` + owns booked cost --- ## Terminology -- Preferred terms: OAS Stack Level S1, convergence, verification, SOPS/age, Goss specification, boundary rule -- Potentially confusing terms: "convergence" = applying Ansible to reach desired state; "verification" = running Goss tests to validate it +- **Adopted host**: an existing provider resource represented and managed here, + but not created by this repo +- **Convergence**: applying Ansible roles to move a host toward declared state +- **Verification**: evaluating live host state with Goss +- **Substrate grant**: an approved host-level network opening supplied by the + owning reef declaration +- **S1**: the infrastructure-substrate layer below the cluster runtime --- ## Related / Overlapping -- `railiance-cluster` (S2) — consumes the OS baseline provided by S1 -- `railiance-hosts` — predecessor or migration-duplicate S1 line; not the - canonical repo for new architecture work -- `ops-bridge` — used to reach local State Hub from remote HostEurope server +- `railiance-cluster` — owns the S2 Kubernetes runtime +- `railiance-platform` — owns S3 shared services and service data protection +- `ops-bridge` — owns workstation-to-host tunnel orchestration +- `resource-control` — owns the resource portfolio assembled from evidence + produced here +- `reef-*` repos — own substrate grouping and exposure intent +- `railiance-hosts` — superseded predecessor; not a second source of truth --- ## Getting Oriented -- Start with: `CLAUDE.md` (session protocol, remote execution), `README.md` (provisioning workflow) -- Key files / directories: `inventory/servers.yaml` (authoritative server - list), `ansible/` (playbooks/roles), `terraform/` (provider configs), `goss/` - (spec + tests), `docs/reef-first-wave-source-map.md`, - `docs/adr/ADR-003-railiance-5repo-stack-architecture.md` -- Entry points: `make tf-plan`, `make tf-apply`, `make converge`, `make verify` +- Session and work routing: `AGENTS.md` and `.custodian-brief.md` +- Current host identities: `inventory/servers.yaml` +- Host convergence: `ansible/playbooks/bootstrap.yaml` and `ansible/roles/` +- Host-specific posture: `ansible/inventory/host_vars/` +- Firewall defaults: `ansible/roles/base/defaults/main.yml` +- Verification: `goss/baseline.yaml.j2`, `docs/verification.md`, and + `docs/conformance-loop.md` +- Hetzner prototype path: `terraform/hetzner/` and `docs/provisioning.md` +- Operator entry points: `make converge`, `make converge-check`, + `make converge-firewall`, `make verify`, and `make goss-status` --- @@ -136,31 +181,43 @@ higher layer (Kubernetes, platform, etc.) can run. ```capability type: infrastructure -title: Server provisioning (Terraform) -description: Provision bare-metal and cloud servers on Hetzner and HostEurope via Terraform with cloud-init first-boot configuration. -keywords: [terraform, server, provisioning, hetzner, hosteurope, cloud-init, infrastructure] -``` - -```capability -type: infrastructure -title: OS hardening and convergence (Ansible) -description: Harden and converge server OS via Ansible (base, security, sops_agent roles) with Goss test suite for baseline validation. -keywords: [ansible, os, hardening, convergence, goss, security, baseline, validation] +title: Adopted-host inventory and convergence +description: Resolve the two current server identities into Ansible inventory and converge supported Debian/Ubuntu host packages, SSH posture, fail2ban, swap, resource limits, and access keys. +keywords: [ansible, inventory, host, convergence, hardening, linux] ``` ```capability type: security -title: Secret management (SOPS/age) -description: Manage encrypted secrets at rest in Git using SOPS/age — encrypt, rotate, and distribute secrets for Railiance infrastructure components. -keywords: [sops, age, secrets, encryption, gitops, key-rotation, credential] +title: Host firewall grant enforcement +description: Converge declared UFW rules, keep the k3s API tunnel-only, restrict Flannel peers, revoke retired sources, and require a reef exposure grant before opening public web ports. +keywords: [ufw, firewall, k3s, tunnel, flannel, reef, exposure] +``` + +```capability +type: infrastructure +title: Recurring host baseline verification +description: Render inventory-aware Goss checks, run them on demand or hourly on-host, retain local failure state, and collect TAP evidence; the current all-host gate has a documented CoulombCore exception. +keywords: [goss, verification, drift, systemd-timer, tap, evidence] +``` + +```capability +type: security +title: SSH trust bootstrap +description: Install automation public keys and OpenBao SSH user-CA trust with host-specific authorized principals; certificate issuance remains with ops-warden. +keywords: [ssh, certificate-authority, openbao, principals, access] +``` + +```capability +type: infrastructure +title: Host capacity evidence interface +description: Collect a bounded, non-secret Railiance01 observation across provider metadata, operating-system capacity, and k3s allocatable state for resource-control. +keywords: [capacity, evidence, hosteurope, resource-control, inventory] ``` --- ## Notes -Targets two current server substrates: `CoulombCore` (`92.205.130.254`) and -`Railiance01` (`92.205.62.239`). The first-wave reef rollout also recognizes a -grouped operator workstation substrate. State Hub access uses ops-bridge — -`bridge up state-hub-coulombcore` or `bridge up state-hub-railiance01` from the -workstation (see ADR-004). +The exact differences between this operational scope and the aspirational +mission in `INTENT.md` are assessed in +`history/2026-08-23-scope-against-intent.md`. diff --git a/history/2026-08-23-scope-against-intent.md b/history/2026-08-23-scope-against-intent.md new file mode 100644 index 0000000..7f2a301 --- /dev/null +++ b/history/2026-08-23-scope-against-intent.md @@ -0,0 +1,124 @@ +# Scope Against Intent Assessment — 2026-08-23 + +## Purpose + +This assessment compares the repository's executable state on 2026-08-23 with +the stable direction in `INTENT.md`. It describes gaps; it does not create or +authorize implementation work. + +The assessment was made after confirming that the repository had no actionable +local work: no ready, active, or blocked workplan; no task assigned to +`railiance-infra` in `todo`, `progress`, or `wait`; no unread inbox message; no +active repo dispatch; and no open Forgejo issue or pull request. + +## Overall Assessment + +The repository is a useful host-convergence and verification substrate, but it +does not yet meet the full intent of reproducibly turning bare machines into a +verified handoff. Its strongest implemented area is post-provisioning host +control. Its largest gap is the seam before that: neither current Host Europe +server is provisioned from source, and the only Terraform module cannot consume +the current inventory. + +The intent is therefore **partially realized**. The repo can maintain important +parts of S1 for existing hosts; it cannot currently rebuild the live S1 estate +from its declared source alone. + +## Principle-by-Principle Assessment + +| Intent principle | Assessment | Current evidence | Gap | +| --- | --- | --- | --- | +| Declarative and reproducible | Partial | Ansible roles, firewall variables, a baseline spec, dynamic inventory, and a Hetzner Terraform module are source-controlled. | The live hosts are adopted Host Europe resources. The Terraform module iterates every inventory entry but requires `type`, `region`, `image`, and `role`, which the current entries lack. Host Europe lifecycle is manual, and no source path can recreate the current estate end to end. | +| Hardened by default | Partial | SSH password/root login are disabled by Ansible; fail2ban and UFW policy exist; k3s API access is tunnel-only; public web ports require a reef grant. | Hardening is applied after access exists rather than guaranteed as one provisioning transaction. `CoulombCore` deliberately bypasses UFW management. The generic cloud-init template still contains a placeholder SSH key and is not the Terraform template. | +| Verified before handoff | Partial | Goss renders inventory-aware checks, emits TAP, and runs hourly plus five minutes after boot. | The uniform baseline intentionally fails on `CoulombCore` because UFW is inactive. Verification is not a mandatory provisioning/handoff gate, and no repository CI exercises the Ansible/Goss contract. | +| Recorded source of truth | Partial | `inventory/servers.yaml` records the two names, IPs, and SSH users; host variables record the CoulombCore exception; Railiance01 has a non-secret evidence interface. | The inventory is authoritative for connection identity but not full resource shape or lifecycle. Current-host provider, location, flavor, image, and role are not modeled there. Evidence coverage is asymmetric and point-in-time. | +| Secure at rest | Partial | The Hetzner token is SOPS/age encrypted and repo hooks check files under `secrets/`. The host role installs SOPS/age without placing the private age key. | Rotation is manual. `inventory/group_vars/secrets.sops.yaml` is a plaintext placeholder outside the hook's checked path, while the docs describe it as encrypted input. The implementation and documentation disagree about the provider-token source. | +| Foundation, not tenant | Mostly met | Active convergence, firewall, SSH, inventory, and verification code stay at host level; cluster and platform ownership are documented elsewhere. | Historical Forgejo migration, runner, and restore-drill artifacts remain in the repo and blur discovery, although they do not form the current S1 execution path. | + +## Direction-of-Evolution Assessment + +### Stronger reproducibility and drift detection — partial + +The hourly Goss timer is a real drift detector and its firewall assertions are +derived from the same inventory variables used for convergence. It records a +host flag and journald event, with an optional State Hub progress post. + +Gaps: + +- No central reaction is guaranteed; paging, task creation, and rollout + blocking belong elsewhere and are not wired as a dependable end-to-end loop. +- `spec/server-baseline.yaml` is described as authoritative but is not consumed + by Ansible or Goss. Maintaining the spec, roles, and template remains a manual + three-way synchronization task. +- The declared spec and convergence are already imperfectly aligned: for + example, the spec requires `htop` and a `tegwick` passwordless-sudo posture, + while the base role does not establish both of those properties. +- A known expected failure means the all-host signal cannot distinguish only + new regressions without host-profile semantics. + +### Broader provider support — gap + +Only Hetzner has Terraform resources. Host Europe is represented by adopted +identity/evidence and manual lifecycle notes, not a provider implementation. +The current single inventory cannot safely mix adopted Host Europe records with +the Hetzner `for_each` model. + +### Continuous baseline verification — partial + +The on-host hourly timer satisfies the cadence part of the direction. It is not +yet a complete continuous assurance capability because deployment does not +require a fresh green result, the failure route is optional, and one managed +host is expected to remain red. + +### Automated rotation of at-rest secret material — gap + +The repository provides `sops --rotate` as an operator command. It has no +scheduled or policy-driven rotation, recipient-age enforcement, rotation +receipt, or consumer rollout verification. + +### Self-evidencing, auditable provisioning — partial + +Goss TAP reports, capacity observations, ADRs, and workplan evidence are useful +records. The actual create/apply/converge sequence has no unified immutable +receipt tying together source revision, provider plan, created resource, +cloud-init completion, Ansible result, and a fresh verification result. + +The local S1 backup command encrypts selected `/etc` files and package +selections, but it is not scheduled, does not copy off-host, and has no restore +procedure or drill in this repo. It should not be treated as proof that the +substrate is recoverable. + +## Concrete Gaps, Ordered by Leverage + +1. **Separate adopted and provisionable inventory semantics.** Add an explicit + provider/lifecycle mode and validate the schema, or give each provider a + filtered declaration. A Terraform plan must never interpret an adopted Host + Europe entry as a Hetzner resource. +2. **Make one baseline contract executable.** Generate convergence and checks + from a shared model, or add automated tests that fail when the human spec, + Ansible roles, and Goss assertions diverge. +3. **Define host profiles and a real handoff gate.** Express the CoulombCore + firewall exception in verification rather than accepting a permanently red + host, and require a fresh result before S2 handoff. +4. **Repair the secret-source contract.** Remove or encrypt the plaintext + placeholder file, extend plaintext checks to every declared secret path, and + make the documentation name the provider-token source actually used by the + Makefile. +5. **Capture provisioning receipts.** Record source revision, plan/apply + identity, provider resource identifiers, convergence result, and verification + result without recording secrets. +6. **Close recovery and rotation loops.** Automate age-recipient rotation with + evidence, and give the local S1 backup a scheduled off-host copy plus a + tested restore procedure if live-state backup remains part of this repo. +7. **Reduce scope noise.** Archive or relocate historical Forgejo/platform + artifacts so current S1 ownership is evident from the file tree as well as + from `SCOPE.md`. + +## Bottom Line + +Today the repo can answer: **"How do we keep these existing Linux hosts closer +to a hardened, observable S1 baseline?"** + +It cannot yet answer the full guiding question in `INTENT.md`: **"Can we rebuild +the live substrate from source and prove, through a mandatory auditable gate, +that it is ready before higher layers use it?"**