ops: complete host-operator ramp-up on railiance01 (WP-0009 T10)
Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
This commit is contained in:
parent
7257e62dba
commit
6ca167ce19
15 changed files with 334 additions and 90 deletions
|
|
@ -1,47 +1,63 @@
|
|||
# Baseline — railiance01
|
||||
|
||||
**Engagement:** eng-coulomb-railiance01-ho-001
|
||||
**Status:** pending first observe session (RU-02)
|
||||
**Last captured:** —
|
||||
**Status:** captured
|
||||
**Last captured:** 2026-07-16 (observe session, T10 ramp-up)
|
||||
**Access:** `ssh railiance01` (user tegwick, IdentityFile id_custodian_agent) — host_observe
|
||||
|
||||
## Identity
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Hostname | railiance01 |
|
||||
| Inventory | railiance-hosts |
|
||||
| Notes | k3s production; forgejo/apps |
|
||||
| Hostname (kernel) | 239.62.205.92.host.secureserver.net |
|
||||
| Inventory name | railiance01 |
|
||||
| Public IP (inventory) | 92.205.62.239 |
|
||||
| Inventory ref | railiance-hosts |
|
||||
| Role | Single-node k3s production (forgejo/apps, activity-core, platform services) |
|
||||
|
||||
## Capture checklist (fill on first session)
|
||||
## OS
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Distro | Ubuntu 24.04.3 LTS (noble) |
|
||||
| Kernel | 6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC |
|
||||
| Arch | x86_64 |
|
||||
| Uptime at capture | 129 days, 13:34 |
|
||||
| CPU cores (nproc) | **2** |
|
||||
| Memory | **3.8 GiB** total |
|
||||
| Swap | **none** |
|
||||
| Root FS | /dev/sda1 ext4 96G, **54% used** (52G/45G free) |
|
||||
|
||||
## Capture commands (non-destructive)
|
||||
|
||||
```bash
|
||||
hostname
|
||||
uptime
|
||||
uname -a
|
||||
cat /etc/os-release
|
||||
nproc
|
||||
free -h
|
||||
df -h
|
||||
ss -tuln | head -40
|
||||
# if k3s:
|
||||
kubectl get node -o wide 2>/dev/null || k3s kubectl get node -o wide 2>/dev/null
|
||||
ssh railiance01 'hostname; uptime; uname -a; nproc; free -h; df -hT; cat /etc/os-release | head -8'
|
||||
```
|
||||
|
||||
## Recorded values
|
||||
|
||||
_To be filled during ramp-up._
|
||||
|
||||
## Known quirks (from docs, pre-session)
|
||||
|
||||
- Forgejo + in-cluster Actions runner documented in railiance-hosts ADR-004
|
||||
- State Hub / activity-core historically deployed on this cluster path
|
||||
- Access often via ops-bridge from workstation
|
||||
|
||||
## Envelope seed
|
||||
## Envelope seed (2026-07-16)
|
||||
|
||||
| Metric | Baseline | Notes |
|
||||
|--------|----------|-------|
|
||||
| Load | | |
|
||||
| Memory | | |
|
||||
| Disk | | |
|
||||
| Top workloads | | |
|
||||
| Load 1/5/15 | ~12 / 11 / 16 | **≫ 2 cores** — overloaded |
|
||||
| MemAvailable | ~0.1–0.5 GiB | severe pressure; PSI full avg60 ~24% |
|
||||
| Disk / | 54% | OK for capacity; watch growth |
|
||||
| /var/log | ~5.7G | journal alone ~4.1G |
|
||||
| Top host processes | k3s server, gitea, temporal-server, activity-core/state-hub python | |
|
||||
| k3s API | ServiceUnavailable at capture | control plane unstable under memory pressure |
|
||||
|
||||
## Known quirks
|
||||
|
||||
- Single-node production: no HA; memory headroom is the binding constraint
|
||||
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
|
||||
- ops-bridge reverse tunnels to localhost ports (state-hub 18000/18001, etc.)
|
||||
- sshd journal noise: reverse-forward port 18765 already in use (tunnel contention)
|
||||
- Inventory hostname vs DNS: reverse DNS is HostEurope generic name
|
||||
|
||||
## Access path (RU-01)
|
||||
|
||||
| Method | Result |
|
||||
|--------|--------|
|
||||
| ICMP to inventory IP | reachable |
|
||||
| `ssh railiance01` (tegwick) | **success** — non-interactive shell |
|
||||
| ops-bridge tunnels to host | connected (state-hub-railiance01, issue-core-railiance01, …) |
|
||||
| Privileged ops | not exercised; sudo -n available for read-only needrestart/ufw/k3s in this session |
|
||||
|
|
|
|||
|
|
@ -1,7 +1,13 @@
|
|||
# Outstanding risks — eng-coulomb-railiance01-ho-001
|
||||
|
||||
_Populated during operate and finalised at ramp-down (RD-03)._
|
||||
Populated during operate; finalise at ramp-down (RD-03).
|
||||
|
||||
| Risk | Severity | Mitigation / owner | Status |
|
||||
|------|----------|--------------------|--------|
|
||||
| _none recorded_ | | | |
|
||||
| RAM undersized (3.8G, no swap) for k3s+platform density | Critical | Human capacity decision (RAM and/or swap); reduce concurrent load | open |
|
||||
| k3s API ServiceUnavailable under pressure | Critical | Stabilize memory; re-check API; avoid heavy kubectl while degraded | open |
|
||||
| Load average ≫ 2 cores | High | Same as capacity; schedule non-urgent work off-host | open |
|
||||
| journald ~4.1G | Medium | Approved journal vacuum | open |
|
||||
| Pending security package updates | Medium | Approved OS security pass | open |
|
||||
| UFW allows 6443/8472 from Anywhere | Medium | Policy review + possible firewall_change | open |
|
||||
| Reverse-forward port 18765 collisions | Low–Medium | Bridge/tunnel cleanup | open |
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ agent: host-operator
|
|||
engagement_id: eng-coulomb-railiance01-ho-001
|
||||
project: coulomb-railiance01
|
||||
last_updated: '2026-07-16'
|
||||
session_count: 1
|
||||
session_count: 3
|
||||
confidentiality: client_owned
|
||||
---
|
||||
|
||||
|
|
@ -16,6 +16,7 @@ confidentiality: client_owned
|
|||
- **Change windows:** prefer low-traffic periods; reboot only with human approval
|
||||
- **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated)
|
||||
- **Out of scope:** app features; other hosts; secret vending
|
||||
- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16)
|
||||
|
||||
## Project Context
|
||||
|
||||
|
|
@ -27,49 +28,77 @@ forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
|||
|
||||
| hostname | role | typical load | services | last review |
|
||||
|----------|------|--------------|----------|-------------|
|
||||
| railiance01 | k3s production | _TBD ramp-up_ | k3s, forgejo/apps (inventory) | never |
|
||||
| railiance01 | k3s production single-node | overload on 2 cores under current mix | k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods | 2026-07-16 |
|
||||
|
||||
## OS & Patch State
|
||||
|
||||
_Pending first OS security pass (RU-05 / weekly protocol)._
|
||||
- Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current)
|
||||
- Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …)
|
||||
- `unattended-upgrades.service` flagged by needrestart for restart
|
||||
- **No package upgrades applied** this session (gated)
|
||||
|
||||
## Security Posture
|
||||
|
||||
_Pending first security snapshot (RU-05)._
|
||||
- UFW **active**: default deny in; allow OpenSSH 22, k3s API **6443/tcp Anywhere**, flannel **8472/udp Anywhere**
|
||||
- Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes
|
||||
- Journal noise: reverse-forward port 18765 already in use (tunnel contention)
|
||||
- Public 6443 exposure: policy review recommended (firewall_change gated)
|
||||
- First snapshot: 2026-07-16 (see health report)
|
||||
|
||||
## Load & Workload Envelope
|
||||
|
||||
_Pending first load review (RU-06)._
|
||||
| Field | Value (2026-07-16 sample) |
|
||||
|-------|---------------------------|
|
||||
| Cores | 2 |
|
||||
| Load 1/5/15 | ~12 / 11 / 16 (**saturated**) |
|
||||
| RAM | 3.8 GiB; no swap; MemAvailable often <0.5 GiB |
|
||||
| PSI memory full | avg60 ~24%, avg300 ~29% |
|
||||
| Disk / | 54% of 96G |
|
||||
| /var/log | ~5.7G (journal ~4.1G) |
|
||||
| Heavy workloads | k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns |
|
||||
| Saturation incident | 2026-07-16 — memory+load critical; k3s API ServiceUnavailable |
|
||||
|
||||
**Envelope intent:** treat load > 4 sustained or MemAvailable < 300Mi as **Watch/Critical**; escalate capacity.
|
||||
|
||||
## Accumulated Findings
|
||||
|
||||
_None yet._
|
||||
- Undersized RAM for workload density is the dominant risk
|
||||
- k3s API instability under memory pressure
|
||||
- Large journald footprint
|
||||
- Pending OS security updates
|
||||
|
||||
## What Worked
|
||||
|
||||
_None yet._
|
||||
- Observe path via `ssh railiance01` (tegwick + id_custodian_agent)
|
||||
- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect
|
||||
- Engagement vault + close-session for durable evidence
|
||||
|
||||
## Watch Points
|
||||
|
||||
- Single-node production: privileged mistakes have full blast radius
|
||||
- DinD / Actions runner privilege model on railiance01 (see railiance-hosts ADRs)
|
||||
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
|
||||
- Disk growth from images, logs, and backups
|
||||
- Do not schedule heavy CI/agents until memory recovers
|
||||
- Bridge reverse-forward port collisions (18765)
|
||||
|
||||
## Recurring Findings
|
||||
|
||||
_None yet._
|
||||
- Memory pressure / no swap · first seen 2026-07-16 · frequency 1
|
||||
- Load ≫ cores · first seen 2026-07-16 · frequency 1
|
||||
|
||||
## Cleared Issues
|
||||
|
||||
_None yet._
|
||||
_None yet (no remediation applied)._
|
||||
|
||||
## Open Threads
|
||||
|
||||
- Complete RU-01 access verification
|
||||
- Capture baseline `vault/baselines/railiance01.md`
|
||||
- First health + load review report
|
||||
- Human decision: add RAM and/or temporary swap
|
||||
- Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443
|
||||
- Re-check k3s API health after memory improves
|
||||
- Align inventory hostname with kernel hostname / DNS naming
|
||||
|
||||
## Session Log
|
||||
|
||||
<!-- YYYY-MM-DD · host(s) · key finding · outcome -->
|
||||
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
|
||||
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
|
||||
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
|
||||
|
|
|
|||
|
|
@ -0,0 +1,62 @@
|
|||
# RU-08 — Privileged action proposal (DRY RUN — not executed)
|
||||
|
||||
**Engagement:** eng-coulomb-railiance01-ho-001
|
||||
**Date:** 2026-07-16
|
||||
**Author:** host-operator (grok session)
|
||||
**Status:** proposal only — **no human approval recorded; no changes applied**
|
||||
|
||||
## Purpose
|
||||
|
||||
Prove the privilege gate path: agent may **propose** privileged work; execution requires recorded human approval.
|
||||
|
||||
## Proposed actions (optional, priority order)
|
||||
|
||||
### P1 — Emergency swap file (if OOM imminent)
|
||||
|
||||
| Field | Value |
|
||||
|-------|--------|
|
||||
| Class | `privileged_ops` |
|
||||
| Blast radius | Host-wide; disk for swapfile; may pause I/O briefly |
|
||||
| Rollback | `swapoff` + remove swapfile |
|
||||
| Command sketch | create 2–4G swapfile on root FS (exact steps only after approval) |
|
||||
| Risk | Disk wear; masks capacity problem |
|
||||
|
||||
### P2 — journald vacuum
|
||||
|
||||
| Field | Value |
|
||||
|-------|--------|
|
||||
| Class | `privileged_ops` |
|
||||
| Blast radius | Historical logs discarded |
|
||||
| Rollback | none (logs gone) |
|
||||
| Command sketch | `journalctl --vacuum-size=500M` |
|
||||
| Risk | Loss of forensic depth |
|
||||
|
||||
### P3 — Package security upgrades
|
||||
|
||||
| Field | Value |
|
||||
|-------|--------|
|
||||
| Class | `package_upgrade` |
|
||||
| Blast radius | Service restarts; possible brief downtime |
|
||||
| Rollback | package-specific; may need restore |
|
||||
| Command sketch | `apt-get update && apt-get upgrade` (or unattended security only) |
|
||||
| Risk | Regression on production single-node |
|
||||
|
||||
### P4 — Restrict UFW sources for 6443/8472
|
||||
|
||||
| Field | Value |
|
||||
|-------|--------|
|
||||
| Class | `firewall_change` |
|
||||
| Blast radius | May lock out nodes/agents if mis-scoped |
|
||||
| Rollback | re-add rules |
|
||||
| Risk | High on single-node misconfiguration |
|
||||
|
||||
## Approval record
|
||||
|
||||
| Approver | Decision | Date | Notes |
|
||||
|----------|----------|------|-------|
|
||||
| _none_ | pending | | Dry-run only for RU-08 |
|
||||
|
||||
## Gate test result
|
||||
|
||||
- Proposal written to vault without executing changes: **PASS (RU-08)**
|
||||
- Access class used this session: **host_observe** only
|
||||
Loading…
Add table
Add a link
Reference in a new issue