tegwick
6ca167ce19
Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
104 lines
4.2 KiB
Markdown
104 lines
4.2 KiB
Markdown
---
|
|
agent: host-operator
|
|
engagement_id: eng-coulomb-railiance01-ho-001
|
|
project: coulomb-railiance01
|
|
last_updated: '2026-07-16'
|
|
session_count: 3
|
|
confidentiality: client_owned
|
|
---
|
|
|
|
# Host Operator Memory — railiance01 pilot
|
|
|
|
## Engagement Charter
|
|
|
|
- **Duty:** Keep railiance01 operational and secure; OS currency; load and workload review
|
|
- **Cadence:** daily health/load (business days); weekly OS/security pass
|
|
- **Change windows:** prefer low-traffic periods; reboot only with human approval
|
|
- **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated)
|
|
- **Out of scope:** app features; other hosts; secret vending
|
|
- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16)
|
|
|
|
## Project Context
|
|
|
|
Coulomb ecosystem production host `railiance01` runs k3s and platform workloads
|
|
(including Forgejo/apps per railiance-hosts docs). This engagement is the first
|
|
forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
|
|
|
## Host Profiles
|
|
|
|
| hostname | role | typical load | services | last review |
|
|
|----------|------|--------------|----------|-------------|
|
|
| railiance01 | k3s production single-node | overload on 2 cores under current mix | k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods | 2026-07-16 |
|
|
|
|
## OS & Patch State
|
|
|
|
- Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current)
|
|
- Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …)
|
|
- `unattended-upgrades.service` flagged by needrestart for restart
|
|
- **No package upgrades applied** this session (gated)
|
|
|
|
## Security Posture
|
|
|
|
- UFW **active**: default deny in; allow OpenSSH 22, k3s API **6443/tcp Anywhere**, flannel **8472/udp Anywhere**
|
|
- Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes
|
|
- Journal noise: reverse-forward port 18765 already in use (tunnel contention)
|
|
- Public 6443 exposure: policy review recommended (firewall_change gated)
|
|
- First snapshot: 2026-07-16 (see health report)
|
|
|
|
## Load & Workload Envelope
|
|
|
|
| Field | Value (2026-07-16 sample) |
|
|
|-------|---------------------------|
|
|
| Cores | 2 |
|
|
| Load 1/5/15 | ~12 / 11 / 16 (**saturated**) |
|
|
| RAM | 3.8 GiB; no swap; MemAvailable often <0.5 GiB |
|
|
| PSI memory full | avg60 ~24%, avg300 ~29% |
|
|
| Disk / | 54% of 96G |
|
|
| /var/log | ~5.7G (journal ~4.1G) |
|
|
| Heavy workloads | k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns |
|
|
| Saturation incident | 2026-07-16 — memory+load critical; k3s API ServiceUnavailable |
|
|
|
|
**Envelope intent:** treat load > 4 sustained or MemAvailable < 300Mi as **Watch/Critical**; escalate capacity.
|
|
|
|
## Accumulated Findings
|
|
|
|
- Undersized RAM for workload density is the dominant risk
|
|
- k3s API instability under memory pressure
|
|
- Large journald footprint
|
|
- Pending OS security updates
|
|
|
|
## What Worked
|
|
|
|
- Observe path via `ssh railiance01` (tegwick + id_custodian_agent)
|
|
- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect
|
|
- Engagement vault + close-session for durable evidence
|
|
|
|
## Watch Points
|
|
|
|
- Single-node production: privileged mistakes have full blast radius
|
|
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
|
|
- Disk growth from images, logs, and backups
|
|
- Do not schedule heavy CI/agents until memory recovers
|
|
- Bridge reverse-forward port collisions (18765)
|
|
|
|
## Recurring Findings
|
|
|
|
- Memory pressure / no swap · first seen 2026-07-16 · frequency 1
|
|
- Load ≫ cores · first seen 2026-07-16 · frequency 1
|
|
|
|
## Cleared Issues
|
|
|
|
_None yet (no remediation applied)._
|
|
|
|
## Open Threads
|
|
|
|
- Human decision: add RAM and/or temporary swap
|
|
- Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443
|
|
- Re-check k3s API health after memory improves
|
|
- Align inventory hostname with kernel hostname / DNS naming
|
|
|
|
## Session Log
|
|
|
|
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
|
|
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
|
|
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
|