tegwick
16cafc3e67
Soft reboot did not pick up reported 4/16 flavor; provider resize still required.
120 lines
5.3 KiB
Markdown
120 lines
5.3 KiB
Markdown
---
|
||
agent: host-operator
|
||
engagement_id: eng-coulomb-railiance01-ho-001
|
||
project: coulomb-railiance01
|
||
last_updated: '2026-07-22'
|
||
session_count: 7
|
||
confidentiality: client_owned
|
||
---
|
||
|
||
# Host Operator Memory — railiance01 pilot
|
||
|
||
## Engagement Charter
|
||
|
||
- **Duty:** Keep railiance01 operational and secure; OS currency; load and workload review
|
||
- **Cadence:** daily health/load (business days); weekly OS/security pass
|
||
- **Change windows:** prefer low-traffic periods; reboot only with human approval
|
||
- **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated)
|
||
- **Out of scope:** app features; other hosts; secret vending
|
||
- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16)
|
||
|
||
## Project Context
|
||
|
||
Coulomb ecosystem production host `railiance01` runs k3s and platform workloads
|
||
(including Forgejo/apps per railiance-hosts docs). This engagement is the first
|
||
forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
||
|
||
## Host Profiles
|
||
|
||
| hostname | role | typical load | services | last review |
|
||
|----------|------|--------------|----------|-------------|
|
||
| railiance01 | k3s production single-node | calmer after reboot (~load 2 on 2 cores) | k3s, gitea, temporal, activity-core, state-hub | 2026-07-16 capacity check |
|
||
|
||
|
||
## OS & Patch State
|
||
|
||
- Ubuntu 24.04 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — no reboot required after 2026-07-16 upgrade)
|
||
- **2026-07-16:** full `apt-get upgrade` applied (approved); certs/ca updated; some service restarts deferred
|
||
- unattended-upgrades / logind may still want restart; re-login SSH sessions recommended
|
||
|
||
|
||
## Security Posture
|
||
|
||
- UFW active: OpenSSH Anywhere; **6443 only from operator IPs** 89.244.90.246 and 85.132.220.102
|
||
- **8472/udp world-open removed** (single-node; re-add if multi-node flannel peers needed)
|
||
- UFW backup: /etc/ufw/user.rules.bak.20260716
|
||
- 2026-07-16 package upgrades applied (security-relevant packages included)
|
||
|
||
|
||
## Load & Workload Envelope
|
||
|
||
| Field | 2026-07-16 capacity check (post operator upgrade claim) |
|
||
|-------|--------------------------------------------------------|
|
||
| Cores (guest) | **still 2** — operator reported 4; not visible yet |
|
||
| RAM (guest) | **still ~3.8 GiB** — operator reported 16 GiB; not visible yet |
|
||
| Load 1m | ~2 (improved vs Critical era) |
|
||
| MemAvailable | ~0.9 GiB |
|
||
| Swap | 4 GiB file, ~300 Mi used |
|
||
| PSI memory full avg60 | ~0.2% |
|
||
| k3s | Ready |
|
||
|
||
Do **not** treat 4/16 as baseline until `nproc` and `free -h` match.
|
||
See reports/2026-07-16-capacity-upgrade-check.md
|
||
|
||
|
||
## Accumulated Findings
|
||
|
||
- Undersized RAM for workload density is the dominant risk
|
||
- k3s API instability under memory pressure
|
||
- Large journald footprint
|
||
- Pending OS security updates
|
||
|
||
## What Worked
|
||
|
||
- Observe path via `ssh railiance01` (tegwick + id_custodian_agent)
|
||
- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect
|
||
- Engagement vault + close-session for durable evidence
|
||
|
||
## Watch Points
|
||
|
||
- Single-node production: privileged mistakes have full blast radius
|
||
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
|
||
- Disk growth from images, logs, and backups
|
||
- Do not schedule heavy CI/agents until memory recovers
|
||
- Bridge reverse-forward port collisions (18765)
|
||
|
||
## Recurring Findings
|
||
|
||
- Memory pressure on 3.8G host · first seen 2026-07-16 · mitigated by swap but capacity still tight
|
||
- Load ≫ cores · improved after remediation; still watch
|
||
|
||
|
||
## Cleared Issues
|
||
|
||
- No swap · cleared 2026-07-16 via /swapfile 4G + fstab
|
||
- journald 4G bloat · vacuumed to ~500M target 2026-07-16
|
||
- World-open k3s API 6443 · restricted to operator IPs 2026-07-16
|
||
- World-open flannel 8472 · removed 2026-07-16 (single-node)
|
||
- Pending package upgrades · apt upgrade applied 2026-07-16
|
||
|
||
|
||
## Open Threads
|
||
|
||
- **Confirm VM resize:** operator reported 4 vCPU / 16 GiB; live guest still 2 / 3.8 GiB (2026-07-16 check)
|
||
- After 4/16 confirmed: update baseline; consider whether to keep full 4G swap or shrink
|
||
- Re-login stale SSH sessions after earlier apt upgrade if not done
|
||
- UFW: add new admin public IPs when they change (see access-plan allowlist procedure)
|
||
|
||
|
||
## Session Log
|
||
|
||
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
|
||
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
|
||
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
|
||
- 2026-07-16 · railiance01 · privileged remediation P1–P4 applied (swap, journal, apt, ufw) · ok
|
||
- 2026-07-16 · railiance01 · deep_assessment · Approved P1-P4 done: 4G swap, journal vacuum, apt upgrade, UFW k3s API allowlist; k3s Ready; RAM still tight · ok
|
||
- 2026-07-16 · railiance01 · capacity check after operator 4CPU/16G report: guest still 2/3.8G · watch
|
||
- 2026-07-16 · railiance01 · standard_review · Operator reported 4CPU/16G upgrade; live guest still 2CPU/3.8G after reboot; k3s Ready; load calmer · ok
|
||
- 2026-07-21 · railiance01 · capacity recheck: guest still 2CPU/3.8G (not 4/16); swap 1.1G used · watch
|
||
- 2026-07-21 · railiance01 · short_assist · Capacity recheck 2026-07-21: guest still 2CPU/3.8G not 4/16; k3s Ready; swap 1.1G used · ok
|
||
- 2026-07-22 · railiance01 · short_assist · Reboot completed; guest still 2CPU/3.8G not 4/16; resize must be finished in provider panel · ok
|