coulomb/kaizen-agentic
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
kaizen-agentic/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md
tegwick 48443a9abb
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 4s
Details
ci / test (push) Failing after 5s
Details
ops: apply approved host-operator remediations on railiance01 
Record operator approval and implement P1–P4: 4G swapfile, journald vacuum,
apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node
Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00

4.5 KiB
Raw Blame History

agent engagement_id project last_updated session_count confidentiality
host-operator eng-coulomb-railiance01-ho-001 coulomb-railiance01 2026-07-16 4 client_owned

Host Operator Memory — railiance01 pilot

Engagement Charter

  • Duty: Keep railiance01 operational and secure; OS currency; load and workload review
  • Cadence: daily health/load (business days); weekly OS/security pass
  • Change windows: prefer low-traffic periods; reboot only with human approval
  • Escalation: coulomb / railiance human operator (Bernd / on-call as designated)
  • Out of scope: app features; other hosts; secret vending
  • Access: ssh railiance01 as tegwick (host_observe verified 2026-07-16)

Project Context

Coulomb ecosystem production host railiance01 runs k3s and platform workloads (including Forgejo/apps per railiance-hosts docs). This engagement is the first forward-deployed host-operator pilot (KAIZEN-WP-0009).

Host Profiles

hostname role typical load services last review
railiance01 k3s production single-node overload on 2 cores under current mix k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods 2026-07-16

OS & Patch State

  • Ubuntu 24.04 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — no reboot required after 2026-07-16 upgrade)
  • 2026-07-16: full apt-get upgrade applied (approved); certs/ca updated; some service restarts deferred
  • unattended-upgrades / logind may still want restart; re-login SSH sessions recommended

Security Posture

  • UFW active: OpenSSH Anywhere; 6443 only from operator IPs 89.244.90.246 and 85.132.220.102
  • 8472/udp world-open removed (single-node; re-add if multi-node flannel peers needed)
  • UFW backup: /etc/ufw/user.rules.bak.20260716
  • 2026-07-16 package upgrades applied (security-relevant packages included)

Load & Workload Envelope

Field Post-remediation 2026-07-16
Cores 2
Load 1m ~6 (was ~12) — still elevated
RAM 3.8 GiB; MemAvailable ~625 Mi
Swap 4 GiB /swapfile, ~1.9 GiB used
PSI memory full avg60 ~8% (was ~24%)
Disk / ~55%
/var/log/journal ~461 Mi (was ~4.1 Gi)
k3s node Ready

Envelope: swap is emergency cushion; plan RAM upgrade if swap stays high.

Accumulated Findings

  • Undersized RAM for workload density is the dominant risk
  • k3s API instability under memory pressure
  • Large journald footprint
  • Pending OS security updates

What Worked

  • Observe path via ssh railiance01 (tegwick + id_custodian_agent)
  • passwordless sudo -n for read-only needrestart/ufw/k3s inspect
  • Engagement vault + close-session for durable evidence

Watch Points

  • Single-node production: privileged mistakes have full blast radius
  • DinD / Actions runner privilege model (railiance-hosts ADR-004)
  • Disk growth from images, logs, and backups
  • Do not schedule heavy CI/agents until memory recovers
  • Bridge reverse-forward port collisions (18765)

Recurring Findings

  • Memory pressure on 3.8G host · first seen 2026-07-16 · mitigated by swap but capacity still tight
  • Load ≫ cores · improved after remediation; still watch

Cleared Issues

  • No swap · cleared 2026-07-16 via /swapfile 4G + fstab
  • journald 4G bloat · vacuumed to ~500M target 2026-07-16
  • World-open k3s API 6443 · restricted to operator IPs 2026-07-16
  • World-open flannel 8472 · removed 2026-07-16 (single-node)
  • Pending package upgrades · apt upgrade applied 2026-07-16

Open Threads

  • Plan hardware/provider RAM increase (swap is temporary relief)
  • Re-login stale SSH sessions after package upgrade
  • Add UFW allow for new admin IPs when they change
  • Monitor if 8472 needed for any multi-node peer (currently none)

Session Log

  • 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
  • 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
  • 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
  • 2026-07-16 · railiance01 · privileged remediation P1–P4 applied (swap, journal, apt, ufw) · ok
  • 2026-07-16 · railiance01 · deep_assessment · Approved P1-P4 done: 4G swap, journal vacuum, apt upgrade, UFW k3s API allowlist; k3s Ready; RAM still tight · ok