coulomb/kaizen-agentic
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
kaizen-agentic/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md
tegwick 6ca167ce19
Some checks failed
CI Smoke / host-smoke (push) Successful in 1s
Details
CI Smoke / container-smoke (push) Successful in 36s
Details
ci / test (push) Failing after 3m41s
Details
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) 
Live observe session verified SSH access, captured baseline and Critical
health/load findings (memory pressure, k3s API unavailable). RU checklist
closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00

4.2 KiB
Raw Blame History

agent engagement_id project last_updated session_count confidentiality
host-operator eng-coulomb-railiance01-ho-001 coulomb-railiance01 2026-07-16 3 client_owned

Host Operator Memory — railiance01 pilot

Engagement Charter

  • Duty: Keep railiance01 operational and secure; OS currency; load and workload review
  • Cadence: daily health/load (business days); weekly OS/security pass
  • Change windows: prefer low-traffic periods; reboot only with human approval
  • Escalation: coulomb / railiance human operator (Bernd / on-call as designated)
  • Out of scope: app features; other hosts; secret vending
  • Access: ssh railiance01 as tegwick (host_observe verified 2026-07-16)

Project Context

Coulomb ecosystem production host railiance01 runs k3s and platform workloads (including Forgejo/apps per railiance-hosts docs). This engagement is the first forward-deployed host-operator pilot (KAIZEN-WP-0009).

Host Profiles

hostname role typical load services last review
railiance01 k3s production single-node overload on 2 cores under current mix k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods 2026-07-16

OS & Patch State

  • Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current)
  • Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …)
  • unattended-upgrades.service flagged by needrestart for restart
  • No package upgrades applied this session (gated)

Security Posture

  • UFW active: default deny in; allow OpenSSH 22, k3s API 6443/tcp Anywhere, flannel 8472/udp Anywhere
  • Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes
  • Journal noise: reverse-forward port 18765 already in use (tunnel contention)
  • Public 6443 exposure: policy review recommended (firewall_change gated)
  • First snapshot: 2026-07-16 (see health report)

Load & Workload Envelope

Field Value (2026-07-16 sample)
Cores 2
Load 1/5/15 ~12 / 11 / 16 (saturated)
RAM 3.8 GiB; no swap; MemAvailable often <0.5 GiB
PSI memory full avg60 ~24%, avg300 ~29%
Disk / 54% of 96G
/var/log ~5.7G (journal ~4.1G)
Heavy workloads k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns
Saturation incident 2026-07-16 — memory+load critical; k3s API ServiceUnavailable

Envelope intent: treat load > 4 sustained or MemAvailable < 300Mi as Watch/Critical; escalate capacity.

Accumulated Findings

  • Undersized RAM for workload density is the dominant risk
  • k3s API instability under memory pressure
  • Large journald footprint
  • Pending OS security updates

What Worked

  • Observe path via ssh railiance01 (tegwick + id_custodian_agent)
  • passwordless sudo -n for read-only needrestart/ufw/k3s inspect
  • Engagement vault + close-session for durable evidence

Watch Points

  • Single-node production: privileged mistakes have full blast radius
  • DinD / Actions runner privilege model (railiance-hosts ADR-004)
  • Disk growth from images, logs, and backups
  • Do not schedule heavy CI/agents until memory recovers
  • Bridge reverse-forward port collisions (18765)

Recurring Findings

  • Memory pressure / no swap · first seen 2026-07-16 · frequency 1
  • Load ≫ cores · first seen 2026-07-16 · frequency 1

Cleared Issues

None yet (no remediation applied).

Open Threads

  • Human decision: add RAM and/or temporary swap
  • Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443
  • Re-check k3s API health after memory improves
  • Align inventory hostname with kernel hostname / DNS naming

Session Log

  • 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
  • 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
  • 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok