coulomb/kaizen-agentic
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
kaizen-agentic/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md

115 lines
4.5 KiB
Markdown
Raw Normal View History

feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
---
agent: host-operator
engagement_id: eng-coulomb-railiance01-ho-001
project: coulomb-railiance01
feat: engagement close-session wires vault, metrics, Kai ledger (WP-0009 T09) Add close_session helpers and CLI to append vault session logs, engagement- scoped metrics, duty Kai charges, and client reports. Rejects sensitive summaries. Pilot smoke-closed once; prepare points operators at close-session.
2026-07-16 12:09:05 +02:00
last_updated: '2026-07-16'
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
session_count: 4
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
confidentiality: client_owned
---
# Host Operator Memory — railiance01 pilot
## Engagement Charter
- **Duty:** Keep railiance01 operational and secure; OS currency; load and workload review
- **Cadence:** daily health/load (business days); weekly OS/security pass
- **Change windows:** prefer low-traffic periods; reboot only with human approval
- **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated)
- **Out of scope:** app features; other hosts; secret vending
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16)
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Project Context
Coulomb ecosystem production host `railiance01` runs k3s and platform workloads
(including Forgejo/apps per railiance-hosts docs). This engagement is the first
forward-deployed host-operator pilot (KAIZEN-WP-0009).
## Host Profiles
| hostname | role | typical load | services | last review |
|----------|------|--------------|----------|-------------|
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
| railiance01 | k3s production single-node | overload on 2 cores under current mix | k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods | 2026-07-16 |
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## OS & Patch State
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- Ubuntu 24.04 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — no reboot required after 2026-07-16 upgrade)
- **2026-07-16:** full `apt-get upgrade` applied (approved); certs/ca updated; some service restarts deferred
- unattended-upgrades / logind may still want restart; re-login SSH sessions recommended
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Security Posture
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- UFW active: OpenSSH Anywhere; **6443 only from operator IPs** 89.244.90.246 and 85.132.220.102
- **8472/udp world-open removed** (single-node; re-add if multi-node flannel peers needed)
- UFW backup: /etc/ufw/user.rules.bak.20260716
- 2026-07-16 package upgrades applied (security-relevant packages included)
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Load & Workload Envelope
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
| Field | Post-remediation 2026-07-16 |
|-------|----------------------------|
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
| Cores | 2 |
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
| Load 1m | ~6 (was ~12) — still elevated |
| RAM | 3.8 GiB; MemAvailable ~625 Mi |
| Swap | **4 GiB /swapfile**, ~1.9 GiB used |
| PSI memory full avg60 | ~8% (was ~24%) |
| Disk / | ~55% |
| /var/log/journal | ~461 Mi (was ~4.1 Gi) |
| k3s node | Ready |
Envelope: swap is emergency cushion; plan RAM upgrade if swap stays high.
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Accumulated Findings
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- Undersized RAM for workload density is the dominant risk
- k3s API instability under memory pressure
- Large journald footprint
- Pending OS security updates
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## What Worked
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- Observe path via `ssh railiance01` (tegwick + id_custodian_agent)
- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect
- Engagement vault + close-session for durable evidence
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Watch Points
- Single-node production: privileged mistakes have full blast radius
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
- Disk growth from images, logs, and backups
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- Do not schedule heavy CI/agents until memory recovers
- Bridge reverse-forward port collisions (18765)
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Recurring Findings
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- Memory pressure on 3.8G host · first seen 2026-07-16 · mitigated by swap but capacity still tight
- Load ≫ cores · improved after remediation; still watch
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Cleared Issues
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- No swap · cleared 2026-07-16 via /swapfile 4G + fstab
- journald 4G bloat · vacuumed to ~500M target 2026-07-16
- World-open k3s API 6443 · restricted to operator IPs 2026-07-16
- World-open flannel 8472 · removed 2026-07-16 (single-node)
- Pending package upgrades · apt upgrade applied 2026-07-16
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Open Threads
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- Plan hardware/provider RAM increase (swap is temporary relief)
- Re-login stale SSH sessions after package upgrade
- Add UFW allow for new admin IPs when they change
- Monitor if 8472 needed for any multi-node peer (currently none)
feat: Phase 1 FDA host-operator Role and railiance01 pilot (WP-0009 T02–T07) Lock DEC-FDA-001 working defaults; add roles/host-operator package with OS/security and load protocols; scaffold eng-coulomb-railiance01-ho-001 with bound agent, vault, ramp checklists, and Kai quote/ledger.
2026-07-16 10:47:52 +02:00
## Session Log
feat: engagement close-session wires vault, metrics, Kai ledger (WP-0009 T09) Add close_session helpers and CLI to append vault session logs, engagement- scoped metrics, duty Kai charges, and client reports. Rejects sensitive summaries. Pilot smoke-closed once; prepare points operators at close-session.
2026-07-16 12:09:05 +02:00
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
ops: complete host-operator ramp-up on railiance01 (WP-0009 T10) Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
2026-07-16 12:43:00 +02:00
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
ops: apply approved host-operator remediations on railiance01 Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
2026-07-16 13:05:25 +02:00
- 2026-07-16 · railiance01 · privileged remediation P1–P4 applied (swap, journal, apt, ufw) · ok
- 2026-07-16 · railiance01 · deep_assessment · Approved P1-P4 done: 4G swap, journal vacuum, apt upgrade, UFW k3s API allowlist; k3s Ready; RAM still tight · ok
Reference in a new issue Copy permalink