ops: apply approved host-operator remediations on railiance01
Record operator approval and implement P1–P4: 4G swapfile, journald vacuum, apt upgrade, and UFW lockdown of k3s API/flannel world-open rules. k3s node Ready post-change; residual risk is still tight RAM.
This commit is contained in:
parent
6730ec0a2c
commit
48443a9abb
8 changed files with 129 additions and 41 deletions
|
|
@ -1,13 +1,10 @@
|
|||
# Outstanding risks — eng-coulomb-railiance01-ho-001
|
||||
|
||||
Populated during operate; finalise at ramp-down (RD-03).
|
||||
|
||||
| Risk | Severity | Mitigation / owner | Status |
|
||||
|------|----------|--------------------|--------|
|
||||
| RAM undersized (3.8G, no swap) for k3s+platform density | Critical | Human capacity decision (RAM and/or swap); reduce concurrent load | open |
|
||||
| k3s API ServiceUnavailable under pressure | Critical | Stabilize memory; re-check API; avoid heavy kubectl while degraded | open |
|
||||
| Load average ≫ 2 cores | High | Same as capacity; schedule non-urgent work off-host | open |
|
||||
| journald ~4.1G | Medium | Approved journal vacuum | open |
|
||||
| Pending security package updates | Medium | Approved OS security pass | open |
|
||||
| UFW allows 6443/8472 from Anywhere | Medium | Policy review + possible firewall_change | open |
|
||||
| Reverse-forward port 18765 collisions | Low–Medium | Bridge/tunnel cleanup | open |
|
||||
| RAM still 3.8G; swap heavily used (~1.9G/4G) | High | Provider RAM upgrade | open |
|
||||
| Load still elevated (~6 on 2 cores) | Medium | Reduce concurrent workloads; capacity | open |
|
||||
| Stale user sessions after apt upgrade | Low | Re-login SSH | open |
|
||||
| UFW 6443 allowlist may miss new admin IPs | Medium | Add `ufw allow from <ip> to any port 6443 proto tcp` | open |
|
||||
| 8472 closed externally | Low | Re-open if multi-node flannel peers appear | accepted (single-node) |
|
||||
| fwupd.service failed restart during upgrade | Low | Investigate if firmware tooling needed | open |
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ agent: host-operator
|
|||
engagement_id: eng-coulomb-railiance01-ho-001
|
||||
project: coulomb-railiance01
|
||||
last_updated: '2026-07-16'
|
||||
session_count: 3
|
||||
session_count: 4
|
||||
confidentiality: client_owned
|
||||
---
|
||||
|
||||
|
|
@ -32,33 +32,34 @@ forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
|||
|
||||
## OS & Patch State
|
||||
|
||||
- Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current)
|
||||
- Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …)
|
||||
- `unattended-upgrades.service` flagged by needrestart for restart
|
||||
- **No package upgrades applied** this session (gated)
|
||||
- Ubuntu 24.04 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — no reboot required after 2026-07-16 upgrade)
|
||||
- **2026-07-16:** full `apt-get upgrade` applied (approved); certs/ca updated; some service restarts deferred
|
||||
- unattended-upgrades / logind may still want restart; re-login SSH sessions recommended
|
||||
|
||||
|
||||
## Security Posture
|
||||
|
||||
- UFW **active**: default deny in; allow OpenSSH 22, k3s API **6443/tcp Anywhere**, flannel **8472/udp Anywhere**
|
||||
- Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes
|
||||
- Journal noise: reverse-forward port 18765 already in use (tunnel contention)
|
||||
- Public 6443 exposure: policy review recommended (firewall_change gated)
|
||||
- First snapshot: 2026-07-16 (see health report)
|
||||
- UFW active: OpenSSH Anywhere; **6443 only from operator IPs** 89.244.90.246 and 85.132.220.102
|
||||
- **8472/udp world-open removed** (single-node; re-add if multi-node flannel peers needed)
|
||||
- UFW backup: /etc/ufw/user.rules.bak.20260716
|
||||
- 2026-07-16 package upgrades applied (security-relevant packages included)
|
||||
|
||||
|
||||
## Load & Workload Envelope
|
||||
|
||||
| Field | Value (2026-07-16 sample) |
|
||||
|-------|---------------------------|
|
||||
| Field | Post-remediation 2026-07-16 |
|
||||
|-------|----------------------------|
|
||||
| Cores | 2 |
|
||||
| Load 1/5/15 | ~12 / 11 / 16 (**saturated**) |
|
||||
| RAM | 3.8 GiB; no swap; MemAvailable often <0.5 GiB |
|
||||
| PSI memory full | avg60 ~24%, avg300 ~29% |
|
||||
| Disk / | 54% of 96G |
|
||||
| /var/log | ~5.7G (journal ~4.1G) |
|
||||
| Heavy workloads | k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns |
|
||||
| Saturation incident | 2026-07-16 — memory+load critical; k3s API ServiceUnavailable |
|
||||
| Load 1m | ~6 (was ~12) — still elevated |
|
||||
| RAM | 3.8 GiB; MemAvailable ~625 Mi |
|
||||
| Swap | **4 GiB /swapfile**, ~1.9 GiB used |
|
||||
| PSI memory full avg60 | ~8% (was ~24%) |
|
||||
| Disk / | ~55% |
|
||||
| /var/log/journal | ~461 Mi (was ~4.1 Gi) |
|
||||
| k3s node | Ready |
|
||||
|
||||
Envelope: swap is emergency cushion; plan RAM upgrade if swap stays high.
|
||||
|
||||
**Envelope intent:** treat load > 4 sustained or MemAvailable < 300Mi as **Watch/Critical**; escalate capacity.
|
||||
|
||||
## Accumulated Findings
|
||||
|
||||
|
|
@ -83,22 +84,31 @@ forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
|||
|
||||
## Recurring Findings
|
||||
|
||||
- Memory pressure / no swap · first seen 2026-07-16 · frequency 1
|
||||
- Load ≫ cores · first seen 2026-07-16 · frequency 1
|
||||
- Memory pressure on 3.8G host · first seen 2026-07-16 · mitigated by swap but capacity still tight
|
||||
- Load ≫ cores · improved after remediation; still watch
|
||||
|
||||
|
||||
## Cleared Issues
|
||||
|
||||
_None yet (no remediation applied)._
|
||||
- No swap · cleared 2026-07-16 via /swapfile 4G + fstab
|
||||
- journald 4G bloat · vacuumed to ~500M target 2026-07-16
|
||||
- World-open k3s API 6443 · restricted to operator IPs 2026-07-16
|
||||
- World-open flannel 8472 · removed 2026-07-16 (single-node)
|
||||
- Pending package upgrades · apt upgrade applied 2026-07-16
|
||||
|
||||
|
||||
## Open Threads
|
||||
|
||||
- Human decision: add RAM and/or temporary swap
|
||||
- Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443
|
||||
- Re-check k3s API health after memory improves
|
||||
- Align inventory hostname with kernel hostname / DNS naming
|
||||
- Plan hardware/provider RAM increase (swap is temporary relief)
|
||||
- Re-login stale SSH sessions after package upgrade
|
||||
- Add UFW allow for new admin IPs when they change
|
||||
- Monitor if 8472 needed for any multi-node peer (currently none)
|
||||
|
||||
|
||||
## Session Log
|
||||
|
||||
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
|
||||
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
|
||||
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
|
||||
- 2026-07-16 · railiance01 · privileged remediation P1–P4 applied (swap, journal, apt, ufw) · ok
|
||||
- 2026-07-16 · railiance01 · deep_assessment · Approved P1-P4 done: 4G swap, journal vacuum, apt upgrade, UFW k3s API allowlist; k3s Ready; RAM still tight · ok
|
||||
|
|
|
|||
|
|
@ -0,0 +1,32 @@
|
|||
# Human approval — privileged remediation batch
|
||||
|
||||
**Engagement:** eng-coulomb-railiance01-ho-001
|
||||
**Date:** 2026-07-16
|
||||
**Approver:** Bernd (operator) — via session instruction: *“approve the suggested tasks… implement them”*
|
||||
**Agent:** host-operator (grok session)
|
||||
|
||||
## Approved items
|
||||
|
||||
| ID | Action | Class |
|
||||
|----|--------|--------|
|
||||
| P1 | Emergency swap file (~4G) on root FS | privileged_ops |
|
||||
| P2 | journald vacuum to ~500M | privileged_ops |
|
||||
| P3 | OS package upgrades (apt) + needrestart awareness | package_upgrade |
|
||||
| P4 | Tighten UFW: remove world-open 6443/8472; allow k3s API from known operator IPs | firewall_change |
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
- Hardware RAM upgrade (provider console — not shell-implementable here)
|
||||
- Unilateral workload kill / pod deletion
|
||||
- Reboot unless needrestart shows kernel requires it **and** operator re-confirms mid-session
|
||||
|
||||
## Rollback notes
|
||||
|
||||
- P1: `swapoff` + remove swapfile + fstab line
|
||||
- P2: logs not recoverable
|
||||
- P3: package-specific rollback / restore
|
||||
- P4: re-add `ufw allow 6443/tcp` and `8472/udp` if access breaks
|
||||
|
||||
## Status
|
||||
|
||||
Approved for immediate implementation under host_observe + gated classes above.
|
||||
Loading…
Add table
Add a link
Reference in a new issue