ops: complete host-operator ramp-up on railiance01 (WP-0009 T10)
Live observe session verified SSH access, captured baseline and Critical health/load findings (memory pressure, k3s API unavailable). RU checklist closed; engagement phase operating; schedule enabled; no privileged changes.
This commit is contained in:
parent
7257e62dba
commit
6ca167ce19
15 changed files with 334 additions and 90 deletions
|
|
@ -3,7 +3,7 @@ agent: host-operator
|
|||
engagement_id: eng-coulomb-railiance01-ho-001
|
||||
project: coulomb-railiance01
|
||||
last_updated: '2026-07-16'
|
||||
session_count: 1
|
||||
session_count: 3
|
||||
confidentiality: client_owned
|
||||
---
|
||||
|
||||
|
|
@ -16,6 +16,7 @@ confidentiality: client_owned
|
|||
- **Change windows:** prefer low-traffic periods; reboot only with human approval
|
||||
- **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated)
|
||||
- **Out of scope:** app features; other hosts; secret vending
|
||||
- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16)
|
||||
|
||||
## Project Context
|
||||
|
||||
|
|
@ -27,49 +28,77 @@ forward-deployed host-operator pilot (KAIZEN-WP-0009).
|
|||
|
||||
| hostname | role | typical load | services | last review |
|
||||
|----------|------|--------------|----------|-------------|
|
||||
| railiance01 | k3s production | _TBD ramp-up_ | k3s, forgejo/apps (inventory) | never |
|
||||
| railiance01 | k3s production single-node | overload on 2 cores under current mix | k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods | 2026-07-16 |
|
||||
|
||||
## OS & Patch State
|
||||
|
||||
_Pending first OS security pass (RU-05 / weekly protocol)._
|
||||
- Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current)
|
||||
- Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …)
|
||||
- `unattended-upgrades.service` flagged by needrestart for restart
|
||||
- **No package upgrades applied** this session (gated)
|
||||
|
||||
## Security Posture
|
||||
|
||||
_Pending first security snapshot (RU-05)._
|
||||
- UFW **active**: default deny in; allow OpenSSH 22, k3s API **6443/tcp Anywhere**, flannel **8472/udp Anywhere**
|
||||
- Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes
|
||||
- Journal noise: reverse-forward port 18765 already in use (tunnel contention)
|
||||
- Public 6443 exposure: policy review recommended (firewall_change gated)
|
||||
- First snapshot: 2026-07-16 (see health report)
|
||||
|
||||
## Load & Workload Envelope
|
||||
|
||||
_Pending first load review (RU-06)._
|
||||
| Field | Value (2026-07-16 sample) |
|
||||
|-------|---------------------------|
|
||||
| Cores | 2 |
|
||||
| Load 1/5/15 | ~12 / 11 / 16 (**saturated**) |
|
||||
| RAM | 3.8 GiB; no swap; MemAvailable often <0.5 GiB |
|
||||
| PSI memory full | avg60 ~24%, avg300 ~29% |
|
||||
| Disk / | 54% of 96G |
|
||||
| /var/log | ~5.7G (journal ~4.1G) |
|
||||
| Heavy workloads | k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns |
|
||||
| Saturation incident | 2026-07-16 — memory+load critical; k3s API ServiceUnavailable |
|
||||
|
||||
**Envelope intent:** treat load > 4 sustained or MemAvailable < 300Mi as **Watch/Critical**; escalate capacity.
|
||||
|
||||
## Accumulated Findings
|
||||
|
||||
_None yet._
|
||||
- Undersized RAM for workload density is the dominant risk
|
||||
- k3s API instability under memory pressure
|
||||
- Large journald footprint
|
||||
- Pending OS security updates
|
||||
|
||||
## What Worked
|
||||
|
||||
_None yet._
|
||||
- Observe path via `ssh railiance01` (tegwick + id_custodian_agent)
|
||||
- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect
|
||||
- Engagement vault + close-session for durable evidence
|
||||
|
||||
## Watch Points
|
||||
|
||||
- Single-node production: privileged mistakes have full blast radius
|
||||
- DinD / Actions runner privilege model on railiance01 (see railiance-hosts ADRs)
|
||||
- DinD / Actions runner privilege model (railiance-hosts ADR-004)
|
||||
- Disk growth from images, logs, and backups
|
||||
- Do not schedule heavy CI/agents until memory recovers
|
||||
- Bridge reverse-forward port collisions (18765)
|
||||
|
||||
## Recurring Findings
|
||||
|
||||
_None yet._
|
||||
- Memory pressure / no swap · first seen 2026-07-16 · frequency 1
|
||||
- Load ≫ cores · first seen 2026-07-16 · frequency 1
|
||||
|
||||
## Cleared Issues
|
||||
|
||||
_None yet._
|
||||
_None yet (no remediation applied)._
|
||||
|
||||
## Open Threads
|
||||
|
||||
- Complete RU-01 access verification
|
||||
- Capture baseline `vault/baselines/railiance01.md`
|
||||
- First health + load review report
|
||||
- Human decision: add RAM and/or temporary swap
|
||||
- Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443
|
||||
- Re-check k3s API health after memory improves
|
||||
- Align inventory hostname with kernel hostname / DNS naming
|
||||
|
||||
## Session Log
|
||||
|
||||
<!-- YYYY-MM-DD · host(s) · key finding · outcome -->
|
||||
- 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok
|
||||
- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok
|
||||
- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue