diff --git a/.custodian-brief.md b/.custodian-brief.md index a1f082f..c05e81a 100644 --- a/.custodian-brief.md +++ b/.custodian-brief.md @@ -2,15 +2,17 @@ # Custodian Brief — kaizen-agentic **Domain:** agents -**Last synced:** 2026-07-16 10:14 UTC +**Last synced:** 2026-07-16 08:48 UTC **State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)* ## Active Workstreams ### Forward-deployed agency: engagements, Kai ledger, host-operator pilot -Progress: 9/11 done | workplan_id: `70f7afb1-78f1-4e3b-8437-48840b5b58cc` +Progress: 7/11 done | workplan_id: `70f7afb1-78f1-4e3b-8437-48840b5b58cc` **Open tasks:** +- · CLI engagement group `52f7d872` +- · Session wire-up `82ac0b5a` - · railiance01 pilot through ramp-up `6b21da0d` - · Absorb and ADR `f212949d` diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/executions.jsonl b/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/executions.jsonl index 5d0823f..d7e011a 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/executions.jsonl +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/executions.jsonl @@ -1,2 +1 @@ {"agent": "host-operator", "duty": "standard_review", "engagement_id": "eng-coulomb-railiance01-ho-001", "execution_time_s": 5.0, "phase": "staffing", "quality_score": 0.7, "success": true, "timestamp": "2026-07-16T10:08:57Z"} -{"agent": "host-operator", "duty": "deep_assessment", "engagement_id": "eng-coulomb-railiance01-ho-001", "execution_time_s": 900.0, "phase": "operating", "quality_score": 0.9, "success": true, "timestamp": "2026-07-16T10:42:51Z"} diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/summary.json b/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/summary.json index ea7b7a2..2443a7f 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/summary.json +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/.kaizen/metrics/host-operator/summary.json @@ -1,9 +1,9 @@ { "agent": "host-operator", - "avg_execution_time_s": 452.5, - "avg_quality_score": 0.8, - "execution_count": 2, - "last_execution": "2026-07-16T10:42:51Z", + "avg_execution_time_s": 5.0, + "avg_quality_score": 0.7, + "execution_count": 1, + "last_execution": "2026-07-16T10:08:57Z", "success_rate": 1.0, "trend": { "quality_score": "stable", diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/ENGAGEMENT.yaml b/engagements/pilots/eng-coulomb-railiance01-ho-001/ENGAGEMENT.yaml index 1edaed6..4ac88cf 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/ENGAGEMENT.yaml +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/ENGAGEMENT.yaml @@ -3,22 +3,24 @@ kind: Engagement metadata: id: eng-coulomb-railiance01-ho-001 request_id: engreq-coulomb-railiance01-ho-001 - created: '2026-07-16' - updated: '2026-07-16' + created: "2026-07-16" + updated: "2026-07-16" workplan: KAIZEN-WP-0009 status: - phase: operating - notes: T10 RU checklist complete 2026-07-16; first health Critical memory/load + phase: staffing + # requested | quoting | funded | staffing | ramp_up | operating + # renewing | ramp_down | closed | cancelled + notes: "Phase 1 scaffold complete; pending access verify to enter ramp_up" spec: client: id: coulomb billing_account: coulomb-ops-kai role: id: host-operator - version: 0.1.0 + version: "0.1.0" targets: - - kind: host - id: railiance01 + - kind: host + id: railiance01 agent_definition: path: agent-host-operator.md derived_from: roles/host-operator/agent-definition.md @@ -36,10 +38,10 @@ spec: contribute_metrics: false contribute_lesson: false human_approval_for: - - privileged_ops - - package_upgrade - - firewall_change - - reboot + - privileged_ops + - package_upgrade + - firewall_change + - reboot redaction_profile: default-host-ops commercial: currency: kai diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/access-plan.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/access-plan.md index af50836..6bcfc56 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/access-plan.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/access-plan.md @@ -9,28 +9,14 @@ | Step | Action | Owner | |------|--------|-------| | 1 | Inventory / facts from `railiance-hosts` (read-only) | operator | -| 2 | SSH as workstation user (tegwick) or cert via ops-warden | operator | +| 2 | SSH cert via ops-warden (`warden sign` / `cert_command`) identity hint `agt` | operator | | 3 | Tunnel if needed (`ops-bridge`, e.g. state-hub-railiance01) | operator | | 4 | Observe session: non-destructive health/load/os checks | host-operator agent | | 5 | Privileged ops only after human approval recorded in vault | human + agent | -## Working path (verified 2026-07-16) - -| Item | Detail | -|------|--------| -| SSH Host alias | `railiance01` in `~/.ssh/config` | -| HostName | 92.205.62.239 | -| User | tegwick | -| IdentityFile | `~/.ssh/id_custodian_agent` (also works with `id_ops` in this environment) | -| Verify command | `ssh -o BatchMode=yes railiance01 'hostname; uptime'` | -| Inventory | `railiance-hosts/inventory/servers.yaml` | -| Bridge | `state-hub-railiance01`, `issue-core-railiance01`, `state-hub-primary` **connected** (static-key) | -| Sudo | passwordless `sudo -n` available for read-only admin inspect (k3s, ufw status, needrestart) | - ## Credential routing -- SSH certificates → **ops-warden** when using cert_command actors -- This pilot used **existing workstation SSH key** path (static key) for observe +- SSH certificates → **ops-warden** - API keys / DB passwords → **OpenBao** via `warden route` (not this agent) - Do **not** message ops-warden for secret values @@ -38,10 +24,10 @@ | Date | Result | Notes | |------|--------|-------| -| 2026-07-16 | **OK** | ICMP OK; `ssh railiance01` interactive-non-batch shell OK; host_observe assessment completed (RU-01) | +| _pending_ | | RU-01 not yet complete | ## Revocation | Date | Action | |------|--------| -| _open_ | On ramp-down: stop renewing agent certs/keys used for this engagement; set schedule disabled; mark here | +| _open_ | On ramp-down: stop renewing agent certs; set schedule disabled; mark here | diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/agent-host-operator.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/agent-host-operator.md index 6d2d305..1390892 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/agent-host-operator.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/agent-host-operator.md @@ -2,24 +2,18 @@ name: host-operator engagement_id: eng-coulomb-railiance01-ho-001 role_id: host-operator -role_version: 0.1.0 +role_version: "0.1.0" description: Host operator for railiance01 (coulomb pilot engagement) category: infrastructure memory: enabled memory_path: engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md targets: -- kind: host - id: railiance01 + - kind: host + id: railiance01 confidentiality: client_owned -phase: operating -access_classes: -- host_observe -- privileged_ops -human_approval_for: -- privileged_ops -- package_upgrade -- firewall_change -- reboot +phase: staffing +access_classes: [host_observe, privileged_ops] +human_approval_for: [privileged_ops, package_upgrade, firewall_change, reboot] base_agent: sys-medic model: inherit --- diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/checklists/ramp-up-status.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/checklists/ramp-up-status.md index 9f7cb6e..d66d2a4 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/checklists/ramp-up-status.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/checklists/ramp-up-status.md @@ -4,15 +4,13 @@ Source checklist: `roles/host-operator/ramp-up.md` | ID | Criterion | Status | Evidence | |----|-----------|--------|----------| -| RU-01 | Access path verified | done | access-plan.md (ssh railiance01 2026-07-16) | -| RU-02 | Host baseline documented | done | vault/baselines/railiance01.md | +| RU-01 | Access path verified | todo | access-plan.md | +| RU-02 | Host baseline documented | todo | vault/baselines/railiance01.md | | RU-03 | Coach / orientation brief filed | done | reports/orientation.md (scaffold) | -| RU-04 | First health review complete | done | reports/2026-07-16-health-review.md | -| RU-05 | Security snapshot initial | done | vault/memory.md Security Posture | -| RU-06 | Load envelope initial | done | vault/memory.md Load & Workload Envelope | +| RU-04 | First health review complete | todo | reports/*-health-review.md | +| RU-05 | Security snapshot initial | todo | vault/memory.md Security Posture | +| RU-06 | Load envelope initial | todo | vault/memory.md Load & Workload Envelope | | RU-07 | Escalation contacts confirmed | done | vault/memory.md Engagement Charter | -| RU-08 | Human approval path tested | done | vault/session-log/2026-07-16-privileged-proposal-dry-run.md | +| RU-08 | Human approval path tested | todo | vault/session-log/ or session log | **Phase gate:** all `done` → set `ENGAGEMENT.yaml` status.phase to `operating` and enable schedule entries. - -**Completed:** 2026-07-16 (T10 live observe ramp-up). diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/commercial/ledger.jsonl b/engagements/pilots/eng-coulomb-railiance01-ho-001/commercial/ledger.jsonl index 1e4497d..251d99e 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/commercial/ledger.jsonl +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/commercial/ledger.jsonl @@ -1,3 +1,2 @@ {"apiVersion":"kaizen.agentic/v1","kind":"KaiLedgerEntry","id":"kai-20260716-quote-open","account":"coulomb-ops-kai","engagement_id":"eng-coulomb-railiance01-ho-001","type":"note","product":"quote_snapshot","capability_tier":4,"amount_kai":0,"currency":"KAI","created_at":"2026-07-16T08:00:00Z","metadata":{"total_quoted_kai":72800,"phase":"staffing","note":"Month-1 estimate recorded; no charge until fund/ramp"}} {"access_surcharge_product": "read_only", "account": "coulomb-ops-kai", "amount_kai": 1600, "apiVersion": "kaizen.agentic/v1", "capability_tier": 4, "created_at": "2026-07-16T10:08:57Z", "currency": "KAI", "engagement_id": "eng-coulomb-railiance01-ho-001", "id": "kai-20260716T100857Z-standard_review", "kind": "KaiLedgerEntry", "metadata": {"access_class": "read_only", "phase": "staffing", "success": true, "target": "railiance01"}, "product": "standard_review", "session_ref": "reports/2026-07-16-standard-review.md", "type": "duty_charge"} -{"access_surcharge_product": "host_observe", "account": "coulomb-ops-kai", "amount_kai": 4900, "apiVersion": "kaizen.agentic/v1", "capability_tier": 4, "created_at": "2026-07-16T10:42:51Z", "currency": "KAI", "engagement_id": "eng-coulomb-railiance01-ho-001", "id": "kai-20260716T104251Z-deep_assessment", "kind": "KaiLedgerEntry", "metadata": {"access_class": "host_observe", "phase": "operating", "success": true, "target": "railiance01"}, "product": "deep_assessment", "session_ref": "reports/2026-07-16-deep-assessment.md", "type": "duty_charge"} diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-deep-assessment.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-deep-assessment.md deleted file mode 100644 index 25a3dc8..0000000 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-deep-assessment.md +++ /dev/null @@ -1,13 +0,0 @@ -# Session report — eng-coulomb-railiance01-ho-001 - -- **Date:** 2026-07-16 -- **Duty:** deep_assessment -- **Targets:** railiance01 -- **Outcome:** success -- **Phase:** operating - -## Summary - -T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating - -_Billing metadata only in commercial/ledger.jsonl; no secrets in this report._ diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-health-review.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-health-review.md deleted file mode 100644 index a6ca430..0000000 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/reports/2026-07-16-health-review.md +++ /dev/null @@ -1,96 +0,0 @@ -# Health & load review — railiance01 — 2026-07-16 - -**Engagement:** eng-coulomb-railiance01-ho-001 -**Phase:** ramp_up (first live observe) -**Access class:** host_observe -**Protocols:** load-workload-review + sys-medic subset - ---- - -## 1. Executive Summary - -railiance01 is a **2-core / 3.8 GiB** Ubuntu 24.04 single-node k3s host under **severe memory pressure and sustained CPU/load overload**. Root disk capacity is acceptable (~54%). The k3s API was **ServiceUnavailable** during assessment (control plane struggling). No privileged remediation was performed. - -**Overall health: Critical** (confidence: High for host-level memory/load; Medium for full pod inventory due to API unavailability) - -## 2. Health Status - -| Dimension | Status | -|-----------|--------| -| Aggregate | **Critical** | -| Memory | Critical (no swap, PSI elevated, ~hundreds of MiB available) | -| CPU / load | Degraded–Critical (load ≫ 2 cores) | -| Disk capacity | Healthy (~54% root) | -| Disk / logs | Watch (`/var/log` ~5.7G, journal ~4.1G) | -| k3s API | Critical / unavailable at sample time | -| Network listeners | Watch (expected k3s + SSH; review exposure of 6443) | - -## 3. Findings - -### F1 — Memory exhaustion / no swap -- **Severity:** Critical -- **Evidence:** MemTotal ~3.8Gi; MemAvailable often <500Mi; SwapTotal 0; PSI memory `full avg60≈24% avg300≈29%`; `kswapd0` long-running -- **Why it matters:** OOM risk; thrashing; control-plane instability -- **Likely cause:** Workload density (k3s + Forgejo + Temporal + activity-core + state-hub edge + platform pods) on undersized RAM -- **Next step:** Human-approved capacity plan (add RAM and/or swap emergency cushion; reduce concurrent workloads). **Do not** kill workloads without owner approval - -### F2 — Load far above CPU capacity -- **Severity:** High -- **Evidence:** load average ~12/11/16 with `nproc=2`; vmstat shows high `sy` and `wa`, heavy block-in -- **Why it matters:** Latency for all services; scheduler saturation -- **Likely cause:** Memory reclaim + I/O + k3s restart recovery -- **Next step:** Correlate with k3s restart time (~10:39 UTC); re-sample after memory relief - -### F3 — k3s API ServiceUnavailable -- **Severity:** Critical -- **Evidence:** `sudo k3s kubectl get node` → server unable to handle request; `k3s server` process high CPU shortly after start -- **Why it matters:** Cluster ops and many apps depend on API -- **Likely cause:** Memory pressure / control-plane restart under load -- **Next step:** Re-check API when MemAvailable improves; avoid concurrent heavy kubectl - -### F4 — Large journald footprint -- **Severity:** Medium -- **Evidence:** `/var/log/journal` ~4.1G of ~5.7G under `/var/log` -- **Why it matters:** Disk growth; I/O; eventual fill -- **Next step:** Propose `journalctl --vacuum-size=` **with approval** (privileged / system change) - -### F5 — Pending OS package updates (incl. security) -- **Severity:** Medium -- **Evidence:** `apt list --upgradable` shows many packages (bind9-*, curl, ca-certificates, dpkg, …) -- **needrestart:** kernel current (KSTA 1); `unattended-upgrades.service` flagged -- **Next step:** Scheduled OS security pass with human approval for upgrades - -### F6 — Reverse tunnel port contention (18765) -- **Severity:** Low–Medium -- **Evidence:** repeated sshd errors binding 127.0.0.1:18765 address already in use -- **Why it matters:** ops-bridge / reverse-forward fragility -- **Next step:** Operator cleanup of stale forwards; bridge config audit - -### F7 — Public exposure of k3s API port in UFW -- **Severity:** Medium (policy) -- **Evidence:** UFW allows 6443/tcp and 8472/udp from Anywhere -- **Next step:** Confirm intended threat model; prefer restricted sources if possible (**firewall_change** gated) - -## 4. Immediate Safe Actions (no approval needed) - -1. Continue observe-only monitoring; re-sample load/memory in 15–30 minutes -2. Prefer not running heavy builds / additional agents on this host until memory recovers -3. Document findings in engagement vault (this report + memory sections) -4. Escalate capacity concern to human operator - -## 5. Escalation - -- **Human operator (Bernd / railiance ops):** capacity (RAM/swap), whether k3s restart was intentional, approval for journal vacuum and package upgrades -- **App owners:** Forgejo, activity-core, Temporal — if service degradation reported - -## 6. Suggested inspect commands - -```bash -ssh railiance01 'free -h; uptime; cat /proc/pressure/memory' -ssh railiance01 'sudo k3s kubectl get node; sudo k3s kubectl get pods -A | head' -ssh railiance01 'sudo du -sh /var/log/journal' -``` - -## 7. Privileged proposals (NOT executed — see RU-08 dry-run) - -See `vault/session-log/2026-07-16-privileged-proposal-dry-run.md`. diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/schedule.yml b/engagements/pilots/eng-coulomb-railiance01-ho-001/schedule.yml index 0021bad..3bc390f 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/schedule.yml +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/schedule.yml @@ -11,8 +11,8 @@ entries: - k3s-node-health-assessment cadence: daily cron: "0 9 * * 1-5" - enabled: true - notes: "Enabled after T10 ramp-up exit 2026-07-16; manual prepare until activity-core wired" + enabled: false + notes: "Enable after ramp_up exit (RU checklist complete)" - id: weekly-os-security agent: host-operator @@ -21,5 +21,5 @@ entries: - os-security-pass cadence: weekly cron: "0 10 * * 1" - enabled: true + enabled: false notes: "Privileged upgrades remain human-gated even when enabled" diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/baselines/railiance01.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/baselines/railiance01.md index 0319b28..5610c62 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/baselines/railiance01.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/baselines/railiance01.md @@ -1,63 +1,47 @@ # Baseline — railiance01 **Engagement:** eng-coulomb-railiance01-ho-001 -**Status:** captured -**Last captured:** 2026-07-16 (observe session, T10 ramp-up) -**Access:** `ssh railiance01` (user tegwick, IdentityFile id_custodian_agent) — host_observe +**Status:** pending first observe session (RU-02) +**Last captured:** — ## Identity | Field | Value | |-------|-------| -| Hostname (kernel) | 239.62.205.92.host.secureserver.net | -| Inventory name | railiance01 | -| Public IP (inventory) | 92.205.62.239 | -| Inventory ref | railiance-hosts | -| Role | Single-node k3s production (forgejo/apps, activity-core, platform services) | +| Hostname | railiance01 | +| Inventory | railiance-hosts | +| Notes | k3s production; forgejo/apps | -## OS - -| Field | Value | -|-------|-------| -| Distro | Ubuntu 24.04.3 LTS (noble) | -| Kernel | 6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC | -| Arch | x86_64 | -| Uptime at capture | 129 days, 13:34 | -| CPU cores (nproc) | **2** | -| Memory | **3.8 GiB** total | -| Swap | **none** | -| Root FS | /dev/sda1 ext4 96G, **54% used** (52G/45G free) | - -## Capture commands (non-destructive) +## Capture checklist (fill on first session) ```bash -ssh railiance01 'hostname; uptime; uname -a; nproc; free -h; df -hT; cat /etc/os-release | head -8' +hostname +uptime +uname -a +cat /etc/os-release +nproc +free -h +df -h +ss -tuln | head -40 +# if k3s: +kubectl get node -o wide 2>/dev/null || k3s kubectl get node -o wide 2>/dev/null ``` -## Envelope seed (2026-07-16) +## Recorded values + +_To be filled during ramp-up._ + +## Known quirks (from docs, pre-session) + +- Forgejo + in-cluster Actions runner documented in railiance-hosts ADR-004 +- State Hub / activity-core historically deployed on this cluster path +- Access often via ops-bridge from workstation + +## Envelope seed | Metric | Baseline | Notes | |--------|----------|-------| -| Load 1/5/15 | ~12 / 11 / 16 | **≫ 2 cores** — overloaded | -| MemAvailable | ~0.1–0.5 GiB | severe pressure; PSI full avg60 ~24% | -| Disk / | 54% | OK for capacity; watch growth | -| /var/log | ~5.7G | journal alone ~4.1G | -| Top host processes | k3s server, gitea, temporal-server, activity-core/state-hub python | | -| k3s API | ServiceUnavailable at capture | control plane unstable under memory pressure | - -## Known quirks - -- Single-node production: no HA; memory headroom is the binding constraint -- DinD / Actions runner privilege model (railiance-hosts ADR-004) -- ops-bridge reverse tunnels to localhost ports (state-hub 18000/18001, etc.) -- sshd journal noise: reverse-forward port 18765 already in use (tunnel contention) -- Inventory hostname vs DNS: reverse DNS is HostEurope generic name - -## Access path (RU-01) - -| Method | Result | -|--------|--------| -| ICMP to inventory IP | reachable | -| `ssh railiance01` (tegwick) | **success** — non-interactive shell | -| ops-bridge tunnels to host | connected (state-hub-railiance01, issue-core-railiance01, …) | -| Privileged ops | not exercised; sudo -n available for read-only needrestart/ufw/k3s in this session | +| Load | | | +| Memory | | | +| Disk | | | +| Top workloads | | | diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/handoff/risks.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/handoff/risks.md index fa8ecfa..ac30771 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/handoff/risks.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/handoff/risks.md @@ -1,13 +1,7 @@ # Outstanding risks — eng-coulomb-railiance01-ho-001 -Populated during operate; finalise at ramp-down (RD-03). +_Populated during operate and finalised at ramp-down (RD-03)._ | Risk | Severity | Mitigation / owner | Status | |------|----------|--------------------|--------| -| RAM undersized (3.8G, no swap) for k3s+platform density | Critical | Human capacity decision (RAM and/or swap); reduce concurrent load | open | -| k3s API ServiceUnavailable under pressure | Critical | Stabilize memory; re-check API; avoid heavy kubectl while degraded | open | -| Load average ≫ 2 cores | High | Same as capacity; schedule non-urgent work off-host | open | -| journald ~4.1G | Medium | Approved journal vacuum | open | -| Pending security package updates | Medium | Approved OS security pass | open | -| UFW allows 6443/8472 from Anywhere | Medium | Policy review + possible firewall_change | open | -| Reverse-forward port 18765 collisions | Low–Medium | Bridge/tunnel cleanup | open | +| _none recorded_ | | | | diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md index 0b0f417..2ade042 100644 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md +++ b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/memory.md @@ -3,7 +3,7 @@ agent: host-operator engagement_id: eng-coulomb-railiance01-ho-001 project: coulomb-railiance01 last_updated: '2026-07-16' -session_count: 3 +session_count: 1 confidentiality: client_owned --- @@ -16,7 +16,6 @@ confidentiality: client_owned - **Change windows:** prefer low-traffic periods; reboot only with human approval - **Escalation:** coulomb / railiance human operator (Bernd / on-call as designated) - **Out of scope:** app features; other hosts; secret vending -- **Access:** `ssh railiance01` as tegwick (host_observe verified 2026-07-16) ## Project Context @@ -28,77 +27,49 @@ forward-deployed host-operator pilot (KAIZEN-WP-0009). | hostname | role | typical load | services | last review | |----------|------|--------------|----------|-------------| -| railiance01 | k3s production single-node | overload on 2 cores under current mix | k3s, gitea/forgejo, temporal, activity-core, state-hub edge, traefik, platform pods | 2026-07-16 | +| railiance01 | k3s production | _TBD ramp-up_ | k3s, forgejo/apps (inventory) | never | ## OS & Patch State -- Ubuntu 24.04.3 LTS (noble); kernel 6.8.0-87-generic (needrestart KSTA=1 — kernel current) -- Many packages upgradable (security-relevant: bind9-*, curl, ca-certificates, dpkg, …) -- `unattended-upgrades.service` flagged by needrestart for restart -- **No package upgrades applied** this session (gated) +_Pending first OS security pass (RU-05 / weekly protocol)._ ## Security Posture -- UFW **active**: default deny in; allow OpenSSH 22, k3s API **6443/tcp Anywhere**, flannel **8472/udp Anywhere** -- Listeners include SSH, k3s components, localhost bridge ports (18000/18001), gitea stack processes -- Journal noise: reverse-forward port 18765 already in use (tunnel contention) -- Public 6443 exposure: policy review recommended (firewall_change gated) -- First snapshot: 2026-07-16 (see health report) +_Pending first security snapshot (RU-05)._ ## Load & Workload Envelope -| Field | Value (2026-07-16 sample) | -|-------|---------------------------| -| Cores | 2 | -| Load 1/5/15 | ~12 / 11 / 16 (**saturated**) | -| RAM | 3.8 GiB; no swap; MemAvailable often <0.5 GiB | -| PSI memory full | avg60 ~24%, avg300 ~29% | -| Disk / | 54% of 96G | -| /var/log | ~5.7G (journal ~4.1G) | -| Heavy workloads | k3s server, gitea, temporal-server, activity-core worker/API, state-hub edge uvicorn, traefik, coredns | -| Saturation incident | 2026-07-16 — memory+load critical; k3s API ServiceUnavailable | - -**Envelope intent:** treat load > 4 sustained or MemAvailable < 300Mi as **Watch/Critical**; escalate capacity. +_Pending first load review (RU-06)._ ## Accumulated Findings -- Undersized RAM for workload density is the dominant risk -- k3s API instability under memory pressure -- Large journald footprint -- Pending OS security updates +_None yet._ ## What Worked -- Observe path via `ssh railiance01` (tegwick + id_custodian_agent) -- passwordless `sudo -n` for read-only needrestart/ufw/k3s inspect -- Engagement vault + close-session for durable evidence +_None yet._ ## Watch Points - Single-node production: privileged mistakes have full blast radius -- DinD / Actions runner privilege model (railiance-hosts ADR-004) +- DinD / Actions runner privilege model on railiance01 (see railiance-hosts ADRs) - Disk growth from images, logs, and backups -- Do not schedule heavy CI/agents until memory recovers -- Bridge reverse-forward port collisions (18765) ## Recurring Findings -- Memory pressure / no swap · first seen 2026-07-16 · frequency 1 -- Load ≫ cores · first seen 2026-07-16 · frequency 1 +_None yet._ ## Cleared Issues -_None yet (no remediation applied)._ +_None yet._ ## Open Threads -- Human decision: add RAM and/or temporary swap -- Human approval: journal vacuum; security package upgrades; UFW source restriction for 6443 -- Re-check k3s API health after memory improves -- Align inventory hostname with kernel hostname / DNS naming +- Complete RU-01 access verification +- Capture baseline `vault/baselines/railiance01.md` +- First health + load review report ## Session Log + - 2026-07-16 · railiance01 · standard_review · T09 wire-up smoke: prepare+close-session path verified (no host access) · ok -- 2026-07-16 · railiance01 · first live observe: Critical memory/load; k3s API unavailable; RU checklist complete · ok -- 2026-07-16 · railiance01 · deep_assessment · T10 ramp-up complete: Critical memory/load, k3s API unavailable, RU all done, phase operating · ok diff --git a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/session-log/2026-07-16-privileged-proposal-dry-run.md b/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/session-log/2026-07-16-privileged-proposal-dry-run.md deleted file mode 100644 index b960da3..0000000 --- a/engagements/pilots/eng-coulomb-railiance01-ho-001/vault/session-log/2026-07-16-privileged-proposal-dry-run.md +++ /dev/null @@ -1,62 +0,0 @@ -# RU-08 — Privileged action proposal (DRY RUN — not executed) - -**Engagement:** eng-coulomb-railiance01-ho-001 -**Date:** 2026-07-16 -**Author:** host-operator (grok session) -**Status:** proposal only — **no human approval recorded; no changes applied** - -## Purpose - -Prove the privilege gate path: agent may **propose** privileged work; execution requires recorded human approval. - -## Proposed actions (optional, priority order) - -### P1 — Emergency swap file (if OOM imminent) - -| Field | Value | -|-------|--------| -| Class | `privileged_ops` | -| Blast radius | Host-wide; disk for swapfile; may pause I/O briefly | -| Rollback | `swapoff` + remove swapfile | -| Command sketch | create 2–4G swapfile on root FS (exact steps only after approval) | -| Risk | Disk wear; masks capacity problem | - -### P2 — journald vacuum - -| Field | Value | -|-------|--------| -| Class | `privileged_ops` | -| Blast radius | Historical logs discarded | -| Rollback | none (logs gone) | -| Command sketch | `journalctl --vacuum-size=500M` | -| Risk | Loss of forensic depth | - -### P3 — Package security upgrades - -| Field | Value | -|-------|--------| -| Class | `package_upgrade` | -| Blast radius | Service restarts; possible brief downtime | -| Rollback | package-specific; may need restore | -| Command sketch | `apt-get update && apt-get upgrade` (or unattended security only) | -| Risk | Regression on production single-node | - -### P4 — Restrict UFW sources for 6443/8472 - -| Field | Value | -|-------|--------| -| Class | `firewall_change` | -| Blast radius | May lock out nodes/agents if mis-scoped | -| Rollback | re-add rules | -| Risk | High on single-node misconfiguration | - -## Approval record - -| Approver | Decision | Date | Notes | -|----------|----------|------|-------| -| _none_ | pending | | Dry-run only for RU-08 | - -## Gate test result - -- Proposal written to vault without executing changes: **PASS (RU-08)** -- Access class used this session: **host_observe** only diff --git a/workplans/kaizen-agentic-WP-0009-forward-deployed-agency.md b/workplans/kaizen-agentic-WP-0009-forward-deployed-agency.md index fa972a7..3783f95 100644 --- a/workplans/kaizen-agentic-WP-0009-forward-deployed-agency.md +++ b/workplans/kaizen-agentic-WP-0009-forward-deployed-agency.md @@ -42,7 +42,7 @@ tasks: status: done title: Wire prepare/session-close to vault paths, metrics, and ledger entries - id: T10 - status: done + status: todo title: Run railiance01 host-operator Phase 1 pilot through ramp-up exit - id: T11 status: todo @@ -216,16 +216,16 @@ points at close-session. Tests cover formula, scrub, and CLI. ```task id: KAIZEN-WP-0009-T10 -status: done +status: todo priority: high state_hub_task_id: "6b21da0d-7ba5-4c64-92cb-9237cf71a102" ``` -**Delivered 2026-07-16:** Live observe on railiance01 via `ssh railiance01` -(tegwick). RU-01–08 complete; phase **operating**. Health: **Critical** memory -pressure (3.8G, no swap), load ≫ 2 cores, k3s API ServiceUnavailable at sample. -Evidence: baselines, health-review, privileged dry-run (RU-08), schedule enabled, -deep_assessment close-session (4900 Kai). No privileged changes applied. +Execute architecture §11 ramp-up runbook against railiance01 (observe-class +access via warden/cert paths). Exit when RU checklist complete and phase +`operating`. Privileged changes only with recorded human approval. + +Evidence: baseline file, first health report, checklist marks, ledger entries. ## Absorb and ADR