kaizen-agentic/docs/forward-deployed-engagement-architecture.md
tegwick 97d0739537
Some checks failed
CI Smoke / host-smoke (push) Successful in 1s
CI Smoke / container-smoke (push) Successful in 1m12s
ci / test (push) Has been cancelled
docs: ADR-007 and FDA playbook; finish WP-0009 (T11)
Absorb railiance01 pilot supplier notes into the forward-deployed engagement
playbook, accept ADR-007 elevating DEC-FDA-001, promote architecture and
business model to v1.0, and mark KAIZEN-WP-0009 finished.
2026-07-16 12:44:56 +02:00

28 KiB
Raw Blame History

Forward-Deployed Engagement Architecture

Status: v1.0 accepted (ADR-007) Date: 2026-07-16 Type: architecture extension Canon ADR: ADR-007 Companion business model: wiki/ForwardDeployedAgencyBusinessModel.md Playbook: integrations/forward-deployed-engagement-playbook.md Decisions: docs/decisions/DEC-FDA-001-working-defaults.md (elevated into ADR-007) Extends: ADR-002 (memory), ADR-003 (protocols), ADR-005 (schedule), ADR-006 (customer engagement) Pilot tree: engagements/pilots/eng-coulomb-railiance01-ho-001/


1. Purpose

Extend kaizen-agentic from installing agent personas into repos to staffing forward-deployed agents for client engagements, with:

  1. A client request → provision path that yields a bound agent definition
  2. Explicit ramp-up and ramp-down phases
  3. Confidential client knowledge separated from supplier Role IP
  4. Hooks for Kai trial currency (and later EUR) settlement events
  5. A concrete first engagement: host-operator for railiance01 (coulomb infra)

This document is the technical contract. Implementation may land across kaizen-agentic, customer engagement repos, activity-core, State Hub, and access systems (ops-warden, OpenBao) without collapsing those ownership lines.


2. Goals and non-goals

Goals

  • Clients can request a Role for named Targets and receive a provisioned engagement
  • Each engagement has machine-readable lifecycle state and phase checklists
  • Operational knowledge accumulates in a client-owned vault, not in the supplier catalog by default
  • Ramp-up produces an operationally aware agent; ramp-down produces a handoff pack and access revocation
  • Duty sessions can emit billing events (Kai) without putting secrets in the ledger
  • First pilot maps cleanly onto existing sys-medic craft + host ops duties

Non-goals (v0.1)

  • Full multi-tenant SaaS control plane
  • Autonomous root changes without human gates
  • Replacing activity-core, State Hub, or ops-warden
  • Public Kai cryptocurrency or payment processor integration
  • Guaranteeing 24/7 unattended SRE autonomy

3. Conceptual model

┌─────────────────────────────────────────────────────────────────┐
│ Client (company / coulomb project)                              │
│  request, fund (Kai), accept handoff, own engagement vault      │
└───────────────┬─────────────────────────────────────────────────┘
                │ EngagementRequest
                ▼
┌─────────────────────────────────────────────────────────────────┐
│ Agency control plane (kaizen-agentic + optional hub records)    │
│  Role catalog · Engagement lifecycle · Quote/ledger hooks       │
└───────┬─────────────────────────┬───────────────────────────────┘
        │ bind Role + policies    │ schedule / prepare
        ▼                         ▼
┌───────────────────┐   ┌─────────────────────────────────────────┐
│ Agent definition  │   │ Execution plane                         │
│ (engagement-bound │   │ coding agent / Claude · activity-core   │
│  prompt + policy) │   │ sessions on allowed Targets             │
└─────────┬─────────┘   └──────────────────┬──────────────────────┘
          │ writes                         │ observes / acts (gated)
          ▼                                ▼
┌───────────────────┐   ┌─────────────────────────────────────────┐
│ Engagement vault  │   │ Targets (repos, hosts e.g. railiance01) │
│ CLIENT OWNED      │   │ access via warden certs / approved paths│
└───────────────────┘   └─────────────────────────────────────────┘

Core entities

Entity Definition Owner
Role Versioned supplier product (host-operator@1.2.0) — craft + protocols Supplier
Client Organization or coulomb project account Client
Target Thing the agent attends (host, cluster, repo, product area) Client
Engagement Binding of Role × Client × Targets × policy × lifecycle Shared record; vault is client
Agent definition Concrete prompt + frontmatter materialised for an engagement Generated; stored per policy
Engagement vault Memory, profiles, session logs, handoff Client
Duty session One execution of the agent under the engagement Ephemeral
Ledger entry Kai charge/credit for seat/duty/access Settlement system

4. Relationship to existing architecture

Existing piece How FDA (forward-deployed agency) uses it
agents/agent-*.md Role templates (generic craft)
.kaizen/agents/<name>/memory.md Becomes vault path under engagement scope (not only project-root generic)
Coach / memory brief Ramp-up orientation producer
Protocols (agents/protocols/...) Role procedures; engagement may pin versions
.kaizen/schedule.yml Duty cadence on target repos / engagement schedule file
ADR-006 customer engagement repo Still holds loop contracts; FDA adds staffed Role engagements
activity-core Optional cron fire for scheduled duties
ops-warden / OpenBao Access class enforcement — never invent secrets in agent prompts
Metrics (ADR-004) Quality + billing correlates; no secret payloads

Important distinction:

  • coulomb-loop style engagement = improvement loop across a repo roster
  • FDA engagement = Role staffing against Targets (hosts, systems, teams)

A client may run both. They share CLI primitives and confidentiality principles but different primary artifacts (loops/ vs engagements/).


5. Data model (schemas)

5.1 Role manifest (supplier)

Path proposal: roles/<role-id>/ROLE.yaml (or package data equivalent).

# roles/host-operator/ROLE.yaml
apiVersion: kaizen.agentic/v1
kind: Role
metadata:
  id: host-operator
  version: "0.1.0"
  title: Host Operator
  category: infrastructure
  capability_tier: 4          # PricingModel multiplier M
spec:
  description: >
    Competent technical administrator for named Linux hosts — OS currency,
    security posture, load and workload review, operational continuity.
  base_agent: sys-medic       # craft inheritance
  protocols:
    - agent: sys-medic
      slug: k3s-node-health-assessment
    - agent: host-operator
      slug: os-security-pass   # new protocol for this Role
    - agent: host-operator
      slug: load-workload-review
  memory_template: host-operator  # extends sys-medic sections
  default_cadence:
    health_review: daily
    os_security_pass: weekly
    load_review: daily
  access_classes_allowed:
    - read_only
    - host_observe
    - privileged_ops          # always gated
  ramp_up:
    checklist_ref: roles/host-operator/ramp-up.md
    exit_criteria:
      - host_baseline_documented
      - access_path_verified
      - first_health_review_complete
      - coach_orientation_filed
  ramp_down:
    checklist_ref: roles/host-operator/ramp-down.md
    exit_criteria:
      - handoff_pack_written
      - open_threads_triaged
      - access_revoked
      - vault_export_confirmed
  billing:
    seat_product: expert_seat_4x
    duty_weights:
      standard_review: 1.0
      deep_assessment: 1.0

5.2 Engagement request (client → agency)

# Example: engagements/requests/2026-07-16-railiance01-host-operator.yaml
apiVersion: kaizen.agentic/v1
kind: EngagementRequest
metadata:
  id: engreq-coulomb-railiance01-ho-001
  created: "2026-07-16"
spec:
  client:
    id: coulomb
    domain_slug: railiance   # or coulomb_social / custodian as applicable
    billing_account: coulomb-ops-kai
  role:
    id: host-operator
    version_constraint: ">=0.1.0 <1.0.0"
  targets:
    - kind: host
      id: railiance01
      inventory_ref: railiance-hosts   # where facts live
      notes: "k3s production; forgejo/apps workloads"
  duty:
    health_review: daily
    os_security_pass: weekly
    load_review: daily
    timezone: Europe/Berlin
  access:
    requested_classes: [host_observe, privileged_ops]
    cert_identity_hint: agt   # ops-warden; never store private keys here
  knowledge:
    vault_placement: client_repo   # see §6
    confidentiality: client_owned
  commercial:
    currency: kai
    trial: true
    quote_requested: true

5.3 Engagement record (lifecycle source of truth)

apiVersion: kaizen.agentic/v1
kind: Engagement
metadata:
  id: eng-coulomb-railiance01-ho-001
  request_id: engreq-coulomb-railiance01-ho-001
  created: "2026-07-16"
  updated: "2026-07-16"
status:
  phase: requested   # see §7
  # requested | quoting | funded | staffing | ramp_up | operating
  # renewing | ramp_down | closed | cancelled
spec:
  client: { id: coulomb, billing_account: coulomb-ops-kai }
  role: { id: host-operator, version: "0.1.0" }
  targets:
    - kind: host
      id: railiance01
  agent_definition:
    path: engagements/eng-coulomb-railiance01-ho-001/agent-host-operator.md
    derived_from: agents/agent-sys-medic.md
  vault:
    root: engagements/eng-coulomb-railiance01-ho-001/vault/
    memory: engagements/eng-coulomb-railiance01-ho-001/vault/memory.md
  schedule:
    path: engagements/eng-coulomb-railiance01-ho-001/schedule.yml
  policy:
    confidentiality: client_owned
    human_approval_for: [privileged_ops, package_upgrade, firewall_change, reboot]
    redaction_profile: default-host-ops
  commercial:
    currency: kai
    seat_product: expert_seat_4x
    ledger_ref: kai://coulomb-ops-kai/eng-coulomb-railiance01-ho-001

5.4 Agent definition (engagement-bound)

Materialised markdown with frontmatter — what the runtime actually loads.

---
name: host-operator
engagement_id: eng-coulomb-railiance01-ho-001
role_id: host-operator
role_version: "0.1.0"
description: Host operator for railiance01 (coulomb engagement)
category: infrastructure
memory: enabled
memory_path: engagements/eng-coulomb-railiance01-ho-001/vault/memory.md
targets:
  - kind: host
    id: railiance01
confidentiality: client_owned
phase: ramp_up
access_classes: [host_observe, privileged_ops]
human_approval_for: [privileged_ops, package_upgrade, firewall_change, reboot]
model: inherit
---

Body = Role prompt (from template) + engagement binding section (targets, escalation contacts, change windows, vault paths, forbidden actions) + phase protocol includes.

Supplier catalog keeps the unbound Role template. The bound definition may live in the client engagement repo (preferred for confidential bindings).


6. Knowledge planes and vault layout

6.1 Placement rules

Content Location Sync to supplier repo?
Role prompt template kaizen-agentic/roles/ or agents/ N/A (source)
Engagement record + bound definition Client engagement repo (or client-private store) No (except non-secret metrics summary if opted-in)
Memory, node profiles, incident notes Engagement vault (client) Never by default
Scrubbed protocol improvement Supplier PR after client license Yes, after review
Kai ledger lines Settlement store Billing metadata only

6.2 Vault directory (client-owned)

engagements/<engagement-id>/
├── ENGAGEMENT.yaml              # lifecycle record
├── agent-host-operator.md       # bound definition
├── schedule.yml                 # duty cadence
├── access-plan.md               # how certs/routes work (no secrets)
├── vault/
│   ├── memory.md                # extended host-operator memory
│   ├── baselines/
│   │   └── railiance01.md       # OS, packages, known load envelope
│   ├── session-log/             # optional per-session files
│   ├── findings/                # durable finding tickets
│   └── handoff/                 # filled during ramp-down
├── reports/                     # client-visible session reports
└── commercial/
    └── quote.yaml               # Kai quote snapshot

6.3 Memory template extension (host-operator)

Inherits sys-medic sections and adds:

## Engagement Charter
<!-- duty, change windows, escalation, out-of-scope -->

## Host Profiles
<!-- hostname | role | typical load | services | last review -->

## OS & Patch State
<!-- distro | kernel | last update pass | deferred packages | reboot needed? -->

## Security Posture
<!-- listening services | firewall notes | cert expiry | known risks -->

## Load & Workload Envelope
<!-- CPU/mem/disk baselines | heavy jobs | saturation incidents -->

## Recurring Findings
## Cleared Issues
## Open Threads
## Session Log

6.4 Redaction gate (supplier craft feedback)

Before any vault content may influence a public Role update:

  1. Run kaizen-agentic engagement scrub <engagement-id> --to-lesson
  2. Output must pass checklist: no hostnames (or allowlisted), no IPs, no user names, no secret material, no tenant identifiers
  3. Client marker contribute_lesson: approved in engagement record
  4. Human maintainer merges into Role/protocol

Without steps 34, scrubbed output is discarded.


7. Lifecycle state machine

requested → quoting → funded → staffing → ramp_up → operating ⇄ renewing
                                              │
                                              ↓
                                          ramp_down → closed

Any non-terminal state → cancelled (with partial ramp-down if vault exists)

7.1 Phase responsibilities

requested

  • Client submits EngagementRequest
  • Agency validates Role exists, Targets well-formed, access classes allowed

quoting

  • Produce Kai quote (seat + ramp + estimated duties)
  • Client accepts or revises scope

funded

  • Ledger holds seat + ramp-up package (or soft-reserve for internal grants)
  • Insufficient Kai → remain in quoting with funding_blocked

staffing

  • Pin Role version
  • Materialise bound agent definition
  • Create vault scaffold + schedule
  • Register optional activity-core definition / hub task links
  • Verify access path without storing secrets (warden route, dry-run SSH, etc.)

ramp_up

Mandatory checklist (host-operator):

ID Criterion Evidence
RU-01 Access path verified access-plan.md + successful observe session
RU-02 Host baseline documented vault/baselines/railiance01.md
RU-03 Coach / orientation brief filed vault or reports/orientation.md
RU-04 First health review complete reports/ + memory session log
RU-05 Security snapshot initial OS & Security Posture sections non-empty
RU-06 Load envelope initial Load & Workload Envelope non-empty
RU-07 Escalation contacts confirmed Engagement Charter
RU-08 Human approval path tested dry-run privileged proposal logged

Exit: all RU-* done → operating. Kai: charge ramp-up package on phase entry or exit (pick one; recommend charge on entry, refund if cancelled before RU-01).

operating

  • Scheduled duties fire (activity-core or manual)
  • Each session: load vault → work → update vault → metrics → ledger duty charge
  • Privileged actions require human approval record
  • Phase may enter renewing near seat period end

ramp_down

Mandatory checklist:

ID Criterion Evidence
RD-01 Open threads triaged memory Open Threads resolved or transferred
RD-02 Handoff pack written vault/handoff/README.md + baselines export
RD-03 Outstanding risks listed handoff/risks.md
RD-04 Access revoked / certs not renewed access-plan.md “revoked at …”
RD-05 Schedule disabled schedule.yml enabled false / activity-core disabled
RD-06 Client confirms vault custody engagement status note
RD-07 Supplier workspace scrubbed no client vault copies in kaizen-agentic tree

Exit: all RD-* → closed. Charge ramp-down package on entry.

closed

  • Terminal. Ledger finalised. Definition retained only in client custody (supplier may keep engagement id + non-secret metrics aggregate if opted-in).

8. Request → provision API / CLI surface

Proposed CLI (kaizen-agentic), additive and offline-friendly where possible:

# Client / operator
kaizen-agentic engagement request --file request.yaml
kaizen-agentic engagement quote   <engagement-id>
kaizen-agentic engagement fund    <engagement-id> --account coulomb-ops-kai

# Agency staffing
kaizen-agentic engagement staff   <engagement-id> --role host-operator
kaizen-agentic engagement render-agent <engagement-id>
kaizen-agentic engagement init-vault <engagement-id>

# Lifecycle
kaizen-agentic engagement phase   <engagement-id> --to ramp_up
kaizen-agentic engagement checklist <engagement-id>
kaizen-agentic engagement phase   <engagement-id> --to operating
kaizen-agentic engagement prepare <engagement-id>   # session bundle like schedule prepare
kaizen-agentic engagement close-session <engagement-id> --metrics ... --kai-duty standard_review

# Knowledge boundary
kaizen-agentic engagement scrub   <engagement-id> --to-lesson
kaizen-agentic engagement export-handoff <engagement-id>
kaizen-agentic engagement phase   <engagement-id> --to ramp_down
kaizen-agentic engagement phase   <engagement-id> --to closed

Session prepare bundle includes:

  1. Bound agent definition
  2. Vault memory + baselines
  3. Relevant protocols (pinned)
  4. Access plan (no secrets)
  5. Phase-specific instructions (ramp-up vs operate vs ramp-down)
  6. Commercial context (duty type for ledger)

This reuses the spirit of schedule prepare (ADR-005) with engagement scope.

State Hub (optional index)

Do not invent workplans by hand in the hub. Options:

  • Engagement markdown/YAML in client repo + fix-consistency if modeled as workplan, or
  • Lightweight hub records: progress events only (add_progress_event) keyed by engagement id in detail JSON

Prefer files as source of truth (ADR-001 spirit).


9. Access control integration

Agents never hold long-lived secrets in definitions or vaults.

Need Route
SSH to railiance01 warden sign / cert_command — ops-warden
API tokens, DB passwords OpenBao via warden route pointers
Tunnel to services ops-bridge
Authorization decisions flex-auth (if applicable)

Access classes in the engagement record constrain what the agent may propose and what the prepare bundle states as allowed. Runtime still depends on human and cert policy; the definition is policy-as-documentation plus checklist gates.

Privileged ops gate

Agent produces Action Proposal
  → Engagement policy requires human approval for class
  → Human records approval in vault/session-log or hub progress event
  → Agent may execute within stated blast radius
  → Session close updates Cleared Issues / Findings

10. Billing event schema (Kai)

Emitted at duty close and phase transitions (implementation may be JSONL local ledger first):

{
  "apiVersion": "kaizen.agentic/v1",
  "kind": "KaiLedgerEntry",
  "id": "kai-20260716T100000Z-001",
  "account": "coulomb-ops-kai",
  "engagement_id": "eng-coulomb-railiance01-ho-001",
  "type": "duty_charge",
  "product": "standard_review",
  "capability_tier": 4,
  "access_surcharge_product": "host_observe",
  "amount_kai": 1700,
  "currency": "KAI",
  "session_ref": "reports/2026-07-16-health-review.md",
  "created_at": "2026-07-16T10:00:00Z",
  "metadata": {
    "target": "railiance01",
    "phase": "operating",
    "tokens_vendor_cost_eur": null
  }
}

Forbidden in ledger: secrets, full command transcripts with sensitive output, raw vault dumps.

Seat charges: type: seat_charge on period boundaries. Ramp packages: type: ramp_up_charge / ramp_down_charge.


11. First example: coulomb host-operator on railiance01

11.1 Context

  • Client: coulomb (platform / railiance ops)
  • Target: host railiance01 — production k3s node (Forgejo, apps, cluster services per railiance-hosts inventory)
  • Role: host-operator (tier 4), craft base sys-medic
  • Duty: keep system operational and secure; OS up to date; regular review of system load and demanding workloads

11.2 Engagement charter (normative for pilot)

In scope

  • OS package currency and reboot planning (with approval)
  • Security hygiene: listeners, basic firewall posture, cert expiry awareness, user/session leftovers
  • Load average, CPU, memory, disk, inode pressure
  • Workload identification: heavy processes, container/pod resource hogs, backup/jobs overlap
  • k3s node health signals (reuse sys-medic protocol where applicable)
  • Documentation of baselines and findings in the engagement vault
  • Recommendations and gated remediation proposals

Out of scope (unless separately engaged)

  • Application feature development
  • Unilateral production data deletion
  • Multi-host fleet automation without engagement expansion
  • Secret value retrieval into chat or git
  • Changing DNS, billing, or legal identity systems

Cadence (pilot)

Duty When Protocol
Health + load review Daily load-workload-review + sys-medic assessment subset
OS & security pass Weekly os-security-pass
Deep assessment On demand / monthly full sys-medic k3s protocol
Incident assist On demand privileged gate as needed

11.3 Suggested inventory anchors (read-only facts)

  • railiance-hosts inventory and SCOPE (host list, ops-bridge notes)
  • railiance-apps / forgejo-on-railiance01 docs for workload expectations
  • Existing sys-medic node profile sections if any prior .kaizen memory exists on related repos

11.4 Ramp-up sequence (pilot runbook)

  1. Create EngagementRequest (YAML above)
  2. Quote + fund from coulomb-ops-kai (or internal grant)
  3. engagement staff → render agent + vault under client engagement store
  4. Verify observe access to railiance01 (cert path)
  5. Capture baseline: uname, uptime, disk, memory, load, top workloads, k3s node status
  6. Coach brief (if other agent memories exist on related projects)
  7. Complete RU checklist → mark operating
  8. Enable daily/weekly schedule entries

11.5 Ramp-down sequence

  1. Freeze new duties; finish in-flight session
  2. Triage open threads (transfer to humans or new engagement)
  3. Write handoff pack: baseline, known risks, deferred patches, “how we operate railiance01”
  4. Disable schedules; revoke agent access
  5. Confirm vault remains in client custody
  6. Scrub supplier working trees
  7. Close engagement; finalise Kai ledger

11.6 Mapping to sys-medic today

Until host-operator Role package exists:

Need Temporary approach
Agent prompt Install/use sys-medic + engagement binding appendix
Memory Init .kaizen/agents/sys-medic/ or engagement vault path via override
Protocols Existing k3s-node-health-assessment + draft os/load protocols in engagement repo
Schedule Manual or activity-core definition owned by client engagement repo

Promotion path: generalize binding appendix into roles/host-operator/ in kaizen-agentic once pilot checklist passes twice.


12. Security & compliance checklist

  • No secrets in git (engagement YAML, agent definition, vault baselines)
  • Vault encryption at rest if hosted on shared infra (client policy)
  • Session transcripts: client retention policy; default not uploaded to supplier
  • Privileged actions require recorded approval
  • Redaction gate before any Role craft PR
  • Access via short-lived certs; no password in prompts
  • Ledger free of sensitive operational detail
  • Ramp-down access revocation verified

13. Implementation phases

Phase 0 — Spec only (this document + business model)

  • Accept as working draft; no code required

Phase 1 — File-based pilot (minimal code)

  • Directory convention under a client engagement repo
  • Manual phase transitions in ENGAGEMENT.yaml
  • sys-medic + appendix as bound definition
  • JSONL Kai ledger in commercial/ledger.jsonl
  • Checklists as markdown

Phase 2 — CLI scaffolding

  • engagement command group (request, staff, render-agent, checklist, prepare, scrub)
  • Validate schemas
  • Integrate memory brief / metrics record with vault paths

Phase 3 — Orchestration

  • activity-core definitions per engagement schedule
  • Optional State Hub progress events
  • Ledger → quote reporting

Phase 4 — Commercial EUR path

  • FX export from Kai history
  • Hybrid billing metadata

14. Acceptance criteria (architecture)

  1. A client can express an EngagementRequest for host-operator @ railiance01
  2. Staffing produces a bound agent definition + client vault + schedule
  3. Ramp-up and ramp-down have testable exit criteria
  4. Operating sessions update client vault, not supplier agents/ by default
  5. Each duty can append a Kai ledger entry without secrets
  6. Pilot runbook in §11 is executable by an operator in one working day for Phase 1

15. Open technical questions

Resolved for Phase 1 by DEC-FDA-001:

Question Working default
Engagement store Pilot under engagements/pilots/ in kaizen-agentic; production vaults client-owned later
Multi-target One engagement, N targets, N baseline files
Metrics to supplier optimizer Opt-in, default off
Kai ledger File JSONL under engagement commercial/

Still open (non-blocking):

  1. Should bound definitions be signed (supply-chain) for production host ops?
  2. When to copy pilot vault to a client-owned repo for pure custody?