2026-06-16 01:58:07 +02:00
|
|
|
# Telemetry and Agent Effectiveness Tracking
|
|
|
|
|
|
|
|
|
|
WP-0001 T04 design — aligned with ADR-004 and WP-0004 ecosystem integration.
|
|
|
|
|
|
|
|
|
|
## Two layers (do not merge)
|
|
|
|
|
|
|
|
|
|
| Layer | Question | Mechanism |
|
|
|
|
|
|-------|----------|-----------|
|
|
|
|
|
| **Project** | How is agent *X* performing in *this repo*? | `kaizen-agentic metrics record` → `.kaizen/metrics/` |
|
|
|
|
|
| **Fleet** | How are coding sessions performing *across repos*? | agentic-resources Helix Forge |
|
|
|
|
|
|
|
|
|
|
kaizen-agentic **does not** ship a parallel session transcript ingestion pipeline.
|
|
|
|
|
|
|
|
|
|
## Project telemetry (implemented)
|
|
|
|
|
|
|
|
|
|
Memory-enabled agents record per-session outcomes at close:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kaizen-agentic metrics record <agent> --success --time <s> --quality <0-1>
|
2026-06-18 08:53:36 +02:00
|
|
|
kaizen-agentic metrics record <agent> --success --time <s> --quality <0-1> --emit-event
|
2026-06-16 01:58:07 +02:00
|
|
|
kaizen-agentic metrics optimize [agent]
|
|
|
|
|
kaizen-agentic memory brief <agent> # includes Performance Summary
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Optional fleet correlation via `HELIX_SESSION_UID` (see
|
|
|
|
|
[integrations/helix-forge-correlation.md](integrations/helix-forge-correlation.md)).
|
|
|
|
|
|
|
|
|
|
## Fleet telemetry (agentic-resources)
|
|
|
|
|
|
|
|
|
|
Helix Forge owns session capture, digest storage, baselines, and weekly retro.
|
|
|
|
|
kaizen-agentic consumes correlation fields only.
|
|
|
|
|
|
|
|
|
|
## CLI install / usage analytics (future)
|
|
|
|
|
|
|
|
|
|
Potential v1.1 additions (not yet implemented):
|
|
|
|
|
|
|
|
|
|
- Opt-in anonymous counters on `install` / `memory init` (no PII, no project paths)
|
|
|
|
|
- Aggregate effectiveness reports via `metrics list` across a monorepo checkout
|
|
|
|
|
|
|
|
|
|
## tele-mcp evaluation (deferred)
|
|
|
|
|
|
2026-08-20 09:43:49 +02:00
|
|
|
[tele-mcp](https://forgejo.coulomb.social/coulomb/tele-mcp) is a candidate MCP adapter
|
2026-06-16 01:58:07 +02:00
|
|
|
for IDE-level telemetry (WP-0001 note). Assess before depending on it. Project and
|
|
|
|
|
fleet layers above satisfy INTENT's "measurable agents" requirement without tele-mcp.
|
|
|
|
|
|
|
|
|
|
## Feedback loop
|
|
|
|
|
|
2026-08-20 09:43:49 +02:00
|
|
|
User experience feedback uses [FEEDBACK.md](FEEDBACK.md) and Forgejo issue templates —
|
2026-06-18 08:53:36 +02:00
|
|
|
separate from execution metrics.
|
