plenty-gobble/research/2026-09-12-IncidentMetricsPrinciples-Agentic-Observability-Research.md

# Research — IncidentMetricsPrinciples for Agentic Observability

**Date:** 2026-09-12  
**Repository:** `plenty-gobble`  
**Status:** Research note / design input  
**Source:** Notes from a DevOps webinar featuring Bronto employees and an observability platform, with a focus on agentic incident evaluation and an OpenTelemetry live demo.

---

## 1. Research Context

The webinar explored observability in a setting where AI agents, rather than only humans, analyze incidents.

The central premise is that traditional observability systems have often been optimized around:

- expected questions;
- known dashboards;
- bounded retention;
- sampling;
- pre-aggregated metrics;
- rollups;
- cardinality reduction.

That optimization model becomes problematic when the analyst is an agent expected to investigate situations that were **not anticipated when the telemetry system was designed**.

The important shift is:

> **Traditional observability optimizes storage around known questions. Agentic observability must preserve enough evidence to survive unknown questions.**

This creates a different set of design priorities.

---

# 2. Central Risk: Silent Incompleteness

The strongest principle from the talk is:

> **The real risk is silent incompleteness: not merely a wrong answer, but a confident answer built on evidence that is not there.**

A human operator may notice that context feels incomplete because of:

- experience;
- intuition;
- domain knowledge;
- recollection of similar incidents;
- awareness of missing systems;
- skepticism toward dashboards.

A human may then explicitly ask:

> "What am I not seeing?"

An agent may not do this naturally if the presented context appears authoritative.

This creates an asymmetry:

```text
Incomplete evidence
       |
       +---------------------+
       |                     |
       v                     v
     human                 agent
       |                     |
experience may          may interpret
trigger doubt           context as complete
       |                     |
       v                     v
asks for more          confident diagnosis
```

Therefore the observability system must not only expose data.

It must expose **what data is absent, transformed, sampled, summarized, expired, or filtered**.

---

# 3. Completeness Is Part of the Data Model

Retention, sampling, filtering, rollups, and cardinality limits should be treated as explicit decisions rather than invisible infrastructure defaults.

The system should be able to answer questions such as:

- Was this stream sampled?
- At what rate?
- Was the sampling uniform?
- Were errors retained preferentially?
- Were successful requests dropped?
- Was the pre-sampling event count retained?
- Was this metric rolled up?
- Which dimensions survived?
- Which dimensions were removed?
- Was the source subject to an attribute allowlist?
- Was cardinality capped?
- What was the original retention window?
- Did the relevant time range partially expire?
- Is the current result based on full-fidelity evidence?
- Is a representative raw sample available?

This suggests a first-class **Completeness Contract**.

Example conceptual model:

```text
EvidenceSet
  source
  time_range
  fidelity
  transformation_history
  sampling_policy
  retention_policy
  rollup_policy
  filter_policy
  coverage
  known_gaps
```

The distinction between:

```text
"no evidence of X"
```

and:

```text
"evidence capable of showing X was not retained"
```

must be machine-visible.

---

# 4. The Case File

The webinar introduced a useful abstraction called the **Case File**.

A Case File is the complete context handed to an analyst or agent for incident analysis.

It contains:

- Observation
- Title
- Comments
- Traces
- Metrics
- Logs
- Prior Incidents

The analysis should produce:

- Root Cause
- Evidence
- Precedent
- Blast Radius

Conceptually:

```text
                         CASE FILE

                      Observation
                          Title
                        Comments
                           |
        +------------------+------------------+
        |                  |                  |
      Traces             Metrics             Logs
        |                  |                  |
        +------------------+------------------+
                           |
                    Prior Incidents
                           |
                           v
                       ANALYSIS
                           |
        +------------------+------------------+
        |                  |                  |
    Root Cause          Evidence          Precedent
                           |
                           v
                     Blast Radius
```

This is a powerful model because it separates:

1. the raw observability estate;
2. the bounded evidence package used for a particular investigation;
3. the conclusions drawn from it.

---

# 5. Case Files Need an Evidence Envelope

The webinar notes imply an important extension.

A Case File should not merely contain evidence.

It should contain a description of **how complete that evidence is**.

A better Case File model is therefore:

```text
CaseFile
  incident identity
  observation
  title
  comments

  traces
  metrics
  logs
  prior incidents

  evidence envelope
    source coverage
    time coverage
    fidelity
    sampling
    rollups
    filters
    retention gaps
    missing systems
    transformations
    known unknowns

  analysis
    root cause
    evidence
    precedent
    blast radius
    confidence
```

This makes confidence dependent not only on reasoning quality but also on evidence quality.

An agent should not be allowed to silently collapse:

```text
high-confidence reasoning
```

and:

```text
high-confidence evidence
```

into one number.

---

# 6. Full Telemetry as a Reliability Strategy

The talk argues that because future questions cannot be predicted reliably, the safest default is to retain the full underlying telemetry.

The reasoning is:

```text
future investigation
        |
        v
question unknown today
        |
        v
required dimensions unknown today
        |
        v
pre-aggregation may destroy future evidence
        |
        v
retain raw / lossless telemetry when economically feasible
```

This directly favors architectures built around inexpensive long-term storage.

The requirement is not necessarily that all data remain on the fastest storage tier.

It is that the data remain **reachable**.

---

# 7. Hot and Cold Tiers

One Bronto-oriented design principle described in the webinar is:

> **Keep everything; put the old part somewhere slow and cheap that the agent can still reach. It will wait.**

This is an important reversal of the traditional assumption that old telemetry becomes disposable because interactive human users will not tolerate slow access.

An agent can often tolerate:

```text
recent incident data    -> milliseconds / seconds
older precedent data    -> seconds / tens of seconds
historical archive      -> slower still
```

if the alternative is missing evidence.

This suggests a latency hierarchy rather than a retention hierarchy:

```text
HOT
recent, frequent, fast
        |
        v
WARM
less frequent, slower
        |
        v
COLD
cheap, high latency
        |
        v
ARCHIVAL
very cheap, still discoverable
```

The crucial property is:

> **Latency may degrade with age. Evidence availability should not silently disappear.**

This aligns strongly with `plenty-gobble`'s object-store-native direction.

---

# 8. Lossless Observability Techniques

The webinar notes identify several techniques for lowering storage cost without throwing away semantic information.

## 8.1 Columnar databases

Column-oriented storage allows queries to read only the required fields.

This reduces:

- physical I/O;
- memory bandwidth;
- decompression work.

It also tends to compress repeated values effectively.

---

## 8.2 Zstandard compression

Zstandard provides high compression efficiency with practical decompression speed.

For telemetry storage this can materially reduce retention cost without altering the represented information.

---

## 8.3 Dictionary encoding

Repeated values such as:

```text
service_name
region
environment
status
method
```

can be stored once in a dictionary and referenced by compact identifiers.

Example:

```text
"production" -> 3

3
3
3
3
3
```

rather than repeatedly storing the full string.

---

## 8.4 Arrow on the wire

Apache Arrow-style columnar transport can reduce serialization and memory-copy overhead between components.

The significance is not just network efficiency.

Keeping data columnar across storage, transport, and execution reduces repeated row/column conversion.

---

## 8.5 Delta-encoded timestamps

Telemetry timestamps are usually ordered and close together.

Instead of storing:

```text
1726000000000
1726000000120
1726000000244
```

the representation can store:

```text
1726000000000
+120
+124
```

The deltas are much smaller and compress more effectively.

---

## 8.6 Repeated attributes stored once

Telemetry frequently repeats the same resource-level attributes across large numbers of events.

Examples:

```text
service.name
service.version
deployment.environment
cluster
host
region
container.image
```

Rather than embedding all attributes in every event, storage may normalize them physically:

```text
AttributeSet #42
  service.name = billing
  environment = production
  region = eu-central-1

events:
  event A -> #42
  event B -> #42
  event C -> #42
```

This can remain logically lossless while reducing physical duplication substantially.

---

# 9. Polymorphic Observability Storage

The webinar describes Bronto's approach as **polymorphic observability storage**.

The underlying principle is:

> **The store should adapt to the data, rather than requiring the data to adapt to one storage shape.**

Telemetry contains radically different field classes:

```text
timestamps
small categorical dimensions
high-cardinality identifiers
numeric measurements
free text
nested attributes
repeated resource metadata
traces
metric series
logs
```

A uniform physical representation leaves efficiency on the table.

A polymorphic store may represent different structures differently while preserving one logical query model.

For `plenty-gobble`, this reinforces the need for:

- field profiling;
- adaptive column promotion;
- dictionary encoding;
- Bloom filters;
- numerical summaries;
- sparse residual storage;
- repeated-attribute factoring;
- signal-specific encoding.

---

# 10. Modern Columnar Storage as an Anti-Rollup Strategy

Traditional observability economics often push operators toward rollups because retaining raw data is expensive.

The webinar's argument is that modern columnar storage and compression can make much more raw data economically feasible.

This changes the decision boundary:

```text
OLD MODEL

raw telemetry
     |
expensive
     |
roll up early
     |
future questions constrained


NEW MODEL

raw telemetry
     |
columnar + compressed
     |
cheap enough to retain
     |
future questions remain possible
```

The principle is:

> **Do not pre-answer future questions merely to make storage affordable if modern lossless storage techniques can avoid that compromise.**

---

# 11. Lossy Observability Techniques

The webinar identifies several common lossy mechanisms.

## Sampling

Only some events are retained.

Information loss can affect:

- rare failures;
- unusual combinations;
- temporal sequences;
- correlations;
- distribution tails.

---

## Retention

Data is completely removed after a time period.

This destroys the ability to revisit:

- long-term trends;
- historical precedent;
- rare recurring incidents;
- behavior before system changes.

---

## Rollups and downsampling

Raw values are replaced by aggregates.

Examples:

```text
1-second measurements
        ->
1-minute average
```

or:

```text
per-customer metric
        ->
per-region metric
```

Future dimensions that were not preserved become unrecoverable.

---

## Severity filters

Example:

```text
keep ERROR
drop INFO
```

This can remove the causal lead-up to a failure.

An INFO event may explain the ERROR event that follows.

---

## Cardinality caps and attribute allowlists

These deliberately remove dimensions that are expensive to index or store.

Unfortunately the removed attribute may later be exactly the discriminating variable needed for diagnosis.

---

# 12. If Loss Is Unavoidable, Preserve the Loss

A major derived principle is:

> **If data must be discarded, retain metadata describing what was discarded.**

This should become a concrete `plenty-gobble` requirement.

Loss transforms should produce a **Loss Manifest**.

Example:

```text
LossManifest

transformation:
  type: sampling

input_count:
  12,441,209

output_count:
  1,244,118

policy:
  normal_sample_rate: 0.10
  error_sample_rate: 1.00

time_range:
  ...

reference_sample:
  ...

reason:
  storage-budget-policy-v3
```

An agent receiving sampled data should also receive this manifest.

---

# 13. Better Sampling Compromise

The notes suggest that if sampling is unavoidable:

> **Keep a floor of normal traffic for reference and preserve the count from before the sampler.**

This is important.

A sampled dataset becomes much more interpretable when it contains:

```text
representative normal examples
+
pre-sampling population count
+
sampling policy
```

For example:

```text
original requests: 10,000,000
retained normal:      100,000
retained errors:       23,411
normal sampling:           1%
error sampling:          100%
```

An agent can then reason about prevalence without confusing retained count with actual count.

---

# 14. Better Rollup Compromise

If rollups are required:

> **Keep the dimensions the business is likely to reason about. Roll up the rest.**

Examples mentioned:

- Customer
- Region
- Build

The deeper principle is to distinguish between:

```text
business identity dimensions
```

and:

```text
operationally disposable dimensions
```

However, `plenty-gobble` should avoid pretending this choice is future-proof.

The dimensions retained and removed must remain explicit in metadata.

---

# 15. Better Edge Filtering

The webinar suggests intelligent edge filtering:

> **Deduplicate, but preserve the count and the first and last occurrence. Drop the successful probe, not the failed one.**

Example transformation:

```text
5,281 identical messages
```

can become:

```text
message: ...
count: 5,281
first_seen: 12:01:03
last_seen: 12:09:44
```

This loses exact inter-arrival timing unless additionally preserved, but retains substantially more semantic value than simply dropping duplicates.

The general principle is:

> **Prefer semantic compaction over blind deletion.**

---

# 16. Better Retention Compromise

If raw telemetry retention must be short:

> **Keep postmortems accessible to the agent.**

This preserves accumulated organizational knowledge even after raw evidence expires.

A useful hierarchy becomes:

```text
raw telemetry
     |
expires eventually
     |
     v
Case File
     |
     v
postmortem
     |
     v
incident precedent knowledge
```

The postmortem should remain linked to:

- the original Case File;
- evidence identifiers;
- telemetry coverage;
- known missing data;
- root cause;
- remediation;
- blast radius.

---

# 17. Incident Memory as a First-Class Data Product

Prior incidents are not merely documents.

For an agent, they form an empirical precedent library.

A historical incident model could expose:

```text
Incident
  symptoms
  affected systems
  root cause
  causal chain
  blast radius
  evidence
  remediation
  recurrence indicators
  telemetry completeness
```

An incident-analysis agent could then search for:

```text
"Have we seen this symptom combination before?"
```

This is more useful than lexical postmortem search alone.

---

# 18. Implications for Agent Confidence

Agent responses should distinguish at least:

```text
Reasoning Confidence
Evidence Completeness
Evidence Quality
```

Example:

```text
Root-cause confidence:      0.84
Evidence completeness:      0.43
Evidence reliability:       0.91
```

A high reasoning score with low evidence completeness should not be presented as a definitive conclusion.

Potential derived status:

```text
Conclusion status:
  PROVISIONAL — HIGH REASONING / LOW COVERAGE
```

The observability platform should supply the evidence metadata required to make this possible.

---

# 19. Proposed Evidence Fidelity Levels

A future `plenty-gobble` model might use an explicit fidelity classification.

Example draft:

```text
F0 UNKNOWN
   No reliable information about transformations.

F1 AGGREGATED
   Only rollups / derived statistics remain.

F2 HEAVILY FILTERED
   Significant source events or dimensions removed.

F3 SAMPLED
   Raw event shape retained, incomplete population.

F4 REPRESENTATIVE
   Sample plus population counts and policy metadata.

F5 NEAR-LOSSLESS
   Minor semantic compaction such as deduplication with counts.

F6 LOSSLESS
   Complete raw logical evidence retained.
```

This is only a candidate model, but some equivalent concept would make evidence quality operationally visible.

---

# 20. Proposed `plenty-gobble` Additions

This webinar research suggests several capabilities that should be considered for the architecture.

## 20.1 Loss Policy Registry

Track every transformation capable of destroying information:

```text
sampling
filtering
rollup
downsampling
cardinality cap
attribute removal
retention
deduplication
```

---

## 20.2 Transformation Lineage

Every derived dataset should record:

```text
source
transform
policy
timestamp
input population
output population
known information loss
```

---

## 20.3 Evidence Completeness API

Given:

```text
dataset + time range + query context
```

return:

```text
coverage
fidelity
known gaps
loss policies
available raw samples
retention boundaries
```

---

## 20.4 Case File Builder

Construct a bounded incident package from:

- observations;
- comments;
- logs;
- traces;
- metrics;
- prior incidents;
- telemetry completeness metadata.

---

## 20.5 Incident Archive

Persist:

- Case Files;
- postmortems;
- root causes;
- blast radius;
- remediation;
- precedent links.

This archive should outlive the shortest telemetry retention windows.

---

## 20.6 Full-Fidelity Reference Samples

Where complete retention is impossible, retain selected time windows or event cohorts at full fidelity.

These serve as calibration material for agents evaluating sampled/aggregated data.

---

# 21. Systems-Theory Interpretation

The observability platform is itself part of the control loop of the production system.

```text
production system
      |
      v
telemetry
      |
      v
observer
      |
      v
diagnosis
      |
      v
decision
      |
      v
intervention
      |
      +------------------> production system
```

Lossy telemetry alters the observer's model.

If that loss is invisible, the controller behaves as though its model were complete.

This is dangerous because the control loop contains an **unobserved distortion**.

The requirement is therefore not necessarily zero loss.

It is:

> **Loss must itself be observable.**

This may be the deepest principle in the webinar notes.

---

# 22. Agentic Observability Principle

A concise formulation is:

> **An observability system for agents must make both evidence and evidence absence observable.**

A stronger version:

> **Never let an agent mistake unavailable evidence for negative evidence.**

This should become a design invariant for incident-analysis systems built on `plenty-gobble`.

---

# 23. Relationship to the Existing `plenty-gobble` Architecture

The previous `plenty-gobble` research emphasized:

```text
prune first
project second
scan third
distribute only when necessary
```

This webinar adds an important constraint:

```text
pruning must reduce reads
without silently reducing retained meaning
```

The two goals can coexist.

### Physical pruning

Safe:

```text
skip data proven irrelevant to this query
```

### Semantic dropping

Potentially unsafe:

```text
discard data because it is believed irrelevant to future queries
```

This distinction is foundational.

`plenty-gobble` should optimize aggressively at **query time** while being conservative about irreversible information destruction at **ingest time**.

---

# 24. Refined Storage Principle

The resulting architectural principle is:

> **Store broadly, query selectively.**

Not:

> **Store selectively, query cheaply.**

That difference is particularly important for agentic investigation.

---

# 25. Candidate Architectural Invariants

The webinar suggests adding the following invariants to `plenty-gobble`.

### O1 — Evidence loss is explicit

No lossy transformation may be invisible.

### O2 — Absence is qualified

A query returning no evidence must distinguish between:

```text
not observed
```

and:

```text
not retained / not collected / filtered
```

### O3 — Retention is policy

Retention duration is explicit, versioned, and queryable.

### O4 — Sampling carries population metadata

Where possible, retain pre-sampling counts and the sampling strategy.

### O5 — Rollups declare removed dimensions

Agents must be able to know what cannot be reconstructed.

### O6 — Historical incident knowledge outlives raw telemetry

Case Files and postmortems remain queryable beyond raw retention.

### O7 — Cold data remains addressable

Moving evidence to a slower tier must not make it semantically disappear.

### O8 — Confidence incorporates evidence completeness

Reasoning confidence alone is insufficient.

---

# 26. Research Hypotheses

The ideas should be tested rather than accepted solely as doctrine.

## H1

For agentic incident analysis, preserving full-fidelity telemetry materially improves root-cause accuracy compared with equivalent analyses over sampled or rolled-up data.

## H2

Explicit completeness metadata reduces false confidence when telemetry is lossy.

## H3

Columnar compression and object storage make long-retention raw telemetry affordable enough that aggressive rollups are often unnecessary.

## H4

Agents tolerate cold-tier latency better than humans, making deeper retention economically useful.

## H5

A small full-fidelity reference sample improves agent interpretation of sampled telemetry.

## H6

Historical Case Files and postmortems materially improve diagnosis after the underlying raw telemetry has expired.

These are good candidates for future benchmark and evaluation programs.

---

# 27. Suggested Evaluation Design

To evaluate the webinar's central claim, create synthetic incident datasets with controlled evidence degradation.

For one incident, produce:

```text
A — full telemetry
B — 10% random sampling
C — tail/error-biased sampling
D — rolled-up metrics
E — severity-filtered logs
F — cardinality-limited attributes
G — degraded data + explicit Loss Manifest
```

Give each to the same incident-analysis agent.

Measure:

```text
root-cause accuracy
blast-radius accuracy
false certainty
number of unsupported claims
ability to identify missing evidence
time to diagnosis
token / compute cost
```

This would turn the principle into measurable engineering evidence.

---

# 28. Practical Design Rule

When considering any storage optimization, ask:

```text
Does this merely change representation?

or

Does this destroy a future question?
```

Examples:

```text
Zstd compression             -> representation
dictionary encoding          -> representation
delta timestamps             -> representation
columnar layout              -> representation
attribute factoring          -> representation

sampling                     -> destroys evidence
retention                    -> destroys evidence
rollup                       -> destroys dimensions
severity filtering           -> destroys events
cardinality caps             -> destroys attributes
```

Prefer the first category aggressively.

Treat the second category as governance decisions.

---

# 29. Resulting `plenty-gobble` Thesis

Combining the previous architecture work with this webinar suggests a stronger purpose for `plenty-gobble`.

It is not only an adaptive, PostgreSQL-centered columnar store.

It can become an infrastructure layer for **evidence-preserving agentic observability**.

The architectural objective becomes:

> **Retain as much semantic evidence as economically possible, compress it losslessly, move old evidence to cheap tiers rather than deleting it, expose every unavoidable loss explicitly, and let query-time pruning—not ingest-time assumptions—determine what an investigation reads.**

---

# 30. Compact Principles

The research can be reduced to ten operating principles:

1. **Store broadly, query selectively.**
2. **Unknown future questions require preserved raw evidence.**
3. **Silent incompleteness is more dangerous than explicit incompleteness.**
4. **Loss must itself be observable.**
5. **Retention, sampling, and rollups are policies, not defaults.**
6. **Prefer lossless physical optimization before semantic reduction.**
7. **Cold and slow is better than deleted.**
8. **Case Files must describe evidence completeness, not only evidence.**
9. **Incident precedent should survive raw-data expiry.**
10. **Agent confidence must be conditioned on evidence coverage.**

---

## Final Research Statement

> **For agentic incident analysis, the observability system is part of the epistemic boundary of the agent. Any hidden data loss becomes hidden model error. `plenty-gobble` should therefore optimize for evidence preservation, explicit loss lineage, and query-time selectivity rather than prematurely deciding which future evidence will matter.**