tegwick
3701d1ae1c
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1370019@bnt-lap001 Assistant-Session: d462dd63-16de-4189-ae70-b9cd6566bf87
25 KiB
Executable file
Research — IncidentMetricsPrinciples for Agentic Observability
Date: 2026-09-12
Repository: plenty-gobble
Status: Research note / design input
Source: Notes from a DevOps webinar featuring Bronto employees and an observability platform, with a focus on agentic incident evaluation and an OpenTelemetry live demo.
1. Research Context
The webinar explored observability in a setting where AI agents, rather than only humans, analyze incidents.
The central premise is that traditional observability systems have often been optimized around:
- expected questions;
- known dashboards;
- bounded retention;
- sampling;
- pre-aggregated metrics;
- rollups;
- cardinality reduction.
That optimization model becomes problematic when the analyst is an agent expected to investigate situations that were not anticipated when the telemetry system was designed.
The important shift is:
Traditional observability optimizes storage around known questions. Agentic observability must preserve enough evidence to survive unknown questions.
This creates a different set of design priorities.
2. Central Risk: Silent Incompleteness
The strongest principle from the talk is:
The real risk is silent incompleteness: not merely a wrong answer, but a confident answer built on evidence that is not there.
A human operator may notice that context feels incomplete because of:
- experience;
- intuition;
- domain knowledge;
- recollection of similar incidents;
- awareness of missing systems;
- skepticism toward dashboards.
A human may then explicitly ask:
"What am I not seeing?"
An agent may not do this naturally if the presented context appears authoritative.
This creates an asymmetry:
Incomplete evidence
|
+---------------------+
| |
v v
human agent
| |
experience may may interpret
trigger doubt context as complete
| |
v v
asks for more confident diagnosis
Therefore the observability system must not only expose data.
It must expose what data is absent, transformed, sampled, summarized, expired, or filtered.
3. Completeness Is Part of the Data Model
Retention, sampling, filtering, rollups, and cardinality limits should be treated as explicit decisions rather than invisible infrastructure defaults.
The system should be able to answer questions such as:
- Was this stream sampled?
- At what rate?
- Was the sampling uniform?
- Were errors retained preferentially?
- Were successful requests dropped?
- Was the pre-sampling event count retained?
- Was this metric rolled up?
- Which dimensions survived?
- Which dimensions were removed?
- Was the source subject to an attribute allowlist?
- Was cardinality capped?
- What was the original retention window?
- Did the relevant time range partially expire?
- Is the current result based on full-fidelity evidence?
- Is a representative raw sample available?
This suggests a first-class Completeness Contract.
Example conceptual model:
EvidenceSet
source
time_range
fidelity
transformation_history
sampling_policy
retention_policy
rollup_policy
filter_policy
coverage
known_gaps
The distinction between:
"no evidence of X"
and:
"evidence capable of showing X was not retained"
must be machine-visible.
4. The Case File
The webinar introduced a useful abstraction called the Case File.
A Case File is the complete context handed to an analyst or agent for incident analysis.
It contains:
- Observation
- Title
- Comments
- Traces
- Metrics
- Logs
- Prior Incidents
The analysis should produce:
- Root Cause
- Evidence
- Precedent
- Blast Radius
Conceptually:
CASE FILE
Observation
Title
Comments
|
+------------------+------------------+
| | |
Traces Metrics Logs
| | |
+------------------+------------------+
|
Prior Incidents
|
v
ANALYSIS
|
+------------------+------------------+
| | |
Root Cause Evidence Precedent
|
v
Blast Radius
This is a powerful model because it separates:
- the raw observability estate;
- the bounded evidence package used for a particular investigation;
- the conclusions drawn from it.
5. Case Files Need an Evidence Envelope
The webinar notes imply an important extension.
A Case File should not merely contain evidence.
It should contain a description of how complete that evidence is.
A better Case File model is therefore:
CaseFile
incident identity
observation
title
comments
traces
metrics
logs
prior incidents
evidence envelope
source coverage
time coverage
fidelity
sampling
rollups
filters
retention gaps
missing systems
transformations
known unknowns
analysis
root cause
evidence
precedent
blast radius
confidence
This makes confidence dependent not only on reasoning quality but also on evidence quality.
An agent should not be allowed to silently collapse:
high-confidence reasoning
and:
high-confidence evidence
into one number.
6. Full Telemetry as a Reliability Strategy
The talk argues that because future questions cannot be predicted reliably, the safest default is to retain the full underlying telemetry.
The reasoning is:
future investigation
|
v
question unknown today
|
v
required dimensions unknown today
|
v
pre-aggregation may destroy future evidence
|
v
retain raw / lossless telemetry when economically feasible
This directly favors architectures built around inexpensive long-term storage.
The requirement is not necessarily that all data remain on the fastest storage tier.
It is that the data remain reachable.
7. Hot and Cold Tiers
One Bronto-oriented design principle described in the webinar is:
Keep everything; put the old part somewhere slow and cheap that the agent can still reach. It will wait.
This is an important reversal of the traditional assumption that old telemetry becomes disposable because interactive human users will not tolerate slow access.
An agent can often tolerate:
recent incident data -> milliseconds / seconds
older precedent data -> seconds / tens of seconds
historical archive -> slower still
if the alternative is missing evidence.
This suggests a latency hierarchy rather than a retention hierarchy:
HOT
recent, frequent, fast
|
v
WARM
less frequent, slower
|
v
COLD
cheap, high latency
|
v
ARCHIVAL
very cheap, still discoverable
The crucial property is:
Latency may degrade with age. Evidence availability should not silently disappear.
This aligns strongly with plenty-gobble's object-store-native direction.
8. Lossless Observability Techniques
The webinar notes identify several techniques for lowering storage cost without throwing away semantic information.
8.1 Columnar databases
Column-oriented storage allows queries to read only the required fields.
This reduces:
- physical I/O;
- memory bandwidth;
- decompression work.
It also tends to compress repeated values effectively.
8.2 Zstandard compression
Zstandard provides high compression efficiency with practical decompression speed.
For telemetry storage this can materially reduce retention cost without altering the represented information.
8.3 Dictionary encoding
Repeated values such as:
service_name
region
environment
status
method
can be stored once in a dictionary and referenced by compact identifiers.
Example:
"production" -> 3
3
3
3
3
3
rather than repeatedly storing the full string.
8.4 Arrow on the wire
Apache Arrow-style columnar transport can reduce serialization and memory-copy overhead between components.
The significance is not just network efficiency.
Keeping data columnar across storage, transport, and execution reduces repeated row/column conversion.
8.5 Delta-encoded timestamps
Telemetry timestamps are usually ordered and close together.
Instead of storing:
1726000000000
1726000000120
1726000000244
the representation can store:
1726000000000
+120
+124
The deltas are much smaller and compress more effectively.
8.6 Repeated attributes stored once
Telemetry frequently repeats the same resource-level attributes across large numbers of events.
Examples:
service.name
service.version
deployment.environment
cluster
host
region
container.image
Rather than embedding all attributes in every event, storage may normalize them physically:
AttributeSet #42
service.name = billing
environment = production
region = eu-central-1
events:
event A -> #42
event B -> #42
event C -> #42
This can remain logically lossless while reducing physical duplication substantially.
9. Polymorphic Observability Storage
The webinar describes Bronto's approach as polymorphic observability storage.
The underlying principle is:
The store should adapt to the data, rather than requiring the data to adapt to one storage shape.
Telemetry contains radically different field classes:
timestamps
small categorical dimensions
high-cardinality identifiers
numeric measurements
free text
nested attributes
repeated resource metadata
traces
metric series
logs
A uniform physical representation leaves efficiency on the table.
A polymorphic store may represent different structures differently while preserving one logical query model.
For plenty-gobble, this reinforces the need for:
- field profiling;
- adaptive column promotion;
- dictionary encoding;
- Bloom filters;
- numerical summaries;
- sparse residual storage;
- repeated-attribute factoring;
- signal-specific encoding.
10. Modern Columnar Storage as an Anti-Rollup Strategy
Traditional observability economics often push operators toward rollups because retaining raw data is expensive.
The webinar's argument is that modern columnar storage and compression can make much more raw data economically feasible.
This changes the decision boundary:
OLD MODEL
raw telemetry
|
expensive
|
roll up early
|
future questions constrained
NEW MODEL
raw telemetry
|
columnar + compressed
|
cheap enough to retain
|
future questions remain possible
The principle is:
Do not pre-answer future questions merely to make storage affordable if modern lossless storage techniques can avoid that compromise.
11. Lossy Observability Techniques
The webinar identifies several common lossy mechanisms.
Sampling
Only some events are retained.
Information loss can affect:
- rare failures;
- unusual combinations;
- temporal sequences;
- correlations;
- distribution tails.
Retention
Data is completely removed after a time period.
This destroys the ability to revisit:
- long-term trends;
- historical precedent;
- rare recurring incidents;
- behavior before system changes.
Rollups and downsampling
Raw values are replaced by aggregates.
Examples:
1-second measurements
->
1-minute average
or:
per-customer metric
->
per-region metric
Future dimensions that were not preserved become unrecoverable.
Severity filters
Example:
keep ERROR
drop INFO
This can remove the causal lead-up to a failure.
An INFO event may explain the ERROR event that follows.
Cardinality caps and attribute allowlists
These deliberately remove dimensions that are expensive to index or store.
Unfortunately the removed attribute may later be exactly the discriminating variable needed for diagnosis.
12. If Loss Is Unavoidable, Preserve the Loss
A major derived principle is:
If data must be discarded, retain metadata describing what was discarded.
This should become a concrete plenty-gobble requirement.
Loss transforms should produce a Loss Manifest.
Example:
LossManifest
transformation:
type: sampling
input_count:
12,441,209
output_count:
1,244,118
policy:
normal_sample_rate: 0.10
error_sample_rate: 1.00
time_range:
...
reference_sample:
...
reason:
storage-budget-policy-v3
An agent receiving sampled data should also receive this manifest.
13. Better Sampling Compromise
The notes suggest that if sampling is unavoidable:
Keep a floor of normal traffic for reference and preserve the count from before the sampler.
This is important.
A sampled dataset becomes much more interpretable when it contains:
representative normal examples
+
pre-sampling population count
+
sampling policy
For example:
original requests: 10,000,000
retained normal: 100,000
retained errors: 23,411
normal sampling: 1%
error sampling: 100%
An agent can then reason about prevalence without confusing retained count with actual count.
14. Better Rollup Compromise
If rollups are required:
Keep the dimensions the business is likely to reason about. Roll up the rest.
Examples mentioned:
- Customer
- Region
- Build
The deeper principle is to distinguish between:
business identity dimensions
and:
operationally disposable dimensions
However, plenty-gobble should avoid pretending this choice is future-proof.
The dimensions retained and removed must remain explicit in metadata.
15. Better Edge Filtering
The webinar suggests intelligent edge filtering:
Deduplicate, but preserve the count and the first and last occurrence. Drop the successful probe, not the failed one.
Example transformation:
5,281 identical messages
can become:
message: ...
count: 5,281
first_seen: 12:01:03
last_seen: 12:09:44
This loses exact inter-arrival timing unless additionally preserved, but retains substantially more semantic value than simply dropping duplicates.
The general principle is:
Prefer semantic compaction over blind deletion.
16. Better Retention Compromise
If raw telemetry retention must be short:
Keep postmortems accessible to the agent.
This preserves accumulated organizational knowledge even after raw evidence expires.
A useful hierarchy becomes:
raw telemetry
|
expires eventually
|
v
Case File
|
v
postmortem
|
v
incident precedent knowledge
The postmortem should remain linked to:
- the original Case File;
- evidence identifiers;
- telemetry coverage;
- known missing data;
- root cause;
- remediation;
- blast radius.
17. Incident Memory as a First-Class Data Product
Prior incidents are not merely documents.
For an agent, they form an empirical precedent library.
A historical incident model could expose:
Incident
symptoms
affected systems
root cause
causal chain
blast radius
evidence
remediation
recurrence indicators
telemetry completeness
An incident-analysis agent could then search for:
"Have we seen this symptom combination before?"
This is more useful than lexical postmortem search alone.
18. Implications for Agent Confidence
Agent responses should distinguish at least:
Reasoning Confidence
Evidence Completeness
Evidence Quality
Example:
Root-cause confidence: 0.84
Evidence completeness: 0.43
Evidence reliability: 0.91
A high reasoning score with low evidence completeness should not be presented as a definitive conclusion.
Potential derived status:
Conclusion status:
PROVISIONAL — HIGH REASONING / LOW COVERAGE
The observability platform should supply the evidence metadata required to make this possible.
19. Proposed Evidence Fidelity Levels
A future plenty-gobble model might use an explicit fidelity classification.
Example draft:
F0 UNKNOWN
No reliable information about transformations.
F1 AGGREGATED
Only rollups / derived statistics remain.
F2 HEAVILY FILTERED
Significant source events or dimensions removed.
F3 SAMPLED
Raw event shape retained, incomplete population.
F4 REPRESENTATIVE
Sample plus population counts and policy metadata.
F5 NEAR-LOSSLESS
Minor semantic compaction such as deduplication with counts.
F6 LOSSLESS
Complete raw logical evidence retained.
This is only a candidate model, but some equivalent concept would make evidence quality operationally visible.
20. Proposed plenty-gobble Additions
This webinar research suggests several capabilities that should be considered for the architecture.
20.1 Loss Policy Registry
Track every transformation capable of destroying information:
sampling
filtering
rollup
downsampling
cardinality cap
attribute removal
retention
deduplication
20.2 Transformation Lineage
Every derived dataset should record:
source
transform
policy
timestamp
input population
output population
known information loss
20.3 Evidence Completeness API
Given:
dataset + time range + query context
return:
coverage
fidelity
known gaps
loss policies
available raw samples
retention boundaries
20.4 Case File Builder
Construct a bounded incident package from:
- observations;
- comments;
- logs;
- traces;
- metrics;
- prior incidents;
- telemetry completeness metadata.
20.5 Incident Archive
Persist:
- Case Files;
- postmortems;
- root causes;
- blast radius;
- remediation;
- precedent links.
This archive should outlive the shortest telemetry retention windows.
20.6 Full-Fidelity Reference Samples
Where complete retention is impossible, retain selected time windows or event cohorts at full fidelity.
These serve as calibration material for agents evaluating sampled/aggregated data.
21. Systems-Theory Interpretation
The observability platform is itself part of the control loop of the production system.
production system
|
v
telemetry
|
v
observer
|
v
diagnosis
|
v
decision
|
v
intervention
|
+------------------> production system
Lossy telemetry alters the observer's model.
If that loss is invisible, the controller behaves as though its model were complete.
This is dangerous because the control loop contains an unobserved distortion.
The requirement is therefore not necessarily zero loss.
It is:
Loss must itself be observable.
This may be the deepest principle in the webinar notes.
22. Agentic Observability Principle
A concise formulation is:
An observability system for agents must make both evidence and evidence absence observable.
A stronger version:
Never let an agent mistake unavailable evidence for negative evidence.
This should become a design invariant for incident-analysis systems built on plenty-gobble.
23. Relationship to the Existing plenty-gobble Architecture
The previous plenty-gobble research emphasized:
prune first
project second
scan third
distribute only when necessary
This webinar adds an important constraint:
pruning must reduce reads
without silently reducing retained meaning
The two goals can coexist.
Physical pruning
Safe:
skip data proven irrelevant to this query
Semantic dropping
Potentially unsafe:
discard data because it is believed irrelevant to future queries
This distinction is foundational.
plenty-gobble should optimize aggressively at query time while being conservative about irreversible information destruction at ingest time.
24. Refined Storage Principle
The resulting architectural principle is:
Store broadly, query selectively.
Not:
Store selectively, query cheaply.
That difference is particularly important for agentic investigation.
25. Candidate Architectural Invariants
The webinar suggests adding the following invariants to plenty-gobble.
O1 — Evidence loss is explicit
No lossy transformation may be invisible.
O2 — Absence is qualified
A query returning no evidence must distinguish between:
not observed
and:
not retained / not collected / filtered
O3 — Retention is policy
Retention duration is explicit, versioned, and queryable.
O4 — Sampling carries population metadata
Where possible, retain pre-sampling counts and the sampling strategy.
O5 — Rollups declare removed dimensions
Agents must be able to know what cannot be reconstructed.
O6 — Historical incident knowledge outlives raw telemetry
Case Files and postmortems remain queryable beyond raw retention.
O7 — Cold data remains addressable
Moving evidence to a slower tier must not make it semantically disappear.
O8 — Confidence incorporates evidence completeness
Reasoning confidence alone is insufficient.
26. Research Hypotheses
The ideas should be tested rather than accepted solely as doctrine.
H1
For agentic incident analysis, preserving full-fidelity telemetry materially improves root-cause accuracy compared with equivalent analyses over sampled or rolled-up data.
H2
Explicit completeness metadata reduces false confidence when telemetry is lossy.
H3
Columnar compression and object storage make long-retention raw telemetry affordable enough that aggressive rollups are often unnecessary.
H4
Agents tolerate cold-tier latency better than humans, making deeper retention economically useful.
H5
A small full-fidelity reference sample improves agent interpretation of sampled telemetry.
H6
Historical Case Files and postmortems materially improve diagnosis after the underlying raw telemetry has expired.
These are good candidates for future benchmark and evaluation programs.
27. Suggested Evaluation Design
To evaluate the webinar's central claim, create synthetic incident datasets with controlled evidence degradation.
For one incident, produce:
A — full telemetry
B — 10% random sampling
C — tail/error-biased sampling
D — rolled-up metrics
E — severity-filtered logs
F — cardinality-limited attributes
G — degraded data + explicit Loss Manifest
Give each to the same incident-analysis agent.
Measure:
root-cause accuracy
blast-radius accuracy
false certainty
number of unsupported claims
ability to identify missing evidence
time to diagnosis
token / compute cost
This would turn the principle into measurable engineering evidence.
28. Practical Design Rule
When considering any storage optimization, ask:
Does this merely change representation?
or
Does this destroy a future question?
Examples:
Zstd compression -> representation
dictionary encoding -> representation
delta timestamps -> representation
columnar layout -> representation
attribute factoring -> representation
sampling -> destroys evidence
retention -> destroys evidence
rollup -> destroys dimensions
severity filtering -> destroys events
cardinality caps -> destroys attributes
Prefer the first category aggressively.
Treat the second category as governance decisions.
29. Resulting plenty-gobble Thesis
Combining the previous architecture work with this webinar suggests a stronger purpose for plenty-gobble.
It is not only an adaptive, PostgreSQL-centered columnar store.
It can become an infrastructure layer for evidence-preserving agentic observability.
The architectural objective becomes:
Retain as much semantic evidence as economically possible, compress it losslessly, move old evidence to cheap tiers rather than deleting it, expose every unavoidable loss explicitly, and let query-time pruning—not ingest-time assumptions—determine what an investigation reads.
30. Compact Principles
The research can be reduced to ten operating principles:
- Store broadly, query selectively.
- Unknown future questions require preserved raw evidence.
- Silent incompleteness is more dangerous than explicit incompleteness.
- Loss must itself be observable.
- Retention, sampling, and rollups are policies, not defaults.
- Prefer lossless physical optimization before semantic reduction.
- Cold and slow is better than deleted.
- Case Files must describe evidence completeness, not only evidence.
- Incident precedent should survive raw-data expiry.
- Agent confidence must be conditioned on evidence coverage.
Final Research Statement
For agentic incident analysis, the observability system is part of the epistemic boundary of the agent. Any hidden data loss becomes hidden model error.
plenty-gobbleshould therefore optimize for evidence preservation, explicit loss lineage, and query-time selectivity rather than prematurely deciding which future evidence will matter.