xperience-grader/ResearchProgram.md

# xperience-grader Research Program

## Status

Initial research program draft.

This document defines the staged research agenda for `xperience-grader`, a self-improving experience grading and optimization service for human–machine and agent–machine interfaces.

The program is intentionally empirical.

Its purpose is not to invent a scoring framework in isolation, but to determine which observations, measurements, dimensions, and experimental methods are actually useful for characterizing interface experience.

---

# 1. Research Objective

The central research question is:

> Can the quality of interaction between an intelligent actor and a machine interface be measured in a sufficiently useful, reproducible, and actionable way to support systematic interface improvement?

The project investigates this question across two related domains:

- **HMIX — Human–Machine Interface Experience**
- **AMIX — Agent–Machine Interface Experience**

The project further investigates whether both can be grounded in a shared interaction model while preserving actor-specific measurements and constraints.

The long-term goal is to establish a service capable of:

```text
observe
→ measure
→ grade
→ diagnose
→ propose improvement
→ re-evaluate
→ learn
```

The research program therefore spans measurement science, human–computer interaction, agent evaluation, software interface design, observability, experimentation, and optimization.

---

# 2. Core Research Hypotheses

The initial program works from a set of explicit hypotheses.

These hypotheses should be challenged experimentally.

## H1 — Interface quality is relational

Interface quality cannot be adequately represented without considering:

```text
actor
task
environment
constraints
interface
```

A useful grade therefore requires an interaction context rather than only static inspection of the interface.

---

## H2 — Task-level observations are more useful than interface-level opinions

A workload executed against an interface produces stronger evidence than a generic assessment of whether an interface is "good".

---

## H3 — Experience quality is multidimensional

A single scalar score is insufficient.

Useful evaluation requires multiple dimensions such as:

```text
success
effort
uncertainty
latency
recoverability
learnability
safety
robustness
```

---

## H4 — Interaction traces contain measurable experience signals

Metrics derived from real interaction traces can reveal meaningful interface friction.

Examples include:

```text
number of actions
retries
failed operations
context consumed
latency
semantic repairs
backtracking
human escalations
```

---

## H5 — Humans and agents share some interface-quality principles

Certain interface properties may systematically benefit both humans and agents.

Candidates include:

```text
explicit state
clear semantics
structured feedback
predictable transitions
reversible actions
discoverable capabilities
```

---

## H6 — Humans and agents also have fundamentally different experience constraints

HMIX and AMIX cannot be collapsed into one identical rubric.

---

## H7 — Experience regressions can be detected automatically

Changes to interfaces should produce measurable differences in workload performance.

---

## H8 — Experience deficiencies can be diagnosed into reusable patterns

Recurring interaction failures may be classifiable into common design deficiencies.

---

## H9 — Interface modifications can produce experimentally demonstrable improvements

Recommended changes should be testable through repeated workload execution.

---

## H10 — The grading system can improve without losing reproducibility

The system can learn better workloads, measurements, and diagnostic models while preserving versioned evaluation methodology.

---

# 3. Research Principles

The research program follows the principles below.

## Evidence First

Claims about experience quality should be grounded in observable evidence wherever possible.

## Separate Observation From Judgment

Maintain an explicit distinction between:

```text
observation
measurement
interpretation
grade
diagnosis
recommendation
```

## Keep Raw Evidence

Derived metrics must not replace underlying interaction traces.

## Prefer Task Success Over Surface Preference

An interface should primarily be evaluated by what actors can accomplish through it.

## Compare Against Alternatives

Absolute scores are less informative than controlled comparisons.

## Preserve Actor Context

Evaluation results should always identify the actor profile and environment used.

## Expose Uncertainty

A grade should be allowed to say:

```text
insufficient evidence
high evaluator variance
unknown
```

## Optimize Conservatively

Changes should preserve capability, safety, and recoverability.

---

# 4. Research Architecture

The research program is organized into six major tracks.

```text
Track A — Foundations and Measurement Model
Track B — HMIX Evaluation
Track C — AMIX Evaluation
Track D — Benchmarking and Calibration
Track E — Diagnosis and Optimization
Track F — Self-Improvement and Continuous Grading
```

These tracks overlap, but should mature approximately in that order.

---

# 5. Track A — Foundations and Measurement Model

## Goal

Establish the minimum common model required to conduct repeatable experiments.

This phase should avoid premature complexity.

The initial goal is not to define the final ontology.

It is to define enough structure to collect comparable evidence.

---

## A1. Define the Experience Observation

Create the first machine-readable schema for:

```text
ExperienceObservation
```

Minimum fields should include:

```text
observation_id

actor
actor_profile

system
interface_surface

task
goal

environment
constraints

starting_state

interaction_events

outcome

measurements

grader_version
workload_version
timestamp
```

---

## A2. Define the Interaction Event Model

An interaction trace should normalize different surfaces sufficiently to support cross-interface analysis.

Candidate event types:

```text
observe
discover
navigate
read
invoke
write
submit
wait
receive
verify
retry
undo
recover
escalate
abort
```

Surface-specific detail should remain available as attached evidence.

---

## A3. Define Workload Schema

A workload should specify:

```text
workload_id
description

goal
initial_state

actor_requirements
allowed_surfaces

constraints

success_conditions
failure_conditions
safety_conditions

expected_side_effects

optional_reference_path
```

The reference path must not become the only acceptable path.

---

## A4. Define Outcome Categories

Candidate outcome classifications:

```text
success
partial_success
recoverable_failure
unrecoverable_failure
unsafe_success
aborted
indeterminate
```

---

## A5. Define Metric Families

Initial metric families should include:

### Outcome

```text
task_success
goal_completion
constraint_compliance
```

### Effort

```text
interaction_count
active_time
wall_time
context_switches
```

### Failure

```text
errors
retries
backtracks
```

### Uncertainty

```text
ambiguous_decisions
exploratory_actions
semantic_repairs
```

### Recovery

```text
recovery_success
recovery_cost
```

### Cost

For agents:

```text
tokens
tool_calls
inference_rounds
compute_cost
```

For humans:

```text
time
clicks
keystrokes
attention shifts
```

---

## A6. Establish Data Storage

Store:

```text
raw trace
normalized observation
metrics
grading result
diagnosis
recommendation
evaluation metadata
```

All derived results should remain traceable to their source evidence.

---

## Deliverable

```text
concepts/ExperienceObservationModel.md
concepts/WorkloadModel.md
schemas/experience-observation.schema.json
schemas/workload.schema.json
```

---

# 6. Track B — HMIX Research

## Goal

Determine which measurements meaningfully characterize human interaction quality.

The first phase should focus on practical usability rather than attempting to model the complete human experience.

---

# B1. Establish HMIX Dimensions

Begin with provisional dimensions:

```text
discoverability
comprehensibility
action clarity
predictability
feedback
efficiency
cognitive load
error prevention
error legibility
recoverability
learnability
trust calibration
control
accessibility
```

Each dimension should define:

```text
meaning
observable indicators
possible measurements
known confounders
evaluation confidence
```

---

# B2. Select Initial Human Workloads

Choose small workflows with objectively verifiable outcomes.

Suggested classes:

```text
account creation
search and retrieval
configuration
data entry
approval
error recovery
administrative operation
```

Avoid aesthetic-heavy evaluations initially.

---

# B3. Compare Expert Review With Measured Interaction

For each workload collect:

```text
expert UX review
human interaction trace
user-reported difficulty
task outcome
quantitative metrics
```

Investigate where these disagree.

---

# B4. Synthetic HMIX Evaluation

Test whether multimodal AI evaluators can identify interface friction before human testing.

Compare:

```text
synthetic prediction
vs
human observation
```

Synthetic grading should initially be treated as a hypothesis generator rather than ground truth.

---

# B5. Human Variance

Repeat identical workloads with different actor profiles.

Candidate profiles:

```text
novice
occasional user
expert user
domain expert
technical administrator
```

Measure how much interface quality depends on expertise.

---

# B6. Learnability Experiments

Repeat tasks over multiple sessions.

Measure:

```text
time reduction
interaction reduction
error reduction
memory dependency
```

Determine which interfaces reward learning.

---

## Deliverables

```text
hmix/HMIXDimensions.md
hmix/HMIXMeasurementProtocol.md
hmix/workloads/
hmix/experiments/
```

---

# 7. Track C — AMIX Research

## Goal

Develop repeatable methods for measuring interface quality for autonomous and semi-autonomous agents.

This is expected to be the most novel part of the program.

---

# C1. Establish AMIX Dimensions

Begin with:

```text
capability discoverability
semantic legibility
contract precision
invocability
composability
state legibility
observability
verifiability
error legibility
recoverability
permission legibility
idempotency
determinism
context efficiency
interaction efficiency
automation depth
distillability
```

---

# C2. Define Agent Profiles

AMIX results should always identify the agent configuration.

Example:

```text
model
model version
reasoning mode
context limit
tool-use capabilities
memory
available documentation
initial prompt
execution constraints
```

---

# C3. Establish Model Diversity

Experiments should not rely exclusively on frontier models.

Use at least three broad capability classes:

```text
frontier model
mid-tier general model
small/local model
```

The purpose is to distinguish:

```text
interface quality
from
model capability
```

---

# C4. Semantic Repair Measurement

Develop a precise taxonomy for semantic repair.

Candidate categories:

```text
documentation lookup
schema reinterpretation
trial invocation
source inspection
parameter guessing
state probing
error reinterpretation
human clarification
```

Measure:

```text
repair count
repair duration
repair tokens
repair success
```

---

# C5. Agent Error Recovery

Deliberately induce:

```text
invalid inputs
stale state
permission failures
timeouts
partial success
ambiguous responses
duplicate invocation risk
```

Observe whether agents can safely recover.

---

# C6. Distillability

After successful exploratory completion, attempt to transform the interaction into:

```text
deterministic script
workflow
tool procedure
compiled plan
```

Measure:

```text
success after distillation
cost reduction
robustness
remaining agent reasoning requirement
```

---

# C7. Permission Legibility

Test whether an agent can distinguish:

```text
can
may
must
must not
```

without unsafe experimentation.

---

# C8. Interface Family Comparison

Run comparable tasks across:

```text
REST API
CLI
MCP
GraphQL
web UI
SDK
```

The goal is not to rank interface families globally.

Instead investigate which characteristics affect AMIX independent of protocol.

---

## Deliverables

```text
amix/AMIXDimensions.md
amix/AgentProfileModel.md
amix/SemanticRepairTaxonomy.md
amix/workloads/
amix/experiments/
```

---

# 8. Track D — Benchmarking and Calibration

## Goal

Determine whether measurements are reproducible and useful enough to support grading.

---

# D1. Build Initial Benchmark Corpus

Select approximately:

```text
5 systems
×
5 representative workloads
×
multiple actor profiles
```

Prefer systems that differ meaningfully in interface quality.

---

# D2. Repeatability

Repeat identical evaluations.

Measure:

```text
outcome variance
metric variance
grade variance
```

---

# D3. Evaluator Variance

Compare:

```text
human evaluator A
human evaluator B
agent evaluator A
agent evaluator B
```

Determine which dimensions are objective enough to automate.

---

# D4. Sensitivity Testing

Introduce controlled interface modifications.

Examples:

```text
remove field descriptions
improve error structure
add explicit state endpoint
rename ambiguous operation
remove confirmation feedback
increase navigation depth
```

Test whether grading responds as expected.

---

# D5. Calibration Anchors

Create small reference examples representing:

```text
excellent
adequate
problematic
severely problematic
```

These should be evidence-backed exemplars, not subjective labels alone.

---

# D6. Correlation Analysis

Investigate relationships among metrics.

Examples:

```text
semantic repairs ↔ task failure

tool calls ↔ token cost

human backtracking ↔ perceived difficulty

state observability ↔ recovery success
```

Avoid assuming causal relationships prematurely.

---

## Deliverables

```text
benchmarks/
calibration/
analysis/
```

---

# 9. Track E — Diagnosis and Optimization

## Goal

Move beyond grades toward actionable interface improvement.

---

# E1. Experience Defect Taxonomy

Create a catalog of recurring interface defects.

Candidate categories:

```text
hidden capability
ambiguous operation
implicit state
weak feedback
unstructured failure
irreversible action
identifier ambiguity
permission opacity
unsafe retry
excessive navigation
context dependency
```

---

# E2. Map Measurements to Diagnoses

Example:

```text
Observation:
Repeated resource searches

Metric:
high discovery cost

Likely diagnosis:
resource identifiers not propagated

Possible improvement:
return canonical resource references
```

---

# E3. Pattern Library

For each successful improvement record:

```text
problem pattern
affected actors
proposed intervention
before metrics
after metrics
tradeoffs
confidence
```

---

# E4. Recommendation Evaluation

Recommendations should be evaluated experimentally.

Required loop:

```text
baseline
→ proposed change
→ implementation
→ repeated workload
→ comparison
```

---

# E5. Multi-objective Optimization

Optimization should account for:

```text
success
efficiency
safety
recoverability
expressiveness
cost
```

Avoid optimization toward a single metric.

---

# E6. Cross-Actor Tradeoffs

Explicitly test whether a change:

```text
improves HMIX
degrades AMIX

or

improves AMIX
degrades HMIX
```

Record these tradeoffs.

---

## Deliverables

```text
diagnostics/ExperienceDefectTaxonomy.md
patterns/
optimizer/
```

---

# 10. Track F — Self-Improving Grading

## Goal

Allow the service to improve its own evaluation methodology while maintaining auditability.

---

# F1. Version Everything

Version:

```text
workloads
actor profiles
rubrics
metric definitions
grader prompts
models
benchmark corpora
```

A grade should always be reproducible against a historical version.

---

# F2. Grader Evaluation

The grader itself should be benchmarked.

Questions include:

```text
Does it identify known defects?

Does it detect controlled regressions?

Does it overreact to irrelevant differences?

Does it agree with human evidence?

Does it generalize across interface families?
```

---

# F3. Workload Evolution

Detect gaps in the benchmark corpus from:

```text
production failures
support incidents
new interface types
optimizer failures
unexplained grading disagreement
```

Add workloads accordingly.

---

# F4. Metric Evolution

A metric should be retained only if it provides explanatory or predictive value.

Track:

```text
usefulness
stability
cost
redundancy
```

---

# F5. Diagnostic Learning

Accumulate evidence about which diagnoses are associated with successful improvements.

Over time:

```text
defect
→ intervention
→ observed outcome
```

should become an empirical knowledge base.

---

# F6. Confidence Model

Every assessment should eventually express:

```text
score
confidence
evidence quantity
evidence diversity
evaluator agreement
```

---

# 11. Initial Experimental Systems

The first research cycle should deliberately use small, inspectable targets.

Recommended target set:

## System 1 — Simple REST API

Focus:

```text
discoverability
schema clarity
errors
idempotency
```

## System 2 — CLI Tool

Focus:

```text
discoverability
help quality
feedback
recovery
```

## System 3 — MCP Service

Focus:

```text
agent discovery
tool descriptions
semantic legibility
composition
```

## System 4 — Human Web Application

Focus:

```text
navigation
feedback
mental model
error recovery
```

## System 5 — Administrative Interface

Focus:

```text
high-impact operations
permission clarity
state visibility
reversibility
```

---

# 12. Minimum Viable Experiment

The first end-to-end experiment should be intentionally modest.

Use:

```text
1 REST API
1 human web workflow
1 MCP service
```

For each:

```text
3 workloads
3 repetitions
at least 2 actor profiles
```

Capture:

```text
full traces
task outcomes
time
interaction count
errors
retries
semantic repairs where applicable
qualitative observations
```

Produce:

```text
dimension assessment
diagnosis
recommendations
confidence
```

The immediate question is:

> Does the resulting report tell us something useful that ordinary testing did not?

If not, improve the research model before scaling.

---

# 13. Milestone Program

## M0 — Research Scaffold

Establish:

```text
INTENT.md
ResearchProgram.md
terminology
research structure
experiment structure
```

Exit condition:

The program can describe how an observation will be captured.

---

## M1 — Observation Kernel

Implement:

```text
ExperienceObservation
Workload
ActorProfile
InteractionTrace
```

Exit condition:

One workload can be recorded end-to-end.

---

## M2 — First AMIX Experiment

Evaluate one API or MCP service.

Exit condition:

A repeatable AMIX report can be generated from evidence.

---

## M3 — First HMIX Experiment

Evaluate one human workflow.

Exit condition:

Human interaction evidence can be represented in the same observation framework.

---

## M4 — Mixed Benchmark Corpus

Create first cross-interface benchmark suite.

Exit condition:

At least five systems and approximately twenty workloads exist.

---

## M5 — Grading v0.1

Introduce initial dimension grading.

Exit condition:

Grades are decomposable, evidence-backed, and reproducible.

---

## M6 — Diagnosis v0.1

Create first experience defect taxonomy.

Exit condition:

Observed problems can be mapped to reusable diagnostic categories.

---

## M7 — Optimization Experiment

Modify an interface based on grader findings.

Exit condition:

A measurable before/after improvement is demonstrated.

---

## M8 — Regression Service

Run benchmark workloads repeatedly across interface versions.

Exit condition:

An experience regression can be detected automatically.

---

## M9 — Self-Improving Grader

Use accumulated experiments to improve rubric, workloads, and diagnostics.

Exit condition:

A new grader version demonstrably outperforms an older version on the benchmark corpus.

---

# 14. Research Artifacts

The repository should accumulate four major forms of research output.

```text
concepts/
experiments/
benchmarks/
patterns/
```

Possible initial structure:

```text
xperience-grader/
│
├── INTENT.md
├── ResearchProgram.md
│
├── concepts/
│   ├── ExperienceObservationModel.md
│   ├── WorkloadModel.md
│   ├── ActorProfileModel.md
│   └── GradingModel.md
│
├── hmix/
│   ├── HMIXDimensions.md
│   ├── workloads/
│   └── experiments/
│
├── amix/
│   ├── AMIXDimensions.md
│   ├── SemanticRepairTaxonomy.md
│   ├── workloads/
│   └── experiments/
│
├── benchmarks/
│
├── diagnostics/
│
├── patterns/
│
├── schemas/
│
└── research/
```

---

# 15. Research Risks

## Premature Scoring

Creating attractive scores before establishing measurement validity.

Mitigation:

Keep dimensions and evidence visible.

---

## Model Bias

AMIX results may become model-specific.

Mitigation:

Use multiple agent classes and explicitly encode actor profiles.

---

## Synthetic UX Illusion

AI-generated HMIX reviews may appear more reliable than they are.

Mitigation:

Calibrate against real human studies.

---

## Benchmark Gaming

Interfaces could be optimized to the benchmark rather than actual usability.

Mitigation:

Maintain hidden and evolving workloads.

---

## Measurement Distortion

Metrics such as clicks or token counts can become optimization targets despite weak relationship to quality.

Mitigation:

Use multi-objective evaluation.

---

## Rubric Drift

Self-improvement could make historical grades incomparable.

Mitigation:

Version methodologies and retain old graders.

---

## Observer Effects

Instrumentation can alter interaction behavior.

Mitigation:

Measure instrumentation overhead.

---

# 16. Success Criteria

The research program becomes practically valuable when it can demonstrate all of the following:

```text
repeatable workload execution

evidence-backed grading

meaningful distinction between interface variants

useful diagnosis

measurable improvement after intervention

experience regression detection

cross-model AMIX comparison

cross-user HMIX comparison

historically reproducible results
```

The strongest validation will be:

> An interface change proposed from xperience-grader evidence produces a repeatable improvement for the target actors and workloads without unacceptable degradation elsewhere.

---

# 17. Long-Term Research Questions

The mature program should explore deeper questions.

## Interface Universals

Are there interface properties that consistently improve interaction across human and artificial actors?

---

## Intelligence-Relative Interfaces

Should interfaces adapt dynamically to actor capability?

---

## Experience Compilation

Can exploratory interaction be progressively compiled into deterministic procedure?

---

## Experience Debt

Can poor interface quality accumulate similarly to technical debt?

---

## Interface Fitness Landscapes

Can interface variants be explored as an optimization landscape?

---

## Adaptive Interfaces

Should interfaces themselves evolve based on observed interaction difficulty?

---

## Experience Economics

What is the economic cost of interface friction?

For agents:

```text
tokens
compute
latency
human escalation
```

For humans:

```text
time
training
support
errors
frustration
```

---

## Mixed Human–Agent Experience

How should interfaces be evaluated when work continuously moves between humans and agents?

This may ultimately require a third perspective beyond HMIX and AMIX:

```text
collaborative MIX
```

---

# 18. Research North Star

The program should continuously return to one question:

> Can we turn interface experience from an opinion into an observable, testable, improvable system property?

If the answer becomes yes, `xperience-grader` should evolve beyond a research tool into infrastructure for continuous experience engineering.

The target loop is:

```text
experience
   ↓
evidence
   ↓
understanding
   ↓
change
   ↓
better experience
   ↓
new evidence
   └──────────────→
```

The research program exists to determine how much of that loop can be made reliable.