24 KiB
xperience-grader Research Program
Status
Initial research program draft.
This document defines the staged research agenda for xperience-grader, a self-improving experience grading and optimization service for human–machine and agent–machine interfaces.
The program is intentionally empirical.
Its purpose is not to invent a scoring framework in isolation, but to determine which observations, measurements, dimensions, and experimental methods are actually useful for characterizing interface experience.
1. Research Objective
The central research question is:
Can the quality of interaction between an intelligent actor and a machine interface be measured in a sufficiently useful, reproducible, and actionable way to support systematic interface improvement?
The project investigates this question across two related domains:
- HMIX — Human–Machine Interface Experience
- AMIX — Agent–Machine Interface Experience
The project further investigates whether both can be grounded in a shared interaction model while preserving actor-specific measurements and constraints.
The long-term goal is to establish a service capable of:
observe
→ measure
→ grade
→ diagnose
→ propose improvement
→ re-evaluate
→ learn
The research program therefore spans measurement science, human–computer interaction, agent evaluation, software interface design, observability, experimentation, and optimization.
2. Core Research Hypotheses
The initial program works from a set of explicit hypotheses.
These hypotheses should be challenged experimentally.
H1 — Interface quality is relational
Interface quality cannot be adequately represented without considering:
actor
task
environment
constraints
interface
A useful grade therefore requires an interaction context rather than only static inspection of the interface.
H2 — Task-level observations are more useful than interface-level opinions
A workload executed against an interface produces stronger evidence than a generic assessment of whether an interface is "good".
H3 — Experience quality is multidimensional
A single scalar score is insufficient.
Useful evaluation requires multiple dimensions such as:
success
effort
uncertainty
latency
recoverability
learnability
safety
robustness
H4 — Interaction traces contain measurable experience signals
Metrics derived from real interaction traces can reveal meaningful interface friction.
Examples include:
number of actions
retries
failed operations
context consumed
latency
semantic repairs
backtracking
human escalations
H5 — Humans and agents share some interface-quality principles
Certain interface properties may systematically benefit both humans and agents.
Candidates include:
explicit state
clear semantics
structured feedback
predictable transitions
reversible actions
discoverable capabilities
H6 — Humans and agents also have fundamentally different experience constraints
HMIX and AMIX cannot be collapsed into one identical rubric.
H7 — Experience regressions can be detected automatically
Changes to interfaces should produce measurable differences in workload performance.
H8 — Experience deficiencies can be diagnosed into reusable patterns
Recurring interaction failures may be classifiable into common design deficiencies.
H9 — Interface modifications can produce experimentally demonstrable improvements
Recommended changes should be testable through repeated workload execution.
H10 — The grading system can improve without losing reproducibility
The system can learn better workloads, measurements, and diagnostic models while preserving versioned evaluation methodology.
3. Research Principles
The research program follows the principles below.
Evidence First
Claims about experience quality should be grounded in observable evidence wherever possible.
Separate Observation From Judgment
Maintain an explicit distinction between:
observation
measurement
interpretation
grade
diagnosis
recommendation
Keep Raw Evidence
Derived metrics must not replace underlying interaction traces.
Prefer Task Success Over Surface Preference
An interface should primarily be evaluated by what actors can accomplish through it.
Compare Against Alternatives
Absolute scores are less informative than controlled comparisons.
Preserve Actor Context
Evaluation results should always identify the actor profile and environment used.
Expose Uncertainty
A grade should be allowed to say:
insufficient evidence
high evaluator variance
unknown
Optimize Conservatively
Changes should preserve capability, safety, and recoverability.
4. Research Architecture
The research program is organized into six major tracks.
Track A — Foundations and Measurement Model
Track B — HMIX Evaluation
Track C — AMIX Evaluation
Track D — Benchmarking and Calibration
Track E — Diagnosis and Optimization
Track F — Self-Improvement and Continuous Grading
These tracks overlap, but should mature approximately in that order.
5. Track A — Foundations and Measurement Model
Goal
Establish the minimum common model required to conduct repeatable experiments.
This phase should avoid premature complexity.
The initial goal is not to define the final ontology.
It is to define enough structure to collect comparable evidence.
A1. Define the Experience Observation
Create the first machine-readable schema for:
ExperienceObservation
Minimum fields should include:
observation_id
actor
actor_profile
system
interface_surface
task
goal
environment
constraints
starting_state
interaction_events
outcome
measurements
grader_version
workload_version
timestamp
A2. Define the Interaction Event Model
An interaction trace should normalize different surfaces sufficiently to support cross-interface analysis.
Candidate event types:
observe
discover
navigate
read
invoke
write
submit
wait
receive
verify
retry
undo
recover
escalate
abort
Surface-specific detail should remain available as attached evidence.
A3. Define Workload Schema
A workload should specify:
workload_id
description
goal
initial_state
actor_requirements
allowed_surfaces
constraints
success_conditions
failure_conditions
safety_conditions
expected_side_effects
optional_reference_path
The reference path must not become the only acceptable path.
A4. Define Outcome Categories
Candidate outcome classifications:
success
partial_success
recoverable_failure
unrecoverable_failure
unsafe_success
aborted
indeterminate
A5. Define Metric Families
Initial metric families should include:
Outcome
task_success
goal_completion
constraint_compliance
Effort
interaction_count
active_time
wall_time
context_switches
Failure
errors
retries
backtracks
Uncertainty
ambiguous_decisions
exploratory_actions
semantic_repairs
Recovery
recovery_success
recovery_cost
Cost
For agents:
tokens
tool_calls
inference_rounds
compute_cost
For humans:
time
clicks
keystrokes
attention shifts
A6. Establish Data Storage
Store:
raw trace
normalized observation
metrics
grading result
diagnosis
recommendation
evaluation metadata
All derived results should remain traceable to their source evidence.
Deliverable
concepts/ExperienceObservationModel.md
concepts/WorkloadModel.md
schemas/experience-observation.schema.json
schemas/workload.schema.json
6. Track B — HMIX Research
Goal
Determine which measurements meaningfully characterize human interaction quality.
The first phase should focus on practical usability rather than attempting to model the complete human experience.
B1. Establish HMIX Dimensions
Begin with provisional dimensions:
discoverability
comprehensibility
action clarity
predictability
feedback
efficiency
cognitive load
error prevention
error legibility
recoverability
learnability
trust calibration
control
accessibility
Each dimension should define:
meaning
observable indicators
possible measurements
known confounders
evaluation confidence
B2. Select Initial Human Workloads
Choose small workflows with objectively verifiable outcomes.
Suggested classes:
account creation
search and retrieval
configuration
data entry
approval
error recovery
administrative operation
Avoid aesthetic-heavy evaluations initially.
B3. Compare Expert Review With Measured Interaction
For each workload collect:
expert UX review
human interaction trace
user-reported difficulty
task outcome
quantitative metrics
Investigate where these disagree.
B4. Synthetic HMIX Evaluation
Test whether multimodal AI evaluators can identify interface friction before human testing.
Compare:
synthetic prediction
vs
human observation
Synthetic grading should initially be treated as a hypothesis generator rather than ground truth.
B5. Human Variance
Repeat identical workloads with different actor profiles.
Candidate profiles:
novice
occasional user
expert user
domain expert
technical administrator
Measure how much interface quality depends on expertise.
B6. Learnability Experiments
Repeat tasks over multiple sessions.
Measure:
time reduction
interaction reduction
error reduction
memory dependency
Determine which interfaces reward learning.
Deliverables
hmix/HMIXDimensions.md
hmix/HMIXMeasurementProtocol.md
hmix/workloads/
hmix/experiments/
7. Track C — AMIX Research
Goal
Develop repeatable methods for measuring interface quality for autonomous and semi-autonomous agents.
This is expected to be the most novel part of the program.
C1. Establish AMIX Dimensions
Begin with:
capability discoverability
semantic legibility
contract precision
invocability
composability
state legibility
observability
verifiability
error legibility
recoverability
permission legibility
idempotency
determinism
context efficiency
interaction efficiency
automation depth
distillability
C2. Define Agent Profiles
AMIX results should always identify the agent configuration.
Example:
model
model version
reasoning mode
context limit
tool-use capabilities
memory
available documentation
initial prompt
execution constraints
C3. Establish Model Diversity
Experiments should not rely exclusively on frontier models.
Use at least three broad capability classes:
frontier model
mid-tier general model
small/local model
The purpose is to distinguish:
interface quality
from
model capability
C4. Semantic Repair Measurement
Develop a precise taxonomy for semantic repair.
Candidate categories:
documentation lookup
schema reinterpretation
trial invocation
source inspection
parameter guessing
state probing
error reinterpretation
human clarification
Measure:
repair count
repair duration
repair tokens
repair success
C5. Agent Error Recovery
Deliberately induce:
invalid inputs
stale state
permission failures
timeouts
partial success
ambiguous responses
duplicate invocation risk
Observe whether agents can safely recover.
C6. Distillability
After successful exploratory completion, attempt to transform the interaction into:
deterministic script
workflow
tool procedure
compiled plan
Measure:
success after distillation
cost reduction
robustness
remaining agent reasoning requirement
C7. Permission Legibility
Test whether an agent can distinguish:
can
may
must
must not
without unsafe experimentation.
C8. Interface Family Comparison
Run comparable tasks across:
REST API
CLI
MCP
GraphQL
web UI
SDK
The goal is not to rank interface families globally.
Instead investigate which characteristics affect AMIX independent of protocol.
Deliverables
amix/AMIXDimensions.md
amix/AgentProfileModel.md
amix/SemanticRepairTaxonomy.md
amix/workloads/
amix/experiments/
8. Track D — Benchmarking and Calibration
Goal
Determine whether measurements are reproducible and useful enough to support grading.
D1. Build Initial Benchmark Corpus
Select approximately:
5 systems
×
5 representative workloads
×
multiple actor profiles
Prefer systems that differ meaningfully in interface quality.
D2. Repeatability
Repeat identical evaluations.
Measure:
outcome variance
metric variance
grade variance
D3. Evaluator Variance
Compare:
human evaluator A
human evaluator B
agent evaluator A
agent evaluator B
Determine which dimensions are objective enough to automate.
D4. Sensitivity Testing
Introduce controlled interface modifications.
Examples:
remove field descriptions
improve error structure
add explicit state endpoint
rename ambiguous operation
remove confirmation feedback
increase navigation depth
Test whether grading responds as expected.
D5. Calibration Anchors
Create small reference examples representing:
excellent
adequate
problematic
severely problematic
These should be evidence-backed exemplars, not subjective labels alone.
D6. Correlation Analysis
Investigate relationships among metrics.
Examples:
semantic repairs ↔ task failure
tool calls ↔ token cost
human backtracking ↔ perceived difficulty
state observability ↔ recovery success
Avoid assuming causal relationships prematurely.
Deliverables
benchmarks/
calibration/
analysis/
9. Track E — Diagnosis and Optimization
Goal
Move beyond grades toward actionable interface improvement.
E1. Experience Defect Taxonomy
Create a catalog of recurring interface defects.
Candidate categories:
hidden capability
ambiguous operation
implicit state
weak feedback
unstructured failure
irreversible action
identifier ambiguity
permission opacity
unsafe retry
excessive navigation
context dependency
E2. Map Measurements to Diagnoses
Example:
Observation:
Repeated resource searches
Metric:
high discovery cost
Likely diagnosis:
resource identifiers not propagated
Possible improvement:
return canonical resource references
E3. Pattern Library
For each successful improvement record:
problem pattern
affected actors
proposed intervention
before metrics
after metrics
tradeoffs
confidence
E4. Recommendation Evaluation
Recommendations should be evaluated experimentally.
Required loop:
baseline
→ proposed change
→ implementation
→ repeated workload
→ comparison
E5. Multi-objective Optimization
Optimization should account for:
success
efficiency
safety
recoverability
expressiveness
cost
Avoid optimization toward a single metric.
E6. Cross-Actor Tradeoffs
Explicitly test whether a change:
improves HMIX
degrades AMIX
or
improves AMIX
degrades HMIX
Record these tradeoffs.
Deliverables
diagnostics/ExperienceDefectTaxonomy.md
patterns/
optimizer/
10. Track F — Self-Improving Grading
Goal
Allow the service to improve its own evaluation methodology while maintaining auditability.
F1. Version Everything
Version:
workloads
actor profiles
rubrics
metric definitions
grader prompts
models
benchmark corpora
A grade should always be reproducible against a historical version.
F2. Grader Evaluation
The grader itself should be benchmarked.
Questions include:
Does it identify known defects?
Does it detect controlled regressions?
Does it overreact to irrelevant differences?
Does it agree with human evidence?
Does it generalize across interface families?
F3. Workload Evolution
Detect gaps in the benchmark corpus from:
production failures
support incidents
new interface types
optimizer failures
unexplained grading disagreement
Add workloads accordingly.
F4. Metric Evolution
A metric should be retained only if it provides explanatory or predictive value.
Track:
usefulness
stability
cost
redundancy
F5. Diagnostic Learning
Accumulate evidence about which diagnoses are associated with successful improvements.
Over time:
defect
→ intervention
→ observed outcome
should become an empirical knowledge base.
F6. Confidence Model
Every assessment should eventually express:
score
confidence
evidence quantity
evidence diversity
evaluator agreement
11. Initial Experimental Systems
The first research cycle should deliberately use small, inspectable targets.
Recommended target set:
System 1 — Simple REST API
Focus:
discoverability
schema clarity
errors
idempotency
System 2 — CLI Tool
Focus:
discoverability
help quality
feedback
recovery
System 3 — MCP Service
Focus:
agent discovery
tool descriptions
semantic legibility
composition
System 4 — Human Web Application
Focus:
navigation
feedback
mental model
error recovery
System 5 — Administrative Interface
Focus:
high-impact operations
permission clarity
state visibility
reversibility
12. Minimum Viable Experiment
The first end-to-end experiment should be intentionally modest.
Use:
1 REST API
1 human web workflow
1 MCP service
For each:
3 workloads
3 repetitions
at least 2 actor profiles
Capture:
full traces
task outcomes
time
interaction count
errors
retries
semantic repairs where applicable
qualitative observations
Produce:
dimension assessment
diagnosis
recommendations
confidence
The immediate question is:
Does the resulting report tell us something useful that ordinary testing did not?
If not, improve the research model before scaling.
13. Milestone Program
M0 — Research Scaffold
Establish:
INTENT.md
ResearchProgram.md
terminology
research structure
experiment structure
Exit condition:
The program can describe how an observation will be captured.
M1 — Observation Kernel
Implement:
ExperienceObservation
Workload
ActorProfile
InteractionTrace
Exit condition:
One workload can be recorded end-to-end.
M2 — First AMIX Experiment
Evaluate one API or MCP service.
Exit condition:
A repeatable AMIX report can be generated from evidence.
M3 — First HMIX Experiment
Evaluate one human workflow.
Exit condition:
Human interaction evidence can be represented in the same observation framework.
M4 — Mixed Benchmark Corpus
Create first cross-interface benchmark suite.
Exit condition:
At least five systems and approximately twenty workloads exist.
M5 — Grading v0.1
Introduce initial dimension grading.
Exit condition:
Grades are decomposable, evidence-backed, and reproducible.
M6 — Diagnosis v0.1
Create first experience defect taxonomy.
Exit condition:
Observed problems can be mapped to reusable diagnostic categories.
M7 — Optimization Experiment
Modify an interface based on grader findings.
Exit condition:
A measurable before/after improvement is demonstrated.
M8 — Regression Service
Run benchmark workloads repeatedly across interface versions.
Exit condition:
An experience regression can be detected automatically.
M9 — Self-Improving Grader
Use accumulated experiments to improve rubric, workloads, and diagnostics.
Exit condition:
A new grader version demonstrably outperforms an older version on the benchmark corpus.
14. Research Artifacts
The repository should accumulate four major forms of research output.
concepts/
experiments/
benchmarks/
patterns/
Possible initial structure:
xperience-grader/
│
├── INTENT.md
├── ResearchProgram.md
│
├── concepts/
│ ├── ExperienceObservationModel.md
│ ├── WorkloadModel.md
│ ├── ActorProfileModel.md
│ └── GradingModel.md
│
├── hmix/
│ ├── HMIXDimensions.md
│ ├── workloads/
│ └── experiments/
│
├── amix/
│ ├── AMIXDimensions.md
│ ├── SemanticRepairTaxonomy.md
│ ├── workloads/
│ └── experiments/
│
├── benchmarks/
│
├── diagnostics/
│
├── patterns/
│
├── schemas/
│
└── research/
15. Research Risks
Premature Scoring
Creating attractive scores before establishing measurement validity.
Mitigation:
Keep dimensions and evidence visible.
Model Bias
AMIX results may become model-specific.
Mitigation:
Use multiple agent classes and explicitly encode actor profiles.
Synthetic UX Illusion
AI-generated HMIX reviews may appear more reliable than they are.
Mitigation:
Calibrate against real human studies.
Benchmark Gaming
Interfaces could be optimized to the benchmark rather than actual usability.
Mitigation:
Maintain hidden and evolving workloads.
Measurement Distortion
Metrics such as clicks or token counts can become optimization targets despite weak relationship to quality.
Mitigation:
Use multi-objective evaluation.
Rubric Drift
Self-improvement could make historical grades incomparable.
Mitigation:
Version methodologies and retain old graders.
Observer Effects
Instrumentation can alter interaction behavior.
Mitigation:
Measure instrumentation overhead.
16. Success Criteria
The research program becomes practically valuable when it can demonstrate all of the following:
repeatable workload execution
evidence-backed grading
meaningful distinction between interface variants
useful diagnosis
measurable improvement after intervention
experience regression detection
cross-model AMIX comparison
cross-user HMIX comparison
historically reproducible results
The strongest validation will be:
An interface change proposed from xperience-grader evidence produces a repeatable improvement for the target actors and workloads without unacceptable degradation elsewhere.
17. Long-Term Research Questions
The mature program should explore deeper questions.
Interface Universals
Are there interface properties that consistently improve interaction across human and artificial actors?
Intelligence-Relative Interfaces
Should interfaces adapt dynamically to actor capability?
Experience Compilation
Can exploratory interaction be progressively compiled into deterministic procedure?
Experience Debt
Can poor interface quality accumulate similarly to technical debt?
Interface Fitness Landscapes
Can interface variants be explored as an optimization landscape?
Adaptive Interfaces
Should interfaces themselves evolve based on observed interaction difficulty?
Experience Economics
What is the economic cost of interface friction?
For agents:
tokens
compute
latency
human escalation
For humans:
time
training
support
errors
frustration
Mixed Human–Agent Experience
How should interfaces be evaluated when work continuously moves between humans and agents?
This may ultimately require a third perspective beyond HMIX and AMIX:
collaborative MIX
18. Research North Star
The program should continuously return to one question:
Can we turn interface experience from an opinion into an observable, testable, improvable system property?
If the answer becomes yes, xperience-grader should evolve beyond a research tool into infrastructure for continuous experience engineering.
The target loop is:
experience
↓
evidence
↓
understanding
↓
change
↓
better experience
↓
new evidence
└──────────────→
The research program exists to determine how much of that loop can be made reliable.