# xperience-grader Research Program ## Status Initial research program draft. This document defines the staged research agenda for `xperience-grader`, a self-improving experience grading and optimization service for human–machine and agent–machine interfaces. The program is intentionally empirical. Its purpose is not to invent a scoring framework in isolation, but to determine which observations, measurements, dimensions, and experimental methods are actually useful for characterizing interface experience. --- # 1. Research Objective The central research question is: > Can the quality of interaction between an intelligent actor and a machine interface be measured in a sufficiently useful, reproducible, and actionable way to support systematic interface improvement? The project investigates this question across two related domains: - **HMIX — Human–Machine Interface Experience** - **AMIX — Agent–Machine Interface Experience** The project further investigates whether both can be grounded in a shared interaction model while preserving actor-specific measurements and constraints. The long-term goal is to establish a service capable of: ```text observe → measure → grade → diagnose → propose improvement → re-evaluate → learn ``` The research program therefore spans measurement science, human–computer interaction, agent evaluation, software interface design, observability, experimentation, and optimization. --- # 2. Core Research Hypotheses The initial program works from a set of explicit hypotheses. These hypotheses should be challenged experimentally. ## H1 — Interface quality is relational Interface quality cannot be adequately represented without considering: ```text actor task environment constraints interface ``` A useful grade therefore requires an interaction context rather than only static inspection of the interface. --- ## H2 — Task-level observations are more useful than interface-level opinions A workload executed against an interface produces stronger evidence than a generic assessment of whether an interface is "good". --- ## H3 — Experience quality is multidimensional A single scalar score is insufficient. Useful evaluation requires multiple dimensions such as: ```text success effort uncertainty latency recoverability learnability safety robustness ``` --- ## H4 — Interaction traces contain measurable experience signals Metrics derived from real interaction traces can reveal meaningful interface friction. Examples include: ```text number of actions retries failed operations context consumed latency semantic repairs backtracking human escalations ``` --- ## H5 — Humans and agents share some interface-quality principles Certain interface properties may systematically benefit both humans and agents. Candidates include: ```text explicit state clear semantics structured feedback predictable transitions reversible actions discoverable capabilities ``` --- ## H6 — Humans and agents also have fundamentally different experience constraints HMIX and AMIX cannot be collapsed into one identical rubric. --- ## H7 — Experience regressions can be detected automatically Changes to interfaces should produce measurable differences in workload performance. --- ## H8 — Experience deficiencies can be diagnosed into reusable patterns Recurring interaction failures may be classifiable into common design deficiencies. --- ## H9 — Interface modifications can produce experimentally demonstrable improvements Recommended changes should be testable through repeated workload execution. --- ## H10 — The grading system can improve without losing reproducibility The system can learn better workloads, measurements, and diagnostic models while preserving versioned evaluation methodology. --- # 3. Research Principles The research program follows the principles below. ## Evidence First Claims about experience quality should be grounded in observable evidence wherever possible. ## Separate Observation From Judgment Maintain an explicit distinction between: ```text observation measurement interpretation grade diagnosis recommendation ``` ## Keep Raw Evidence Derived metrics must not replace underlying interaction traces. ## Prefer Task Success Over Surface Preference An interface should primarily be evaluated by what actors can accomplish through it. ## Compare Against Alternatives Absolute scores are less informative than controlled comparisons. ## Preserve Actor Context Evaluation results should always identify the actor profile and environment used. ## Expose Uncertainty A grade should be allowed to say: ```text insufficient evidence high evaluator variance unknown ``` ## Optimize Conservatively Changes should preserve capability, safety, and recoverability. --- # 4. Research Architecture The research program is organized into six major tracks. ```text Track A — Foundations and Measurement Model Track B — HMIX Evaluation Track C — AMIX Evaluation Track D — Benchmarking and Calibration Track E — Diagnosis and Optimization Track F — Self-Improvement and Continuous Grading ``` These tracks overlap, but should mature approximately in that order. --- # 5. Track A — Foundations and Measurement Model ## Goal Establish the minimum common model required to conduct repeatable experiments. This phase should avoid premature complexity. The initial goal is not to define the final ontology. It is to define enough structure to collect comparable evidence. --- ## A1. Define the Experience Observation Create the first machine-readable schema for: ```text ExperienceObservation ``` Minimum fields should include: ```text observation_id actor actor_profile system interface_surface task goal environment constraints starting_state interaction_events outcome measurements grader_version workload_version timestamp ``` --- ## A2. Define the Interaction Event Model An interaction trace should normalize different surfaces sufficiently to support cross-interface analysis. Candidate event types: ```text observe discover navigate read invoke write submit wait receive verify retry undo recover escalate abort ``` Surface-specific detail should remain available as attached evidence. --- ## A3. Define Workload Schema A workload should specify: ```text workload_id description goal initial_state actor_requirements allowed_surfaces constraints success_conditions failure_conditions safety_conditions expected_side_effects optional_reference_path ``` The reference path must not become the only acceptable path. --- ## A4. Define Outcome Categories Candidate outcome classifications: ```text success partial_success recoverable_failure unrecoverable_failure unsafe_success aborted indeterminate ``` --- ## A5. Define Metric Families Initial metric families should include: ### Outcome ```text task_success goal_completion constraint_compliance ``` ### Effort ```text interaction_count active_time wall_time context_switches ``` ### Failure ```text errors retries backtracks ``` ### Uncertainty ```text ambiguous_decisions exploratory_actions semantic_repairs ``` ### Recovery ```text recovery_success recovery_cost ``` ### Cost For agents: ```text tokens tool_calls inference_rounds compute_cost ``` For humans: ```text time clicks keystrokes attention shifts ``` --- ## A6. Establish Data Storage Store: ```text raw trace normalized observation metrics grading result diagnosis recommendation evaluation metadata ``` All derived results should remain traceable to their source evidence. --- ## Deliverable ```text concepts/ExperienceObservationModel.md concepts/WorkloadModel.md schemas/experience-observation.schema.json schemas/workload.schema.json ``` --- # 6. Track B — HMIX Research ## Goal Determine which measurements meaningfully characterize human interaction quality. The first phase should focus on practical usability rather than attempting to model the complete human experience. --- # B1. Establish HMIX Dimensions Begin with provisional dimensions: ```text discoverability comprehensibility action clarity predictability feedback efficiency cognitive load error prevention error legibility recoverability learnability trust calibration control accessibility ``` Each dimension should define: ```text meaning observable indicators possible measurements known confounders evaluation confidence ``` --- # B2. Select Initial Human Workloads Choose small workflows with objectively verifiable outcomes. Suggested classes: ```text account creation search and retrieval configuration data entry approval error recovery administrative operation ``` Avoid aesthetic-heavy evaluations initially. --- # B3. Compare Expert Review With Measured Interaction For each workload collect: ```text expert UX review human interaction trace user-reported difficulty task outcome quantitative metrics ``` Investigate where these disagree. --- # B4. Synthetic HMIX Evaluation Test whether multimodal AI evaluators can identify interface friction before human testing. Compare: ```text synthetic prediction vs human observation ``` Synthetic grading should initially be treated as a hypothesis generator rather than ground truth. --- # B5. Human Variance Repeat identical workloads with different actor profiles. Candidate profiles: ```text novice occasional user expert user domain expert technical administrator ``` Measure how much interface quality depends on expertise. --- # B6. Learnability Experiments Repeat tasks over multiple sessions. Measure: ```text time reduction interaction reduction error reduction memory dependency ``` Determine which interfaces reward learning. --- ## Deliverables ```text hmix/HMIXDimensions.md hmix/HMIXMeasurementProtocol.md hmix/workloads/ hmix/experiments/ ``` --- # 7. Track C — AMIX Research ## Goal Develop repeatable methods for measuring interface quality for autonomous and semi-autonomous agents. This is expected to be the most novel part of the program. --- # C1. Establish AMIX Dimensions Begin with: ```text capability discoverability semantic legibility contract precision invocability composability state legibility observability verifiability error legibility recoverability permission legibility idempotency determinism context efficiency interaction efficiency automation depth distillability ``` --- # C2. Define Agent Profiles AMIX results should always identify the agent configuration. Example: ```text model model version reasoning mode context limit tool-use capabilities memory available documentation initial prompt execution constraints ``` --- # C3. Establish Model Diversity Experiments should not rely exclusively on frontier models. Use at least three broad capability classes: ```text frontier model mid-tier general model small/local model ``` The purpose is to distinguish: ```text interface quality from model capability ``` --- # C4. Semantic Repair Measurement Develop a precise taxonomy for semantic repair. Candidate categories: ```text documentation lookup schema reinterpretation trial invocation source inspection parameter guessing state probing error reinterpretation human clarification ``` Measure: ```text repair count repair duration repair tokens repair success ``` --- # C5. Agent Error Recovery Deliberately induce: ```text invalid inputs stale state permission failures timeouts partial success ambiguous responses duplicate invocation risk ``` Observe whether agents can safely recover. --- # C6. Distillability After successful exploratory completion, attempt to transform the interaction into: ```text deterministic script workflow tool procedure compiled plan ``` Measure: ```text success after distillation cost reduction robustness remaining agent reasoning requirement ``` --- # C7. Permission Legibility Test whether an agent can distinguish: ```text can may must must not ``` without unsafe experimentation. --- # C8. Interface Family Comparison Run comparable tasks across: ```text REST API CLI MCP GraphQL web UI SDK ``` The goal is not to rank interface families globally. Instead investigate which characteristics affect AMIX independent of protocol. --- ## Deliverables ```text amix/AMIXDimensions.md amix/AgentProfileModel.md amix/SemanticRepairTaxonomy.md amix/workloads/ amix/experiments/ ``` --- # 8. Track D — Benchmarking and Calibration ## Goal Determine whether measurements are reproducible and useful enough to support grading. --- # D1. Build Initial Benchmark Corpus Select approximately: ```text 5 systems × 5 representative workloads × multiple actor profiles ``` Prefer systems that differ meaningfully in interface quality. --- # D2. Repeatability Repeat identical evaluations. Measure: ```text outcome variance metric variance grade variance ``` --- # D3. Evaluator Variance Compare: ```text human evaluator A human evaluator B agent evaluator A agent evaluator B ``` Determine which dimensions are objective enough to automate. --- # D4. Sensitivity Testing Introduce controlled interface modifications. Examples: ```text remove field descriptions improve error structure add explicit state endpoint rename ambiguous operation remove confirmation feedback increase navigation depth ``` Test whether grading responds as expected. --- # D5. Calibration Anchors Create small reference examples representing: ```text excellent adequate problematic severely problematic ``` These should be evidence-backed exemplars, not subjective labels alone. --- # D6. Correlation Analysis Investigate relationships among metrics. Examples: ```text semantic repairs ↔ task failure tool calls ↔ token cost human backtracking ↔ perceived difficulty state observability ↔ recovery success ``` Avoid assuming causal relationships prematurely. --- ## Deliverables ```text benchmarks/ calibration/ analysis/ ``` --- # 9. Track E — Diagnosis and Optimization ## Goal Move beyond grades toward actionable interface improvement. --- # E1. Experience Defect Taxonomy Create a catalog of recurring interface defects. Candidate categories: ```text hidden capability ambiguous operation implicit state weak feedback unstructured failure irreversible action identifier ambiguity permission opacity unsafe retry excessive navigation context dependency ``` --- # E2. Map Measurements to Diagnoses Example: ```text Observation: Repeated resource searches Metric: high discovery cost Likely diagnosis: resource identifiers not propagated Possible improvement: return canonical resource references ``` --- # E3. Pattern Library For each successful improvement record: ```text problem pattern affected actors proposed intervention before metrics after metrics tradeoffs confidence ``` --- # E4. Recommendation Evaluation Recommendations should be evaluated experimentally. Required loop: ```text baseline → proposed change → implementation → repeated workload → comparison ``` --- # E5. Multi-objective Optimization Optimization should account for: ```text success efficiency safety recoverability expressiveness cost ``` Avoid optimization toward a single metric. --- # E6. Cross-Actor Tradeoffs Explicitly test whether a change: ```text improves HMIX degrades AMIX or improves AMIX degrades HMIX ``` Record these tradeoffs. --- ## Deliverables ```text diagnostics/ExperienceDefectTaxonomy.md patterns/ optimizer/ ``` --- # 10. Track F — Self-Improving Grading ## Goal Allow the service to improve its own evaluation methodology while maintaining auditability. --- # F1. Version Everything Version: ```text workloads actor profiles rubrics metric definitions grader prompts models benchmark corpora ``` A grade should always be reproducible against a historical version. --- # F2. Grader Evaluation The grader itself should be benchmarked. Questions include: ```text Does it identify known defects? Does it detect controlled regressions? Does it overreact to irrelevant differences? Does it agree with human evidence? Does it generalize across interface families? ``` --- # F3. Workload Evolution Detect gaps in the benchmark corpus from: ```text production failures support incidents new interface types optimizer failures unexplained grading disagreement ``` Add workloads accordingly. --- # F4. Metric Evolution A metric should be retained only if it provides explanatory or predictive value. Track: ```text usefulness stability cost redundancy ``` --- # F5. Diagnostic Learning Accumulate evidence about which diagnoses are associated with successful improvements. Over time: ```text defect → intervention → observed outcome ``` should become an empirical knowledge base. --- # F6. Confidence Model Every assessment should eventually express: ```text score confidence evidence quantity evidence diversity evaluator agreement ``` --- # 11. Initial Experimental Systems The first research cycle should deliberately use small, inspectable targets. Recommended target set: ## System 1 — Simple REST API Focus: ```text discoverability schema clarity errors idempotency ``` ## System 2 — CLI Tool Focus: ```text discoverability help quality feedback recovery ``` ## System 3 — MCP Service Focus: ```text agent discovery tool descriptions semantic legibility composition ``` ## System 4 — Human Web Application Focus: ```text navigation feedback mental model error recovery ``` ## System 5 — Administrative Interface Focus: ```text high-impact operations permission clarity state visibility reversibility ``` --- # 12. Minimum Viable Experiment The first end-to-end experiment should be intentionally modest. Use: ```text 1 REST API 1 human web workflow 1 MCP service ``` For each: ```text 3 workloads 3 repetitions at least 2 actor profiles ``` Capture: ```text full traces task outcomes time interaction count errors retries semantic repairs where applicable qualitative observations ``` Produce: ```text dimension assessment diagnosis recommendations confidence ``` The immediate question is: > Does the resulting report tell us something useful that ordinary testing did not? If not, improve the research model before scaling. --- # 13. Milestone Program ## M0 — Research Scaffold Establish: ```text INTENT.md ResearchProgram.md terminology research structure experiment structure ``` Exit condition: The program can describe how an observation will be captured. --- ## M1 — Observation Kernel Implement: ```text ExperienceObservation Workload ActorProfile InteractionTrace ``` Exit condition: One workload can be recorded end-to-end. --- ## M2 — First AMIX Experiment Evaluate one API or MCP service. Exit condition: A repeatable AMIX report can be generated from evidence. --- ## M3 — First HMIX Experiment Evaluate one human workflow. Exit condition: Human interaction evidence can be represented in the same observation framework. --- ## M4 — Mixed Benchmark Corpus Create first cross-interface benchmark suite. Exit condition: At least five systems and approximately twenty workloads exist. --- ## M5 — Grading v0.1 Introduce initial dimension grading. Exit condition: Grades are decomposable, evidence-backed, and reproducible. --- ## M6 — Diagnosis v0.1 Create first experience defect taxonomy. Exit condition: Observed problems can be mapped to reusable diagnostic categories. --- ## M7 — Optimization Experiment Modify an interface based on grader findings. Exit condition: A measurable before/after improvement is demonstrated. --- ## M8 — Regression Service Run benchmark workloads repeatedly across interface versions. Exit condition: An experience regression can be detected automatically. --- ## M9 — Self-Improving Grader Use accumulated experiments to improve rubric, workloads, and diagnostics. Exit condition: A new grader version demonstrably outperforms an older version on the benchmark corpus. --- # 14. Research Artifacts The repository should accumulate four major forms of research output. ```text concepts/ experiments/ benchmarks/ patterns/ ``` Possible initial structure: ```text xperience-grader/ │ ├── INTENT.md ├── ResearchProgram.md │ ├── concepts/ │ ├── ExperienceObservationModel.md │ ├── WorkloadModel.md │ ├── ActorProfileModel.md │ └── GradingModel.md │ ├── hmix/ │ ├── HMIXDimensions.md │ ├── workloads/ │ └── experiments/ │ ├── amix/ │ ├── AMIXDimensions.md │ ├── SemanticRepairTaxonomy.md │ ├── workloads/ │ └── experiments/ │ ├── benchmarks/ │ ├── diagnostics/ │ ├── patterns/ │ ├── schemas/ │ └── research/ ``` --- # 15. Research Risks ## Premature Scoring Creating attractive scores before establishing measurement validity. Mitigation: Keep dimensions and evidence visible. --- ## Model Bias AMIX results may become model-specific. Mitigation: Use multiple agent classes and explicitly encode actor profiles. --- ## Synthetic UX Illusion AI-generated HMIX reviews may appear more reliable than they are. Mitigation: Calibrate against real human studies. --- ## Benchmark Gaming Interfaces could be optimized to the benchmark rather than actual usability. Mitigation: Maintain hidden and evolving workloads. --- ## Measurement Distortion Metrics such as clicks or token counts can become optimization targets despite weak relationship to quality. Mitigation: Use multi-objective evaluation. --- ## Rubric Drift Self-improvement could make historical grades incomparable. Mitigation: Version methodologies and retain old graders. --- ## Observer Effects Instrumentation can alter interaction behavior. Mitigation: Measure instrumentation overhead. --- # 16. Success Criteria The research program becomes practically valuable when it can demonstrate all of the following: ```text repeatable workload execution evidence-backed grading meaningful distinction between interface variants useful diagnosis measurable improvement after intervention experience regression detection cross-model AMIX comparison cross-user HMIX comparison historically reproducible results ``` The strongest validation will be: > An interface change proposed from xperience-grader evidence produces a repeatable improvement for the target actors and workloads without unacceptable degradation elsewhere. --- # 17. Long-Term Research Questions The mature program should explore deeper questions. ## Interface Universals Are there interface properties that consistently improve interaction across human and artificial actors? --- ## Intelligence-Relative Interfaces Should interfaces adapt dynamically to actor capability? --- ## Experience Compilation Can exploratory interaction be progressively compiled into deterministic procedure? --- ## Experience Debt Can poor interface quality accumulate similarly to technical debt? --- ## Interface Fitness Landscapes Can interface variants be explored as an optimization landscape? --- ## Adaptive Interfaces Should interfaces themselves evolve based on observed interaction difficulty? --- ## Experience Economics What is the economic cost of interface friction? For agents: ```text tokens compute latency human escalation ``` For humans: ```text time training support errors frustration ``` --- ## Mixed Human–Agent Experience How should interfaces be evaluated when work continuously moves between humans and agents? This may ultimately require a third perspective beyond HMIX and AMIX: ```text collaborative MIX ``` --- # 18. Research North Star The program should continuously return to one question: > Can we turn interface experience from an opinion into an observable, testable, improvable system property? If the answer becomes yes, `xperience-grader` should evolve beyond a research tool into infrastructure for continuous experience engineering. The target loop is: ```text experience ↓ evidence ↓ understanding ↓ change ↓ better experience ↓ new evidence └──────────────→ ``` The research program exists to determine how much of that loop can be made reliable.