diff --git a/INTENT.md b/INTENT.md new file mode 100644 index 0000000..20df6da --- /dev/null +++ b/INTENT.md @@ -0,0 +1,1119 @@ +# xperience-grader + +## Intent + +`xperience-grader` is a research and engineering project for building a **self-improving experience evaluation service for machine interfaces**. + +Its purpose is to inspect, exercise, measure, explain, and ultimately help optimize interfaces through which humans and artificial agents interact with software systems. + +The project treats interface quality not primarily as a property of visual design, API syntax, protocol choice, or implementation quality, but as a property of the **interaction experience between an acting intelligence and a machine system while pursuing a goal**. + +`xperience-grader` therefore asks: + +> How well does this interface enable an actor to understand the system, pursue an intended goal, act safely and efficiently, observe consequences, recover from problems, and improve through repeated interaction? + +The initial project distinguishes two principal perspectives: + +- **HMIX — Human–Machine Interface Experience** +- **AMIX — Agent–Machine Interface Experience** + +These perspectives share a common conceptual and measurement framework while retaining different actor models, constraints, failure modes, and quality criteria. + +The long-term ambition is to establish `xperience-grader` as an **independent, continuously improving grading and diagnostic service** that can be engaged against APIs, services, applications, command-line tools, agent tools, workflows, protocols, graphical interfaces, administrative surfaces, data interfaces, and combinations of these. + +The project should progress from evaluation toward **closed-loop interface optimization**, while keeping grading and optimization conceptually and operationally separate. + +--- + +# 1. Mission + +Build a system that can answer, with evidence: + +> What is it like for this kind of actor to use this machine interface for this task? + +And subsequently: + +> What changes would measurably improve that experience without degrading important capabilities, safety properties, or other actor experiences? + +The system should become increasingly capable of: + +1. discovering an interface, +2. understanding its available capabilities, +3. deriving or accepting representative tasks, +4. interacting with the system, +5. observing complete interaction traces, +6. measuring relevant properties, +7. diagnosing experience deficiencies, +8. explaining those deficiencies, +9. proposing improvements, +10. testing modified interfaces, +11. learning from accumulated evaluations, +12. improving its own grading methodology. + +`xperience-grader` should itself become an adaptive measurement instrument. + +--- + +# 2. Scope + +The project is concerned with **interaction surfaces between intelligences and computational systems**. + +These include, but are not limited to: + +- web applications, +- desktop applications, +- mobile applications, +- administrative interfaces, +- command-line interfaces, +- APIs, +- REST interfaces, +- GraphQL, +- RPC interfaces, +- MCP servers and tools, +- SDKs, +- libraries, +- workflow systems, +- messaging interfaces, +- event interfaces, +- data access surfaces, +- configuration interfaces, +- conversational interfaces, +- autonomous-agent tools, +- multi-step service interactions, +- compound interfaces spanning several systems. + +An evaluated experience may cross interface families. + +For example: + +```text +human + → web UI + → asynchronous job + → notification + → administration interface +``` + +or: + +```text +agent + → MCP discovery + → API call + → event stream + → object storage + → verification endpoint +``` + +The unit of concern is therefore not necessarily a screen, endpoint, API, or tool. + +It is an **experience path through a system in pursuit of a goal**. + +--- + +# 3. Foundational Model + +The project starts from a general interaction loop: + +```text +Goal + ↓ +Discover + ↓ +Interpret + ↓ +Plan + ↓ +Act + ↓ +Observe + ↓ +Verify + ↓ +Recover + ↓ +Learn + └──────────────→ future interaction +``` + +Different interfaces support different portions of this loop explicitly. + +Weak interfaces force actors to reconstruct missing parts through experimentation, inference, documentation search, external knowledge, or human intervention. + +A strong interface reduces unnecessary uncertainty while preserving expressive power. + +The project therefore treats experience quality as a multidimensional phenomenon. + +A preliminary abstraction is: + +```text +ExperienceQuality = + f( + goal_success, + effort, + latency, + uncertainty, + recoverability, + learnability, + robustness, + safety, + expressive_power, + adaptability + ) +``` + +No single scalar score should be considered sufficient to represent the complete interaction experience. + +Scores are projections over richer observations. + +--- + +# 4. Experience Observation as the Primitive + +The fundamental data object of `xperience-grader` should not be an interface score. + +It should be an **experience observation**. + +Conceptually: + +```text +ExperienceObservation + actor + actor_profile + system + interface_surface + environment + goal + task + constraints + starting_state + interaction_trace + outcome + measurements + incidents + uncertainties + evidence +``` + +A grade is derived from one or more observations. + +Therefore: + +```text +grade(interface) +``` + +is considered incomplete. + +The more meaningful form is: + +```text +grade( + interface, + actor_profile, + task, + environment, + constraints +) +``` + +Aggregated grades may subsequently be computed across workloads, actor populations, or operational environments. + +--- + +# 5. Human–Machine Interface Experience — HMIX + +HMIX evaluates interfaces from the perspective of human actors. + +It is broader than visual design or conventional usability. + +Relevant dimensions may include: + +## Discoverability + +Can the person determine what functionality exists and where to find it? + +## Comprehensibility + +Can the person form an adequate mental model of the system and its current state? + +## Action Clarity + +Is it apparent what actions are available and what they mean? + +## Predictability + +Can the user reasonably anticipate the consequences of an action? + +## Feedback + +Does the system make progress, state transitions, success, and failure visible? + +## Efficiency + +How much time, attention, navigation, repetition, and manual work are required? + +## Cognitive Load + +How much state must the user remember, infer, translate, or reconstruct? + +## Error Prevention + +Does the system make unintended actions unnecessarily easy? + +## Error Legibility + +Can users understand what went wrong? + +## Recoverability + +Can mistakes and exceptional states be repaired safely and economically? + +## Learnability + +Does repeated interaction result in increasing competence and reduced effort? + +## Accessibility + +Can users with different capabilities interact effectively with the system? + +## Trust Calibration + +Can the user correctly judge what the machine knows, has done, intends to do, and guarantees? + +## Control + +Can the user meaningfully intervene, inspect, reverse, or redirect system behavior where necessary? + +HMIX should eventually incorporate both empirical human measurements and synthetic evaluations by capable multimodal models. + +Synthetic evaluation must not be mistaken for human evidence. + +--- + +# 6. Agent–Machine Interface Experience — AMIX + +AMIX evaluates interfaces from the perspective of artificial agents. + +Agent experience quality is not equivalent to API quality. + +Agents interact through APIs, command-line programs, MCP tools, browser interfaces, documents, files, databases, event streams, workflows, messages, or mixed surfaces. + +Relevant dimensions may include: + +## Capability Discoverability + +Can the agent determine which relevant capabilities exist? + +## Semantic Legibility + +Can the agent determine what operations, fields, states, and objects mean? + +## Contract Precision + +Are expectations represented sufficiently precisely for reliable execution? + +## Invocability + +Can the agent reliably formulate valid interactions? + +## Composability + +Can operations be safely incorporated into larger plans? + +## State Legibility + +Can relevant machine state be inspected rather than inferred? + +## Observability + +Can the agent determine what happened after an action? + +## Verifiability + +Can the agent independently determine whether the intended outcome was achieved? + +## Error Legibility + +Are failures represented in a form that permits machine reasoning and corrective action? + +## Recoverability + +Can the agent identify a safe next action after failure? + +## Permission Legibility + +Can the agent understand what it can, may, must, and must not do? + +## Idempotency and Replay Safety + +Can uncertain operations be retried without unintended duplication or corruption? + +## Determinism + +How reliably do equivalent interactions produce operationally equivalent results? + +## Context Efficiency + +How much context, documentation, token consumption, and state reconstruction are required? + +## Interaction Efficiency + +How many tool calls, retries, planning rounds, or inference cycles are required? + +## Automation Depth + +How far can a task proceed without human semantic repair or intervention? + +## Distillability + +Can successful exploratory behavior later be converted into deterministic, cheaper, reusable interaction procedures? + +--- + +# 7. Semantic Repair + +A central AMIX research concept is **semantic repair**. + +Semantic repair occurs when an agent cannot continue reliable execution based on the interface contract and available machine-readable state alone. + +Examples include: + +- searching external documentation, +- inspecting implementation source code, +- experimentally probing parameter semantics, +- interpreting ambiguous error messages, +- guessing undocumented state transitions, +- asking a human what an operation means, +- retrying alternative formats, +- reverse engineering implicit conventions. + +Possible measures include: + +```text +semantic_repairs / successful_task + +semantic_repair_tokens + +semantic_repair_latency + +semantic_repair_tool_calls + +human_semantic_escalations +``` + +Semantic repair rate may become one of the strongest indicators of poor agent-facing interface quality. + +--- + +# 8. Workloads + +Interfaces should be evaluated against **workloads**, not isolated operations. + +A workload describes representative goals actors attempt to accomplish. + +Example: + +```text +Create a customer account, +assign billing information, +activate a subscription, +verify successful provisioning, +and recover correctly if payment authorization fails. +``` + +A workload may define: + +```text +goal +actor profile +initial conditions +allowed interfaces +constraints +success conditions +safety conditions +acceptable side effects +evaluation dimensions +``` + +Workloads may originate from: + +- product use cases, +- documentation, +- support incidents, +- production telemetry, +- human experts, +- agent-generated exploration, +- interface schemas, +- acceptance tests, +- historical traces. + +A mature `xperience-grader` should maintain evolving workload libraries. + +--- + +# 9. Evidence + +Every grade should be evidence-backed. + +Possible evidence includes: + +- interaction traces, +- screenshots, +- video recordings, +- DOM state, +- accessibility trees, +- API requests and responses, +- schemas, +- tool definitions, +- logs, +- traces, +- events, +- timing measurements, +- token usage, +- tool-call counts, +- retries, +- error states, +- user feedback, +- human review, +- agent explanations, +- task outcomes. + +Grades should preserve provenance. + +A future observer should be able to ask: + +> Why did this interface receive this grade? + +and inspect the underlying evidence. + +--- + +# 10. Measurement Before Judgment + +The project should distinguish: + +```text +observation +measurement +interpretation +grade +diagnosis +recommendation +``` + +These are not equivalent. + +For example: + +```text +Observation: +Agent performed 14 tool calls. + +Measurement: +Median comparable workload requires 6 calls. + +Interpretation: +Interaction requires unusually high exploratory effort. + +Diagnosis: +Resource identifiers are not discoverable from preceding responses. + +Recommendation: +Expose canonical resource references in mutation responses. +``` + +Maintaining this separation is essential for auditability and methodological improvement. + +--- + +# 11. Grading + +The project may provide grades for communication and comparison, but grading should never destroy the underlying multidimensional evidence. + +A possible hierarchy is: + +```text +raw observations + ↓ +metrics + ↓ +dimension assessments + ↓ +task grades + ↓ +workload grades + ↓ +interface profile + ↓ +aggregate grade +``` + +An aggregate grade should therefore remain decomposable. + +The project should strongly prefer statements such as: + +```text +Excellent discoverability +Moderate semantic ambiguity +Poor recovery behavior +Very high execution efficiency after successful discovery +``` + +over unexplained scalar judgments. + +--- + +# 12. Reference Actors + +Evaluation requires explicit actor models. + +HMIX reference actors might vary by: + +- domain experience, +- application experience, +- language, +- accessibility requirements, +- frequency of use, +- technical expertise, +- cognitive workload. + +AMIX reference actors might vary by: + +- model, +- model capability class, +- context size, +- tool-use ability, +- planning strategy, +- memory configuration, +- available documentation, +- autonomy level, +- token budget. + +An interface can therefore legitimately have: + +```text +high AMIX for frontier coding agents + +moderate AMIX for small local agents + +poor AMIX for deterministic automation +``` + +without contradiction. + +--- + +# 13. Capability Preservation + +Optimization must not reduce the problem to minimizing clicks, tokens, calls, or latency. + +An interface may become cheaper while losing important expressive power. + +Every optimization should therefore consider capability preservation. + +Relevant dimensions include: + +```text +goal success +expressive power +robustness +safety +recoverability +efficiency +latency +cost +learnability +``` + +Optimization should seek Pareto improvements where possible. + +A change that improves one dimension while materially degrading another should be visible as a tradeoff, not silently classified as an improvement. + +--- + +# 14. The Grader and the Optimizer + +The project should maintain a strict conceptual boundary between grading and optimization. + +## Grader + +The grader asks: + +> Given this evidence, what can we conclude about the quality of this experience? + +The grader should be conservative, repeatable, explainable, and evidence-oriented. + +## Optimizer + +The optimizer asks: + +> Given these deficiencies and constraints, what changes might improve the experience? + +The optimizer may generate: + +- interface redesigns, +- schema changes, +- documentation changes, +- error model improvements, +- interaction shortcuts, +- new endpoints, +- richer machine-readable state, +- UI modifications, +- workflow restructuring, +- protocol changes. + +Proposed changes should be evaluated by running workloads again. + +The basic optimization loop becomes: + +```text +interface + ↓ +workload execution + ↓ +observation + ↓ +grading + ↓ +diagnosis + ↓ +candidate changes + ↓ +modified interface + ↓ +workload execution + └──────────────→ repeat +``` + +--- + +# 15. Self-Improvement + +`xperience-grader` should itself improve through accumulated experience. + +Self-improvement may occur at several levels. + +## Better Workloads + +Observed production behavior reveals missing or unrealistic evaluation scenarios. + +## Better Measurements + +Repeated evaluation reveals more predictive metrics. + +## Better Rubrics + +Some dimensions may prove redundant, weak, or incorrectly weighted. + +## Better Reference Baselines + +Accumulated observations establish empirical comparison distributions. + +## Better Diagnoses + +Recurring interaction failures can be classified into reusable patterns. + +## Better Recommendations + +Interface modifications can be evaluated against subsequent measurements. + +## Better Evaluators + +Different models, tools, simulations, and human reviewers can be compared for grading reliability. + +## Better Methodology + +The project should continuously challenge whether its own grading correlates with actual experience outcomes. + +Self-improvement must not mean uncontrolled rubric drift. + +Versions of methodologies, graders, benchmark sets, and reference actors should remain identifiable and reproducible. + +--- + +# 16. The Experience Pattern Library + +A long-term output of accumulated grading should be a catalog of interface patterns. + +Examples may include: + +```text +Explicit State Pattern +Structured Failure Pattern +Discoverable Capability Pattern +Safe Retry Pattern +Progressive Disclosure Pattern +Canonical Identifier Pattern +Verifiable Mutation Pattern +Reversible Action Pattern +Inspectable Permission Pattern +Machine-Readable Enumeration Pattern +``` + +Each pattern should accumulate evidence across HMIX and AMIX workloads. + +Patterns may therefore eventually be classified as: + +```text +human-beneficial +agent-beneficial +universally-beneficial +human-agent tradeoff +context-dependent +``` + +This may form the basis of a broader **MIX Interface Design Canon**. + +--- + +# 17. Universal Interface Properties + +One of the deeper research questions of the project is whether certain properties systematically improve interaction quality for both human and artificial intelligence. + +Candidate properties include: + +- explicit state, +- stable concepts, +- clear feedback, +- predictable semantics, +- reversible operations, +- visible consequences, +- inspectability, +- structured errors, +- progressive disclosure, +- meaningful defaults, +- deterministic identifiers, +- discoverable capabilities. + +If such properties can be empirically demonstrated across actor classes, they may represent more fundamental principles of machine-interface design. + +--- + +# 18. Service Model + +The intended operational form of `xperience-grader` is a service. + +Conceptually: + +```text +submit target + ↓ +identify interface surfaces + ↓ +select actor profiles + ↓ +derive/select workloads + ↓ +execute evaluation + ↓ +collect evidence + ↓ +calculate measurements + ↓ +grade dimensions + ↓ +generate diagnosis + ↓ +produce improvement candidates +``` + +A user might eventually request: + +```text +grade this API for autonomous agents +``` + +or: + +```text +review this application onboarding flow for first-time human users +``` + +or: + +```text +compare the HMIX and AMIX consequences of these two interface designs +``` + +or: + +```text +continuously grade every release and detect experience regressions +``` + +--- + +# 19. Continuous Experience Testing + +A mature implementation should integrate naturally into engineering workflows. + +Possible execution modes include: + +```text +ad hoc review +pull-request evaluation +release qualification +nightly regression testing +production telemetry assessment +continuous interface optimization +``` + +This transforms experience quality from an occasional design-review activity into an observable engineering property. + +A future pipeline might be: + +```text +commit + ↓ +build + ↓ +functional tests + ↓ +security tests + ↓ +HMIX workloads + ↓ +AMIX workloads + ↓ +experience regression gate + ↓ +deployment +``` + +--- + +# 20. Relationship to Existing Practices + +`xperience-grader` should reuse rather than replace mature disciplines. + +Relevant adjacent practices include: + +- usability testing, +- accessibility testing, +- UX research, +- human-computer interaction, +- API design, +- API linting, +- contract testing, +- integration testing, +- agent benchmarking, +- software observability, +- distributed tracing, +- reliability engineering, +- performance testing, +- security testing, +- formal interface specifications. + +The distinctive contribution of `xperience-grader` is to combine these into an **actor-centered empirical model of interface experience**. + +--- + +# 21. Architecture Direction + +An initial conceptual architecture may include: + +```text +xperience-grader +│ +├── core +│ ├── observation-model +│ ├── workload-model +│ ├── evidence-model +│ ├── grading-model +│ └── actor-model +│ +├── hmix +│ ├── graders +│ ├── actors +│ ├── workloads +│ └── metrics +│ +├── amix +│ ├── graders +│ ├── actors +│ ├── workloads +│ └── metrics +│ +├── runners +│ ├── browser +│ ├── api +│ ├── cli +│ ├── mcp +│ └── workflow +│ +├── evidence +│ +├── benchmarks +│ +├── diagnostics +│ +├── patterns +│ +└── optimizer +``` + +This structure is provisional. + +The repository should evolve from evidence rather than architecture aesthetics. + +--- + +# 22. Initial Research Questions + +The first research phase should answer at least the following: + +1. What dimensions best predict perceived and actual interaction quality? +2. Which dimensions can be measured objectively? +3. Which require interpretation? +4. How stable are grades across different evaluators? +5. How dependent is AMIX on the chosen model? +6. How dependent is HMIX on expertise and familiarity? +7. Which interface characteristics benefit both humans and agents? +8. Which characteristics create human-agent tradeoffs? +9. How should multi-interface workflows be graded? +10. How should uncertainty be represented? +11. How should failed tasks affect grades? +12. How should safety restrictions interact with efficiency measurements? +13. Can semantic repair be measured reliably? +14. Can experience regressions be detected automatically? +15. Can successful exploratory agent interactions be distilled into deterministic procedures? +16. Can optimizer-generated interface changes produce reproducible improvements? +17. How should grading methodology itself be evaluated? + +--- + +# 23. Initial Experimental Program + +Early experiments should remain deliberately small. + +Candidate targets should include: + +- one conventional REST API, +- one MCP service, +- one CLI, +- one administrative web interface, +- one end-user web workflow. + +Each target should receive: + +```text +3–10 representative workloads +multiple observations per workload +explicit actor profiles +trace capture +initial measurements +human or expert review +diagnostic report +``` + +The goal is not initially to establish a universal score. + +The goal is to discover whether the proposed measurement model produces useful, reproducible distinctions. + +--- + +# 24. Non-Goals + +The project is not intended to become: + +- a generic visual-design rating service, +- an aesthetic preference engine, +- an API style linter, +- a replacement for accessibility standards, +- a replacement for functional tests, +- a model benchmark leaderboard, +- a single universal experience number, +- an optimizer that silently modifies production systems. + +These may become inputs or integrations, but they are not the defining purpose. + +--- + +# 25. Guiding Principles + +## Experience Is Relational + +There is no interface quality independent of actor, goal, context, and constraints. + +## Evidence Before Opinion + +Grades should be supported by observable interaction evidence. + +## Workloads Before Widgets + +Meaningful tasks matter more than isolated screens or endpoints. + +## Multidimensional Before Scalar + +A score is a projection, not reality. + +## Diagnose Before Optimize + +Changes should address identified interaction failures. + +## Preserve Capability + +Efficiency is not improvement if expressive power, safety, or robustness is lost. + +## Humans and Agents Are Different + +HMIX and AMIX should share structure without pretending their constraints are identical. + +## Learn from Interaction + +Every grading run should be capable of improving future evaluation. + +## Version the Method + +Self-improvement must remain reproducible and auditable. + +## Optimize the Coupling + +The deeper object of study is neither the actor nor the machine alone, but the quality of the interface coupling between them. + +--- + +# 26. Long-Term Vision + +`xperience-grader` should evolve into infrastructure for **experience engineering**. + +A sufficiently mature system would allow interface quality to become: + +- observable, +- measurable, +- comparable, +- testable, +- regression-controlled, +- diagnosable, +- experimentally optimizable. + +Software teams should eventually be able to ask not merely: + +```text +Does the interface work? +``` + +but: + +```text +How well can different forms of intelligence work through it? +``` + +The long-term research hypothesis is that good interfaces are not merely pleasant surfaces or convenient protocols. + +They are effective **coupling mechanisms between adaptive systems**. + +Humans adapt. + +Agents adapt. + +Machines change. + +Tasks evolve. + +A durable interface therefore cannot only encode commands and data. It must support the formation, execution, observation, correction, and refinement of purposeful interaction. + +`xperience-grader` exists to make that quality measurable. + +> **Measure the coupling. Explain the friction. Improve the experience.** diff --git a/ResearchProgram.md b/ResearchProgram.md new file mode 100644 index 0000000..cf56680 --- /dev/null +++ b/ResearchProgram.md @@ -0,0 +1,1616 @@ +# xperience-grader Research Program + +## Status + +Initial research program draft. + +This document defines the staged research agenda for `xperience-grader`, a self-improving experience grading and optimization service for human–machine and agent–machine interfaces. + +The program is intentionally empirical. + +Its purpose is not to invent a scoring framework in isolation, but to determine which observations, measurements, dimensions, and experimental methods are actually useful for characterizing interface experience. + +--- + +# 1. Research Objective + +The central research question is: + +> Can the quality of interaction between an intelligent actor and a machine interface be measured in a sufficiently useful, reproducible, and actionable way to support systematic interface improvement? + +The project investigates this question across two related domains: + +- **HMIX — Human–Machine Interface Experience** +- **AMIX — Agent–Machine Interface Experience** + +The project further investigates whether both can be grounded in a shared interaction model while preserving actor-specific measurements and constraints. + +The long-term goal is to establish a service capable of: + +```text +observe +→ measure +→ grade +→ diagnose +→ propose improvement +→ re-evaluate +→ learn +``` + +The research program therefore spans measurement science, human–computer interaction, agent evaluation, software interface design, observability, experimentation, and optimization. + +--- + +# 2. Core Research Hypotheses + +The initial program works from a set of explicit hypotheses. + +These hypotheses should be challenged experimentally. + +## H1 — Interface quality is relational + +Interface quality cannot be adequately represented without considering: + +```text +actor +task +environment +constraints +interface +``` + +A useful grade therefore requires an interaction context rather than only static inspection of the interface. + +--- + +## H2 — Task-level observations are more useful than interface-level opinions + +A workload executed against an interface produces stronger evidence than a generic assessment of whether an interface is "good". + +--- + +## H3 — Experience quality is multidimensional + +A single scalar score is insufficient. + +Useful evaluation requires multiple dimensions such as: + +```text +success +effort +uncertainty +latency +recoverability +learnability +safety +robustness +``` + +--- + +## H4 — Interaction traces contain measurable experience signals + +Metrics derived from real interaction traces can reveal meaningful interface friction. + +Examples include: + +```text +number of actions +retries +failed operations +context consumed +latency +semantic repairs +backtracking +human escalations +``` + +--- + +## H5 — Humans and agents share some interface-quality principles + +Certain interface properties may systematically benefit both humans and agents. + +Candidates include: + +```text +explicit state +clear semantics +structured feedback +predictable transitions +reversible actions +discoverable capabilities +``` + +--- + +## H6 — Humans and agents also have fundamentally different experience constraints + +HMIX and AMIX cannot be collapsed into one identical rubric. + +--- + +## H7 — Experience regressions can be detected automatically + +Changes to interfaces should produce measurable differences in workload performance. + +--- + +## H8 — Experience deficiencies can be diagnosed into reusable patterns + +Recurring interaction failures may be classifiable into common design deficiencies. + +--- + +## H9 — Interface modifications can produce experimentally demonstrable improvements + +Recommended changes should be testable through repeated workload execution. + +--- + +## H10 — The grading system can improve without losing reproducibility + +The system can learn better workloads, measurements, and diagnostic models while preserving versioned evaluation methodology. + +--- + +# 3. Research Principles + +The research program follows the principles below. + +## Evidence First + +Claims about experience quality should be grounded in observable evidence wherever possible. + +## Separate Observation From Judgment + +Maintain an explicit distinction between: + +```text +observation +measurement +interpretation +grade +diagnosis +recommendation +``` + +## Keep Raw Evidence + +Derived metrics must not replace underlying interaction traces. + +## Prefer Task Success Over Surface Preference + +An interface should primarily be evaluated by what actors can accomplish through it. + +## Compare Against Alternatives + +Absolute scores are less informative than controlled comparisons. + +## Preserve Actor Context + +Evaluation results should always identify the actor profile and environment used. + +## Expose Uncertainty + +A grade should be allowed to say: + +```text +insufficient evidence +high evaluator variance +unknown +``` + +## Optimize Conservatively + +Changes should preserve capability, safety, and recoverability. + +--- + +# 4. Research Architecture + +The research program is organized into six major tracks. + +```text +Track A — Foundations and Measurement Model +Track B — HMIX Evaluation +Track C — AMIX Evaluation +Track D — Benchmarking and Calibration +Track E — Diagnosis and Optimization +Track F — Self-Improvement and Continuous Grading +``` + +These tracks overlap, but should mature approximately in that order. + +--- + +# 5. Track A — Foundations and Measurement Model + +## Goal + +Establish the minimum common model required to conduct repeatable experiments. + +This phase should avoid premature complexity. + +The initial goal is not to define the final ontology. + +It is to define enough structure to collect comparable evidence. + +--- + +## A1. Define the Experience Observation + +Create the first machine-readable schema for: + +```text +ExperienceObservation +``` + +Minimum fields should include: + +```text +observation_id + +actor +actor_profile + +system +interface_surface + +task +goal + +environment +constraints + +starting_state + +interaction_events + +outcome + +measurements + +grader_version +workload_version +timestamp +``` + +--- + +## A2. Define the Interaction Event Model + +An interaction trace should normalize different surfaces sufficiently to support cross-interface analysis. + +Candidate event types: + +```text +observe +discover +navigate +read +invoke +write +submit +wait +receive +verify +retry +undo +recover +escalate +abort +``` + +Surface-specific detail should remain available as attached evidence. + +--- + +## A3. Define Workload Schema + +A workload should specify: + +```text +workload_id +description + +goal +initial_state + +actor_requirements +allowed_surfaces + +constraints + +success_conditions +failure_conditions +safety_conditions + +expected_side_effects + +optional_reference_path +``` + +The reference path must not become the only acceptable path. + +--- + +## A4. Define Outcome Categories + +Candidate outcome classifications: + +```text +success +partial_success +recoverable_failure +unrecoverable_failure +unsafe_success +aborted +indeterminate +``` + +--- + +## A5. Define Metric Families + +Initial metric families should include: + +### Outcome + +```text +task_success +goal_completion +constraint_compliance +``` + +### Effort + +```text +interaction_count +active_time +wall_time +context_switches +``` + +### Failure + +```text +errors +retries +backtracks +``` + +### Uncertainty + +```text +ambiguous_decisions +exploratory_actions +semantic_repairs +``` + +### Recovery + +```text +recovery_success +recovery_cost +``` + +### Cost + +For agents: + +```text +tokens +tool_calls +inference_rounds +compute_cost +``` + +For humans: + +```text +time +clicks +keystrokes +attention shifts +``` + +--- + +## A6. Establish Data Storage + +Store: + +```text +raw trace +normalized observation +metrics +grading result +diagnosis +recommendation +evaluation metadata +``` + +All derived results should remain traceable to their source evidence. + +--- + +## Deliverable + +```text +concepts/ExperienceObservationModel.md +concepts/WorkloadModel.md +schemas/experience-observation.schema.json +schemas/workload.schema.json +``` + +--- + +# 6. Track B — HMIX Research + +## Goal + +Determine which measurements meaningfully characterize human interaction quality. + +The first phase should focus on practical usability rather than attempting to model the complete human experience. + +--- + +# B1. Establish HMIX Dimensions + +Begin with provisional dimensions: + +```text +discoverability +comprehensibility +action clarity +predictability +feedback +efficiency +cognitive load +error prevention +error legibility +recoverability +learnability +trust calibration +control +accessibility +``` + +Each dimension should define: + +```text +meaning +observable indicators +possible measurements +known confounders +evaluation confidence +``` + +--- + +# B2. Select Initial Human Workloads + +Choose small workflows with objectively verifiable outcomes. + +Suggested classes: + +```text +account creation +search and retrieval +configuration +data entry +approval +error recovery +administrative operation +``` + +Avoid aesthetic-heavy evaluations initially. + +--- + +# B3. Compare Expert Review With Measured Interaction + +For each workload collect: + +```text +expert UX review +human interaction trace +user-reported difficulty +task outcome +quantitative metrics +``` + +Investigate where these disagree. + +--- + +# B4. Synthetic HMIX Evaluation + +Test whether multimodal AI evaluators can identify interface friction before human testing. + +Compare: + +```text +synthetic prediction +vs +human observation +``` + +Synthetic grading should initially be treated as a hypothesis generator rather than ground truth. + +--- + +# B5. Human Variance + +Repeat identical workloads with different actor profiles. + +Candidate profiles: + +```text +novice +occasional user +expert user +domain expert +technical administrator +``` + +Measure how much interface quality depends on expertise. + +--- + +# B6. Learnability Experiments + +Repeat tasks over multiple sessions. + +Measure: + +```text +time reduction +interaction reduction +error reduction +memory dependency +``` + +Determine which interfaces reward learning. + +--- + +## Deliverables + +```text +hmix/HMIXDimensions.md +hmix/HMIXMeasurementProtocol.md +hmix/workloads/ +hmix/experiments/ +``` + +--- + +# 7. Track C — AMIX Research + +## Goal + +Develop repeatable methods for measuring interface quality for autonomous and semi-autonomous agents. + +This is expected to be the most novel part of the program. + +--- + +# C1. Establish AMIX Dimensions + +Begin with: + +```text +capability discoverability +semantic legibility +contract precision +invocability +composability +state legibility +observability +verifiability +error legibility +recoverability +permission legibility +idempotency +determinism +context efficiency +interaction efficiency +automation depth +distillability +``` + +--- + +# C2. Define Agent Profiles + +AMIX results should always identify the agent configuration. + +Example: + +```text +model +model version +reasoning mode +context limit +tool-use capabilities +memory +available documentation +initial prompt +execution constraints +``` + +--- + +# C3. Establish Model Diversity + +Experiments should not rely exclusively on frontier models. + +Use at least three broad capability classes: + +```text +frontier model +mid-tier general model +small/local model +``` + +The purpose is to distinguish: + +```text +interface quality +from +model capability +``` + +--- + +# C4. Semantic Repair Measurement + +Develop a precise taxonomy for semantic repair. + +Candidate categories: + +```text +documentation lookup +schema reinterpretation +trial invocation +source inspection +parameter guessing +state probing +error reinterpretation +human clarification +``` + +Measure: + +```text +repair count +repair duration +repair tokens +repair success +``` + +--- + +# C5. Agent Error Recovery + +Deliberately induce: + +```text +invalid inputs +stale state +permission failures +timeouts +partial success +ambiguous responses +duplicate invocation risk +``` + +Observe whether agents can safely recover. + +--- + +# C6. Distillability + +After successful exploratory completion, attempt to transform the interaction into: + +```text +deterministic script +workflow +tool procedure +compiled plan +``` + +Measure: + +```text +success after distillation +cost reduction +robustness +remaining agent reasoning requirement +``` + +--- + +# C7. Permission Legibility + +Test whether an agent can distinguish: + +```text +can +may +must +must not +``` + +without unsafe experimentation. + +--- + +# C8. Interface Family Comparison + +Run comparable tasks across: + +```text +REST API +CLI +MCP +GraphQL +web UI +SDK +``` + +The goal is not to rank interface families globally. + +Instead investigate which characteristics affect AMIX independent of protocol. + +--- + +## Deliverables + +```text +amix/AMIXDimensions.md +amix/AgentProfileModel.md +amix/SemanticRepairTaxonomy.md +amix/workloads/ +amix/experiments/ +``` + +--- + +# 8. Track D — Benchmarking and Calibration + +## Goal + +Determine whether measurements are reproducible and useful enough to support grading. + +--- + +# D1. Build Initial Benchmark Corpus + +Select approximately: + +```text +5 systems +× +5 representative workloads +× +multiple actor profiles +``` + +Prefer systems that differ meaningfully in interface quality. + +--- + +# D2. Repeatability + +Repeat identical evaluations. + +Measure: + +```text +outcome variance +metric variance +grade variance +``` + +--- + +# D3. Evaluator Variance + +Compare: + +```text +human evaluator A +human evaluator B +agent evaluator A +agent evaluator B +``` + +Determine which dimensions are objective enough to automate. + +--- + +# D4. Sensitivity Testing + +Introduce controlled interface modifications. + +Examples: + +```text +remove field descriptions +improve error structure +add explicit state endpoint +rename ambiguous operation +remove confirmation feedback +increase navigation depth +``` + +Test whether grading responds as expected. + +--- + +# D5. Calibration Anchors + +Create small reference examples representing: + +```text +excellent +adequate +problematic +severely problematic +``` + +These should be evidence-backed exemplars, not subjective labels alone. + +--- + +# D6. Correlation Analysis + +Investigate relationships among metrics. + +Examples: + +```text +semantic repairs ↔ task failure + +tool calls ↔ token cost + +human backtracking ↔ perceived difficulty + +state observability ↔ recovery success +``` + +Avoid assuming causal relationships prematurely. + +--- + +## Deliverables + +```text +benchmarks/ +calibration/ +analysis/ +``` + +--- + +# 9. Track E — Diagnosis and Optimization + +## Goal + +Move beyond grades toward actionable interface improvement. + +--- + +# E1. Experience Defect Taxonomy + +Create a catalog of recurring interface defects. + +Candidate categories: + +```text +hidden capability +ambiguous operation +implicit state +weak feedback +unstructured failure +irreversible action +identifier ambiguity +permission opacity +unsafe retry +excessive navigation +context dependency +``` + +--- + +# E2. Map Measurements to Diagnoses + +Example: + +```text +Observation: +Repeated resource searches + +Metric: +high discovery cost + +Likely diagnosis: +resource identifiers not propagated + +Possible improvement: +return canonical resource references +``` + +--- + +# E3. Pattern Library + +For each successful improvement record: + +```text +problem pattern +affected actors +proposed intervention +before metrics +after metrics +tradeoffs +confidence +``` + +--- + +# E4. Recommendation Evaluation + +Recommendations should be evaluated experimentally. + +Required loop: + +```text +baseline +→ proposed change +→ implementation +→ repeated workload +→ comparison +``` + +--- + +# E5. Multi-objective Optimization + +Optimization should account for: + +```text +success +efficiency +safety +recoverability +expressiveness +cost +``` + +Avoid optimization toward a single metric. + +--- + +# E6. Cross-Actor Tradeoffs + +Explicitly test whether a change: + +```text +improves HMIX +degrades AMIX + +or + +improves AMIX +degrades HMIX +``` + +Record these tradeoffs. + +--- + +## Deliverables + +```text +diagnostics/ExperienceDefectTaxonomy.md +patterns/ +optimizer/ +``` + +--- + +# 10. Track F — Self-Improving Grading + +## Goal + +Allow the service to improve its own evaluation methodology while maintaining auditability. + +--- + +# F1. Version Everything + +Version: + +```text +workloads +actor profiles +rubrics +metric definitions +grader prompts +models +benchmark corpora +``` + +A grade should always be reproducible against a historical version. + +--- + +# F2. Grader Evaluation + +The grader itself should be benchmarked. + +Questions include: + +```text +Does it identify known defects? + +Does it detect controlled regressions? + +Does it overreact to irrelevant differences? + +Does it agree with human evidence? + +Does it generalize across interface families? +``` + +--- + +# F3. Workload Evolution + +Detect gaps in the benchmark corpus from: + +```text +production failures +support incidents +new interface types +optimizer failures +unexplained grading disagreement +``` + +Add workloads accordingly. + +--- + +# F4. Metric Evolution + +A metric should be retained only if it provides explanatory or predictive value. + +Track: + +```text +usefulness +stability +cost +redundancy +``` + +--- + +# F5. Diagnostic Learning + +Accumulate evidence about which diagnoses are associated with successful improvements. + +Over time: + +```text +defect +→ intervention +→ observed outcome +``` + +should become an empirical knowledge base. + +--- + +# F6. Confidence Model + +Every assessment should eventually express: + +```text +score +confidence +evidence quantity +evidence diversity +evaluator agreement +``` + +--- + +# 11. Initial Experimental Systems + +The first research cycle should deliberately use small, inspectable targets. + +Recommended target set: + +## System 1 — Simple REST API + +Focus: + +```text +discoverability +schema clarity +errors +idempotency +``` + +## System 2 — CLI Tool + +Focus: + +```text +discoverability +help quality +feedback +recovery +``` + +## System 3 — MCP Service + +Focus: + +```text +agent discovery +tool descriptions +semantic legibility +composition +``` + +## System 4 — Human Web Application + +Focus: + +```text +navigation +feedback +mental model +error recovery +``` + +## System 5 — Administrative Interface + +Focus: + +```text +high-impact operations +permission clarity +state visibility +reversibility +``` + +--- + +# 12. Minimum Viable Experiment + +The first end-to-end experiment should be intentionally modest. + +Use: + +```text +1 REST API +1 human web workflow +1 MCP service +``` + +For each: + +```text +3 workloads +3 repetitions +at least 2 actor profiles +``` + +Capture: + +```text +full traces +task outcomes +time +interaction count +errors +retries +semantic repairs where applicable +qualitative observations +``` + +Produce: + +```text +dimension assessment +diagnosis +recommendations +confidence +``` + +The immediate question is: + +> Does the resulting report tell us something useful that ordinary testing did not? + +If not, improve the research model before scaling. + +--- + +# 13. Milestone Program + +## M0 — Research Scaffold + +Establish: + +```text +INTENT.md +ResearchProgram.md +terminology +research structure +experiment structure +``` + +Exit condition: + +The program can describe how an observation will be captured. + +--- + +## M1 — Observation Kernel + +Implement: + +```text +ExperienceObservation +Workload +ActorProfile +InteractionTrace +``` + +Exit condition: + +One workload can be recorded end-to-end. + +--- + +## M2 — First AMIX Experiment + +Evaluate one API or MCP service. + +Exit condition: + +A repeatable AMIX report can be generated from evidence. + +--- + +## M3 — First HMIX Experiment + +Evaluate one human workflow. + +Exit condition: + +Human interaction evidence can be represented in the same observation framework. + +--- + +## M4 — Mixed Benchmark Corpus + +Create first cross-interface benchmark suite. + +Exit condition: + +At least five systems and approximately twenty workloads exist. + +--- + +## M5 — Grading v0.1 + +Introduce initial dimension grading. + +Exit condition: + +Grades are decomposable, evidence-backed, and reproducible. + +--- + +## M6 — Diagnosis v0.1 + +Create first experience defect taxonomy. + +Exit condition: + +Observed problems can be mapped to reusable diagnostic categories. + +--- + +## M7 — Optimization Experiment + +Modify an interface based on grader findings. + +Exit condition: + +A measurable before/after improvement is demonstrated. + +--- + +## M8 — Regression Service + +Run benchmark workloads repeatedly across interface versions. + +Exit condition: + +An experience regression can be detected automatically. + +--- + +## M9 — Self-Improving Grader + +Use accumulated experiments to improve rubric, workloads, and diagnostics. + +Exit condition: + +A new grader version demonstrably outperforms an older version on the benchmark corpus. + +--- + +# 14. Research Artifacts + +The repository should accumulate four major forms of research output. + +```text +concepts/ +experiments/ +benchmarks/ +patterns/ +``` + +Possible initial structure: + +```text +xperience-grader/ +│ +├── INTENT.md +├── ResearchProgram.md +│ +├── concepts/ +│ ├── ExperienceObservationModel.md +│ ├── WorkloadModel.md +│ ├── ActorProfileModel.md +│ └── GradingModel.md +│ +├── hmix/ +│ ├── HMIXDimensions.md +│ ├── workloads/ +│ └── experiments/ +│ +├── amix/ +│ ├── AMIXDimensions.md +│ ├── SemanticRepairTaxonomy.md +│ ├── workloads/ +│ └── experiments/ +│ +├── benchmarks/ +│ +├── diagnostics/ +│ +├── patterns/ +│ +├── schemas/ +│ +└── research/ +``` + +--- + +# 15. Research Risks + +## Premature Scoring + +Creating attractive scores before establishing measurement validity. + +Mitigation: + +Keep dimensions and evidence visible. + +--- + +## Model Bias + +AMIX results may become model-specific. + +Mitigation: + +Use multiple agent classes and explicitly encode actor profiles. + +--- + +## Synthetic UX Illusion + +AI-generated HMIX reviews may appear more reliable than they are. + +Mitigation: + +Calibrate against real human studies. + +--- + +## Benchmark Gaming + +Interfaces could be optimized to the benchmark rather than actual usability. + +Mitigation: + +Maintain hidden and evolving workloads. + +--- + +## Measurement Distortion + +Metrics such as clicks or token counts can become optimization targets despite weak relationship to quality. + +Mitigation: + +Use multi-objective evaluation. + +--- + +## Rubric Drift + +Self-improvement could make historical grades incomparable. + +Mitigation: + +Version methodologies and retain old graders. + +--- + +## Observer Effects + +Instrumentation can alter interaction behavior. + +Mitigation: + +Measure instrumentation overhead. + +--- + +# 16. Success Criteria + +The research program becomes practically valuable when it can demonstrate all of the following: + +```text +repeatable workload execution + +evidence-backed grading + +meaningful distinction between interface variants + +useful diagnosis + +measurable improvement after intervention + +experience regression detection + +cross-model AMIX comparison + +cross-user HMIX comparison + +historically reproducible results +``` + +The strongest validation will be: + +> An interface change proposed from xperience-grader evidence produces a repeatable improvement for the target actors and workloads without unacceptable degradation elsewhere. + +--- + +# 17. Long-Term Research Questions + +The mature program should explore deeper questions. + +## Interface Universals + +Are there interface properties that consistently improve interaction across human and artificial actors? + +--- + +## Intelligence-Relative Interfaces + +Should interfaces adapt dynamically to actor capability? + +--- + +## Experience Compilation + +Can exploratory interaction be progressively compiled into deterministic procedure? + +--- + +## Experience Debt + +Can poor interface quality accumulate similarly to technical debt? + +--- + +## Interface Fitness Landscapes + +Can interface variants be explored as an optimization landscape? + +--- + +## Adaptive Interfaces + +Should interfaces themselves evolve based on observed interaction difficulty? + +--- + +## Experience Economics + +What is the economic cost of interface friction? + +For agents: + +```text +tokens +compute +latency +human escalation +``` + +For humans: + +```text +time +training +support +errors +frustration +``` + +--- + +## Mixed Human–Agent Experience + +How should interfaces be evaluated when work continuously moves between humans and agents? + +This may ultimately require a third perspective beyond HMIX and AMIX: + +```text +collaborative MIX +``` + +--- + +# 18. Research North Star + +The program should continuously return to one question: + +> Can we turn interface experience from an opinion into an observable, testable, improvable system property? + +If the answer becomes yes, `xperience-grader` should evolve beyond a research tool into infrastructure for continuous experience engineering. + +The target loop is: + +```text +experience + ↓ +evidence + ↓ +understanding + ↓ +change + ↓ +better experience + ↓ +new evidence + └──────────────→ +``` + +The research program exists to determine how much of that loop can be made reliable.