coulomb/xperience-grader
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
xperience-grader/INTENT.md

24 KiB
Raw Permalink Blame History

xperience-grader

Intent

xperience-grader is a research and engineering project for building a self-improving experience evaluation service for machine interfaces.

Its purpose is to inspect, exercise, measure, explain, and ultimately help optimize interfaces through which humans and artificial agents interact with software systems.

The project treats interface quality not primarily as a property of visual design, API syntax, protocol choice, or implementation quality, but as a property of the interaction experience between an acting intelligence and a machine system while pursuing a goal.

xperience-grader therefore asks:

How well does this interface enable an actor to understand the system, pursue an intended goal, act safely and efficiently, observe consequences, recover from problems, and improve through repeated interaction?

The initial project distinguishes two principal perspectives:

  • HMIX — Human–Machine Interface Experience
  • AMIX — Agent–Machine Interface Experience

These perspectives share a common conceptual and measurement framework while retaining different actor models, constraints, failure modes, and quality criteria.

The long-term ambition is to establish xperience-grader as an independent, continuously improving grading and diagnostic service that can be engaged against APIs, services, applications, command-line tools, agent tools, workflows, protocols, graphical interfaces, administrative surfaces, data interfaces, and combinations of these.

The project should progress from evaluation toward closed-loop interface optimization, while keeping grading and optimization conceptually and operationally separate.

1. Mission

Build a system that can answer, with evidence:

What is it like for this kind of actor to use this machine interface for this task?

And subsequently:

What changes would measurably improve that experience without degrading important capabilities, safety properties, or other actor experiences?

The system should become increasingly capable of:

  1. discovering an interface,
  2. understanding its available capabilities,
  3. deriving or accepting representative tasks,
  4. interacting with the system,
  5. observing complete interaction traces,
  6. measuring relevant properties,
  7. diagnosing experience deficiencies,
  8. explaining those deficiencies,
  9. proposing improvements,
  10. testing modified interfaces,
  11. learning from accumulated evaluations,
  12. improving its own grading methodology.

xperience-grader should itself become an adaptive measurement instrument.

2. Scope

The project is concerned with interaction surfaces between intelligences and computational systems.

These include, but are not limited to:

  • web applications,
  • desktop applications,
  • mobile applications,
  • administrative interfaces,
  • command-line interfaces,
  • APIs,
  • REST interfaces,
  • GraphQL,
  • RPC interfaces,
  • MCP servers and tools,
  • SDKs,
  • libraries,
  • workflow systems,
  • messaging interfaces,
  • event interfaces,
  • data access surfaces,
  • configuration interfaces,
  • conversational interfaces,
  • autonomous-agent tools,
  • multi-step service interactions,
  • compound interfaces spanning several systems.

An evaluated experience may cross interface families.

For example:

human
  → web UI
  → asynchronous job
  → notification
  → administration interface

or:

agent
  → MCP discovery
  → API call
  → event stream
  → object storage
  → verification endpoint

The unit of concern is therefore not necessarily a screen, endpoint, API, or tool.

It is an experience path through a system in pursuit of a goal.

3. Foundational Model

The project starts from a general interaction loop:

Goal
  ↓
Discover
  ↓
Interpret
  ↓
Plan
  ↓
Act
  ↓
Observe
  ↓
Verify
  ↓
Recover
  ↓
Learn
  └──────────────→ future interaction

Different interfaces support different portions of this loop explicitly.

Weak interfaces force actors to reconstruct missing parts through experimentation, inference, documentation search, external knowledge, or human intervention.

A strong interface reduces unnecessary uncertainty while preserving expressive power.

The project therefore treats experience quality as a multidimensional phenomenon.

A preliminary abstraction is:

ExperienceQuality =
    f(
        goal_success,
        effort,
        latency,
        uncertainty,
        recoverability,
        learnability,
        robustness,
        safety,
        expressive_power,
        adaptability
    )

No single scalar score should be considered sufficient to represent the complete interaction experience.

Scores are projections over richer observations.

4. Experience Observation as the Primitive

The fundamental data object of xperience-grader should not be an interface score.

It should be an experience observation.

Conceptually:

ExperienceObservation
    actor
    actor_profile
    system
    interface_surface
    environment
    goal
    task
    constraints
    starting_state
    interaction_trace
    outcome
    measurements
    incidents
    uncertainties
    evidence

A grade is derived from one or more observations.

Therefore:

grade(interface)

is considered incomplete.

The more meaningful form is:

grade(
    interface,
    actor_profile,
    task,
    environment,
    constraints
)

Aggregated grades may subsequently be computed across workloads, actor populations, or operational environments.

5. Human–Machine Interface Experience — HMIX

HMIX evaluates interfaces from the perspective of human actors.

It is broader than visual design or conventional usability.

Relevant dimensions may include:

Discoverability

Can the person determine what functionality exists and where to find it?

Comprehensibility

Can the person form an adequate mental model of the system and its current state?

Action Clarity

Is it apparent what actions are available and what they mean?

Predictability

Can the user reasonably anticipate the consequences of an action?

Feedback

Does the system make progress, state transitions, success, and failure visible?

Efficiency

How much time, attention, navigation, repetition, and manual work are required?

Cognitive Load

How much state must the user remember, infer, translate, or reconstruct?

Error Prevention

Does the system make unintended actions unnecessarily easy?

Error Legibility

Can users understand what went wrong?

Recoverability

Can mistakes and exceptional states be repaired safely and economically?

Learnability

Does repeated interaction result in increasing competence and reduced effort?

Accessibility

Can users with different capabilities interact effectively with the system?

Trust Calibration

Can the user correctly judge what the machine knows, has done, intends to do, and guarantees?

Control

Can the user meaningfully intervene, inspect, reverse, or redirect system behavior where necessary?

HMIX should eventually incorporate both empirical human measurements and synthetic evaluations by capable multimodal models.

Synthetic evaluation must not be mistaken for human evidence.

6. Agent–Machine Interface Experience — AMIX

AMIX evaluates interfaces from the perspective of artificial agents.

Agent experience quality is not equivalent to API quality.

Agents interact through APIs, command-line programs, MCP tools, browser interfaces, documents, files, databases, event streams, workflows, messages, or mixed surfaces.

Relevant dimensions may include:

Capability Discoverability

Can the agent determine which relevant capabilities exist?

Semantic Legibility

Can the agent determine what operations, fields, states, and objects mean?

Contract Precision

Are expectations represented sufficiently precisely for reliable execution?

Invocability

Can the agent reliably formulate valid interactions?

Composability

Can operations be safely incorporated into larger plans?

State Legibility

Can relevant machine state be inspected rather than inferred?

Observability

Can the agent determine what happened after an action?

Verifiability

Can the agent independently determine whether the intended outcome was achieved?

Error Legibility

Are failures represented in a form that permits machine reasoning and corrective action?

Recoverability

Can the agent identify a safe next action after failure?

Permission Legibility

Can the agent understand what it can, may, must, and must not do?

Idempotency and Replay Safety

Can uncertain operations be retried without unintended duplication or corruption?

Determinism

How reliably do equivalent interactions produce operationally equivalent results?

Context Efficiency

How much context, documentation, token consumption, and state reconstruction are required?

Interaction Efficiency

How many tool calls, retries, planning rounds, or inference cycles are required?

Automation Depth

How far can a task proceed without human semantic repair or intervention?

Distillability

Can successful exploratory behavior later be converted into deterministic, cheaper, reusable interaction procedures?

7. Semantic Repair

A central AMIX research concept is semantic repair.

Semantic repair occurs when an agent cannot continue reliable execution based on the interface contract and available machine-readable state alone.

Examples include:

  • searching external documentation,
  • inspecting implementation source code,
  • experimentally probing parameter semantics,
  • interpreting ambiguous error messages,
  • guessing undocumented state transitions,
  • asking a human what an operation means,
  • retrying alternative formats,
  • reverse engineering implicit conventions.

Possible measures include:

semantic_repairs / successful_task

semantic_repair_tokens

semantic_repair_latency

semantic_repair_tool_calls

human_semantic_escalations

Semantic repair rate may become one of the strongest indicators of poor agent-facing interface quality.

8. Workloads

Interfaces should be evaluated against workloads, not isolated operations.

A workload describes representative goals actors attempt to accomplish.

Example:

Create a customer account,
assign billing information,
activate a subscription,
verify successful provisioning,
and recover correctly if payment authorization fails.

A workload may define:

goal
actor profile
initial conditions
allowed interfaces
constraints
success conditions
safety conditions
acceptable side effects
evaluation dimensions

Workloads may originate from:

  • product use cases,
  • documentation,
  • support incidents,
  • production telemetry,
  • human experts,
  • agent-generated exploration,
  • interface schemas,
  • acceptance tests,
  • historical traces.

A mature xperience-grader should maintain evolving workload libraries.

9. Evidence

Every grade should be evidence-backed.

Possible evidence includes:

  • interaction traces,
  • screenshots,
  • video recordings,
  • DOM state,
  • accessibility trees,
  • API requests and responses,
  • schemas,
  • tool definitions,
  • logs,
  • traces,
  • events,
  • timing measurements,
  • token usage,
  • tool-call counts,
  • retries,
  • error states,
  • user feedback,
  • human review,
  • agent explanations,
  • task outcomes.

Grades should preserve provenance.

A future observer should be able to ask:

Why did this interface receive this grade?

and inspect the underlying evidence.

10. Measurement Before Judgment

The project should distinguish:

observation
measurement
interpretation
grade
diagnosis
recommendation

These are not equivalent.

For example:

Observation:
Agent performed 14 tool calls.

Measurement:
Median comparable workload requires 6 calls.

Interpretation:
Interaction requires unusually high exploratory effort.

Diagnosis:
Resource identifiers are not discoverable from preceding responses.

Recommendation:
Expose canonical resource references in mutation responses.

Maintaining this separation is essential for auditability and methodological improvement.

11. Grading

The project may provide grades for communication and comparison, but grading should never destroy the underlying multidimensional evidence.

A possible hierarchy is:

raw observations
    ↓
metrics
    ↓
dimension assessments
    ↓
task grades
    ↓
workload grades
    ↓
interface profile
    ↓
aggregate grade

An aggregate grade should therefore remain decomposable.

The project should strongly prefer statements such as:

Excellent discoverability
Moderate semantic ambiguity
Poor recovery behavior
Very high execution efficiency after successful discovery

over unexplained scalar judgments.

12. Reference Actors

Evaluation requires explicit actor models.

HMIX reference actors might vary by:

  • domain experience,
  • application experience,
  • language,
  • accessibility requirements,
  • frequency of use,
  • technical expertise,
  • cognitive workload.

AMIX reference actors might vary by:

  • model,
  • model capability class,
  • context size,
  • tool-use ability,
  • planning strategy,
  • memory configuration,
  • available documentation,
  • autonomy level,
  • token budget.

An interface can therefore legitimately have:

high AMIX for frontier coding agents

moderate AMIX for small local agents

poor AMIX for deterministic automation

without contradiction.

13. Capability Preservation

Optimization must not reduce the problem to minimizing clicks, tokens, calls, or latency.

An interface may become cheaper while losing important expressive power.

Every optimization should therefore consider capability preservation.

Relevant dimensions include:

goal success
expressive power
robustness
safety
recoverability
efficiency
latency
cost
learnability

Optimization should seek Pareto improvements where possible.

A change that improves one dimension while materially degrading another should be visible as a tradeoff, not silently classified as an improvement.

14. The Grader and the Optimizer

The project should maintain a strict conceptual boundary between grading and optimization.

Grader

The grader asks:

Given this evidence, what can we conclude about the quality of this experience?

The grader should be conservative, repeatable, explainable, and evidence-oriented.

Optimizer

The optimizer asks:

Given these deficiencies and constraints, what changes might improve the experience?

The optimizer may generate:

  • interface redesigns,
  • schema changes,
  • documentation changes,
  • error model improvements,
  • interaction shortcuts,
  • new endpoints,
  • richer machine-readable state,
  • UI modifications,
  • workflow restructuring,
  • protocol changes.

Proposed changes should be evaluated by running workloads again.

The basic optimization loop becomes:

interface
   ↓
workload execution
   ↓
observation
   ↓
grading
   ↓
diagnosis
   ↓
candidate changes
   ↓
modified interface
   ↓
workload execution
   └──────────────→ repeat

15. Self-Improvement

xperience-grader should itself improve through accumulated experience.

Self-improvement may occur at several levels.

Better Workloads

Observed production behavior reveals missing or unrealistic evaluation scenarios.

Better Measurements

Repeated evaluation reveals more predictive metrics.

Better Rubrics

Some dimensions may prove redundant, weak, or incorrectly weighted.

Better Reference Baselines

Accumulated observations establish empirical comparison distributions.

Better Diagnoses

Recurring interaction failures can be classified into reusable patterns.

Better Recommendations

Interface modifications can be evaluated against subsequent measurements.

Better Evaluators

Different models, tools, simulations, and human reviewers can be compared for grading reliability.

Better Methodology

The project should continuously challenge whether its own grading correlates with actual experience outcomes.

Self-improvement must not mean uncontrolled rubric drift.

Versions of methodologies, graders, benchmark sets, and reference actors should remain identifiable and reproducible.

16. The Experience Pattern Library

A long-term output of accumulated grading should be a catalog of interface patterns.

Examples may include:

Explicit State Pattern
Structured Failure Pattern
Discoverable Capability Pattern
Safe Retry Pattern
Progressive Disclosure Pattern
Canonical Identifier Pattern
Verifiable Mutation Pattern
Reversible Action Pattern
Inspectable Permission Pattern
Machine-Readable Enumeration Pattern

Each pattern should accumulate evidence across HMIX and AMIX workloads.

Patterns may therefore eventually be classified as:

human-beneficial
agent-beneficial
universally-beneficial
human-agent tradeoff
context-dependent

This may form the basis of a broader MIX Interface Design Canon.

17. Universal Interface Properties

One of the deeper research questions of the project is whether certain properties systematically improve interaction quality for both human and artificial intelligence.

Candidate properties include:

  • explicit state,
  • stable concepts,
  • clear feedback,
  • predictable semantics,
  • reversible operations,
  • visible consequences,
  • inspectability,
  • structured errors,
  • progressive disclosure,
  • meaningful defaults,
  • deterministic identifiers,
  • discoverable capabilities.

If such properties can be empirically demonstrated across actor classes, they may represent more fundamental principles of machine-interface design.

18. Service Model

The intended operational form of xperience-grader is a service.

Conceptually:

submit target
    ↓
identify interface surfaces
    ↓
select actor profiles
    ↓
derive/select workloads
    ↓
execute evaluation
    ↓
collect evidence
    ↓
calculate measurements
    ↓
grade dimensions
    ↓
generate diagnosis
    ↓
produce improvement candidates

A user might eventually request:

grade this API for autonomous agents

or:

review this application onboarding flow for first-time human users

or:

compare the HMIX and AMIX consequences of these two interface designs

or:

continuously grade every release and detect experience regressions

19. Continuous Experience Testing

A mature implementation should integrate naturally into engineering workflows.

Possible execution modes include:

ad hoc review
pull-request evaluation
release qualification
nightly regression testing
production telemetry assessment
continuous interface optimization

This transforms experience quality from an occasional design-review activity into an observable engineering property.

A future pipeline might be:

commit
  ↓
build
  ↓
functional tests
  ↓
security tests
  ↓
HMIX workloads
  ↓
AMIX workloads
  ↓
experience regression gate
  ↓
deployment

20. Relationship to Existing Practices

xperience-grader should reuse rather than replace mature disciplines.

Relevant adjacent practices include:

  • usability testing,
  • accessibility testing,
  • UX research,
  • human-computer interaction,
  • API design,
  • API linting,
  • contract testing,
  • integration testing,
  • agent benchmarking,
  • software observability,
  • distributed tracing,
  • reliability engineering,
  • performance testing,
  • security testing,
  • formal interface specifications.

The distinctive contribution of xperience-grader is to combine these into an actor-centered empirical model of interface experience.

21. Architecture Direction

An initial conceptual architecture may include:

xperience-grader
│
├── core
│   ├── observation-model
│   ├── workload-model
│   ├── evidence-model
│   ├── grading-model
│   └── actor-model
│
├── hmix
│   ├── graders
│   ├── actors
│   ├── workloads
│   └── metrics
│
├── amix
│   ├── graders
│   ├── actors
│   ├── workloads
│   └── metrics
│
├── runners
│   ├── browser
│   ├── api
│   ├── cli
│   ├── mcp
│   └── workflow
│
├── evidence
│
├── benchmarks
│
├── diagnostics
│
├── patterns
│
└── optimizer

This structure is provisional.

The repository should evolve from evidence rather than architecture aesthetics.

22. Initial Research Questions

The first research phase should answer at least the following:

  1. What dimensions best predict perceived and actual interaction quality?
  2. Which dimensions can be measured objectively?
  3. Which require interpretation?
  4. How stable are grades across different evaluators?
  5. How dependent is AMIX on the chosen model?
  6. How dependent is HMIX on expertise and familiarity?
  7. Which interface characteristics benefit both humans and agents?
  8. Which characteristics create human-agent tradeoffs?
  9. How should multi-interface workflows be graded?
  10. How should uncertainty be represented?
  11. How should failed tasks affect grades?
  12. How should safety restrictions interact with efficiency measurements?
  13. Can semantic repair be measured reliably?
  14. Can experience regressions be detected automatically?
  15. Can successful exploratory agent interactions be distilled into deterministic procedures?
  16. Can optimizer-generated interface changes produce reproducible improvements?
  17. How should grading methodology itself be evaluated?

23. Initial Experimental Program

Early experiments should remain deliberately small.

Candidate targets should include:

  • one conventional REST API,
  • one MCP service,
  • one CLI,
  • one administrative web interface,
  • one end-user web workflow.

Each target should receive:

3–10 representative workloads
multiple observations per workload
explicit actor profiles
trace capture
initial measurements
human or expert review
diagnostic report

The goal is not initially to establish a universal score.

The goal is to discover whether the proposed measurement model produces useful, reproducible distinctions.

24. Non-Goals

The project is not intended to become:

  • a generic visual-design rating service,
  • an aesthetic preference engine,
  • an API style linter,
  • a replacement for accessibility standards,
  • a replacement for functional tests,
  • a model benchmark leaderboard,
  • a single universal experience number,
  • an optimizer that silently modifies production systems.

These may become inputs or integrations, but they are not the defining purpose.

25. Guiding Principles

Experience Is Relational

There is no interface quality independent of actor, goal, context, and constraints.

Evidence Before Opinion

Grades should be supported by observable interaction evidence.

Workloads Before Widgets

Meaningful tasks matter more than isolated screens or endpoints.

Multidimensional Before Scalar

A score is a projection, not reality.

Diagnose Before Optimize

Changes should address identified interaction failures.

Preserve Capability

Efficiency is not improvement if expressive power, safety, or robustness is lost.

Humans and Agents Are Different

HMIX and AMIX should share structure without pretending their constraints are identical.

Learn from Interaction

Every grading run should be capable of improving future evaluation.

Version the Method

Self-improvement must remain reproducible and auditable.

Optimize the Coupling

The deeper object of study is neither the actor nor the machine alone, but the quality of the interface coupling between them.

26. Long-Term Vision

xperience-grader should evolve into infrastructure for experience engineering.

A sufficiently mature system would allow interface quality to become:

  • observable,
  • measurable,
  • comparable,
  • testable,
  • regression-controlled,
  • diagnosable,
  • experimentally optimizable.

Software teams should eventually be able to ask not merely:

Does the interface work?

but:

How well can different forms of intelligence work through it?

The long-term research hypothesis is that good interfaces are not merely pleasant surfaces or convenient protocols.

They are effective coupling mechanisms between adaptive systems.

Humans adapt.

Agents adapt.

Machines change.

Tasks evolve.

A durable interface therefore cannot only encode commands and data. It must support the formation, execution, observation, correction, and refinement of purposeful interaction.

xperience-grader exists to make that quality measurable.

Measure the coupling. Explain the friction. Improve the experience.