Measuring the User Experience: Collecting, Analyzing, and Presenting UX Metrics

Copyright
Contents
Dedication
Preface
References
Acknowledgments
A Special Note From Cheryl Tullis Sirois
Biographies
1
Introduction
1.1 What is User Experience?
1.2 What are User Experience Metrics?
1.3 The Value of UX Metrics
1.4 Metrics for Everyone
1.5 New Technologies in User Experience Metrics
1.6 Ten Myths About UX Metrics
Myth 1: Metrics Take Too Much Time to Collect
Myth 2: UX Metrics Cost Too Much Money
Myth 3: UX Metrics Are Not Useful When Focusing on Small Improvements
Myth 4: UX Metrics Don’t Help Us Understand Causes
Myth 5: UX Metrics Are Too Noisy
Myth 6: You Can Just Trust Your Gut
Myth 7: Metrics Don’t Apply to New Products
Myth 8: No Metrics Exist for the Type of Issues We Are Dealing With
Myth 9: Metrics Are Not Understood or Appreciated by Management
Myth 10: It’s Difficult to Collect Reliable Data With a Small Sample Size
2
Background
2.1 Independent and Dependent Variables
2.2 Types of Data
2.2.1 Nominal Data
2.2.2 Ordinal Data
2.2.3 Interval Data
2.2.4
Ratio Data
2.3 Descriptive Statistics
2.3.1 Measures of Central Tendency
2.3.2 Measures of Variability
2.3.3 Confidence Intervals
2.3.4
Displaying Confidence Intervals as Error Bars
2.4 Comparing Means
2.4
Independent Samples
2.4.2 Paired Samples
2.4.3 Comparing More Than Two Samples
2.5 Relationships Between Variables
2.5.1 Correlations
2.6 Non-Parametric Tests
2.6.1 The Chi-Square Test
2.7 Presenting Your Data Graphically
2.7.1 Column or Bar Graphs
2.7.2 Line Graphs
2.7.3 Scatterplots
2.7.4 Pie or Donut Charts
2.7.5 Stacked Bar Graphs
2.8 Summary
3
Planning
3.1 Study Goals
3.1.1 Formative User Research
3.1.2 Summative User Research
3.2 UX Goals
3.2.1 User Performance
3.2.2 User Preferences
3.2.3 User Emotions
3.3 Business Goals
3.4 Choosing the Right UX Metrics
3.4.1 Completing an eCommerce Transaction
3.4.2 Comparing Products
3.4.3 Evaluating Frequent Use of the Same Product
3.4.4 Evaluating Navigation and/or Information Architecture
3.4.5 Increasing Awareness
3.4.6 Problem Discovery
3.4.7 Maximizing Usability for a Critical Product
3.4.8 Creating an Overall Positive User Experience
3.4.9 Evaluating the Impact of Subtle Changes
3.4.10 Comparing Alternative Designs
3.5 User Research Methods and Tools
3.5.1 Traditional (Moderated) Usability Tests
3.5.2 Unmoderated Usability Tests
3.5.3 Online Surveys
3.5.4 Information Architecture Tools
3.5.5 Click and Mouse Tools
3.6 Other Study Details
3.6.1 Budgets and Timelines
3.6.2 Participants
3.6.3 Data Collection
3.6.4 Data Cleanup
3.7 Summary
4
Performance Metrics
4.1 Task Success
4.1.1 Binary Success
Calculating Confidence Intervals for Binary Success
4.1.2 Levels of Success
How to Collect and Measure Levels Of Success
How to Analyze and Present Levels of Success
4.1.3 Issues in Measuring Success
4.2 Time-On-Task
4.2.1 Importance of Measuring Time-on-Task
4.2.2 How to Collect and Measure Time-on-Task
Turning on and off the Clock
Tabulating Time Data
4.2.3 Analyzing and Presenting Time-on-Task Data
Ranges
Thresholds
Distributions and Outliers
4.2.4 Issues to Consider When Using Time Data
Only Successful Tasks or all Tasks?
Using a Concurrent Think-Aloud Protocol
Should you tell the Participants about the Time Measurement?
4.3 Errors
4.3.1 When to Measure Errors
4.3.2 What Constitutes an Error?
4.3.3 Collecting and Measuring Errors
4.3.4 Analyzing and Presenting Errors
Tasks with a Single Error Opportunity
Tasks with Multiple Error Opportunities
4.3.5 Issues to Consider When Using Error Metrics
4.4 Other Efficiency Metrics
4.4.1 Collecting and Measuring Efficiency
4.4.2 Analyzing and Presenting Efficiency Data
Lostness
4.4.3 Efficiency as a Combination of Task Success and Time
4.5 Learnability
4.5.1 Collecting and Measuring Learnability Data
4.5.2 Analyzing and Presenting Learnability Data
4.5.3 Issues to Consider When Measuring Learnability
What is a Trial?
Number of Trials
4.6 Summary
5
Self-Reported Metrics
5.1 Importance of Self-Reported Data
5.2 Rating Scales
5.2.1 Likert Scales
5.2.2 Semantic Differential Scales
5.2.3 When to Collect Self-Reported Data
5.2.4 How to Collect Ratings
5.2.5 Biases in Collecting Self-Reported Data
5.2.6 General Guidelines for Rating Scales
5.2.7 Analyzing Rating-Scale Data
5.3 Post-Task Ratings
5.3.1 Ease of Use
5.3.2 After-Scenario Questionnaire
5.3.3 Expectation Measure
5.3.4 A Comparison of Post-Task Self-Reported Metrics
5.4 Overall User Experience Ratings
5.4.1 System Usability Scale
5.4.2 Computer System Usability Questionnaire
5.4.3 Product Reaction Cards
5.4.4 User Experience Questionnaire
5.4.5 AttrakDiff
5.4.6 Net Promoter Score
5.4.7 Additional Tools for Measuring Self-Reported User Experience
5.4.8 A Comparison of Selected Overall Self-Reported Metrics
5.5 Using SUS to Compare Designs
5.6 Online Services
5.6.1 Website Analysis and Measurement Inventory
5.6.2 American Customer Satisfaction Index
5.6.3 OpinionLab
5.6.4 Issues With Live-Site Surveys
5.7
Other Types of Self-Reported Metrics
5.7.1 Assessing Attribute Priorities
5.7.2 Assessing Specific Attributes
5.7.3 Assessing Specific Elements
5.7.4 Open-Ended Questions
5.7.5 Awareness and Comprehension
5.7.6 Awareness and Usefulness Gaps
5.8
Summary
6
Issues-Based Metrics
6.1 What is a Usability Issue?
6.1.1 Real Issues Versus False Issues
6.2 How to Identify an Issue
6.2.1 Using Think-Aloud From One-on-One Studies
6.2.2 Using Verbatim Comments From Automated Studies
6.2.3 Using Web Analytics
6.2.4
Using Eye-Tracking
6.3 Severity Ratings
6.3.1 Severity Ratings Based on the User Experience
6.3.2 Severity Ratings Based on a Combination of Factors
6.3.3
Using a Severity Rating System
6.3.4 Some Caveats About Rating Systems
3.4 Analyzing and Reporting Metrics for Usability Issues
6.4.1 Frequency of Unique Issues
6.4.2 Frequency of Issues per Participant
6.4.3 Percentage of Participants
6.4.4 Issues by Category
6.4.5 Issues by Task
6.5 Consistency in Identifying Usability Issues
6.6 Bias in Identifying Usability Issues
6.7 Number of Participants
6.7.1 Five Participants Is Enough
6.7.2 Five Participants Is Not Enough
6.7.3 What to Do?
6.7.4 Our Recommendation
6.8 Summary
7
Eye Tracking
7.1 How Eye Tracking Works
7.2 Mobile Eye Tracking
7.2.1 Measuring Glanceability
7.2.2 Understanding Mobile Users in Context
7.2.3 Mobile Eye Tracking Technology
7.2.4 Glasses
7.2.5 Device Stand
7.2.6 Software-Based Eye Tracking
7.3 Visualizing Eye Tracking Data
7.4 Areas of Interest
7.5 Common Eye Tracking Metrics
7.5.1 Dwell Time
7.5.2 Number of Fixations
7.5.3 Fixation Duration
7.5.4 Sequence
7.5.5 Time to First Fixation
7.5.6
Revisits
7.5.7 Hit Ratio
7.6 Tips for Analyzing Eye Tracking Data
7.7 Pupillary Response
7.8 Summary
8
Measuring Emotion
8.1 Defining the Emotional User Experience
8.2 Methods to Measure Emotions
8.2.1 Five Challenges in Measuring Emotions
8.3 Measuring Emotions Through Verbal Expressions
8.4 Self-Report
8.5 Facial Expression Analysis
8.6 Galvanic Skin Response
8.7 Case Study: the Value of Biometrics
8.8 Summary
9
Combined and Comparative Metrics
9.1 Single Ux Scores
9.1.1 Combining Metrics Based on Target Goals
9.1.2 Combining Metrics Based on Percentages
9.1.3 Combining Metrics Based on Z-Scores
9.1.4 Using SUM: Single Usability Metric
9.2 Ux Scorecards and Framework
9.2.1 UX Scorecards
9.2.2 UX Frameworks
9.3 Comparison to Goals and Expert Performance
9.3.1 Comparison to Goals
9.3.2 Comparison to Expert Performance
9.4 Summary
10
Special Topics
10.1 Web Analytics
10.1.1 Basic Web Analytics
10.1.2 Click-Through Rates
10.1.3 Drop-off Rates
10.1.4 A/B Tests
10.2 Card-Sorting Data
10.2.1 Analyses of Open Card-Sort Data
10.2.1.1 Hierarchical Cluster Analysis
10.2.1.2 Multidimensional Scaling
10.2.1.3 How Many Participants Are Enough for a Card-Sorting Study?
10.2.2 Analyses of Closed Card-Sort Data
10.3 Tree Testing
10.4 First Click Testing
10.5 Accessibility Metrics
10.6 Return-on-Investment Metrics
10.7 Summary
11
Case Studies
11.1 Thinking Fast and Slow in the Netflix TV User Interface
 Background
 Methods
Participant Interviews
Materials
Procedure
Example scenarios
Eye-Tracking
 Results
 Discussion
Thinking Fast
Thinking Slow
 Impact
Biography
11.2 Participate/Compete/Win (Pcw) Framework: Evaluating Products and Features in the Marketplace
11.2.1 Introduction
11.2.2 Outlining Objective Criteria
Participate
Compete
Win
11.2.3 Feature Analysis
Generating the Feature List
Calculating a Feature Importance Score
Choosing a Competitive Product
Calculating a Feature Availability/Value Score
11.2.4 “PCW” (Summative) Usability Testing
Overall Versus Workflow Success Rate
Prototype Versus Live Site Testing
Overall Program Success
Biographies
11.3 Enterprise UX Case Study: Uncovering the “UX Revenue Chain”
11.3.1 Introduction
11.3.1 Metric Identification and Selection
Participants
11.3.2 Methods
Top Task Identification
Top Task Force Ranking Survey
Task-based Benchmark Study Pre-redesign #1
11.3.3 Analysis
11.3.4 Results
11.3.5 Conclusion
Biography
11.4 Competitive UX Benchmarking of four Healthcare Websites
11.4.1 Methodology
Data collection through UserZoom for both studies
Experimental Design for the Quantitative N = 200 study
Metrics and Key Performance Indicators
11.4.2 Results
qxScore
Overall Results
Task Details
Pre-Versus Post-Site Perception
11.4.3 Summary and Recommendations
11.4.4 Acknowledgment and Contributions
11.4.5 Biography
11.5 Closing the SNAP Gap
11.5.1 Field Research
11.5.2 Weekly Reviews
11.5.3 Application Questions
11.5.4 Surveys
11.5.5 Testing Prototypes
11.5.6 Success Metric
11.5.7 Organizations
GoInvo
Massachusetts Department of Transitional Assistance
11.5.8 Biography
12
Ten Keys to Success
12.1 Make the Data Come Alive
12.2 Don’t Wait to be Asked to Measure
12.3 Measurement is Less Expensive than you Think
12.4 Plan Early
12.5 Benchmark Your Products
12.6 Explore Your Data
12.7 Speak the Language of Business
12.8 Show Your Confidence
12.9 Don’t Misuse Metrics
12.10 Simplify Your Presentation
Index