This book is your definitive guide to the rapidly growing role of Quantitative User Experience (Quant UX) Research in product development. The book provides an overview of the skills you need on the job, presents hands-on projects with reusable code, and shares advice on starting and developing a career. The book goes beyond basic skills to focus on what is unique to Quant UX. The authors are two of the most widely recognized practitioners in Quant UX research, and this book shares insights from their combined decades of experience.
Organizations today have more data about user needs and behaviors than ever before. With this large-scale data, Quant UX researchers work to understand usage patterns, measure the impact of design changes, and inform strategic decisions. In the Quant UX role, interdisciplinary researchers apply analytical skills to uncover user needs, inform engineering and design, answer strategic business questions, and optimize software and hardware products for human interaction. This book provides guidance around customer satisfaction surveys, understanding user behavior from log analysis, and the statistical methods that are commonly used to assess user outcomes.
What You Will Learn:
Discover the role of Quantitative User Experience (Quant UX) research
Understand how Quant UX research differs from other disciplines such as data science
Plan common research projects and know how to achieve success
Position Quant UX activities in product development, engineering, and UX organizations
Apply the HEART framework to measure user experience outcomes
Evaluate your skills and potential to be hired as a Quant UX researcher
Know what to expect during job interviews
Find examples of common Quant UX projects with shared R code and data sets
Who This Book Is For:
Practitioners and managers who seek a comprehensive guide to the new field of Quantitative User Experience Research. Readers will understand the Quant UX role, build research skills, find examples of hands-on code and analyses, learn about UX organizations and stakeholders, and receive advice on job interviews and career paths. Data scientists, social scientists, and other researchers will learn how their skills transfer to Quant UX, where they can help teams build better, more successful products.
Author(s): Chris Chapman, Kerry Rodden
Publisher: Apress
Year: 2023
Language: English
Pages: 384
Table of Contents
About the Authors
About the Technical Reviewers
Acknowledgments
Introduction
Part I: User Experience and
Introduction to Part I
Chapter 1: Getting Started
1.1 Who Are We? Why Should You Listen to Us?
1.2 What Is Different About This Book?
1.3 Who Is Our Audience?
1.3.1 A Quick Check on Your Interests
1.4 What Will You Learn?
1.5 How to Use This Book
1.5.1 Assumptions
1.5.2 A Note About Jargon
1.5.3 End of Chapter Exercises
1.6 Online Materials
1.6.1 Code and Data Sources
1.6.2 Help! Updates and Errata
1.7 Key Points
Chapter 2: User Experience and UX Research
2.1 User Experience
2.1.1 UX Roles
2.1.2 UX Design and Software Engineering
2.1.3 Product Management
2.2 UX Research
2.2.1 Categories of UX Researchers
2.2.2 The Research Lifecycle for UXRs
2.2.3 Typical Research Projects in the Product Lifecycle
2.3 Key Points
2.4 Learning More
Chapter 3: Quantitative UX Research: Overview
3.1 Quantitative UX Research
3.2 Week-to-Week Practice of Quant UX Research
3.2.1 Typical Activities in a Week
3.2.2 Common Research Questions for Quant UXRs
3.2.3 Stakeholder Questions
3.3 Varieties of Quant UXRs
3.4 Quant UXR Differences from Other Roles
3.4.1 Quant UXR vs. General UXR
3.4.2 Quant UXR vs. Mixed Methods UXR
3.4.3 Quant UXR vs. Survey Scientist
3.4.4 Quant UXR vs. Marketing Researcher
3.4.5 Quant UXR vs. Data Scientist
3.4.6 Quant UXR vs. Business or Product Analyst
3.4.7 Quant UXR vs. Research Scientist
3.4.8 Quant UXR vs. Academic Research
3.5 Will You Like a Quant UXR Role?
3.6 Key Points
3.7 Learning More
Part II: Core
Introduction to Part II
Chapter 4: UX Research
4.1 Foundational and In-Depth Skills for Quant UXR
4.1.1 “T-Shape” Skills
4.2 Focus on the User
4.2.1 Adopt the User’s Perspective
4.2.2 Assess User-Centric Variables and Outcomes
4.2.3 Answer “Why?” with a Cognitive Approach
4.2.4 Focus on Unmet Needs
4.2.5 Relate to UX Actions and Stakeholders
4.3 Research Validity
4.4 Assessing Users and Assessing Products
4.5 Research Ethics
4.5.1 Research Risks and Benefits
4.5.2 Privacy and Legal Requirements
4.5.3 Minimum Collection
4.5.4 Scientific Standards
4.5.5 Impact on Society
4.5.6 The Newspaper Test
4.6 Research Planning
4.7 Key Points
4.8 Learning More
Chapter 5: Statistics
5.1 Why Statistics?
5.1.1 Statistics vs. Machine Learning
5.2 The Foundation: Sampling and Data Quality
5.3 Core Statistical Analysis Skills
5.3.1 Exploratory Data Analysis and Visualization
5.3.2 Descriptive Statistics
5.3.3 Inferential Tests and Practical Significance
5.3.4 Fundamentals of A/B Testing
5.3.5 Linear Models
5.4 Frequently Observed Issues
5.4.1 Bad or Biased Data
5.4.2 Focusing on Discovery, Losing Sight of Decisions
5.4.3 Prematurely Assuming an Outcome of Interest
5.4.4 Interpreting Statistical Significance
5.4.5 Applying Fancy Models
5.5 Key Points
5.6 Learning More
5.7 Questions and an Exercise
Chapter 6: Programming
6.1 Overview
6.1.1 Is Programming Required?
6.1.2 What Language?
6.2 Procedural Programming Basics
6.2.1 Algorithms
6.2.1.1 Logical Steps
6.2.1.2 Control Structures
6.2.1.3 Functions
6.2.2 Data Structures
6.2.2.1 Vectors
6.2.2.2 Quick Check: What Do You Think?
6.2.2.3 Arrays and Data Frames
6.2.2.4 Hash Tables
6.3 SQL
6.4 Other Coding Topics
6.4.1 Reproducibility of Code
6.4.2 Performance and Scale
6.5 Key Points
6.6 Learning More
6.7 Exercises
Part III: Tools and
Introduction to Part III
Chapter 7: Metrics of User Experience
7.1 The HEART Framework
7.1.1 Happiness
7.1.2 Engagement
7.1.3 Adoption
7.1.4 Retention
7.1.5 Task Success
7.2 The Goals-Signals-Metrics Process
7.2.1 Goals
7.2.2 Signals
7.2.3 Metrics
7.3 Applying the Methods Together
7.4 Example: Redesigning Labels in Gmail
7.5 Lessons Learned From Experience
7.5.1 Individual Pitfalls
7.5.1.1 Not Enough Team Involvement
7.5.1.2 Starting Too Big
7.5.1.3 Underestimating the Next Steps
7.5.1.4 Too Many Metrics
7.5.2 Organizational Issues
7.5.2.1 Unwillingness to be Evaluated
7.5.2.2 Optimizing for a Single Metric
7.5.2.3 Failure to Consider Ethical Consequences
7.6 Key Points
7.7 Learning More
7.8 Exercises
Chapter 8: Customer Satisfaction Surveys
8.1 Goals of a Customer Satisfaction Program
8.2 The Components of Listening to Customers
8.2.1 Customer Population and Sample
8.2.2 Survey Mechanism
8.2.3 Ordinal Ratings
8.2.3.1 Top 2 Box and Proportional Scores
8.2.3.2 What About Net Promoter Scores?
8.2.4 Open-Ended Comments
8.2.5 Demographic and Behavioral Information
8.2.6 Don’t Compare Groups, Compare Over Time
8.2.7 Follow-up with Stakeholders and Customers
8.2.7.1 What to Report
8.2.7.2 Closing the Loop
8.3 Common Problems in CSat Analysis
8.4 Example Analysis in R
8.4.1 Initial Data Inspection
8.4.2 CSat for One Time Period
8.4.3 CSat over Time
8.4.4 Top 2 Box Proportions
8.4.5 Is CSat Changing? Initial Analysis
8.4.6 Examination by Country
8.4.7 A Better Model of CSat Change in These Data
8.5 Key Points
8.6 Learning More
8.7 Exercises
Chapter 9: Log Sequence Visualization
9.1 Example Sequence Data
9.1.1 Sunburst Chart for the Buffet Data
9.2 Sunburst Visualization of Website Data
9.2.1 Transforming the Logs to Sequences
9.2.1.1 Loading and Sessionizing the Data
9.2.1.2 Creating Sequences from the Sessions
9.2.2 Sunburst Visualization of the EPA Data
9.2.3 Next Steps in Analysis
9.3 Key Points
9.4 Learning More
9.5 Exercises
Chapter 10: MaxDiff: Prioritizing Features and User Needs
10.1 Overview of MaxDiff
10.1.1 Illustration of MaxDiff Analysis
10.1.2 Calculating Pizza Demand
10.1.3 Summary of MaxDiff Advantages
10.2 Detailed Introduction to MaxDiff Estimation
10.2.1 Common UX Topics for MaxDiff Surveys
10.2.2 Writing and Fielding a MaxDiff Survey
10.2.2.1 Writing the Question and Column Headers
10.2.2.2 Developing an Item List
Simple Items
Length
Commonality of Items
Developing the Item List
Check the Trade-offs
Maximum Number of Items
10.2.2.3 Prohibitions: Items That Can’t Appear Together
10.2.2.4 Number of Tasks
Survey Length for Average, Sample-Level Results
Survey Length for Individual-Level Results
10.2.2.5 Sample Size
10.2.2.6 Survey Fielding Methods
10.2.2.7 Mixed Qualitative-Quantitative Group Interview
10.2.3 Survey Authoring Platforms
10.2.3.1 MaxDiff in Qualtrics
10.2.4 MaxDiff and Accessibility
10.2.5 MaxDiff Statistical Models
10.2.5.1 Counts and Difference Scores
10.2.5.2 Multinomial Logit Model
10.2.5.3 Hierarchical Bayes Model
10.2.5.4 Using and Reporting the Scores
10.3 Example: Information Seeking Use Cases
10.3.1 Overview: MaxDiff for Information Seeking
10.3.2 Survey Format
10.3.3 Data Format
10.3.3.1 Data Sets in Other Formats
10.3.4 Estimation with the choicetools Package
10.3.4.1 Setup for Estimation
10.3.4.2 Check the Data
10.3.4.3 Load the Data
10.3.4.4 Estimate the Model
10.3.4.5 Plot the Results
More on HB Iterations
10.3.5 Next Steps
10.4 Key Points
10.5 Learning More
10.6 Exercises
Part IV: Organizations and
Introduction to Part IV
Chapter 11: UX Organizations
11.1 Typical UX Organization Models
11.1.1 Role-Centric Organization
11.1.1.1 Quant UXRs in Role-Centric Organizations
11.1.1.2 Notes on Success with Role-Centric Organizations
11.1.2 Product-Centric Organization
11.1.2.1 Quant UXRs in Product-Centric Organizations
11.1.2.2 Notes on Success with Product-Centric Organizations
11.2 Other Organizational Models for Quant UXRs
11.2.1 Centralized Quant UX Research Teams
11.2.1.1 Recommendations for Centralized Quant UX Research Teams
Managing a Centralized Quant Team
Recommendations to Individual Quant UXRs
11.2.2 Quant UX in a Data Science or Analytics Team
11.2.2.1 Recommendations for Quant UX in a Data Science or Analytics Team
11.3 Advice for Managers of Quant UXRs
11.3.1 Access to Stakeholders and Data
11.3.2 Shield from Immediate Requests
11.3.3 Growth Opportunity
11.3.4 Help with Determining Impact
11.3.5 Stay Out of the Way
11.4 Key Points
11.5 Learning More
Chapter 12: Interviews and Job Postings
12.1 General Quant UXR Interview Process
12.2 Two Formats for Interview Panels
12.2.1 Format 1: Interview Loops
12.2.2 Format 2: Hands-On Interviews
12.2.3 What Happens Among Interviewers?
12.2.4 Who Makes the Hiring Decision?
12.3 Before, During, and After an Onsite Interview
12.3.1 Before: What Happens at the Company
12.3.1.1 Panel Membership
12.3.2 Before: Your Preparation
12.3.2.1 Research Presentations
12.3.2.2 Requests for Analyses in Advance
12.3.2.3 Do and Don’t
12.3.3 During Interviews
12.3.3.1 A General Approach to Questions
12.3.3.2 Bring a List of Questions
12.3.4 Afterward
12.3.4.1 Thank You
12.3.4.2 Fit Calls
12.3.4.3 What You Can Negotiate
12.3.4.4 Red Flags
12.3.4.5 When the Answer is “No”
12.4 Job Postings and Applications
12.4.1 Finding Jobs
12.4.2 Additional Suggestions for Applications
12.4.2.1 Informational Interviews
12.4.2.2 Referrals and References
12.4.2.3 Cover Letter
12.4.2.4 CV vs. Résumé
12.4.2.5 Personal Websites and Open Source Projects
12.5 Key Points
12.6 Learning More
Chapter 13: Research Processes, Reporting, and Stakeholders
13.1 Initial Engagement
13.1.1 What Stakeholders Want…and What They Need
13.1.2 Focus on Decisions
13.1.3 Work Backward
13.2 Delivering Results
13.2.1 Stakeholders Are the Users of Your Research
13.2.2 Two Models: Presentations and Documents
13.2.2.1 Presentation Slide Decks
Advantages of Slides
Disadvantages of Slides
13.2.2.2 Research Report Documents
Advantages of Documents
Disadvantages of Documents
13.2.2.3 Recommendation for Reporting
13.3 Principles of Good Deliverables
13.3.1 Short and Focused on Action
13.3.2 Minimally Technical Reports
13.3.3 Remain Unbiased
13.3.4 Reproducible and Generalizable
13.4 Research Archives
13.5 Common Problems with Stakeholders
13.5.1 Lack of a Decision Criterion
13.5.2 Ad Hoc Projects
13.5.3 Opportunity Cost
13.5.4 Validation Research
13.5.5 Statistical Significance
13.5.6 Cherry Picking Results
13.5.7 Conflicting Results
13.5.8 Challenge Only If Negative (COIN)
13.6 Finding a Great Stakeholder
13.7 Key Points
13.8 Learning More
Chapter 14: Career Development for Quant UX Researchers
14.1 Elements of Career Paths in Industry
14.1.1 Job Levels
14.1.1.1 Levels, Responsibility, and Expertise
14.1.1.2 Levels and Compensation
14.1.2 Career Ladder
14.1.3 Tracks: Individual Contributor and Manager
14.1.4 Distribution of Levels
14.1.5 The Choice of IC vs. Manager
14.2 The Problems with Levels
14.3 Performance Reviews and Promotion
14.3.1 Performance Reviews
14.3.2 Impact
14.3.3 Promotion
14.4 Personal Styles and Goals
14.4.1 Maximizing vs. Satisficing
14.4.2 Builder vs. Explorer
14.5 Building Skills Throughout a Career
14.5.1 Areas for Skills Development
14.5.1.1 Quant or UX?
14.5.1.2 Qualitative Research Skills
14.5.1.3 General UX Research…and UX Generally
14.5.1.4 Programming
14.5.1.5 Statistics
14.5.2 Find Mentors
14.6 Paths for Senior ICs
14.6.1 Staff Level Pattern 1: Tech Lead
14.6.2 Staff Level Pattern 2: Evangelist
14.6.3 Staff Level Pattern 3: Strategic Partner
14.7 Key Points
14.8 Learning More
Chapter 15: Future Directions for Quant UX
15.1 Future 1: UX Data Science
15.2 Future 2: Computational Social Science
15.3 Future 3: Mixed Methods UX
15.4 Future 4: Quant UX Evolution
15.5 Learning More
15.6 Finally
Appendix A: Example Quant UX Job Description
Quantitative User Experience Researcher
Appendix B: Example Quant UX Hiring Rubrics
Rubrics to Assess Quant UXR Candidates
Appendix C: References
Index
