Second Edition. Measuring. the User Experience. Collecting, Analyzing, and Presenting Usability Metrics TOM TULLIS BILL ALBERT

Transcription

1 Second Edition Measuring the User Experience Collecting, Analyzing, and Presenting Usability Metrics TOM TULLIS BILL ALBERT

2 \ \ PREFACE TO THE SECOND EDITION ACKNOWLEDGMENTS BIOGRAPHIES xili xv xvli CHAPTER 1 Introduction What Is User Experience What Are User Experience Metrics? The Value of UX Metrics Metrics for Everyone New Technologies in UX Metrics Ten Myths about UX Metrics 11 Myth 1: Metrics Take Too Much Time to Collect 11 Myth 2: UX Metrics Cost Too Much Money 12 Myth 3: UX Metrics Are Not Useful When Focusing on Small Improvements 12 Myth 4: UX Metrics Don't Help Us Understand Causes 12 Myth 5: UX Metrics Are Too Noisy 12 Myth 6: You Can Just Trust Your Gut 13 Myth 7: Metrics Don't Apply to New Products 13 Myth 8: No Metrics Exist for the Type of Issues We Are Dealing with 13 Myth 9: Metrics Are not Understood or Appreciated by Management 14 Myth 10: It's Difficult to Collect Reliable Data with a Small Sample Size 14 CHAPTER 2 Background Independent and Dependent Variables Types of Data Nominal Data Ordinal Data Interval Data Ratio Data Descriptive Statistics Measures of Central Tendency Measures of Variability Confidence Intervals Displaying Confidence Intervals as Error Bars Comparing Means Independent Samples Paired Samples 27

3 2.4.3 Comparing More Than Two Samples 2.5 Relationships Between Variables Correlations 2.6 Nonparametric Tests The x 2 Test 2.7 Presenting your Data Graphically Column or Bar Graphs Line Graphs Scatterplots Pie or Donut Charts Stacked Bar or Column Graphs 2.8 Summary CHAPTER 3 Planning 3.1 Study Goals Formative Usability Summative Usability 3.2 User Goals Performance Satisfaction 3.3 Choosing the Right Metrics: Ten Types of Usability Studies Completing a Transaction Comparing Products Evaluating Frequent Use of the Same Product Evaluating Navigation and/or Information Architecture Increasing Awareness Problem Discovery Maximizing Usability for a Critical Product Creating an Overall Positive User Experience Evaluating the Impact of Subtle Changes Comparing Alternative Designs 3.4 Evaluation Methods Traditional (Moderated) Usability Tests Online (Unmoderated) Usability Tests Online Surveys 3.5 Other Study Details Budgets and Timelines Participants Data Collection Data Cleanup 3.6 Summary CHAPTER 4 Performance Metrics 4.1 Task Success Binary Success Levels of Success

4 / Contents :«4.1.3 Issues in Measuring Success Time on Task Importance of Measuring Time on Task How to Collect and Measure Time on Task Analyzing and Presenting Time-on-Task Data Issues to Consider When Using Time Data Errors When to Measure Errors What Constitutes an Error? Collecting and Measuring Errors Analyzing and Presenting Errors Issues to Consider When Using Error Metrics Efficiency Collecting and Measuring Efficiency Analyzing and Presenting Efficiency Data Efficiency as a Combination of Task Success and Time Learnability Collecting and Measuring Learnability Data Analyzing and Presenting Learnability Data Issues to Consider When Measuring Learnability Summary 96 CHAPTER 5 Issue-Based Metrics What Is a Usability Issue? Real Issues versus False Issues How to Identify an Issue In-Person Studies Automated Studies Severity Ratings Severity Ratings Based on the User Experience Severity Ratings Based on a Combination of Factors Using a Severity Rating System Some Caveats about Rating Systems Analyzing and Reporting Metrics for Usability Issues Frequency of Unique Issues Frequency of Issues Per Participant Frequency of Participants Issues by Category Issues by Task Consistency in Identifying Usability Issues Bias in Identifying Usability Issues Number of Participants 11R Five Participants Is Enough Five Participants Is Not Enough Our Recommendation Summary HQ

5 CHAPTER 6 Self-Reported Metrics Importance of Self-Reported Data Rating Scales Likert Scales Semantic Differential Scales When to Collect Self-Reported Data How to Collect Ratings Biases in Collecting Self-Reported Data General Guidelines for Rating Scales Analyzing Rating-Scale Data Post-Task Ratings Ease of Use After-Scenario Questionnaire (ASQ) Expectation Measure A Comparison of Post-task Self-Reported Metrics Postsession Ratings Aggregating Individual Task Ratings System Usability Scale Computer System Usability Questionnaire Questionnaire for User Interface Satisfaction Usefulness, Satisfaction, and Ease-of-Use Questionnaire Product Reaction Cards A Comparison of Postsession Self-Reported Metrics Net Promoter Score Using SUS to Compare Designs Online Services Website Analysis and Measurement Inventory American Customer Satisfaction Index OpinionLab Issues with Live-Site Surveys Other Types of Self-Reported Metrics Assessing Specific Attributes Assessing Specific Elements Open-Ended Questions Awareness and Comprehension Awareness and Usefulness Gaps Summary 161 CHAPTER 7 Behavioral and Physiological Metrics Observing and Coding Unprompted Verbal Expressions Eye Tracking How Eye Tracking Works Visualizing Eye-Tracking Data Areas of Interest Common Eye-Tracking Metrics Eye-Tracking Analysis Tips 174

6 7.2.6 Pupillary Response Measuring Emotion Affectiva and the Q-Sensor Blue Bubble Lab and Emovision Seren and Emotiv Stress and Other Physiological Measures Heart Rate Variance Heart Rate Variance and Skin Conductance Research Other Measures Summary 185 CHAPTER 8 Combined and Comparative Metrics Single Usability Scores Combining Metrics Based on Target Goals Combining Metrics Based on Percentages Combining Metrics Based on Z Scores Using Single Usability Metric Usability Scorecards Comparison to Goals and Expert Performance Comparison to Goals Comparison to Expert Performance Summary 208 CHAPTER 9 Special Topics Live Website Data Basic Web Analytics Click-Through Rates Drop-Off Rates A/B Tests Card-Sorting Data Analyses of Open Card-Sort Data Analyses of Closed Card-Sort Data Tree Testing Accessibility Data Return-On-lnvestment Data Summary 236 CHAPTER 10 Case Studies Net Promoter Scores and the Value of a Good User Experience Methods Results Prioritizing Investments in Interface Design Discussion Conclusion 243 References 244 Biographies 244

7 * Contents 10.2 Measuring the Effect of Feedback on Fingerprint Capture Methodology Discussion Conclusion 253 Acknowledgment 253 References 253 Biographies Redesign of a Web Experience Management System Test Iterations Data Collection Workflow Results Conclusions 262 Biographies Using Metrics to Help Improve a University Prospectus Example 1: Deciding on Actions after Usability Testing Example 2: Site-Tracking Data Example 3: Triangulation for Iteration of Personas Summary 270 Acknowledgments 270 References 270 Biographies Measuring Usability Through Biometrics Background Methods Biometric Findings Qualitative Findings Conclusions and Practitioner Take-Aways 275 Acknowledgments 276 References 276 Biographies 277 CHAPTER 11 Ten Keys to Success Make Data Come Alive Don't Wait to Be Asked to Measure Measurement Is Less Expensive Than You Think Plan Early Benchmark Your Products Explore Your Data Speak the Language of Business Show Your Confidence Don't Misuse Metrics Simplify Your Presentation 287 REFERENCES 289 INDEX 297 /