Remote Testing Methods and Tools Bill Albert December 13, 2011 walbert@bentley.edu 781.891.2500 www.bentley.edu/usability
Agenda Introduction to remote usability testing Quantitative-based tools Qualitative-based tools 2
About me Director, Design and Usability Center, Bentley University Experience with remote testing tools I am data driven! 3
I am a big fan of usability testing in the lab! 4
Introduction to remote usability testing
Defining Remote Testing Tools What do we mean by remote testing tools? A remote testing tool is any technology that allows a researcher to collect data from users about their experience in using an interface, without direct, face-to-face contact. There are two ways of organizing remote testing tools Moderated vs. Unmoderated Qualitative vs. Quantitative-based Moderated Qualitative-based Quantitative-based Unmoderated Qualitative-based Quantitative-based 6
Unmoderated Testing Tools Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 7
Our UX Toolkit 8
Why Should You Care? Saves time o Lab study takes 2-4 weeks from start to finish, unmoderated typically takes hours to a few days* Saves money o Participants compensation typically a lot less ($10 vs. $100) o Tools are becoming very inexpensive Reliable metrics o Only (reasonable) way to collect UX data from large sample sizes Geography is not a limitation o Collect feedback from customers all over the world Greater Customer insight o Richest dataset about the customer experience 9
Why is this important NOW? Questions from senior management are becoming more complex and time sensitive Traditional usability is no longer enough Push to measure the UX Convergence with market research and web analytics to paint a more complete picture of the UX Budgets/timeline constraints 10
Which one goes first? Lab first, then Unmoderated Identify/fix low hanging fruit, then focus on remaining tasks with large sample size Unmoderated first, then Lab Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues Generate new concepts, ideas, questions through lab testing, then test/validate online Validate attitudes/preferences observed in lab testing Collect video clips or more quotes of users to help bring metrics to life Gather all the metrics to validate design if it tests well, then no need to bring users into the lab 11
Quantitative-based tools
Unmoderated Testing Tools - Quantitative Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 13
Overview Common Research Questions: What are the usability issues, and how big? Which design is better, and by how much? How do customer segments differ? What are user design preferences? Is the new design better than the old design? Where are users most likely to abandon a transaction? Types of Studies: Comprehensive evaluation UX benchmark Competitive evaluation Live site vs. prototype comparison Feature/function test Discovery Typical Metrics: Task success Task time Self-report ratings such as ease of use, confidence, satisfaction SUS Click paths Abandonment 14
Strengths / Limitations / Myths Strengths: Comparing products Measuring user experience Finding the right participants Focusing on design improvements Insight into the user s real experience Limitations: Not well suited to rapid, iterative design Need a deep understanding of issues Studies that require long sessions Lose control over prototypes Internet access Myths: Only test with websites Very expensive Only gather quantitative data A lot of noise in the data Same as any market research study Comparing products 15
Full Service Tools 16
Self Service Tools 17
Card Sorting / IA Tools 18
Online Surveys 19
Click and Mouse Tools 20
Qualitative-based tools
Unmoderated Testing Tools - Qualitative Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 22
Overview Common Research Questions: What are the big pain usability pain points, and wins? How do users think, and feel about the design? Are we on the right (design) track? What is the overall experience like? Why might users abandon a transaction? Types of Studies: Low fidelity prototype High fidelity prototype Competitive evaluation Comprehensive evaluation Feature/function test Discovery Common Metrics: Frequency of issues Verbatim comments Task success (via video) Task time (via video) 23
Video Tools 24
Report Tools 25
Expert Review 26
Additional Resources Online: www.measuringux.com www.measuringusability.com http://usability.bentley.edu http://www.usefulusability.com/24-usability-testing-tools/ http://remoteresear.ch/tools/ Follow me: @UXMetrics @BentleyDUC LinkedIn Group: Bentley Design and Usability Center 27
Thank You! Bill Albert, Ph.D. Director, Design and Usability Center walbert@bentley.edu 1.781.891.2608 http://usability.bentley.edu