Remote Testing Methods and Tools

Transcription

1 Remote Testing Methods and Tools Bill Albert December 13,

2 Agenda Introduction to remote usability testing Quantitative-based tools Qualitative-based tools 2

3 About me Director, Design and Usability Center, Bentley University Experience with remote testing tools I am data driven! 3

4 I am a big fan of usability testing in the lab! 4

5 Introduction to remote usability testing

6 Defining Remote Testing Tools What do we mean by remote testing tools? A remote testing tool is any technology that allows a researcher to collect data from users about their experience in using an interface, without direct, face-to-face contact. There are two ways of organizing remote testing tools Moderated vs. Unmoderated Qualitative vs. Quantitative-based Moderated Qualitative-based Quantitative-based Unmoderated Qualitative-based Quantitative-based 6

7 Unmoderated Testing Tools Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 7

8 Our UX Toolkit 8

9 Why Should You Care? Saves time o Lab study takes 2-4 weeks from start to finish, unmoderated typically takes hours to a few days* Saves money o Participants compensation typically a lot less ($10 vs. $100) o Tools are becoming very inexpensive Reliable metrics o Only (reasonable) way to collect UX data from large sample sizes Geography is not a limitation o Collect feedback from customers all over the world Greater Customer insight o Richest dataset about the customer experience 9

10 Why is this important NOW? Questions from senior management are becoming more complex and time sensitive Traditional usability is no longer enough Push to measure the UX Convergence with market research and web analytics to paint a more complete picture of the UX Budgets/timeline constraints 10

11 Which one goes first? Lab first, then Unmoderated Identify/fix low hanging fruit, then focus on remaining tasks with large sample size Unmoderated first, then Lab Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues Generate new concepts, ideas, questions through lab testing, then test/validate online Validate attitudes/preferences observed in lab testing Collect video clips or more quotes of users to help bring metrics to life Gather all the metrics to validate design if it tests well, then no need to bring users into the lab 11

12 Quantitative-based tools

13 Unmoderated Testing Tools - Quantitative Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 13

14 Overview Common Research Questions: What are the usability issues, and how big? Which design is better, and by how much? How do customer segments differ? What are user design preferences? Is the new design better than the old design? Where are users most likely to abandon a transaction? Types of Studies: Comprehensive evaluation UX benchmark Competitive evaluation Live site vs. prototype comparison Feature/function test Discovery Typical Metrics: Task success Task time Self-report ratings such as ease of use, confidence, satisfaction SUS Click paths Abandonment 14

15 Strengths / Limitations / Myths Strengths: Comparing products Measuring user experience Finding the right participants Focusing on design improvements Insight into the user s real experience Limitations: Not well suited to rapid, iterative design Need a deep understanding of issues Studies that require long sessions Lose control over prototypes Internet access Myths: Only test with websites Very expensive Only gather quantitative data A lot of noise in the data Same as any market research study Comparing products 15

16 Full Service Tools 16

17 Self Service Tools 17

18 Card Sorting / IA Tools 18

19 Online Surveys 19

20 Click and Mouse Tools 20

21 Qualitative-based tools

22 Unmoderated Testing Tools - Qualitative Full-service Self-service Unmoderated tools Qualitativebased Quantitativebased Card Sorting/IA Surveys Click/Mouse Video tools Reporting Expert Reviews 22

23 Overview Common Research Questions: What are the big pain usability pain points, and wins? How do users think, and feel about the design? Are we on the right (design) track? What is the overall experience like? Why might users abandon a transaction? Types of Studies: Low fidelity prototype High fidelity prototype Competitive evaluation Comprehensive evaluation Feature/function test Discovery Common Metrics: Frequency of issues Verbatim comments Task success (via video) Task time (via video) 23

24 Video Tools 24

25 Report Tools 25

26 Expert Review 26

27 Additional Resources Online: LinkedIn Group: Bentley Design and Usability Center 27

28 Thank You! Bill Albert, Ph.D. Director, Design and Usability Center