Usability Testing with Howard Kiewe Seven Infrequently Asked Questions 2006, Howard Kiewe The Seven IAQs We will cover the following questions: 1. What is a Usability Test? 2. Who are the Test Participants? 3. Why Usability Test? 4. When to Usability Test? 5. Why Doesn t Everyone Test? 6. Where to Usability Test? 7. How to Usability Test? Usability Testing, 1/26/2006 McGill University HCI 304-424B 2 Howard Kiewe 1
1. What is a Usability Test? The evaluation of a product s usability through direct observation of user behavior during a structured task. Also called a usability evaluation or simply an evaluation. Key Concepts: User not Usability Expert (unlike a heuristic evaluation) User Behavior not User Opinion (unlike a survey) Direct Observation not Indirect Obser. (unlike a server log) Structured Task not Any Task (unlike an ethnography) Usability Testing, 1/26/2006 McGill University HCI 304-424B 3 2. Who are the Test Participants? A representative sample of the user population Usability Testing, 1/26/2006 McGill University HCI 304-424B 4 Howard Kiewe 2
3. Why Usability Test? Reasons Your users are not like you Your users cannot explain their usability issues Goals Find usability issues Verify that a user interface (UI) meets its usability goals Choose between competing designs Bring all members of the development team on board Usability Testing, 1/26/2006 McGill University HCI 304-424B 5 4. When to Usability Test? Business Objectives User Research UX Requirements Prerequisites User experience (UX) requirements Prototype (UX design) Phases Low fidelity High fidelity UX Evaluation UX Design Usability Testing, 1/26/2006 McGill University HCI 304-424B 6 Howard Kiewe 3
5. Why Doesn t Everyone Test? It takes time and costs money: Step Hours Design test plan & materials 32 Design test environment 8 Run pilot test 8 Revise test tasks/materials 8 Run test/collect data 32 Summarize data 16 Document/present results 40 Total 144 Why invest the resources when: My users are like me or My users can explain their usability issues *Adapted from Mayhew (1999) The Usability Engineering Lifecycle Usability Testing, 1/26/2006 McGill University HCI 304-424B 7 6. Where to Usability Test? User s Environment High ecological validity Poorly controlled Inexpensive Convenient for the user Potential interruptions Videotaping inconvenient Third-party observations intrusive Usability Lab Low ecological validity Well controlled Expensive Inconvenient for the user No interruptions Videotaping convenient Third-party observations unintrusive Usability Testing, 1/26/2006 McGill University HCI 304-424B 8 Howard Kiewe 4
7. How to Usability Test? Seven Steps: Complete Prerequisites Plan Develop Materials Pilot Test Recruit Participants Test Summarize, Analyze, & Report Usability Testing, 1/26/2006 McGill University HCI 304-424B 9 Prerequisites & Plan Prerequisites 1. Usability requirements & goals 2. Prototype Plan 1. Choose test focus Ease or learning vs. Ease of use 2. Choose a user focus Cannot test everyone 3. Design a test task Cannot test all tasks Usability Testing, 1/26/2006 McGill University HCI 304-424B 10 Howard Kiewe 5
Develop Materials 1. Observer briefing 2. User introduction 3. Informed consent 4. Non Disclosure Agreement 5. Pretest questionnaire 6. User or training documentation (optional) 7. Test tasks 8. Data collection sheet 9. Posttest questionnaire 10. Test script Usability Testing, 1/26/2006 McGill University HCI 304-424B 11 Develop Materials (2) User introduction Thank you & about the product We are testing the interface, not you It s weird. I m just going to observe. Data collection sheet Action, User comments, Observations Posttest questionnaire (mostly affect, usually a Likert Scale) The instructions and prompts are helpful: Strongly Agree 1 2 3 4 5 Strongly Disagree Standard instruments available, ie: Software Usability Measurement Inventory (SUMI) Usability Testing, 1/26/2006 McGill University HCI 304-424B 12 Howard Kiewe 6
Pilot Test 1. Recruit pilot user 2. Set up test environment 3. Run pilot test 4. Revise materials & environment 5. Adjust timing Don t skip the pilot! Usability Testing, 1/26/2006 McGill University HCI 304-424B 13 Recruit Participants 1. 3-10 users per test run 2. 1-2 hours per test, 30 min between them. 3. Consider participant motivation 4. Consider an incentive 5. Recruitment is work. Be persistent. Usability Testing, 1/26/2006 McGill University HCI 304-424B 14 Howard Kiewe 7
Test Facilitator Role Consider think aloud, pair testing, or posttest review Don t lead or help Ask questions to gain insight Avoid distracting users Ask posttest questions Observer Role Take notes Usability Testing, 1/26/2006 McGill University HCI 304-424B 15 Questions to Gain Insight Ask What are you thinking? Is that what you expected? What would you like to accomplish? Avoid Why (encourages justification) Usability Testing, 1/26/2006 McGill University HCI 304-424B 16 Howard Kiewe 8
Summarize, Analyze, & Report Summarize & Analyze Count # of errors, time per task or per error Use only descriptive statistics like averages Report List elements that work List issues, including frequency & severity Make suggestions for improvement Usability Testing, 1/26/2006 McGill University HCI 304-424B 17 Exercise: 1) Scripting Develop Short Script 1. User introduction (1 paragraph) 2. Test tasks (1 paragraph) 3. Data collection sheet (action, user comment, observations) 4. Posttest questionnaire (a few Likert questions) For These Goals 1. Quickly and easily define a preset region 2. Quickly and easily navigate to a preset region 3. Rate the camera as easy to use, efficient, and enjoyable Usability Testing, 1/26/2006 McGill University HCI 304-424B 18 Howard Kiewe 9
Exercise: 2) Testing Facilitator Role Read introduction & test tasks Use paired testing Ask questions to gain insight Ask posttest questions Observer Role Take notes on data collection sheet Usability Testing, 1/26/2006 McGill University HCI 304-424B 19 Exercise: 3) Reporting To Class: List elements that work List issues, including frequency & severity Make suggestions for improvement Usability Testing, 1/26/2006 McGill University HCI 304-424B 20 Howard Kiewe 10
Thank you! Contact Information Howard Kiewe User Experience Consultant Research Associate Montreal, Canada McGill University hkiewe@videotron.ca hkiewe@cim.mcgill.ca (514) 485-6373 (514) 398-4400 x 089888 On the Web www.howardkiewe.com www.user-experience.info Usability Testing, 1/26/2006 McGill University HCI 304-424B 21 References & Reading BSI. (1998). BS EN ISO 9241-11:1998 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs). Part 11: Guidance on usability. London: British Standards Institution. The ISO standard that defines usability and how it should be integrated into computer products. Kiewe, H. (2006). The UXE White Paper: User Experience Engineering Essentials [Electronic Version]. user-experience.info, 2006. Retrieved September 15, 2006 from http://www.user-experience.info/uxe. User experience engineering (UXE) is a structured research, design, and evaluation process whose goal is to make user interactions with a product or service easy, efficient, and enjoyable. This white paper summarizes the UXE process and evaluates its business case. Mayhew, D. J. (1999). The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design San Diego: Morgan Kaufmann. Detailed presentation of the user-centred design process. NCI. (2006). Research-Based Web Design & Usability Guidelines. Retrieved March 25, 2006, from http://usability.gov/guidelines/. Lists web design and usability principles and rates them according to the strength of supporting research. A quick way to learn about and evaluate design principles. Updated regularly. Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: Wiley. A thorough usability testing handbook. Stone, D. L., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User Interface Design and Evaluation. San Francisco, CA: Elsevier/Morgan Kaufmann. A recently published textbook on user interface design from a user-centred and human factors perspective. Broad and current coverage in an accessible style. Usability Testing, 1/26/2006 McGill University HCI 304-424B 22 Howard Kiewe 11