COMP 3220 Human-Computer Interaction Usability Testing Questions 4.2. Usability Testing Alexander Nikov 1. What is a Usability Test? 2. Who are the Test Participants? 3. Why Usability Test? 4. Why Doesn t Everyone Test? 5. When to Usability Test? 6. Where to Usability Test? 7. How to Usability Test? 1. What is a Usability Test? The evaluation of a product s usability through direct observation of user behavior during a structured task. Also called a usability evaluation or simply an evaluation. Key Concepts: User not Usability Expert (unlike a heuristic evaluation) User Behavior not User Opinion (unlike a survey) Direct Observation not Indirect Obser. (unlike a server log) Structured Task not Any Task (unlike an ethnography) 2. Who are the Test Participants? A representative sample of the user population
3. Why Usability Test? 4. When to Usability Test? Reasons Your users are not like you Your users cannot explain their usability issues Goals Find usability issues Verify that a user interface (UI) meets its usability goals Choose between competing designs Bring all members of the development team on board 5. Why Doesn t Everyone Test? 6. Where to Usability Test?
7. How to Usability Test? Prerequisites & Plan Seven Steps: 1. Complete Prerequisites 2. Plan 3. Develop Materials 4. Pilot Test 5. Recruit Participants 6. Test 7. Summarize, Analyze, & Report Develop Materials 1. Observer briefing 2. User introduction 3. Informed consent 4. Non Disclosure Agreement 5. Pretest questionnaire 6. User or training documentation (optional) 7. Test tasks 8. Data collection sheet 9. Posttest questionnaire 10. Test script Develop Materials (cont.)
Pilot Test 1. Recruit pilot user 2. Set up test environment 3. Run pilot test 4. Revise materials & environment 5. Adjust timing Don t skip the pilot! Recruit Participants 1. 3-10 users per test run 2. 1-2 hours per test, 30 min between them. 3. Consider participant motivation 4. Consider an incentive 5. Recruitment is work. Be persistent. Test Questions to Gain Insight Facilitator Role Consider think aloud, pair testing, or posttest review Don t lead or help Ask questions to gain insight Avoid distracting users Ask posttest questions Observer Role Take notes
Summarize, Analyze, & Report Formal Usability Testing Summarize & Analyze Count # of errors, time per task or per error Use only descriptive statistics like averages Report List elements that work List issues, including frequency & severity Make suggestions for improvement Users given everyday webpage tasks to complete Users are asked to speak aloud, explaining their thought process Users actions and audio feedback are recorded Post-test questionnaire Analysis of usability test results Categorize and identify problems with the website Identify and propose possible solutions Sample Usability Testing Tasks Usability Evaluation Tools 1. Find and write the price of a Microsoft Xbox 360 Console: $3500.00 2. Find and write the ESRB Rating of Spiderman 3 Nintendo Wii : Teen 3. Find and write the best resolution to view this website:..x 4. Find and write the last date the website was updated:. 5. Find and write the total playing time for the Gears of War trailer:.minutes..seconds 6. Click on the Cool Links tab, locate the link for X-Play and visit their website. 7. Buy a Playstation 3 and Gears of Wars video game. Heuristics and a template so that you can evaluate usability aspects of different kinds of systems http://www.id-book.com/secondedition/catherb/index.htm Web-Based User Interface Evaluation with Questionnaires http://www.acm.org/~perlman/question.html Online Survey Generator: students can create their own surveys http://www.ifsm.umbc.edu/djenni1/osg/ Pew Internet & American Life Survey http://www.pewinternet.org/
Interactive Heuristic Evaluation Toolkit http://www.id-book.com/secondedition/catherb/index.htm Suggested heuristics for Websites, All people and All types applications (cont.) Sample Usability Testing Survey
Sample Usability Testing Survey (cont.) Two-room usability lab at UWI One-way mirrors for overview Dome cameras with remote pan/tilt/zoom Live observation + digital video recording of multiple video streams How to use The Observer XT Possibilities with collected data Create a data selection and: Visualize behaviors in time Calculate statistics Create highlight videos Export data as ASCII file Systematic Observation Recording who does what, when, where and with whom Who - Subject (one or more) What - Behavior When Time (recorded automatically) Where - Modifier With whom - Modifier
How to use The Observer XT FOUR steps Observer XT - User interface How to use The Observer XT STEP 2 - Create coding scheme Subjects Settings Subjects (names of subjects) Behaviors (actions that you want to observe) Modifiers (specify subjects and behaviors more precisely) Connections (subjects/behaviors - modifiers) Channels (subjects behaviors) Check coding scheme
Behaviors Mutually exclusive behaviors How to use The Observer STEP 3 - Collect Data Score data after creating the coding scheme
Live and Offline scoring Scoring Behaviors - with coding scheme OBSERVER XT Video recorded 2-40
How to use The Observer STEP 4 - Data Analysis Data Analysis