1 How to Run an End User Feedback Session on Your Product Teena Singh, Principal Product Manager, HCM User Experience Lulit Bezuayehu, User Experience Architect, HCM User Experience 1 Copyright 2012, 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
2-Hour User E xperience Workshop Hands-On Session Steps for running a user feedback session Example materials to help you run your own usability study Do s and don ts of moderating a feedback session Key Takeaways How to collect valuable user feedback on your application Tips & tricks for running a successful usability test 2
Teena 3
UX Direct FREE Toolkit: Usability Tools How to conduct usability testing Usability Testing Fact sheet How to conduct Heuristic Evaluation Usability Rating Scales CIF Usability Checklist 4
User-Centered Implementation Cycle - Validate 5 Copyright 2012, Oracle and/or its affiliates. All rights reserved.
What is Usability Testing? 6
Question 1: Usability Study Experience What is your experience with usability testing? A. Ran a usability study B. Was a Participant for a usability study C. Observed a usability study D. No, I have not been exposed to usability testing at all 7
What is Usability Testing? Usability: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. (Source: ISO 9241-11) Usability testing is a technique used to evaluate a product by testing it with representative users. (Source: usability.gov) 8
Definition of Usability testing Essential features Measuring a) Efficiency rate/speed completing a task Efficiency is doing things right b) Effectiveness ability to complete the task Effectiveness is doing the right things c) Satisfaction of users - delight Gives direct input from real users Iterative in nature 9
Benefits of Usability testing ORACLE Do it right the first time. Reduce costly errors CUSTOMER Save time & cost of implementation Less customer & Oracle liability Understand user patterns & habits Users less likely to complain Improves the product End-user completes tasks faster More Oracle product sold Compliance w/ gov. regs, contracts, unions- Decrease customer support, training, & docs Increase customer satisfaction. Fewer errors- managers are happier. Increased customer satisfaction & loyalty Avoid fines & penalties Cheaper total cost of ownership for customer Teena 10
What is a user feedback session? 11
Discount Usability User Feedback Session What is Discount Usability: An approach to usability that seeks ways to optimize usability methods for cost-effectiveness. - User Feedback Session is an example of Discount Usability Data Collection Method - We will focus on this methodology as an example and give you the tools and knowledge to run this kind of usability activity 12
What is a User Feedback Session? Task Based Qualitative Assessment Method Task-based method for evaluation of products at an early stage of development Conducted with a small group of participants from a target user group Used to validate designs Performance data is typically not gathered for this kind of evaluation 13
What resources do you need? Required and Recommended Required: - A computer connected to the website, prototype (even paper-based) or code you are testing - A well thought out script of tasks and scenarios that you will ask the user to complete - A Moderator to ask questions and request tasks - MOST IMPORTANTLY: An end user participant to provide feedback Recommended: - A note taker to take notes - Resources to either record audio or video of the test to record responses and reactions and analyze website navigation 14
What resources do you need? How many end users do you need? Specify key characteristics of user types when you recruit Recommended: 8-10 Usability Study for HCM: Time and Labor: 5 employees 3 managers Provide an incentive 15
Non-Lab User Feedback Sessions The method is flexible with respect to: - Where the study is conducted, (in the lab, at a customer location, at a trade show, or through remote collaboration tools). 16
Question 2: User Feedback Session Which resource is NOT REQUIRED to run a user feedback session? A. Well thought out script of tasks and scenarios B. Computer connected to the website or prototype you are testing C. Moderator D. Participant E. Note taker 17
Planning Your User Feedback Session 18
Planning Your User Feedback Session - Step 1: Determine Design Focus - Step 2: Assemble your Moderator Script - Step 3: Create Click Path for Prototype - Step 4: Review Moderator Script with your Project Team - Step 5: Run a Pilot Test 19
Step 1: Determine Design Focus Areas to Review with Participants 1: Identify key scenarios and tasks that you want to validate with customer 2: Select prototype to show customer based on task frequency, importance, priority, etc. 3: Determine areas of the prototype that you have design questions about or feature/functionality that you need to verify 4: Translate the design-focused questions/areas into tasks and questions for participants. 20 Oracle Confidential
Step 1: Determine Design Focus Choosing what features to test Important features to test are typically New Used often Problematic, based on customer s feedback To gain consensus on design direction Likely to cause the product to fail if used incorrectly Commonly used job tasks 21 Oracle Confidential
Step 1: Determine Design Focus Identify Key Scenarios & Tasks Identify key scenarios (or high level use cases) Talk through the key scenario that you will describe to your customers. Make it simple. A quick story that you can always refer to when showing your prototype. List core tasks that your prototype supports Determine which tasks you want to validate! You won t be able to do all of them. Which tasks are the most important or frequently completed by users? 22 Oracle Confidential
S tep 2: Assemble your Moderator S cript What is the Moderator Script? - A guide that helps to steer the conversation A cheat sheet for the discussion with the customer - Outlines the test protocol - Includes the Participant scenario and task list - Includes all follow up questions you want to ask each participant - Helps you ask all customers the same questions so that you have a basis for comparison. - You do not need to stick to it word for word 23
Step 2: Assemble your Moderator Script Guide format Section 1: Introduction/Introductory Script - Test Objective - Outlines the process for the Participant Section 2: Describe the overall background scenario Section 3: First task - Task Scenario - Task-specific questions Section 4: Second task Section 5: Third task, etc. Section 6: Any overall or final questions Section 7: Wrap-Up 24
Step 2: Assemble your Moderator Script Introduction/Introductory Script The introduction to the session should contain the following elements: Introduce yourself, your role, and any additional team members attending Have participants introduce themselves and their roles Set expectations by describing the format of the meeting and what will be expected of participants Let participants know that rather than just show them a demo we will have them test drive the design by describing how they would attempt to complete the tasks Let participants know that you will be asking questions after each task about what they think of what they are seeing 25 Oracle Confidential
Step 2: Assemble your Moderator Script Introduction/Introductory Script Emphasize that we are looking for ways to improve the design by having customer reviews and we want their honest feedback If recording the session, informed consent is required. You must ask their permission to record the session You must read or show the Safe Harbor Statement to participants This is fully scripted. See template. 26 Oracle Confidential
Step 2: Assemble your Moderator Script Describe the overall background scenario Why set up the scenario? To provide context for what the participants are about to see Allows you to refer back to the scenario throughout the session Initial scenario set-up Verify user profile(s) and scenario with customer Describe the scenario that lead the end-user up to the first step of the flow State any assumptions or dependencies relevant to this flow 27 Oracle Confidential
Step 2: Assemble your Moderator Script Describe the overall background scenario Overall scenario should focus on the user role Write the scenario in a way that reflects test participants real life work situations (e.g. employee name, employer name, location) A good scenario is Clear (all participants will understand it) Short Include relevant information Gives background and context to the whole script (and not the first task only) 28 Oracle Confidential
Step 2: Assemble your Moderator Script Describe the overall background scenario Sample Background Scenario: You are Albert Katz, a Quality Assurance (QA) Engineer with six years of experience. You aren t happy with your current position, so you are thinking about changing jobs. You have heard great things about Vision Corporation, and have navigated to the Vision Corporation careers page from a link on the company s website. 29
Step 2: Assemble your Moderator Script Tasks Writing tasks Tasks need to be focused on one specific feature or a cluster of features Reasonable Represent a realistic situation Write the task in a way that is related to the test participant s work Specific Avoid using terms that exist on the interface (don t lead) Sequential (maintain appropriate flow) 30 Oracle Confidential
Step 2: Assemble your Moderator Script Tasks Writing tasks S ample Tas k S cenario: You want to see if Vision Corporation is currently hiring QA Engineers. You are willing to relocate, so you want to find out about QA Engineer openings in San Francisco. - S ample Tas k 1: Describe how you would find QA Engineer jobs at Vision Corporation in San Francisco. 31
Step 2: Assemble your Moderator Script Writing Tasks with Design Focus in Mind UI Layout If you want to know If participants correctly interpret the purpose of the page based on the UI layout Ask What is the primary task to be accomplished on this page? What is the first thing you would do on this page? If users can find features on the page If you were to do [task], where would you go? 32 Oracle Confidential
Step 2: Assemble your Moderator Script Writing Tasks with Design Focus in Mind Is important information missing? If you want to know if the design provides all the information users would need to complete typical tasks Ask Would you have all the information you need to complete [task]? If not, what else would you need? Are there any analytics or metrics you would like to see on this screen? Validation of new features If you want to know how a new feature would be received by customers Ask Would this feature be useful to you? Would you use this [new feature]? How so? 33 Oracle Confidential
Step 2: Assemble your Moderator Script Writing Tasks with Design Focus in Mind Information Architecture If you want to know if data is categorized in logical groupings Ask Do these groupings [point to a set of items] make sense to you? If not, how would you want them to be grouped? What s missing? Navigation If you want to know do participants understand the navigation structure if participants have the navigation options they would need Ask How would you expect to get from here to the next page? How would you launch [this feature] from here? What other tasks or parts of the application would you want to have quick access to from this page? Are there related tasks that you would go to from here? 34 Oracle Confidential
Step 2: Assemble your Moderator Script Writing Tasks with Design Focus in Mind Terminology If you want to know if the terminology used makes sense and is intuitive Ask What is the typical wording you would use for [concept]? Mental Model If you want to know if the design matches the user s mental model Ask What would you expect to see next after [taking an action]? 35 Oracle Confidential
Step 2: Assemble your Moderator Script Writing Tasks with Design Focus in Mind Expectations of how UI elements and widgets work If you want to know if users understand how to interact with UI elements/widgets Ask What would you expect this <UI widget> to do? 36 Oracle Confidential
Step 2: Assemble your Moderator Script Any overall or Final Questions Ask any questions overall or final questions that you might have - Ask each participant the same questions - What did you think of the Prototype overall? - Can you tell me your top likes and dislikes of the designs you saw today? - How does this compare to what you are currently using? 37 Oracle Confidential
Step 2: Assemble your Moderator Script Wrap-Up Is there anything we missed that you want to talk about About what you saw today Or the product overall Thank the Participant - Give them a gift/incentive for their time 38 Oracle Confidential
Quiz Question 3: Can you find what is WRONG? Background Scenario: You are Paul McCartney and often to go Amazon.com to purchase household products. Task Scenario: He recently purchased a new digital camera on Amazon.com and you are disgusted with it. - Task 1: Visit the uniform resource locator for Amazon and see if there is a link for Returns are Easy. Hint: The link is towards the bottom of the page. - Task 2: Determine the number of total products that Amazon sells. 39 Oracle Confidential
Discussion Points (Results) Did you find all these things that were WRONG? The Scenario and tasks were not written using the you perspective Name should have no bias Strong phrasing like disgusted detracts from the task Uniform resource locator is using technical jargon/language that an end user may not understand Providing too much information Link name Returns are easy gives away answer to user False scenarios not recommended 40 Oracle Confidential
Step 3: Create Click Path for the Prototype Create a Click Path document Identify the actions the user should take to successfully complete the tasks and what the state of the prototype should be at each point along the way. Share the click path with the Note Taker. It is a good tool to help the Note Taker stay oriented as the participant works through the tasks. 41 Oracle Confidential
Step 3: Create Click Path for the Prototype - Actions to successfully complete the tasks - Documents the state of the prototype/application as the user clicks through it - Share the click path with the Note Taker - Helps the Note Taker stay oriented 42 Oracle Confidential
Step 4: Review Moderator Script Solicit feedback on Moderator Script with project team. Ask your Project Team if they want to add any task-based or overall questions Finalize Moderator Script 43 Oracle Confidential
Step 5: Run a Pilot Test Pilot the session once or twice with an internal Oracle employee, not involved in the task creation process to get an idea of the session length and clarity of the tasks Based on Pilot Test finalize task and questions in your Moderator Script 44 Oracle Confidential
Conducting Your Us er Feedback S es s ion 45
Conducting User Feedback Session Prepare for your test Set up testing area - Have the prototype ready - Make sure your recording device is ready - Have the Participant Tasks and Scenarios printed out and ready Prepare yourself mentally for the test - Try and get a good night of sleep - Review Moderator Script one last time - Remember to: Be ready to expect the unexpected 46
Conducting User Feedback Session Preliminary Details Meet the participant - Relax, introduce yourself, make small talk and help the Participant feel at ease - Make eye contact, thank them for coming - Offer the Participant a refreshment Transition to the testing station - Explain how the station is setup - Position yourself and the Participant 47
Conducting User Feedback Session Go through the Introductory Script Allows you to go systematically through details - Introductions - Outlines the structure of the test and general protocol Test Objective Procedure - Explain and demonstrate thinking aloud - Advise the Participant to be Honest - We are not testing the participant but rather the design - Ask them if they have any questions - Confirm if you have permission to record 48
Conducting User Feedback Session Introductory Script: Explain Thinking Aloud Concept As you work with the product, we would like you to think aloud about what you are experiencing, what you like and dislike about it, what your expectations are, and what you are trying to do with the prototype. Please continue working as you think aloud however; don t stop working to provide comments. I will give you the opportunity to provide more detailed comments at the end of each task. 49
Conducting User Feedback Session Thinking Aloud Concept: Advantages Preference information in real time Improves focus and concentration for participants Clues about why something is confusing - Identifies the source of confusion 50
Conducting User Feedback Session Thinking Aloud Concept: Disadvantages Not a common behaviour Increases mindfulness 51
Conducting User Feedback Session Run the Participant Double check to see if your Note taker is ready Ask Participant to Read Background Scenario aloud Ask participant to read first task aloud Confirm with the participant that they understand the task - Could you go ahead and summarize the task please? - Do you have any questions about what the task is asking you to do? 52 Copyright 2012, 2013, Oracle and/or its affiliates. All rights reserved.
Conducting User Feedback Session Run the Participant Remind the Participant to Think Aloud - You may go ahead and start the task. - Remember to think aloud Ask the Participant to tell you when they feel they have completed the task - Do you believe you have completed the task? Moderate only when Necessary - That is helpful feedback Ask Task specific Follow-up Questions 53
Conducting User Feedback Session Post Test Activities Ask any questions overall or final questions that you might have - Ask each participant the same questions - What did you think of the Prototype overall? - Can you tell me your top likes and dislikes of the designs you saw today? - How does this compare to what you are currently using? Thank the Participant - Give them a gift/incentive for their time 54
Conducting User Feedback Session Prepare for your next Participant Tests should be scheduled so there is at least a half hour break between sessions Get away from the test area, if only briefly Reset the prototype or application Save the recording and notes Prepare for your next participant and test 55
Question 4: Usability Study Experience Poll 4: The think aloud methodology enables the Participant to give you real time preference data. True or False 56
Do s and Don ts of Moderating a User Feedback Session 57
Guidelines for Moderator: Be Unbiased/Impartial Give the same feedback for negative and positive comments on the product - That is helpful feedback. Thank you. You don t have to answer every question React to mistakes in exactly the same way as you do to correct behavior 58
Guidelines for Moderator: Let the Participant Speak Minimize interruptions - Participants should be speaking 80% of the time - When in doubt, wait before saying anything Saying nothing is still communicating Your response should be unbiased if the Participant makes a comment on the application - That is helpful feedback. 59
How to Deal with: Participant Frustration Since frustration is usually related to a critical deficiency of the product, if you move on too quickly you can miss an opportunity to understand how the Participant learns to use the product The key is to find just that point where the Participant is frustrated but still willing to try Encourage the Participant to verbalize what is happening. Often, if the Participant is allowed to vent while performing the task, their frustration can be minimized I can see you are getting frustrated, but would you please try a bit longer. 60
Guidelines for Moderator: Be aware of the effects of your voice and body language Never indicate through your speech or mannerisms that you approve or disapprove of any actions or comments offered by a particular Participant - Moving closer to someone indicates acceptance of what the person is saying, moving further away indicates rejection - Raising the pitch of your voice usually signifies agreement, while lowering it communicates the opposite Record your pilot test so you can observe/listen to your behavior Remember: Its good to see Participants have problems with the application during testing situation. It provides and opportunity to identify and correct issues in the design. 61
Guidelines for Moderator: Make sure that the Participant is really finished task Very often, especially if a Participant is unsure of a task, there is a moment of indecision after completing the task when a Participant is not sure if he or she has completed the task correctly If you jump in too soon, because the Participant is finished, you are confirming that they completed the task correctly and might be cutting into their moment of indecision If you are sitting close to the Participant be careful not to mark the task as completed which can be an inadvertent signal to them The best way to prevent this problem all together is to have the Participant signal when they are finished as part of the protocol 62
Questions 5: Usability Study Experience Do you think you will run a usability study in the next 3-6 months? Please provide an open ended response 63
User Feedback S ession R ole Playing E xample 64
User Feedback Session Role Playing Example 65
Example 1: Role Play Example Background/Use Case Scenario You are Terry Luca and work as a VP of Information Systems for a small college. You are currently thinking of conducting some user research activities to ensure that the link to your campus portal is visible and the content you have in it is appropriate for your customers. 66
Example 1: Role Play Example Scenario A colleague of yours noted that they had seen an article in the July 2010 Oracle s Customer Participation Newsletter that discussed how MICA (Maryland Institute College of Art) conducted user research to better understand the needs of their users when redesigning their web portal. 67
Example 1: Role Play Example Task 1 Find the July 2010 Newsletter that is archived on the Usable Apps website 68
Example 1: Role Play Example Task 2 Task 2: Who wrote the article and see if you can contact them directly. 69
Example 2: Role Play Example Background/Use Case Scenario You are Terry Luca and work as an IT Professional at a small college. You have been asked by your manager to see if Oracle is doing any usability testing on HCM mobile applications. 70
Example 2: Role Play Example Scenario A colleague of yours noted that they had seen an article in the September 2011 Oracle s Customer Participation Newsletter that discussed how Oracle conducted user research at a user conference to better understand the needs of their mobile users. 71
Example 2: Role Play Example Task 1 Find the September 2011 Newsletter that is archived on the Usable Apps website 72
Example 2: Role Play Example Task 2 Task 2: Who wrote the article and see if you can contact them directly. 73
Discussion Points Moderator reminded Participant to think aloud When Participant asked if she could find the Newsletters on the Subscribe to Our Newsletter Moderator did not give her the exact answer When Participant got frustrated Moderator did not immediately jump in to help her Moderator waited until Participant stated she had completed the task 74
Q&A 75
76 Copyright 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8