Ux test SU Bookstore SAINT CHRISTOPHER MONTERROSO 2012 By: St. Christopher Monterroso
Contents INTRODUCTORY SUMMARY 2 The Test About The Seattle University Online Bookstore Test Plan 4 Purpose, Goals, Location & Setup Questions Participants Roles, Method Schedule Measures, Report Contents Project Schedule Deliverables Test Scripts / Materials 12 Observer Guidelines Orientation Script Pre-test & Post-test Questionnaire Task Scenarios Results 16 Pre-test Tabled Post-test Tabled Time on Tasks Findings 24 Conclusion 25 Appendix 26 St. Christopher Monterroso Usability Testing 1
Intro Seattle University, founded in 1891, is a Jesuit Catholic university located on 50 acres in Seattle s Capitol Hill neighborhood. More than 7, 700 students are enrolled in undergraduate and graduate programs within eight different schools and colleges. Seattle University allows its students the option of purchasing their textbooks online. THE TEST Usability testing is essential for determining whether or not a website will succeed in capturing and retaining a strong user base. It is exponentially better for a website to be tested earlier on, however it is never too late to improve upon an existing site. With that in mind I designed and ran some usability studies on Seattle University s online bookstore feature in order to discover areas of success and inadequacy. The findings of this study will serve to help improve the site s user experience. The actual test consisted of 5 participants; 2 that are or have been in college, and 3 that have never attended. Each participant was asked to complete 5 simple tasks that a Seattle University student might normally do during their attendance at this school. The tasks included: finding back to school items, buying new copies of books, comparing the new and rented prices of books, checking out new and rented books, and buying a gift. From these tests I gathered both quantitative and qualitative data by recording everything that each participant said or did. I used the information gathered to expose common usability errors and ways in which they can be resolved. THE ONLINE BOOKSTORE Seattle University s online bookstore is an all-in-one resource for students material needs in regards to school. From here they can obtain anything from course material to gifts for friends and family members. Its main purpose is to provide an easy way for students to get their required texts for class; however, it has secondary features for buying: gifts, memorabilia and more. It is fairly comparable to online bookstores from other universities and has potential to be a very useful tool for new or returning students. St. Christopher Monterroso Usability Testing 2
Test Plan On September 8th, 2012, usability testing occurred on the bookstore feature so that objective data could be collected. The purpose of testing was to take an in depth look at how the online experience functioned for each user. All commonalities will be applied to develop set standards for future versions of the website. OBJECTIVE The goal of the testing was to assess the bookstore feature by: How easily each user got what they needed Difficulty based on nationality What obstacles were encountered Did the flow of the site meet expectations LOCATION AND SETUP The setting of the usability test was a controlled environment free of distraction so each participant could focus solely on the tasks they were given. The specific location was 1338 Terry Avenue Seattle, Washington 98155 in a room that was comfortable and easy to get to. One moderator was present to encourage thought, not help, and create impetus. As far as setup there was one computer which remained on a neutral screen until testing had begun. There were cameras present to record the screen of the computer and the audio from each participant for playback assessment. St. Christopher Monterroso Usability Testing
RESEARCH QUESTIONS These questions were answered: 1. How easy was the site to navigate? 2. What expectations did the users have for the feature? 3. Was it difficult to find each item through basic navigation? 4. What obstacles are encountered along the way? 5. Was the amount of time for each goal appropriate? 6. What steps did each participant take on their path to their goals? 7. What questions did the users ask as they work through their goals? QUANTITATIVE DATA Gathered at the end of the session was: Errors in navigation due to misinterpretation of links Errors in checkout process due to the dual cart system Whether or not a process was completed Was numbers of errors affected by locality Time taken to complete each task QUALITATIVE DATA Gathered at the end of the session was: All verbal communications between moderator and participant A holistic experience discussion 4
Test Plan PARTICIPANTS For the majority of the participants I was looking for people who have experience in college or have been considering attending. They were to be inexperienced with the Seattle University online bookstore, but have experience shopping online. I ended up with 5 subjects with characteristics broken down as follows: CHARACTERISTICS NUMBER OF PARTICIPANTS PARTICIPANT TYPE International Student Local Student 1 4 STUDENT STATUS Attending Seattle University Attended/Attending College Never in College 1 2 2 AGE RANGE 18-20 21-25 25 and Older 1 3 1 GENDER Male Female 1 4 ONLINE PURCHASING Frequent (2+ times a month) Occasional (1 2 times a month) Seldom/Never (once every 2+ months) 2 1 2 St. Christopher Monterroso Usability Testing
MODERATOR ROLE As the moderator I sat quietly and took notes as each participant went through their tasks. I made sure to properly inform each individual on what was being asked of them, conduct a background interview, read them the session script, and then introduce tasks appropriately. Since the outline of the tasks gave each participant free reign over how they achieved their goals, follow-up questions were asked at the end of each session to gather qualitative data. PARTICIPANT ROLE All participants used Seattle University s website to rent or purchase books for fictitious courses that they were enrolled in for Fall 2012. Each course had multiple teachers and times to allow for the maximum complexity possible so that the most information could be obtained. Every participant received a course, book and purchasing order. METHOD This usability study was primarily a discovery test in which the effectiveness of the bookstore feature would be assessed and analyzed. Therefore there was only one set method for each participant so that the results could be easily compared. The method for testing that was used was a between-subjects structure with a slight counter-balanced order. Counter-balancing was only slight because certain tasks had to happen in a certain order due to their inherent nature. Each participant session lasted about 30 minutes, and 10 minutes of that time was dedicated to explaining the study as well as expectations to the participants. 6
Test Plan SESSION SCHEDULE Below is how the official testing day was broken down: TIME SEPTEMBER 8 th DETAILS 11:00 am - 12:00 pm Setup 12:00 pm - 12:15 pm First Debriefing 12:15 pm - 1:00 pm Revision of Script 1:30 pm - 10:00 pm Participant Testing INDIVIDUAL Before the Session: I had the participants: Read and sign non-disclosure and recording consent forms Fill out a brief background questionnaire INDIVIDUAL Introduction to the Session (8 minutes): We went over: Previous experience with this method of testing Why full cooperation is vital The setup of the environment My role as the moderator The role the participant plays and what is expected of them Session protocol Importance of verbalizing everything that they are thinking St. Christopher Monterroso Usability Testing
INDIVIDUAL Pre-Test Interview (2 minutes): We discussed the participant s: Knowledge of similar sites Expectations of this sort of site Reason for using a site such as this one INDIVIDUAL Tasks (15-30 minutes): Participants were asked to: Find the back to school basics set located in the bookstore Add the book Helping for MGMT 572 to the shopping cart Find and compare Essentials of Management Information Systems (10th edition) for IS 315 new and rented prices and add the cheaper to the cart Add the rented book Ulysses by James Joyce to the cart and checkout everything Find a 1/4 zip mom fleece sweatshirt and add it to the cart INDIVIDUAL Post-Test Debrief (6 minutes): I gathered as much: Feedback as possible about the experience they had on the website Information about problems noticed during the observation 8
Test Plan MEASURES To address these questions: Is there any impact based on locality of the participant? Will it make a difference if the order of steps is slightly adjusted per user? What obstacles are encountered? How successful are the participants in finding what they are asked to? What kinds of questions are asked by each individual? How does each person feel about how long each process took? How well does the site support the path of the user? I gathered both quantitative and qualitative data by: Errors of omission Responses/comments from users Time taken on each task Number of tasks completed without the help of the moderator Number of errors How many wrong turns were taken REPORT CONTENT Everything that was recorded for reporting in terms of this usability test were: Summaries of all interactions as well as all stages of the study Questions derived for standardization of experience Data collection Data layout Discussion of results St. Christopher Monterroso Usability Testing
PROJECT TIMELINE Everything that went into this project was laid out as follows: WHAT WHEN INITIAL PLANNING OF Objectives, Scope and Deliverables Identification of Participants Preparation of Schedule Aug. 2 nd 2012 Delivery of Screener and Test Plan Aug. 9 th 2012 Recruitment of Participants Sept. 2 nd 2012 Delivery of Final Session Script Sept. 6 th 2012 Test Run of Session Script Sept. 6 th 2012 Participant Sessions Sept. 8 th 2012 Delivery of Final Report Sept. 15 th 2012 Presentation of Findings Sept. 17 th 2012 10
Test Plan DELIVERABLES Everything that was submitted in the final report and presentation: All meetings in the pre-planning processes drafted and everything else that has been said and done including: questions, participant recruiting, etc. A typed copy of the session script Final copies of participant criteria A test plan that includes everything that has taken place Checklist of things observed during each session Test run of the session Pre-test questionnaires Usability sessions of the 5 participants Post-test interviews All recordings of the sessions A final report of all findings and comments Presentation of process and discoveries St. Christopher Monterroso Usability Testing
Scripts / Materials Being in a testing environment can be intimidating for participants and users at times may feel under a lot of pressure to deliver what they think is needed from them. Having someone watching them as they do required tasks can add to that pressure so it is imperative that all observers follow these guidelines. OBSERVER GUIDELINES Rules for every moderator and/or observer: Make sure that the orientation script is read to each participant Do not include participants names inside findings (notes are to remain anonymous) Do not distract the participants with noise, interruptions, etc. Even if you know the participant(s) treat them as you would everybody else Be respectful to everyone involved in the study Do not make small talk even if the participant wants to because it will inversely affect the time it takes to complete a task and will compromise statistics Refrain from being too demanding and overbearing because this might make the participant even more uneasy Make sure everything is in order and everybody is comfortable before testing 12
Scripts / Materials ORIENTATION SCRIPT Good (morning/afternoon/evening), my name is. Before we get started on anything today, I would like you to fill out a brief questionnaire. This will be used to assess where everybody is at in terms of background and skills so that everything can be kept in order later. Anything that will be stated and recorded in this session will be anonymous so please be one hundred percent honest with everything said and done. [wait until the questionnaires are complete] [Thank participant]. Now just to reaffirm, this study is an observation of the Seattle University s online bookstore feature and how useful and intuitive it is. Today you will be asked to perform a given number of tasks on this website and I will record any problems that you encounter with the site as well as any errors in its process. This test has nothing to do with your ability. Everything recorded will be an evaluation of the website and its capacity to perform in a predictable way. Just do your best and do not be overly concerned with the results you are not the one being tested the website is. So approach each task as you would see fit and do not do anything you would not normally do. Any error that occurs is solely reflective of the websites usability and therefore is not your fault. My role here is just to document the advantages and short comings of the online bookstore based on a per user system. Please say what is going through your mind while you go through this process and think aloud. Again, because we really want to know what can be improved upon or what works great I would ask that you be completely honest. Feel free to ask as many questions as you would like but know that I may not be able to answer them until the end of the session. While you are going through the tasks I will be taking notes and the session will also be recorded so that nothing is overlooked. If at any time you feel uneasy feel free to stop the test. Before we get started are there any questions you would like to ask? If not then I would like to have you sign this non-disclosure and consent form so that we can begin. St. Christopher Monterroso Usability Testing
PRE-TEST QUESTIONNAIRE To be conducted before any testing begins. Are you a student in college? (if not have you ever attended?) Are/were you an international student or a resident/out-of state student? How often do you purchase items online? A. Everyday B. Frequently C. On occasion D. Seldom E. Never Do you prefer to buy books online or in a store? If number 4 is online then where do you shop for textbooks online? (list up to five) Have you ever bought gifts online before? Do you prefer to rent or buy a textbook? POST-TEST INTERVIEW To be conducted after every session is over. What are your overall impressions of the website? Which task was the most difficult to complete? What made that task difficult? Please scale each task individually using a 1-5 level of difficulty (5 being the hardest): Would you continue to purchase your textbooks this way if you attended this school? Do you have any suggestions for this website to make it more useful? Do you have any parting comments or questions about the online bookstore? 14
Scripts / Materials TASK SCENARIOS To be completed by each participant. 1. Find your back to school basics set in the online bookstore and add it to the cart. 2. Add the book Helping (in new condition) for MGMT 572 to your shopping cart. 3. Find Essentials of Management Information Systems (10 th edition) for IS 315 (in new condition) and compare it to the renting price. Add the cheaper option to the cart. 4. Add the rented book Ulysses by James Joyce to your cart and checkout everything. 5. Find the 1/4 zip mom fleece sweatshirt and add it to your cart. St. Christopher Monterroso Usability Testing
Results PRE-TEST TABLED Below is all the data collected from the questionnaires ARE YOU A STUDENT IN COLLEGE? (IF NOT HAVE YOU EVER ATTENDED?) Participant 1 Yes Participant 2 No Participant 3 Yes Participant 4 No Participant 5 Yes ARE/WERE YOU A RESIDENT OR INTERNATIONAL STUDENT? Participant 1 International Participant 2 N/A Participant 3 Resident Participant 4 N/A Participant 5 Resident 16
Results HOW OFTEN DO YOU BUY ITEMS ONLINE? Participant 1 Frequently (2+ times a month) Participant 2 Never Participant 3 On occasion (1 2 times a month) Participant 4 On occasion (1 2 times a month) Participant 5 Seldom (once every 2+ months) DO YOU PREFER TO BUY BOOKS ONLINE OR IN A STORE? Participant 1 Online Participant 2 In the store Participant 3 In the store Participant 4 In the store Participant 5 In the store St. Christopher Monterroso Usability Testing
IF ONLINE NAME UP TO 5 SITES THAT YOU VISIT Participant 1 Amazon Participant 2 N/A Participant 3 N/A Participant 4 N/A Participant 5 N/A HAVE YOU EVER BOUGHT GIFTS ONLINE? Participant 1 Yes Participant 2 No Participant 3 Yes Participant 4 Yes Participant 5 No 18
Results DO YOU PREFER TO RENT OR BUY TEXTBOOKS? Participant 1 Depends on the price, but I prefer to buy textbooks. Participant 2 Buy Participant 3 Rent Participant 4 Rent Participant 5 Buy St. Christopher Monterroso Usability Testing
POST-TEST TABLED Below is all the data collected from the exit interviews WHAT WERE YOUR OVERALL IMPRESSIONS OF THE SITE? Participant 1 It was difficult to find the stuff asked. Too much work to checkout twice. Participant 2 It didn t seem easy to navigate so I had to search everything. Participant 3 The imagery was not appropriate and I would default to search. Participant 4 Professional like any other site. Likes navigation but can be forgotten at times. Participant 5 Pretty confusing at points and is not that easy to navigate. WHICH WAS THE MOST DIFFICULT? Participant 1 Four Add the rented book Ulysses to the cart and checkout Participant 2 One Find the back to school basics Participant 3 One Find the back to school basics Participant 4 Two Add the book Helping for MGMT 572 to the shopping cart Participant 5 Four Add the rented book Ulysses to the cart and checkout 20
Results WHAT MADE THE TASK DIFFICULT? Participant 1 The double sign-in and double cart. Participant 2 Could not find it where I thought it would be located. Participant 3 Did not know where to click to get to where it was. Participant 4 Could not figure out where to look for the book. Participant 5 The double cart. PLEASE SCALE EACH TASK INDIVIDUALLY 1 5 (5 BEING THE HARDEST) Participant 1 1 1 3 5 4 Participant 2 5 2 3 1 1 Participant 3 4 3 3 1 2 Participant 4 4 4 2 4 1 Participant 5 3 2 3 5 2 Task 1 Task 2 Task 3 Task 4 Task 5 St. Christopher Monterroso Usability Testing
WOULD YOU CONTINUE TO PURCHASE YOUR TEXTBOOKS THIS WAY? Participant 1 No Participant 2 No Participant 3 Yes, but would complain. Participant 4 Yes, instead of amazon. Participant 5 No, absolutely not! DO YOU HAVE ANY SUGGESTIONS? Participant 1 Make a single checkout, and clean up the textbook home page. Participant 2 Make things easier to find so that I do not have to use the search feature. Participant 3 Unclutter the home page, combine shopping carts and add more appropriate images. Participant 4 No Participant 5 One cart with all the items and make the bookstore easier to find. 22
Results PARTING COMMENTS OR QUESTIONS? Participant 1 No Participant 2 Make it more logical Participant 3 Why are the home page ads so huge and obnoxiously clunky? Participant 4 No Participant 5 The textbook page felt like you could only rent the textbooks. TIME TAKEN All the times from each participant is averaged in the table below. AVERAGED TIMES IN MINUTES FOR EACH TASK (ALL PARTICIPANTS) Mean Average 2:24 2:27 2:47 5:26 1:14 Median Average 1:31 0:45 2:57 4:43 1:10 Task 1 Task 2 Task 3 Task 4 Task 5 St. Christopher Monterroso Usability Testing
Findings POSITIVE FEEDBACK In response to completing the session participants expressed these positive comments: Searching is a life saver. Once you get the hang of everything the steps get easier. The site felt like it had everything you needed. Checkout on the main page was easy to follow and each button made sense. The renting textbook page was clear and easy to navigate. NEGATIVE FEEDBACK The comments the participants offered of the site that were critical included: The home screen of the bookstore is not that useful. Comparison of new, used and rented prices is cumbersome. Imagery in the RedHawks shop is not helpful. Opinions were made that the visual look left something to be desired. Although, no one participant ever knew what exactly bothered them about it. There is far too much text on the textbook tab page. Everyone was confused by the fact that there are two carts. Location of the bookstore link was not in the right spot for 3 of the participants. The textbook page makes it only seem like you can rent textbooks. Placement of the online bookstore link is not where some thought it would be. 24
Conclusion KEY CHANGES For the majority of the participants this was the first time that they had used the Seattle University s online bookstore. It was a struggle for most and even though everyone got through each task everyone expressed some level of frustration throughout their session. The biggest frustration caused by any of the tasks was the one that came from the checkout process. One user who figured out the dual cart system early on summed up everyone s thoughts best when they said, I m still in the bookstore but this [renting section] has its own shopping cart. So I m going to have to make two separate purchases which is annoying. With this in mind the changes to the site this research shows to be appropriate are: A merging of the two cart systems into one that can be accessed everywhere Options to rent the book if availability exists directly on the buy new page A search feature like the one on the rent page for the main textbook/home page Only one link to buy/rent on the textbook tab page Addition of online bookstore links to the academics tab and the bottom of the home page More appropriate: images and labels on buttons St. Christopher Monterroso Usability Testing 25
Appendix NONDISCLOSURE Understanding your participation. Please read over the following carefully. I,, am asking for your participation in the evaluation of Seattle University s online bookstore feature. Your participation in this study is to help us gather information about how useful and intuitive the process is. Together we will uncover usability flaws which in turn will help us to improve the website. Everything that you do will be observed for proper documentation. By doing so we can discover how each user reacts with the product individually. Before any of the process begins we will be asking you to fill out brief questionnaires and answer a couple short introductory interviews. In addition to information provided by forms, observations, and interviews we will also be requiring video and vocal recordings. By signing this form you give permission to I,, and Seattle University, to use your voice, recordings, and all statements for purposes of evaluating the website and presenting the results of our findings. Any and all information acquired by us will remain anonymous. Participating parties are asked that any information acquired by them through these sessions that is not already public information be kept within their session. Your signature guarantees that you will not discuss anything learned today outside of your study. You may however retain the right to say that you did participate in this study. If you agree to these terms then please indicate so by signing and dating the below form. Printed Name: Signature: Date: 26