0 DCOM 131-01 Group Project 2: Usability Testing Usability Test Report Tim Harris, Zach Beidler, Sara Urner, Kacey Musselman
1 Table of Contents Introduction... 2 Purpose... 2 Heuristic... 3 Participants... 3 Methodology... 4 Test Type... 4 Testing Conditions... 5 Characteristics of Users... 5 Length and Format of Tests... 7 Findings/Test Results... 7 Post-task, post-test results... 18 Recommendations:... 19 Conclusion:... 20 Appendices... 21 Appendix A.... 21 Appendix B.... 23 Appendix C.... 23 Appendix D.... 25 Appendix E.... 26
2 Introduction Job Center is a website that most Lebanon Valley College students will utilize during their time here. Run by Career Services and available through MyLVC, it is a career database with resources to obtain help with resumes, interviews, etc, databases of both possible internship and job openings as well as alumni willing to be career mentors, and much more. One chooses to attend college in order to ultimately go out into the workforce and have a career, and this site is meant to help students further themselves professionally. When you know how to conduct yourself in a work setting and demonstrate the communications skills and knowledge required of your desired field, you will stand out amongst other potential hires. It is Job Center s goal to make the professional world accessible for students of Lebanon Valley College. Purpose Our purpose in conducting this usability test was to make sure that the site functions properly, and most importantly that students were able to make sense of the site s many features and accomplish their goals. We had several concerns related to the site s structure and were curious to see how real LVC students would interact with the site. Our testing consisted of several subjects who had not visited the Job Center site before, so that allowed us to see if students could successfully navigate the site to accomplish certain tasks without any prior instruction.
3 Heuristic The main heuristic that we chose to focus on was efficiency. College students want to find internships, but they can t take long doing it. They have papers to write and exams to study for. We wanted to test to make sure that the structure of the site allowed for quick and simple use, and that a student could get in and out within reasonable time and get what they wanted accomplished. Participants This sections describes the test subjects for our usability test, reasons why selected them for participation, and information gathered during pre-testing. The Lebanon Valley College Job Center s primary audiences are LVC students. The sites purpose is to help find future jobs/internships for students, along with creating career connections, and providing other services. Our group decided to gather five to seven test subjects that were students at Lebanon Valley College in order to observe whether the Job Center is accomplishing its purpose effectively and efficiency. Since any student can use this resource, our goal was to get a diversified group of participants so that we can test if students in any major or class year can accomplish their goal when using the Job Center. In order to record pre-test screen data, we asked each test subject to answer several pre-tests questions.
4 Our group asked for: For their current class year. What their major(s) and minor(s) they are currently. Where their hometown was located. Whether they used or have not used Job Center before. How often they used Job Center if they did use it before. If they have not used before, what functions do they think the site will have. From the pre-test date, test subjects ranged from freshman to senior class standings and most were not in the same major. Every participant lived in a different hometowns, even one subject was from Connecticut. Surprisingly, our group found that only three out of eight participants have used the Job Center before. Methodology Test Type The usability test we performed for the Job Center website was used to inform us on what some of the issues with it could be. Prior to running the tests, we gathered information from different sources along with doing our own research. We spoke directly with Career Services and asked them what it was on Job Center that they specifically wanted us to test. We then went through on our own and wrote down what all the reasons are that people go on job center. After that we ordered those things and decided which are the most important things that we wanted to focus on.
5 After the usability tests were finished, we gathered all of the information and organized the results in a way that was easy for us to understand. From there we looked at how difficult it seemed for people to complete the task and that helped us judge on what needed improvements. We also took track of time to see how difficult it was for a user to figure out how to do a task, which then brought into legibility with the site and how well mental models were followed. Testing Conditions All of our usability testing was done in the usability lab. We had the test subject in the testing room with a moderator, while the other group members were observing in the observation room. The moderator did change because we wanted to make sure who ever knew the test subject would be the one to moderate them. All who moderated followed a specific script to keep things consistent. The moderator sat behind the test subject and out of the way while being there just in case any help or guidance was needed with the tasks. We also each explained to our test subjects to do the thing out loud method just to give us a better idea of their brain process through the test. Each subject used Internet Explore so everyone was consistent on the browser type. Testing was done on the usability lab computer and we used Morae for recording the test along with the date. The observers were the ones who were using Morae by taking notes and using the timer for tasks during the test. Nothing extra was then used except for what we were given. Characteristics of Users We had a total of eight test subjects and these were all friends of people within our groups. All test subjects were students at LVC and they ranged from freshman to seniors.
6 We wanted to make sure that we got users who have used the site before along with those who have never used it before. It was also important that we got a mix of majors and interests because depending on what that is, they may be more comfortable with navigating unknown websites and they will also get different search results. Below is a table description of our test subjects and the questions summing up the pre-test that we gave them. Test Subject Year Major(s) Hometown Have you used Job Center? How often? Why? 1 FR Actuarial Science Millersville No 2 FR Accounting/ Reading No Business 3 SO Actuarial Science South Harrison, NJ No 4 JR Elementary Ed./ Milford, CT No Special Ed. 5 SO Psychobiology Jonestown No 6 JR Digital Communications New Cumberland Yes Not often Internships 7 SR Math Allentown Yes Once or Twice Resume help 8 SR Math Cape May Yes Often Job search/job fairs
7 Length and Format of Tests Each test we stayed consistent with how we ran it and we kept scripts for the ones directly involved with the test subject. When it was time for the test to start, the moderator would go out and get the subject, then bring them into the test room. At that point the moderator would begin to read the script on what the test was for and the process of the test. We then gave them the consent form and asked the subject to fill out a pre-test questionnaire. When the subject was done they would be told to turn the page and begin the test. Once it was over we asked for them to fill out a post-test questionnaire and then they were free to leave. We left 30-minute time periods between each test and had no problems. Generally each test lasted anywhere from 5 to 15 minutes depending on the subject. Findings/Test Results This section gives an overview of our findings from the usability tests and test results. Test Subjects provided our group with qualitative data in the form of comments, behaviors, and facial expressions. During observation, many findings and result were found to be surprising. The very first startling finding during each test is that many students were unable to find the LVC Job Center without assistance. As seen on Table 1, four out of the eight test subjects only knew where and how to get to the Job Center correctly, often showing faces of confusion. Participants that were lost in finding the Job Center received assistance in order to further
8 the test. Also three of the four that did not know how to get Job Center went to the Career Services page on the Lebanon Valley College site. These actions made by the participants show that they are aware of the connection between Career Services and Job Center. Table 1. Task A: Navigation Success and Failure (needed assistance) Participant Success Failure (needed assistance) #1 X #2 X #3 X #4 X #5 X #6 X #7 X #8 X Simple and broad informational tasks such as finding the Resume Creator and Resource Library had the largest successful rate (Table 2 and Table 3) with both resulting in seven out of eight subject able to complete the tasks.
9 Table 2. Task B: Resume Creator Success and Failure Participant Success Failure #1 X #2 X #3 X #4 X #5 X #6 X #7 X #8 X Table 3. Task C: Resource Library Success and Failure Participant Success Failure #1 X #2 X
10 #3 X #4 X #5 X #6 X #7 X #8 X Unexpectedly, once the user was able to get on the Job Center site, some participants still had trouble to get access to the site. This was due to the fact they never have been on the Job Center before. Two out of eight users saw a page that needed them to submit a profile in order to access the Job Center. Participant #5 and #6 had this issue and even though there was a paragraph of instructions below a Submit Profile button, they decided to go back to the Career Services page for answers. When it came to the efficiency in accomplishing each task, two out of eight participants were able to finish finding the Resume Creator within 12 seconds. According to Table 4, two out of eight participants found the Resume Creator within 30 to 45 seconds. Participant #6, failed to find the correct Resume Creator, and found other info on resumes on the Career Services page. Table 4. Task B: Resume Creator Time on Task
11 Participant Under 30 sec. 30 sec. 1 min. 1 min. 1 1/2 min. Over 2 min. 1 1/2 min. 2 min. #1 46 sec. #2 1 min. and 15 sec. #3 1 min. and 2 sec. #4 48 sec. #5 3 min. #6 1 min. and 31 sec. #7 43 sec.
12 #8 32 sec. Six out of eight users completed the Resource Library task under a minute and easily found the Resource Library link on the Home page. Seven out of eight subjects, shown on Table 6 were able to find the correct folder on the Resource Library within a couple seconds after they found the correct page. Table 6. Task C: Resource Library Time on Task Participant Under 30 sec. 30 sec. 1 min. 1 min. 1 1/2 min. Over 2 min. 1 1/2 min. 2 min. #1 32 sec. #2 52 sec. #3 6 sec. #4 48 sec.
13 #5 20 sec. #6 1 min. and 59 sec. #7 19 sec. #8 11 sec. When looking for information on On-campus interview, only four out of eight subjects were able to find the information as shown on Table 7. Most of the participants would often hover over every nav link and still miss the On-campus interview link. Table 7. Task D: Search Success and Failure on Finding On-Campus Interview Participant Success Failure #1 X
14 #2 X #3 X #4 X #5 X #6 X #7 X #8 X Navigation or information architecture caused a problem for users in order to find the correct page like Mentor (Career Connections). Overwhelming the subject with three nav bars also confused the user in order to find the on-campus interviews or resume creator. Even worse results were found when subjects attempted to find a job with the ID number. Only one out of eight subjects was successful at using the advance search and finding the Job ID search box. The other participants received no results because they used the Keyword search box with the ID number in it. Most of the subject had a face of disgust and were perplexed from the result feedback. Resulting in several to skip the task and move on to the next task. As a result, three of those seven found the correct job when moving forward with tasks instructing to open advance search. Participants had also trouble trying to find narrow or specific information like a job/internship they would be interested in. As shown on Table 8, five out of eight subjects
15 couldn t find a job/internship they would be interested in because of various reasons. Several participants used Job Location and were unable to find a job in their location. For example, participant #4 input a Zip code in Job Location and couldn t find anything as a job near their hometown. Participant #2 was the only user able to find a job/internship near their hometown. Most subjects utilized their majors and were unable to stumble on any job/internship they would be interested in. Table 8. Task D: Search Success and Failure on Finding Job/Internship Participant Success Failure #1 X #2 X #3 X #4 X #5 X #6 X #7 X #8 X Table 9. Task D: Search Categories Used Searching for Job/Internship
16 Category #1 #2 #3 #4 #5 #6 #7 #8 Keyword Position X X X X X Type Job ID Majors X X X X Job Category X X Type of Business X X Classificatio n X Job Location X X Mentor (Career Connection) search resulted with six out of eight (Table 10) able to find an alumnus in a field of their interest. The other two couldn t find one for various reasons, most were from the use of Industry and not having the industry they desired. For example, subject #5 was confused on with the broad industry name sciences. Table 10. Task D: Search Success and Failure on Finding Mentor
17 Participant Success Failure #1 X #2 X #3 X #4 X #5 X #6 X #7 X #8 X Overall, Job Center s flexibility and efficiency of use relies on the user s experience with the system. Participants are able to quickly find simple and broad information like a Resume Creator and Resource Library. Yet, at times navigation can get in the way when accomplishing the task as fast as possible. On the other hand, most participants are unable to find more specific information like a job/internship due to the available search options to the user. Advance search can be effective when correctly used; however, the industry and job location search bars can be faulty. Throughout the tests, we were able to gain rich qualitative feedback from many subjects. For example, participant #5 could not find his/her correct industry and explained it only says sciences. In addition, a couple search
18 categories confused the participants. For example, participant #3 stated there is a lot of the same things that are serving the same purpose like Type of Business seems to be very similar to Job Category. Post-task, post-test results With our post-task questionnaire, we decided to ask five questions with a rating system of 1 to 5. 1 meant that you strongly disagree and 5 meant that you strongly agreed with 3 being neutral. Most of our questions had answers that either strongly agreed with the question or strongly disagreed, which then balanced the average out to neutral. This is also because some of our test subjects either really struggled or were pretty successful in doing the tasks. We did get a pretty good idea of what tasks were a struggle and which were not, by doing our tests, but this was another way to see exactly what the test subject thought. Below are all of our findings and the questions that we asked for the post-test. Each participant s answer was averaged out to one number, which is where the average rate for each question comes from. Post-test questions 1. To what extent did you utilize the instructions on the career services website?
19 2. How difficult did you find it to complete these tasks? 3. After doing these tasks, how comfortable are you with Job Center? 4. Do you think this site would accomplish your goal of job searching? 5. Would you use this site again for job searching? Question # Average Rating 1 3.1 2 3.2 3 3.6 4 2.8 5 3.3 Recommendations:
20 When examining the findings of the usability test conducted on the LVC job center site, it becomes clear that the site should make several changes. These changes will decrease the amount of kinematic and cognitive load on users while improving the overall efficiency of the site. Additionally, by making the following changes it will increase the likelihood of students actually using the job center site for its intended purposes. The first recommendation for the LVC job center site would be to make it easier to locate. Of all the tasks listed in the usability test, having students simply navigate to the job center site resulted in a significant amount of failures. This could be solved by making a link to the job center site on the actual LVC website. Also, an additional recommendation to improve the job center page would be to limit the content on the home page. The immense amount of content contributed to confusion amongst participants and decreased the overall efficiency of the site. A way that this could be improved upon would be to use a few key words to describe the links used on the home page. A third and final recommendation for the site would be to eliminate the multiple navigation bars. By having multiple navigation bars, users were unsure of where they should begin looking for information regarding their tasks. Developing a hierarchy of needs and establishing several key categories to organize the pages content could certainly resolve this issue. Conclusion:
21 Overall, after conducting a usability test on the LVC job center website it became clear that the page has several inefficiencies. This was determined by having participants complete several tasks that included: navigating to the website, finding the resume creator, locating the resource library, locating an on campus interview, discovering a potential job and finally, finding a mentor. This test evaluated the efficiency heuristic by looking at the time it took each participant to complete each task. However, by taking into consideration the several recommendations for the LVC job center site, students will then be enable to fully utilize all of the features and resources of the website. Appendices Appendix A. Test Script 1. Take Subject into testing room 2. Say:
22 My name is and I want to thank you for coming here to help our group help us test the flexibility and efficiency of use of LVC s Job Center. Our purpose is to see how efficient the Job Center at accomplishing several goals for users like yourself. As you can see, there are cameras set up to record your audio and actions that you do on this computer. Also behind this one mirror is a group of observers that will observe, analyze, and take notes on the how you complete each task during this test. First, you will fill out a pre-test form. Then, for the actual test you will be performing several tasks that are listed on a sheet instructing what to do on Job Center. After, you then fill out a quick post-test form. I realize that thinking out loud is not normal when using sites; however, doing it throughout will provide many insights on the efficiency of the Job Center. For example, you could say something like I really like this because. or I am confused in what to do here. I would appreciate if you could read and sign this consent form for further information on this usability test Thank you. Do you have any questions or concerns before we start the test? 3. Hand the participant the pre-test form. 4. After pre-test: hand them the test tasks. 5. After test: hand them the post-test questions.
23 Appendix B. Usability Pre-Test Questions What year are you? FR SO JR SR What is your major(s)? What is your minor(s)? Where is your hometown? Have you ever used the LVC Job Center website? Yes No If yes how often? For what purpose do you use it? If no, what functions do you expect this site do have? Appendix C. Usability Test Tasks Test
24 Please perform each task as stated. You may have difficulty with certain tasks, but it is important to at least attempt each one. Pre-Test: Open Internet Explorer from the Desktop Navigate to LVC.edu Test: Task A: Navigation 1. Find Job Center Task B: Resume 1. Find the Resume Creator 2. Go back to the home page Task C: Resource Library 1. Navigate to Resource Library 2. Find help on how to make a cover letter Task D: Search 1. When is the Advertising Specialty Institute holding an interview at LVC? 2. Do a search for Job 7328
25 3. Using the Advanced Search, find a job/internship that you would be interested in 4. Using Career Connections, find an alumnae currently working in a field you are interested in 5. Close out the career services website ---------- End of Test ---------- Appendix D. Usability Post-Test Questions Please rate the questions 1-5. 1 Being you strongly disagree, 5 being you strongly agree and 3 being neutral. 1. To what extent did you utilize the instructions on the career services website? 1 2 3 4 5 2. How difficult did you find it to complete these tasks? 1 2 3 4 5 3. After doing these tasks, how comfortable are you with Job Center? 1 2 3 4 5 4. Do you think this site would accomplish your goal of job searching?
26 1 2 3 4 5 5. Would you use this site again for job searching? 1 2 3 4 5 Appendix E. Usability Test Notes 4/14/14 Subject 1 Went to Career Services site Didn t see Job Center- Job Center like on site doesn t take you to Job Center Found Resume Creator, took a while Does lots of scrolling Didn t find on campus interview- went to job/internship section instead of On-Campus Used Keyword incorrectly Really struggling Subject 2 Went to Student Life to find Job Center Went into My Profile once on Job Center- unnecessary for tasks presented Accidentally clicked on a link to take him to LVC.edu when he wanted to go to the home page of Job Center
27 Found cover letter help Keeps going to the Resource Library to look for things Went to/ can t find on campus interviews Used directions to find 6378 Found a job he would be interested in, had trouble with the search options (too narrow) Subject 3 Clicked on Student Life Back to the home page Offices Moderator instructs to use MyLVC Resource Library Clicks down arrow on Resume Builder Then finds Resume builder Doesn t use advanced search at first then is able to find the 6378 job Was not successful in finding a job/ internship related to her interests (actuarial science) Subject 4 Found Job Center easily One mistake before Resume Confused faces Found Resource Library and Cover letter help Command+F- Used to search the words so they would be highlighted on the screen?
28 Flustered faces Couldn t find ASI or 6378 Whoa - Accidentally clicked on link that took her to the Career Services site and off Job Center Not even using Job Center anymore- completely lost Searching the school site for things- not Job Center Puts Job ID # into the keyword box Accidentally typed in the wrong ID number Doesn t know what Job Category is Too specialized of a search- no good results Put zip code in job location Didn t find job or alumni Subject 5 Is it on MyLVC? Found Job Center Hasn t ever logged in- Submit Profile page Confused, keeps leaving page and coming back to it Finally clicks submit profile Found Resume Creator Found cover letter help
29 Going through all jobs page by page instead of typing in the ID Everyone keeps typing the ID # in the Keyword slot Couldn t find a job Making the search more specific Observant of the links- unlike previous subjects it seemed Subject 6 Doesn t know where Job Center is Went to About LVC??? Offices, Student Life, etc Finally goes to My LVC (after assistance) Has never been on it Doesn t understand to submit his profile??? Doesn t take time to read the directions below the Submit Profile button FINALLY submits profile Thought he found Resume Creator but didn t Found Resource Library Really struggling Found Career Connections right away- Couldn t find Graphic Design Didn t find a mentor Most seem to not utilize the Advanced Search Subject 7 Got right into Job Center
30 Found Resume Creator Using Keyword for Job ID Really fast- has used the site before Subject 8 *I didn t take notes on this subject because she was the extra person after Jess Finlayson*