Big Kahuna Project User Testing for A Journey through Fantasy Land Jennifer Jones USER PROFILE The audience for my Big Kahuna site consists of sixth grade language arts students in South Lyon. In terms of demographics, 51 students are male and 39 students are female. Over 90% of the students are Caucasian. Approximately 33% come from low income families. This is representative of the overall demographics of the school. In terms of technological know-how, the students are quite diverse. Some students are skilled users, while others have only had access to computers at school. In an informal survey, about 87% of my current students stated that they have computers at home. Of these 78 students with computers, all but 8 have the Internet at home as well. However, even with a high number of home computers, these students still vary greatly in their skills. Even so, at the point in the year in which students would use my BK site, they would have been exposed to at least two WebQuests. The design of these sites is purposefully similar in terms of navigation to the BK site in order to scaffold and build on what they already know how to do. Additionally, the WebQuests have the students complete such tasks as opening documents and visiting other sites, just as this project asks. Just as their technological skill level varies, their motivation and goals will vary as well. Some of my students are working for the grade, just as Pirsig described. They will be extrinsically motivated to do well in order to earn the grade. Other students will be intrinsically motivated to explore the journeys on the website. However, there are still the students who are not as motivated to complete their school tasks. Hopefully, the variety of resources used on the site will aid in capturing their attention. Prior knowledge is yet another aspect that will vary for the users. Although many have been exposed to fantasy stories in books or movies, many will not have analyzed the features of the genre as we will be doing. Their interest levels will vary as well, which affects how much exposure they have had to the topic. USER-TESTING CHECKLIST Below is an adapted version of the user-testing checklist developed by Plastic Glasses. What we will do and how we'll go about it Ask students to help you with a grad school project after school. Explain the assignment(s) to your users and what they would do if they were doing the assignment and then have them go through the BK as if they were actually doing the assignment(s). Sit with students and observe them as they go through the site. Be sure to focus on functionality, aesthetics, usability, and engagement. Have students answer the students complete an online survey on their own after they use the site. Main Focuses of Checklist Navigability - Do users have an easy time navigating through the lesson? Are the steps for the lesson clearly laid out? Does the user get lost? Aesthetics - Do the users seem to like the layout and the look of the site? Engagement - Do the users like the resources you employ? Do they find them interesting? Why or why not? Understanding - Do the users understand the assignment and/or purpose of the site?
Questions to Ask Users The questions will be divided into a ratings section and a suggestions section as described below. Ratings Students will first be asked to rate each of the following statements. Their response choices will be: strongly agree, agree, neutral, disagree, and strongly disagree. The words in red appear merely to show the focus of each question and will not appear in the actual survey. 1. The design of the site is pleasing to look at. (When answering this, consider the colors, fonts, and images that were used.) Aesthetics 2. I could easily find my way through the site. In other words, I could move from one journey to the next without any problems. Navigability 3. I understood the goals and expectations for each journey. Understanding 4. The directions were clear for each journey. Navigability 5. I think the journeys are interesting. Engagement 6. I like the resources that were used in the lessons (i.e. YouTube videos, audio of the stories). Engagement Open-Ended Questions Students will then be given the following open-ended questions in order to provide further feedback on the site. 1. Did everything work correctly? Did you find any flaws, like typos, links that don t work, or images that don't load? 2. Is there anything you would change to make this site better or easier for you to use? 3. Do you think it s beneficial to explore this topic online? Why? 4. What do you like about the site? What do you dislike about the site? Ensuring Honesty Select students or users who you know will give you open and honest feedback and who will not be afraid to tell you if they find something about the site difficult to navigate or understand, or if they just plain don't like something. Ask questions and request users' total openness. Users will provide feedback via an online survey that does not require their name. This may make them feel more at ease with being honest because it truly does take away from identifying them. We will also offer space for students to comment in response to each question. USER FEEDBACK Six people provided feedback via Survey Monkey. Four of these responders were students. Two of these students were able to take the survey while at school; while two others were unable to stay and completed the review at home. The profiles of the students are summarized in the table on the following page.
Gender Academics Technological Skills Where Survey was Taken Student 1 Female Above-average Above-average School Student 2 Male Above-average Average Home Student 3 Female Average Average School Student 4 Female Below-average Below-average Home A fifth student, a male with average academics and below-average technological skills, was supposed to evaluate the site but was unable. In addition to the four students, two colleagues provided feedback as well. One is another sixth grade language arts teacher. The other is a sixth grade special education teacher who co-teaches with me. Their feedback is summarized by question on the following pages.
Suggestions 1. Did everything work correctly? Did you find any flaws, like typos, links that don t work, or images that don't load? 2. Is there anything you would change to make this site better or easier for you to use? 3. Do you think it s beneficial to explore this topic online? Why? 4. What do you like about the site? What do you dislike about the site?
CHANGES BASED ON FEEDBACK I received a lot of positive feedback about the site. It seemed that the most positive comments were regarding the variety of resources, such as YouTube, VoiceThread, etc. Additionally, their constructive criticism provided some great insight into how I could improve my site. The comments provided both in the ratings section and the open-ended question section proved to be the more useful than the tally of their ratings. One student who reviewed the site at home stated, I couldn't get the checklist to open on my computer. Although I converted other documents on the site into PDFs, I hadn t done so for that one; it was still saved as a.docx. I will be converting it and adding the PDF to the site to ensure that users can access the files. Another comment was, The movie clips took a little while to show up. Journey 4 contains five different YouTube clips embedded into the page. However, I think that each clip is valuable enough to stay and that I will not make any changes to that page. One student was very concerned with grades. At two different points in the survey, she had commented, I wasn't sure what parts we would be graded on and Add the rubric for how we'll be graded. Honestly, I am undecided on whether or not I would like to accept this feedback. As we have discussed in class, I want the focus to be on the experience, not the grade. A student had commented, I needed more directions for the last one and a colleague had a similar suggestion. I would like to add some brainstorming questions to help the students get started on collaboratively writing a story. GoogleDocs is something that I hope to introduce in the classroom though, so I don t feel that directions will be needed for that. Another comment was, Tell me if I'm suposed to keep going or stop at the end of each page. I combined this comments with my own thoughts about the Welcome page for the project. Although no one specifically commented on that welcome page, I felt it was a little bland. In the online unit that I created in Teaching K-12 Students Online, my homepage included a brief video in which I introduced the lesson as well as a Google calendar of when the students were to have each part completed. I am considering adding this to my site as well. One student commented, The story sounds like a lot of work. To that feedback, I would just say, Yes, it will be work. As the culminating project for this assignment, I am ok with the students realizing that it will require effort on their part.