Background Minnesota Tablet Usability Study Report May 2015 The Minnesota Comprehensive Assessments (MCA) are currently administered using desktop and laptop computers. With the recent rise in the use of tablet computers, the Minnesota Department of Education (MDE) will allow students to test on tablets beginning in the 2015-2016 school year. MDE and Pearson conducted a usability study to evaluate whether there are features of the test items or of the tablets that might disadvantage students testing on tablets. Previous research on the use of tablets for testing suggests three major differences between desktop/laptop computers and tablets that may pose challenges for students using tablets to respond to long reading questions and technology-enhanced reading, math and science items. 1. Screen size The reduced screen size of tablets compared to desktop/laptop computers often requires additional scrolling and paging through long reading passages, or difficulty interacting with technology-enhanced reading, math, and science items. 2. Touch-screen keyboards rather than physical keyboards Tablet computers offer no resting state for fingers when using the touch-screen keyboard, thereby limiting one s ability to use keyboarding skills. Touch-screen keyboards pop-up and cover screen real estate while in use, making the effective screen size even smaller. 3. Lack of a mouse Fingers are less precise than a mouse when used as a pointer. Without a mouse, it is more difficult to place the cursor at the desired point for text revision. Previous Pearson research on student experiences with tablets is found at the following address: http://researchnetwork.pearson.com/tablets Purpose Because none of the tablet research to date has been conducted specifically using Minnesota students and Minnesota test content, MDE commissioned a study. The purpose of the Minnesota Student Tablet Study was to capture and summarize student experiences taking Minnesota test questions on a tablet. The results of the study would inform implementation decisions regarding allowable devices for testing. The study was aimed at 4 th and 9 th graders. Method A brief training was initially provided by Pearson staff for district and school technology staff and administrators. Using standardized scripts and materials, participating students were administered a set of exemplar MCA test items using a tablet and then responded to a brief survey about their experience. The test items were not scored. The districts / schools were advised that they had the option to use external keyboards. Pearson did not collect that detail from the participating districts or schools as it was not defined initially as a key requirement. The items presented to the students included long reading passages, and technology-enhanced items from reading, math, and science. Items that were selected for the study were those that required the student to interact with the questions by scrolling, using the touch-screen keyboard, and using their finger for response selections. 1 M innesota Tablet Study
Students were only permitted to use tablets that met the requirements of the TestNav 8 testing platform. The table below lists the features of the eligible tablets. All participating districts/schools were responsible for providing the tablets for student test-taking. Tablet Requirements for Study Participation Tablet Type Screen Size Minimum Operating System ipad 9.7 inches ios 7 and ios 8 Android 10.1 inches Android Lollipop Participants Participation in the study was voluntary. The Minnesota District or School Consent Form was provided to each participating institution and was returned signed by one or more administrative staff members. The district (or school) level consent form indicates that notification for student participation was handled by each district/school. MDE was provided the consent form prior to Pearson distributing for the study. Signed forms were received from the following districts: Minnetonka, Wadena-Deer Creek, Wayzata, Win-E-Mac. A total of 110 students representing 5 schools and 4 districts participated in the tablet study. Twenty-two (20%) were from grade 4 and eighty-eight (80%) were from grade 9. Of the participants, 55% were female and 45% were male. The sample was composed predominately of white students (85%), followed by Asian (6%), Hispanic (6%), black (2%) and Indian (1%) students. Results A summary of responses to all survey questions can be found in the appendix of this document. Key findings are summarized below: Previous Tablet Use 58% of students reported using tablets at home for fun 75% of students reported using tablets for school work 74% of students reported having an arrangement with their school that allows them to take tablets home 82% of students reported having used a tablet to take a test or quiz at school Responses to Tablet Administration of MCA Items 88% of students said something unexpected happened never or rarely when clicking on something in the test 78% said they could see everything they needed on the test most or all of the time 89% of students reported zooming in never or rarely 9% said zooming made it more difficult to answer the question 70% used the onscreen keyboard only 91% reported that it was easy or very easy to use the keyboard 5% of students thought scrolling made answering the items very hard and an additional 35% thought it was somewhat hard 74% reported having no problem seeing the animations and simulations 81% had no problems using the tablet to answer math questions 2 M innesota Tablet Study
Discussion/Conclusion The results from this small-scale study suggest that the participants were relatively adept at using tablet computers both at home and at school. In addition, the majority of students surveyed had little difficulty using the tablet to take the test and respond to the items. However, a non-trivial proportion of students reported problems with scrolling and seeing animations/simulations. Limitations The results of this study should be interpreted with caution for several reasons. First, the sample used is not of sufficient size or representativeness to permit drawing strong inferences about the full population of Minnesota students. The sample size did not permit the disaggregating of results to investigate patterns among various subgroups. In addition, the participants comprised a convenience sample, including only those schools who volunteered to participate. It is possible, if not likely, that the participating districts are the most ardent users of tablets. It is unclear at this point whether the results would generalize to the state as a whole, or to students/schools that are not as adept with using tablets. Furthermore, only students in two grades were surveyed and the sample was heavily weighted toward 9 th graders, with roughly four times as many 9 th graders as 4 th graders. It is not known whether the results observed here will generalize to other grade levels. Additionally, it is not known whether students who did not use an external keyboard chose not to, or were not given that option by their school or district. Finally, students may not have been motivated to take the survey seriously, which might limit the validity of the results. 3 M innesota Tablet Study
Appendix: Survey Responses for All Questions The tables below provide the number and percentage of students who selected each option for each survey question. Note that percentages will add up to 100 (within rounding) in cases where only one choice was permitted. However, for multiple response questions (i.e., check all that apply), the percentages may exceed 100 since multiple selections were possible. 6. Which of the following do you regularly use at home for fun (for example: playing games, talking with friends, etc.)? (check all that apply) Desktop computer 24 21.82% Laptop computer 43 39.09% Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, 64 58.18% Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) 85 77.27% None of the above 3 2.73% 7. Which of the following do you regularly use for school work (for example: homework, classroom work, research, etc.)? (Check all that apply) Desktop computer 39 35.45% Laptop computer 45 40.91% Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, 83 75.45% Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) 25 22.73% None of the above 5 4.55% 8. In which classes in school do you use a tablet? (Check all that apply) English/Language Arts class 75 68.18% Math class 88 80.00% Science class 83 75.45% Social Studies class 54 49.09% Music or Art class 17 15.45% I do not use a tablet in school 7 6.36% 9. Which of the following best describes the tablet you use for school work? The tablet is assigned to me at the beginning of the year and I can 81 73.64% take it home with me. The tablet is assigned to me at the beginning of the year and it 3 2.73% stays at school. The tablet is checked out to me for certain classes or activities but 16 14.55% returned at the end of the class or activity. The tablet I use for school is one that I personally own. 2 1.82% I do not use a tablet for school work. 8 7.27% 4 M innesota Tablet Study
10. For what types of school activities do you use your tablet? (Check all that apply) Research/looking things up 92 83.64% Creating projects 68 61.82% Taking pictures 49 44.55% Watching videos 53 48.18% Checking grades 81 73.64% Doing homework 83 75.45% E-mailing or messaging the teacher 57 51.82% E-mailing or messaging other students 30 27.27% Reading books or stories 47 42.73% Writing essays or papers 70 63.64% Group projects 62 56.36% None of the above 6 5.45% 11. How many text messages do you usually send each day? I do not send text messages 18 16.36% Fewer than 10 text messages 25 22.73% Between 10 and 50 text messages 35 31.82% Between 50 and 100 text messages 16 14.55% More than 100 text messages 16 14.55% 12. Which of the following have you EVER used to take a test or quiz at school? (Check all that apply) Desktop computer 87 79.09% Laptop computer 69 62.73% Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, 90 81.82% Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) 9 8.18% None of the above 1 0.91% 13. When taking the test how often did something unexpected (something other than what you meant to do) happen when you clicked or tapped on part of a test question? Never (0 times) 39 35.45% Rarely (1-2 times) 49 44.55% Sometimes (3-5 times) 18 16.36% Often (more than 5 times) 4 3.64% 14. When taking the test could you see everything on the screen at the same time that you need to answer the questions on the test (the story, the question itself, the place where you put your answer, etc.)? No, I could not ever see everything I needed 4 3.64% Yes, I could see what I needed some of the time 21 19.09% Yes, I could see what I needed most of the time 49 44.55% Yes, I could see what I needed all of the time 36 32.73% 5 M innesota Tablet Study
15. How often did you have to make the parts of the test item larger (zoom in) or smaller (zoom out) in order to answer the question? Never (0 times) 63 57.27% Rarely (1-2 times) 36 32.73% Sometimes (3-5 times) 9 8.18% Often (more than 5 times) 2 1.82% 16. On which tests did you have to make images larger (zoom in) or smaller (zoom out)? (Check all that apply) Reading 21 19.09% Science 20 18.18% Math 16 14.55% No subject areas 70 63.64% 17. In your opinion, did the process of making the test larger (zoom in) or smaller (zoom out) seem to make it harder to answer the test item? Yes 10 9.09% No 31 28.18% I am not sure 19 17.27% I did not have to make any test items larger or smaller 50 45.45% 18. In your opinion, did it seem like you had to spend more time to answer the test items that needed to be made larger (zoom in) or smaller (zoom out)? Yes 18 16.36% No 28 25.45% I am not sure 15 13.64% I did not have to make any test items larger or smaller 48 43.64% Missing response 1 0.91% 19. Did you use the onscreen keyboard or external keyboard? Onscreen only 77 70.00% External keyboard 11 10.00% I used both 22 20.00% 20. How easy or hard was it to use the keyboard to take your test? Very Easy 34 30.91% Easy 66 60.00% Hard 10 9.09% Very Hard 0 0.00% 6 M innesota Tablet Study
21. Did scrolling through the reading passages make it hard to answer the test questions? Scrolling did not make it hard to answer the reading questions 66 60.00% Scrolling made it somewhat hard to answer the reading questions 38 34.55% Scrolling made it very hard to answer the reading questions 6 5.45% 22. On the Science test did you have any problems seeing the animations and simulations clearly enough to answer the questions? I did not have any problems seeing the animations and simulations 81 73.64% I had some problems seeing the animations and simulations 25 22.73% I had a lot of problems seeing the animations and simulations 3 2.73% Missing response 1 0.91% 23. On the Math test did you have any problems using the tablet to answer the questions? I did not have any problems using the tablet to answer the math 89 80.91% questions I had some problems using the tablet to answer the math questions 17 15.45% I had a lot of problems using the tablet to answer the math 2 1.82% questions Missing response 2 1.82% 7 M innesota Tablet Study