Minnesota Tablet Usability Study Report

Transcription

1 Background Minnesota Tablet Usability Study Report May 2015 The Minnesota Comprehensive Assessments (MCA) are currently administered using desktop and laptop computers. With the recent rise in the use of tablet computers, the Minnesota Department of Education (MDE) will allow students to test on tablets beginning in the school year. MDE and Pearson conducted a usability study to evaluate whether there are features of the test items or of the tablets that might disadvantage students testing on tablets. Previous research on the use of tablets for testing suggests three major differences between desktop/laptop computers and tablets that may pose challenges for students using tablets to respond to long reading questions and technology-enhanced reading, math and science items. 1. Screen size The reduced screen size of tablets compared to desktop/laptop computers often requires additional scrolling and paging through long reading passages, or difficulty interacting with technology-enhanced reading, math, and science items. 2. Touch-screen keyboards rather than physical keyboards Tablet computers offer no resting state for fingers when using the touch-screen keyboard, thereby limiting one s ability to use keyboarding skills. Touch-screen keyboards pop-up and cover screen real estate while in use, making the effective screen size even smaller. 3. Lack of a mouse Fingers are less precise than a mouse when used as a pointer. Without a mouse, it is more difficult to place the cursor at the desired point for text revision. Previous Pearson research on student experiences with tablets is found at the following address: Purpose Because none of the tablet research to date has been conducted specifically using Minnesota students and Minnesota test content, MDE commissioned a study. The purpose of the Minnesota Student Tablet Study was to capture and summarize student experiences taking Minnesota test questions on a tablet. The results of the study would inform implementation decisions regarding allowable devices for testing. The study was aimed at 4 th and 9 th graders. Method A brief training was initially provided by Pearson staff for district and school technology staff and administrators. Using standardized scripts and materials, participating students were administered a set of exemplar MCA test items using a tablet and then responded to a brief survey about their experience. The test items were not scored. The districts / schools were advised that they had the option to use external keyboards. Pearson did not collect that detail from the participating districts or schools as it was not defined initially as a key requirement. The items presented to the students included long reading passages, and technology-enhanced items from reading, math, and science. Items that were selected for the study were those that required the student to interact with the questions by scrolling, using the touch-screen keyboard, and using their finger for response selections. 1 M innesota Tablet Study

2 Students were only permitted to use tablets that met the requirements of the TestNav 8 testing platform. The table below lists the features of the eligible tablets. All participating districts/schools were responsible for providing the tablets for student test-taking. Tablet Requirements for Study Participation Tablet Type Screen Size Minimum Operating System ipad 9.7 inches ios 7 and ios 8 Android 10.1 inches Android Lollipop Participants Participation in the study was voluntary. The Minnesota District or School Consent Form was provided to each participating institution and was returned signed by one or more administrative staff members. The district (or school) level consent form indicates that notification for student participation was handled by each district/school. MDE was provided the consent form prior to Pearson distributing for the study. Signed forms were received from the following districts: Minnetonka, Wadena-Deer Creek, Wayzata, Win-E-Mac. A total of 110 students representing 5 schools and 4 districts participated in the tablet study. Twenty-two (20%) were from grade 4 and eighty-eight (80%) were from grade 9. Of the participants, 55% were female and 45% were male. The sample was composed predominately of white students (85%), followed by Asian (6%), Hispanic (6%), black (2%) and Indian (1%) students. Results A summary of responses to all survey questions can be found in the appendix of this document. Key findings are summarized below: Previous Tablet Use 58% of students reported using tablets at home for fun 75% of students reported using tablets for school work 74% of students reported having an arrangement with their school that allows them to take tablets home 82% of students reported having used a tablet to take a test or quiz at school Responses to Tablet Administration of MCA Items 88% of students said something unexpected happened never or rarely when clicking on something in the test 78% said they could see everything they needed on the test most or all of the time 89% of students reported zooming in never or rarely 9% said zooming made it more difficult to answer the question 70% used the onscreen keyboard only 91% reported that it was easy or very easy to use the keyboard 5% of students thought scrolling made answering the items very hard and an additional 35% thought it was somewhat hard 74% reported having no problem seeing the animations and simulations 81% had no problems using the tablet to answer math questions 2 M innesota Tablet Study

3 Discussion/Conclusion The results from this small-scale study suggest that the participants were relatively adept at using tablet computers both at home and at school. In addition, the majority of students surveyed had little difficulty using the tablet to take the test and respond to the items. However, a non-trivial proportion of students reported problems with scrolling and seeing animations/simulations. Limitations The results of this study should be interpreted with caution for several reasons. First, the sample used is not of sufficient size or representativeness to permit drawing strong inferences about the full population of Minnesota students. The sample size did not permit the disaggregating of results to investigate patterns among various subgroups. In addition, the participants comprised a convenience sample, including only those schools who volunteered to participate. It is possible, if not likely, that the participating districts are the most ardent users of tablets. It is unclear at this point whether the results would generalize to the state as a whole, or to students/schools that are not as adept with using tablets. Furthermore, only students in two grades were surveyed and the sample was heavily weighted toward 9 th graders, with roughly four times as many 9 th graders as 4 th graders. It is not known whether the results observed here will generalize to other grade levels. Additionally, it is not known whether students who did not use an external keyboard chose not to, or were not given that option by their school or district. Finally, students may not have been motivated to take the survey seriously, which might limit the validity of the results. 3 M innesota Tablet Study

4 Appendix: Survey Responses for All Questions The tables below provide the number and percentage of students who selected each option for each survey question. Note that percentages will add up to 100 (within rounding) in cases where only one choice was permitted. However, for multiple response questions (i.e., check all that apply), the percentages may exceed 100 since multiple selections were possible. 6. Which of the following do you regularly use at home for fun (for example: playing games, talking with friends, etc.)? (check all that apply) Desktop computer % Laptop computer % Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, % Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) % None of the above % 7. Which of the following do you regularly use for school work (for example: homework, classroom work, research, etc.)? (Check all that apply) Desktop computer % Laptop computer % Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, % Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) % None of the above % 8. In which classes in school do you use a tablet? (Check all that apply) English/Language Arts class % Math class % Science class % Social Studies class % Music or Art class % I do not use a tablet in school % 9. Which of the following best describes the tablet you use for school work? The tablet is assigned to me at the beginning of the year and I can % take it home with me. The tablet is assigned to me at the beginning of the year and it % stays at school. The tablet is checked out to me for certain classes or activities but % returned at the end of the class or activity. The tablet I use for school is one that I personally own % I do not use a tablet for school work % 4 M innesota Tablet Study

5 10. For what types of school activities do you use your tablet? (Check all that apply) Research/looking things up % Creating projects % Taking pictures % Watching videos % Checking grades % Doing homework % ing or messaging the teacher % ing or messaging other students % Reading books or stories % Writing essays or papers % Group projects % None of the above % 11. How many text messages do you usually send each day? I do not send text messages % Fewer than 10 text messages % Between 10 and 50 text messages % Between 50 and 100 text messages % More than 100 text messages % 12. Which of the following have you EVER used to take a test or quiz at school? (Check all that apply) Desktop computer % Laptop computer % Touch-screen tablet (for example: ipad, Kindle Fire, Galaxy Tab, % Nexus 7, etc.) Smart phone (a phone with a touch screen and/or keyboard) % None of the above % 13. When taking the test how often did something unexpected (something other than what you meant to do) happen when you clicked or tapped on part of a test question? Never (0 times) % Rarely (1-2 times) % Sometimes (3-5 times) % Often (more than 5 times) % 14. When taking the test could you see everything on the screen at the same time that you need to answer the questions on the test (the story, the question itself, the place where you put your answer, etc.)? No, I could not ever see everything I needed % Yes, I could see what I needed some of the time % Yes, I could see what I needed most of the time % Yes, I could see what I needed all of the time % 5 M innesota Tablet Study

6 15. How often did you have to make the parts of the test item larger (zoom in) or smaller (zoom out) in order to answer the question? Never (0 times) % Rarely (1-2 times) % Sometimes (3-5 times) % Often (more than 5 times) % 16. On which tests did you have to make images larger (zoom in) or smaller (zoom out)? (Check all that apply) Reading % Science % Math % No subject areas % 17. In your opinion, did the process of making the test larger (zoom in) or smaller (zoom out) seem to make it harder to answer the test item? Yes % No % I am not sure % I did not have to make any test items larger or smaller % 18. In your opinion, did it seem like you had to spend more time to answer the test items that needed to be made larger (zoom in) or smaller (zoom out)? Yes % No % I am not sure % I did not have to make any test items larger or smaller % Missing response % 19. Did you use the onscreen keyboard or external keyboard? Onscreen only % External keyboard % I used both % 20. How easy or hard was it to use the keyboard to take your test? Very Easy % Easy % Hard % Very Hard % 6 M innesota Tablet Study

7 21. Did scrolling through the reading passages make it hard to answer the test questions? Scrolling did not make it hard to answer the reading questions % Scrolling made it somewhat hard to answer the reading questions % Scrolling made it very hard to answer the reading questions % 22. On the Science test did you have any problems seeing the animations and simulations clearly enough to answer the questions? I did not have any problems seeing the animations and simulations % I had some problems seeing the animations and simulations % I had a lot of problems seeing the animations and simulations % Missing response % 23. On the Math test did you have any problems using the tablet to answer the questions? I did not have any problems using the tablet to answer the math % questions I had some problems using the tablet to answer the math questions % I had a lot of problems using the tablet to answer the math % questions Missing response % 7 M innesota Tablet Study