White Paper. The Changing Landscape of Technology and its Effect on Online Survey Data Collection. by Nicole Mitchell, Knowledge Specialist, June 2014

Transcription

1 White Paper The Changing Landscape of Technology and its Effect on Online Survey Data Collection by Nicole Mitchell, Knowledge Specialist, June 2014 Survey Sampling International, 2014

2 ABOUT THE AUTHOR Nicole Mitchell has been with SSI for 14 years. She began as a Project Director delivering telephone research projects, then became a Senior Project Manager delivering sample and data collection solutions for online projects. In her current role as a knowledge specialist on the SSI Knowledge Team she uses her multi-mode expertise to advise clients on the best solutions for their research, and to conduct research on new data collection modes, especially mobile. Nicole has a BA in Psychology from University of Connecticut and a MA in Psychology from Fairfield University. (continued) 2

3 According to a report published in January of 2014 by Pew Research Center, 58% of US adults own a smartphone. Since 2011, smartphone ownership has increased by 23 percentage points. More shockingly, the growth of tablet ownership increased from 3% in 2010 to 42% in As ownership of these devices increases, it is essential to include these respondents in our samples if the samples are to continue to represent the online population frame. Limited research has been conducted regarding the questionnaire design aspect of online survey research via mobile devices, despite the fact that a growing number of respondents are attempting to take surveys on their mobile devices (Cazes et al. 2011). Previous research suggests that online surveys taken on mobile devices tend to have high drop off rates especially if the survey has not been optimized for mobile devices. In addition, past research demonstrates that respondents who took an online survey via mobile web took longer to complete the survey than those who took the survey using a mobile app or PC web. The most recent contribution to this research was by Nielsen and NPD Group (2013). These studies observed how respondents took surveys on a smartphone versus on a computer. Smartphone users were administered the survey via a survey app which was programmed for all the major smartphone operating systems. Non-smartphone respondents were sent an invitation containing the survey link to the same questionnaire. The questionnaire contained 24 questions regarding consumer behavior, internet usage and TV viewing habits. Although the questionnaire included short questions and response lists and did not contain any grid type questions, it was not optimized for smartphone web administration. Seven hundred and five respondents completed using the app and 771 complete the survey online. Eighteen percent of respondents who were instructed to take the survey using a PC or laptop completed the survey using a smartphone. In addition, 4.6% of mobile respondents who were instructed to complete the survey via the smartphone app completed the survey using a tablet. Consistent with previous findings, dropout rates were higher among mobile web respondents and mobile app respondents. No significant difference was found between tablet and computer respondents. Findings regarding completion rate were also consistent with previous findings. Mobile web respondents took more time than others to complete the survey. Lastly, the study found no significant differences in item non-response by mode or platform. SSI S NEW RESEARCH INTO MOBILE POPULATIONS > SSI examined the SSI system survey traffic (representing all respondents available to take online surveys) to understand the demographic characteristics of survey respondents who utilize these devices to take surveys. > Additionally, SSI undertook research which uncovered key differences and similarities between smartphone and tablet behaviors, such as: o Activities conducted on each device o Behaviors while using the device to complete a survey o Abandon rate o Length of time to complete the questionnaire > Further, the research compared the data quality obtained via smartphone and tablet responses to that obtained from laptop and desktop users. 3

4 > Lastly, SSI analyzed the consequence of excluding data received from tablets and smartphones. METHODOLOGY Participants SSI online sample consisting of 5501 US respondents age 18+: > 1435 tablet respondents > 1869 smartphone respondents > 2197 laptop/desktop respondents Instruments The questionnaire included questions on demographics, activities done on the device, where they use the device, other tasks performed while using the device, ownership of other technology, and quality control questions. There were three different survey treatments: Mobile unfriendly > The survey page is a smaller version of a traditional desktop/laptop webpage. > The survey page is not responsive to different screen sizes and it needs zoom-in or horizontal scrolling (e.g. the screenshot below only shows half the page). > The font size is small and survey elements are tiny and difficult to read and select. Mobile friendly > The survey page is rendered properly from different screen sizes but loading speed is not optimized. > It is clear to read and easy to select. > Horizontal scrolling is minimized. > Survey has a similar look and feel as a PC browser but adapted to mobile. > The traditional grid question is asked as multiple single answer questions across all devices. Mobile optimized > Designed specifically for a mobile device where the survey program recognizes the device and optimizes the survey for mobile respondents. > The survey layout is optimized by removing unnecessary elements. > There is no horizontal scrolling. > The grid question is shown as multiple single answer questions on the mobile phone. However, on tablets, laptops and desktops the traditional grid format is shown. 4

5 PROCEDURE In addition to self-reported data regarding the device they are using to take the survey, we captured and stored operating system information for all respondents to determine which device they were using to take the survey. Respondents were randomly assigned to one of the three survey treatments. Age and gender quotas based on the US Census were created to ensure each treatment group would have a representative number of respondents from each device category: tablet, smartphone, laptop/desktop. Characteristics of the online survey-taking population When looking at the overall US stream of SSI survey respondents who began a survey in the past six months by device, the use of laptops and desktops still prevails among online survey takers with 79% of respondents taking a survey using a laptop or desktop computer. However, 21% percent of respondents used a mobile phone or tablet to participate in an SSI survey in the past six months. Eleven percent used their mobile phone, while 10% used their tablet to participate. Again looking at all US respondents who began an SSI survey in the past six months, as age increases, the number of participants utilizing their mobile phone to take an online survey decreases. Fifteen percent of respondents age utilized their mobile phone to take a survey. Conversely, only 3% of respondents age 65 and older used a mobile phone to take a survey. Age % Desktop/Laptop % Tablet % Mobile Phone % 10% 15% % 10% 11% % 10% 10% % 9% 7% % 9% 5% % 7% 3% Although laptop/desktop still reigns as the device most used to take a survey, use of mobile devices is more prevalent among younger people. Forty-nine percent of respondents who access surveys using their mobile phone were between the ages of 18 and 34 while 35% using a tablet to access the survey were between the ages of 18 and 34. 5

6 Age by Device Mobile Phone Tablet Desktop/Laptop 27% 22% 13% 11% 22% 20% 21% 19% 17% 20% 18% 19% 18% 16% 10% 10% 14% 4% When we examined the US traffic by ethnicity, we saw that African Americans (17% of all mobile phone participants) and Hispanics (16% of all mobile phone participants) were more likely to participate using their mobile phone. Race/Ethnicity by Device Mobile Phone Tablet Desktop/Laptop 59% 71% 74% 17% 9% 9% 4% 6% 5% 16% 12% 9% 5% 3% 3% African American Asian Caucasian Hispanic Other Lastly, we examined income across device and found that respondents accessing the survey with their mobile phone tend to have lower incomes than respondents who took the survey using a tablet or laptop/desktop. This finding is expected since younger people and ethnic minorities tend to have lower incomes than their counterparts. 6

7 Income by Device Mobile Phone Tablet Desktop/Laptop 44% 28% 22% 22% 23% 21% 16% 23% 22% 20% 14% 15% 13% 8% 9% Less than $29,999 $30,000-$49,999 $50,000-$74,999 $75,000-$99,999 $100,000+ Technology ownership and attitudes Now we turn to the survey data, where we asked respondents about their attitudes and opinions about technology. Respondents who used a mobile phone or tablet to take the survey were more likely to identify with the statement I know I m not the first to adopt a new technology, but I m faster than the average person. I like to see it mature a little bit before I take the plunge. Often I ll check products out with other users or research it a bit than respondents who took the survey on their laptop/desktop (35% mobile and 37% tablet vs. 27% for laptop/desktop). Thirty two percent of laptop/desktop respondents noted that the statement that best described them in regards to adopting new technologies was I like to wait for a new technology to mature a lot before I adopt it. I guess I m a little skeptical. Often I ll wait until I have to buy a replacement or my friends will push me into buying one. Tablet survey takers (19%) were more likely than mobile phone (13%) and laptop/desktop (14%) survey takers to state that they embrace new technology before most other people they know and that others often ask them their opinion on new products and technologies. 7

8 Tech Adoption by Device Mobile Tablet Laptop/Desktop 37% 35% 32% 27% 27% 7% 10% 9% 13% 19% 14% 20% 13% 9% 16% I usually know about new products before they are launched and I am eager to try them. It often feels like I m the first person to have a new product. I d say I embrace new technology before most other people I know do. Others often ask me about my opinion on new products and technologies. I know I m not the first to adopt a new technology, but I m faster than the average person. I like to see it mature a little bit before I take the plunge. Often I ll check products out with other users or research it a bit. I like to wait for a new technology to mature a lot before I adopt it. I guess I m a little skeptical. Often I ll wait until I have to buy a replacement or my friends will push me into buying one. I m really not into new things. I m perfectly happy with the old things. In fact by the time I adopt a new technology it s probably not so new anymore. In addition to indicating that they are faster than the average person when it comes to adopting new technology, mobile phone (62%) and tablet (65%) survey takers are less likely to need someone else to set it up or show them how to use a new electronic device when they get one than respondents taking a survey on their laptop/desktop (52%). 8

9 When I get a new electronic device, I usually need someone else to set it up or show me how to use it. 100% 80% 60% 40% 20% 62% 65% 52% 11% 10% 13% 28% 35% 25% Mobile Phone Tablet Laptop/Desktop 0% Strongly/Somewhat Disagree Neither agree nor disagree Strongly/Somewhat Agree Mobile phone survey takers demonstrated a slightly significant difference in their attitude towards managing multiple electronic devices than respondents taking the survey using their tablet or laptop/desktop. Fourteen percent of mobile phone respondents indicated that they agreed with the statement It is stressful to own and manage all of the different electronic devices I have. In contrast, 18% of tablet survey takers and 18% of laptop/desktop survey takers agreed with the statement. It is stressful to own and manage all of the different electronic devices I have 100% 90% 80% 70% 60% 50% 40% 30% 20% 65% 62% 58% 21% 21% 23% 14% 18% 18% Mobile Phone Tablet Laptop/Desktop 10% 0% Strongly/Somewhat Disagree Neither agree nor disagree Strongly/Somewhat Agree 9

10 In-survey behaviors Respondents who received the mobile unfriendly version were more likely to abandon the survey if they were on a mobile phone (21%) or tablet (9%) than respondents using a laptop/desktop (4%) to access the survey. The mobile unfriendly and mobile friendly implementations of the questionnaire had the lowest rate of abandonment among respondents accessing the survey utilizing a laptop/desktop. Conversely, the abandon rate for mobile optimized and mobile friendly versions of the questionnaire was lowest among respondents who used a mobile phone or tablet to access the survey. 25% 20% 21% Abandon Rate by Device 15% 10% 5% 9% 4% 11% 11% 6% 7% 6% 4% 0% Mobile Unfriendly Mobile Friendly Mobile Optimized Mobile Phone Tablet Laptop/Desktop Length of interview (LOI) by device The overall median completion time (LOI) was 9.75 minutes across all devices. The median time among respondents who accessed the survey using a mobile phone was minutes, while the median time for tablet respondents was minutes. Laptop/desktop respondents took the least amount of time to complete the survey with a median time of 7.62 minutes. Median LOI Device (minutes) Overall 9.75 Mobile Tablet Laptop/Desktop

11 Minutes Length of Interview by questionnaire implementation and device Across all devices, the mobile optimized questionnaire design took the least amount of time to complete. Respondents on a mobile device completed the optimized version of the question in minutes, tablet respondents completed this version in 9.60 minutes and laptop/desktop users completed the optimized version in 7.02 minutes. Not surprisingly, the laptop/desktop users also completed the mobile unfriendly version in the same amount of time as the mobile optimized version. Overall, respondents who completed the survey using a laptop/desktop took the least amount of time to complete the survey for all questionnaire versions. Median LOI (minutes) Implementation Mobile (1) Tablet (2) Laptop/Desktop (3) Mobile Unfriendly (1) Mobile Optimized (2) Mobile Friendly (3) Median LOI Mobile Phone Tablet 4 Laptop/Desktop 2 0 Mobile Unfriendly Mobile Friendly Mobile Optimized Location while taking survey Across all devices the majority of respondents were at home when they were taking the survey. However, respondents taking the survey on a laptop/desktop (12%) were almost twice as likely to be at work than respondents taking the survey on a mobile phone (6%) or tablet (5%). As expected, mobile phone respondents were more likely to be on the go than respondents who accessed the survey using their tablet or laptop/desktop. On the go respondents were either in a public enclosed space (e.g. a bar, a café, a library), open space (e.g. your backyard or a park), on public transportation (bus, plane, train, etc.), in a car, or somewhere else. 11

12 Location by Device Mobile phone Tablet Laptop/Desktop 87% 82% 82% 6% 5% 12% 0% 1% 0% 12% 8% 6% At home At work At school On the go Multitasking while taking survey Overall, 78% of respondents stated that they were just doing the survey, while the remainder said they were doing other things at the same time. Respondents taking the survey on a laptop/desktop (18%) were less likely to engage in multi-tasking behavior than their cohorts using a mobile phone (26%) or tablet (24%). 12

13 Multi-tasking by Device Mobile Tablet Laptop/Desktop 74% 76% 82% 26% 24% 18% Just doing this survey Doing other things at the same time We then examined how many activities respondents were involved in while taking our survey. The findings suggest that there were no statistically significant differences found in the number of activities among those who engaged in multi-tasking behavior and the device they took the survey on. Number of Additional Activities While Doing Survey Mobile Tablet Laptop/Desktop 56% 60% 59% 27% 24% 25% 17% 16% 16% Behaviors while multi-tasking 13

14 Among those who multi-tasked while taking our survey, almost half (48%) were watching a TV program. Respondents using a tablet (56%) were more likely to watch a TV program while taking the survey than those who took the survey on a mobile phone (49%) or laptop/desktop (39%). 56% 49% 39% Activities by Device Mobile Phone Tablet Laptop/Desktop 23% 23% 22% 17% 19% 18% 15% 16% 17% 13% 14% 14% 13% 10% 9% 10% 11% 13% 9% 10% 7% 8% 10% 6% 7% 7% 8% 4% 5% 5% 2% 2% 3% DATA QUALITY An important question in our research was Will the data quality of smartphone and tablet responses be worse than that of laptop and desktop users? The following quality control tests were implemented to examine data quality: 1. Speeding 2. Straightlining 3. Selecting low incidence answers 4. Conflicting answers Speeding 14

15 Speeders were defined as any respondent who completed the study in less than 48% of the median length of interview (LOI). Using the overall median time of 9.75, we flagged any respondent regardless of device who completed the survey in less than 48% of the overall median time. 5.6% of respondents were flagged for speeding. However when we examined completion speed by device, we found that respondents who accessed the survey using their tablet or laptop/desktop were disproportionately flagged as speeders. Speeder Check Based on Overall Median LOI Total Mobile Tablet Laptop/Desktop 10.90% 5.60% 1.40% 4.10% Failed As noted earlier, tablet and laptop/desktop respondents completed the survey in less time than respondents who accessed their survey using a mobile phone. So we examined what would happen if we created a speeding rule for each device based on each device s median time. When we calculated the speeder check based on the mobile median LOI of 12.36, we flagged 28.1% of laptop/desktop respondents as speeders almost three times the number of respondents who were flagged using the overall median time. We also flagged 9.2% of tablet respondents as speeders which is more than double the amount flagged via the overall median LOI. Speeder Check Based on Mobile Median LOI Total Mobile Tablet Laptop/Desktop 28.10% 14.20% 3.70% 9.20% Failed 15

16 Calculating the speeder check rule based on the tablet median LOI flagged fewer laptop/desktop respondents than the mobile speeder check, but it still flagged more than the overall speeder check. The tablet speeder check flagged nearly the same number of tablet and mobile phone respondents as the overall speeder check. Speeder Check Based on Tablet Median LOI Total Mobile Tablet Laptop/Desktop 12.60% 6.40% 1.70% 4.20% Failed In comparison to the mobile phone and tablet speeding rule, the speed check based on the laptop/desktop median LOI flagged the fewest respondents across all devices. Speeder Check Based on Laptop/Desktop Median LOI Total Mobile Tablet Laptop/Desktop 3.90% 2.00% 0.50% 1.60% Failed Does this mean that the laptop/desktop median LOI is the best speeder check or is it safe to assume the overall speeder check is fair enough? The laptop/desktop speeder check may be too lax in determining data quality for the other two devices. As demonstrated in the chart below, when we created a speeding rule for each device based on flagging those who completed in less time than the median time for each device, we found that we flagged approximately the same number of 16

17 respondents for each device. Thus, we recommend using a speeder rule for each device based on the median time of that device and flagging respondents accordingly If this is not practical, we do not recommend using a speeder check alone, but only in conjunction with at least one other quality control check before removing respondents for providing poor data quality. Speeder Check Based on Median Time for Each Device Mobile Speeding Check Tablet Speeding Check Laptop/Desktop Speeding Check 3.7% 4.2% 3.9% % Failed Straightlining A respondent was flagged for straightlining behavior if they provided the same answer for a set of nine statements that either appeared in the traditional grid format, in a series of single questions, or in a grid template designed for all devices. Only 2% of respondents failed the straightlining check. Overall, mobile respondents were less likely to engage in straightlining behavior than tablet and laptop/desktop respondents. 17

18 Straightline Check by Device Total Mobile Tablet Laptop/Desktop 2.00% 1.50% 2.10% 2.30% Failed Across all devices, the mobile unfriendly questionnaire had the largest percentage of respondents engaging in straightlining behavior. This is not surprising since regardless of device, respondents assigned to the mobile unfriendly version were exposed to the traditional grid question design. The questionnaire versions that were more accommodating to a mobile device had significantly fewer respondents engaging in straightlining behavior. Straightline Check by Survey Type Mobile Unfriendly Mobile Optimized Mobile Friendly 4.00% 1.70% 1.50% % Failed Although not statistically significant, we did find differences when we examined straightlining behavior among respondents who received the mobile unfriendly version by device. Respondents who accessed the survey using a mobile phone were slightly more likely to engage in straightlining behavior than respondents using a tablet or laptop/desktop. These results are consistent with previous findings by my colleague Pete Cape which suggest that the traditional grid encourages fast processing of items which can lead to straightlining behavior. 18

19 Straightlining: Mobile Unfriendly Across Device Mobile - Mobile Unfriendly PC/Laptop - Mobile Unfriendly Tablet - Mobile Unfriendly 4.30% 4.10% 3.70% Failed Conflicting answers The third quality control check flagged respondents who provided answers that were in opposition to one another. Respondents were presented with a five point or 10 point scale where they were asked to indicate whether the below statements either Described me completely or Doesn t describe me at all : > Most things are either black or white, there is rarely any middle ground > Everything is shades of grey, it all depends on your viewpoint Mobile phone respondents (9.10%) were more likely to provide conflicting answers when asked how much the above statements described them. Quality Check: Conflicting Answers by Device Total Mobile Tablet Laptop/Desktop 6.90% 9.10% 6.80% 5.00% Failed 19

20 Based on the above result and that the question sometimes appeared in a grid format, we examined this quality check among mobile phone respondents and the survey version they received. The results demonstrate that 14.8% of mobile phone respondents failed this check when they received the mobile unfriendly version of the survey. Conflicting Answers: Device by Survey Type Mobile - Mobile Unfriendly Mobile - Mobile Friendly 14.80% 9.60% Mobile - Mobile Optimized Mobile - Mobile Friendly Hybrid 6.70% 6.40% Failed When we removed the mobile phone respondents who received the mobile unfriendly traditional grid question, the percent of mobile phone respondents failing the conflicting answers quality check declined significantly. Presenting the traditional grid to respondents who access surveys on their mobile phone may result in poor data quality. Quality Check: Conflicting Answers by Device Total Mobile Tablet Laptop/Desktop 6.30% 7.50% 6.80% 5.00% Failed Selecting low incidence answers The final quality control check flagged respondents who selected two sets of statements that have a relatively low likelihood of occurring. If a respondent selected both Purchased a new house in the 20

21 past year and Purchased a condo in the past year OR selected both Purchased a new automobile in the past month and Purchased a used automobile in the past month they were flagged. Overall, less than 1 percent failed the low incidence quality control check. Overall quality So how does the quality picture look overall? Using the two or more rule to determine whether a respondent should be removed from the data due to poor behavior, we found no statistically significant differences between devices. Overall Quality by Device 5.0% 4.0% Total 3.0% 2.0% 1.0% 1.5% 1.1% 1.6% 1.7% Mobile Tablet Laptop/Desktop 0.0% Failed 2 or more In the above measurement of quality, we used three quality control checks plus the separate speeder checks for each device. Consequences of excluding mobile respondents Relying solely on respondents who utilize a laptop/desktop to access surveys significantly decreases the number of younger respondents available to participate in an online survey. Therefore, designing a survey that is compatible across all devices is an ideal way to increase feasibility for difficult to reach populations. As discussed earlier when we examined survey traffic by device, we saw that including mobile phone respondents affords greater access to younger respondents. Twenty-two percent of SSI s May 2014 mobile traffic consists of respondents By including these younger respondents from our mobile (tablet and phone) traffic stream we are able to reach out to an additional 25% of respondents aged In addition to increasing access to younger respondents, including mobile phone respondents increases reach and capacity when targeting African American and Hispanic populations. By including mobile phone respondents, we are able to increase our reach and capacity by 16% for African American respondents and 14% for Hispanic respondents. 21

22 Mobile Increases Feasibility Desktop Tablet Mobile Phone 15% 11% 16% 14% 10% 10% 8% 11% 75% 79% 76% 75% African American Hispanic In addition to greater feasibility, including mobile phone respondents expands the diversity of the sample in regards to opinions and attitudes. When we excluded respondents who used a mobile device (mobile phone or tablet) to access the survey, we see that the data tells two different stories. The data that contains mobile and non-mobile respondents indicates that one third of respondents feel they are faster than the average person when it comes to adopting new technologies. In contrast, the data that excludes mobile respondents suggests that almost a third of respondents were more skeptical and like to wait for the technology to mature a lot before they adopt it. 22

23 Attitudes Towards Technology Adoption 40% 35% 30% 25% 20% 15% 10% 5% 0% 9% 9% Eager to try & First to have 16% 15% Embrace before most other people 34% 28% Not the first to adopt but faster than the average 33% 28% Wait until tech matures a lot before adopting Not into new things All Devices 9% 16% 34% 28% 14% Excluding mobile devices 9% 15% 28% 33% 16% 16% 14% We also noticed differences when excluding mobile respondents when we examined attitudes about accessibility and mobile devices and how confident respondents are when setting up a new electronic device. Among those who agreed that they liked that cell phones and other mobile devices allow me to be more available to others, we see an eight percentage point difference between the sample that includes respondents regardless of device (79%) and the data that excludes respondents who access the survey utilizing a mobile device (71%). 23

24 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% I like that cell phones and other mobile devices allow me to be more available to others. 11% 8% Strongly/Somewhat Disagree 18% 14% Neither agree nor disagree 79% 71% Strongly/Somewhat Agree All Devices Excluding mobile devices A difference was also apparent when asked how much respondents agreed or disagreed with the statement When I get a new electronic device, I usually need someone else to set it up or show me how to use it. Fifty-nine percent of respondents from the all-device sample disagreed with this statement. When we exclude mobile device respondents, the percentage of respondents who disagreed with this statement decreases by seven percentage points. 24

25 70% 60% 50% When I get a new electronic device, I usually need someone else to set it up or show me how to use it 59% 52% 40% 30% 20% 10% 13% 11% 35% 30% All Devices Excluding mobile devices 0% Strongly/Somewhat Disagree Neither agree nor disagree Strongly/Somewhat Agree Lastly, we found a slightly significant difference between the two data sets when respondents were asked to indicate how much they agreed or disagreed with the statement It is stressful to own and manage all of the different electronic devices I have. The data set that included all device respondents resulted in 62% of respondents either strongly or somewhat disagreeing with this statement. When we examined this statement excluding mobile respondents we noticed a slight decrease, where 58% of respondents disagreed with the statement. 25

26 70% 60% It is stressful to own and manage all of the different electronic devices I have 62% 58% 50% 40% 30% 20% 23% 22% 18% 17% All Devices Excluding mobile devices 10% 0% Strongly/Somewhat Disagree Neither agree nor disagree Strongly/Somewhat Agree Although we see data shifts when we compare the results that include mobile respondents to the results that exclude mobile respondents, it is important to note that the differences are not very large. However, it is also important to remember that these differences are based on the proportion of mobile traffic in the sample being at 21%. If the proportion of mobile traffic in the sample was higher, we would expect to see greater shifts, which is why we recommend including mobile respondents in samples now while the differences are still small. In addition to examining attitudes and opinions toward technology, we also asked about general life activities. If we exclude mobile respondents how will this affect data that is not focused on technology? We asked all respondents to select the following activities that applied to them: > Purchased a new house in the past year > Purchased a condo in the past year > Purchased a new automobile in the past month > Purchased a used automobile in the past month > Got married in the past 6 months > Visited a museum in the past year > Purchased a new smartphone in the past 6 months (e.g. Blackberry, Android, iphone, HTC, etc.) > Purchased pet insurance in the past 6 months > Researched a product or service online in the past month > Flown in a helicopter in the past 6 months When we compared the sample that included respondents regardless of device to the sample that excluded respondents who utilized a mobile device to access the survey we did not find any significant 26

27 differences for the following activities: > Purchased a new house in the past year > Purchased a condo in the past year > Purchased a new automobile in the past month > Purchased a used automobile in the past month > Got married in the past 6 months > Visited a museum in the past year > Purchased pet insurance in the past 6 months > Flown in a helicopter in the past 6 months However, we did observe differences in the data for the below activities: > Purchased a new smartphone in the past 6 months (e.g. Blackberry, Android, iphone, HTC, etc.) > Researched a product or service online in the past month When all respondents are included 35% indicated that they have Purchased a new smartphone in the past 6 months (e.g. Blackberry, Android, iphone, HTC, etc). However, when we exclude respondents who access the survey utilizing a mobile device the percentage of respondents selecting this activity decreases by eight percentage points. These results are similar to the technology results; the differences are not large, but something to be mindful about as the proportion of mobile traffic in the sample increases over time. In addition, we observed data differences between the all-device sample and the non-mobile sample in regards to researching products or services online in the past month. The data that included respondents regardless of device demonstrated that 52% of respondents had engaged in this activity in the past month. However, when we exclude mobile respondents the percentage of respondents who reported that they researched a product or service online in the past month increased to 55%. 27

28 60% 50% Life Activities 52% 55% 40% 30% 20% 10% 35% 27% All Devices Excluding Mobile 0% Purchased a new smartphone in the past 6 months (e.g. Blackberry, Android, iphone, HTC, etc) Researched a product or service online in the past month The final finding appears to be quite a conundrum as to why more laptop/desktop respondents research products online. There are a couple of hypotheses which could explain why this occurred. Researching products or services online may be easier to do on a laptop/desktop because laptop/desktops have larger keyboards which make it easier to type. Furthermore, it is easier to toggle back and forth between web pages on a larger screen when comparing products and services. Or perhaps the explanation lies with the wording of the statement Researched a product or service online in the past month. If we change the wording to on your phone instead of online we may have seen a different result. Mobile phone respondents may technically be online but, may not think of themselves in that way. As ever in research we need to be really careful that we use the same language and the same concepts as the people we are researching. CONCLUSION The mobile survey-taking population is growing very fast; especially among difficult to reach groups such as young people and minorities. Therefore, including mobile populations increases research project feasibility, sometimes dramatically among these groups. Today, the mobile population is similar to the general population, with differences only seen in answers to questions related to technology. Therefore, now is the time to integrate them into an online sample. When doing so, questionnaires must be rethought and redesigned. In addition, quality controls and related algorithms must be redesigned specifically with the mobile device in mind to avoid biasing the data by overexcluding interviews from mobile participants. If we acknowledge these changes now and adapt surveys to be device-agnostic, we will be able to reach respondents on their terms. If we do not adapt our surveys for mobile devices now, then we risk losing access to more participants in the future, with consequences for our data. 28

29 REFERENCES Buskirk, T.D. and Andrus, C. Smart Surveys for Smart Phones: Exploring Various Approaches for Conducting Online Mobile Surveys via Smartphones. Available at: Cazes, J., L. Townsend, H. Rios and J. Hughes Evolving best practices in mobile surveys and online administration. [Kinesis Survey Technologies Whitepaper.] Available at: Guidry, K.R Response quality and demographic characteristics of respondents using a mobile device on a web-based survey. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Orlando, FL. Available at: 0Respondents.pdf Mumby, H. Are Millenials Paying Attention. Available at: Nielsen. America's New Mobile Majority: a Look at Smartphone Owners in the U.S. Mobile Insights, Q Available at: Rainie, L Tablet and e-book reader ownership nearly double over the holiday gift-giving period. [Pew Internet and American Life Project Report.] Available at: Smith, A Nearly half of American adults are smartphone owners. [Pew Internet and American Life Project Report.] Available at: /Findings.aspx. Wells, T., Bailey, J., Link, M. Filling the Void: Gaining a Better Understanding of Tablet-Based Surveys. Survey Practice, North America, 6, mar Available at: ABOUT SSI SSI is the premier global provider of sampling, data collection and data analytic solutions for consumer and business-to-business survey research, reaching respondents in 86 countries via Internet, telephone, mobile/wireless and mixed-access offerings. SSI staff operates from 25 offices in 18 countries, offering CATI, questionnaire design consultation, programming and hosting, online custom reporting and data processing. SSI s 3,300 employees serve more than 3,000 clients worldwide. Visit SSI at 29