DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS?

Transcription

1 WHITE PAPER DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? OCTOBER 2012 JON PULESTON AND MITCH EGGERS

2 CONTENTS CONTENTS 2 INTRODUCTION 3 THE HEADLINES 4 OVERVIEW OF THE BESPOKE RESEARCH CONDUCTED FOR THIS PAPER 8 FACTORS INFLUENCING DATA QUALITY UNDERSTANDING BASIC CROSS-CULTURAL FACTORS VARIANCE CAUSED BY UNTRUTHFULNESS SPEEDING VARIANCE QUESTION DESIGN VARIANCE SAMPLE VARIANCE 36 LANGUAGE & INTERPRETATION ISSUES 38 APPENDIX: REFERENCES 43 AUTHORS 45 ABOUT US 46 DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 2

3 INTRODUCTION This paper is an examination of the relative impact of survey design, panel quality, and cross-cultural behavior on the accuracy of data when conducting international online survey research. We study all three of these general influences on data quality using seven specific quality improvement techniques in 15 countries. Over the past few years, a great deal of research has been conducted independently to understand the impact on data integrity of panel quality, survey engagement techniques, and cross-cultural response biases. Few studies, however, have examined all three factors together using a comparative crossnational approach. In addition, this research examines other major factors thought to impact data quality, such as imposing age-gender quota cells to drive distributional consistency, setting speed limits that remove those who are inappropriately rushing though surveys, and requiring sample sourcing consistency to remove variance caused by differential unobserved panel characteristics. The latter wellresearched aspects of quality are familiar to most researchers and provide an ideal comparative baseline that allows us to answer our primary research question: Which of these various data quality improvement techniques make the biggest difference and how do they interact with each other to produce double trouble (multiplicative) for some topics and question types? Relatively little is understood about the differing impact of these factors. Therefore, researchers do not have a practical guide to what they should be worried about when conducting research in various countries on various topics using various participants. Is it more important to focus on the quality of the survey design or the quality of the panel? What aspects of panel quality matter? Should researchers worry more about speeders, untruthfulness, or authenticating identity? When balancing on demographics, is age or gender more important? Should researchers focus on question design to make research more engaging, or on the psychology of respondents and their willingness to answer truthfully? How do all these factors vary by country and survey topic? To explore and ground these issues in baseline measurements, we used a comparative empirical approach in which we conducted two large-scale multi-county survey experiments interviewing more than 11,000 respondents in 15 countries. We used a treatment versus control group approach and systematically varied questioning techniques, panel sourcing, and a new lie detection technique developed by Mitch Eggers, Lightspeed GMI s Chief Scientist, and Eli Drake, Lightspeed GMI s Director of Modeling, to measure the authenticity of responses from each respondent (This technique was first described at the IIR/TMRE conference in Orlando Florida, 2011). This piece of research also extends a line of research reported in a paper presented at the 2012 ESOMAR APAC conference entitled, Can Survey Gaming Techniques Cross Continents? It examined the cross-cultural differences in the way respondents answer surveys and explored how well creative survey techniques perform in different countries. The goal was to determine whether various design techniques could help reduce the cultural response style bias that creates problems when conducting multi-country surveys. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 3

4 THE HEADLINES 1. ALL TECHNIQUES MATTER: The main finding from this research exploring the relative impact of different factors is that all data quality improvement techniques are important. The chart below sums up the results from this research: speeding, lying, and panel sourcing all impact data quality at levels approximately equal to demographic balance and question design techniques. While their impact varies greatly question-to-question, country-to-country, and panel-to-panel, they all impact data quality to a similar degree when averaged across the 11 countries we studied in the second wave 1. Figure 1 (above) shows the degree to which results change for all the question types and topics in the countries tested in the second wave of our research when each of these data quality measures is imposed or removed. In other words, it answers the question, How much would my data change if I did not implement this technique to improve my data accuracy? 2. BEWARE OF UNTRUTHFULNESS, PARTICULARLY WITH LOW INCIDENCE QUESTIONS. Beyond basic cross-cultural response bias, untruthfulness presents the largest potential corrupting influence to accurately measuring any form of low incidence personal behavior. In fact, if researchers don t guard against untruthful reporting on low incidence questions (below 20%), the variance is potentially larger than the size of the behavior being measured. For example, if there is a 15% incidence of untruthfulness on a question about recent past purchase that has a true population incidence of 15%, the survey will measure an approximately 30% incidence, and nearly 50% of the completed surveys will be from untruthful respondents. The figure below illustrates this point. 1 Wave one included seven counties: Australia, China, India, Japan, South Korea, Singapore, and the United States. Wave two included eleven countries: China, India, U.S., Brazil, Germany, Mexico, Russia, South Africa, Spain, Sweden, and the United Kingdom. The countries were in both waves: China, India and the United States. We have a total of 15 different countries across the two waves. The figure below shows the inflation of reported brand ownership attributable to liars when the brand had a true incidence of under 20%. Note that lying did not have nearly as large an impact on other topic areas. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 4

5 3. SPEEDING IS ENDEMIC: Current research confirms that nearly all people participating in nearly all surveys speed at some point. Speeding is a behavior researchers cannot deal with simply by removing a small group of people who speed a lot, although that is a good start. Speeding varies by style of question and the location within the survey; it is endemic in every country and on every question. At the end of a survey it holds the potential to become the leading source of data variance (see the figure below). 4. QUESTION DESIGN FACTORS ARE THE SINGLE MOST IMPORTANT MEANS OF IMPROVING THE OVERALL QUALITY OF DATA BUT YES BIAS CAN IMPART DATA VARIANCE: Effective respondent sourcing and imposing tight demographic controls can provide only a baseline of quality improvement in survey responses. Question design factors have the highest overall impact (all questions and all topics) on improving data quality. Reducing reliance on traditional grid questions in favour of more creative approaches is the key recommendation for improving data quality. The figure below shows the difference in predictable pattern answering in two different question design types: one standard html grid and the other more interesting visual rotations for each evaluation. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 5

6 5. AGE BALANCING IS MORE IMPORTANT THAN GENDER BALANCING: Younger age groups show stronger tendency to speed and lie, which are two primary corrupting influences of survey data. Without these two influences the impact of age compared to gender depends on the topic being surveyed. In general, because of the interaction with other data degrading behaviors, age balancing is more important than gender balancing when defining quotas and looking for the most consistent cross-country feedback. 6. PANEL SOURCING VARIANCE IS ON PAR WITH DEMOGRAPHIC VARIANCE: The data we compared in this research was from various industry suppliers of proprietary panels and river sample. We imposed quotas on the panel sources but left the river sources without quotas. Rarely did any variance between these sources fall outside the boundaries of the variance caused by basic demographic differences. However, we have not tested a comprehensive range of river sources so this observation must be considered anecdotal. We believe it will vary depending the quality and natural balance of both the panel and the river sources being compared. 7. CROSS CULTURAL VARIANCE IS A LARGE SOURCE OF DATA DIFFERENCES OVERALL: While average cross-country variance is 7.1%, variance from individual countries on a question-toquestion level can be significantly higher, As the chart below demonstrates, variances of up to 15% are not uncommon. Therefore, understanding the underlying factors that dictate basic cultural responses in different countries is critical if researchers are to compare cross-national data effectively. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 6

7 This paper is structured to explore each of these main sources of data variance in more detail. It aims to give researchers a thorough understanding of the factors that affect the authenticity of data in crosscountry studies. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 7

8 OVERVIEW OF THE BESPOKE RESEARCH CONDUCTED FOR THIS PAPER The analysis for this paper was collected in two large-scale multi-country studies. The first fielded in seven countries to 3,900 respondents using the double opt-in panels from mid-december 2011 to early January The second fielded to 7,450 respondents in 11 countries using multiple double opt-in panels and multiple river sources and was conducted between May and June Both surveys shared a common set of benchmarking questions. Countries Wave 1 Wave 2 Australia 613 Brazil 533 China Germany 854 India Japan 695 Korea 217 Mexico 484 Russia 531 Singapore 552 South Africa 499 Spain 491 Sweden 297 UK 830 US Total Across the two survey waves we included 21 mini-experiments, (12 shared across both waves) during which respondents were randomly allocated to either (a) a set of 3-4 standard market research survey questions, or (b) a question set measuring the same constructs on the same scales but using creative questioning techniques such as: Role-play scenarios that turn questions into quests Gambling and competitive point scoring techniques Graphics (in addition to words) in questions, scales and response options More visually-oriented response formats, e.g., drag and drops, slider scales, caterpillar segment scales. These techniques were pioneered and studied in previous research with Western audiences (see Puleston papers in Box 1). The dependent variables in the experiments included six key metrics, compared among the seven countries: DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 8

9 Overall response distribution Individual-level response variance (a measure of straight-lining) Average time taken to consider and complete the question Lie detection test Self-reported enjoyment of completing each survey question 2 The first four performance metrics are known for all respondents. The latter are known only for one-third of the sample who took part in the first wave of research to benchmark each question. Pilot testing found that constant checking of how much each respondent liked each question reduced respondent engagement in the overall exercise, so it was significantly reduced in the second wave. 2 Recorded on a 10-point scale and analysed using a summary measure calculated in a similar fashion to the net promoter score : the proportion that gave a score of 9 or 10 for interest, minus the proportion that gave it a 1, 2, 3 or 4. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 9

10 Table 1: Experiments included: Experiment Topic Constants Lie detection Morality Radio buttons vs Flag and drag Gaming scenario Words vs visuals Words vs visuals Combination of pictorial and scenario Combination of pictorial and scenario Various low incidence activities Morality indicators Positivity Mobile phone brands Shopping emotions Preferred TV genres Preferred activities Olympic sports broadcasting Choice of media channels Yes/no ratings L-R scale Semantic points Concurrent L-R scale Semantic tiles Auto-next after selection Pick any, mix of +/- emotions 5pt smooth slider scale, starting on midpoint 5pt L-R semantic scale (dislike to like). Sequential judgements Open ender Same size box Pick any Experimental conditions Format A Format B Single binary choice questions to all Radio button 7pt scale (grid) Rating questions to all Three single options re expectations of change (decline, stay the same, grow) List of words Ends and points unlabeled Tiles containing written labels Format A Simple question Words only Scenario question text only Pictorial with words Flag and drag with 100pt variation but 7pt scale markers Gaming scenario: Bet $10/$20/$30 for each option (3x3=9 options), then told market reaction Words on colored stars, drag and drop Ends and points unlabeled, but man in chair moves from being asleep to giving a standing ovation Faces above written labels Format B Format C Scenario question with visuals DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 10

11 FACTORS INFLUENCING DATA QUALITY These dimensions of quality exhibit different impacts and can interact with each other to produce very large effects on results. For example country, panel source, age and gender may interact so that a particular source in a specific country may deliver males under age 25 years who are far more problematic than the same source in the same country for all other age and genders. Similarly, speeding and lying may prove far more problematic for certain question types in certain countries. In the next section we will examine each of the key causes of data variance in more detail. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 11

12 1. UNDERSTANDING BASIC CROSS- CULTURAL FACTORS The original impetus for this research was to gain a better understanding of basic cross-cultural factors that influence online research results, a topic covered in legendary detail by Geert Hofstede, who explored the fundamental personality differences among people in different countries. He identified four dimensions by which researchers could differentiate cultures, and which may affect how national populations on average respond to surveys. Hofestede s four main personality dimensions are: 1. Social inequality and relationships with authority 2. The relationship between the individual and the group (sensitivity to group decision making) 3. Concepts of masculinity and femininity and the implications of being born male or female 4. Ways of dealing with uncertainty and ambiguity, which is related to the control of aggression and expression of emotion. Without attempting to paraphrase whole books written on this topic, the key finding relevant to this research is that cultural differences in personality traits result in cultural differences in how participants answer survey questions. We are able to observe these differences by aggregating answers from large numbers of questions from many different surveys across many different countries, a task we undertook as part of the research we conducted and reported in earlier papers. The key cultural and personality differences we are able to observe in cross-country online survey research are the following: 1.1 PROPENSITY TO SAY YES The propensity to answer yes to a simple yes-no question varies by country. The chart below summarizes the main regional differences, which are based on aggregated data from 60 yes-no questions asked in 30 countries. There is a general trend for western and more developed markets in Asia to be less likely to answer yes to a question. Yes Index Japan/Korea 85% North America 88% Northern Europe & UK 90% Australia 93% Southern Europe 101% Eastern Europe 101% South America 113% China 113% 111% Asia (excl. Japan/Korea/China) Africa 115% India 132% DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 12

13 1.2 PROPENSITY TO LIKE SOMETHING Participants propensity to report liking something varies dramatically across the world. Below is an outline of the main regional differences based on the aggregated self-reported liking scores from 90 different questions. The Japanese and Northern Europeans rarely stick their necks out and say they like something. They are more comfortable saying something is average or OK. Southern Europeans, Latin Americans and Asians more readily express their liking for a wide range of things. Whist the Japanese and Koreans, Northern Europeans and British all say Yes to a similar degree, Liking draws out more measurable differences. Like Japan 7% Northern Europe 10% Korea 12% UK 13% China 13% Australia 14% North America 15% Eastern Europe 16% Asia (excl. Japan/Korea/China) 16% Southern Europe 18% Africa 20% South America 22% India 23% 1.3 WILLINGNESS TO AGREE WITH A STATEMENT We have aggregated the agreement scores from 580 questions asked in various surveys across 30 countries. Agreement patterns are different from liking and propensity for saying yes. For example, the Chinese are very likely to say they agree with something but relatively less likely to say they like something. Once again, Japan and Northern Europeans are the least likely to agree with anything. Agree Japan 38% Northern Europe 44% UK 47% Australia 48% Southern Europe 53% South America 58% Eastern Europe 50% USA 49% Asia (exec Japan/China) 52% China 63% India 66% DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 13

14 1.4 WILLINGNESS TO DISAGREE When it comes to expressing disagreement, a division arises amongst Northern Europeans that does not exist with positive scoring indicators. The Dutch are much more willing to disagree than others and measurably out score other Northern European countries. At the other end of the scale are the Chinese who are extremely reluctant to disagree with anything. Disagree China 14% India 15% Asia (exec Japan/China) 16% South America 26% Eastern Europe 28% Southern Europe 28% UK 30% Germany 31% USA 31% Other Nth Europeans 33% Japan 36% Netherlands 38% Cultural differences result in massive skews in the relative scoring attained across different countries when using likert range scales. Below is a summary chart illustrating the dramatic differences in net positive rating of some representative countries. At one end we have the Japanese who very rarely present a positive opinion and at the other end of the scale the Indians who very rarely don t! DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 14

15 1.5 SITTING ON THE FENCE ON ISSUES Across many countries in Asia, with the exception of India and China, there is a strong reticence to express opinions, which results in a tendency to give neutral scores. Closely behind Asians (e.g., China) are the Northern Europeans who also have a high neutral score. Neutral scoring South America 22% India 22% China 23% Africa 25% Southern Europeans 27% USA 28% Eastern Europe 29% Australia 30% Northern Europeans 31% Asia (ex China) 35% USING THIS INFORMATION TO WEIGHT RESPONSES With knowledge of these differences in the basic character of responses across countries it is possible to re-weight certain types of questions to deliver a more comparable cross-country data. The example below summarizes individuals rating of their happiness from a range of different countries. Self-rated happiness was one of the test questions asked in these experiments. Mexicans appear to be the happiest and Swedes and Germans the least happy. When the data is weighted to account for question agreement bias, however, the data show personal happiness ratings similar in most countries with the exception of China, India and Brazil. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 15

16 This is an illustrative point, but in most questions and topics there are fundamental underlying differences in reported opinions, and the appropriate weighting procedure is not clear. It does demonstrate the importance of exercising caution when making any sort of cross-cultural comparisons without adjusting for cultural differences in propensity to agree, like, say yes, or express dislike. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 16

17 2. VARIANCE CAUSED BY UNTRUTHFULNESS Truthfulness varies by country and it too exhibits significant cross-national differences. We show in this section the propensity to not give fully truthful responses when taking online surveys correlates with cultural norms associated with corruption. In cultures with high levels of corruption, as measured by the World Bank s corruption index 3, higher percentages of online respondents fail our lie detector. We define an untruthful respondent as one who answers yes to a high number of improbable questions, which we calibrate using known sample and population benchmarks. Ideally, the researcher places the lie detection question in the typical early screening position in the questionnaire. Our untested hypothesis is that placing these questions where the screener questions are typically found makes it clear to those who want to lie to defeat the screener questions, this is the place to commit the infraction. We continue to experiment with various low incidence questions because our working hypothesis is that uncorrelated low-incidence questions most clearly identify untruthful answer behavior. The lie detection questions we used in this research were the six improbable true-false questions listed below: Do you have three or more dogs as domestic pets? Have you been skydiving in the past three months? Do you have a subscription to the opera? Can you juggle four or more oranges simultaneously? Have you been to Las Vegas, USA in the past six months? Have you have purchased a new washing machine or tumble dryer in the past three months? There are a wide range of low incidence questions that are appropriate within any particular culture. Ideally the library of improbable lie detection questions will be similar to screener questions and be plentiful enough to rotate so participants are not exposed to the same sequence repeatedly. In addition to these six questions, we include a calibration question in the lie detector that has two special properties. First, the calibration question should be randomly distributed within the population and unrelated to any of the low incidence lie-detection questions. Second, the calibration question should have a known incidence in the population. Examples of questions that meet these two requirements are: Are you left-handed? Where you born in the month of March? Were you born in the year of the rabbit? Researchers should keep in mind the difficulty of identifying questions that meet these two criteria. Selfreported left-handed percentage varies by country and may vary by each country s tolerance of lefthanded preference in elementary schools. Similarly, it is likely that older cohorts experienced less tolerance than younger cohorts and more progressive schools may have allowed left-handed preference prior to less progressive schools. All these factors may render left-handed preference less random than researchers might think. Similarly, eye color is not random; those of northern and central European ancestry have a higher percentage of blue eyes which may result in non-random distribution between core populations and recent immigrant populations. Month of birth exhibits seasonality but is random with respect to nearly all other variables, which makes it one of our favorite calibration questions. In addition, the percentage of the population born in a particular month is well documented in most human populations. 3 Daniel Kaufmann, (September 2004) Global Competitiveness Report 2004/ ~theSitePK: ,00.html Download Corporate Corruption/Ethics Indicies. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 17

18 Using these techniques, we measured the propensity to answer untruthfully among various respondent sources spanning the 15 countries under study. Answering a high number of improbable questions correlates with over reporting of two randomly distributed characteristics left handedness and month of birth. Among seven countries for which we collected data, the correlation between untruthfulness and over reporting left-handed preference is.66. Among 14 online sources spread across nine countries the correlation between untruthfulness and over reporting born in March is.16. It is important to note that no correlation is expected to exist between the set of low incidence questions and either left-handed preference or birth month, but a relationship is created as differential proportions of respondents being untruthful when answering the low incidence questions and also being untruthful when asked if they are left handed or if they were born in March. The liars create the correlation, and the propensity to be untruthful when taking online surveys varies greatly by country. 30% 25% 20% 15% 10% 5% 0% Relative levels of untruthfulness Australia China India Japan Korea s'apore USA (panel) USA (river) Brazil GMI Mexico Russia South Africa GMI Sweden UK (panel) UK (River) Untruthfulness among participants in online research is heavily influenced by culture. Below is the scatter plot showing the relationship between the World Bank s corruption index and the propensity of participants to be untruthful when participating our online surveys. The slope of the regression line is.15 and the explained variance is 39%, clear empirical evidence the propensity to lie varies by country and is directly related to culture. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 18

19 World Bank Corruption Index Countries and Sources Countries and Sources Linear (Countries and Sources) 0% 10% 20% 30% Percent who Lie We are exploring other factors that also influence untruthfulness like economic incentives to lie given anticipated awards for qualification and completion, respondents views of market research, and whether participants care if their individual voice is heard. Research to date indicates lying varies between 3% and 30% from country-to-country and is a major risk factor to accurate data. HOW MANY PEOPLE LIE? Across the 11 countries sampled in the second wave of this research, here is the breakdown of the proportion of people who answered yes to the low incidence questions in our lie detector test. We estimated that there is only an 18% chance that they would say yes to at least one of the low incidence questions but at the screening stage 37% said yes to at least one, 27% to more than one (chances less than 1 in 10) and 8% to more than 3 (chances less than 1 in 1000). We would not wish to call a third of the sample who said yes to one or more low incidence question liars, but for the purposes of this analysis and other analyses indicates answering yes to 3 of 6 captures a large percentage of all untruthful respondents. For comparative purposes, however, we examine data variance among those who answered yes to one or more screening questions compared to those who DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 19

20 said no to all screening questions, a group we classify as totally truthful. The figure below shows there is a younger male bias to those that said yes to one or more low incidence questions. WHAT TYPE OF QUESTIONS ARE AFFECTED BY LEVELS OF UNTRUTHFULNESS? It impacts some questions much more than others. It is most prominent when asking about a high status activity like owning ipads, visiting Harrods or reading Vogue magazine. In our experiment we measured upwards of 100% overclaim amongst the untruthful for these types of questions compared to the totally truthful group. It has less dramatic, but nevertheless significant general influence on any question surrounding personal behavior (e.g., have you washed your hair today), but is a fairly benign issue when recording attitudes toward issues. The question with the lowest level of data variance as a result of untruthfulness, interestingly, was for the gambling question, where respondents had to bet some imaginary money on their choices. HOW TO DEAL WITH UNTRUTHFUL RESPONDENTS? DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 20

21 It would be easy to recommend simply screening all but the totally truthful, but the problem is a significant proportion of the population answer yes to one or more low incidence questions, so to remove all of them would miss balance the overall data. Our suggestion is to think about screening out respondents who answer 3 or more low incidence questions and flag any respondent in the data who answers more than one and take note of any differences in how this group answers any sensitive questions that may be effected by untruthful respondents. Working from the other side of the data and removing egregious overreporters, researchers should remove respondents who answered yes to a high proportion of low incidence questions (e.g., remove those who answered yes to six out of six, then five out of six, then four out of six, etc.) until the percent who indicated they were born in March drops below 10%. In many cases the data will improve greatly. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 21

22 3. SPEEDING VARIANCE Speeding issues are probably the biggest general problem we face when conducting online research. In these research experiments we have examined speeded effects in quite some detail. The first point to keep in mind is that speeding is a generic, not specific issue. We examined each question and measured the number of people who sped through at least one question and the figure was 85%* (i.e., less than one in 5 people actually complete a survey without speeding at all at any stage) 4. For the purpose of this analysis and to understand the impact and errors caused by speeding, we divided the sample from each country into two groups, the people who answered the survey and were above and below the median times. We then compared their answers to a wide cross section of 10 different question types across the survey and measured the variance. The general error speeding introduced was +/-6.2%, which is roughly on par with the differences you may see in answers between gender or two broad age groups. As you might expect, speeding variance did increase as respondents progressed through the survey. It was averaging 4% on the first few questions and rose to around 8% at the end, which ranked this factor as the highest source of overall variance. The second observation is that there was a measurable difference between the variance of personal compared to impersonal style questions. 4 We defined a speeder as someone who answered a question in under half of the median question answer time. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 22

23 DEMOGRAPHIC VARIATION From a demographic perspective, it is the younger age groups and men who tend to answer survey questions faster. NATIONAL DIFFERENCES IN SPEEDING On the surface, if you examine the average completion time of surveys, you can usually observe significant differences between countries which could lead you to the conclusion that some nationalities speed through surveys a lot more than others. However, to compare effectively you need to take account of relative reading and comprehension times in different countries, as these are the primary source of reading time variance. When this is taken into account, thinking times become quite remarkably similar country to country. Below is a chart that really exemplifies the issue. It was a repetitive set of questions and on the first question it had the instructions and then the following pages a repetitive set of identical questions. You can see that there were quite dramatic differences in reading and comprehension time on the first question but almost identical time spent answering the follow-on questions. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 23

24 To examine this further, we looked at the average reading time of a series of text-only frames in our survey and average answer times of follow-on questions where there was little or no reading/comprehension needed. We were then able to create an index weighting for each country. As you can see from the chart below, Japanese, for example, is read at an index of 60% compared to the average, yet taking this into account their thinking time is on par with most other countries. AVERAGE READING / COMPREHENSION SPEEDS PER COUNTRY: Words per minute (English cross translated*) Japan 7.5 Korea 5.8 China 5.7 UK 5.6 Sweden 5.5 Australia 5.2 USA 5.0 Spain 4.9 Germany 4.9 India 4.6 Russia 3.9 South Africa 3.7 Brazil 3.6 Mexico 3.5 Average 4.5 *Please note this is not native language reading speeds but English cross-translated reading speeds (i.e., if you translate 100 English words into Japanese how long will it take to read these). AVERAGE QUESTION THINKING TIMES DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 24

25 From this analysis we were able to calculate thinking time for specific questions and we were then able to compare the relative thinking time of different types of questions. AVERAGE THINKING TIMES (ALL COUNTRIES) Single attitude statement Repeated single attitude statement Attitude statement in grid Binary yes no list choice Multi choice question (per option) Single choice question (per option) 7 seconds 5 seconds 2.1 seconds 1.1 seconds 0.81 seconds 0.41 seconds What you can see from this is the rapid decay in thinking time given to questions that are presented in repetition. We will think for 7 seconds for the first instance, 5 for the second and then this drops to an average of 2 seconds for all subsequent questions. For simple yes-no questions like are you aware of this brand thinking time is about 1 second and if these choices are presented in a multi-choice list it drops to under 1 second. This difference in thinking time as a result is the root cause of speed-related data variance. Comparing the answers from the slowest and fastest half of the sample you can see there are really quite significant differences in answers for binary yes-no and multi-choice questions. THE IMPACT OF SPEEDING ON LIKERT SCALE QUESTIONS The impact of speeding on likert-scale questions, while it is a lot lower on the surface compared to multichoice questions, is often hidden in the data because of the way we answer questions. When we speed, we tend to answer questions more randomly, or in a pattern, the effect of which is simply to water down the data rather than distort it. It would be a mistake, however, to think there is inherently more thought put into these types of questions compared to multi-choice style questions. There appears to be an inherent pattern in the way we speed. It is often clustered around the slightly positive. For example, on a likert scale this means the speeding error biases results toward the positive and is more pronounced in questions where there is natural disagreement. In conducting detailed question-specific analysis and selecting the questions where there is an underlying negative opinion, the data exhibit the corrupting influence of speeding. See the chart below which is an anecdotal example but DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 25

26 taken from a real survey of over 2,000 respondents. The speeding effects of the disengaged respondents can be seen in the second hump that appears in the answer pattern of the fastest half of the sample. Furthermore, the natural answer style differs across countries, which can introduce a general error between countries and complicate cross-national data comparisons. The chart below, which reports the typical answer patterns of Japan compared to India, illustrates this point. AVERAGE ANSWER TIME VARIANCE BY QUESTION CONTENT We noted earlier there were differences in speeding data variance based on the content of the question, and the anomaly is more apparent when you compare the thinking time given to these different questions. Example 1: We asked respondents 2 questions of identical nature in our survey. One was about the respondent and one was about a brand. Specifically, we asked, Thinking about yourself, how much do you feel the following words describe you? and similarly we asked, How much do the following words describe this brand? Respondents invested 70% more time answering the question about themselves. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 26

27 Example 2: We asked 2 sets of likert-scale questions almost identical in nature, but the first phrased in the personal like this, How much like you are these types of people? In the second question, we asked in the impersonal like this, How much do you agree or disagree with these statements? The phrasing lesson is clear for all types of questions: the more you can contextualize a question in the mind-set of the respondent the more effort respondents will put into thinking about their answers. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 27

28 4. QUESTION DESIGN VARIANCE Question design effects lead to a broader look at the overall impact that question design has on data quality. Changing the context of a question to be framed in the personal is an example of how question design can potentially improve the quality of data but there are myriad question designs that impact the data. Having covered this issue in detail in several earlier papers, we wish to focus here on the relative effects of different techniques on answers. Question design variance can be broken down into 5 broad factors: 1. Different balance of data as a result of range labelling issues and the way questions are interpreted 2. Differences attributable to the use of imagery and iconography 3. Differences attributable to the dynamic format presentation, e.g., dragging and dropping versus button selection 4. Differences attributable to how respondents are motivated to answer a question 5. Linked to item 4, differences attributable to the way questions are worded What type of data variance do these different factors introduce, and can it be controlled or does it introduce random or more complex data variation that is not easily understood? RANGE LABELLING AND QUESTION INTERPRETATION As Bernie Malinoff has observed, you only have to blow on a likert scale and the data will change. It is important to realise that what may look like quite innocuous changes to the way you label a question can have a big knock-on effect on the balance of the data gathered. To exemplify this issue, below is a chart taken from a paper published by Harris Interactive, written by Randall K. Thomas, John Bremer, George Terhanian, and Renee M. Smith exploring the impact of scaling methods on the balance of data from likert questions. Essentially there are 2 ways you can ask a likert-scale question: 1. Bi-polar = strong disagree strongly agree 2. Unipolar = completely disagree completely agree And 2 ways of labelling them: 3. End anchored bi-polar = strongly disagree strongly agree 4. Fully anchored bi-polar = strongly disagree, slightly disagree, neither agree or disagree, slightly agree, strongly agree Different labelling protocols deliver measurably different scores, with bi-polar peaking at 4/5 and a unipolar peaking at 3/5. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 28

29 You can see that fully anchored scales deliver data that is more evenly spread across the range, which is usually the objective of likert-scale questions. It is therefore recommended that researchers make sure to use anchor labels where possible. Similar types of errors occur based on the positioning of the anchor point on slider questions. Comparing the data from sliders positioned at the 0% delivers roughly 10% lower scores than the data from those with a start position of 50%. In general, this type of variance can be controlled for, so as long as you are consistent with question design approaches and labelling protocols, the effects are not expected to interfere with your data. IMAGERY BASED VARIANCE Using images to support choice selection can help respondents make decisions and enable researchers to communicate concepts more effectively than words, but this clarity comes with a trade off. At a basic level, using images instead of words for brand recognition will change the recall scores of the brands. In the current experiment, it introduces a data variance of 34%, which, compared to other forms of variance is very high. For example, data variance attributable to speeding was 35%. On one level, the imagery helped respondents pay more attention to their answers. On another level it distorted the recall levels due to the differences in the way we mentally process words and logos. In this increasingly digital-visual world, many names by themselves do not resonate, and it is only the attachment of the logo that triggers brand associations. The variance increases significantly when you switch from brand names to pictures of the products. Increase in recall using brand imagery DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 29

30 In this example, you can see that for some brands the product picture recall was up to 60% higher than the word recall for some of the less well-known brands. There is, however, another factor at play. People tend to interpret imagery literally, and as the data display, comparing the scores for words versus logos when measuring awareness of Muller Vitality and Ski Yogert very different indexing and memory are apparent. Both Muller and Ski are very familiar brands, and with just the words displayed, respondents focus on the name Muller. But when we present a picture of the specific product, people start focusing on Vitality in the case of Muller, and on strawberry flavor in the case of Ski, and the recall scores actually drop off. This literal interpretation effect can be even more dangerous when asking people to make word associations using representative imagery. The choice of imagery can have a dominating and leading effect. To test this, we conducted an experiment where we mixed male versus only female imagery to support a range of emotional word choices about attitudes to football and shopping. We found the choice of imagery had a dominating effect on the emotions selected. Below is an example of the imagery. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 30

31 We also compared liking scores for a range of different music artists using different images, and compared the rating of the artist when we showed respondents the lowest and highest rated pictures. We measured some effects of over 100% differences in the resulting liking scores. The bottom line is that using imagery in surveys without calibration can introduce data variance effects that are overwhelming. Image selection has to be very carefully implemented. Our recommendation when using imagery to support attitude statements is to work with baskets of images like the example below and avoid any form of imagery if you are asking people to make emotional associations. ICONOGRAPHY We also looked at the impact of using iconography to support a range-based selection process. Here the steering impact is relatively benign but had a measurably positive impact on improving attention levels and thus reducing speeding data variance and improved data granularity. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 31

32 Example use of iconography in a question: We measured an average of less than 2% data variance but up to a 20% reduction in speeding and speeding data variance. Also, there was a lot more cross-country consistency in responses among countries. This issue was explored in some detail in our previous paper, an example of which can be seen below. It shows a lot less pattern answering using more visually based questioning techniques. QUESTION FORMAT We have also explored the impact of asking questions in generally more creative ways, such as dragging and dropping, flag sorting, star rating, gambling, list building and various forms of dynamic animated grids. Examples of alternative creative approaches: DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 32

33 Only two formats in this experiment delivered measurable data variance. The first was the gambling methodology, where respondents were asked to bet imaginary money on various choices*. Gambling encouraged respondents to be slightly more circumspect and minimally shift the balance of data. The second format that caused a significant difference was the list building format. This method requires further investigation, but the explanation seems to be that respondents have a tendency to sort things into even groups. *Details of this technique are outlined in the game experiments paper Generally speaking, we found that in every case, these more creative questioning methods reduced straightlining and increased data granularity. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 33

34 QUESTION WORDING How questions are worded can significantly impact how a question is interpreted. There is controversy around the use of certain types of gamification motivation techniques and what impact these techniques might have on the character of data. Often this impact is hard to control. Below is an example from our paper, The Game Experiments, where one group was asked to name places they liked to go on holiday and the second group was asked to imagine they were the editor of a travel magazine drawing up a list of places they like to go on holiday: In this example, the volume of feedback from the virtual magazine editors doubled, but the answers were significantly altered by the technique used to ask the question. Here is another example taken from this experiment, where one group of people was asked to, make a list of your favorite shops and the second group to, imagine you could design your perfect shopping centre only with the shops you wanted. Can you draw up a list of the shops you would have in your shopping centre DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 34

35 This revised wording technique increased the volume of feedback fourfold, but in this case had a less of an impact on the character of the answers. There were just a lot more shops listed. Here is a third example, where we asked people to undertake a media planning task. One group was asked in a traditional way what media they would choose. The second group was asked to imagine they had $1m of their own money to spend to reach people like them. What media they would choose? As you can see, their choice of media changed measurably and the appeal of less top-of-mind media came to the fore like outdoor advertising and cinema advertising. It begs the question of which data are more reliable? It may or may not help to note that the second group spent 50% more time thinking about their answers. Finally, here is a very famous example of how wording of a question can affect answers. It was taken from what could be described as the seminal text on this topic, written in 1951 by Stanley Payne, The Art of Asking Questions. He cites a case from a WW2 US population census: When asking how many bedrooms people had, there was a tendency to under report. They added the phrase to the question including rooms you don t sleep in and as a result, 500,000 more rooms we reported. What all these examples serve to illustrate is that how you word a question is a creative skill of the researcher. But it is also a choice and clearly, a critical part of the whole process of survey design. It is difficult to design any rules that can instruct you how to do it effectively. The most important thing is to be conscious of the impact changes like these can have and ensure if it is a sensitive issue that you pilot test the different wording or different question styles. SUMMARY OF THE IMPACT OF QUESTION DESIGN FACTORS Design of a question is really the only weapon you have to tackle speeding effects. An adjusted and improved approach can be clearly demonstrated to reduce speeding and increase the quality of answers, but will also often deliver a measurable difference on answers. Ultimately, being informed of these differences is the researcher s responsibility and choice about which approach to use. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 35

36 5. SAMPLE VARIANCE The final issue we explored in this research was the impact of sampling sources on the data. Recognize that as a sample provider, we cannot be seen as objective on this issue. We appreciate the data from this aspect of the paper will be treated as well intentioned anecdotal evidence. To evaluate data sources, we compared the answer to this survey from the two unique and independently sourced panels in UK, US, Germany and Spain, along with some raw river sample (no quotas) in the US and UK. We observed somewhat different levels of panel variance country to country and marginally higher variance when comparing panel to panel versus panel to river. The level of variance was roughly the same as was observed between different age groups or gender and roughly half the variance observed country to country. Panel variance by question type We examined the sourcing variance by question format and this revealed stronger differences in data when recording personal behavior compared to measuring attitudes or making predictions. This is in line with what might be expected, as personal behavior questions are far more sensitive to demographic variation than attitudinal questions. DIMENSIONS OF ONLINE SURVEY DATA QUALITY: WHAT REALLY MATTERS? 36