White Paper Best Practices in Online Verification and Quality Control By Jackie Lorch, Vice President, Global Knowledge Management September 2015 Survey Sampling International, 2015
ABOUT THE AUTHOR Jackie Lorch Vice President, Global Knowledge Management Jackie Lorch is vice president of SSI s Global Knowledge Management group and has been with SSI for 25 years. She was a member of the team which developed SSI s first online panel and managed the panel for several years. In her current role, she conducts research on research in support of SSI client research objectives. She is a frequent writer and speaker on research topics, including at AMA, ARF, CASRO, ESOMAR and MRA Annual conferences. Lorch is an ESOMAR Representative for the US and a member of the Board of Directors of the Marketing Research Institute International. 2
PART 1: VERIFICATION BEST PRACTICES The majority of research participants are engaged and honest in their responses. Although bad participants have been shown to usually make no material difference to the survey results, this is not a reason not to be concerned about them, especially since their impact is increased in low-incidence projects. Technology enables increasingly accurate identity verification, which is effective in deterring frauds. Verification procedures in telephone and face-to-face interviewing (both at the interview and data entry stage) are usually done as a check on the truthfulness of the interviewer, not the participant. It is the interviewer who has the means, motive, and opportunity to cheat. In online surveys, of course, it is the participant who has the opportunity to cheat. The question is: Why would they do that? You could imagine that a person would cheat to qualify for a survey, in order to claim a reward. In fact there is a great deal of evidence to suggest that hard-core fraudsters over-qualify for interviews. Research also suggests that a large majority of such people are physically located in places where the small rewards offered by market research are worth a great deal places like India, China and the Philippines. For this reason, many research agencies do not allow participants into surveys with out-of-area IP addresses. One needs to be careful doing this, however, because if the fraudster finds they are blocked they will find another route into the survey possibly proxy-hopping to look like they are in area. SSI s suite of quality control products has IP verification functionality built in, as do many other proprietary systems. SSI prevents the same person doing the same survey more than once, either because they are coming from multiple panels being used on a project or exist multiple times within the same panel. Otherwise good participants might be tempted to cheat to get into a survey because a high reward is offered, especially if the invitation gives the game away by telling you how to qualify! We know, from our own research, that large incentives do not encourage generally better response rates, so we discourage clients from trying to offer them. We also advise against invitation text that identifies the topic or reward for a survey. There is much that panel companies can do to prevent frauds from entering questionnaires. SSI uses the following controls, among others: Recruiting from trusted partnerships with loyalty programs whose memberships are verified at source. Data certification program for all sources Feedback loop with action taken on any poor-quality respondent reported to SSI by a client Two-factor claim authentication required for reward redemption SSI also recommends using the questionnaire itself, where possible, to verify that the person is indeed who they claim to be by asking them something that they should know. This is especially effective in business-to-business settings where knowledge-based questions can be asked. 3
PART 2: BEST PRACTICES FOR IN-SURVEY QUALITY CONTROL QUESTIONS There are two types of data quality issues we seek to identify via in-survey quality control questions. First, there are a very small number of people who begin a survey with the intention of not providing honest answers. A larger group intend to be honest but become fatigued or bored and do not expend enough attention and effort when completing the survey. This group is known as satisficers. SSI employs a suite of tools to identify and remove frauds and educate satisficers on the importance of paying attention and answering carefully and thoughtfully. However, it is impossible to prevent every fraudulent person from entering a survey, or to predict who will satisfice on a particular day. Therefore, the use of in-survey quality control questions is an important quality measure. Effective quality control questions should be reasonably disguised from the respondent. They should not attract attention nor be so obvious that they start to annoy people. A researcher s desire should be to remove all the culprits from the data set, while trying to remove as few of the falsely-accused as possible. To determine the optimal number and type of quality control, SSI has tested 15 different quality control measurements: 12 quality-control questions, a speeding check, a straightlining check and an open-end assessment. These were tested among 2,100 online participants in the U.S. The survey containing the quality controls covered a mixture of topics including entertainment, social issues, and lifestyle and general behavior questions. The survey used 12 offline benchmarks as measurements of quality to compare the data against. The median survey time was 12.5 minutes; short enough not to encourage fatigue. None of the quality-control measurements removed all of the culprits but almost all of the qualitycontrol measurements removed some of the falsely accused. SSI recommends using multiple quality control measurements and removing participants who fail one or more. SSI found the following quality check types most effective in removing poor quality while minimizing false positives: 1. Low-incidence items done in the past week, check travel to remote location 2. Open-end quality check 3. Conflicting answers given in a short grid 4. Speeder check 5. Grid check Check 6 instruction within a short grid Flagging every participant who failed one of these questions, resulted in 195 falsely-accused and captured all 65 culprits. By removing people who fail only two of the five, we flag only 35 falsely accused and capture 60 of the 65 culprits. In summary, researchers should not use badly designed quality-control questions, which hurt feasibility and create a poor user experience. Misdirects or true traps (e.g. false brands very similar to real brands) fall into this category. Quality control measures that throw out a large number of participants do so at random and do not improve data quality. 4
Researchers should also keep in mind that any survey participant can become disengaged in the moment and fail a single quality-control question. Removing these participants does not improve data quality. For this reason, participants flagged for removal should have failed multiple quality-control checks. Quality control questions are useful tools to catch fraudulent and inattentive respondents from spoiling data, however they need to be used correctly. Because anyone s attention can wander throughout the course of a long survey, it is important that a minimum of two quality control questions are used and that participants are only flagged as failing if both are answered incorrectly. If multiple people are satisficing by the end of the survey, it is strongly suggested to evaluate the questionnaire design and length and look for ways to make the survey taking experience within the questionnaire more engaging. And lastly, respondents are human too. Is it possible they selected the wrong option or read the question wrong while taking the survey? Of course it is! ABOUT SSI SSI is the premier global provider of data solutions and technology for consumer and business-tobusiness survey research, reaching respondents in 100+ countries via Internet, telephone, mobile/wireless and mixed-access offerings. SSI staff operates from 30 offices in 21 countries, offering sample, data collection, CATI, questionnaire design consultation, programming and hosting, online custom reporting and data processing. SSI s 3,600 employees serve more than 2,500 clients worldwide. Visit SSI at www.surveysampling.com. 5