Validation Techniques For B2B Panels

Transcription

1 Reaching for the Holy Grail: Validation Techniques for Online B2B Panels Paul Johnson, Director of Analytics, Survey Sampling International Introduction With the advent of the Internet and the explosion of the digital age more and more communication is occurring online. B2B surveys have also moved in this direction for a variety of reasons: online allows for display of visuals, allows respondents to self-complete the survey at their leisure, and avoids costly interviewer time. However, moving to the self-completion mode allows people to claim false credentials in order to qualify. While these fraudsters are a small percentage (1-2%) of a well-managed online panel, they can still cause problems in certain surveys. In particular business to business (B2B) research is susceptible to this problem because B2B studies are low incidence and offer higher rewards. Whenever you have a low incidence study most of the good panelists are screened out because they don t honestly qualify. On the other hand, the few fraudsters are able to guess the qualifications needed in the survey so they are able to qualify for the survey more often. For example, if the population incidence of a B2B target is 4% and of the fraudsters are able to qualify the resulting sample will have (.015*.8)/(.015* *.04) 25% of the survey completes coming from the very few fraudsters. The second problem is that those who are manipulating the survey system specifically look for surveys with higher rewards. Because these target audiences time is at a premium, the incentives for B2B surveys are typically higher than the average consumer survey. These two factors combine to make online B2B research fraught with difficulties. Some companies doing B2B research have moved to using their own customer lists rather than an online panel, but that solution presents its own difficulties. Customer lists are a skewed representation of the whole market and can lead to unrealistic market numbers. Use of these lists can also stifle innovation because they miss those who do not frequently use the company and might have new or different needs. Competitor lists are hard to get as companies are normally unwilling to share their precious contacts with others. Thus, the need for a high quality B2B online panel will not disappear. In fact many market research agencies are searching for this type of product as one might search for the Holy Grail. Methodology Survey Sampling International (SSI) is finding ways to reach this Holy Grail of high quality B2B sample through a variety of validation techniques. The goal is to reduce the fraud percentage to a manageable level such that even in low incidence B2B audiences they cannot become a significant part of the online completes. Ideally all validation techniques would be done before panelists start to take surveys. In this paper we explore four different techniques for validation: Recruiting panelists from existing reward and loyalty programs

2 Data mining the B2B signup profile data Leveraging social media linkage Phone validation We evaluated the effectiveness of the sources by comparing the validation rates of the data mining and phone steps among the different recruitment types. We would expect that the panelists recruited from these existing sources would validate at a higher rate on each of these checks. Those that did not pass the checks in the data mining of the B2B profile were excluded from the panel, so we did not test this effectiveness with a survey, but we do report on the exclusion rates by source. The social media linkage and the phone validation effectiveness were both tested through quality control surveys. We used two different surveys to assess the effectiveness of these validation techniques. The first is a high incidence survey about general business topics which all full-time employees qualify for. We do not expect to see a large amount of fraud here, but we hope to see less in the validated groups than in the groups that did not go through the validation process. The second survey was a low incidence survey specific to computer hardware decision makers. We expect to see a much larger occurrence of fraud in this survey. To make sure that the questionnaire design would not impact results the survey was kept under 10 minutes and optimized for a mobile device. The first survey contained 17 quality control checks and the second survey contained 12 quality control checks of various types detailed in Appendix A. We deemed a person a fraudster if they failed four or more of these checks in both surveys, but we report results for each category and check. During fielding it was necessary to make one important adjustment. The phone validation was required by everyone, but the social media linkage (done through an API with LinkedIn) was voluntary. We did not require panelists to connect through LinkedIn because not all business professionals have a LinkedIn account and we did not want to exclude those respondents who did not. However as it doesn t take long to provide a work number and those in the B2B panel should all have a business number we did require everyone to go through the phone validation. Unfortunately, we only had 600 panelists voluntarily connect their account through LinkedIn. As a result we could not do the low incidence survey test on this group. Data Mining and Sourcing We used a proprietary algorithm to mine the data in the profile. The algorithm had 15 rules that looked for inconsistencies in the 27 profile questions and assigned a severity score to each one. If the rule violated should have very few exceptions (for example if the number of direct or indirect reports were greater than the company size) it was given a large severity score. On the other hand if the rule could conceivably happen in some circumstances (for example if the panelist was over 45 with less than one year of experience in the industry they were working in) they were given a small severity score. Those with a severity score over 100 or that violated multiple rules were not allowed into B2B surveys. This step was important in removing those who were obvious frauds. In the initial test last December it removed 8% of the sample. As the sourcing for the profile expanded and improvements have been made in the

3 algorithm to make it more powerful and precise, it has become even more important, and today removes around 15% of respondents. We were able to see that sourcing for existing communities with their own strict validations immediately increased the quality of respondents as seen in Figure 1. Only 3% of those respondents who came from such sources were removed through the data mining, whereas over 2 of those who were not members of communities already were removed. The phone validation increases in validation and contact rates for those from community sources. It is important to recruit respondents from both types of sources to increase the diversity of the sample. The chart below also clearly demonstrates that the biggest barrier to phone validation is contacting the respondents. When a respondent was contacted they were very likely to validate (at a 3:1 rate for directly recruited and a 7:1 rate for community sources). Figure 1. Validation Rates by Technique and Source % 2 4% 5% % 67% Invalid Unknown Valid % 15% Community Non-Community Community Non-Community Profile Data Mining Phone Confirmation LinkedIn We used LinkedIn data to test the possibility of using social media sites to validate online B2B panelists. LinkedIn has many different fields available. It is challenging to turn those data fields into useful information that can be easily matched to validated against panel profile data. We chose to test seven fields that matched well to the profile and we thought would provide effective validation.

4 Name Date of Birth Phone Number Name Company Connections (10+) Skills (5+) Figure 2 shows the validation rates of those who volunteered to connect through LinkedIn. Unfortunately, most people did not put in a telephone number or a date of birth (including even just year of birth) on their LinkedIn account. Name was not useful because after manual coding for close matches (such as Sandy and Sandra) almost everyone validated. was also not deemed as good of a metric as people often have multiple addresses and they might keep a personal one for taking survey and a business one on the LinkedIn profile or vice versa. Figure 2. Availability and Validation Percentage of LinkedIn information % 20.8% 30.1% 25.1% 36.8% 37.8% 73.1% 16.6% 92.5% 12.3% 96.8% 79.2% 69.9% 58.2% 63.2% 9.6% 49.9% 17.3% 2.5% 5. Not Available Fail Pass Automatic Manual Numeric The remaining three possibilities (having more than five skills, more than ten connections, or matching the company name from the profile) and compare the data to our high incidence survey we see the results shown in Figure 3. From this data you can see only marginal improvements in these variable matching. In particular the percentage that we consider fraudsters (over four quality failures in the survey) rather than just inattentive respondents were all below 1%. The most likely reason for this that the fraudsters in the panel did not bother to connect to a social media account. If we have no fraud to begin with we are unable to measure the effectiveness of LinkedIn for eliminating fraud. While we aren t seeing a great deal of value in using LinkedIn variables, the fact that they did confirm our other validations is an indicator of good quality as seen in the next section when compared to phone validation on the high incidence survey.

5 Figure 3. Number of High Incidence Quality Check Failed by LinkedIn Variables % 1% 1% 8% 5% 3% 3% 7% 5% 3% 4% 9% 1 11% 9% 13% 15% 19% 18% % 38% 38% 35% 45% 39% 38% 37% % 37% 49% 52% 39% 37% 48% 45% Pass Fail No Info Pass Fail No Info Pass Fail Company Name Skills Connections Phone Because we forced everyone to provide a telephone number and attempted each number up to five times we have plenty of sample to do both the high incidence and low incidence check. The validation percentages were shown previously in Figure 1 with the majority of phone numbers still unable to be validated. Figure 4 shows how the phone validation did in comparison to those that connected through LinkedIn. As predicted we are not seeing a large problem with fraudsters in any of the sample because it is a high incidence survey. We do see what we were expecting in that either voluntarily connecting through LinkedIn or validating on the phone seem to be equaly effective in reducing this problem to half of a percentage point. This conclusion becomes even more apparent in Figure 5 with the low incidence test. Those that validated over the phone show half the fraud rate of either the invalid or the unknown categories. Also, importantly even the unknown and invalid groups are showing less fraud than we would expect from a 1-2% base of fraud in the panel. This could be because the data mining of the profile is preventing some of that fraud from entering the survey. However, we don t know that conclusively because we never invited those that did not pass the data mining validation.

6 Figure 4. Number of High Incidence Quality Check Failed by Validation Status % 3.3% 0.4% 4.4% 4.2% 0.5% 3.3% 4.8% % 10.2% 14.1% 12.8% % 42.5% 38.3% 37.4% 38.9% 38.9% 42.7% 44.9% Invalid Unknown Valid All Phone LinkedIn Figure 5. Number of Low Incidence Quality Check Failed by Phone Validation Status % 1 4% 5% 14% 12% 15% % 33% 43% 27% 16% Invalid Unknown Valid

7 Discussion and Conclusions The sourcing of respondents is an important component of recruiting and maintaining a high quality B2B online panel. Leveraging these existing reward mechanisms is important in attracting a wide base of respondents. Furthermore these respondents have already invested time and energy with the prior relationship and thus are less likely to engage in fraudulent behavior. They have also already gone through some validation with the exisiting relationship before they even come to the panel. Thus it is no surprise that they perform much better on both the data mining validation and the phone validation. Data mining the profile is a useful tool to weed out fraudsters, especially with the panelists that did not join the panel directly from a known community. It is a good entry test it allows growth of a sizeable panel by only including those that pass this hurdle. However, it isn t a good confirmation test, because there are still noticable amounts of fraudulent behavior in low incidence surveys among those who pass it. More work could be done to refine the algorithm to do a better job at eliminating fraud while preserving good panelists. Social media linkage can be an indicator of good quality. Fraudsters do not go through the extra effort to voluntarily connect their account to social media sites. Thus if they have done so of their own volition there is a measure of trust associated with that. However, adding incentives to connect to a social media site could attract those fraudsters, so we do not recommend this approach. A more promising option is to allow those who connect through LinkedIn to not provide a phone number as currently the quality of phone validated and social media validated sample is comparable. However more testing needs to be done on this including a higher sample size. We were unable to determine the effectiveness of many of the social media validation points on LinkedIn because we did not have fraudsters attempt to validate through this technique. However we were able to determine that many of the variables we identified early were not viable to use for validation because users do not provide key information such as phone number of year of birth on their LinkedIn profile. More testing needs to be done in this area especially as LinkedIn expands the variables and starts to standardize the formats for easier search and application. It is important to note that a panel company validating in this method would need to compensate the social media site appriopriately for the extra value they are adding by validating panelists. While currently still in testing it could still hold promise because the cost of social media validation is likely much lower than the current alternative of phone validation. Phone validation proved to be effective at significantly reducing fraud. However building a fully phone validated panel would be a long endeavor because the majority of the online respondents are still difficult to reach over the phone. Even with a high response rate over 2 there are significant costs associated with removing all the recruits who cannot be validated. We do not currently recommend this process because a large majority of the invalid and unknown respondents are still not fraudsters and are behaving appriopriately in the surveys. However appending phone validation status to an online B2B data set could be useful

8 as a point of reference if results are questioned. This can help build confidence in the sample and demonstrate that the results come from respondents qualified to answer the survey of interest. Phone validation can also be an effective way of evaluating and comparing sources of recruits. Lastly it is important to note that no validation process is foolproof. Even with phone validation there was a small number of respondents who engaged in fraudulent behavior. The goal is to minimize the number of people who do this so that they cannot significantly impact the results of the study, allowing key decision makers to still take informed actions to help their customers on the basis of trusted research data. While the perfect validation might be as elusive as the Holy Grail, the pursuit of validation techniques can help us improve quality. Even though perfection is never possible, even the pursuit of perfection itself can improve results

9 Appendix A Quality control checks for the high incidence B2B survey are detailed below. Full details of the questionnaire are available upon request. 1) GOBACK We allowed those who said they were unemployed now to go back and change their answer for a potential to qualify for the survey. This would catch those that purposely change their answer in order to try and earn rewards. 2) AGEC We checked for age consistency to see if the number of years that they have been working plus the age they got their first job is greater than the age we had on profile 3) JOBC We checked inconsistency in number of year works to the number of jobs they have had to see if they are employed, but never had a job or if they have had more than 1 job for every year of employment. 4) FSIMP We checked for simple following instructions to select a punch. 5) SATENJC We checked if they were consistent in their satisfaction and enjoyment rating of their current job. 6) COMPK We checked those that said they were knowledgeable about computers to see if they knew their computer operating system. Those that said they did not or that selected a fake operating system were caught. 7) DEPART We checked to see if they worked in different departments that would likely conflict (for example IT and Sales). 8) OWNREP We checked to see if they were an owner with no reports from heads of departments. 9) CLEVEL We looked for consistency in their response to a specific C-level question and their general role question (basically middle management or non-managers stating they are C-level). 10) COMPIND We compared the number of years in the industry to the number of years they were with the company for consistent answers. 11) RECSAT We checked to see if they recommended a job they were not satisfied with or didn t recommend one they were satisfied with. 12) COMPC We checked to see if they thought they would switch companies in 5 years, but be back in 10 years. 13) JOBC We checked to see if they thought they would switch responsibilities in 5 years, but be back in 10 years. 14) DESCC We checked to see if they described themselves in contradictory attributes (such as ambitious and settled). 15) REASONC We check that the reasons they chose for job enjoyment were consistent with the reasons they would not leave their current job. 16) SPEED We checked to see that they took at least 3 minutes (approximately half the median time) on the survey if they were employed. 17) OPEN We looked at the verbatim responses given for open ends or other specifies that didn t make sense.

10 The following are the quality control checks for the low incidence B2B survey. Full details of the questionnaire are available upon request. 1) GOBACK We allowed those who said they were not in charge of purchasing computer hardware to go back and change their answer to qualify for the survey. This would catch those that purposely change their answer in order to try and earn rewards. 2) FSIMP We checked for simple following instructions to select a punch. 3) SIZECON We checked for consistency in the size of their company to what we had on profile 4) DREPORT We checked to see if they were a manager in a company with over 10 people, but had no direct reports. 5) SATENJC We checked if they were consistent in their satisfaction and enjoyment rating of their current job. 6) CORMODEL We compared the make and model of the tablet they owned for consistency. 7) TABCON We checked to see if they thought they would switch companies to only tablets and phones in 5 years, but be back to laptops and desktops 10 years. 8) RAREMAG We identified those that say they read an industry magazine that is long out of print called Ahoy! 9) AGECON We checked that the age provided in the survey matched the age in the profile (within 1 year). 10) GENCON We checked that the gender provided in the survey matched the age in the profile. 11) SPEED We checked to see that they took at least 2.5 minutes (approximately half the median time) on the survey if they received all sections. 12) OPEN We looked at the verbatim responses given for open ends or other specifies in the particular industry category that didn t make sense.