Looking Beyond Quality Differences: How Do Consumer Buying Patterns Differ by Sample Source?

Looking Beyond Quality Differences: How Do Consumer Buying Patterns Differ by Source? The CASRO Panel Conference February 24 th & 25 th, 2010 New Orleans, LA

Background Since their proliferation began in the 1990s, opt-in, online panels have been lab- and road-tested to verify their suitability for use in commercial research. Differences in results from online versus phone data collection Examples of non-coverage error with online samples Survey-taking patterns that evidence mindless responding More recently, debate has turned to the replicability of responses patterns across opt-in panels. Gittleman (2009) acknowledged the non-probablistic nature of data of opt-in panels and supported a focus on consistency of results as a quality measure. Examination of the quality and consistency of data from opt-in panels has also recently focused on demographic variables and other variables that have externally verifiable measures. Yeager, Krosnick, Chang, Javitz, Levendusky, Simpser, and Wang (2009) concluded that non-probability, online sample surveys are always less accurate than probability surveys such as RDD phone surveys. 2

Background Less published work has examined the extent of consumer-behavior related disparities in data collected using different online sample sources. Therefore, the current research explored how reported consumer behavior and related variables differ by online sample source. 3

Key Research Question Do different online sample sources evidence materially different patterns of consumer behavior? 4

Research Method This exploratory research examined data from the following samples: Two opt-in panel samples Two river samples Two social networking samples Analysis will determine whether material differences exist in Product category purchasing Brand awareness Brand usage Demographic variables Cheating and related behaviors 5

as a Source Ads are sold through an auction process. offers a recommended bid range, and it changes daily. Higher bidders enjoy more impressions or exposures. reports that on average,.04% to.08% of people exposed to an ad on click on it. users can be targeted by age, gender, geography or birthday. Advertisers can control costs by setting a maximum daily budget. On average, Burke paid $1.21 per click. Ads must be approved by, and approval typically takes between six and 24 hours. 7

Survey Landing Page 8

Click Rates by Age/Gender Groups Age/Gender Group Ad Click Rate 65+ females 0.043% 65+ males 0.037% 35-44 females 0.034% 45-64 females 0.033% 45-64 males 0.031% 18-34 males 0.029% 35-44 males 0.028% 18-34 females 0.015% Average 0.031% 9

Demographic Profiles and Panelist Behaviors 10

Demographics 100% Female 100% Employed Full Time 90% 90% 80% 80% 70% 70% 60% 60% 50% 50% 40% 30% 20% 10% 53% 53% 54% 56% 68% 45% 40% 30% 20% 10% 44% 35% 51% 28% 49% 34% 0% Panel A Panel B River A River B Social 0% Panel A Panel B River A River B Social 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Married 52% 52% 54% 52% 39% Panel A Panel B River A River B Social 52% 11

Demographics 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% College Graduates 29% 24% 23% 20% 29% Panel A Panel B River A River B Social 11% 50 45 Age $50,000 Household Income 40 35 $40,000 30 25 20 15 45 43 45 46 30 50 $30,000 $20,000 $47,630 $49,877 $51,938 $46,293 $49,738 $39,688 10 5 $10,000 0 Panel A Panel B River A River B Social $0 Panel A Panel B River A River B Social 12

Panelist Behaviors 35 30 25 Hours Online Weekly 10 9 8 7 Surveys Taken in Past Month 20 15 10 5 24 25 24 25 28 32 6 5 4 3 2 1 6 8 3 8 4 2 0 Panel A Panel B River A River B Social 0 Panel A Panel B River A River B Social 13

Fraudulent Responding and Mental Cheating 14

Incidence of Fraudulents 35% 30% 25% 20% 15% 10% 5% 0% 2% 6% 4% 4% 11% Panel A Panel B River A River B Social 4% 15

Unusually Fast Survey Completion 35% Average Survey Length: 27 minutes 30% 25% 20% 15% 10% 5% 0% 13% 22% 7% 13% 17% Panel A Panel B River A River B Social 3% Speeders were defined in this research as respondents completing the survey in fewer than ten minutes. 16

Incidence of Mental Cheaters 35% 30% 25% 20% 15% 10% 5% 0% 12% 11% 6% 6% 14% Panel A Panel B River A River B Social 7% 17

Measures Related to Consumer Behavior 18

Category Purchasers Beer 100% 90% 80% 70% 60% 50% 40% 30% 20% 61% 58% 61% 64% 73% 55% 10% 0% Panel A Panel B River A River B Social 19

Purchase Frequency Beer Panel A Higher Volume Lower Panel B River A River B Social 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Once a week or more often 2-3 times per month Once a month Every two months Every 3-4 months Less often than every 3-4 months 20

Brand Awareness Miller Beer 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 40% 37% 39% 33% 60% 60% 59% 63% 41% 27% 49% Panel A Panel B River A River B Social 54% Unaided Awareness Aided Awareness 21

Brand Usage Miller Beer 100% 90% 80% 70% 60% 50% 40% 30% 20% 40% 41% 54% 47% 41% 39% 10% 0% Panel A Panel B River A River B Social 22

Category Purchasers Gasoline 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 96% 91% 93% 95% 94% 92% 0% Panel A Panel B River A River B Social 23

Purchase Frequency Gasoline Panel A Panel B River A River B Social 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% More often than twice a week Twice a week Once a week Every 2-3 weeks Once a month Less often than once a month 24

Brand Awareness BP 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 45% 47% 40% 36% 37% 38% 44% 46% 37% 57% 33% Panel A Panel B River A River B Social 25% Unaided Awareness Aided Awareness 25

Brand Usage BP 100% 90% 80% 70% 60% 50% 40% 30% 20% 47% 41% 58% 50% 57% 47% 10% 0% Panel A Panel B River A River B Social 26

Category Purchasers Drug Stores 100% 90% 80% 70% 60% 50% 40% 97% 89% 97% 97% 97% 96% 30% 20% 10% 0% Panel A Panel B River A River B Social 27

Purchase Frequency Drug Stores Panel A Panel B River A River B Social 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Once a week or more often 2-3 times per month Once a month Every two months Every 3-4 months Less often than every 3-4 months 28

Brand Awareness CVS Walgreens 29

Category Purchasers Insurance 100% 90% 80% 70% 60% 50% 40% 30% 70% 65% 75% 71% 75% 75% 20% 10% 0% Panel A Panel B River A River B Social 30

Purchase Volume Insurance Panel A Panel B River A River B Social 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Purchased 1 policy Purchased 2 policies Purchased 3 or more policies 31

Brand Awareness Allstate Geico 32

Brand Usage Allstate Geico 33

Category Purchasers Cellular Telephone Service 100% 90% 80% 70% 60% 50% 40% 78% 78% 82% 81% 96% 84% 30% 20% 10% 0% Panel A Panel B River A River B Social 34

Average Monthly Bill Wireless Telephone Service $100 $90 $80 $70 $60 $50 $40 $30 $63 $61 $72 $63 $84 $87 $20 $10 $0 Panel A Panel B River A River B Social 35

Brand Usage Sprint Verizon 36

Category Purchasers Credit Cards 100% 90% 80% 70% 60% 50% 40% 30% 20% 73% 76% 83% 83% 82% 49% 10% 0% Panel A Panel B River A River B Social 37

Average Monthly Expenditure Credit Cards $500 $450 $400 $350 $300 $250 $200 $441 $451 $541 $418 $488 $447 $150 $100 $50 $0 Panel A Panel B River A River B Social 38

Brand Awareness Visa 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 17% 19% 20% 83% 80% 80% 14% 86% 23% 74% Panel A Panel B River A River B Social 16% 69% Unaided Awareness Aided Awareness 39

Category Purchasers Notepads 100% 90% 80% 70% 60% 50% 40% 30% 80% 75% 88% 89% 92% 76% 20% 10% 0% Panel A Panel B River A River B Social 40

Brand Usage Post-It 100% 90% 80% 70% 60% 50% 40% 30% 20% 58% 46% 64% 66% 76% 63% 10% 0% Panel A Panel B River A River B Social 41

Category Purchasers Panel A Panel B River A River B Beer Disposable Diapers Cellular Service Facial Tissue Fast Food/Casual Dining Donut or Coffee Shops Batteries Credit Cards Drug Stores Non-401K Investments Insurance Social Notepads Car Rental Personal Computers Gasoline Higher Than Average Average Lower Than Average 42

Purchase Frequency/Volume `` Panel A Panel B River A River B Social Beer Disposable Diapers Cellular Service Facial Tissue Fast Food/Casual Dining Donut or Coffee Shops Batteries Credit Cards Drug Stores Non-401K Investments Insurance Notepads Car Rental Personal Computers Gasoline Higher Than Average Average Lower Than Average 43

Brand Awareness Miller Beer Panel A Panel B River A River B Social Energizer VISA CVS Walgreens Fidelity Investments Allstate Geico Post-It Avis Dell Huggies HP Gateway BP Pampers Sprint Verizon Kleenex Puffs Burger King Dunkin' Donuts Higher Than Average Average Lower Than Average 44

Brand Awareness Verizon 82% 97% Miller Beer 90% 81% Sprint 82% 96% Allstate 93% 84% Dell 98% 85% Walgreens 94% 87% Burger King 98% 85% Gateway 87% 80% Post-it 80% 93% Puffs 87% 81% Kleenex 96% 84% CVS 88% 86% VISA 97% 85% Huggies 92% 92% Energizer 95% 84% Pampers 92% 92% Geico 93% 84% Avis 76% 83% HP 93% 84% BP 70% 82% Dunkin's Donuts 97% 88% Fidelity Investments 60% 74% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Social 45

Brand Usage Miller Beer Panel A Panel B River A River B Social Energizer VISA CVS Walgreens Fidelity Investments Allstate Geico Post-It Avis Dell Huggies HP Gateway BP Pampers Sprint Verizon Kleenex Puffs Burger King Dunkin' Donuts Higher Than Average Average Lower Than Average 46

Summary Demographic differences exist based on sample source, with the most notable differences existing between the social networking samples and the other sample sources. Respondents from the Social tended to be younger, they are less likely to be married, and they are more likely to be female. In contrast, respondents from are more likely to be male, are less educated, and they have lower household incomes. Consistent with previous research, rates of fraudulent responding and mental cheating do differ by panel source. Of particular note: The Social and the sample differ materially on both demographic and fraudulent/cheating behavior. generally resembles other river samples in that fraudulent/cheating behaviors are minimal. However, using a ad as the sole sample source and as executed for this research is not a tenable approach for most commercial research. 47

Production Data from the Ad click rate 10,509,614 impressions generated 3,233 clicks to the survey landing page.031% ad click rate Survey start rate 264 people of the 3,233 people who clicked on the ad answered at least one survey question 8% start rate Survey completion 70 of the 264 who started the survey finished it 26% completion rate CPI for completes: $56 CPI for partial interviews (though 15 minutes of survey): $26 Interviewing spanned 20 days, albeit over the end-of-year holidays 48

Summary While some variation does exist in incidence of category purchasing, aided and unaided awareness, and brand purchase, the differences tend to be relatively small. Significant differences on omnibus tests were not found across the vast majority of consumer-behavior/marketing-related variables examined in this research. However, many of the differences that do exist involve the Social and the sample. Sometime these two sample sources differed in the same direction, and sometimes they did not. 49

Implications More research is needed to determine whether these findings are replicable in lower incidence product categories. For many product categories, sample quota controls based on demographic variables must be applied because those variables have known relationships with attitudes that impact consumer behavior. Because samples drawn from social networks tend to produce different results from those obtained from traditional opt-in and river samples, organizations should do due diligence to determine where their use is most warranted. The research industry would be well served to look for ways to introduce additional scientific rigor into sample blending. 50