The Economic and Cognitive Costs of Annoying Ads Daniel G. Goldstein Siddharth Suri Fernando Diaz Microsoft Research, New York City R. Preston McAfee Microsoft Corporation Matthew Ekstrand-Abueg Northeastern University
Questions of the day How can online labor markets be used for field studies? What are the economic and cognitive costs of annoying ads?
Field studies in online labor markets Mechanical Turk Widely used as subject pool (much like the early ones of Reips, Birnbaum, Baron, Göritz, Goldstein, etc.) However, MTurk was created as a bona fide labour market Work as DV. Dropout as a good thing. Compensating differential Amount needed to pay someone to compensate for risk or unpleasantness Innovative method of Toomim et al for testing user interfaces
Publisher: New York Times Citibank paid New York Times Display Ad Advertiser: Citibank
Display advertising Online advertising about as large as broadcast TV Display: $13 billion USD in 2013 33% of online advertising revenue Display expected to overtake sponsored search ads on mobile devices
Good Ads
Bad Ads
Mobile annoying ads
Tension " Advertisers pay publishers to display annoying ads " Annoying ads cost publishers page views through user abandonment
Impacted parties " Publishers " Lost traffic " Apparent desperation: Warning of service quality " Users " Interfere with content consumption. 200 million AdBlock downloads on Firefox alone. " Advertisers " Appear disreputable " Undermines money-to-burn signal
Research questions " What makes ads annoying? " What are the economic costs of annoying ads? " What are the cognitive costs of annoying ads?
Related Work " Yoo & Kim 2005, Burke et al 2005 " Moderate animation increases brand recognition and attitude " Heavy animation decreases brand recognition and attitude " Goldfarb & Tucker 2011 " Ads that are intrusive or match content increase intent to purchase " Ads that are both decrease attention decrease intent to purchase " Calculating compensating differentials, Toomim et al 2011
Preparatory study: Identifying annoying ads
Ranking Ads by Annoyingness " 72 animated ads, 72 static ads generated from their final frame " Users randomly shown either the static or animated version " First users previewed all ads
Ranking Ads by Annoyingness " 72 animated ads, 72 static ads generated from their final frame " Users randomly shown either the static or animated version " First users previewed all ads " Then rated each on a 1-5 scale of annoyingness
Ranking Ads by Annoyingness " 72 animated ads, 72 static ads generated from their final frame " Users randomly shown either the static or animated version " First users previewed all ads " Then rated each on a 1-5 scale of annoyingness " Finally, commented on why
Preparatory study participants " Conducted on Amazon Mechanical Turk " $0.25 flat rate, $0.02 per ad rated " 141 participants in 18 hours " Restricted to US workers with 95% approval rating
Identifying Good & Bad Ads bad ads good ads " Animated ads are most annoying, static are least " Used 10 most/least annoying ads in study 2
The Impact of Animation " Animation caused ads to be rated as substantially more annoying
What Makes Ads Annoying?
Study 1: The economic cost of annoying ads" "
Measuring the " Compensating Differential " Task: users classify email, can stop any time " Randomly assigned into " 3 pay conditions:.2,.4,.6 cents per email " 3 ad conditions: no ads, good ads, bad ads " Treatment not revealed until acceptance " Circumvented ad blockers
" Ads were randomly selected from the set of good or bad ads " New ads with each email " Emails from the Enron data set
Study 1 participants " Conducted on Amazon Mechanical Turk " $0.25 flat rate, 0.2, 0.4, 0.6 cents/email " 1223 workers in two weeks " Restricted to workers with 90% approval rating
Compensating Differential: " Bad Ads to No Ads " Bad ads caused 12.7 fewer impressions compared to no ads ".2 cents per impression yields an additional ~16.6 impressions " To get an additional 12.7 impressions need to pay 0.2 * 12.7/16.6 =.153 40 52 50 68 65 85
Calculation If you get 16.6 additional impressions for every.2 cents you pay, how much does it cost to make up the 12.7 impressions lost to annoying ads? X ( 16.6 impressions/.2 cents )= 12.7 impressions X =12.7 impressions(.2 cents/16.6 impressions ) X =.153 cents/ impression X = $1.53 / 1k impressions =$1.53 CPM
Compensating Differential: " Bad Ads to Good Ads " Bad ads caused 9.5 fewer impressions compared to good ads ".2 cents per impression yields an additional ~16.6 impressions " To get an additional 9.5 impressions need to pay 0.2 * 9.5/16.6 =.115 " $1.15 CPM 40 47 52 62 68 81
Robustness " Data are skewed: " Many people categorized a few emails " Few people categorized a lot " Means drop as observations are excluded reflecting skew, annoyingness effect remains " Good ads and no ads yield same number of impressions
Accuracy " Enron data set has emails categorized as spam or ham Treatment Bad Ads Good Ads No Ads Accuracy 90% ±1 92% ±1 93% ±1 " Statistically significant difference between bad ads and good ads/no ads
Study 2: The cognitive cost of annoying ads" "
Mousetracking Mousetracking correlates with eye tracking Chen, Anderson, & Sohn, 2001; Guo & Agichtein, 2010; Huang, White & Buscher, 2012 Mousetracking proxies for user interest Steiger & Reips, 2010; Willemsen & Johnson, 2010; Navalpakkam, & Churchill, 2012
Study 2 participants " Conducted on Amazon Mechanical Turk " $0.50 flat rate, $0.10 per question answered " 2,840 participants
Task Good or Bad Ad
Manipulation check 42% listed ad as annoying in bad condition 5% did so in good condition
Good ad fixations
Bad Ad Fixations
Mousetracking metrics Fixations of the mouse in an area Distance mouse travels in an area Number of entrances of the mouse into an area Time mouse spends in an area
Mousetracking metrics:" Ad area Measure Bad Ads Good Ads P-Val (log OLS) Fixations 4.45 (0.67) 1.57 (0.26) <.001 Distance 182.6 (7.9) 157.9 (8.0).003 Entrances 1.31 (0.05) 1.13 (0.05).004 Time (ms) 1873 (321) 1101 (186).001
Mousetracking metrics:" Text area Measure Bad Ads Good Ads P-Val (log OLS) Fixations Distance Entrances Time (ms) 135.7 (9.07) 128.0 (9.05).047 1,492 (55.20) 1,570 (66.60).677 1.51 (.06) 1.50 (.07).322 38,268 (1,207.00) 36,367 (1,085.00).596
Mouse reading Rated by 3 judges About 7% incidence of mouse reading in good and bad conditions, no significant difference
Accuracy Accuracy was significantly lower in bad condition compared to the good ad condition (2.5 percentage point delta, 6 percentage points for answer at bottom of page)
Summary of mousetracking studies Annoying ads do capture user attention Annoying ads have negative affect on accuracy However, absence of evidence that people read differently when annoying ads present. Why? Could be depletion, however, in Experiment 1, the accuracy difference between the bad and good ad conditions held constant. Our hypothesis is that annoying ads make people care less about what they re reading or the questions they are answering
" Conclusions & Future Work"
Conclusions " Animation has a causal effect on annoyance " Annoying ads hamper accuracy in cognitive tasks " Annoying ads cause dropout " Compensating differential of bad ads to no ads/good ads is at least $1 CPM, which would price some ads out of the market " Good ads lead to about the same number of impressions as no ads " We would lose money by running annoying ads
Conclusions " Mousetracking shows that people attend to annoying ads and they the impact accuracy " However, they don t appear to affect reading behavior and may impact accuracy for motivational reasons
Research Agenda " Towards a psychologically informed display ad pricing scheme " Display ads should be priced in terms of time in view instead of impressions (Economics and Computation, 2011) " Two short ads are better than one long ad (Economics and Computation, 2012) " What is the cost of annoying ads? (Journal of Marketing Research, 2014)
" Thank you!"