How Effective is Targeted Advertising?

How Effective is Targeted Advertisig? Ayma Farahat Yahoo! afarahat@yahoo-ic.com Michael Bailey Departmet of Ecoomics, Staford Uiversity mcbailey@staford.edu ABSTRACT Advertisers are demadig more accurate estimates of the impact of targeted advertisemets, yet o study proposes a appropriate methodology to aalyze the effectiveess of a targeted advertisig campaig, ad there is a dearth of empirical evidece o the effectiveess of targeted advertisig as a whole. The targeted populatio is more likely to covert from advertisig so the respose lift betwee the targeted ad utargeted group to the advertisig is likely a overestimate of the impact of targeted advertisig. We propose a differece-i-differeces estimator to accout for this selectio bias by decomposig the impact of targetig ito selectio bias ad treatmet effects compoets. Usig several large-scale olie advertisig campaigs, we test the effectiveess of targeted advertisig o brad-related searches ad clickthrough rates. We fid that the treatmet effect o the targeted group is about twice as large for bradrelated searches, but aively estimatig this effect without takig ito accout selectio bias leads to a overestimatio of the lift from targetig o brad-related searches by almost 1,000%. Categories ad Subject Descriptors J.4 [Social ad Behavioral Scieces]: Ecoomics ;G.3[Probability ad Statistics]: Experimetal Desig Geeral Terms Experimetatio, Measuremet, Ecoomics Keywords Behavioral targetig BT, Clickthrough rate CTR, Olie advertisig, Advertisig effectiveess, Field experimets, Selectio bias 1. INTRODUCTION As olie advertisig is proliferatig at a ever icreasig pace, clickthrough rates for olie advertisemets have decreased from 3% to far less tha 1% [15]. To improve the effectiveess of their campaigs, advertisers ad cotet providers are icreasigly turig to targeted advertisig, or Copyright is held by the Iteratioal World Wide Web Coferece Committee IW3C2. Distributio of these papers is limited to classroom use, ad persoal use by others. WWW 2012,. ACM 978-1-4503-1229-5/12/04. advertisig methods that deliver idividually catered advertisemets based upo the cotet of the website, locatio of the user, browsig history, demographics, the user profile, or ay other available iformatio. Purveyors of targeted advertisig ofte promise improved performace, ot oly i beig able to deliver the advertisemet to desired user segmets, but also icreased performace metrics like clickthrough rates CTR ad sales coversios. Nevertheless, there are few studies to date that measure the effectiveess of targeted advertisig. Give that targeted campaigs carry a premium over other advertisig products [5], advertisers are demadig more accurate estimates of the impact of targetig to be able to evaluate whether the additioal cost is greater tha the margial retur o a targeted advertisemet. Advertisers target their advertisemets to the group of users they expect are most likely to respod to the advertisig. This provides a major challege i estimatig the effect of targeted advertisig because the populatio is chagig simultaeously with the ad, ad this selectio bias will cause ay study that aively looks at respose lifts betwee the targeted ad utargeted group to greatly overestimate the effect of advertisig [4]. To effectively aalyze the impact of targeted advertisig, we must ot oly measure the respose of the targeted ad o-targeted populatios to the advertisig, but also measure their respose i the absece of the advertisig itervetio, allowig oe to measure the treatmet effects of advertisig. There is oly value i targetig if the treatmet effect o the targeted group is greater tha the effect o the utargeted group. I this paper, we discuss previous theoretical ad empirical work o targeted advertisig ad discuss how our methods accout for the selectio bias igored i previous work o targetig. We the itroduce a differece-i-differeces estimator to evaluate the effectiveess of targetig that cotrols for the selectio bias ad usig a large-scale atural field experimet ivolvig several olie advertisig campaigs ad a specific iterest-based targetig product, we compare our bias-corrected estimates of the impact of targeted advertisig with aive estimates. Fially, we estimate a model of targeted CTRs that decomposes the effect of the advertisemet, clickiess the propesity to click o ay ad of users, ad brad ad category iterest of users o targeted CTRs. We fid that brad search lifts from targetig are almost etirely selectio bias 77% of the lift o average, but the treatmet effect for the targeted populatio is double that 111

of the utargeted populatio. Clickthrough rate lifts are mostly the treatmet effect oly 11% is selectio bias, but the media bias-corrected CTR lift is oly 1/3 that of the aive CTR lift ad we argue that this is a lower boud of the selectio bias give the targetig algorithm we aalyze. Fially, we fid that brad iterest is by far the most importat determiat of targeted CTRs, greatly outweighig the clickiess of targeted users ad the attributes of the advertisemet. 2. TARGETED ADVERTISING Although our evaluatio methods could be applied to ay type of targetig, we will focus o behavioral targetig BT for two reasos: 1 Most of the work i evaluatig the effectiveess of targeted advertisig has focused o behavioral targetig ad 2 sice targeted users are chose based upo similar behavior, traditioal measures of advertisig effectiveess are very likely to igore a strog selectio bias; the targeted users behavior is very likely to be highly correlated with the measured respose. Ya, et. al. [23] offer oe of the first looks ito whether there is ay value i targetig i olie media. Their goal is to see if targetig ads based upo user behavior leads to a sigificat improvemet i clickthrough rates CTRs. First, they segmet their sample of users ito groups defied by similar browsig ad query behavior. For each ad i their sample, they fid the segmet that had the highest CTR o that ad ad estimate the potetial CTR lift from targetig as: CTR segmet CTR ALL CTR Lift CTR ALL where CTR segmet is the highest CTR o the ad amogst all segmets ad CTR ALL is the average CTR o the ad. They fid that through segmetatio the CTR ca be improved by as much as 670% ad argue that with more ovel segmetig approaches CTRs could be improved as much as 1000%. However, oe caot say whether the icreased CTR is because the advertisemet was a good fit for that segmet or whether it just so happeed that the particular segmet cotaied the users with the most clickig ad olie activity. For example, the user segmet with the highest CTR o a ad could coceivably click more o several ads or most ads i a particular category, idicatig that segmetatio or targetig did t deliver clicks from users iterested i the product promoted by the ad, but just delivered users more likely to click o ay ad. The CTR lift could be a valid measure of the value of targetig if all the advertisers cared about is clicks, but if advertisers care about iterest i their category or brad, there is o way to tell how much of this lift was due to a good match betwee the product i the ad ad the iterests of the high CTR segmet, or whether the segmet would have a higher CTR lift for ay geeric ad. Additioally, because their aalysis is doe i a ex-post fashio simply selectig the segmet with the highest CTR, it igores the problem that advertisers must choose their targetig model or segmet before the ad is show geerously assumig that advertisers always target the optimal segmet. A priori, uless most groups of users have similar clickig propesities for all ads, we should expect this methodology to estimate a large CTR lift from targetig for ay ad, but this CTR lift overestimates the value of targetig to advertisers, especially if advertisers care about which users are clickig o the ads 1. Chag ad Vijay [7] use historical data o Yahoo! properties to compare the CTR of users who would have qualified for a particular BT category for a ad versus the CTR of all users. For example, a BT category could be users iterested i fiace, ad the authors would the see if those who qualify for that category have a larger CTR o the fiace ad the all users. They estimate a variat of CTR Lift: CTR qualified ad lift ad = 1 100 CTR Lift 2 CTRad all ad fid that the CTR lift is 39% over typical Yahoo! users ad eve more o sites with less cotextual iformatio like the Frot Page 56% or Mail 61%. Agai, the problem with usig this measure of the lift is that users who qualify for the BT category differ systematically from all users ad the CTR lift could be just as high o ay radom advertisemet because the users who qualify for the BT segmet might have more olie activity ad are more likely to click o ay give advertisemet. This paper exteds their work i several ways: 1 We lay dow a rigorous theoretical framework ad ecoometric methodology to distill the effectiveess of targeted advertisig cotrollig for selectio bias 2 Exploitig a large scale atural experimet that exposes targeted ad utargeted users to both targeted ad utargeted ads, we ca measure how much search ad CTR lifts are due to the advertisemets, the targetig, ad the clickiess of users 3 We provide a empirical model of CTRs to explai the variatio i CTRs due to targetig. 3. IDENTIFYING THE IMPACT OF TAR- GETING USING TREATMENT EFFECTS A treatmet effect TE is the average causal effect of some treatmet, policy, or program o some measurable outcome of iterest, e.g. the effect of a job-traiig program o future employmet rates. Our goal is to measure the treatmet effect of targeted advertisig keepig i mid that with targetig, the populatio receivig the treatmet might differ from the populatio ot receivig the treatmet, ad a appropriate methodology must accout for this populatio differece. Followig the stadard otatio o treatmet effects [12], let Y i1 be the respose of idividual i whe idividual i receives the treatmet ad Y i0 be the respose of idividual i whe idividual i is utreated, or assiged to the cotrol group for example if the outcome of iterest is brad related queries the Y i1 = 1 if the user makes a brad related query after seeig the advertisemet ad Y i1 =0otherwise. Let D i be a idicator variable equal to 1 if idividual i 1 The authors also itroduce a F measure to measure the effectiveess of targetig. However, there is o atural iterpretatio for this ordial F measure ad it caot iform a advertiser of the margial reveue from a targeted advertisemet. More importatly, usig F as our estimate for the value of targetig does ot elimiate the selectio bias problem; a advertisemet might have a very high F measure, but it could be that the same targeted group would have a similar F measure o ay ad meaig the users have a high clickiess. 112

receives the treatmet ad equal to 0 otherwise i our case D i = 1 idicates the user saw the advertisemet. We wat to measure the impact of the advertisemet, or the idividual treatmet effect, Y i1 Y i0, for each idividual so that we ca costruct the average treatmet effect ATE: EY i1 Y i0 ATE expectatio is take over the populatio. The ATE is the average differetial i respose betwee the users who saw the ad ad those who did ot. For each idividual we observe Y i = Y i0 + D iy i1 Y i0, so we ca ever estimate Y i1 Y i0, the idividual treatmet effect, or the ATE because we ca t simultaeously put the idividual i the treatmet ad cotrol. We ca t see what the user would do after seeig the ad, ad the simultaeously measure what the user would have doe had they ot see the ad. Aother widely used measure of the impact of the treatmet is the average treatmet effect o the treated ATET: EY i1 Y i0 D i =1=EY i1 D i =1 EY i0 D i =1 ATET which is the average treatmet effect for those who are assiged to receive the treatmet. Because Y i0 Di =1isot observed, the estimatio of the ATET is impossible to estimate directly there is o such thig as a group that is treated ad does ot receive the treatmet. Oe approach to measurig the treatmet effect is to measure the differece i outcome betwee the treated ad the utreated. The Naive estimator, which compares the average outcome betwee the treatmet ad cotrol, ca be writte as: NAIV E = EY i D i =1 EY i D i =0 Naive = EY i1 D i =1 EY i0 D i =0 = EY i1 D 1 EY i0 D 1+EY i0 D 1 EY i0 D 0 = {EY i1 D 1 EY i0 D 1} + {EY i0 D 1 EY i0 D 0} } {{ } } {{ } ATET Selectio Bias Our Naive estimate is equal to the ATET, plus a term we deote the Selectio Bias. The selectio bias is the differece i respose betwee the selected ad uselected populatios from beig left utreated. The Naive estimator completely igores what the treated would have doe had they ot see the ad, ad the utreated had they see the ad. If there is o differece betwee the treated ad utreated populatio, for example if D i was chose at radom through a cotrolled experimet, the EY i0 D 1=EY i0 D 0adthe selectio bias would be 0. The radom assigmet of cotrol allows oe to just look at differece i outcome betwee the treated ad utreated to measure the treatmet effect. I our case, assigmet to treatmet ad cotrol is determied by the targetig criterio of the advertiser which makes D i far from radom. Whe the treated ad utreated populatio differ remarkably, the selectio bias could be very large. For example, cosider site retargetig where the targeted populatio are those who visited the advertiser s website. These users have already displayed iterest i the brad or product beig promoted ad are much more likely to covert tha the geeral populatio i the absece of treatmet. The Naive estimate igores the fact that the targeted populatio is more likely to covert eve without seeig the ad ad overestimates the ATET by the amout of selectio bias. 3.1 Differece-i-differeces The value of targetig to advertisers is the ATE, or how much larger the treatmet effect is o the treated tha o the utreated. If the treatmet effect is the same size for the targeted ad utargeted group, the targetig will ot improve the margial reveue of the ad. To get the ATE, we must take the ATET ad subtract the ATE o the utreated ATEU: AT E = AT ET AT EU = EY i1 D 1 EY i0 D 1 EY i1 D 0 EY i0 D 0 μ 11 μ 01 μ 10 μ 00 DID Where μ kl is the average respose of the group with treatmet assigmet D i = l who receive treatmet k =0, 1 = utreated, treated. To estimate this differece-i-differeces DID, we must see how targeted ad utargeted users respod to the advertisig, ad how targeted ad utargeted users respod i the absece of advertisig. We ca the fid the effect of the treatmet o the targeted group, ad differece out the treatmet effect o the utargeted group to fid the margial impact of showig a advertisemet to the targeted group over the utargeted group. We ca rearrage the DID ad write the ATE as: AT E =μ 11 μ 10 μ 01 μ 00 DID2 DID fids the ATE by fidig the ATET ad the subtractig off the ATEU. DID2 offers a alterative iterpretatio of DID, we ca obtai the DID estimate by first estimatig the differece i respose betwee the targeted ad utargeted populatio to the treatmet. However this differece might be due solely to differeces i the populatios, ad have othig to do with the advertisig, so we must correct for this bias by differecig μ 01 μ 00, the differece i respose betwee the targeted ad utargeted group i the absece of the advertisig itervetio, which tells us how much of the differece is due solely to the differeces betwee the populatio. It should be oted that matchig estimators, which match similar users who received differet treatmets, will oly help i estimatig the ATET; to obtai the ATE we eed a proper differece-i-differeces to accout for the selectio bias. The DID estimator also falls aturally out of a ecoometric model as follows. 4. ECONOMETRIC MODEL We model the process of users respodig to a advertisemet as a repeated Beroulli trial with probability of success μ. This probability depeds o whether the user saw the ad or ot, ad other observable characteristics of the user, so we assume that users who are observably idetical have the same probability of coversio. Suppose we ru a experimet ad radomly expose some subset of the populatio to the advertisemet, ad leave some part of the populatio uexposed. Additioally, there is a subset of the populatio that the advertiser is targetig for the advertisig campaig, but exposure to the ad is radom ad irrespective of targetig. Let y i =1ifuseri coverts ad y i = 0 otherwise. If user i has probability of 113

coversio, μ, the : Pry i =1=Ey i=μ Vary i=μ1 μ If Y is the umber of users who covert out of total users who covert with probability μ, a sum of the Beroulli successes, the Y is distributed biomial with mea μ ad variace μ1 μ. Therefore, the expected coversio rate, E Y µ1 µ, has mea μ ad variace. We model the expected value, E Y =μ, as a liear combiatio of the idepedet variables, X which icludes whether the user is targeted ad whether the user is exposed to the ad through a lik fuctio f : Y E = μ = f 1 Xβ Sice Y is distributed biomial, the appropriate lik fuctio is the logit, fx = x. If the umber of users i the 1 x experimet is large, Y is approximately distributed ormal with mea μ ad variace µ1 µ. The ormal distributio correspods to the lik fuctio fx =x. Additioally, if Y the probability of coversio is small, is also approximately distributed Poisso with mea μ ad variace µ if μ is small the μ μ1 μ. The Poisso lik fuctio is fx =lx. μ l = Xβ 1 μ Biomial μ = Xβ Normal lμ =Xβ Poisso Igorig other idepedet variables, suppose the probability the user clicks o the advertisemet is expaded as: fμ i=xβ = β 0 + β 1Ad i + β 2Target i + β 3Ad i Target i Where Ad i is a dummy variable equal to uity if the user i saw the ad ad Target i is a dummy variable that is equal to uity if user i is targeted for the ad, ad they are both equal to 0 otherwise. β 0 is the baselie coversio percetage, or the coversio rate of users who do t see the ad ad are utargeted. β 1 is the differece i the probability of coversio betwee those seeig the ad ad those ot seeig the ad, it is the margial effect of the ad holdig targetig costat. β 2 is the differece i probability of coversio betwee the targeted ad utargeted group. This is the Selectio Bias. β 3 is the iteractio term betwee beig show the targeted ad as well as beig i the targeted segmet. This measure is the margial icrease i the probability of coversio whe the user is show the ad ad is part of the targeted group or the ATE. This is the value of targetig to advertisers aswerig, how much larger is the treatmet effect o the targeted group? Note that β 1 + β 3 is the ATET ad β 1 is the ATEU ad that whe computig averages over populatios we would fid that: fey Ad =1,Target=1=β 0 + β 1 + β 2 + β 3 fey Ad =1,Target=0=β 0 + β 1 fey Ad =0,Target=1=β 0 + β 2 fey Ad =0,Target=0=β 0 If we were to use the aive estimates of the impact of targetig we would fid that : NAIVE = fey Ad = 1,Target= 1 fey Ad =0,Target=0 = β 0 + β 1 + β 2 + β 3 β 0 = β 1 + β 2 + β 3 =β 1 + β 3+β 2 = Treatmet Effect + Selectio Bias If we had show all targeted users the advertisemet, ad withheld the ad from all the utargeted users, we would have a fudametal idetificatio problem; wheever Target=1, we would have that Ad =1, so there is o way to separately idetify β 2 ad β 3. By determiig who sees the ad at radom however, we ca properly idetify β 3 usig DID: DID = fey Ad = 1,Target= 1 fey Ad =0,Target=1 fey Ad =1,Target=0 + fey Ad = 0,Target= 0 = β 0 + β 1 + β 2 + β 3 β 0 + β 2 β 0 + β 1+β 0 = β 3 which yields a estimate of the desired parameter. The test statistic for the differece i differeces estimate of the impact of targetig for the three lik fuctios is: [ l μ11 [ l =l 1 μ 11 μ01 1 μ 01 µ 11 1 µ 11 l l 1 μ10 1 μ 10 μ00 ] ] 1 μ 00 = β 3 µ 01 1 µ 01 1 Logit μ 11 μ 10 μ 01 μ 00 =β 3 Idetity l μ 11 l μ 10 l μ 01 l μ 00 =l µ11 µ 01 = β 3 Log Whe usig the idetity lik fuctio, we ca iterpret β 3 as the margial icrease i coversio probability. The iterpretatio of β 3 is ot as straightforward for the other lik fuctios as the margial icrease is i terms of the log odds ratio. As a alterative to the differeces i differeces estimator, we use the followig approximatio to the logarithmic fuctio, l1 + x x for x small, to show that a quotiet-i-quotiets QQ, which has a more atural iterpretatio tha the other estimators for β 3, ca sometimes approximate the Poisso lik estimator: 114

µ11 µ 01 µ11 µ 01 µ11 µ 01 l 1for 1 small Quotiet-of-quotiets The Quotiet-of-quotiets is iterpreted as percetage deviatios from the base coversio rate. The aive quotiet, µ 11, istheatetipercetageterms,adwhedivided by µ 01 yields the ATE i percetage terms. The mai takeaway from the model is that to properly evaluate the effectiveess of targeted advertisig, the appropriate experimet would be to show the advertisemet to some members of the utargeted sample, ad to withhold the advertisemet from some members of the targeted sample, to see how they respod i the presece/absece of advertisig. Oly the ca the proper differece-i-differeces be estimated. The samples do ot have to be evely sized, but the power of the test to determie sigificat differeces is depedet o the sample sizes of the differet groups. 5. NATURAL FIELD EXPERIMENT We exploit a atural field experimet from the large rectagular ad uit o the Yahoo! Frot Page www.yahoo.com. Yahoo! Frot Page advertisemets are sold i roadblocks every user is delivered the advertisemet ad occasioally are split evely betwee two advertisers such that a visitor will be show a advertisemet from the first advertiser if the time of arrival to the Frot Page is o a eve secod, ad from the secod advertiser if the arrival time is o a odd secod, regardless of the characteristics of the visitor. A impressio for oe of the two advertisers is show for every visit to the frot page o that particular date. This is kow as a frot page split. This provides the perfect experimetal setup to compute the Naive ad DID estimates of the advertisig campaigs. For each frot page split we coduct two experimets; we pick oe of the two campaigs as our target campaig, ad defie the other campaig i the split as our cotrol campaig. We the switch the roles of the advertisers, givig us two experimets per frot page split. We observe how the targeted ad utargeted populatios respod to the test advertisemet ad the cotrol advertisemet to compute the DID ad Naive estimates of targetig. Although oe of the frot page campaigs we aalyze are a targetig campaig, for each advertiser we chose the iterest category they would have bee most likely to target. For each user we observe their behavioral targetig profile ad kow to what behavioral targetig segmets they belog, so we ca compute the hypothetical search lift that would have occurred had the advertisemet ru as a targetig oly campaig for the advertiser s iterest category. For example, if the advertisemet is fiace related, we ca idetify which users belog to the fiace BT segmet ad the compute the aive ad DID estimators of the search lift for that category of users. Motivated by recet results that highlight the impact of search o advertisig metrics [17], we begi by examiig the impact of display ads o both brad ad geeric category search terms. We follow with a aalysis of clickthrough rate lifts for the same campaigs. Aalyzig search lifts follows the setup of the ecoometric model exactly. We assume that each user has a costat probability of makig a brad or category-related search, ad this probability differs upo whether the user sees the advertisemet ad is i the target advertisers BT category. Our results are specific to the targetig product chose ad we would expect differet results for differet targetig products ad differet kids of targetig. Differet ways of segmetig ad clusterig users might iduce larger amouts of selectio bias whe computig the effectiveess of targetig. For our experimet, we chose to use a iterestbased targetig product, Yahoo! s BT Egager. BT Egager places users ito broad iterest-based categories like Chiese laguage or fiace othebasisofsearches, pageviews, clicks, ad other browsig behavior. Because this model is meat to idetify broad iterest istead of maximizig clickthrough rate, it provides a coservative estimate of the selectio bias that oe would fid usig more aggressive CTR maximizig targetig strategies. 5.1 Targetig s Impact o Search All data, tables, ad results from the paper ca be foud i the olie appedix icluded i the supplemetary materials for the paper. The experimet icludes 18 advertisig campaigs o 9 frot page splits from May through August, 2011 See table 1 2. For each user, we track searches made o Yahoo! search by the user after their first visit to the frot page, but before their secod visit the secod impressio ad we also exclude searches made 10 miutes after they view the advertisemet. The choice of 10 miutes was motivated by the fact that most searches occur withi 10 miutes of the first visit to the Yahoo! home page. Searches after 10 miutes oly add oise to the estimates. Removig searches made after the secod impressio allows us to clealy idetify which advertisemet iflueced the search ad to igore frequecy effects 3. For the 18 campaigs there were 332 millio such impressios with a average of about 18.4 millio impressios per campaig. We defie a target segmet search as a search that is related to the category of the target advertisemet 4. Similarly, a cotrol segmet search is a search related to the category of the cotrol advertisemet. For the brad related searches, we idetified the most saliet brad associated with each advertisemet ad defie a brad search either target or cotrol as a search that icludes the brad ame. We fid that the ads have a sizeable effect o search; 9/18 ads have a statistically sigificat ad positive effect o brad searches, ad oly oe of the ads had a egative effect 2 The radom assigmet of ads eve secod arrivals to advertisemet 1, odd secod arrivals to advertisemet 2, failed for oe hour durig the 06/20 frot page split which is why the impressios for the two advertisers is ot evely distributed. Because we have o reaso to assume that users arrivig durig that oe hour differ substatially from the rest of the users, it should t ifluece the outcome of the two experimets for that day. 3 we also repeated the experimet icludig all searches ad the results are substatively the same 4 Precisely, Yahoo! has compiled a list of caoical searches for every BT category, ad we defie a segmet search as a search that is i the top 100 search terms for the BT category of the target advertiser. For example, if auto isurace was our target advertisemet campaig with a BT category of isurace, the top 100 caoical search terms would iclude terms such as isurace, geico, progressive, auto isurace, ad prudetial. 115

Table 1: Targeted Brad Search Lift ad CTR Lift, Diff-i-diff Lifts, ad % of Naive Lift that is Selectio Bias for the 18 Campaigs i Aalysis. Brad Search Lifts CTR Lifts Target Category Naive Lift Diff-i-diff SB % of Lift Naive Lift Diff-i-diff SB % of Lift Credit Card 1,647%*** a 0.00% 101% 159%*** 148%*** 7% Isurace 1 b 773%*** 0.02% 83% 123,375%*** 119,901%*** 3% Credit Card 218%*** 0.01% 90% 777,474%*** 761,598%*** 2% Isurace 2 113%*** -0.01% 174% 22%*** 22%*** -2% College 477%*** 0.00% 87% 17%** 4% 77% Isurace 1 738%*** 0.01% 92% 39%*** 8%*** 81% Notebooks 679%*** 0.03% 93% 136%*** 136%*** 1% Digestive System -100%** -0.01% - 126%* -44%** 135% Notebooks 2,010%*** 0.11%*** 77% 5,650%*** 5,643%*** 0% Isurace 1 618%*** 0.01% 92% 31%*** 37% -18% Reality TV -80%** -0.01% -18% 3% 3% 22% Credit Card -40%** -0.01% 41% 79%*** 276%*** -250% Notebooks 1,148%*** 0.09%*** 42% 207%*** -46%*** 122% Adveture Movies 1,281%*** 0.38%*** 62% 39%*** 38%*** 1% Adveture Movies 1,589%*** 0.46%*** 1% 183%*** 171%*** 7% Isurace 1 730%*** -0.01% 109% 131,134%*** 131,385%*** 0% College 510%*** 0.00% 78% 14%** 22%*** -55% Isurace 1 663%*** 0.00% 102% 52%*** 21%*** 59% MEAN 721% 0.06% 77% 57,708% 56,629% 11% MEDIAN 672% 0.00% 87% 102% 37% 2% The Naive Estimate is the CTR differece betwee the target BT segmet ad all users ot i the target BT segmet o the target ad both excludig the cotrol BT segmet. The Diff-i-diff is the differece betwee the ad impact o the targeted group, ad the ad impact o the utargeted group SB % of Lift is the percet of the aive lift that ca be attributed to selectio bias a Two sided t-test p-value usig ormal approximatio; * = p < 0.05; ** = p < 0.01; *** = p < 0.001; the ull hypothesis is that the estimate equals 0. b Oe advertiser, desigated Isurace 1, appears 5 times i our sample with similar advertisemets. All other brads are uique. 116

o searches for the brad of the ad. The average populatio treatmet effect lift i brad related searches is 44.7%. For those i the targeted BT category, four of the ads have a positive ad statistically sigificat effect o brad searches ad the average treatmet effect search lift is 1.8% ad 79.9% for segmet ad brad searches respectively. Table 2 presets the average search lifts for category ad brad searches for the targeted ad utargeted ads ad for the targeted ad utargeted populatios. The Naive estimates are the search lifts betwee the target group seeig the target ad ad the utargeted group seeig the target/cotrol ad colums of table 2. The aive estimate of the effect of targetig o searches is a lift of almost 3,000% for category searches ad 800% for brad searches! A overestimate o the order of 1000%. The targeted group is much more likely to make a brad related search o matter what ad is see, so the lift betwee the targeted ad utargeted group is mostly selectio bias media of 87% selectio bias. Table 1 shows the aive brad search lift ad DID brad search lift for each of the 18 campaigs. The average DID estimate of the effectiveess of targetig is -0.013% for a category search ad 0.067% for a brad search, or media lifts of about 4% ad 51% respectively. The selectio bias accouts for 98% of the aive search lift for segmet searches, ad 77% of the search lift for brad searches. Three of the campaigs are associated with a egative aive search lift, i.e. the targeted group made fewer searches after seeig the ad. This gives evidece that iterest category targetig does ot always yield the optimal target audiece, although sice 2 of the 3 campaigs occurred o 06/23 there could be a aomaly o that date. There is a eormous variatio i the aive lifts, ad to a lesser extet the DID lifts betwee the campaigs. Oe reaso for this variatio is that we throw out all users who belog to the BT group of the cotrol advertiser. Because there are differet levels of overlap betwee the target ad cotrol BT group, we keep a majority of the target group for some campaigs ad ot others. Aother reaso for the variatio is that there is variatio i the BT egager models. For segmets i which demad is high, there is likely more sophisticatio i the modelig ad parameters are adjusted so that supply meets demad. 5.2 Aalyzig Clickthrough Rates with Treatmet Effects The treatmet effects results we derived earlier are ot as portable to a clickthrough rate aalysis. Users ca make a search or make some kid of coversio without seeig the advertisemet, but they caot click o a advertisemet without seeig the advertisemet. The Naive estimator ca o loger be used because we caot observe clicks from the utreated group. I this aalysis, we defie the aive estimator as the differece i CTR betwee the targeted group ad the utargeted group o the targeted advertisemet. This is the CTR Lift estimate discussed previously [23] ad [7]. Suppose we also showed the targeted ad o-targeted populatio a placebo, or cotrol ad. A ad for which targetig should ot be a factor i geeratig a CTR lift betwee the targeted ad o-targeted populatio. If there is ay CTR lift betwee the groups o this placebo ad, the it ca be attributed to a higher click propesity by the targeted populatio, ad ot due to the ad beig a better fit or beig more appropriate for the targeted group. This is how we defie selectio bias i this problem, the CTR lift betwee the targeted ad o-targeted group o a placebo ad for which the lift is solely attributed to a higher click propesity perhaps because of more happy clickers i the targeted populatio. The ideal experimet the ivolves showig target ads ad placebo ads to both the targeted ad o-targeted populatios. The value of targetig is the DID estimator: DID = fey Ad = 1,Target= 1 fey Ad =1,Target=0 fey Ad =0,Target=1 + fey Ad = 0,Target= 0 = NAIV E SELECTION BIAS The value of targetig is the CTR lift subtractig the baselie click propesity of the targeted group. Aother way to get this DID is to first compute the CTR lift of the targeted group betwee the targeted ad utargeted ad. However, this differece could be attributed solely to ad differeces, so we must differece form this the CTR lift of the utargeted group betwee the targeted ad utargeted ad as a measure of the baselie ad differece. There are some importat caveats/assumptios with this approach: 1 We caot estimate the ATET or ay ad impact measures sice it is impossible to click i the absece of seeig the ad; the impact of the ad o clicks has o meaig. 2 Whe the DID estimator is 0 for the search/coversio lift, it meas that the targeted group receives o extra lift from the advertisig above ad beyod the lift i the geeral populatio, this tells us that employig targetig is ot ay more effective i geeratig searches/coversios tha a ruof-etwork campaig. I the CTR case, if the DID estimator is 0, it has the differet iterpretatio that the CTR lift o the target ad is idetical to the CTR lift o the placebo ad, but depedig o the advertisers goals this does ot ecessarily mea that targetig is ot effective. 3 This strategy relies o the placebo ad beig a ad for which ay CTR differece is due to populatio differeces i propesity to click oly. This assumptio fails if prefereces for the target ad cotrol advertisemet differ by treatmet group. As a example, suppose that the target ad is for cologe ad the placebo ad is for perfume, ad the targeted segmet is 20-30 year old males. The targeted group is less likely to click o the placebo advertisemet, but this is ot because of differig click propesities, it is because of differig tastes, which is the problem we are tryig to solve o the target ad, how to separate tastes from clickiess. Formally, this assumptio states that usig our regressio methodology there are o iteractio terms of the form β 41 Ad i 1 Target i, such that the targeted users respod differetly to the placebo ad compared to the geeral populatio for reasos ot related to clickiess. Similarly, if the placebo ad is very similar to the target advertisemet, the targeted group might be more likely to click o it tha the geeral populatio due to tastes ad ot due to clickiess. The frot page split is a ideal veue for testig the aive ad DID estimators for CTR lifts. Because the ads are assiged at radom, we ca defie oe of the two ads as the 117

Table 2: Average Search Lifts Betwee Populatios ad Advertisemets. Category Searches Brad Searches Target Ad Cotrol Ad Lift Target Ad Cotrol Ad Lift Targeted 0.86% 0.87% 1.8% 0.19% 0.13% 79.9% Utargeted 0.03% 0.03% 2.7% 0.02% 0.02% 44.7% Lift 3,157% 3,272% 896% 742% Lifts uder the colums are the search lifts betwee the targeted group ad utargeted group for the ad i the colum. Lifts after the rows are the search lifts betwee the targeted ad utargeted ad for the give group targeted/utargeted. target ad ad the other as the placebo ad ad estimate the aive ad DID estimator for CTR lift. 5.3 CTR Results We restrict our attetio to oly the first advertisemet served so we ca measure the margial impact of a sigular impressio o clicks 5. The 18 campaigs had about 18.4 millio uique users with a mea CTR of 0.15%. Table 1 shows the aive CTR lift ad DID CTR lift for the 18 campaigs. The media aive CTR lift betwee the targeted ad o-targeted group is 102%, ad the DID estimator is 89% of the Naive estimate, meaig that selectio bias oly accouts for 11% of this CTR lift o average. For the ads i our aalysis, the aive CTR lift is a good approximatio of the treatmet effect lift. The Quotiet-of-quotiets model yields similar results. There are a few reasos our estimates of the selectio bias is small for CTR estimatio. It could be that the BT egager model is less proe to assigig users with high clickiess to lots of BT categories. 99.5% of our sample of users is icluded i at least oe BT category ad the BT categories oly overlap by 5% o average, thus a very small portio of our sample is users belogig to several BT categories geeratig lots of clicks ad drivig up the selectio bias. Oe of the reasos the search selectio bias is high is that users are assiged to BT categories based o both predicted ad observed category search patters, but this pheomeo does ot maifest itself for CTR because clicks are a much less commo occurrece ad the model is oly based loosely o CTR. Aother explaatio could be that for several of the ads i our sample there was a egative correlatio i tastes betwee the two ads i the frot page split such that the target BT s clickiess was outweighed by their distaste for the placebo ad. This would be supported by the fact that there is such a small overlap betwee BT category assigmet. However, the categories appear to be ulikely to geerate such a egative correlatio i tastes. 6. A MODEL FOR CLICKTHROUGH RATES After accoutig for the clickiess of the targeted group, we still fid large CTR lifts o frot page advertisemets. Ca we attribute the residual lift to iterest i the brad or category? To make this causal claim we eed to lay dow a behavioral model of clickig that describes why the targeted group is more proe to click o a advertisemet tha 5 the results from lookig at all frot page impressios are substatially idetical ad will be provided upo request. the geeral populatio of users. We posit that CTRs for a targeted group of users is depedet o three thigs: 1 Creative: Attributes of the ad itself that drives a higher CTR for everyoe, like quality of the advertisemet or geeral appeal of the brad. This ca be measured by the CTR of all users o the ad. 2 Clickiess: Click propesity of the group, this captures how much more likely the targeted group clicks o ay ad. We measure clickiess by the selectio bias, or how much more likely the target group clicks o the placebo ad. 3 Iterest: How much additioal iterest does the targeted group have for the brad or category. We measure this by lookig at the lift i category or brad searches for the target group after seeig a placebo ad which is a measure of the pre-determied iterest. We follow the steps outlied i our ecoometric setup ad estimate this relatioship as a geeralized liear model. The CTRs are modelled as beig distributed Gaussia, Biomial, or Poisso by choosig the appropriate lik fuctio with liear coditio mea: fμ i=xβ = β 0 + β 1Creative i + β 2Clickiess i + β 3CategoryIterest i + β 4B rad Iterest i The appropriate lik fuctio or caoical lik fuctio for the Gaussia, Biomial, ad Poisso distributios are the idetity fuctio, logit fuctio, ad log fuctio respectively. We also ru a Box-Cox regressio [Box & Cox, 1964] to determie the best power trasformatio of the depedet variable that would satisfy the liear coditioal mea assumptio. The regressio suggests the data is best modelled as log-liear, with a estimated lambda close to 0 λ = 0.1, i.e. Poisso. We ormalize all idepedet variables to have mea 0 ad a stadard deviatio of 1 the idepedet variables are stadard deviatios from the mea ad estimate all models with robust stadard errors. For the Poisso ad Biomial models, we preset expoetiated coefficiets. Coefficiets for the poisso model ca be iterpreted as the percetage chage i the CTRs per stadard deviatio icrease i the idepedet variable mius oe. For the Biomial model, the iterpretatio is the CTR percetage chage i the odds, ot percetage 1 CTR chage i CTRs; however, sice CTR 0, the odds is approximately equal to the probability, CTR, so the CTR 1 CTR same iterpretatio ca be applied. The coefficiets are ot expoetiated for the Gaussia model ad are the stadard margial effects. The results of the regressio are preseted i table 3. 118

Brad iterest by far is the most importat determiat of segmet CTR ad is statistically sigificat i two of the models i all the models usig oe-tail p-values. A oe stadard deviatio icrease i brad iterest leads to a 138% icrease i CTRs i the Poisso model. Table 3: Regressio Results for CTR Model. Biomial Poisso Gaussia Costat 0.006*** 0.001 Creative 0.69 0.69-0.001 0.2 0.2 0.003 Clickiess 1.15 1.14 0.0003 0.23 0.23 0.001 Category Iterest 1.21 1.21 0.0007 0.44 0.44 0.002 Brad Iterest 2.4* 2.38* 0.006 0.88 0.86 0.001 Lik Logit Log Idetity N 18 18 18 Expoetiated coefficiets reported for Biomial ad Poisso models. Two sided t-test p-value; * = p < 0.05; ** = p < 0.01; *** = p < 0.001; the ull hypothesis is that the estimate equals 0. Robust stadard errors show i paretheses. Clickiess ad Category Iterest are both positive, but ot statistically differet from 0. Creative is egative but ot statistically sigificat. Although this seems to suggest that better ads lead to lower CTRs for the targeted group oce cotrollig for iterest ad clickiess, it is probably more likely the case that the Creative variable is better iterpreted as a measure of geeral iterest i the advertisemet, ad the more broad appeal a ad has, the lower precisio there is i the targeted category. Cotrollig for brad ad category iterest, targetig will ot be as precise for a geeral appeal ad like a blockbuster movie. The target populatio respods to the ad similarly to the geeral populatio oce we ve accouted for clickiess ad category iterest, but without accoutig for brad iterest which drives most of the variatio i CTRs. Our sample of creatives is small =18, limitig the geeral applicability of our results, but it presets strog evidece oetheless that brad iterest is a sigificat determiat of variatio i targeted CTRs. 7. MARGINAL VALUE OF A TARGETED ADVERTISEMENT If the cost per 1000 impressios CPM is $1, for the campaigs we aalyze it would cost a advertiser o average 72 cets per click ad $5.12 for each brad related search. If targetig was used, ad CPM for targeted impressios was also $1, a click would oly cost 16 cets ad a brad related search would oly cost 53 cets. However, for searches, the figure advertisers care about is ot the cost per search, but the margial cost of a brad search, or how much it would cost the advertiser to iduce someoe to search for the brad by displayig a ad. The margial cost of a brad-related search o Yahoo! search from ay user is $15.65, but is oly $1.69 for a targeted user. This oly cosiders searches o Yahoo! ad overstates the cost of all brad-related searches o ay search egie. Uless showig targeted advertisig is 9 times more expesive, targeted advertisig is more cost effective i geeratig brad searches. If all a advertiser cares about is clicks, ad ot about the clickiess of the targeted group, the targeted advertisig is more cost effective at geeratig clicks as log as it is o more tha 4.5 times as expesive as displayig the ad to everyoe. Give a idustry average of a 3 times price premium for targetig [5], we might coclude targetig is more cost effective, but of course this depeds greatly o the targetig product. If advertisers oly value clicks that are t due to selectio bias, for example if they have a iche product ad they oly wat clicks from those who are uiquely iterested i their category, the it would cost the advertiser 22 cets per click, or about 1/3 the cost of a click for a ru-of-etwork campaig. 8. CONCLUSION AND IMPLICATIONS FOR ADVERTISERS We coclude by discussig what advertisers ca lear from our model ad results. Whe computig the effectiveess of a targeted advertisig campaig, it is critical to ot oly compare how the targeted ad o-targeted populatios respod to advertisig, but how they respod i the absece of advertisig. This is because the targeted segmet is more likely to covert i the absece of advertisig tha the utargeted segmet, ad to truly measure the effect of advertisig this selectio bias must be accouted for. We fid large selectio bias i brad related search lifts. If a compariso was made betwee the searches of the targeted ad utargeted group after seeig a advertisemet, we would coclude the advertisemet lifted brad searches 721% o average. I reality, that same lift is observed after seeig a urelated advertisemet, the true effect of advertisig oce we accout for this lift is aroud 79%. We fid that for a iterest-based behavior targetig model, selectio bias or clickiess of the targeted group ca oly accout for about 11% of the CTR lift; we believe this is more a fuctio of a targetig product that is less proe to selectio bias ad the results are a coservative estimate for a more aggressive behavioral targetig model that seeks to maximize CTRs. If advertisers are discrimiate about the types of users clickig o their ads, ad ot just the umber of clicks, our methodology offers isight ito how much of the CTR lift from the targetig group is uique to their advertisemet category. Whe we model CTRs for the targeted groups as a fuctio of advertisemet characteristics, clickiess of the group, ad pre-determied iterest i the category ad brad of the advertisemet, we fid that brad iterest overwhelmig explais variatio i CTRs, so the lift i CTRs from targeted users is beig drive by idetifyig users who have iterest i the brad, ad ot from characteristics of the advertise- 119

mets. Oce we cotrol for category iterest ad clickiess, targeted users respod to the ad like the geeral populatio. This study opes up several questios about the effectiveess of targeted advertisig. Advertisers are seekig more ad more to target their ads to the segmets most likely to covert as a result of the advertisig; however, this strategy may ot be cost effective as this segmet is likely to covert i the absece of ay advertisig. Our results idicate that more sophisticated targetig algorithms might ot gai, ad might eve harm, the advertiser as those seeig the ad would covert i the absece of advertisig. Targeted advertisig is projected to grow at a eormous rate; we hope research o targeted advertisig keeps up the pace. 9. REFERENCES [1] B. Aad ad R. Shachar. Targeted advertisig as a sigal. Quatitative Marketig ad Ecoomics, 73:237 266, 2009. [2] J. Agrist ad G. Imbes. Two-Stage least squares estimatio of average causal effects i models with variable treatmet itesity. Joural of the America Statistical Associatio, 90430:431 442, 1995. [3] J. Agrist, G. Imbes, ad D. Rubi. Idetificatio of causal effects usig istrumetal variables. Joural of the America Statistical Associatio, 91434:444 455, Jue 1996. [4] K. Bagwell. Chapter 28 the ecoomic aalysis of advertisig. Hadbook of Idustrial Orgaizatio, 3:1701 1844, 2007. [5] H. Beales. The value of behavioral targetig. Network Advertisig Iitiative, 2010. [6] G. Box ad D. Cox. A aalysis of trasformatios. Joural of the Royal Statistical Society. Series B Methodological, page 211 252, 1964. [7] K. L. Chag ad V. K. Narayaa. Performace aalysis of behavioral targetig at yahoo! Techical report, Advertisig Scieces, Yahoo! Labs, 2010. [8] J. Che ad J. Stallaert. A ecoomic aalysis of olie advertisig usig behavioral targetig. Techical report, Uiversity of Coecticut, 2010. [9] T. Che, J. Ya, G. Xue, ad Z. Che. Trasfer learig for behavioral targetig. I Proceedigs of the 19th iteratioal coferece o World wide web, pages 1077 1078, 2010. [10] Y. Che, D. Pavlov, ad J. F. Cay. Large-scale behavioral targetig. I Proceedigs of the 15th ACM SIGKDD iteratioal coferece o Kowledge discovery ad data miig, pages 209 218, 2009. [11] E. Gal-Or, M. Gal-Or, J. May, ad W. Spagler. Targeted advertisig strategies o televisio. Maagemet Sciece, 525:713 725, May 2006. [12] G. Imbes ad J. Agrist. Idetificatio ad estimatio of local average treatmet effects. Ecoometrica, 622:467 475, 1994. [13] G. Iyer, D. Soberma, ad M. V. Boas. The targetig of advertisig. Marketig Sciece, 243, 2005. [14] M. Joo, K. Wilbur, ad Y. Zhu. Televisio advertisig ad olie search. Available at SSRN: http://papers. ssr. com/sol3/papers. cfm, 2010. [15] P. Kazieko ad M. Adamski. AdROSA-Adaptive persoalizatio of web advertisig. Iformatio Scieces, 17711:2269 2295, Jue 2007. [16] A. Lambrecht ad C. Tucker. Whe does retargetig work? timig iformatio specificity. SSRN elibrary, July 2011. [17] J. Leciski. Zero Momet of Truth. http://www.zeromometoftruth.com/ [18] R. A. Lewis, J. M. Rao, ad D. H. Reiley. Here, there, ad everywhere: correlated olie behaviors ca lead to overestimates of the effects of advertisig. I Proceedigs of the 20th iteratioal coferece o World wide web, page 157 166, 2011. [19] P. Melville, S. Rosset, ad R. D. Lawrece. Customer targetig models usig actively-selected web cotet. Proceedig of the 14th ACM SIGKDD iteratioal coferece o Kowledge discovery ad data miig, pages 946 953, 2008. [20] D. Rubi. Estimatig causal effects of treatmets i radomized ad oradomized studies. Joural of Educatioal Psychology, 665:688 701, 1974. [21] D. Vakratsas, F. Feiberg, F. Bass, ad G. Kalyaaram. The shape of advertisig respose fuctios revisited: A model of dyamic probabilistic thresholds. Marketig Sciece, 231:109 119, Ja. 2004. [22] T. Wiesel, K. Pauwels, ad J. Arts. Practice prize Paper-Marketig s profit impact: Quatifyig olie ad off-lie fuel progressio. Marketig Sciece, 304:604 611, 2011. [23] J. Ya, N. Liu, G. Wag, W. Zhag, Y. Jiag, ad Z. Che. How much ca behavioral targetig help olie advertisig? I Proceedigs of the 18th iteratioal coferece o World wide web, pages 261 270, 2009. 120