Real-Time Evaluation of Campaign Performance

Transcription

1 Singapor Managmnt Univrsity Institutional Knowldg at Singapor Managmnt Univrsity Rsarch Collction L Kong Chian School Of Businss L Kong Chian School of Businss Ral-Tim Evaluation of Campaign Prformanc Andr Bonfrr Singapor Managmnt Univrsity, andrbonfrr@smu.du.sg Xavir Drz Wharton School, Univrsity of Pnnsylvania Follow this and additional works at: Part of th Advrtising and Promotion Managmnt Commons, and th Markting Commons Citation Bonfrr, Andr and Drz, Xavir. Ral-Tim Evaluation of Campaign Prformanc. (2008). Markting Scinc., 28(2), 251. Rsarch Collction L Kong Chian School Of Businss. Availabl at: This Journal Articl is brought to you for fr and opn accss by th L Kong Chian School of Businss at Institutional Knowldg at Singapor Managmnt Univrsity. It has bn accptd for inclusion in Rsarch Collction L Kong Chian School Of Businss by an authorizd administrator of Institutional Knowldg at Singapor Managmnt Univrsity. For mor information, plas mail libir@smu.du.sg.

2 Ral-Tim Evaluation of Campaign Prformanc Octobr 2006 André Bonfrr* Xavir Drèz * Corrsponding author. André Bonfrr is Assistant Profssor of Markting at th Singapor Managmnt Univrsity, L Kong Chian School of Businss, 50 Stamford Road #05-01, Singapor , andrbonfrr@smu.du.sg, Tl , Fax Xavir Drèz is Assistant Profssor of Markting at th Wharton School of th Univrsity of Pnnsylvania, 3730 Walnut Strt, Philadlphia, PA , , Fax , xdrz@wharton.upnn.du. This rsarch was fundd in part by th Wharton-SMU rsarch cntr, Singapor Managmnt Univrsity and in part by a WBI - Mac Cntr Grant.

3 Ral-Tim Evaluation of Campaign Prformanc W dvlop a tsting mthodology that can b usd to prdict th prformanc of mail markting campaigns in ral tim. W propos a split-hazard modl that maks us of a tim transformation (a concpt w call virtual tim) that allows for th stimation of straightforward paramtric hazard functions to gnrat arly prdictions of an individual campaign s prformanc (as masurd by opn and click rats). W apply our mthod to 25 mail campaigns and find that th mthod is abl to produc in lss than two hours stimats that ar mor accurat and mor rliabl than what th traditional mthod (doubling tim) can produc aftr 14 hours. Othr bnfits of our mthod ar that w mak tsting indpndnt of th tim of day and w produc maningful confidnc intrvals. Thus, our mthodology can b usd not only for tsting purposs, but also for liv monitoring. W show that a campaign slction rul basd on our modl rathr than on th doubling mthod can improv ovrall rspons rats by 20%. Kywords: Databas markting, mail, pr-tsting, advrtising campaigns. 2

4 Introduction can b a powrful vhicl for markting communications. Many marktrs favor this nw mdium bcaus it provids thm with a chapr and fastr way to rach thir customrs. Furthr, th onlin nvironmnt allows marktrs to masur consumrs actions mor accuratly. This is a boon for markting scintists in thir dsir to incras th ffctivnss of markting fforts and masur th ROI of markting xpnditurs. Although mail rspons rats startd out high (spcially whn compard with thos rportd for onlin and offlin advrtising), thy dclind ovr tim and ar now blow 2.5% (DMA 2005). Finding ways to rais ths rspons rats is critical for mail marktrs. A usful tool to achiv this is an ffctiv mail tsting mthodology. Idntifying potntial strngths and waknsss of th contnt (th mail crativ) and th targt population bfor th mail is snt out at full scal can hlp marktrs improv th rspons rats for thir campaigns. As a motivating xampl for th problm w ar intrstd in, considr th cas of a product managr at on of th major movi studios. With two or thr nw DVDs coming out vry wk, studios oftn rly on mail markting to gnrat intrst for upcoming titls. This onlin promotion is particularly important for smallr titls (.g., Transportr 2) that will not bnfit from mass advrtising and will not b havily pushd by Amazon or Blockbustr. For such titls, th product managr would typically ask hr crativ agncy to com up with a fw concpts, and pick on of th dsigns to snd out. Sh would also us a sris of critria (.g., movi gnr, gndr) to slct a subst of hr databas as a targt for th mail. Givn th numbr of nw titls to promot vry wk, sh would work on a short production cycl and snd mails without formally tsting th quality of th dsign or th targt sampl (this is diffrnt from larg rlass such as X-Mn III which ar plannd months in advanc and 3

5 rciv broad advrtising and channl support). Th succss of our managr s mails might b much improvd if sh could tst multipl crativs and targt slction in a fast (sh has short lad tims) and inxpnsiv (th titls do not warrant larg xpnss) way. Thr ar no modls in th xtant markting scinc litratur that can b dirctly applid to provid such a tst. Th importanc of tsting lmnts of th markting mix is not nw to markting scintists. For xampl, in nw product dvlopmnt (and distribution) th ASSESSOR modl (Silk and Urban 1978, Urban and Katz 1983) has bn usd for dcads to forcast th succss of nw products basd on laboratory basd tst markting. Mthods hav also bn dvlopd to prform multipl paralll tsting as prdicatd by th nw product dvlopmnt litratur (Dahan and Mndlson, 2001; Smith and Rinrtsn, 1995). A novl approach usd by Mo and Fadr (2002) utilizs advanc purchas ordrs mad via th CDNOW wbsit to gnrat arly indicators of nw product sals for music CDs. In advrtising, th fficacy of an advrtising campaign is assssd using a battry of tsts dsignd to idntify th bst crativ to us (.g., th most prsuasiv or th most mmorabl) using slctd mmbrs of th targt audinc. Fild xprimnts with split-cabl tlvision tchnology hav also bn usd to study th impact of advrtising on brand sals (Lodish t al 1995, Blair 1988). In dirct markting, modling tchniqus hav bn dvlopd to hlp marktrs slct th right customr group for a givn contnt (Gönül and Shi 1998, Bult and Wansbk 1995; Bitran and Mondschin 1996, Gönül, Kim, and Shi 2000, Gönül and Hofstd 2006). Bult and Wansbk (1995) build a rgrssion modl to prdict ach customr s liklihood of rsponding to a dirct markting communication, and thn slct which customrs should b contactd by xplicitly maximizing th xpctd profits gnratd by ach communication. Using a dynamic programming approach rathr than using rgrssion, Bitran and Mondschin (1996) tak th 4

6 profit maximization a stp furthr by incorporating invntory policis (invntory and out of stock costs) into th dcision. Gönül and Shi (1998) xtnd this dynamic programming approach by allowing customrs to optimiz thir own purchass bhavior ovr multipl priods (i.., both th firm and th customr ar forward looking). In Gönül, Kim, and Shi (2000), th authors rcogniz that customrs can ordr from old catalogs and that on can still garnr nw sals from old customrs who wr not snt a nw catalog. Thus, thy propos a hazard function modl of purchas whr customrs ar snt a catalog only if th xpctd profits with th additional mailing xcds th profits without th mailing. Elsnr, Krafft, and Huchzrmir (2004) us Dynamic Multilvl Modling (DMLM) to optimiz customr sgmntation and communication frquncy simultanously. In practic, fw dirct markting campaigns ar rolld-out untstd. Th traditional approach (Nash 2000) is to us th doubling mthod to prdict th ultimat rspons rat to a dirct markting offr. In th doubling mthod, on uss past campaign to stimat th tim ndd for half of th rsponss to b rcivd (th doubling tim). Thn, whn prforming a tst, on waits th doubling tim, and multiplis by two th numbr of rsponss rcivd at that tim to stimat th ultimat rspons rat. Th waiting priod dpnds on th mdium usd. For first class mailing, firms will wait a minimum of two wks; for third class mailing, thy will wait four wks. Onc th tsts rsults ar in th dcision makr nds to mak a go/no-go dcision basd on profitability. Morwitz and Schmittlin (1998) show that whn making such projctions, managrs typically do not sufficintly rgrss tst rspons rats to th man. Tsting is also popular in Intrnt markting applications. Onlin advrtisrs track bannr ad prformanc in ral tim to idntify th appal (click-through) of various advrtising crativ. Click-stram modls can b implmntd to tst th appal of contnt by masuring th 5

7 click-through rats or wbsit stickinss (Bucklin and Sismiro 2003). Ey tracking tchnology may b usd to idntify whr (and if) a customr is viwing th advrtising mssag mbddd on a wbpag (Drèz and Husshrr 2003). Whil som of ths tsting mthodologis might b adaptd to th contxt of mail markting, som uniqu faturs of mail prsnts svral nw modling challngs. First, many firms hav implmntd tracking tchnologis for mail campaigns that can accuratly masur (to th scond) whn a customr rsponds to th mail. Givn th goal of ral-tim tsting, it is ssntial that w mak full us of this continuous tim data. Th data tlls us both whthr and whn a customr rsponds to an mail, and w nd our mthodology to mak us of this information prtaining to campaign lvl succss. Scond, in contrast to a typical clickstram stting, mail communications ar initiatd by th firm rathr than th customr. This adds a layr of complxity in that, whil th dlivry of an mail is oftn clos to bing instantanous, thr is an obsrvabl dlay btwn th tim th mail is snt out and th tim it is opnd. Th opning of th mssag will dpnd on how oftn customrs chck thir mail. Thus, although th marktr has dirct control ovr whn th mail is snt out, thr is littl control ovr whthr and whn a customr rsponds to th mail. This is diffrnt from traditional clickstram modls whr th usr rqusts th contnt, and w can assum that th contnt is bing procssd immdiatly. A third diffrnc in mail markting involvs th lad tim for gnrating both th crativ and th xcution of th campaign. Evn a larg mail campaign can b snt out at rlativly low cost and dlivrd in a mattr of hours, consquntly campaigns ar short livd and oftn run with short lad tims and consquntly with comprssd dadlins such as wkly (.g., Amrican Airlins, Travlocity, Th Tir Rack), bi-wkly (.g., Longs Drugstor on th 6

8 Wst Coast), or vn daily basis (.g., Sun Microsystms). Ths short lad tims plac significant constraints on tsting, simply bcaus answrs to a tst ar typically ndd in just hours to b usful. For ths rasons, ffctiv mail markting communication rquirs a tsting mthodology that is abl to gnrat actionabl rsults in as short a tim as possibl. Th goal of such a tsting procdur is to gnrat prdictions of opn incidnc and click-through rats of any mail campaign as quickly and accuratly as possibl. Our papr dscribs th dvlopmnt and tst of an mail pr-tsting modl. W bgin by dvloping a split-hazard log-logistic modl of opn bhavior. Th split-hazard componnt modls th incidnc of opn (vrsus not opn); a Log-Logistic hazard rat is usd to prdict th distribution of opn tims. Click bhavior is thn modld using both a cnsord split hazard modl and a simplr Binomial modl. To hlp us produc stabl stimats vn whn data is spars (a common occurrnc whn trying to tst campaigns in a short amount of tim) w us Baysian shrinkag stimation. This allows us to tak advantag of th information containd in past campaigns and wight this past information with rspons data obsrvd in th focal campaign. Both sts of modls ar compard with th doubling mthod usd by dirct markting practitionrs. In our application of th modls to actual data, w find it ncssary to account for intraday variations in customr rsponss (.g., to account for fwr mails opnd at night). Consquntly, w dvlop a concpt of virtual tim that allows us to produc a modl that fits th data wll whil kping th spcification simpl. Virtual tim involvs adjusting th spd of tim to adapt to th marktrs and customrs availability throughout th day. Using virtual tim allows us to kp th modl spcification simpl. This maks shrinkag straightforward and allows for an asy intrprtation of th modl paramtrs. 7

9 W apply th tsting procdur to data obtaind from a larg ntrtainmnt company. Th analysis shows that using our approach, w can rduc tsting tim from 14 hours to lss than two without any dcras in tsting rliability. It also highlights th pitfalls inhrnt with comprssing tsting tim. Indd, th mor comprssd th tsting tim, th mor snsitiv th quality of th rsults ar to th spcifications of th hazard function and to intra-day sasonality. Our modl provids a numbr of substantial practical bnfits. (1) Th fast valuation of a campaign allows for arly warnings about th probabl succss or failur of th campaign. This can lad to timly go/no-go dcisions for ithr campaigns or crativ. (2) Our modl provids diagnostic information that can hlp improv th rsults of an undr-prforming campaign. A simpl dcision modl could discard any campaign that dos not prform abov som thrshold lvl of rspons. (3) Our tsting procdur coupld with such a dcision procss can gnrat highr avrag (cross-campaign) rspons rats (20% highr in our simulation). (4) An important additional advantag of our tsting procdur is that only a small sampl is rquird for tsting. Th small sampl siz maks it asy to tst th ffctivnss of multipl advrtising copis. Svral sub sampls can b gnratd, and a diffrnt crativ snt to ach sub sampl. (5) Our procss formalizs th us of th company s knowldg bas in modling futur campaigns. Indd, as th numbr of campaigns grows, th mail marktr larns mor about th distribution of rspons rats for campaigns snt. This lads to mor accurat forcasts. 1. Rsarch Stting and Data Dscription W will calibrat and tst our modls using a databas of twnty-fiv mail campaigns snt as part of th onlin nwslttr of an ntrtainmnt company. Most of th mails ar in th form of promotions aimd at inducing customrs to purchas a movi titl onlin or offlin, or to click on 8

10 links to accss furthr contnt (diffrnt campaigns hav diffrnt purposs). Each campaign has a subjct lin displaying th purpos of th promotion. Th main body of th mail is only visibl aftr th rcipint has opnd th mail. Within th body of th mail, rcipints ar abl to click on various links to activat th promotion or to dirct thm to a wbsit. It is important to not that clicks can only occur if a rcipint opns th mail. Summary statistics for th mail campaigns ar rportd in Tabl 1. Th campaigns vary in siz from around 5,000 to 85,000 mails snt. Our databas consists of 617,037 mails snt, of which 111,419 wr opnd, and 9,663 of thos mails wr clickd on at last onc. Th opn rat is thus 18.1% and th click-through rat is 8.7%. Th unconditional click rat (dfind as th numbr of clicks dividd by th numbr of mails snt) is about 1.6%. Thr is a wid rang in both opn and click-through rats across campaigns. Clarly, th mor succssful campaigns ar th ons that hav both high opn rats and high clickthrough rats. Using a mdian split on opn and click-through rats, w find that 20% of our campaigns fall in this uppr right quadrant (high opn and click-through rats). Som campaigns (32% of our data) njoy high opn rats but hav low click-through rats. This is probably an indication that ths campaigns hav broad appal but ar poorly xcutd. Th firm might b abl to mov ths campaigns to th uppr right quadrant by using bttr crativ. Anothr 28% of campaigns ar in th opposit quadrant; thy fail to attract nough attntion to b opnd, but gnrat high click-through givn thy wr opnd. In such cass, th xcution is good, but th bas appal is low; th firm might improv th campaign through bttr targting. Th rmaining campaigns (20%) hav both low opn and click-through rats; improving ths campaigns would rquir improving both th targting and th contnt. If this cannot b don, it may b bst to drop th campaign altogthr. 9

11 Givn th numbr of campaigns in ach quadrant, it is clar that it is difficult to prdict th succss of a campaign x ant. Th goal of our modl is thus to prdict out-of-sampl opn and click rats quickly and with small sampls. Providing a forcast in a timly mannr allows th firm to adjust th crativ or targting of th campaign whn ndd, thrby improving ovrall rsults. Figurs 1a and 1b prsnt histograms of th tim (in hours) for customrs to opn th mail sinc it was snt, and th tim (in minuts) it taks th customr to click on th mail sinc it was opnd. Givn our objctiv of rducing th tim allocatd to tsting, svral faturs of our data ar highly prtinnt to modl construction: 1. Opns usually occur within 24 hours of snding; clicks occur within a minut of bing opnd. 2. Thr is a rlativly low lvl of mail activity during th first fw hours aftr an mail campaign is snt (s th first fw hours of Figur 1a), followd by a build up. 3. Th histogram of th dlay btwn snd and opn (Figur 1a) rvals a distinct multi-modal pattrn undrlying particularly during th first 24 hours aftr an mail is snt. This pattrn is also visibl on individual campaign histograms. Th first fatur rquirs that a rapid tsting modl of opn and click rat work wll with cnsord data. Indd, by shortning th tsting tim, w rduc th amount of uncnsord data availabl to us. Th scond fatur suggsts that w must b carful about our assumptions rgarding th data gnration procss whn building our modl. This is particularly important in our cas as w ar trying to mak prdictions about th ntir distribution of mail activity basd on only a fw hours of activity. 10

12 Th multimodal pattrn found in th tim until opning is troublsom as it dos not appar to conform to any standard distribution and might b difficult to captur with a simpl modl. To undrstand what may b driving this multi-modal pattrn, w plot th distribution of opns throughout th day (s Figur 2). This graph shows considrabl variation in activity through th day. Thr ar fwr mails opnd lat at night and arly in th morning than during th day. W rfr to this pattrn as intraday sasonality. W show in th nxt sction how this sasonality is th caus of th multimodal fatur of Figur 1a. Givn ths rsults, w will accommodat th following faturs in our modling framwork. To gnrat stimats within th first fw hours aftr snding, w will hav to work with cnsord data and only a small amount of data will b availabl for stimation. W also nd to tak into account intraday sasonality to allow parsimonious paramtric approach to modl th numbr of opns. In th nxt sction, w dvlop a mthodology that accommodats ths issus. 2. Modl Stup W dvlop a rapid tsting mthodology for a spcific application: th tsting of onlin mail campaigns. Th rapid tsting mthodology rquirs a modl that provids arly fdback on whthr a campaign is likly to b succssful or not. In th spirit of traditional tsting modls, it is important that our mthodology consums as fw rsourcs as possibl. Idally, th modl would also b parsimonious (i.., hav fw paramtrs), and would stimat quickly such that a tst can b implmntd in ral tim and would allow for th monitoring of an mail campaign as it is bing snt. Indd, an ovrly complx or ovr-paramtrizd modl that taks hours to gnrat prdictions would dfat th purpos of rapid tsting. 11

13 W first dscrib in mor dtail how th modl accommodats intra-day sasonality. Nxt, w dvlop a split hazard modl of opn and click probabilitis that taks into account th possibility that som mails ar nvr opnd or clickd on (givn opn). W thn driv th shrinkag stimators for th opn and click modls and stat th liklihood function usd in th stimation From physical tim to virtual tim A traditional approach to handling sasonality, such as that displayd in Figurs 1 and 2, is to introduc tim-varying covariats in th modl. Thr ar two main problms with this approach. First, th covariats ar oftn ad-hoc (.g., hourly dummis). Scond, thy oftn mak th othr paramtrs lss intrprtabl (.g., a low opn rat during pak hour could b largr than a high opn rat during off-pak hours). To allviat ths concrns, w build on th approach dvlopd by Radas and Shugan (1998). Radas and Shugan (hraftr RS) d-sasonalizd a procss by changing th spd at which tim flows. Thy showd that by spding up tim during high sasons, and slowing down tim during low sasons, on can crat a nw (virtual) tim sris that is dvoid of sasonality. Th bnfits of this approach, assuming that on has th right sasonality pattrn, is that on can us straightforward modls in virtual tim and asily intrprt th maning of th paramtrs of ths modls. Th ffctivnss of th RS approach hings on having a good handl on th sasonal pattrn prsnt in th data. In thir application (th movi industry) thy produc sasonal adjustmnts by combining past sals data with industry knowldg (.g., prsnc of major holidays with high movi dmand). A shortcoming of this approach is that som of th sasonality may b ndognous to th firms dcisions. For instanc, if movi studios bliv that Thanksgiving wknd is a big wknd, thy may always choos to rlas thir bst 12

14 movis during that wknd (Ainsli, Drèz, and Zufrydn 2005). Thus, part of th sasonality obsrvd during Thanksgiving will b du to th fact that mor consumrs hav th tim and dsir to s movis on that wknd (consumr inducd sasonality) and part of th sasonality will b du to th fact that firms rlas thir biggr movis on that wknd (firm inducd sasonality). If on uss past data as a bas for sasonal adjustmnt without considring th dcisions of th firm, on can potntially ovrcorrct and attribut all th sasonal ffcts to consumr dmand whil it is in fact also partly du to firm supply. In our cas, w also hav a potntial for both consumr- and firm-inducd sasonality. For instanc, th avrag consumr is much lss likly to opn mails at four in th morning than at four in th aftrnoon. Similarly, firms do not work 24 hours a day. If w look at whn th firm snds its mail (Figur 3), w obsrv littl (but som) activity during th night, thn a pak at ight in th morning, a pak at noon, an a lot of activity in th aftrnoon. It is likly that ths paks ar rsponsibl for som of th incras in activity w s in Figur 1 at similar tims. To sparat consumr inducd sasonality from firm inducd sasonality, w bnfit from two faturs of our modling nvironmnt not prsnt in RS. First, w hav continuous tim individual lvl data. Whil RS had to work with aggrgat wkly masurs, w know th xact tim ach mail is snt and opnd. Scond, whil a movi can opn on th sam day throughout th country, mails cannot all b snt at th sam tim. s ar snt squntially; for xampl, a million-mail campaign can tak 20 hours to snd. Thus, w can simulat an nvironmnt that is dvoid of firm basd sasonality by r-sampling our data such that th numbr of mails snt at any point in tim is constant through th say (i.., Figur 3 for such a firm would b flat). 13

15 To rsampl th data, w procd in thr stps. First, for ach minut of th day, w collct all mails that wr snt during that minut. Scond, w randomly slct with rplacmnt 100 mails from ach minut of th day (144,000 draws). Third, w ordr th opn tims of ths 144,000 mails from 0:00:00 to 23:59:59 and associat with ach actual opn tim a virtual tim qual to its rank dividd by 144,000. Th rlationship btwn ral and virtual tim basd on thir cumulativ dnsity functions is shown in Figur 4. This rprsnts th passing of tim as sn by consumrs indpndnt of th actions of th firm. W can us th rlationship dpictd in Figur 3 to comput th lapsd virtual tim btwn any two vnts. For instanc, if an mail wr snt at midnight and opnd at two in th morning, w would comput th lapsd virtual tim btwn snd an opn by taking th diffrnc btwn th virtual quivalnt of two a.m. (i.., 00:29:44 virtual) and midnight (i.., 00:00:00 virtual) to com up with 29 minuts and 44 sconds. Similarly, if th mail had bn snt at noon and opnd at two p.m., thn th lapsd virtual tim would b 11:05:10 09:08:30 = 1 hour 56 minuts and 40 sconds. Applying th virtual tim transformation to th lapsd tim btwn snd and opn for all mails in our datast rsults in th histogram shown in Figur 4. Comparing this histogram to Figur 1, w can s th ffct of using a virtual tim transformation. Th undrlying sasonal pattrn has all but disappard. What was a multimodal distribution is now unimodal A split-hazard modl of opn and click tim Th tim it taks for customrs to opn an mail from th tim it is snt, or th tim it taks to click on an mail from th tim th customr opns it ar both modld using a standard duration modl (.g., Mo and Fadr 2002, Jain and Vilcassim 1991). Sinc both actions can b modld using a similar spcification, w discuss thm intr-changably. Starting with opns, w account 14

16 for th fact that in an acclratd tst, a failur to opn an mail is indicativ of on of two things. Eithr rcipints ar not intrstd in th mail, or thy hav not had a chanc to s it yt (i.., th data is cnsord). Of cours, th shortr th amount of tim allocatd to a tst, th highr th liklihood that a non-rspons is indicativ of cnsoring rathr than lack of intrst. To account for this bias, w modl th opn probability and th opn tim simultanously in a right-cnsord split hazard modl (similar to Kamakura, Kossar, and Wdl (2004) and Sinha and Chandrashkaran (1992)). Th probability that a customr will opn or click an mail varis from campaign to campaign, and is dnotd withδ, whr is a subscript idntifying diffrnt campaigns, and k is a suprscript dnoting an opn ( δ ) or click ( δ ). k o c Th liklihood function is constructd as follows. W start with a basic cnsord hazard rat modl of th opn tim distribution: N k ( ) ( ) 1 k k k k Ri k Ri = i Θ i Θ, (1) i= 1 L f t S T whr: is a subscript that idntifis a spcific mail campaign, k is a suprscript that idntifis th modl usd ( k { o= opn, c= click} ), i is an indx of rcipints, N is th numbr of rcipints for mail, k R i is 1 if rcipint i opnd/clickd mail bfor th cnsoring point T, T i is th cnsoring point of mail i of campaign, k t i is th lapsd tim btwn snd and opn (opn and click) in th vnt that th rcipints opnd (clickd) th mail, 15

17 f( t Θ ) is th pdf for tim t, givn a st of paramtrs Θ, St ( Θ ) is th corrsponding survival function. W adjust th hazard rat to account for th fact that som rcipints will nvr opn th mail. If w call δ th liklihood that mail will b opnd ( δ for clicks), w hav: o c N k R k k k k k k k i k k k ( Θ ) = δ Θ i δ i Θ + δ i= 1 L t, T, R f( t ) S( T ) (1 ) N i= 1 k R k (1 R ) i k (1 R ) i k k k i k k = δ f( t Θ ) 1 (1 ( )). i δ S T i Θ (2) k k Th stimation of δ and Θ for any paramtric hazard function can b prformd by maximizing this gnral liklihood function Shrinkag stimators As in most practical applications, w bnfit by having data availabl from past campaigns and w can us this information to improv th prformanc of our modl. Spcifically, w can us paramtrs from past campaigns to build a prior on th opn and click hazard functions, as wll as th split hazard componnt. This is spcially usful at th bginning of a campaign whn data is spars. Th implmntation of th shrinkag stimator dpnds on th spcific hazard functions usd in th modl. W thrfor postpon our discussion of th shrinkag stimators until th mpirical sction of th papr whr w valuat diffrnt possibl hazard functions An Altrnativ Approach to Estimating Click Rats Although thortically sound, using a split-hazard modl to stimat th paramtrs of th click tims (conditional on an opn) might b ovrly complx. Indd, sinc most consumrs click on an mail within sconds of opning it, it is likly that fw click obsrvations ar right-cnsord. In our sampl, ovr 95 prcnt of all mails opnd bfor th doubling point ar also clickd on 16

18 bfor th doubling point. Thrfor, w dvlop and tst an altrnativ modl for stimating click rats. W us a traditional binomial procss with a Bta distributd prior in an mpirical Bays framwork. Our hop is that a mor parsimonious modl will prform bttr at th bginning of a tst, whn fw data points ar availabl. Formally, w us th man of th postrior of th Bta distribution as th stimat for as follows. Lt c b th numbr of clicks, o b th numbr of opns, and Bta( υ, ω ) b our prior on th distribution of th click rat (built using prior campaigns). Thn th postrior distribution of th conditional click liklihood is distributd Bta( υ + c, ω + o c). Th stimat for th postrior man is: c δ δ ˆc υ + c = (3) υ + ω + o As bfor, stimats of th prior distribution paramtrs ( υ, ω ) ar gnratd from availabl past campaigns. Th valus for opn (o) and click (c) ar gnratd from th data at th tim th tst is conductd. W rfr to this click modl as th Empirical Bays Binomial Distribution (EBB) Th Doubling Mthod Bfor discussing an application of our modl, w would lik to draw a comparison with xisting approachs to prdicting th succss rat of dirct markting campaigns. Th most common modl usd by practitionrs is th doubling mthod (Nash 2000). This mthod involvs first xamining th rsponss of past dirct markting campaigns and computing th amount of tim it taks for 50% of th rsponss to b rcivd (th doubling tim). Th analyst thn uss th huristic that for any futur campaigns, th prdictd total numbr of rsponss is qual to doubl th numbr of rsponss obsrvd at th doubling tim. In our cas, th doubling tim is 14 hours, ranging from 4 to 29 hours (s Tabl 1). 17

19 Th doubling mthod is a powrful and simpl huristic. It maks thr implicit assumptions. First, it assums that not vrybody will rspond. Scond, it assums that it taks tim for popl to rspond. Third, it assums that th timing of th rsponss is indpndnt from th rat of rspons and approximatly constant across campaigns. As a non-paramtric mthod, it dos not mak any assumption about th undrlying rspons procss, nor dos it provid any ways to tst whthr th currnt campaign conforms to th data collctd from prvious campaigns or runs fastr or slowr than xpctd. Hnc, it dos not provid any ways to valuat whthr a currnt tst should b run for a longr priod or could b finishd arly; an important pic of information our tchniqu provids. In ssnc, th doubling mthod aggrgats tim into two bins; ach containing half of th rsponss. This aggrgation loss vital timing information that could b usd to bttr modl th rspons procss. 3. Application of th modl to mail campaign pr-tsting W now apply our modls to th data from th mail campaigns dscribd arlir and valuat th rlativ prdictiv validity of various modls. Our application involvs two main phass, a calibration and modl spcification phas and an stimation and validation stag. W compar th prdictions of our modls with thos of th doubling huristic oftn usd in dirct markting applications. Calibration and Modl Spcification: In th calibration phas, w ar intrstd in larning which of th paramtric functional forms w should us to driv our prdictiv modls. To implmnt this w compar a st of commonly usd hazard rat distributions including Wibull/Exponntial, Log-Logistic and Log-Normal. Th functional form that fits th data bst 18

20 is usd in th simulation and calibration phas. Th spcification chosn for th hazard function also drivs th spcification rquird for shrinkag stimation. Simulation and validation: th main purpos of th simulation and validation stags is to validat th modls proposd in th papr by studying th accuracy of th prdictions thy mak out-ofsampl. W also want to find out which of th modls has th bst prdictiv prformanc and whthr w can gnrat stimats that ar usful for dcision making within a short amount of tim (say hours) such that tsting is fasibl for ral-tim campaign planning. Givn th bst spcification for th hazard rat found in th calibration phas, svral modls ar compard for both th opn and th click procsss. W compar th modls basd on ral tim (no tim transformation) vrsus virtual tim, and basd on shrinkag vrsus noshrinkag stimation. In summary, w fit and validat ach of th following modls for both th opn and click procsss in th simulation and validation phass: 1) No-shrinkag stimation with ral tim 2) No-shrinkag stimation with virtual tim 3) Shrinkag stimation with ral tim 4) Shrinkag stimation with virtual tim Th EBB modl is tstd only for th clicks and rprsnts our fifth spcification tstd: 5) Empirical Bays Binomial modl for clicks only Thr is no nd to tst th virtual tim transformation for th EBB modl bcaus this spcification uss only a count of th numbr of clicks rlativ to th numbr of opns it is thrfor not basd on th amount of tim it taks for customrs to click on th mail givn it was opnd. 19

21 3.1. Calibration and Estimation Shap of th Hazard Function Rsarchrs in markting hav mployd svral spcifications for th hazard function whn doing survival analysis s Jain and Vilcassim 1991, Sawhny and Eliashbrg 1996, Chintagunta and Haldar 1998, Drèz and Zufrydn 1998). Following thir work, w considrd th following four spcifications: Exponntial, Wibull, Log-Normal, and Log-Logistic. This st of distribution ncompasss a wid rang of consumr bhavior. Our final choic of hazard function is basd on how wll it agrs with th data (goodnss of fit). W stimatd a campaign lvl hazard rat modl for ach distribution using th complt st of opns and clicks availabl for ach campaign (i.., this is a straight hazard modl that is nithr split nor cnsord). W rport th fit statistics for all four spcifications in Tabl 2a (opn modl) and 2b (click modl). Th analysis suggsts that th Log-Logistic distribution fits th data bst ovrall for both opn and click. Th Log-Normal is a clos scond, but has th drawback of not having a closd form xprssion for its survivor function. It is important to not that th Exponntial distribution prforms rlativly poorly, mphasizing th nd for a nonconstant hazard rat that allows for a dlay btwn rcption and opn of an mail, or btwn opn and click (i.., allows for nough tim for consumrs to procss th mssag). Th rlativly poor fit of th Wibull distribution (which allows for a ramping up priod) furthr shows that on also nds to accommodat for a dcras in th hazard rat aftr nough tim has passd. Making th right assumptions rgarding th chang in hazard rat ovr tim is thus crucial. This is spcially tru sinc much of th data availabl during th tst will com from th first fw hours of th tst, rprsnting th incrasing part of th Log-Logistic hazard function. 20

22 Estimating this basd on a Wibull or Exponntial hazard function would clarly misspcify th modl. Th probability dnsity function and th survivor function for th Log-Logistic ar (s Kalbflisch and Prntic (1985) for dtails about th Log-Logistic and othr distributions mntiond in this papr): f( t αλ, ) = λα( λt) α 1 α ( 1 + ( λt) ) 2, (4) 1 St ( αλ, ) = α 1 +( λt) whr λ > 0 is a location paramtr and α > 0 is a shap paramtr. Consistnt with prvious notation, w rfr to th shap and location paramtrs for any givn campaign () and mail rspons action ( k { o, c} ) as k α, and k λ, rspctivly. Dpnding on th valu of α, th Log- Logistic hazard is ithr monotonically dcrasing ( α 1) or invrtd U-shap ( α > 1) with a turning point at ( α 1) 1/ α t =. λ Shrinkag Mthodology Sinc w us a Log-Logistic hazard function, our split-hazard modls hav thr paramtrs ( α, λδ, ). W build informativ priors for ( α, λδ, ) using th stimats obtaind from othr campaigns. Basd on an inspction of th mpirical distribution of ths paramtrs, w spcify our prior forδ as a Bta distribution and α and λ ar as a bivariat Log-Normal distribution: δ ~ Bta( a, b ) δ ( a, λ) ~ Log-Normal( μ, Σ) δ (5) 21

23 Th paramtrs ( aδ, bδ, μ, Σ ) for a givn campaigns ar stimatd using th mthod of momnts from th paramtrs ( α, λδ, ) obtaind from all othr campaigns. To comput th corrlation btwn paramtrs α and λ of th Log-Normal distribution w us th corrction factor dscribd in Johnson and Kotz (1972, pag 20) to adjust for possibl small sampl bias. Th corrlation for th Log-Normal is: ρ xp( ρ σ σ ) 1 N LN αλ, α λ αλ, = 2 2 { ( σα) } ( σλ) { } xp 1 xp 1 (6) N whr ρ α, λ is th corrlation cofficint for th Normal distribution of th two paramtrs. Not that this corrlation cofficint is indpndnt of th mans of th paramtrs, dpnding only on th standard dviations of th paramtrs (rspctivly, σ, σ ) Simulation and validation Split-Hazard Modl With th hazard function proprly dfind and th shrinkag mthodology in plac, w ar now rady to fit and validat our modls. Th final liklihood functions ar as follows: 1) Liklihood function for th Non-Shrinkag Modl: L k R i N α 1 k k k k λα( λti) k 1 ( t T R Θ ) = δ δ 2 i= 1 +( λti α λ k (1 R ) i,, 1 (1 ) α (7) α ( 1 + ( λt ) ) 1 ) i 2) Liklihood function for th Shrinkag Modl (,, ) LN ( μ, ) Bta ( δ, δ ) L t T R Θ = Σ a b k k k N i= 1 k R i α 1 ( ti) k 1 δ 2 λti k λα λ δ 1 (1 ) α α ( 1 ( λt ) ) 1 +( ) + i k (1 R ) i (8) 22

24 whr Σ is th varianc-covarianc matrix for th Log-Normal prior distribution. Th EBB stimator is computd dirctly from (3) without rsorting to its liklihood function. In ach simulation, w adopt th prspctiv of a marktr who wishs to prtst his campaigns bfor committing to th final snd. To this nd, w look at ach campaign assuming that th rmaining (E - 1) campaigns hav alrady bn compltd. Thus, prior to th tst, w know nothing about a focal campaign xcpt th information containd in th priors, and th numbr of mails that nd to b snt out. Basd on company policy, w st our sampl tst siz at 2,000 mails (w varid th tst siz btwn 1,000 and 2,000 in incrmnts of 200 mails but did not find any substantiv diffrnc in rsults). W also st diffrnt cnsor points, in 30 minut incrmnts, ranging from 30 minuts to 8 hours. At ach cnsor point, any mail that had bn opnd prior to th cnsor point was usd in th non-cnsord componnt of th logliklihood. All obsrvations byond th cnsor point (rgardlss of whthr thy wr ultimatly opnd or not) wr codd as cnsord. Basd on this st of cnsord and uncnsord obsrvations, w thn stimatd th paramtrs of th split-hazard cnsord Log-Logistic rat modl using priors constructd basd on all othr campaigns. As a practical mattr, th distributions of opn and click xprinc long tails, such that som rsponss continu to com in long aftr a campaign has run its cours. Our data rvals that 99% of all mail rsponss ar obsrvd within 3 wks of snding out th mail communication. Typically, th company conducts post-campaign dbrifing 2-3 wks aftr th mails ar snt out. Thus, w st a cut-off dat of 3 wks (504 hours) and us th numbrs of opns and clicks obsrvd at that tim as th tru valu w ar trying to prdict. Our forcast of th numbr of opns and clicks at 3 wks is constructd using th paramtr stimats for ach 23

25 of th cnsord sampls. Th cumulativ distribution of opns and clicks at 504 hours is calculatd using: ( 504 hours ˆ, ˆ ) ( Numbr Snt ) ˆ δ o F α o λ o for th numbr of opns at 3 wks, and ( 504 hours ˆ, ˆ ) ( Estimatd opns ) ˆ δ c F α c λ c for th stimatd numbr of clicks. Rsults for th opn modls Th full rsults for ach campaign and for ach cnsoring point consists of a st of paramtrs o o o c c c ( δ, α, λ ) for opns and ( δ, α, λ ) for clicks. Sinc for ach modl w hav 16 tim points (btwn 30 minuts and 8 hours) and 25 campaigns, this mans our analysis gnrats a total of 6x16x30 = 2,880 stimats. Givn this larg numbr of stimats it is difficult to prsnt all th rsults in on tabl. It is also not that maningful to prsnt any sufficint statistic of ths stimats sinc thy ar cnsord at a diffrnt point in tim. W thrfor summariz our rsults by looking only at th stimatd opn and click rat o c rspons rats for ach campaign ( δ, δ ). Our summary includs a comparison of th prdictions basd on ach st of stimats and at any givn tim point with thir tru (postcampaign) valus. Th summary also includs th corrsponding avrag dviations of th prdictions from thir tru valus. Figur 6 graphically prsnts this summary of th Man Absolut Dviation (MAD) for th prdictd numbr of opns across campaigns and in half hour incrmnts. Th vrtical axis rprsnts th MAD, across campaigns, or th avrag absolut diffrnc btwn th prdictd numbr of opns from th modl and th actual numbr of opns obsrvd in th data. Th horizontal axis rprsnts th half hour incrmnts, starting at 30 minuts. Th prdiction from th 14 hour doubling point is also plottd on th graph, th MAD is indicatd by th hight of th 24

26 vrtical lin plottd at six hours. It is plottd at six hours bcaus that is th tim at which th bst prforming modl yilds an improvmnt ovr th doubling tim modl. W obsrv a gnral downward trnd for all modls as th cnsoring point movs to th right and a largr proportion of th total sampl is availabl for calibration vrsus prdiction. Th rsults show a clar dominanc in prdictiv prformanc of th shrinkag modls ovr th non-shrinkag modls. Th MAD valus for th shrinkag modls ar about a quartr of th MAD valus for th non-shrinkag modls. Basd on prdiction rror, th virtual tim/shrinkag modl tnds to outprform all othr modls. Furthr, it achivs th sam lvl of prdiction rror as th doubling mthod in as littl as six hours compard to th 14 hours of th doubling mthod. To gt a bttr sns of th bnfits and drawbacks of ach modl, w plot th prdictd numbr of opns for an illustrativ campaign for ach modl (Figur 7). Th solid lins in th middl of ach graph in Figur 7 rprsnt th stimatd numbr of opns at 504 hours, basd on th cnsor point listd on th horizontal axis. Th small dottd lins tracking abov and blow ach of th solid lins rprsnt th confidnc intrvals. Th constant dashd lin in th middl is th tru valu for th numbr of opns at 504 hours. W find that, for this campaign, th nonshrinkag modls tnd to prform quit badly compard with th shrinkag modls. W s also that thy hav som difficulty convrging spcially within only a fw hours of snding th mails. A comparison across campaigns for ach of ths modls rvals svral bnfits of th shrinkag ovr th non-shrinkag modls. In short w find that non-shrinkag modls xhibit thr typs of problms. First, thy oftn fail to convrg during th first coupl of hours of 25

27 tsting. Scond, whn thy do convrg, thy oftn produc confidnc intrvals that ar too tight. Third, thy traditionally undrstimat rspons rats during th first day of tsting. Shrinkag allviats ths problms, both in ral tim and in virtual tim. Modls convrg quickly and rliably vn with fw data points. Th confidnc intrvals producd ar ralistic. Th bnfits of th virtual tim modl ovr th ral tim modl is that virtual tim producs tightr and mor stabl stimats and producs th stimat fastr. Thus th virtual tim modl is wll suitd to tst th prformanc of a campaign. Rsults for th click modls Whn prdicting click activity in our simulations, w procd in two stps. First, w fit our modls of click-through rat (using quations (7), (8) or (3) as appropriat). W produc stimats for ˆc δ (fiv stimats total, four for th log-logistic hazard rat modls plus on for th EBB modl). Scond, w prdict th ultimat numbrs of click for ach modl by multiplying δ with th numbr of opns ( ˆo δ ) forcast obtaind from th virtual tim/shrinkag modl of ˆc opns at th sam point in tim. Th virtual tim/shrinkag modl for opn was usd as a basis for this prdiction bcaus this is th on that prformd bst in th opn modl tst and thrfor rprsnts th most ralistic comparison. W compar th stimatd clicks with th actual clicks and comput th MAD of ach modl. Figur 8 rports MAD in 30-minut incrmnts across all 25 campaigns for th four basic modls and for th EBB modl. Th pictur dpictd hr is similar to th cas of th opn prdiction. Th major diffrnc is that th EBB modl rvals itslf to b th bst prforming modl. Indd, by combining th EBB modl with th virtual tim opn modl, w can achiv a prformanc similar to th doubling mthod in only thr and a half hours a 75% rduction in tsting tim. 26

28 3.3. A Campaign Slction Dcision Rul Th prcding two sctions show that our modls can produc prdictions of th opn and click rats fastr and mor accuratly than th doubling mthod. Whil th rsults do show a clar improvmnt in spd and accuracy, a natural qustion that ariss is: ar th bnfits of th nw modls substantial from a managrial standpoint? To numrat ths bnfits in th contxt of our application, w tak th prspctiv of a campaign managr who uss a simpl huristic to liminat any undr-prforming campaigns. Lt us say that th managr only wants to run campaigns that will produc an unconditional click-through rat (CTR) of 2% or mor. On possibl rason for not wanting to snd a low yild campaign is that it costs mor to snd than its xpctd rturns. Such undrprforming campaigns also rprsnt an opportunity cost in that thy ti up rsourcs that could b usd to snd mor profitabl campaigns. Furthrmor, snding undsirabl matrial could lad to highr customr attrition. W considr thr diffrnt dcision ruls and compar th aggrgat prformanc of th accptd campaigns undr ach dcision rul. First, w us th doubling mthod, whr w wait for 14 hours and slct any campaign with a prdictd CTR of 2% or mor. Scond, w us our bst prforming modl and slct any campaign with a prdictd CTR of 2% or mor aftr thr and a half hours of virtual tim rgardlss of th confidnc intrval around th stimat. Third, w us our bst prforming modl and run th tst until th prdictd CTR is significantly diffrnt than 2% at p =.05. Using this third rul, th tim ndd to tst a campaign will vary dpnding on th obsrvd data. Th rsults of this tst ar shown in Tabl 3. As xpctd, our modl prforms bttr than th doubling mthod. Th avrag campaign rspons rat can b incrasd by 9% simply by 27

29 applying our modl and waiting for 3.5 hours of virtual tim (3.63% CTR vs. 3.34%). On can gain a furthr 10% (4.00% vs. 3.63%, for a total improvmnt of 20%) by running th tst to statistical significanc rathr than using a fixd tim stopping rul. On should not that whn using a statistical tst rathr than a rul of thumb to dcid whn to stop th tst, th tsting tim varis widly across campaigns. Nin of th 25 campaigns rach statistical significanc in as littl as an hour. In contrast, on campaign taks th bttr part of thr days (63 hours) bfor raching statistical significanc (it rachs 90% significanc lvl aftr 90 minuts) Whn Is th Bst Tim To Tst? Our comparison of th prdictiv ability for th split hazard rat modl dmonstrats that, on avrag, w can larn as much in thr and a half hours as w can larn from th doubling mthod in 14 hours. Howvr, it is important to rmmbr that ths thr and a half hours ar masurd in virtual tim. In ral tim, th tst will rquir mor or lss tim dpnding on whn it is prformd. Figur 9 shows how long thr and half virtual hours corrspond to in ral tim, dpnding on th tim of day whn th tst commncs. Thr appars to b a swt spot in th aftrnoon, btwn 1pm and 7pm whr a thr and half virtual hour tst can b carrid out in much lss than thr actual hours (th shortst it could tak would b 1:51 hours if it starts at 5:31 p.m.). Starting aftr 7pm will impos dlays as th tst is unlikly to b finishd bfor popl go to bd; if th tst is startd at 10:33 p.m. it will actually tak 7 and a half hours to complt. 4. Discussion and Conclusion Th valu of information incrass with its timlinss. Knowing quickly whthr a campaign is going to b succssful provids th opportunity to corrct potntial problms bfor it is too lat or vn stop th campaign bfor it is compltd. It is thrfor imprativ to dvlop mthods 28

30 that improv both th accuracy and th spd with which campaign tsting can b don. In this articl, w study a modling procdur that can b implmntd for th fast valuation of mail campaign prformanc. Th prformanc of an mail campaign is dfind by its opn and click rats. Th mthodology w propos prdicts ths rats quickly using a small sampl prtst. Rducing th sampl siz and tsting priod to a minimum producs multipl modling challngs. Indd, w propos to snd 2,000 mails, and wait lss than two hours to produc stimats of how th campaign will prform aftr thr wks. In two hours, w typically obsrv fwr than a hundrd opns and fwr than tn clicks. Th ky to succssful prdiction of th ultimat rsults of an mail campaign basd on so fw data points lis in using th information to its fullst potntial. Thr ar thr lmnts that mak our mthodology succssful: (1) using th appropriat modl spcification, (2) transforming tim to handl intra-day sasonality, and (3) using informativ prior information. Each of ths thr lmnts provids its own uniqu contribution to th ovrall fit of th modl. Th appropriat hazard function is critical bcaus our comprssd-tim tsts produc obsrvations that ar havily right cnsord. Thus, w ar oftn fitting a whol distribution basd only on its first quartil (or vn lss). A misspcification of th hazard function could caus svr rrors in prdiction. In othr words, th valu of th rsponss of th first fw popl to rspond to th mail campaign is an important indicator of th succss of th ovrall campaign. W find that th traditional xponntial hazard function usd in many modls of onlin bhavior is a poor fit for our procss. Our rsults provid strong vidnc to suggst that mail rspons rats (both opn and click-through) ar drivn by a non constant hazard rat. Rathr w s th 29

31 hazard rat riss in th arly phas of an mail campaign and thn dcrass as tim progrsss. W find that th bst fitting paramtric modl for opn tims is th Log-Logistic. For modling click-through rats, w compar th sam Log-Logistic hazard rat modl with a Binomial procss. W find that th straight binomial procss is a good dscriptor of th phnomnon givn that th quick consumr rspons aftr an mail is opnd limits cnsoring to th point whr it is not a factor. Thus, w find that th click-through rat (th total numbr of clicks for a campaign, unconditional on opn) is bst prdictd using a combination of th Bta binomial modl for th click rat, and th Log-Logistic split hazard modl for th opn rat. W apply our split-hazard modl to a virtual tim nvironmnt. Th virtual tim transformation rmovs intra-day sasonality and maks our tsting procdur invariant to th tim of day at which it is prformd. This is a ky factor in th robustnss of our modl in that it allows us to bypass th nd to handl sasonality dirctly in th modl and allows for a straightforward spcification with fw paramtrs. By limiting th numbrs of paramtrs w must stimat to thr for th opn modl and on for th click modl, w mak th bst us of our limitd data (w hav a high ratio of data points to paramtrs, or high dgrs of frdom) and w produc paramtrs that ar dirctly intrprtabl (th click and opn rats or stimat dirctly without th nd for transformation). Anothr bnfit of our tim transformation is that by making ach campaign indpndnt of th tim of day, w can compar rsults across campaign, and asily build informativ priors for ach of th paramtrs w nd to stimat. This yilds a procdur that producs maningful stimats and confidnc intrvals with a minimum amount of data. It also allows a firm to conduct tsts srially. That is, if thy chos to modify a campaign s crativ of targt population as th rsult of a tst, thy can rtst th campaign and compar th nw rsults to th first ons. 30