RESEARCH PAPER SERIES

Transcription

1 Research Paper No Dyamc Customzato of Marketg Messages Iteractve Meda Chrstopher G. Gooley James M. Latt RESEARCH PAPER SERIES GRADUATE SCHOOL OF BUSINESS STANFORD UNIVERSITY

2 Dyamc Customzato of Marketg Messages Iteractve Meda Chrstopher G. Gooley James M. Latt Graduate School of Busess Staford Uversty Staford, CA Aprl, 1998 Revsed, October 2000 The authors wsh to thak Dgtal Impact, Ic., for provdg the data used ths study.

3 "Dyamc Customzato of Marketg Messages Iteractve Meda" Abstract As a cosequece of sgfcat advaces formato techology, the marketg commuty has become creasgly terested the possbltes afforded by teractve meda. The exploso of the World Wde Web s the most otable example of such terest. Iteractve meda allow the marketer to 1) detfy the cosumer ad characterstcs of the cosumer, 2) decde o the marketg message real-tme, ad 3) capture respose to marketg commucatos. I cotrast to some tradtoal meda (e.g., televso, rado, prt), whch sed oe stadardzed message to all cosumers, teractve meda allow the marketer to delver customzed messages talored to the dvdual cosumer. I addto, ulke most other marketg evromets whch requre meda plag decsos to be made advace, teractve meda allow the marketer to make decsos o the fly, usg formato about prevous decsos to gude the curret decso. I other words, marketg decsos through teractve meda ca be truly dyamc. I ths paper, we formulate a uque procedure to explot the beefts of teractve meda. We study the geeral problem of a marketer whose obectve s to maxmze expected retur (e.g., respose rate) over the course of a drect respose marketg campag. The marketer has the ablty to dyamcally allocate two or more uque marketg messages (e.g., ads, Web pages) to acheve ths obectve. Cosumer respose to a partcular message may deped o a set of covarates (e.g., demographc characterstcs). I gorace of the true relatoshp betwee respose ad the covarates, the marketer ca use regresso techques to lear about the parameters of the respose fuctos. Moreover, because the marketer ca cotually update the parameter estmates, he ca also cotually adapt the decso of whch message to select. The theoretcal framework we draw from s the mult-armed badt problem statstcal decso theory whch there s a fudametal dlemma betwee formato, such as the eed to lear about the parameter values goverg respose for each ad, ad cotrol, such as the obectve of maxmzg respose rate. I such problems, t may be wse to sacrfce some potetal early payoff for the prospect of gag formato about cosumers that wll allow for more formed decsos later. A mportat dfferece our problem s that we corporate covarates (although our soluto ca hadle the stadard o-covarate case). Suppose that the marketer has two or more uque marketg messages avalable ad weak pror belefs as to the effectveess of each message. The marketer ca radomly assg the messages for a bref talzato perod to collect trag samples for each of the messages. From the o, the marketer ca estmate respose fuctos, whch ca be perodcally updated. I decdg whch message to select for ay partcular cosumer, the marketer ca compare the regresso estmates ad select the message wth the hghest predcted respose. We refer to ths approach as the myopc rule. However, sce the predctos are estmated wth mprecso, t may be worthwhle for the marketer to calculate ucertaty adustmets, ad corporate these adustmets ts message selecto decsos. The ucertaty adustmets reflect the mprecso the parameter estmates of the regresso models. The adustmets are larger for cosumers that have extreme characterstcs (covarates) tha for typcal cosumers because a extreme cosumer wll provde more formato about the ature of the respose fucto tha a typcal cosumer wll. Our proposed procedure s easy to mplemet, ad wll eable marketers to crease the success of ther marketg campags. I addto, the approach ca be used several dfferet marketg applcatos, cludg drect respose advertsg, Web page desg, ad product developmet.

4 1. Itroducto Image a hypothetcal catalog retaler dog busess the early 1990s. Every quarter, the retaler seds ts ew catalog to thousads of customers across the coutry, hghlghtg ew products as well as dscouts o curret ad dscotued merchadse. The oe day, a employee the mal-order departmet suggests that the compay move ts catalog operato ole. Wth ole techology, the employee explas, the compay ca dsplay ts catalog ad make t avalable to mllos of potetal cosumers a flash. The employee also clams that the cataloger ca customze ts catalogs, askg ew cosumers some basc questos (wth ther permsso) ad the dsplayg oly tems that are lkely to be of terest to those specfc dvduals. I addto, the cataloger ca cotue to teract wth cosumers by collectg the emal addresses of cosumers ad formg them of mportat developmets at practcally o cost to the cataloger. The boss lowers hs head to hs desk. That s brllat, he sghs. If oly I had a clue about what you are talkg about. I the year 2000, vrtually every drect marketer ow uderstads what the employee was talkg about. Iteractve meda, the most otable example beg the World Wde Web, have revolutozed marketg two mportat ways. Frst, teractve meda have sgfcatly lowered the barrers to etry ad have reduced the costs of sellg products to cosumers. I the catalog example, prtg, postage ad paper are elmated, ad replaced by a oe-tme producto setup cost ad small updatg ad revso costs. Secod, teractve meda allow the marketer to automatcally delver customzed messages to dfferet cosumers stead of oe stadardzed message to all cosumers. These customzed messages are delvered realtme, ad the decso of whch customzed message to preset to each cosumer ca be made a educated maer. Specfcally, because the marketer ca capture respose to ts marketg actos ad detfy the cosumer ad characterstcs of the cosumer, t has the ablty to model the relatoshp betwee respose ad cosumer characterstcs. The model, tur, ca be used to decde o the customzed message a partcular cosumer receves. The mplcato s that the marketer ca dyamcally adapt a marketg 1

5 campag, makg ad updatg decsos o the fly. I a sese, the meda plag process becomes seamless: t happes oe fell swoop. Oe example of a customzed message evromet occurs the cotext of ole retalg. We.com (formerly Vrtual Veyards) s a o-le merchat sellg we ad related accessores o the Iteret. Oe of the messages dsplayed o the We.com home page was Peter s Istat Pck (amed after co-fouder ad master sommeler Peter Graoff), a hypertext dsplay featurg a dfferet we recommedato each moth. If the vstor s terested, he/she ca clck through to ether purchase the recommeded we or get more formato about t. If the vstor s a regstered customer, We.com ca ot oly detfy the vstor whe he/she arrves at the home page (by meas of a "cooke" fle), but also have access to formato about behavor durg past vsts, cludg pages searched ad wes purchased. Thus, t makes sese for We.com to cosder delverg a dfferet "Istat Pck" to dfferet customers based o ther avalable formato. For example, Peter ca specfy that cosumers who prefer red wes (as reflected ther purchase hstores at the ste) are exposed to a stat pck featurg a caberet, whle cosumers who prefer whte wes are exposed to a stat pck featurg a chardoay. I summary, advaces formato techology have made t possble for marketers to commucate wth cosumers a more advatageous way, cosequetly rederg these ew meda more attractve to marketers. As a result, the marketg commuty has wtessed a paradgm shft from broadcast to teractve meda, ad ths tred from broadcast to teractve meda s acceleratg. Perhaps the most vsble sg of ths tred s the growth of marketg expedtures o ole meda, especally the World Wde Web. Jupter Commucatos estmates that U.S. ole advertsg reveue wll grow from $1.1 bllo 1997 to over $5 bllo The explosve growth of marketg o the Web ad other teractve meda s leadg to a creased terest amog marketers as to how to measure ad mprove marketg effectveess ths ew marketg evromet. The marketer s problem: whch message should I preset? I ths artcle, we formulate a uque procedure to crease cosumer respose whe the marketer s techologcally eabled to delver customzed marketg commucatos, ether through 2

6 teractve meda, or through tradtoal meda coupled wth a sophstcated database ad provso of a drect respose mechasm to cosumers (e.g., a toll-free umber). The prmary problem we study has the followg basc characterstcs. Over the course of a fte marketg campag, a marketer presets messages to cosumers sequetal fasho (.e., ot all at oce). The marketer has the potetal to preset two or more dstct marketg messages (e.g., baer ads, recommedatos, web cotet), ad ca choose to expose a cosumer to ay oe of these messages. Respose to a message (e.g., clckg o a baer ad, tme spet o a web page) ca be captured ad tracked by the marketer. Cosumer respose to a message may deped o a set of covarates (e.g., demographc varables). Ths relatoshp betwee respose ad the covarates may be dfferet for each message, ad the parameters of the respose fuctos are ukow to the marketer. Sad dfferetly, the marketer has weak prors as to the effcacy of each message. However, the marketer ca lear about the parameters of the respose fuctos. Sce the marketer ca cotually update the parameter estmates a teractve evromet, he/she ca also cotually adapt the decso of whch message to preset. The marketer s obectve s to maxmze total expected respose over the course of the marketg campag. The problem, formally stated Secto 2, s a varato o the classc multarmed badt problem. The mult-armed badt problem s prototypcal of a geeral class of adaptve cotrol ad desg problems whch there s a fudametal dlemma betwee formato, such as the eed to lear about the parameter values goverg respose for each message, ad cotrol, such as the obectve of maxmzg respose rate. I such problems, t may be wse for the marketer to sacrfce some potetal early payoff for the prospect of gag formato about cosumers that wll allow for more formed decsos later. The ature of the soluto: use a model to form the decso. Because we cosder the value of covarates, the specfc problem we study s eve more complcated tha the mult-armed badt problem. Nevertheless, the basc tuto s the same. Suppose that the marketer has two or more uque messages avalable. Early o the campag, the marketer wats to ga formato about the drvers of cosumer respose for each of the messages, ad therefore t eeds to expermet by allocatg each of the 3

7 messages suffcet umbers (e.g., through radom assgmet). After ths tal perod of collectg a trag sample for each message, the marketer ca estmate respose fuctos usg regresso techques. I gorace of the true respose fuctos, the marketer ca compare the predcted respose estmates from the regresso models ad decde to preset the message wth the hghest predcted respose. We refer to ths approach as the myopc rule. However, sce the predctos are estmated wth mprecso, t may be worthwhle for the marketer to calculate ucertaty adustmets, ad corporate these adustmets ts message selecto decsos. The ucertaty adustmets reflect the mprecso the parameter estmates of the regresso models. The adustmets are larger for cosumers that have extreme characterstcs (covarates) tha for typcal cosumers because a extreme cosumer wll provde more formato about the ature of the respose fucto tha a typcal cosumer wll. Over the durato of the campag, the parameters of the respose fucto ca gradually be estmated wth a creasgly hgher degree of precso. For that reaso, the ucertaty adustmets ca be gradually reduced. I summary, we resolve the apparet dlemma betwee formato ad cotrol by troducg sutable ucertaty adustmets to the myopc rule. Our approach s smlar sprt to Asar ad Mela (2000), who model respose to e-mal marketg cotet ad show that modeled respose to customzed cotet s much hgher tha respose to stadardzed cotet. Much of the focus of Asar ad Mela s to capture ther model the sources of uobserved heterogeety across dvduals ad across e-mal vehcles. They the use smulated aealg to fd a optmal customzed desg (.e., rearragg the umber ad order of lks preseted each e-mal) based o the posteror parameter estmates from ther model. Oe dfferece our approach s that we are focused o the problem of whch cotet to preset to whom at the very early stages of the process (.e., maagg the trade off betwee gatherg formato about respose ad maxmzg respose based o the avalable formato). By cotrast, Asar ad Mela's model s calbrated usg formato whch all e-mals ad e-mal cotet are preseted to a maorty of the dvduals the sample (75 percet o average). Aother dfferece s that our model s valdated based o actual respose behavor (rather tha modeled probablty of respose). 4

8 The remader of ths paper s orgazed to hghlght the research framework ad the maageral relevace of our cotrbutos. I Secto 2, we state our problem formally, preset two examples that llustrate the tuto of the theoretcal framework, ad the revew the relevat theory ad lterature. We elaborate o our methodology Secto 3. I Secto 4, we test the performace of our approach usg data collected by a Iteret marketg compay. Secto 5 sketches several research extesos ad offers cocludg remarks. 5

9 2. Statemet of Problem ad a Revew of the Relevat Theory 2.1 Formal Statemet of the Problem Suppose a marketer has the potetal to preset two (or more) uque messages (e.g., baer ads, Web pages) for a partcular marketg campag that wll be termated whe N cosumers have bee exposed to oe of the messages. The marketer treats cosumers sequetally (oe at a tme), exposg the cosumer to oe of the messages. The marketer s obectve s to maxmze total expected respose, whe respose to a message ca be measured a umber of ways (e.g., the decso to purchase a tem from the catalog, the decso to clck o a baer ad, the decso to vst the page featurg Peter s Istat Pck, or the durato tme of a customer's vst to a Web ste). The true probablty of respose depeds o a set of covarates (e.g., demographcs, Web avgato hstory), where the marketer may have some weak prors (e.g., based o prevous experece) about the mpact of the covarates. Some of the parameters of the respose fucto (also kow as a "lk fucto") may be commo across messages. Ths mples that the messages are ot depedet the sese that kowledge about the respose fucto for oe message provdes formato about respose to oe or more of the other messages. The marketer geeral wll have access to oly a small subset the covarates that fluece respose. The marketer lears about the effectveess of the messages as they are preseted. The message selected at ay pot tme depeds o the prevous selectos, the outcomes (respose/o respose) from these prevous selectos, ad the curret ad prevous values of the covarates. Logstc (or probt) regresso models ca be used to calbrate respose fuctos to lear about the parameters goverg respose for each of the messages. The respose fuctos ca the be perodcally updated. 2.2 Badt Problem Examples Badt problems are, geeral, very dffcult to solve closed form. The two hghly stylzed examples that follow ca be solved aalytcally ad are oly teded to hghlght the ature of the soluto to such problems. I the frst example, we demostrate how a marketer ca use covarate formato to form the message 6

10 selecto decso ad acheve a hgher expected respose rate. I other words, we compare a customzed message strategy to a stadardzed message strategy. I the secod example, we demostrate the tuto behd the theory of badt problems (that follows Secto 2.3). We show that, codtoal o beg able to delver customzed messages, the myopc strategy -- whch selects the ad wth the greatest expected mmedate ga -- s ot ecessarly optmal. That s, there may some beeft sacrfcg some potetal early payoff for the prospect of gag formato about cosumers that wll allow for more formed decsos later ad a hgher expected total payoff Settg Up the Problem: Iteret Advertsg Suppose that a marketer has two baer ads avalable. Vstors arrve at the marketer s Web ste oe at a tme, ad the marketer ca choose to dsplay ether Ad A or Ad B. The probablty of clckg o each ad depeds upo a sgle bary varable (e.g., whether the vstor s marred or ot), deoted X. If a marred dvdual arrves at the Web ste, X = 1, whle f a umarred perso arrves, X = 0. The probablty that the vstor s marred s 0.9 at ths Web ste. We wll assume that a bary logt model specfes respose to each ad: (2.1) p X exp( α + β X ) = 1+ exp( α + β X ) where p X s the probablty vstor clcks o ad, = A, B, α s the base level of effectveess, ad β s a parameter capturg the effect of martal status. The expected probablty of success s smply the weghted average of codtoal respose probabltes for marred ad umarred vstors: (2.2) E ( p ) P( X = 1) ( p X = 1) + P( X = 0) ( p X = 0) = = 0.9 ( p X = 1) ( p X = 0) 7

11 We assume that the parameters descrbg vstor respose to Ad B are kow wth certaty. Specfcally, the marketer kows that α = ad β = 0 (.e., respose to Ad B does ot deped o martal status). Usg equato (2.2), we fd that the expected respose probablty for Ad B s We further assume that there s ucertaty about the values of the parameters descrbg respose to Ad A. Specfcally, the marketer kows the value of α A = 3. 45, but s ucerta about β A, the effect of martal status. The marketer s pror belef s that there s 0.05 probablty that β = 3.2 ad a 0.95 probablty that β = 0. I other words, there s a low (5%) chace A A that beg marred has a large postve mpact o clckg, ad a hgh (95%) chace that martal status has o effect o clckg. To keep thgs smple, we wll assume that there are ust two tme perods, t 1 ad t 2, the ad campag, ad oe vstor arrves at the Web ste each perod. The marketer ca use past formato to decde how to proceed. Hece, the ad selected for mpresso t 2 depeds o the respose (clck or ot) at tme t 1. Moreover, the ad selected for t 2 depeds o the martal status of the vstor t 1 ad the martal status of the vstor t 2. A strategy specfes whch ad to select, =1, 2. A strategy s optmal f t yelds the maxmal expected respose rate. We defe the worth of a (two-perod) strategy as the expected total umber of clcks for all possble hstores resultg from that strategy. For the sake of brevty, we restrct our atteto to strategy selecto at oe pot tme the examples. I the frst example, we focus o the decso wth oe perod ( t 2 ) remag. I the secod example, we restrct atteto to the tal selecto ( t 1 ) the two perod problem. t B B Choce of Message for Perod t 2 Assume that the marketer selected Ad B the frst perod, ad therefore the oly decso to cosder s whch ad to select t 2. Whch ad should the marketer choose? Wth oly oe perod left the campag, the marketer should choose the ad that has the hgher expected respose probablty. Sad dfferetly, the myopc decso s optmal a oe-perod problem, ad the worth of the strategy s smply the expected outcome. 8

12 However, the calculato of expected respose dffers depedg o whether or ot the marketer has access to covarate formato. If the marketer has access to the covarate, he ca calculate expected respose probablty codtoal o the value of the covarate. I the absece of the covarate, the calculato of expected respose probablty s ucodtoal. No covarates. Suppose that the marketer does ot have access to X. I the absece of covarate formato, ad wth oly oe perod left the campag, the marketer should choose the message that has the hghest expected respose rate. The marketer kows that the expected respose probablty for Ad B s ; for Ad A, the expected respose probablty s I the absece of covarate formato, the marketer should choose Ad B t 2. The worth of ths strategy s clcks. Covarates. Now, assume the marketer has access to vstor s marred, expected respose to Ad A s gve by: X. If we kow the arrvg E exp( ) exp( 3.45) ( p + A X = 1) = = exp( ) 1+ exp( 3.45) Sce the expected respose to Ad A for marred cosumers s hgher tha the respose to Ad B ( >.0500), the optmal decso for t 2 gve X = 1 s to dsplay Ad A. The optmal decso, gve the vstor s umarred (.e., X = 0) s to dsplay Ad B, sce the expected respose to Ad A by a umarred vstor s Thus, the presece of covarate formato, the ucodtoal worth of our strategy s clcks. I ths frst example, we have show that f the marketer has access to covarate formato, t ca talor ads such that expected respose rate s creased. Whe the marketer dd ot have access to the covarate, the best strategy (wth oe perod to go) had a worth of.0500 clcks. Whe the marketer was able to observe the covarate, the best strategy had a worth of.0510 clcks, a two percet mprovemet. 9

13 2.2.3 Choce of Message Perod t 1 Assume that the respose parameters are the same as the prevous example, except that ow α = 3. 5 stead of α = Suppose a marred cosumer arrves A A t 1. Should the marketer show Ad A or Ad B t 1? Expected respose to ad A for marred vstors ca be calculated as follows: E exp( ) exp( 3.5) ( p + A X = 1) = = exp( ) 1+ exp( 3.5) Sce the expected respose to Ad A s lower tha the respose to Ad B ( <.0500), the myopc decso for t 1 s to dsplay Ad B. Sce the marketer would ga o addtoal formato about Ad A f t showed Ad B t 1, the optmal decso for t 2 (gve Ad B was chose tally) s to show Ad B aga. Hece, the worth of the strategy that volves selectg Ad B s smply 2 (.0500) =. 10 clcks. I ths case, startg out wth the ad wth the hgher expected respose turs out to be myopc. Usg the backwards ducto method of dyamc programmg, we ca show that dsplayg Ad A the frst perod yelds a expected worth of.1058 clcks, whch s hgher tha the worth of tally selectg Ad B (.10 clcks). Why s the optmal tal selecto the ad wth the lower expected respose rate? The tuto uderlyg ths surprsg result s that there s a potetally large payoff for learg more about the effect of martal status. Wthout gog through all of the mathematcal detals, we wll focus o oe calculato. There s ucertaty about the effect of β A ; specfcally, the marketer has a small pror that there s a large payoff to showg Ad A (.e., β A =3.2). By startg wth Ad A, the marketer s able to observe respose to the ad, update hs/her prors ad reduce hs ucertaty. The probablty of observg a respose to Ad A whe α = 3. 5 ad β = 3. 2 s If the marred A vstor clcks o Ad A, the marketer updates hs/her belefs accordg to Bayes rule: A P( β A P( clck o A β A = 3.2) P( β A = 3.2 clck o A) = P( clck o A) (.426) (.05) = = = 3.2)

14 Ths large crease (from fve percet to 43.3 percet) the belef about the probablty of a hgh payoff makes t worthwhle to make the sacrfce of gog wth a ad that has a lower expected respose rate. I ths secod example, we have show that the myopc strategy s ot ecessarly the optmal customzed message selecto strategy. Next, we provde the theoretcal framework that led to such a cocluso. 2.3 A Theoretcal Framework: The Badt Problem wthout Covarates Igorg the covarates for the tme beg, the problem of determg the optmal allocato of messages to N cosumers ca be cast the framework of the classcal mult-armed badt problem, whch has bee extesvely studed the statstcs ad egeerg lterature. The ame derves from a maged slot mache wth k 2 arms. Whe a arm s pulled, the player ws a radom reward, whch may be weghted by a dscout factor takg o a value betwee 0 ad 1. For each arm, there s a ukow probablty dstrbuto of the reward, ad the player s problem s to choose N successve pulls o the k arms so as to maxmze the total expected reward. A classc example that motvated much of the research ths area s the cotext of sequetal medcal trals, where there are k treatmets wth ukow probabltes of success, p,., p, to be chose sequetally to treat a large class of N 1 k patets. The obectve s to mmze the expected umber of patets assged to a feror treatmet. Note that the treatmets produce a reward of 1 or 0, ad so the dstrbuto of the reward s Beroull. Furthermore, there s o dscoutg of the rewards, ad the horzo s fte. Thus, the sequetal medcal trals problem s almost detcal to the marketg problem we stated Secto 2.1, except that covarates are ot volved. I statstcal decso theory, the most wdely adopted approach to solvg badt problems s the Bayesa approach. Berry ad Frstedt (1985) catalog vrtually all of the maor results up to the md-1980s. Wth a Bayesa approach, a badt s a typcal problem dyamc programmg. Whe the horzo s fte, backwards ducto ca 11

15 be used to determe optmal strateges. The example we preseted secto llustrated the decso theoretc approach. I that example, we solved a two-perod dyamc program to obta the optmal soluto (but to coserve space, ot all of the computatos were preseted). Whe there are more tha two arms ad the tme horzo s large the badt problem, solutos ca become computatoally tractable. Oe of the most mportat results the badt lterature s Gtts (1979) soluto to the k -armed badt problem through what he called dyamc allocato dces (DAIs). Gtts ad Joes (1974) ad Gtts (1979) showed that the desrablty of a arm ca be determed by fdg a kow arm such that both the arm uder cosderato ad the kow arm are optmal a two-armed badt. I other words, a k-dmesoal badt problem ca be decomposed to k dfferet two-armed badts, each volvg oe kow ad oe ukow arm. 2.4 Alteratve Approaches The assumpto of geometrc dscoutg s requred to obta Gtts results, ad there are several dffcultes applyg the optmal polces usg Gtts framework. DAIs are ofte dffcult to compute, sestve to small devatos the prors, ad may be sestve to the choce of the dscoutg factor. Ths last pot s especally dsturbg sce, practce, oe may wat to use the geometrc dscouted problem as a approxmato to the uform fte horzo problem. I summary, DAIs ca be utlzed to reduce the dmesoalty of the problem, they are dffcult to compute ad/or requre strog assumptos. As a result, more practcal (ad tutvely appealg) alteratves have bee proposed. These alteratve asymptotcally optmal approaches guaratee that the observed proporto of successes coverges to the true proporto of successes whe the total umber of trals becomes fte. They apply a basc prcple of flatg the myopc estmator (.e., the estmator that has the hghest expected outcome for the curret observato) by a sutable adustmet that reflects oe s ucertaty about future observatos. I a sese, the ucertaty adustmet reflects the mportace of vestg formato that could be worthwhle makg better decsos later. Whe the umber of trals becomes very large, the ucertaty adustmet goes to zero, ad the myopc estmator approaches 12

16 optmalty. The decso rules assocated wth these approaches are coceptually smple: choose the treatmet wth the hghest ucertaty-adusted probablty of success. We ext dscuss oe specfc allocato rules that have bee proposed. La (1987) poted out the usefuless of sequetal testg theory makg ucertaty adustmets to the so called certaty-equvalece rule the egeerg lterature. He proposed a class of smple adaptve allocato rules that corporate these ucertaty adustmets for the mult-armed badt. These allocato rules are based o certa upper cofdece bouds, whch are developed from boudary crossg theory, for the k populato parameters. Suppose the true parameters for the k treatmets are θ, = 1,., k, ad that these parameters have a commo desty fucto that belogs to the expoetal famly. Rather tha samplg at stage + 1 from the populato wth the largest θ ˆ, T ( ) (.e. the myopc rule), where T ( ) deotes the umber of tmes oe has sampled from Π up to stage, La proposes the followg smple modfcato: sample at stage + 1 from the populato Π wth the largest upper cofdece boud U, T ( ). The upper cofdece boud s defed as: (2.5) { ˆ ˆ 1 U ( g, N) = f θ : θ > θ ad I( θ, θ g( / )},,, N where s the umber of observatos take from populato Π, N s the total sample sze, I ( θ, λ) s the Kullback-Lebler formato umber, ad g ( 0) satsfes certa assumptos. To llustrate La s deas, we cosder the case of Normal destes. Suppose that 2 Y 1, Y2,. are..d. radom varables wth mea θ ad varace σ. I ths setup, La shows that cofdece boud reduces to (2.6) U 2 g N = ˆ 2σ (, ) θ + g, N 13

17 2 where g( / N) f ( / N) /[ 2( / N) ] =. The fucto ( / N) f s a approxmato of the optmal stoppg boudary for the aalogous cotuous tme Normal two-armed badt problem wth oe arm kow that was solved by Cheroff ad Ray (1965). We ca rewrte equato (2.6) terms of stadard errors, whch has more tutve appeal. If we let: (2.7) K = N N f, ad N (2.8) σ se( θ ˆ ) =, the (2.9) U, ( g, N) = θˆ = θˆ + N f N + K se( θˆ N σ ) Fgure 1 s a graph of K ( / N) ad K ( N) / /. It s oteworthy that K / decays very rapdly. The mplcato s that vrtually all of the learg should occur up frot, after whch tme oly eglgble adustmets to the myopc rule are requred. For example, whe / N =. 20, K / =. 22. I summary, the ucertaty adustmets quckly asymptote to 0. 14

18 K as a fucto of /N /N 1 K K/sqrt() Fgure 1. Graph of ucertaty adustmet for proposed dyamc allocato dex. La s proposed rules have a ce heurstc terpretato. The upper cofdece boud U, flates the estmator θ ˆ, r by a amout that decreases wth the umber r of observatos already take from the populato. Thus, U, depeds ot oly o the estmator θ ˆ, but also o the sample sze, ad comparg the k populatos o the bass of U, volves ot oly the parameter estmates but also the sample szes of all populatos. 2.5 The Covarate Badt: Results to Date Despte the vast badt problem lterature, we are aware of oly a few publshed artcles that cosder models wth covarates. These few studes all cosder a case whch there s oly oe ukow arm ad oe covarate. Furthermore, each of these studes take a Bayesa approach: computato of the optmal strateges mply backwards ducto va dyamc programmg solutos. I hs poeerg work, Woodroofe (1979) cosders a hghly stylzed covarate model. A key assumpto hs model s that the support of the dstrbuto of the covarate s ubouded above. Uder the assumptos of hs model, Woodroofe proves 15

19 that the myopc strategy s asymptotcally optmal. The reaso such a result s possble s that the presece of a covarate that s ubouded above assures that a myopc strategy wll dcate the ukow arm ftely ofte. Woodroofe (1982) ad Sarkar (1990) exteded ths result to more geeral models (they dd ot, however, relax the assumpto of the covarate havg fte support). Clayto (1988) vestgated a fte horzo uformly dscouted Beroull badt where the probablty of success depeded o the covarate through a lk fucto such as the logt. Hs focus was o descrbg the structural propertes of the optmal strateges for varous covarate models. Our problem s much more geeral tha the oe examed the few studes metoed. Oe, we allow the support of the covarate(s) to be bouded. Two, we seek smple, computatoally tractable rules. Three, we am to be less restrctve about the dstrbutoal assumptos volved. Ad four, we allow for multdmesoalty terms of multple ukow arms ad multple covarates. The method we propose the ext secto addresses all of these ssues. 16

20 3. A Method for Hadlg More Geeral Covarate Models 3.1 The Basc Approach Here, we preset a example that llustrates the key features of our proposed approach. Suppose we have two marketg messages (Message 1, Message 2) for whch respose s govered by a smple regresso model wth oe covarate: 2 (3.1) y = α + β X + ε, ε N(0, σ ), 1, 2. = We cosder two cases: parallel les ad tersectg les. Parallel Les The two les wll be parallel f β 1 = β 2. I ths case, the problem reduces to oe of determg the dffereces mea respose of the two messages. Hece, the problem reduces to the two parameters cosdered the stadard badt problem! I fact, ths specal case makes t clear why Woodroofe s result that the myopc rule s asymptotcally optmal caot be true f we relax certa assumptos. If the two les are parallel, the covarate badt reduces to the stadard badt; ad we kow that the stadard badt, the myopc rule s, geeral, ot optmal. Itersectg Les The frst key sght our approach s that, rather tha estmate four parameters { α, β, α β } 1 1 2, 2, we ca re-parametrze the problem as volvg three parameters. I the case of two tersectg les, we wat to estmate {, β β } x o 1, 2, where 0 tersecto of the two les. Wth a lttle bt of algebra, oe ca show that x s the pot of (3.2) x 0 α 2 α1 = β β

21 If x 0 were kow, our allocato rule would be smple: at each stage, select Message 1 f the covarate X s to rght of x0 ad show Message 2 f the covarate s to the left of x 0. However, gorace of the true parameters, oe ca obta a estmate of ths pot of tersecto by pluggg the parameter estmates: (3.3) x ˆ αˆ αˆ = ˆ β ˆ 2 β1 Hece, the myopc rule would be: at stage, select Message 1 f the covarate X s to rght of ˆx 0 ad show Message 2 f the covarate s to the left of ˆx 0. However, there s ucertaty ths estmate of x 0, ad we wat to corporate ths ucertaty our rules Calculatg a ucertaty measure for x 0 usg the Delta Method Although there other measures of parameter ucertaty (e.g., Kullback Lebler formato), the stadard error s perhaps the most wdely used measure, ad so we wll adopt t here. Sce 0 x s a fucto of the ukow parameters { α, β, α β } 1 1 2, use the Delta method to compute the asymptotc covarace. Let x = f ( ˆ β, αˆ, ˆ β, ˆ ). The Delta method allows us to estmate the ˆ α1 ˆ( ˆ0 asymptotc covarace matrx of ˆx 0, deoted V x ), as: 2, we ca (3.4) V ˆ( xˆ ) = Vˆ( f ( ˆ β, αˆ, ˆ β, αˆ )) = GVˆ( ˆ β, αˆ, ˆ β, ˆ ) G α1 where (3.5) f f f f G =,,, ˆ ˆ ˆ β α 2 2 β αˆ

22 Havg calculated V ˆ( ˆ β, αˆ, ˆ β, ˆ ) adg, we ca the compute the quadratc form α1 V ˆ( xˆ0 ). Our proposed measure of ucertaty s the stadard error of ˆx 0 at stage : (3.6) SE xˆ ) = Vˆ( xˆ ). ( How may stadard errors? Sce we have reduced the problem to fdg oe measure of ucertaty, we ca apply La s method for determg the sze of the ucertaty adustmet. I our smple model wth Normally dstrbuted dsturbaces, we ca apply equato (3.8) to fd a ucertaty adustmet (UA): (3.7) UA ( ˆ ) ( ˆ x0 K SE x0 ) N = where K ( / N) s specfed equato (3.6) The Oe Covarate Rule Now we are a posto to formulate a rule that captures the formato versus cotrol tradeoff. Our oe covarate rule s: at stage, f the covarate the terval [ x UA ( xˆ ), xˆ + UA ( xˆ )] ˆ X falls, the choose the message wth the smaller curret sample sze (umber of selectos); f the covarate les outsde ths terval, choose the message wth the hgher value of ŷ. The dea s to try to lear by mprovg the formato cotet, whe the value of the covarate s wth the gray zoe (.e., the ucertaty adustmet dstace from ˆx 0 ), ad to be aggressve (myopc) whe the value of the covarate value s outsde the gray zoe. The basc rule s llustrated Fgure 2, where the oval represets the gray zoe. 19

23 Fgure 2. Dagram descrbg the "oe covarate rule:" the oval represets the gray zoe where t makes sese to trade off respose for formato. 3.2 Multdmesoal Badt Models We have specfed a rule for the case of two ukow messages ad oe covarate. A mportat exteso of our rule s to allow for multple covarates ad three or more messages. We wll ext show how to accomplsh ths by re-characterzg the problem terms of the dffereces predcted resposes. I the stadard multple lear regresso framework wth k covarates, a estmate of the respose varable s gve by: (3.7) yˆ = αˆ + ˆ β x ˆ β k xk A estmate of the stadard error of predcto for a ew observato s: SE = s x X X (3.8) [ ] 1 x where 20

24 (3.9) s = ( y yˆ ) k 1 2, y s the true respose, ŷ s the estmate of respose, ad x s a vector for the locato of the ew data pot. We ow tur to descrbg our multdmesoal approach. We frst defe the leader as the arm wth the hghest predcted respose, ad a coteder as a feror arm for whch the dfferece betwee the leader s predcted respose ad ts respose s less tha or equal to a approprate ucertaty adustmet. The rule for the multdmesoal approach essetally specfes the followg: f all of the dffereces betwee the leader ad the coteders are greater tha ad the correspodg ucertaty adustmets, we play myopc by stckg wth the leader. If oe or more of the dffereces the predctos are less tha the correspodg ucertaty adustmet, the the rule specfes the coteder wth the smallest sample sze. To formalze the rule, let ŷ L deote the predcted respose of the leader, ad the estmate of the dfferece betwee respose for the leader ad the predcted respose for arm (at stage ). Further, we deote the ucertaty adustmet for ths estmated dfferece as UAd ˆ are gve by the ext two equatos:. The formulas for ths estmate ad the stadard error of the estmate dˆ (3.10) dˆ = ( yˆ yˆ ), where yˆ = max( yˆ,, yˆ ), k 1 L L 1. k > (3.11) UAˆ = K SE( yˆ ˆ L y ) d N Assumg the arms are depedet, 21

25 (3.12) SE( yˆ = L yˆ ) = L Var( yˆ Var( yˆ ) Var( yˆ ) L yˆ ) = s x 2 L [ X L X L] x s x [ X X ] x If all of the dˆ s are greater tha the correspodg UAd ˆ s, there are o coteders ad the rule selects the leadg arm. Otherwse, there s at least oe coteder ad the rule specfes choosg the coteder wth smallest sample sze. We are ow a posto to descrbe our Basc Rule, whch ca be appled multdmesoal lear model settgs. Basc Rule: At stage, f dˆ UA smallest sample sze; f ˆ d for at least oe, choose the coteder wth the dˆ > UA for all, the choose the leader. ˆ d Note that the sprt of the basc rule s the same as our oe covarate rule: lear by mprovg the formato cotet whe (myopc). dˆ s ot too large, otherwse, be aggressve 3.3 Extedg the Basc Rule to Hadle Other Dstrbutoal Assumptos I the basc approach, we assumed that the error terms followed a Normal dstrbuto. Here we geeralze the basc rule to a oparametrc rule whch dstrbutoal assumptos are ot ecessary. I ths way, our model ca be appled to other models (e.g., logstc regresso for bary respose). The key modfcato s to devote a small umber of tal observatos (message mpressos) for strctly expermetal purposes. The marketer could ether radomly assg the messages or follow a rotato scheme. The purpose of ths short expermetal perod s to adust for the fact that, ear / N = 0, the assumpto of Normalty s crtcal ( techcal terms, we are appealg to the theory of large devatos). Therefore, we wll be slghtly less precse ths very sestve rage. As we move towards larger values of / N, the assumpto of Normalty becomes more approprate, ad we ca swtch to the basc rule. We suggest the legth of expermetal perod be o the order of 22 log N. So, for example,

26 the marketer could alterate selecto of two messages for swtch to the basc rule. To summarze, the o-parametrc rule s: 2 log N observatos ad the No-parametrc Rule: Step 1: Radomly assg or rotate the two messages for 2log N observatos. Step 2: Apply Basc Rule for the remader of the marketg campag. 23

27 4. Implemetato of the Ideas Testg our method would ordarly volve gettg agreemet advace from a Iteret marketer to track the effectveess of the proposed procedure geeratg respose (relatve to a exstg polcy or polces). Fortuately, we were preseted wth a stuato whch, wth a few reasoable assumptos, we could test our approach versus ay umber of bechmarks. Normally, our procedure volves makg a decso about whch sgle respose opportuty (e.g., a clckable ad baer) to preset to each customer. We foud a compay that had already collected cosumer respose formato across multple respose opportutes. To evaluate the performace of ay gve polcy (cludg our ow approach), we smply have to decde whch alteratve we would have preseted to each customer ad the look at actual respose. Our goal s to use these data to establsh the superor performace of a approach that uses realstc (ot smulated) covarate formato wth dyamc updatg. 4.1 Data Our data provder s Dgtal Impact, Ic., a Iteret-focused drect marketg compay that provdes tools ad servces such as Java-based product catalogs for Iteret marketers. Wth Dgtal Impact s Merchat Mal product, a ole marketer ca delver persoalzed, graphcally rch, dgtal catalogs to customers based upo ther purchases, terests ad prefereces. Oe of Dgtal Impact s clets s a leadg ole musc ste. For cofdetalty, we wll refer to ths clet as Apollo. O Apollo s behalf, Dgtal Impact desgs ad delvers completely customzed emal promotos o a b-weekly bass. The catalog album promotos come the form of album descrptos ad possbly other cotet (e.g., pctures, specfc marketg messages). The customer ca clck-through o ay or all of the assocated lks for the dvdual albums ad arrve at Apollo s Web ste to get further formato (ad possbly order the album). The albums cluded each customer s catalog are chose by Dgtal Impact accordg to a propretary "market basket" algorthm that uses customer formato ad past purchase hstory. 24

28 I oe partcular marketg campag, Dgtal Impact set a e-mal "catalog" to over 13,000 Apollo customers. Each customer saw a total of te dfferet albums; however, ot all customers saw the same te albums (a cosequece of the propretary algorthm used by Dgtal Impact). Noetheless, there was cosderable covergece o the te albums preseted most frequetly to Apollo customers. Of the 13,000 customers reached ths campag, 10,684 saw at least three of the te albums lsted Table 1. As show Table 2, more tha half of these customers saw at least seve of the top te lsted albums. These te albums form the set of respose alteratves we wll cosder for presetato to each customer. Table 1: Top 10 Albums Terms of Clck-Throughs for March 8 E-Mal Campag Artst Album Ttle Gere Presetatos Clckthroughs Prce Crystal Ball R&B / Soul Mles Davs Kd Of Blue Jazz Thrd Eye Bld Thrd Eye Bld Pop/Rock Savage Garde Savage Garde Pop/Rock Orgal TV Soudtrack Upstars At Melrose Place Jazz Jazz Radohead OK Computer Pop/Rock Pat Metheey Group Imagary Day Jazz Deep Forest Comparsa R&B / Soul Paul McCartey Stadg Stoe:McCartey Pop/Rock Aretha Frakl The Delta Meets Detrot... R&B / Soul TOTAL Table 2: Number of top 10 albums cluded Catalog Number of Top 10 Albums Icl. Choce Set Number of Customers Percet of Customers Total 10, % Our goal s to decde whch three albums to preset to each customer. We choose to preset three albums (rather tha a sgle respose alteratve) for two reasos. Frst, 25

29 as show Table 1, the overall respose rate s low: there are 478 clcks o 69,314 presetatos, whch traslates to a clck rate of 0.69%. Presetg three albums gves us more data wth whch to calbrate our respose models, ad stll forces us to exclude over half the data. Secod, decdg o a set of albums s closer sprt to the catalog malg practced by Dgtal Impact, ad affords us the opportuty to select a subset of albums by gere, recordg artst, etc. Our approach assumes that each dvdual has the opportuty to respod to each of the albums preseted the e-mal. Ths s probably aïve. We kow from recet studes (e.g., Asar ad Mela, 2000) that placemet wth a e-mal dlueces the probablty of respose, ad the albums preseted lower the e-mal get a lower respose, ceters parbus. Noetheless, we feel that ths effect probably works agast us, troducg ose to the process ad dampeg the predcted respose rates from our models. 4.2 Methodology We coducted a Mote Carlo study to assess the performace of three dfferet polces for selectg albums: radom assgmet, a pror assgmet (based o album gere), ad the polcy mpled by the procedure developed ths paper, whch we label dyamc customzato. We descrbe each of these polces tur. As a bechmark (a bass agast whch we ca compare the performace of the other two polces), we use a radom umber geerator to decde whch albums to preset to each customer. We assg a radom umber to each album that the customer actually saw, ad the select the three albums wth the hghest radom umbers. Ths esures that we preset each customer wth three albums that he or she actually was preseted (ad had the opportuty to respod to). The a pror assgmet polcy s based o a approach smlar to (but much smpler tha) the oe Dgtal Impact actually uses whe costructg ts malgs. I ths stuato, we do ot have ay drect formato or eve strog prors regardg the atcpated resposes to the te albums o our lst. However, we do have formato about how much each customer o the lst has spet o albums that fall to dfferet muscal categores or geres. Sce the te albums fall to three geres, Pop/Rock, 26

30 R&B/Soul, ad Jazz, we classfed each customer to oe of three gere groups by calculatg hs/her hstorcal percetage of purchases these three geres ad the detfyg the gere wth the maxmum percetage. If for example, a customer had 20% of pror purchases Pop/Rock, 15% Jazz, ad 30% R&B/SOUL, he/she would be classfed to the R&B/SOUL group. The, we selected the three albums from the gere correspodg to the customer s group (see Table 3a). If the customer was ot actually preseted oe of the albums, we swtched to a radom assgmet polcy. Table 3a: A Pror Polcy Groupgs Group Album (Artst) Number of Customers Pop/Rock Thrd Eye Bld Savage Garde 5163 Radohead Paul McCartey R&B/Soul Prce Deep Forest 1188 Aretha Frakl Jazz Mles Davs Pat Metheey Orgal TV Soudtrack 1133 Table 3b: Percet of Album Clck-throughs by Hstorcal Gere Percetage Mea Percet of Clcker Purchases Gere R&B/ Album Artst Album Gere Pop/Rock Soul Jazz Prce R&B / Soul 30.82% 15.27% 6.84% Mles Davs Jazz 31.93% 5.90% 19.74% Thrd Eye Bld Pop/Rock 69.30% 6.34% 0.81% Savage Garde Pop/Rock 38.60% 9.32% 0.00% Orgal TV Soudtrack Jazz 21.51% 3.53% 16.12% Radohead Pop/Rock 51.40% 7.51% 11.81% Pat Metheey Group Jazz 18.53% 5.59% 33.82% Deep Forest R&B / Soul 23.30% 18.51% 1.09% Paul McCartey Pop/Rock 58.26% 7.71% 4.29% Aretha Frakl R&B / Soul 12.53% 15.91% 11.68% Note: Rows do ot add to 100% sce there are may other geres ot cluded ths lst (e.g. Coutry, Iteratoal, Heavy Metal). To llustrate the mportace of ths varable, Table 3b shows the mea percet of prevous purchases the album s gere for those customers who clcked o exactly oe album. From the table, t s clear that the mea percetage of pror purchases the gere of the album clcked s dsproportoately hgher. For example, o average, the Pat Metheey clckers had 33.5% of ther pror purchases Jazz ad oly 18.5% of ther 27

31 pror purchases the Pop/Rock gere. I cotrast, Paul McCartey clckers bought 58.3% of ther pror purchases the Pop/Rock gere ad oly 4.3% of ther pror purchases Jazz. The last polcy, whch we label dyamc customzato, s cosstet wth the deas uderlyg the model developed ths paper. To mmc the effect of beg able to use early respose behavor to calbrate the model ad update the polcy for later malgs, we play out the selecto approach a seres of fve "waves." Specfcally, we radomly assged albums for 20% of customers ( wave 1). We the calbrated stepwse logstc regresso models for each of the te albums, allowg covarates to eter the model f they passed a threshold sgfcace level. Usg the models thus calbrated, we selected the three albums for the ext wave (20-40%) based o the predctos of the models (.e., by orderg the p-hats). We cotued ths process for the three remag waves. Oly two covarates cosstetly etered to the logstc regresso models. The most mportat covarate we detfed s geper, the percetage of customer s prevous purchases the gere of the artst uder cosderato. Ths gves us some added cofdece that the a pror strategy s based upo a reasoable decso rule. A secod mportat determat of clck through s HTML: a dcator varable represetg whether or ot the cosumer had HTML e-mal capablty. Respose rates for most of the albums were sgfcatly hgher for customers who were HTML-eabled. Of the 10,684 customer the sample, 2646 (24.8%) had HTML= Results The results of the Mote Carlo study are reported Table 4. I the study, we ra 50 smulatos for each codto. The average clck-throughs were 222, 240, ad 274 for the radom assgmet, a pror, ad dyamc customzato codtos, respectvely. Frst of all, we ote that the dfferece betwee the radom polcy ad the a pror polcy s statstcally sgfcat ad amouts to a dfferece of roughly 8% respose. Ths suggests that the formato o pror purchase plays a valuable role targetg respose opportutes to cosumers future malgs. More mportatly, there s also a statstcally sgfcat dfferece betwee the performace of the a pror polcy ad our 28

32 proposed dyamc customzato strategy. Ths suggests that t s also valuable to be able to calbrate the effects of the covarate formato ad allow the model predctos to fluece future targetg decsos. The results are at least suggestve of the superor performace of a approach that uses covarate formato a dyamc way. Table 4: Results Polcy Descrpto Radom Assgmet 1. Radomly assg 3 albums from lst of those Top 10 albums that actually appeared customer s catalog A Pror Assgmet 1. Classfy each customer to oe of three gere groups (R&B, Jazz, Pop/Rock) by calculatg hs/her hstorcal percetage of purchases ad detfyg the gere wth the maxmum percetage. 2. Sed each customer a specally selected set of three albums desged to match the gere classfcato Step 1. Dyamc Customzato 1. Radom assgmet for 20% of customers 2. Calbrate stepwse logstc regresso models for each album usg covarates 3. Rak order albums by estmates of clck-through percetage (phats). 4. Select the three albums customer s set wth the three hghest p-hats for ext 20% of customers (wave 2). 5. Repeat steps 2-4 for remag waves (40-60%, 60-80%, %). Clcks (std error) 222 (1.61) 240 (0.83) 274 (1.42) 29

33 5. Research Extesos ad Cocludg Remarks 5.1 Research Extesos Message wearout. Thus far, we have ot cosdered the possblty that a customer may be exposed to a message more tha oce. Advertsg theory would predct a wearout effect assocated wth a creasg umber of repettos of a message. Oe way a marketer could hadle ad wearout would be to smply expose a customer to a dfferet message after a certa umber of exposures to the orgal message. A alteratve method would be to corporate ad wearout drectly the regresso models by allowg for a curvlear effect of a addtoal covarate represetg the umber of tmes the customer has bee exposed to the message. Of course, both approaches would requre that the marketer track the umber of exposures to each message for each customer. Decdg whe to ed a pre-test. Although our research s motvated by the opportutes afforded by ew meda, our approach has a more tradtoal applcato that ca be exploted by marketers. The typcal approach for coductg a pretest s to choose a arbtrary expermetal perod ad the use the pretest results to select a message for the remader of the marketg campag. Ths s smlar to the approach take medcal trals, whch researchers choose a expermetal phase ad a termal phase whch the treatmet phase wth the hgher mea the expermetal phase s used exclusvely durg the termal phase. Ths sequetal medcal trals problem has bee extesvely studed the statstcs lterature. La, Lev, Robbs ad Segmud (1980) show how to choose the legth of the expermetal perod (.e., determe a stoppg rule) such as to maxmze the expected reward for the etre tral (total umber of patets treated). A key sght ther approach s that the legth of the expermetal perod should deped o the total umber of patets treated. I certa marketg stuatos t may ot be possble to cotually lear ad update the parameters of the customer respose models. Nevertheless, the marketer ca adopt a two-stage pretest approach whch the legth of the expermetal perod s chose accordg to La et al s stoppg rule. We could adopt ther approach, ad also exted the theory to hadle covarates. 30