Media Mix Modeling vs. ANCOVA. An Analytical Debate

Transcription

1 Meda M Modelng vs. ANCOVA An Analytcal Debate

2 What s the best way to measure ncremental sales, or lft, generated from marketng nvestment dollars? 2

3 Measurng ROI From Promotonal Spend Where possble to mplement, an epermental desgn that uses a randomly selected holdout group provdes the most statstcal power and relablty n marketng measurement Sample Drect Marketng Campagn Plan: 22 June July August September October November December TMS DM TMS EM TMS EM 2 DM3 DM4 DM2 Event Market Events and Eternal Factors 3

4 Use Randomzaton to Increase Statstcal Power Holdout sample should be selected from the lowest level of the epermental desgn and each treatment cell should have a correspondng randomly selected holdout group Segment Targets N=5,6 Segment 2 Targets N=35,2 Segment 3 Targets N=2, TREATMENT GROUP (TEST) RANDOM CONTROL TREATMENT GROUP (TEST) RANDOM CONTROL TREATMENT GROUP (TEST) RANDOM CONTROL NOTE: Control group can be much smaller than treatment group. Use sample sze calculator to determne mnmum possble sample sze for control group Sample Sze Calculator 4 Opton to apply fnte populaton correcton adjustment

5 Verfy Pre-Campagn Holdout Valdty PRE PERIOD CAMPAIGN RUNS POST PERIOD Random selecton of the control group from the target group should guarantee that the test and control groups are equvalent on key metrcs pror to the campagn start, but Statstcal testng of the dfference between pre-perod test and control groups on key metrcs s an mportant valdaton step and adds to confdence n the post-perod measurement P-Values for the Dfference Between Test and Control Group Mean Values on Key Metrcs n Pre-Perod Sample Metrcs Group Group2 Group3 Pre-Perod Trends: Test and Control Groups Avg. Sales$ Meda Impressons Other Promotonal Spend Compettor Spend Demographcs p >.5 s non-sgnfcant

6 Measure Pre-Post Launch Change If the test group and control group are statstcally equvalent pror to the campagn launch, then the dfference n sales between the groups after the campagn represents the ncremental sales contrbuton of the campagn ANCOVA (Analyss of Covarance) test wll measure the sgnfcance of the dfference and also control for other potental factors that could dfferentally mpact test and control groups durng the campagn perod PRE PERIOD CAMPAIGN POST Test Control Campagn Effect?? Average Weekly Sales per Target: Test v Control, Pre and Post Pre-Perod Post- Perod Growth From Pre to Post Test Control Test-Control..5.4 ANCOVA Adjusted. Campagn Effect 6 Sales by Week: Test and Control Groups ANCOVA Adjusted dfference s after controllng for covarates and, f sgnfcant (pvalue less than.5), s the measure of true ncremental sales from the campagn

7 Summary: ANCOVA for Marketng Measurement Benefts Etremely relable results Conservatve test Control for other factors that may mpact volume growth of target relatve to holdout Able to scale to calculate overall ROI from marketng program Epect replcable results f same condtons and weghts apply n repeated treatment 7

8 There s another opton Meda M Modelng can overcome many lmtatons of ANCOVA-based analyss 8

9 Lmtatons of ANCOVA Feasblty of holdout group Opportunty cost of beng out of market wth ncremental meda Selecton of test perod length s subjectve Dffcult to measure mass & dgtal meda No gudance on cross-tactc decsons Does not provde nsght nto future budget allocaton decsons Does not eplan base factor contrbutons As we wll see, Meda M Modelng wll overcome all of these lmtatons 9

10 How s Meda M Modelng Dfferent? Meda M Models can be used to understand the ncremental, layered effect of crosstactc marketng over tme Incremental Meda Contrbuton Base

11 Revenue Incremental Revenue Revenue Incremental Revenue What are the Requrements and Process? Input Data Statstcal Models Response Curves & Optmzaton TV Tme Drect Mal Tme y t p Sales Decomposton p t t TV Drect Mal Rado Prnt Base Investment Amount Prnt Rado Drect Mal TV Rado Tme Prnt Drect Mal % Rado 8% Prnt 4% Segment TV 6% Base 5% - Tme Segment Segment 2 TV Drect Mal Rado Prnt Base Investment Amount Investment Prnt Rado Drect Mal TV Tme Tme

12 How Does Meda M Modelng Work? 2 L l lt l y t X 2 2 y y k L y... k L y n n y y y 2 ) ( ) )( ( y y y u 2 2 ) ( n n y u Functonal forms of model equatons Estmaton of equatons ) ep( y

13 Functonal Forms of Equatons Functonal form of a relatonshp between response and eplanatory varables s determned by factors such as dmnshng/ncreasng returns to scale, (a)symmetry n response, etc. Some of the most frequently used functonal forms are: Functonal Form Representaton Return to Scale Lnear Quadratc Power addtve Multplcatve (log-log) l L y t X y y l lt y... 2 L k Constant Dmnshng Dmnshng Dmnshng 3 Log-Recprocal y ep( ) S-Shaped

14 Estmaton of Equatons Eample equaton y Estmaton of the betas y n Resdual ( n ( )( y ) u 2 y) u y y y Populaton equaton ndcatng relatonshp between and y; estmated usng sample of data representng the populaton Intercept equals the sample average of y plus the sample estmate of Sample covarance between and y dvded by the sample varance of Dfference between actual and predcted, estmate of the unknown error n the populaton equaton 4 Sum of Squared Resduals n u 2 n ( y ) 2 Ordnary Least Squares estmates mnmze the sum of squared resduals

15 Applcaton of Parameter Estmates How do we calculate contrbuton for each varable n the model? Multply coeffcent from model ( beta ) by weekly model nputs (mpressons) Sum weekly values to get total contrbuton attrbutable to each meda Model Coeffcent ( Beta ) for Dsplay: Week Dsplay Impressons Contrbuton 5/3/29,972, /6/29 2,226, /3/29 2,483, /7/2 5,55, /4/2 7,6, Sum contrbuton across weeks to get total ncremental sales due to Dsplay 53,45 5/2/2 4,937,

16 GRPs GRPs Measurement of tme-varyng mpacts Adstock refers to the effect of advertsng etendng several perods after the orgnal eposure Estmate usng Dstrbuted Lag Model. Estmate model wth lagged effects for all meda terms coeffcents represent % decay at each lag 2. Smooth wth estmaton of gamma dstrbuton to the lagged effect coeffcents Estmate usng varous eponental decays TV TV Decay - Step Lag Model TV Decay - Step 2 Gamma Dstr Week TV Adstock t TV Adstock ( TV.#) Week t t 6 TV Adstock (2%) Adstock (4%) Adstock (6%) Adstock (8%)

17 Toolset of econometrc methodologes Methodologes used n MMM Analyses Ordnary Least Squares (OLS) Med (Bayesan Shrnkage, Random Coeffcents) Unobserved Components Models (UCM) Two Stage: UCM-Med Seemngly Unrelated Regresson (SUR) Structural Equaton Modelng (SEM) Tme Seres Data (.e. Natonal Week) OLS Methodology Selecton UCM Panel Data (.e. DMA Week) Med Herarchcal Relatonshps Two Stage: UCM - Med SUR SEM 7

18 Case Study Comparsons What does each approach offer n these nstances? 8

19 NR per Physcan per Week Case Study : Drect Campagn Typcal mult-channel campagn to physcans wth m of tactcs deployed n rapd successon across long tmeframe Captalzes on use of unversal control group of non-marketed holdout EM EM2 DM DM Tele-detal DM2 EM Resend EM Resend EM2 Resend - 9

20 NR per Physcan per Week Case Study : ANCOVA APPROACH. Defne multple pre-post perods 2. Conduct holdout-valdty tests for each pre-perod and each set of test/control groups 3. Measure ANCOVA-Adjusted change n volume usng double dfference PRE PERIOD 2 POST PERIOD 2 EM EM2 DM ANCOVA: PRE PERIOD DM Tele-detal ANCOVA: POST PERIOD DM2 EM Resend EM Resend 2 EM2 Resend - 2

21 Case Study : ANCOVA Ancova : Tme Perod Ancova 2: Tme Perod 2 Change n Per Physcan Prescrpton Volume from Pre to Post Change n Volume from Pre2 to Post2 Test Control Dfference. 2.7 ANCOVA-Adjusted Sgnfcance.5 p <. ANCOVA could provde sold measurement of overall campagn mpact for two dfferent tme perods whle controllng for other factors Per physcan ncrease could be scaled to measure total mpact and calculate overall ROI 2

22 Case Study : Med Modelng Approach. Use correlogram approach ftted wth gamma curves to calculate decay curves per channel 2. Transform nput varables to account for decay 3. Buld model at the physcanweek level over 3 weeks of hstory and all physcans, whether targeted or not n campagn 4. Ft model usng best functonal form 5. Calculate response curves for each tactc 6. Input nto plannng tool for optmzaton 22

23 Case Study : Campagn Plannng From MMO Output MMO equaton creates outputs that can be used n a scenaro plannng tool to test the mpact of dfferent nvestment levels by tactc and calculate epected ROI from varyng budget levels 23 and ANCOVA confrmed lft estmates

24 Case Study 2: Web Support Program Stuaton: Launched consumer support webste where consumers regster onlne for product support and nformaton. Consumers only provde zp code n onlne support regstraton. Sales not able to be ted drectly to consumers but only to geography (zp code) Key queston: Does consumer support program drve future sales? ANCOVA Approach: ) Match consumer regstered zp code to most lkely purchase zp code 2) Identfy control zp codes wth no consumer regstratons n promty 3) Test lft after web program launches Volume per household per month Pre Launch Sep-9 Oct-9 Nov-9 Dec-9 Jan- Feb- Control Test Web Vsts per HH per Month Market Share Pre Launch Sep-9 Oct-9 Nov-9 Dec-9 Jan- Feb- Control Test 5% % 5% % Sep-9 Oct-9 Nov-9 Dec-9 Jan- Feb- Control Test 24

25 Case Study 2: Possble ANCOVA Output ANCOVA may demonstrate a lnk between sales and web support program use Test pre-post perod dfferences between zp codes wth regstratons and wth no regstratons Control for covarates that mght nfluence test zp codes 25

26 Case Study 2: Med Model Approach. Collect zp-level data on all programs n place, by week, over long tme perod 2. Calculate contrbuton of each of the tactcs, ncludng the web regstratons 3. Compare relatve contrbuton to sales and relatve ROI levels of each tactc Input Model ROI Model Output: Quadratc Form Effect Estmate StdErr tvalue Probt Intercept log_trend total_mkt_grp sq_total_mkt_grp (.39) decay_regster sq_decay_regster (8.5) decay_actvaton sq_decay_actvaton (4.6) decay_actvaton sq_decay_actvaton Baselne 52% TV GRP 25% 2: Regstratons 3% 4: Actvaton 6% 3: Actvaton 2 7% 6: Promoton 7% : 26 Effect Estmate StdErr tvalue Probt Intercept <. log_trend <. total_mkt_grp <.

27 Case Study 3: Cross-Tactc Measurement Stuaton: Large advertsng spend objectve s to optmze spend by tactc and geography 27 Base 43% Contrbuton % FY 2 Incremental 57% Meda m modelng ndcates ncrementalty of meda along wth ndcaton of ROI across tactcs Spend Incremental Sales $2M FY 2 5% 2 % 8% 7% 28 9% Streamng Vdeo 55% 5% 2% % 9% 32% Spend % Increm. Sales % ROI Inde Newspaper Drect Mal TV Rado Dsplay

28 Case Study 3: MMM provdes nsghts nto promotonal performance by regon Meda m modelng ndcates promotonal messagng s more effectve n the Mdwest than all other regons % Promo Tactcs by Regon 9% 8% 33 FY 2 7% % 5% 4% 3% 2% % % 3% 4% 3% 7% 3% % 3% 9% 2% 7% 9% % 2% % 2% 4% 6% 44 9% 4% 4% 28 % Cost % Increm Sales ROI Inde

29 A Best Practce Approach.. Takng marketng measurement to the net level 29

30 Comparson of Methods = Wnner on ths Attrbute ANCOVA Meda M Modelng Cost Hdden Costs Complety of Eecuton Data Requrements (Depends on # groups) Cost of wthholdng promoton from control Statstcs smpler, but test desgn more comple Measurement Ablty Scenaro Plannng Only to repeat eact Best for... 3

31 Best Practce Measurement Framework Meda M Modelng gves best practce estmates of meda mpacts both overall and at the vehcle level. The methodology s also etensble to the tactc level, and can be appled n cases where ndrect or drect attrbuton s not feasble. Indrect/Drect Attrbuton s best employed n relatve analyses wthn a meda vehcle, at levels of granularty not possble va tradtonal m modelng (.e. search keywords). Measurement Level: Legend: Category Meda Vehcle TV Meda DRTV -8 Dsplay Base Search Drect Measure usng Meda M Modelng Tactc Creatve Daypart Duraton Creatve Daypart Duraton Creatve Ste Landng Page Ste Keyword Creatve Segment Measure usng Indrect/Drect Attrbuton Last-clck In-market testng (ANCOVA) Ad trackng 3

32 Merkle Analytcs Bran Demtros Assocate Drector Integrated Meda Optmzaton Practce Merkle Analytcs Lynda S Gordon Senor Drector Lfe Scences Analytcs Practce Merkle Analytcs LSGordon@merklenc.com