Understanding multivariate testing techniques and how to apply them to your email marketing strategies An Experian Marketing Services white paper
Testing should be standard operating procedure Whether it is in search, display, direct mail, email, mobile or social, one would be hard-pressed to find a marketer today who isn t aware of the concepts of A/B or multivariate testing (MVT) and the value they offer. With the amount of data that is being captured, stored, processed and analyzed in today s world, combined with the speed at which results boomerang back to the tester, we operate in a time and place where there is no excuse not to continually optimize your marketing campaigns via some sort of testing procedure. Even the president tests Testing has indeed become mainstream. Highlighting this fact is a recent article in Businessweek, which details the use of statistical testing by the Obama reelection campaign when trying to solicit donations from their email subscribers. The article describes how the campaign staff would initially experiment with a handful of different subject lines for their emails. After sending these seed emails out to a limited number of subscribers, the team would wait for the results (amount of money donated) to trickle in before it became clear which email subject lines performed best. Now equipped with the knowledge of what emails would perform significantly better than others, the team would send the winning emails out to the rest of the millions of email subscribers on the president s list, arguably increasing the donations haul from email marketing by millions of dollars. What s being tested? Testing ranges in complexity. A/B and A/B/C testing are the most common forms of testing used in marketing today. These types of tests are fairly straightforward to perform and interpret, as they focus solely on measuring the effects of changing values from only one factor. As shown in the graph on the following page from Experian Marketing Services December 2012 Email Market Study, where email marketers across eight verticals were surveyed about their email marketing initiatives, most companies today are leveraging A/B and A/B/C testing to optimize different facets of their email campaigns (subject line, frequency, time of day, day of week, call-to-action, etc.), but very few (11 percent) are leveraging the more complicated, yet more powerful, tool of multivariate testing to optimize their email marketing efforts. Page 1 Understanding multivariate testing techniques and how to apply them to your email marketing strategies
What type of testing do you perform on email campaigns? 100% 97% 80% 81% 60% 40% 39% 50% 36% 43% 28% 20% 11% 21% 11% 11% 0% Subject line Creative Frequency Time of day Day of week HTML versus text Call-toaction Product Number placement of products Friendly from Multivariate Source: Experian Marketing Services Email Market Study, Acquisition and engagement tactics, December 2012 A common A/B testing mistake A common mistake when implementing A/B testing is performing sequential A/B tests in an effort to arrive at optimal levels for multiple factors. These experiments often start with the standard or status quo settings of the key factors to be tested. The levels of the factor that is believed to be the most responsible for performance are tested first, while the other factor levels remain constant. After the responses have been gathered and the optimal level for the first factor is determined, the factor regarded as the second most influential is tested next, with the optimal first level factor remaining fixed for the rest of the experiment. This process continues to repeat itself until each factor level has been individually tested. To better illustrate this flawed process, consider the following example with two factors, each with just two levels (the simplest case possible). Suppose an organization wants to determine the best image and ad copy to put in an email with the click-to-open ratio being the metric to maximize. We will denote the different images as I 1 and I 2 and the different ad copy as C 1 and C 2. In their first email blast to 20,000 customers, the company decides to send half of these customers an email with the combination (I 1, C 1 ) and the other half with the combination of (I 1, C 2 ). An Experian Marketing Services white paper Page 2
The results are as follows: Click to open percent: (I 1, C 1 ) = 7.5 percent (I 1, C 2 ) = 8.5 percent Based on these results, the company believes that Copy 2 is the preferred ad copy and fixes the next email blast to 10,000 more customers at this level so that the next test in the sequence will only vary the image level not already tested. Click to open percent: (I 2, C 2 ) = 9.5 percent Seeing these results, the company decides that (I 2, C 2 ) is the optimal combination in terms of being able to generate the highest click-to-open ratio. The problem is that the company may have missed out on finding the global optimum by not testing the fourth combination of (I 2, C 1 ), which may have yielded a click-to-open ratio even greater than 9.5 percent. A/B tests assess one level of one factor versus the control group, but cannot measure the interaction effect across factors. The reason this method of sequentially testing one factor at a time fails to find the optimal factor levels is that an interaction effect exists between factors 1 and 2. That is, the factor effects are not additive, but rather the combinations among different factors and their levels produce an additional effect (interaction) when used simultaneously. By not being able to capture interaction effects, this sequential approach may miss the optimum altogether. In situations such as these, it is more appropriate to perform a multivariate test (factorial test to be exact), where all factors are changed together and all combinations are accounted for. There is a wide array of different kinds of multivariate tests available, but when the number of factors and the number of factor levels to be tested are limited, the full factorial approach is the best option, as it retains the most amount of information about the factors. Page 3 Understanding multivariate testing techniques and how to apply them to your email marketing strategies
Multivariate testing: a brief history Multivariate testing has its roots in the statistical discipline of experimental design. Experimental design is the process of planning a study to meet specified objectives. Planning an experiment properly is important in order to ensure that the right data and sufficient sample sizes are available to answer the research question of interest, as clearly and efficiently as possible. One of the first recorded applications of experimental design comes from the early 20th century and a statistician named R.A. Fisher. Fisher was hired by an agricultural research center to determine the effects that various factors (soil type, sun exposure, rainfall, fertilizer, etc.) had on plant growth. By designing proper, randomized controlled experiments before any seeds had been planted into the ground, Fisher was able to harvest results that allowed him to determine the optimal levels of each factor that would maximize plant growth and crop output. His work proved seminal, and others soon found good use of experimental design in a wide variety of industrial and manufacturing problems. Only recently have brands discovered the power of experimental design (now termed multivariate testing) in optimizing marketing campaigns. Advantages of MVT over A/B testing Multivariate testing has several advantages over that of A/B or A/B/C testing. In A/B or A/B/C testing, it is easy to determine which particular creative performed best at generating the most amount of conversions, but this type of testing can t tell you why one creative performed better than the next. Was the increase in conversions due to the change in background color, the call to action, the main image, etc.? MVT allows you to measure the size of the effect that each of these factors has in generating conversions and provides guidance on where to focus your next round of testing efforts. An Experian Marketing Services white paper Page 4
The graph below is an example of what may be found through MVT testing: Factor influence on click rates Layout 42.0% Main image 39.0% Factors Call to action 10.8% Size 5.9% Color 2.3% 0% 10% 20% 30% 40% 50% Percent effect MVT also saves the tester both time and money, as it tests multiple factor-level settings simultaneously. Instead of being limited to varying the values of only one factor, MVT, as its name implies, allows you to test multiple factors at the same time, A/B thereby eliminating the need to run multiple MVT A/B tests. This increased efficiency gives the tester the ability to find the optimal factorlevel settings more quickly, and the quicker the optimal settings are found, the sooner results improve and return on investment increases. Lastly, as noted in the earlier example, MVT provides the decision maker with better information as potential factor-level interaction effects can be captured. It is most likely that the different factors that make up your email campaigns don t work in isolation, meaning synergistic effects do exist among the different images, colors, calls to action, sizes, etc. that make up your email messages. The phrase The whole is greater than the sum of its parts rings true in these situations. For example, you may have a particular image that performs OK on its own, and a particular background color that also performs OK on its own, but when used together, these particular settings create an email that performs amazingly well. This type of interaction information cannot be captured via A/B or A/B/C testing because only one factor is being analyzed at a time. By setting up tests that more Page 5 Understanding multivariate testing techniques and how to apply them to your email marketing strategies
accurately account for the true multidimensional nature of your data, you will be able to gather deeper analytic insights that just aren t possible to collect via A/B or A/B/C testing. Applications of multivariate testing Email creative testing There are many different dimensions of an email creative that could possibly affect subscriber engagement: layout, main image, call to action, color, size, etc. Depending on the number of factors you are interested in testing and the amount of variations you have to test for each factor, the overall number of potential creative combinations can quickly become rather daunting. For example, if you are interested in assessing the effects of two different layouts, two different main images, five different calls to action, two different background colors and two different sizes, you are left with 80 different email combinations to test. However, through the science of experimental design, there exists ways to reduce this number of combinations to test down to a much more reasonable figure. In fact, a properly designed multivariate test can reduce the number of combinations to test from 80 down to as few as 16, a much more manageable situation. Another benefit to this approach is that the tester loses very little in terms of back end, actionable insights when reducing the number of creative combinations to be tested in an intelligent manner. In other words, the ability to identify which factors have a significant effect (and conversely, which factors show little to no effect) on user engagement is still able to be captured, along with the identification of what the optimal settings of the email factors should be in order to maximize subscriber response/engagement. Single factor A/B Full factorial design Fractional factorial design Layout Layout X X X Call to action Buy now Act now Get more info Image X X X X X X X X X Call to action Image X X Call to action X Easy to manage, but slow and narrow Broadest learning, but expensive and often infeasible Complex to design, but most efficient and effective Source: Experian Marketing Services An Experian Marketing Services white paper Page 6
Time of day/day of week testing The following graphs show at an aggregate level what was historically observed as the best days and times for an email deployment in Q4 2012. Charts with the best days and times by major industry can be found in the appendix section of this report. Days Monday Thuesday Wednesday Thursday Friday Saturday Sunday 0.0% 20.0% Unique opens Day of week performance Q4 2012 all industry Volume % 17% 16.6% 2.7%.12% $.17 $173 15% 16.8% 2.7%.12% $.16 $189 16% 16.7% 2.6%.11% $.13 $188 15% 16.5% 2.5%.11% $.14 $195 17% 16.4% 2.6%.13% $.16 $179 9% 17.8% 2.9%.16% $.20 $176 10% 17.8% 2.9%.14% $.19 $189 0.0% 4.0% Unique clicks 0.10%.20% $.10 $.20 Trans. rate Rev. per email $0 $100 $200 Average order % of Range name Volume Transactions 12:00a.m. 3:59a.m. 12% 14% 4:00a.m. 7:59a.m. 20% 22% 8:00a.m. 11:59a.m. 40% 42% 12:00p.m. 3:59p.m. 15% 12% 4:00p.m. 7:59p.m. 10% 9% 8:00p.m. 11:59p.m. 2% 2% Time of day performance Q4 2012 all industry % of 17.6% 3.2%.20% $.22 $135 16.2% 2.5%.15% $.18 $164 16.1% 2.4%.13% $.17 $174 17.6% 2.8%.13% $.15 $188 18.2% 2.9%.15% $.16 $182 21.7% 4.2%.34% $.48 $246 0.0% 30% 0.0% 6.0% 0.0%.40% $0 $.60 $0 $300 Unique opens Unique clicks Trans. rate Rev. per email Average order Source: Experian Marketing Services Page 7 Understanding multivariate testing techniques and how to apply them to your email marketing strategies
Looking at these results, we see that Saturday and Sunday had higher open, click and transaction rates, but much lower volume. For weekdays, Monday had the highest revenue per mail, but Friday had the better click rate. Volume also plays a role in time of day, as 8 p.m. to midnight and midnight to 4 a.m. had higher responses than the more popular 4 a.m. to noon times. However, as every brand s customers behave differently, it is most likely the case that the optimal day of week and time of day for your email deployments will show a different pattern of performance. It is important to remember that the data from the aggregate graph is retrospective, not a controlled study. When designing a time of day/day of week study for your subscribers, it is imperative that the various day/time groups are defined ahead of time and in a manner that incorporates random sampling techniques. A properly designed MVT will give you clarity into what matters most in terms of deployment decisions and which day/time combinations have proven themselves to be statistically significant in their ability to provide improved metrics for your business. Frequency and recency testing The question of How often to mail? cannot be answered without a comprehensive testing plan. Trying to assess the effects of frequency or recency on retrospective data has a much higher risk of confounding or misallocating performance due to other subscriber attributes like engagement or tenure. To help remove this noise from the process, MVT can be leveraged to create proper test groups beforehand, as well as identify what sample size requirements need to be met in order for results to be statistically significant. The use of forwardlooking multivariate test designs versus backward-looking retrospective analyses will provide you with the assurance that the conclusions drawn from your study are accurate and statistically meaningful determinations of the optimal delivery settings for your subscribers. When testing for optimal frequencies, it is also important to remember that not all engagement segments should be exposed to the same amount of experimentation. Your engaged audience is already demonstrating desirable levels of performance, and you want this behavior to remain. You don t want to run the risk of disrupting this routine with too many frequency tests. At the other end of the spectrum, you do not have much to lose when testing against your inactive audience; there is only upside to be gained. Take advantage of this situation, and test against a wider range of subscribers to see if any different patterns of frequency tend to spur added levels of engagement. An Experian Marketing Services white paper Page 8
Email cadence Highly engaged Test conservatively Moderately engaged Inactive Test liberally Segment Highly engaged Moderately engaged Inactive Sample learnings Current frequency/ recency 12x per month/ evenly staggered 12x per month/ evenly staggered 12x per month/ evenly staggered Optimal frequency/ recency 15x per month/ evenly staggered 10x per month/ evenly staggered 2x per month/ clustered For more information on how to re-engage inactives, download Experian Marketing Services reactivation white paper. Page 9 Understanding multivariate testing techniques and how to apply them to your email marketing strategies
Conclusion Remember, testing all the various aspects of your email program for optimal performance is an important tactic for any successful email marketing campaign. The testing process is also iterative, with no prescribed end. As the business environment and goals of your brand change alongside the needs and wants of your customers, it is important to continually test and fine-tune user experiences on a regular basis to ensure your marketing messages are primed for maximum effect. For more information on how to design and execute multivariate testing with your email marketing programs, please visit our website at www.experian.com/marketingservices or call us at 1 866 626 6479. An Experian Marketing Services white paper Page 10
Experian Marketing Services 955 American Lane Schaumburg, IL 60173 1 866 626 6479 www.experian.com/marketingservices 2013 Experian Information Solutions, Inc. All rights reserved Experian and the Experian marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc. Other product and company names mentioned herein are the property of their respective owners. 03/2013