DIRECT MAIL: MEASURING VOLATILITY WITH CRYSTAL BALL

Transcription

1 Proceedings of the 2005 Crystal Ball User Conference DIRECT MAIL: MEASURING VOLATILITY WITH CRYSTAL BALL Sourabh Tambe Honza Vitazka Tim Sweeney Alliance Data Systems, 800 Tech Center Drive Gahanna, OH ABSTRACT This paper covers the concept of and need for a stochastic profitability evaluation to appropriately assess the risk associated with retail direct marketing efforts. Through the application of Crystal Ball simulation, the measures necessary to build a financial model are identified and volatility measurement in direct mail performance is demonstrated. Examples are used to demonstrate how Crystal Ball can be used to improve customer targeting and messaging. 1 INTRODUCTION 1.1 About Alliance Data Systems Alliance Data Systems is a provider of transaction services, credit services and marketing services in North America. The Company focuses on facilitating and managing electronic transactions between its clients and their customers through multiple distribution channels including in-store, catalog, and the Internet. Its credit and marketing services assist clients in identifying and acquiring new customers, as well as helping to increase the loyalty and profitability of their existing customers. Alliance Data works with over 350 specialty retailers, petroleum retailers, utilities, supermarkets and financial services companies. Alliance Data s marketing services group manages direct mail initiatives for many retail clients, which has proven to be an effective and reliable way of marketing to retail customers. The personalized method of direct marketing is efficient for increasing brand awareness as well as helping to create additional revenue. 1.2 Direct Marketing and Incremental Revenue While evaluating the performance of a direct mail campaign, it is important to note that only revenue that would not have been generated without direct mail communication is incremental. Therefore, correct measurement of the direct mail impact requires a measure of baseline performance. This can be achieved by holding out a statistically matched group of customers from the direct mail promotion. This group is called a control group. The incremental measurement is performed by comparing this control group against a matched test group or mailed group. First, it is important to select the correct customers for a promotional opportunity since executing a direct mail campaign can involve significant costs. Identifying profitable customers and targeting them in a correct way is the key to a productive campaign. To achieve this, the customer base is usually divided into clusters or segments of similar customers based on different characteristics that can include transactional behavior, demographic information or attitudinal attributes. To identify profitable segments and to mitigate risk, direct marketers normally test a small group of customers before a new direct mail offer is rolled out to the total population. Incremental revenue in direct mail campaigns can be associated with the increase in traffic driven by the promotional offer and/or by the additional spend of the responding consumers. The combined effect of these two factors determines the success of the effort. Point-of-sale data collected at the registers can be stored on a marketing database from where a detailed financial analysis of the direct mail campaign can be generated. The financial model is based on the actual response rate to the direct mail offer and spend of the responders. The product of these two metrics determines the total revenue generated by each segment. The difference between the revenue generated by the test group and the revenue generated by the matched control group (receiving no incentive) equals the incremental revenue generated by the direct mail campaign. Direct marketers can calculate profit for the campaign by subtracting the costs associated with the campaign from incremental revenue. The

2 ratio of profit to the costs incurred gives the return on investment (ROI) for the campaign. Therefore, in the financial model for the direct mail campaign, response rate and spend per responder are the two inputs, while incremental revenue, profit and ROI are the performance metrics. 1.3 The Volatility Problem While this analysis gives detailed insight into the campaign performance, it is also illogical to assume that the exact same performance can be repeated. Marketers must also validate that the performance metrics achieved have some level of associated certainty. This is necessary to eliminate the possibility of achieving results merely by chance. This certainty level issue becomes even more important when there are limited quantities available for the test. The solution is a stochastic evaluation of the direct mail campaign performance. Direct marketers should account for the volatility associated with the campaign performance metrics in order to assess the true associated risk and perform some Worst - Best case scenario type of analysis. 2 STOCHASTIC MODEL AND TOOL SELECTION A true stochastic profitability evaluation model looks at the effect of randomness the two inputs (i.e., response rate and spend per responder) have on the final incremental revenue and profit numbers. This randomness can be captured using the appropriate probability distributions from the historical data. An important thing to note here is that the distribution parameters can change considerably across different segments in a particular campaign. Segmentation creates an intelligent bias in the whole customer universe in such a manner that no segment is a true representative of the whole customer population. This creates a need to account for randomness at the individual segment level. Figure 1: Conceptual stochastic profitability model Figure 1 represents the conceptual stochastic model for the measurement of direct mail campaign profitability. The figure represents one segment within a campaign and the same logic applies to all other segments. In this example, response rates for the test segment and control segment overlap, with the mean response rate for the test segment being higher. On the other hand, mean spend per responder for the control segment is slightly higher than that of the test segment; this is because test segments receive discounts which drive their spend down. The combined distribution of response rate and spend per responder generates the distribution for spend per customer in the campaign. (This is an important metric because costs incurred are for all the customers in the campaign and not only responders.) In this example, the resultant mean spend per customer for the test group is higher than the mean spend for the control group. This is because the positive response lift generated by additional test responders is more significant than the negative spend per responder. Therefore, the original deterministic financial model will show that this segment generated additional revenue. However, the stochastic model looks at the resultant distribution generated by subtracting spend per customer distribution for the control group from the same distribution for the test group. This new distribution is represented as the distribution for incremental revenue per customer, which tells much more than the deterministic value and gives the range for the incremental revenue. Breakeven analysis allows individuals to calculate the probability of getting positive revenue from this segment. This is very useful when making decisions about optimal campaign circulation. Issues can arise when implementing this model using pure statistical theory so specific steps must be taken to mitigate the risk. First, each segment requires customized distribution. Additionally, the whole calculation of incremental revenue requires multiplications and subtractions of different distributions. This is very difficult to achieve analytically and also requires lot of approximations. These approximations also make the model less accurate and less representative of the real world. On the other hand, implementing this model using Monte Carlo simulation is a more accurate solution as it can handle different distributions and the required algebraic operations on them without any approximations. With a sufficient number of trials, this method can be trusted to give accurate results.

3 Crystal Ball was our obvious choice for this implementation because of the following two reasons: Crystal Ball can handle Monte Carlo simulation and gives the direct marketers the ability to use many probability distributions. Crystal Ball can be integrated with Microsoft Excel and can be automated easily. This was very important because the original financial model was already built in Microsoft Excel. 3 MODEL BUILDING IN CRYSTAL BALL 3.1 Defining Assumptions The first step in the modeling process is to decide the assumption and forecast cells and to determine if any of the correlations need to be incorporated in the model. For each segment there are two input assumptions that drive the total incremental revenue for that segment. Figure 2 shows how these distributions were defined in the final model. The response rate is a binary variable and follows a binomial distribution. To model this in Crystal Ball, however, we had to use a normal approximation of binomial because number of trials parameter for binomial distribution in our cases was higher than the Crystal Ball limit. With the higher number of customers in a segment, this approximation is very close to the real distribution. In most cases, having considerably higher sample sizes, spend per responder also follows the normal distribution. Mean and standard deviation for both of these parameters are calculated at the segment level using the transactional data stored on a marketing database. Assumption cells are defined in the model using these numbers. Figure 2: Defining assumptions in the model

4 3.2 Defining Forecasts There are three forecasts in the model at the segment level: Revenue per customer for the test group Revenue per customer for the control group Incremental revenue per customer. Figure 3 represents how these forecasts are generated with the given assumptions. The product of the response rate and spend per responder generates the distribution for revenue per customer. The difference between the revenue per customer for test group and control group generates the distribution for incremental revenue per customer. It is important to note here that the response rate and spend per responder are independent of each other. Based on the offer and seasonality effects, it is possible to have more response with less spend and vice versa. Therefore, in the model, no correlation has been put between the two input assumptions. Cost is subtracted from the incremental revenue to generate distributions for the profit and ROI. Test Group Response Rate Revenue / Responder Revenue / Customer Control Group Inc Revenue / Customer Response Rate Revenue / Responder Revenue / Customer Figure 3: Report flow 3.3 Run Parameters and Automation We tested the model to determine the optimum number of trials. In the process we found that after 10,000 trials, results seem to be stable and there is almost no change in the resulting distribution beyond this point. Therefore, we decided to run 10,000 trials for every simulation. The final report generated by the model is in a spreadsheet format and does not contain any graphics. To facilitate the run speed all the graphics are suppressed and for the ease of use, the simulation run has been automated using Visual Basic for Application macros. Even at 10,000 trials and a high number of segments (up to 100), the run time remained reasonable (less than 5 min) on a PC with 2.26 GHz processor and 512 MB RAM.

5 4 APPLICATION The stochastic model using Crystal Ball simulation has several applications in direct marketing. The first is assessing the volatility of direct mail campaign results. Our initial financial model supported results as point estimates, which are the actual results of a campaign. For smaller segments, traditional financial analysis can be misleading; therefore stochastic modeling analysis becomes a necessity to assess the risk and opportunity associated with those segments. To illustrate this, see example in Figure 4. This example shows two customer segments from the same campaign. As a point estimate, traditional analysis would show that these two segments performed exactly the same in terms of incremental revenue. However, if we analyze the volatility of these segments, we can see that the segment A has much more volatility than segment B; Therefore segment B has a greater probability of achieving positive incremental revenue. The stochastic model helps us avoid the mistake of stating that both of these segments will perform equally, a distinction that is critical when deciding to move from test to rollout. 35% 65% - Incremental 5% + Incremental 95% Figure 4: Illustrating the effect of volatility In direct marketing, companies tend to anniversary their marketing campaigns, so another application of stochastic analysis can be assessing the repeatability of a campaign. Stochastic analysis makes the forecasting much more reliant on previous performances than intuition. For example, a holiday gift with purchase campaign in which customers receive a free item with a purchase requires stores to a have an inventory of gift items in stock. Forecasting through stochastic modeling can better estimate the gift units that will be needed to avoid excess inventory or dissatisfied customers. 5 SUMMARY Volatility assessment of a direct marketing campaign s performance metrics provides important insights for future campaign planning. The stochastic evaluation model explained here is an effective way to assess volatility in direct marketing efforts. Implementing this model using classical statistical methods is difficult due to analytical complexities and approximations. Monte Carlo simulation is used to implement this model in the Crystal Ball to help direct marketers understand the true risk associated with their direct marketing efforts. BIOGRAPHIES Tim Sweeney is the Senior Manager of Customer Reporting and Analysis within Alliance s Consumer Database Marketing Services Group. Tim is primarily responsible for developing innovative analytics and analytical processes to further clients insight into their customers behavior. His team performs a wide variety of analytics including Customer Profitability Studies, Demographic Profiling, Demand Forecasting and a variety of consumer-product interaction studies. Prior to managing this team, Tim worked with several large retail soft goods clients performing high-end analytics to drive customer insights.

6 Tim also played an instrumental role in the development and implementation of a new credit marketing database to support Alliance Data s retail business and the more than 85 million private label credit accounts the company owns. Prior to joining Alliance Data Systems, Tim worked for Star Bank in Cincinnati as a Financial Analyst. Tim earned a BBA in Finance and Real Estate from the University of Cincinnati in In 1999 he returned to UC where he earned an MBA with concentrations in Finance and Marketing. Honza Vitazka is a Manager of Adhoc Reporting and Analysis in the Consumer Database Marketing Group at Alliance Data Systems. Honza has nearly five years of experience in CRM analytics in the specialty retail industry. Honza leads a team that focuses on ad hoc customer analytics and direct marketing campaign analysis. Honza earned an MBA degree from the University of Cincinnati in Sourabh Tambe is a Lead Database Marketing Analyst in Consumer Database Marketing Services at Alliance Data Systems. Sourabh s primary responsibilities include conducting direct mail campaign analysis and designing new analytics for campaign performance measurement. Sourabh has played a key role in consolidating and improving Alliance Data s campaign reporting suite. Prior to joining Alliance Data, Sourabh worked in performance measurement and improvement using simulation modeling in various software packages. Sourabh earned an MS degree in Industrial Engineering with a focus on applied statistics and operations research from the University of Cincinnati in 2004.