Business Process Services. White Paper. Price Elasticity using Distributed Computing for Big Data

Transcription

1 Business Process Services White Paper Price Elasticity using Distributed Computing for Big Data

2 About the Authors Rajesh Kavadiki Rajesh is part of the Analytics and Insights team at Tata Consultancy Services (TCS). He has over ten years of experience in Big Data analytics and machine learning, and is currently working on parallelizing machine learning algorithms on the Big Data platform. He has worked extensively on Java, MapReduce, and Big Data related technologies with specialization in retail and social media.

3 Abstract As retailers deal with intense competition and changing customer preferences globally, pricing remains a critical factor that can determine their success. In today's uncertain economy, the consumer decision making cycle is complex and price is a key influencer of their buying decisions. Retailers are compelled to make price changes frequently due to competition, seasonality, offers and promotions, and so on. This requires them to accurately estimate and forecast the responsiveness of demand to these price changes. They need to be able to gauge the price sensitivity, as an increase in price could lead to a decline in sales. Similarly, a decrease in price could lead to excessive demand leading to out of stock scenarios. Price elasticity is a common measure of price sensitivity used by retailers to develop appropriate pricing strategies and enhance sales revenues. Predicting sales or volumes in response to price changes is particularly challenging in the case of large retailers with a global presence and multiple stock keeping units (SKUs). Traditional analytics tools are limited in their capabilities and unable to support the analysis of large data sets with agility across multiple SKUs. This paper proposes the use of log-linear models on distributed computing or Big Data to measure price elasticity of items across millions of transactions. The aim of this method is to help retailers assess price elasticity with agility and accuracy across all the SKUs in their repertoire.

4 Contents Is the Price Right? 5 Interpreting Price Elasticity 6 Choosing the Recommended Model and Framework 7 Addressing Price Elasticity with the Log-linear Model 7 Validating the Log-linear Model 9 Applying the Log-linear Model to Data 10 Using Analytical Models to Determine Pricing 11

5 Is the Price Right? Globally, retailers acknowledge that pricing is a vital factor in consumers' decision making process. Depending on the type of product, consumers can be very resistant to any price changes. Pricing also has an impact on the positioning of a product. Consumers seek value in a product, which is a function of the perceived benefits and the product's price. We may commonly assume that more consumers will purchase a product when the price is lower. But this notion is negated when consumers assume that a higher priced product provides more quality and value. Hence, pricing is an all-important variable from a retailer's perspective. Intense competition, seasonality, and other such factors compel retailers and e-commerce vendors to amend prices of stock keeping units (SKUs) on a daily basis. There is also great pressure on store managers to predict the sale quantity before the price changes in order to forecast demand and inform suppliers or vendors accordingly to prevent out of stock situations. Price elasticity is a statistical technique that can be incorporated into a software model to predict the sale quantity depending on the percentage of the change in price. The model is designed to predict the sale quantities depending on the price change, promotion offers, seasonality, and changes in competitors' SKU prices. Determining price elasticity is extremely important for supporting retailer pricing decisions aimed at reducing price resistance. The extent to which a product is price elastic impacts the volume of sales and the revenues of a retailer. Therefore, marketing managers need to consider price sensitivity of the market and the impact of price change on the sales revenue. We have the example of an e-commerce retailer who offered huge discounts, resulting in most of the products going out of stock within minutes. This happened because the retailer was not able to judge the surge in demand caused by the drastic reduction in price. This in turn led to customer dissatisfaction and negative feedback all over social media. Retailers thus need to consider building price elasticity models for every SKU across stores to avoid such scenarios. With multiple SKUs in their portfolio, retailers are dealing with large data sets. Retail stores, particularly those spread across countries, have millions of transactions every day with data sizes ranging in hundreds of terabytes. To process this amount of data, traditional statistical software or systems would need to run for days, and in most cases, would run out of memory while analyzing them. Traditionally, therefore, retailers use models built only to handle a few sets of SKUs. Alternatively, store managers rely on their intuition and experience to predict sales volumes. Now retailers can realize better outcomes and accurately evaluate price elasticity of products by leveraging a statistical model and framework that uses distributed computing or Big Data. 5

6 Interpreting Price Elasticity The following formula is generally used to find if the product is price elastic, inelastic or neutral: Price elasticity of demand = Percentage change in quantity demand / Percentage change in price Price Elasticity of demand = (Q₂-Q₁)/[(Q₂+Q₁)] where (P₂-P₁)/[(P₂+P₁)] Q₂ is the current sold quantity after the price change Q₁ is the previous sold quantity before the price change P₂ is the current price after the price change P₁ is the price before the price change This formula gives us three possible scenarios: Scenario 1: Product is Price Elastic If price elasticity is less than -1 it means that the product is price elastic; that is, if there is an increase in price the total revenue falls, and if there is a decrease in price the total revenue increases. Scenario 2: Product is Price Inelastic If price elasticity is greater than -1 and less than 0, it represents that the product is price inelastic; that is, if the price increases the total revenue increases, and if the price decreases the total revenue decreases. Scenario 3: Product is Unit Price Elastic If price elasticity is equal to -1, it represents that the product is unit elastic. This means if the price increases the total revenue remains unchanged, and if price decreases the total revenue remains unchanged. The products that belong to the price inelastic category are those for which there are very few competitive alternatives, such as unique irreplaceable spare parts. So, there is an opportunity for a retailer to increase the price of these commodities and gain more revenue. The products that belong to the price elastic category are those for which there are competitive alternatives available and an increase in price could cause the customers to switch to other brands. An example of this could be consumer goods such as personal grooming products or household products. The total revenue here is the function of price and quantity. The products that belong to the unit price elastic category are the ones for which a unit increase or decrease in price would cause the same unit decrease or increase in demand. Hence a price change would not make any impact on the revenues. 6

7 Price elasticity is not only applicable to retail, but also to various other industries and sectors such as government and taxation. Cigarettes as a product category would fall under the price inelastic category, since there are no alternatives to cigarettes as a product. So if the government tries to increase the price of the cigarettes, the demand would almost remain the same or would decrease by a negligible margin, thus increasing revenue. Another example would be an increase in tax by the income tax department, which affects everyone in the supply chain differently. If there is an increase in tax on the purchase of raw materials, the supplier might choose to pass on the price change to the manufacturer. Now the manufacturer or retailer might decide to pass on the tax burden to the customer or may try to find different ways to absorb the price change, which would be completely dependent on the price elasticity of the product. Choosing the Recommended Model and Framework A distributed computing model using MapReduce framework can be used to measure price elasticity of items across stores and millions of transactions. Since using traditional methods and systems (like SAS, R, MATLAB) would be difficult to process big data sets, this approach uses Hadoop Distributed File System (HDFS) along with MapReduce. Hadoop can scale horizontally, enabling retailers to use a cluster of systems to build models for every SKU in the store. These predictive models can then be used to predict the sales quantity for a price change, and thus, help store managers ensure stock availability before the price change occurs. Furthermore, the recommended solution uses Mahout, an open source distributed machine learning library to overcome the limitation of existing traditional systems. The design and implementation methodology demonstrates the feasibility of this approach to solve price elasticity model using log-linear regression. The solution is generic and can be used with various predictor variables as per business needs. In addition, we also suggest a parallel implementation of the log-linear regression model using the MapReduce programming concept. Log-linear is a powerful technique in regression models, which have become popular in solving price elasticity for retailers. Addressing Price Elasticity with the Log-linear Model Price elasticity generally follows log-linear models as compared to linear models, since the change in quantity does not change linearly with price. Log-linear models are similar to linear models; however both independent and dependent variables are log transformed. Since log is applied on both dependent and independent variables, some researchers call it the log-log model. Price Price Linear Quantity Log Linear Quantity Figure 1 : Comparison of linear and log-linear regression 7

8 Log-linear models can be depicted as follows: Log(sales) = α+β log(target SKU price)+β * I(TPR)+β * I(frontPage)+β * I(Coupons)+β * I(ads) β log(competitor SKU1 price)+β log(competitor SKU2 price)+ β log(competitor SKU3 price) where Total Price Reduction (TPR), front page, coupons and ads are some of the promotional offers that are applicable for a store on a particular day α βn Represents base sales Indicates coefficients of different independent variables I(X) Indicator function that takes value 1 or 0 depending on the promo offer The above equation 1.1 can generally be represented as Y=α+β X β X β X β X β X n n Here, Y is a single dimensional matrix with log normalized sales, and X 0.. n represents n dimensional matrix with each column having unique log normalized feature. α and β coefficients can be found by ordinary T -1 T least squares formula (X X) X Y. T -1 T β = (X X) X Y 1.3 T -1 T The above equation (X X) X Y has an inversion operation which is highly computation intensive for a very large matrix. This can be avoided by parallelizable singular value decomposition. Singular value decomposition (SVD) is the factorization of a matrix into three different matrices (U, D and V). Multiplication of these three matrices (U, D and V) would result in the initial matrix. Equation 1.3 can further be decomposed using SVD as: -1 T β = (V*D *U ) Y 1.4 Here, U is an m by m orthogonal matrix, V is an n by n orthogonal matrix, and D is an m by n diagonal matrix with singular values (U, D, V), which can be obtained from a singular value decomposition of X. SVD is a parallelizable algorithm in Big Data environment. The algorithm to get α and β coefficients using MapReduce is as follows: 1. MapReduce job to append 1s as the initial column of X. T 2. MapReduce job to calculate transpose of X to get X T 3. MapReduce matrix multiplication to get X X T 4. To get inverse of the above matrix, MapReduce job to perform singular value decomposition of X X 5. MapReduce job to take the reciprocal of the diagonal elements in matrix D 8

9 6. MapReduce job to take the transpose of U T 7. MapReduce job to take the transpose of V -1 T T 8. MapReduce job to perform multiplication of V * D * U * X * Y Validating the Log-linear Model The data set is randomly sampled with a 70 to 30 ratio. 70 percent of the data set is used to train the model as described in the previous section, and the remaining 30 percent is used to validate the model. Once α and β coefficients are computed as described, they are substituted back into the test data sample to check the accuracy of the model. As the solution described in this paper doesn't use any sampling techniques and uses distributed computing to decrease the computational time, the accuracy would be the same as the traditional systems. Equation 1.1 can be rewritten as follows to predict sales with change of price α S = e + β (Target SKU₁ Price) ₀+ SKU Price factor β * I(TPR) e ₁ + β * I(Front Page) e ₂ + β * I(Coupons) e ₃ + β * I(Ads) e ₄ + Promotion Offers β (Competitor SKU₁ Price) ₅+ β (Competitor SKU₂ Price) ₆+ Competitor Prices β (Competitor SKU₃ Price) ₇ The sum of square error term is calculated by Where S true N SSE = Sqrt((S true S predicted ) /N) i=1 is the true sales for the day and for a store S is the predicted sales from the equation predicted N is the total number of records or rows in the test data. 9

10 If the accuracy of the predictions is found to be low, the analyst can try to plug in different independent variables that contribute to the sales such as weekends, holidays, seasonality, cannibalization, and so on. Applying the Log-linear Model to Data With the objective of predicting sales quantity, the data set would have log normalized sales quantity as a target vector with three major categories of predictor variables: 1. Price of the targeted SKU 2. Promotion offers for the targeted SKU 3. Prices of competitor SKUs As price elasticity is solved using log-linear regression, all the continuous variables are log normalized and indicator variables are left intact. The prices of targeted SKU and competitor SKU are log normalized. The promotion offers are not log normalized as these are discrete. Each row of the data describes day level characteristics for a store (see Table 1). Store # SKU_Price Total Sales Date TPR Front Page Coupons Ad TR_SKU 1 Price TR_SKU 2 Price TR_SKU 3 Price Table 1: Indicative Dataset Row with Day Level Characteristics for a Store The total sales in Table 1 reflect the sales quantity for the SKU in a store on a particular day. Total Price Reduction (TPR), front page, coupons and ads are some of the promotional offers that are applicable for a store on a particular day. TR_SKU1 price, TR_SKU2 price, TR_SKU3 price are the competitor SKU prices on that day. The recommended model writes a hive SQL script, which in turn runs a MapReduce job to join different tables and aggregate sales on a particular day for a store from millions of transactions. Table 1 is for representation purposes only and the real world scenarios could be different. In the real word, in case of large multinational stores, millions of transactions are conducted in each store and there could be thousands of such stores. In such cases, the model needs to: Aggregate millions of transactions with respect to sales for each store and for each SKU every day Aggregate the total number of daily sales for every SKU across different stores in each row (example: with approximately 1000 stores and for 730 days, the model would generate (1000*730) approximately 0.7 million rows) Perform the above step for every SKU (example: with approximately 10,000 SKUs, the model would have 10,000 different data sets each with approximately 0.7 million rows) 10

11 Perform log-linear regression for each of the datasets (approximately 10,000), which would result in alpha and beta coefficients Substitute the alpha and beta coefficients in real time to predict the sales quantity Using Analytical Models to Determine Pricing With the retail industry witnessing intensifying competition, it has become imperative to gauge the impact of price on the sales volume. Therefore, formulating accurate pricing strategies with the help of a scientific approach is critical from a profitability perspective, for both brick and mortar as well as online retailers. With exponential growth in data, traditional systems are increasingly constrained in processing huge datasets. This has resulted in the shift to distributed computing for analyzing large data volumes. The analyses recommended in this paper were carried out using the three-stage analytical framework including data aggregation, model building, and validation. Log regression has been employed to build models at SKU-day level for computing sales. The columns or independent variables used in this model are for indicative purpose and these variables can be extended to include other characteristics such as seasonality, store level characteristics, and indicators for weekend or holiday sales. Retailers can use these characteristics and many more depending on their business model to accurately predict sales for a given price change. This accuracy could help separate the winners from losers in an increasingly competitive retail environment. 11

12 About TCS Business Process Services Unit Enterprises seek to drive business growth and agility through innovation in an increasingly regulated, competitive, and global market. TCS helps clients achieve these goals by managing and executing their business operations effectively and efficiently. TCS Business Process Services (BPS) include core industry-specific processes, analytics and insights, and enterprise services such as finance and accounting, HR, and supply chain management. TCS TM creates value through its FORE simplification and transformation methodology, backed by its deep TM domain expertise, extensive technology experience, and TRAPEZE governance enablers and solutions. TCS complements its experience and expertise with innovative delivery models such as using robotic automation and providing Business Processes as a Service (BPaaS). TCS' BPS unit has been positioned in the leaders' quadrant for various service lines by many leading analyst firms. With over four decades of global experience and a delivery footprint spanning six continents, TCS is one of the largest BPS providers today. Contact For more information about TCS' Business Process Services Unit, visit: [email protected] Subscribe to TCS White Papers TCS.com RSS: Feedburner: About Tata Consultancy Services (TCS) Tata Consultancy Services is an IT services, consulting and business solutions organization that delivers real results to global business, ensuring a level of certainty no other firm can match. TCS offers a consulting-led, integrated portfolio of IT and IT-enabled infrastructure, engineering and TM assurance services. This is delivered through its unique Global Network Delivery Model, recognized as the benchmark of excellence in software development. A part of the Tata Group, India s largest industrial conglomerate, TCS has a global footprint and is listed on the National Stock Exchange and Bombay Stock Exchange in India. For more information, visit us at IT Services Business Solutions Consulting All content / information present here is the exclusive property of Tata Consultancy Services Limited (TCS). The content / information contained here is correct at the time of publishing. No material from here may be copied, modified, reproduced, republished, uploaded, transmitted, posted or distributed in any form without prior written permission from TCS. Unauthorized use of the content / information appearing here may violate copyright, trademark and other applicable laws, and could result in criminal or civil penalties. Copyright 2015 Tata Consultancy Services Limited TCS BPS Design Services I 03 I 15