The Operational Value of Social Media Information. Social Media and Customer Interaction

Transcription

1 The Operational Value of Social Media Information Dennis J. Zhang (Kellogg School of Management) Ruomeng Cui (Kelley School of Business) Santiago Gallino (Tuck School of Business) Antonio Moreno-Garcia (Kellogg School of Management) Social Media and Customer Interaction Social media is computer-mediated tool that allow people to create, share or exchange information in virtual communities and networks 83% of Forbes 5 companies use at least one of the social media sites to interact with customers. Top 5 retailers in U.S. earned $3.3 billion from social shopping in 214 (26% higher than 213) 2 1

2 11/17/15 Facebook Facebook accounts for 65% of total social revenue. 3 Research Question In practice with limited data, can we improve sales forecast accuracy and in turn inventory management by incorporating publicly available social media information? If so, by how much? How should we select forecasting models when incorporating social media information? How should we extract features from social media information to improve forecast accuracy? Roadmap Data Features Algorithms Framework Results and Implications 4 2

3 Sales and Social Data (213/1 213/8) Online Apparel Company Sales Operational Data Daily sales and revenue data (2 years) The company s promotion and marketing data (1 year) The company s own daily sales forecast Jan Feb Mar Apr May Jun Jul Sales, Comments, Posts Normalized Comments, Posts and Sales (Data with Forecast) Social Media Data (from Facebook API) The company's 1.5k posts, 24k comments, and 22k likes from more than 7k Facebook users (2 years). Jan Feb Mar Apr May Jun Jul 5 Forecasting Framework Features Features: Basic Features (per day) Past sales, advertisement, promotion data, seasonality Non-textual Features Number of posts, number of comments from users, number unique users visited, etc. Social Features (per day) Textual Feature Number of unique words in comments, average sentiment of comments, etc. For each day: Features i 3 Features i 2 Features i 1 Sales i Sales i = f(features i 1, Features i 2, Features i 3 ) In our algorithm, we use past 7 days data, each day has 4 features. Therefore, our forecasting function takes 7 * 4 = 28 features in total. Socher et al. (213) 6 3

4 Forecasting Framework Forecast Algorithm Forecasting models (linear and nonlinear, with variable selections): Linear models: linear regression, Lasso and forward selection Support vector machine: Linear and Radial Kernel Ensemble models: Gradient boosting model and Random forest. Sales at t-1 Tree 1 Tree 2 Tree N < 5 5 Sales at t = 5 Sentiment at t-1 < 1 1 Sales at t = 4 Sales at t = 6 Voting Leo Breiman (21), Anyd Liaw and MaBhew Wiener (215). 7 Forecasting Framework Forecast Framework Training Period: 9 days Testing Period: 45 days Training Period: 1-fold cross validation with 3 repeats (Industry standard) to tune hyper-parameters. Performance Metrics: Mean Absolute Percentage Error: MAPE = 1 P N N i=1 P N Root Mean Squared Error: RMSE = i=1 (F i S i ) 2 q 1 N Fi Si Si Testing Period: We update our forecast models every 5 days. (Test period = 45 -> We have /5 = 1 models in each testing period). Prediction Horizon (1 7): the number of days ahead we predict. (i.e., Prediction Horizon = 2 ó We predict the sales of the day after tomorrow) 8 4

5 . Sales Jan Feb Mar Apr May Jun Jul Jan Feb Mar Apr May Jan Jun Feb Jul Mar Apr May Jun Jul Jan Feb Mar Apr May Jun Jul Comments Posts Sales Sales 11/17/15 Forecasting Framework Information Value Basic Features Basic Features Basic Features Social Features ML model+ Framework Unknown model + Framework Company Proprietary Information Social Media Forecast Baseline Forecast Company Forecast Value of Social Media Information Value of Machine Learning Models 9 Forecasting Main Results Model: Random Forest (with number of trees and tree depth as tunable variables) Training: 213/1/1 213/4/1 (9 days) Testing: 213/4/2 213/5/6 (45 days) 12 MAPE of Sales Forecasts over Different Prediction Horizaons MAPE (%) %* 23.%*** 9.5% 2.4%** 27.3%*** 15.5%** 18.5%** model Base Forecast Social Forecast Company Forecast Prediction Horizon (days) 1 5

6 Different Starting times MAPE of Sales Forecasts over Different Starting s (Prediction Horizon = 1) 2 MAPE (%) %** model Base Forecast Social Forecast Company Forecast 1 23.%*** 18.7%** 14.%* 13.6%** 5 Jan 1 Jan 15 Feb 1 Feb 15 Mar 1 Starting Value of Social Information is significantly positive throughout different training and testing periods. 11 Different Machine Learning models MAPE of Sales Forecasts over Different Models (Prediction Hoizon = 1) 15 MAPE (%) 1 5 Linear Models SVM Ensemble Models Data base social Linear RegressionForward Selection Lasso SVM (Linear) SVM (Radial) GBM Random Forest Social media Information is valuable if the statistical model : Is nonlinear in features Has variable selection 12 6

7 Model Inspection How important is social media information in forecasting models? Random Forest: Weighted-average Gini Index. 12 Top 2 Features with Highest Gini Importance 9 Past Sales Gini Importance 6 Sentiment Past Comments Type Base Non textual Textual 3 Features Social media information is important in forecasting. Textual processing of social media information is helpful. 13 Take-Away Social Media information helps! In practice with limited data, we can reduce MAPE by 15% when incorporating social media information. Forecasting models that are nonlinear and can actively select variables will benefit more from incorporating social media information. Only counting of social media activities are not enough. Textual analysis of comments and posts are also, if not more, important. 14 7

8 Model Inspection How important is social media information in forecasting models? Random Forest: Weighted-average Gini (impurity) Index. For example, there is a box of balls, half of them are red and half of them of green. We want to predict the color of the ball base on two binary features A and B. All balls with A = 1 are red. Half of balls with B = 1 are red. Which feature is more important?.5*.5= A 1 GI = B 1 GI = - Gini index: Gini impurity index: G = P n c i=1 p i(1 p i ) Gini index of a node: I n = G parent Pi G split i 13 Model Inspection Forecast Algorithm How important is social media information in forecasting models? Lasso: how many variables left from basic features and social features? 1 lasso = 1 NX X px (y i px ij j ) 2 + j A 2 i=1 j=1 j=1 We use a simple and efficient Lasso implementation: Least Angle Regression (Efron et al. 23) We can inspect the model now to see which variables are more correlated with our forecasted sales. 15 8

9 Model Inspection How important is social media information in forecasting models? Lasso: how many variables left from basic features and social features? 3 Number of Features in Lasso Regression Number of Features % 44.1% 44.1% Base Non textual Textual Feature Type Social media information is important in forecasting. Textual processing of social media information is helpful. 16 InLasso FALSE TRUE Least Angle Regression lasso = 1 2 NX X (y i px ij j ) 2 + i=1 1. Find the most correlated x i. 2. Move i to the least square coe cient until some x j becomes as correlated with the residuals as x i. 3. Move ( i, j) to the joint least square coe cients of the residuals until some x k becomes as correlated as x i and x j. 4. Repeat 2, 3 until we selected k variables. j=1 1 px j A j=1 18 9