Page 1 of 20 ISF 2008 Models for Product Demand Forecasting with the Use of Judgmental Adjustments to Statistical Forecasts Andrey Davydenko, Professor Robert Fildes a.davydenko@lancaster.ac.uk Lancaster University Management School 1 Contents The Importance of Demand Forecasting at SKU Level... 2 Demand Forecasting Software... 3 Reasons for Using Judgmental Adjustments... 4 A Typical Process of Making Adjustments... 5 A Typical Process of Making Adjustments (Forecast Pro)... 6 The Need for Modelling of Adjustments... 7 The Formulation of the Problem... 8 Typical Features of the Data... 9 Previously Suggested Approaches...10 The Proposed Approach...11 Main Modelling Features...12 Handling Special Situations...13 Data Used for Evaluation...14 Results of Modelling and Evaluation...15 Research Summary...16 Next Steps...17 Innovative User Interface to Support Demand Forecasting with Judgmental Adjustments...18 References...20
Page 2 of 20 Importance of Demand Forecasting at SKU level Accurate forecasts are essential for efficient operations & high levels of customer service Consequences of inaccurate forecasts: Underestimation Poor service Additional ordering expenses Overestimation Dead stock Losses of investments 2 Nowadays the role of demand forecasting is becoming more important as companies are trying to find more efficient ways of managing their operations in modern highly-competitive business environments. In particular, this relates to forecasting at the stock-keeping unit (SKU) level or at the level of single products. Inaccurate forecasts of product demand in case of underestimation can lead to consequences, such as poor level of customer service, out-of-stock items, and corresponding additional expenses; while overestimation leads to overproduction or overstocking, which eventually means lost investments and lost resources.
Page 3 of 20 Demand Forecasting Software allows to automatically generate forecasts using extrapolative methods, such as ARIMA, exp. smoothing, curve fitting, etc. 3 In the view of the importance of the problem, a large number of software products have become available on the market to address the task of the automation of demand forecasting. Such software products allow to automatically generate predictions of future demand using one of statistical procedures for time series extrapolation, which, for instance, can be ARIMA, various kinds of exponential smoothing techniques, or other well-known algorithms for automatic time series extrapolation. This methodology in many cases can bring satisfactory results and is particularly useful for large-scale forecasting.
Page 4 of 20 Reasons for Using Judgmental Adjustments Situations when extrapolative methods can occur inapplicable: promotion campaigns, advertising price changes, new products with a lack of historical data, new competitive products on the market, new regulations, etc. 61.2 % of users make regular managerial adjustments to software generated forecasts (Sanders and Manrodt, 2003) 4 However, current experience of using demand forecasting software reveals that purely extrapolative algorithms often become inapplicable due to their inability to sufficiently handle forthcoming events and to the lack of relevant historical data. For example, this can happen in such situations as promotions, price changes, advertising, introduction of new products, new governmental regulations, political events, and other situations where past historical patterns will not hold true in the future, and therefore cannot be automatically extrapolated. One of widely adopted means for overcoming this imperfection is to use judgmental adjustments to statistically generated forecasts and thereby to take into consideration additional information about the environment. In fact, according to the results of the survey by Sanders and Manrodt, 2003, the majority of software users indicate making regular managerial adjustments to model-based predictions.
Page 5 of 20 A Typical Process of Judgmental Adjustments 1. Software package is used to generate a statistical forecast s t,i for a specified period t and SKU i 2. Experts examine the model-based forecast s t,i and can adjust it in the light of their domain knowledge. The final forecast becomes where a t,i - size of adjustment, s t,i =f (past data) f t,i =s t,i + a t,i a t,i =0 in case of no adjustment. 5 Most typically, the process of making judgmental adjustments is performed sequentially and only includes the two following steps. At first, for a given period and given SKU, a statistical forecast is generated by means of a special software package. Usually it is accomplished by applying a simple univariate forecasting method, and the source dataset for that method concerns only past values of sales. After that the model-based forecast is reviewed by experts (representatives from marketing, sales, logistics or production departments). As a result of their revision, the statistical forecast may be adjusted in order to take into account the previously mentioned exceptional circumstances. This approach is now very widely adopted because it is simple to use, to understand and to implement, it allows to incorporate the latest information rapidly (Sanders and Ritzman, 2004).
Page 6 of 20 A Typical Process of Judgmental Adjustments (Forecast Pro) Statistical forecast Final Final forecast s t,i f t,i Managerial adjustment a t,i 6 The described process is now supported by most of demand forecasting packages, and can be illustrated on the latest Forecast Pro version. The shown interface enables users to obtain statistical forecasts, and to adjust them using percentages or increments in full correspondence with the previously described procedure. Other packages use analogous approaches to incorporate judgmental inputs.
Page 7 of 20 The Need of Modelling of Adjustments Judgmentally adjusted forecasts are highly subjective biased and inefficient Statistical modelling of adjustments is needed to improve the quality of the final forecasts by optimally handling the available information obtain objective interval forecasts 7 Several recent studies have been aimed to explore the features of the adjusted forecasts and showed that while judgmental adjustments do improve overall accuracy, they also to a large extent make forecasts biased and inefficient. This means that the final forecasts contain systematic errors and do not optimally utilise available information, due to the fact that they are highly subjective. This suggests the necessity of applying statistical techniques in order to improve the properties of demand estimations. Statistical models of the adjustments are also needed because unlike the extrapolative algorithms, the methodology of judgmental correction itself does not allow obtaining confidence intervals. Eventually, the modelling is needed to provide the decision maker with adequate probabilistic representation of future demand in the presence of the adjustments.
Page 8 of 20 The Formulation of the Problem Available dataset includes: statistical forecasts s t,i final forecasts f t,i history of actual demand values x t,i where t period, i SKU index To find: procedure for statistical estimation of demand for future period if s,i and f,i are known, x,i,i =? Desired aim: to improve properties of statistical and final forecasts 8 More precisely, using special notation, the addressed problem can be formulated as follows. The available data for each SKU contains the history of the computergenerated forecasts, the corresponding judgmentally adjusted forecasts and actual outcomes. The modelling task is to find a good estimation of demand value for a given period in future if statistical and final forecasts for that period are known. One of the desired aims of the modelling is to improve the properties of the constituent forecasts. But even if it is impossible to achieve significant improvements in accuracy, the modelling still remains crucial, as it is the only means to obtain probabilistic estimations using all the available information.
Page 9 of 20 Typical Features of the Data limited amount of relevant observations data related to a particular SKU adjustments are made only occasionally, in most cases statistical forecasts are left unchanged various classes of adjustments exhibit different properties the features of forecasts may vary over time Estimations cannot be done at a level of SKU individually. Models should be based on aggregated estimations related to groups of data with similar properties 9 The source data for modelling adjustments usually possesses certain set of challenging features. In particular, The amount of relevant data at a level of single SKU is very limited, partly because the judgmental adjustments are made only occasionally. Previous works also suggest that various classes of adjustments exhibit different properties. In addition, the possibility that the features of forecasts may vary over time should also be taken into account by introducing additional parameters or by limiting the used data. The above considerations make it impossible to build and estimate separate models for each SKU individually. Therefore, while specifying a model some of the parameters are needed to be introduced at an aggregated level and to be related to a set of data with similar properties.
Page 10 of 20 Previously Suggested Approaches (Fildes at al., 2008) optimal adjust model: x t,i t,i = 1 s t,i t,i + 2 a t,i t,i + t,i t,i where x t,i actual demand, s t,i system forecast, a t,i size of judgmental adjustment, 1, 2 model parameters to be estimated, v t,i error term, t period, i SKU index Estimation procedure: Variables are normalised using the std. dev. of sales for each SKU in order to model v t,i more adequately Groups of data with positive and negative adjustments a t,i are modelled separately The parameters are found by means of OLS 10 One of recently proposed ways of modelling adjustments is based on building a linear regression where the actual demand is represented as a linear function of the statistical forecast and the corresponding adjustment size. The main feature of this model is that the parameters are estimated using data for a set of SKUs instead of a single SKU. But since variances of error term differ between products, variables are divided by the standard deviation of sales for each SKU, which then allows to apply OLS. Another feature is that parameters are estimated separately for positive and negative adjustments, since it was found that they have significantly different properties. Although this approach performed well, a set of questions remain which relate to the ways of obtaining more adequate estimations, in particular, interval estimations, and to the ways of improving the model specification by choosing more disaggregated data groups with similar properties and by tackling with the time-varying nature of coefficients.
Page 11 of 20 The Proposed Developments The research aim: to create a unified statistical framework for automatic handling of judgmental adjustments in demand forecasting software The proposed developments are based on Bayesian approach, which allows to use more flexible and robust modes obtain exact interval forecasts find optimal estimations according to a specified loss function Estimation procedure: forecasts are found in a form of predictive probability density function using MCMC algorithms 11 This research aims to develop existing approaches in order to create a unified statistical framework for automatic handling of the adjustments, which, in term, is needed to improve the design of demand forecasting software. The chosen ways to accomplish this task are based on robust parametric modelling, Bayesian framework and Monte-Carlo numerical estimation methods. This allows to use flexible model specifications, to obtain forecasts in a form of predictive probability density function, and, as a result of it, to calculate exact interval and point estimations, and to find optimal solutions for a given loss function.
Page 12 of 20 Main Modelling Features Form of regression equations: where x t,i t,i = k s t,i t,i + k f t,i t,i + t,i,k x t,i actual demand, s t,i system forecast, f t,i final forecast, k, k model parameters, v t,i,k error term, t period, i SKU index, k data group index Data is ordered into k groups according the sign and size of adjustments (in percentages), best results obtained for k = 4 v t,i,k are assumed to be uncorrelated and represented by t-distribution, variances of error terms are modelled separately for each i and k: v t,i,k ~ t ( 0, i,k, d ), d = 3 or 4, number of d.f. 12 The proposed model specification has the following features. The actual demand value is represented as a weighted sum of statistical and final forecasts. The available data is split into several groups according to the size and direction of adjustments. For each of the groups separate coefficients are used. The error term is modelled by means of heavily-tailed distributions since the analysis showed that they represent errors more adequately. Variances for each SKU are modelled separately and are represented by introducing additional variables.
Page 13 of 20 Handling Special Situations Properties of data change in time: Limit the amount of data to L past obs-ns (L can be found automatically) Use dynamic models for coefficients (random-walk changes): k,t k,t-1 = + k,t k,t k,t-1 k,t k,t, ~ N (0,W k ) k,t Data on one SKU level is insufficient to estimate the variance of the error term, i,k: Use additional assumptions about the ratio of variances: i,1 / i,k = r k-1 (for any i, any k>1) Demand values are only positive and their variances depends on value: Use log transformed values of x t,i, s t,i and f t,i 13 The described specification can be extended in order to take into account several special cases. Firstly, to represent the time-varying nature of parameters, a window of past observations can be used with automatically selected length. In addition, parameters can be represented as evolving in time stochastic processes, specifically, as a randomwalk processes. Another challenging situation arises if the amount of data within the level of SKU is insufficient to obtain estimations of error term variances. The proposed means to tackle with this is to use additional assumptions that ratios of variances will remain the same between SKUs. Finally, to take into account the heteroscedasticity of demand time series it is useful to apply logarithmical transformations to the exsanguinous variables.
Page 14 of 20 Data Used for Modelling and Evaluation 4 companies 3 in manufacturing with monthly forecasts (pharmaceuticals, food, and household products) 1 retailer forecasting weekly one-step ahead statistical systems forecasts, the final forecasts and the corresponding actual outcomes observations history: 3 years about 60,000 cases in total 14 The analysis was based on the data that had been collected from four companies, three with monthly forecasts and one forecasting weekly. The source data contained one-step ahead forecasts and actual outcomes. The history of observations was about three years and contained more than 60,000 cases in total.
Page 15 of 20 Results of Modelling and Evaluation Mean MdAPE for different methods (for manufacturing companies): Statistical forecast 30.88 % Final forecast 25.80 % Average of statistical and final 25.28 % Optimal Adjust Model 23.43 % Proposed methodology 22.72 % Different categories of adjustments exhibit different properties Negatively adjusted forecasts are often close to optimal values Positively adjusted forecasts are largely overestimated, as a rule, positive adjustment is about 50% more than needed Firms should pay attention to situations where positive adjustments are made 15 In order to assess possible gains in accuracy, the alternative approaches were compared using the median absolute percentage error (MdAPE) measure. The results show that modelling adjustments allows to improve the performance of final forecasts, and the improvement is comparable to the difference in accuracy between statistical and final forecasts. Modelling also helped better understand features of adjustments and results here are similar to those obtained in (Fildes et al., 2008). It was confirmed that various categories of adjustments exhibit different statistical properties, and this fact should be taken into account while modelling. In particular, negatively adjusted forecasts are often close to optimal values, while positively adjusted forecasts are usually largely overestimated. Firms should therefore pay more attention to situations where positive adjustments are made. In addition, initial testing showed that the properties of forecasts evolve in time, which makes it necessary to apply corresponding types of models.
Page 16 of 20 Research Summary The proposed solutions allow to: improve the accuracy of final forecasts obtain unbiased and adequate estimations of future demand obtain confidence intervals in the presence of judgmental adjustments find optimal solutions based on a specified loss function Addressed practical issue: to enhance the design of demand forecasting software packages by providing efficient means for handling judgmental adjustments 16 Though the increase in accuracy is important, the main result of the research is that a set of modelling and estimation procedures was developed which allows to provide decision-maker with high-quality, adequate and objective predictions of future product demand in the presence of judgmental adjustments to statistical forecasts. The adopted methods make it possible to obtain exact point and interval estimations as well as optimal solutions for a specified loss function. The solved tasks are crucial to the improvements in the design of demand forecasting systems.
Page 17 of 20 Next Steps Explore possibilities of organisational improvements: the use of structured adjustments better support for making notes and accessing relevant historical data Develop and evaluate other methodologies: using judgmental information as an input to statistical model combining independent judgmental and statistical estimations 17 One of the next steps of the research is to explore the possibilities of improving the methodology of judgmental adjustments not only by enhancements in modelling of the available data, but also by organisational changes and innovative software functionality, including the use of structured adjustments, and better support for making notes and accessing relevant historical data. Other less adopted but more methodologically justified means for incorporating judgmental information into forecasting involve using judgmental information as an input to statistical model and combining independent judgmental forecasts and statistical estimations. The development and evaluation of these approaches is also planned to be conducted in the course of the further research.
Page 18 of 20 Innovative User Interface to Support Demand Forecasting with Judgmental Adjustments As a part of the current work a prototype of user interface was developed in order to improve the performance the adjustment process. The produced here results are based on the following considerations. Often forecasting systems lack the functionality of keeping notes which would allow better understanding and tracking the rationales for making adjustments. Also, recent publications (Lee at al., 2007; Fildes et al., 2008) emphasise the need for improved facilities to enable users to sort cases in the database and to recall and analyse analogous previously occurred situations. However, on the other hand, while those facilities are beneficial and desirable, the amount of information relevant to product demand forecasting can be considerably large and organising and presenting it to a user in a convenient way is a challenging task. Eventually, the success of innovations in the software functionality to a large extent depends on the quality of software interface which should be easy to use and, in the same time, flexible and versatile. The proposed solution is aimed to implement desired functional improvements of the forecasting support systems while meeting the outlined usability requirements as well. The screen shown on Fig. 1 corresponds to the task of preparation and submission of final forecasts. Fig. 2 shows interface for accessing previously submitted forecasts along with related to them notes. These two screens support the main activities of the forecasting process and can be used to illustrate the introduced new features of the interface: Improved support for notes and comments enables managers to add notes in a multi-user mode using templates or as an unstructured text, represent notes in a form of a categorised list, and relate them to different forecasts and periods, different SKUs or groups of SKUs. Support for structured adjustments to model-based forecasts. Improved access to previously submitted forecasts along with corresponding notes and comments, including the support for multiple-horizons forecasting scenario. Search facility for finding similar past cases based on constructing queries to database (including search across SKUs). Analysis of forecasting performance and system feedback. The proposed prototype is fully developed and is ready for the implementation within demand forecasting support systems.
Page 19 of 20 Fig. 1 Fig. 2
Page 20 of 20 References Fildes, R., Goodwin, P., Lawrence, M., & Nikolopoulos, K. (2008). Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning. International Journal of Forecasting. [Forthcoming] Lee, W.Y., Goodwin, P., Fildes, R., Nikolopoulos, K., & Lawrence, M. (2007). Providing support for the use of analogies in demand forecasting tasks. International Journal of Forecasting, 23, 377-390. Sanders, N.R., & Manrodt, K.B. (2003). Forecasting Software in Practice: Use, Satisfaction and Performance. Interfaces, 33 (5), 90-93. Sanders, N.R., & Ritzman, L.P. (2004). Integrating judgmental and quantitative forecasts: methodologies for pooling marketing and operations information. International Journal of Operations & Production Management, 24, 514-529.