THINK DEVISE CALCULATE ENVISION INVENT INSPIRE THINK DEVISE CALCULATE ENVISION INV

Transcription

1 Issue 4 June 2006 FORESIGHT The International Journal of Applied Forecasting THINK DEVISE CALCULATE ENVISION INVENT INSPIRE THINK DEVISE CALCULATE ENVISION INV SPECIAL FEATURES Forecasting for Call Centers Forecast Accuracy Metrics for Inventory Control Lessons From Successful Companies Breaking Down Barriers to Forecast Process Improvement Transformation Lessons From Coca-Cola Enterprises Inc. A PUBLICATION OF THE INTERNATIONAL INSTITUTE OF FORECASTERS IIF

2 ACCURACY AND ACCURACY-IMPLICATION METRICS FOR INTERMITTENT DEMAND by John Boylan and Aris Syntetos Preview: John and Aris distinguish between forecast-accuracy metrics, which measure the errors resulting from a forecast method, and accuracy-implication metrics, which measure the achievement of the organization s stock-holding and service-level goals. Both measurements are important. The correct choice of a forecast-accuracy metric depends on the organization s inventory rules and on whether accuracy is to be gauged for a single item or across a range of items. The authors recommend specific accuracy and accuracy-implication metrics for each context. John Boylan is Professor of Management Science at Buckinghamshire Chilterns University College. Previously, he worked in OR at Rolls-Royce and at the Unipart Group. His research and publications (Journal of the OR Society, International Journal of Production Economics, International Journal of Forecasting) have increasingly focused on the challenges of forecasting slow, intermittent and lumpy demands. Aris Syntetos is a Lecturer in Operations Management and Operational Research at the University of Salford, UK. His research interests include intermittent-demand forecasting and the interface between forecasting and stock control. On behalf of the Salford Business School, he is currently involved in two inventory-management projects, one with an engineering firm and one with an international wholesaling company. In considering forecasting-accuracy metrics for intermittent demand, we should begin by looking at the inventory method. Depending on that method, we may need estimates of mean demand, variance of demand, percentiles of demand, and probabilities of high-demand values. When a forecast of mean demand is needed, the accuracy of the forecast for an individual item can be judged by the mean absolute error (MAE). To assess forecast accuracy across a range of items, a scale-independent metric, such as the ratio of the mean absolute error to the mean demand, is appropriate. Alternatively, the geometric mean absolute error (GMAE) may be used. If forecasts of percentiles of demand or probabilities of high-demand values are needed, then an appropriate chi-square test should be used, concentrating on the upper end of the distribution (for example, the 95 th percentile). No matter which inventory system is used, the accuracy-implication metrics of stock-holding and service levels should always be considered. Introduction In the February 2006 edition of Foresight, Kenneth Kahn poses the following question: Should we view forecast accuracy as an end in itself or rather as a means to an end? (Kahn, 2006, p. 25). Most commonly, intermittent-demand forecasting is a means toward the twin ends of lowering stock-holding costs (including costs of stock obsolescence) and maintaining or improving stock availability ( service level ). The achievement of these goals depends not only on the accuracy of the forecasting method but also on the suitability of the inventory rules determining the timing and size of orders. The relationship between these factors and the system s goals is shown in Figure 1, next page. If we regard the design and implementation of a stockmanagement system as a means toward an end, then the outcome measures on the right-hand side of Figure 1 should not be ignored. These measures ensure that forecasters and inventory managers do not lose sight of the system s purpose. Accuracy-Implication Metrics Most managers would regard stock-holding costs and service level as outcome measures rather than accuracy measures. But if we keep the inventory rules fixed and try different forecasting methods, these outcome measures become accuracy-implication measures. The term accuracy implication is used instead of accuracy because metrics such as service level do not measure the accuracy of a forecasting method, but they do measure the implication of its accuracy under a given inventory rule. A good example of this approach is the study by Eaves and Kingsman (2004). They estimate the effect of forecastingmethod choice on 18,750 line items, including intermittent June 2006 Issue 4 FORESIGHT 39

3 Figure 1. Relationship Among Forecasting, Inventory Rules and Performance Measures Forecasting Method Inventory Rules STOCK MANAGEMENT SYSTEM 40 FORESIGHT Issue 4 June 2006 Stock-holding Costs Service Level items. Importantly, they assume a constant service-level requirement, allowing accuracy implications to be assessed. For example, for quarterly data, they find that using single exponential smoothing instead of Croston s method requires an additional stock investment of 1.28m. Therefore, instead of simply reporting that Croston s method is more accurate than smoothing, they show the cost implication of making the wrong choice. What Types of Forecasts Are Required? To identify the most appropriate accuracy metrics, we must first ask what is to be forecast. The variables to be forecast depend on the inventory method. For example, suppose that we use a periodic (R, s, S) inventory rule. This means that we review the inventory system every R periods, and when the stock level drops to a certain reorder point (called s) or lower, then we order enough stock to take us back up to the reorder level (called S). Some of the most effective methods for finding s and S, including Naddor s heuristic (Naddor, 1975), require only estimates of the mean and variance of demand. To take a second example, suppose that we use an (s, Q) inventory method. In this case, we review the stock continuously and place an order of fixed quantity Q if the stock drops to the reorder level s or below. This system is also known as (r, Q). In some systems, we wish to ensure that there is no more than a 10% chance of stockout during the replenishment cycle (review time plus lead time). For this case, we need to estimate the 90 th percentile of the distribution of demand over the replenishment cycle, rather than the mean and the variance. Another alternative is that we wish to ensure that at least 90% of demand is satisfied directly off the shelf please note that this is not the same as a 90% chance of no stock outs; this point is discussed in greater detail by Silver, Pyke, & Peterson (1998, ). In this case, we need to estimate the probabilities of any demands that exceed the reorder level. Here, instead of estimates of percentiles, we want estimates of the probabilities of high demand. Should We Forecast the Entire Demand Distribution? Willemain (this issue) suggests that the general problem is to forecast the whole distribution of demand. It is true that this is the most general statement of the problem. However, as we have already noted, some inventory systems require estimates of only the mean and variance. For other systems, estimates of high percentiles and probabilities of high-demand values are needed; even in these cases, we do not need a forecast of the entire distribution. Measures based on the entire distribution can be misleading. A good overall goodness of fit statistic may result from excellent forecasts of the chances of lowdemand values, which can mask poor forecasts of the chances of high-demand values. It may be that for other applications (for example, revenue forecasts), forecasts of low percentiles are required (Willemain et al., 2004). However, for inventory calculations, we suggest that attention be restricted to the upper end of the distribution (the 90 th or 95 th percentiles). In summary, percentile forecasts and estimates of probabilities of demand are required for some inventory systems. For other systems, we need forecasts of the mean and variance of demand. All these quantities are features of the overall distribution of demand. It is the accuracy of determining the key quantities (for example, mean demand, 90 th percentile) required for the inventory rules that is important, rather than the accuracy across the entire demand distribution. Estimates of Mean Demand When it is necessary to forecast the mean demand level, there are two issues to address: What is the best forecasting method for a particular stock-keeping unit (SKU)? What is the best forecasting method across a range of SKUs? The second problem is more common in practice, but answering the first question gives us some insight into how to answer the second. The case of a single SKU For a single SKU, we may use a simple measure such as the mean absolute error (MAE) to measure a method s accuracy in forecasting mean demand. (The mean absolute error is calculated by noting each of the errors and treating them all as positive in sign, and then averaging them.) The mean squared error is not suitable for intermittent-

4 demand items because it is sensitive to the occurrence of very high forecast errors. The accuracy of a method s mean-demand forecasts can be compared with another method by calculating the percentage of series for which it has a lower MAE. This approach is known as the Percentage Better method, which is discussed in more detail by Boylan (2005). The approach can be easily extended to the comparison of more than two methods; in that case, it would be termed Percentage Best. A limitation of the Percentage Better method is that, although it summarizes the frequency with which one method outperforms another, it does not inform the user of the degree of improvement in accuracy. Averaging the values of mean absolute error across series would seem to be the obvious answer. Unfortunately, such measures can be dominated by a small number of SKUs with large errors. This problem is known as scale dependence. For non-intermittent data, an effective way of addressing the scale-dependence problem is to calculate the mean absolute percentage error (MAPE). However, the MAPE measure fails for intermittent data because the denominator (actual value) is frequently zero. Amending the denominator to unity when the actual value is zero, as suggested by Jim Hoover in this section, is a pragmatic idea, but it is without any foundation in statistical theory. Another option mentioned by Hoover is the symmetric MAPE (smape), in which the numerator is the absolute value of the actual minus forecast, and the denominator is the average of the actual and forecast values. However, whenever the actual value is zero, the smape entry will have a value of two, regardless of the forecast. If the actual is zero and our forecast is 1, then smape = 1 / ((0+1) / 2)) = 2. If our forecast is 100, then smape = 100 / ((0+100) / 2)) = 2. Therefore, smape cannot be recommended because when actual demand is zero, it does not discriminate between forecasting methods. Scale-independent metrics Scale-independent metrics are required to assess forecast accuracy across a range of items. For intermittent data, a good scale-independent measure is the ratio of the mean absolute error to the mean demand, as suggested by Jim Hoover. A variation on this approach is to compare the accuracy of one method to another by taking the ratio of mean absolute errors. Alternatively, instead of MAEs, we can compute the ratio of the geometric root mean square error (GRMSE) of one method to that of another. Although this metric is more complex, it is even more robust (less sensitive) than the MAE regarding outlying observations. Fildes (1992) showed that, in the GRMSE calculation, the distorting effect of large errors cancels out. For details on the application of the GRMSE to intermittent-demand items, see Syntetos and Boylan (2005). In his accompanying article, Rob Hyndman correctly notes that the geometric root mean square error is identical to the geometric mean absolute error (GMAE). Because the GMAE is easier to calculate than the GRMSE, and delivers the same result, we will use it in the example that follows. The geometric mean is an alternative to averaging by the arithmetic mean, in which we multiply all observations and then find the nth root. For example, suppose we have three observations: 1, 4, and 16. The geometric mean is 4, as this number is the cube root of 64 (= 1 x 4 x 16). This approach can also be applied to the absolute forecast errors. Suppose we have four forecast errors: -3, 1, -5, and 4. The absolute errors are 3, 1, 5, and 4. Then the geometric mean is the fourth root of 60 (= 3 x 1 x 5 x 4), namely This is the geometric mean absolute error, and it is identical to the geometric root mean square error. A potential problem with the GMAE is that if any one forecast error is zero, then the GMAE is also zero, regardless of the size of the other forecast errors. Zero forecast errors can arise in two ways for intermittent demand: 1. Non-zero demand: identical non-zero is forecast. 2. Zero actual demand: zero is forecast. In our experience, the first case does not arise frequently in practice, and it never occurred on the dataset of 3,000 SKUs analyzed by Syntetos and Boylan (2005). Methods based on exponential smoothing (ES), such as Croston s method and the Syntetos-Boylan approximation, do not generally produce whole-number estimates of the mean demand; therefore they do not typically generate zero errors. Consequently, series with zero GMAEs will be rare if ES-based methods are compared, and they can be excluded from an across-series analysis. If other methods are used, such as the naïve method, then the GMAE will not always be well defined. However, the naïve method is sensitive to large demands and will generate high forecasts in such instances, making it inappropriate for practical inventory applications. June 2006 Issue 4 FORESIGHT 41

5 The second case, highlighted to us in a private correspondence by Jack Hayya, can occur when it has been a very long time since there have been any non-zero observations. This may signal that the item is at the end of its life and should be reviewed for classification as obsolescent, requiring no subsequent forecasts. If the item is nearing obsolescence (but is not yet obsolete), there would have been some evidence of demand in recent years, and a zero mean demand forecast is inappropriate and should be reviewed. Estimates of Demand Variance Why does the variance of forecast error need to be estimated? There are two reasons: (1) in some cases, the variance of demand is estimated as an intermediate step in finding a percentile of forecast demand, or the probability of high values of demand; (2) in other cases, the variance of demand is input to a formula that will be used to estimate inventory parameters, such as the reorder point (s). It is not possible to assess the accuracy of variance estimates directly, unless assumptions are made about the demand distribution. However, indirect approaches are available. If the variance is estimated to find a percentile of demand, we can examine the accuracy of the resulting percentile estimate. To do this, we identify the percentile of interest (for example, the 90 th percentile) and compare how many observations exceed the percentile estimate against the expected value. This can be achieved using the chi-square test, as discussed by Tom Willemain. A similar approach can be adopted if the variance estimate is used to calculate probabilities of high values of demand. If the variance is used as an input to an inventory formula, we can look at the measures of inventory cost and service. This would enable different approaches to variance estimation to be compared indirectly and is an example of the accuracy-implication approach advocated in this paper. Conclusions In considering forecasting-accuracy metrics for intermittent demand, we should begin by looking at the inventory method. Which forecasts are required for the particular inventory method? The answer may be estimates of mean demand, variance of demand, percentiles of demand, or probabilities of high-demand values. There are appropriate accuracy metrics for each type of estimate. No matter which inventory system is in use, the accuracyimplication metrics of stock-holding costs and service levels should always be considered because these are of prime importance to the organization. The use of these measures should not be limited to situations in which it is difficult to assess forecast error directly. Accuracyimplication metrics also offer a basis for the comparison of different forecasting methods. References Boylan, J. (2005). Intermittent and lumpy demand: A forecasting challenge, Foresight: The International Journal of Applied Forecasting, Issue 1, Eaves, A. H. C. & Kingsman, B. G. (2004). Forecasting for the ordering and stock-holding of spare parts, Journal of the Operational Research Society, 55, Fildes, R. (1992). The evaluation of extrapolative forecasting methods, International Journal of Forecasting, 8, Kahn, K. B. (2006). Commentary: Putting forecast accuracy into perspective, Foresight: The International Journal of Applied Forecasting, Issue 3, Naddor, E. (1975). Optimal and heuristic decisions on single and multi-item inventory systems, Management Science, 21, Silver, E. A., Pyke, D. F. & Peterson, R. (1998). Inventory Management and Production Planning and Scheduling, 3 rd ed., New York: John Wiley & Sons. Syntetos, A. A. & Boylan, J. E. (2005). The accuracy of intermittent demand estimates, International Journal of Forecasting, 21, Willemain, T. R., Smart, C. N. & Schwarz, H. F. (2004). A new approach to forecasting intermittent demand for service parts inventories, International Journal of Forecasting, 20, Contact Info: John Boylan Buckinghamshire Chilterns University College, UK jboyla01@bcuc.ac.uk Aris Syntetos University of Salford, UK a.syntetos@salford.ac.uk 42 FORESIGHT Issue 4 June 2006