Improving Demand Forecasting

Transcription

1 Improving Demand Forecasting 2 nd July 2013 John Tansley - CACI

2 Overview The ideal forecasting process: Efficiency, transparency, accuracy Managing and understanding uncertainty: Limits to forecast accuracy, including the Poisson limit The Forecast Value Add approach: From simple to more complex models Types of forecasting model: Econometric, quantitative, and combined Types of quantitative models: Time series, explanatory, combined Case study 1: Improved call volume forecasting for financial services debt management Case study 2: 30 year water demand forecasting 2

3 The ideal forecasting process Goal of forecasting process: Provide the best possible forecast, given the information available Best means: Efficiency: Automate data feeds as much as possible Transparency: Understandable (avoiding black box or overly complicated Excel) Accuracy: Self explanatory! Also provide estimates of forecast accuracy if possible 3

4 Managing and understanding uncertainty Best possible accuracy is outside the control of the analyst Factors that affect accuracy: Lack of all necessary information: Only have access to limited data Problem changes rapidly over time Inaccuracies in known information: Inaccurate data Incorrect mental model of the business problem Fundamental limits to accuracy Poisson noise 4

5 Poisson accuracy limit When forecasting counts, there is a fundamental limit of achievable accuracy the Poisson limit 100 calls in a 10h day - equally spaced Call 08:00:00 09:00:00 10:00:00 11:00:00 12:00:00 13:00:00 14:00:00 15:00:00 16:00:00 17:00:00 18:00: calls in a 10h day - random Call 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 5

6 Poisson accuracy limit When forecasting counts, there is a fundamental limit of achievable accuracy the Poisson limit 1000 calls in a 10h day Call 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18: Calls per hour :00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 6

7 Poisson accuracy limit Demand in a particular time period or location is generally distributed according to a universal distribution the Poisson distribution. The spread of this distribution is around the square root of the demand volume Demand Mean Spread Spread (%) % % % % % Understanding this limit helps to Set reasonable accuracy expectations Prioritise where analysts should spend their time 7

8 The Forecast Value Add approach Start simple Additional complexity is only worth it if it increases accuracy Can measure by how much each incremental step improves the forecast No point adding complexity if forecast error increases Error Forecast error versus model complexity Error Best model Best model Naïve model performance Poisson limit performance Model complexity 8

9 Types of forecasting models Econometric Models Manually built models Small datasets (if any) Manual setup Based on business knowledge Bayesian Econometric Models Manual model structure Model parameters set from user constraints and data Based on both business knowledge and data Quantitative Models Automatic models Larger datasets Little user control over parameters Based on data Examples: Regression, Decision Trees Knowledge Data 9

10 Types of quantitative models Time. input1 input2 input3 Target explanatory Regression, Decision Trees, Neural Networks Use drivers, add insights time series Weekly profile, moving average, ARIMA, Exponential smoothing, Good for trends 4 combined ARIMA with drivers, Decomposition Forecasting More powerful, but more complex 10

11 Case study 1 Improving the forecasting process Improved call volume forecasting for the debt management function CFS were creating forecasts in large Excel sheets, populated manually CFS had a desire to improve process, and remove single man dependency Solution: Software: statistical forecasting models Automation of data feeds Results: Reduced single man dependency Immediately showed increased speed (from 2.5 to 1 day) and transparency Over time, increased insight, increased accuracy (from 60% to 70% of days within target) 11

12 Case study 2 Long term demand forecasting 30 year water demand forecasting for a large water board Yearly forecasts across geographical areas, and business sectors A few years worth of demand, economic and weather data Approach: Bayesian Econometric Models Allows the model structure to be set by the user Model parameter estimates are set by the user Model parameters are then calibrated on existing data Result: Forecasts that combine the best business knowledge and the data Parameters can be set by the user if needed, or dictated solely by the data 12

13 Wrap-up A large number of techniques are currently available for forecasting, the key is choosing right technique for right problem A good forecasting approach should add insight as well as accuracy Forecast Value Add approach: keep it as simple as possible Key is to keep on top of models keep them understandable and easy to update A good forecast sets the foundation for good planning 13

14