Forecasting in multi-skill call centers

Transcription

1 Forecasting in multi-skill call centers A multi-agent multi-service (MAMS) approach: research in progress Gianmario Motta 1, Thiago Barroero 2, Daniele Sacco 3, Linlin You 4 Dept. of Industrial and Information Engineering University of Pavia Pavia, Italy 1 [email protected], 2 [email protected], 3 [email protected], 4 [email protected] Abstract Workforce management is critical in call center business. Human resources are the highest cost, and therefore efficiency is a key success factor. On the other side relevant peaks of incoming calls have to be served. We here consider a complex case, with a many-to-many relationship between agents and services, i.e. the same agent serves many customers and the same customer may be served by many agents. In this perspective, we propose a model to forecast calls in long- and mid-term by ARIMA (Auto-Regressive Integrated Moving Average), and to size workforce in mid-term by integrating an Erlang model. Finally, we have developed a tool to forecast calls in a multi-agent multi-service call center. Field tests are running and first results validate our model. Keywords Call center; ARIMA; Box and Jenkins; Regression analysis; Time Series; Forecasting; Service level agreement; Service level management INTRODUCTION Nowadays, call centers are a large investment for many organizations which need to manage contacts with customers. In 2012, an UK survey reports that the mean call center size is 115 seats; large call centers (over 250 seats) employ 51 per cent of all agents, and small call centers represent 75 per cent of the sites but employ only 27 per cent of staff [1]. Call centers need to staff their operations accurately to provide a satisfactory level of service at a reasonable cost [2]. Weinberg et al. identify the accurate prediction of the call arrivals as the most hard to assess primitives [21]. Generally, a profitable call center targets over 80% load of the available time, or, equivalently, an average idle time lower than 20%. To reach such results a management system is critical. With a poor management, service levels may be low and employees may be short, resulting in an ineffective inefficiency [3]. To have an efficient effectiveness, the call center workforce has to be balanced in front of a double Gaussian curve, with two peaks, respectively in morning and afternoon, and with a complex seasonality, with peaks in the first days of the week and in some months of the year [4]. The workload issue is manageable in call centers [1] with over seats and few customers, and it is handled by a variety of practical means. First, historical records suggest the right sizing of workforce. Second, peaks are served by part-time i.e. overflow staff [5]. Furthermore, in larger centers, the mix of services is not critical, even if calls differ in multiple characteristics, as communication language, technical skills, domain experience, etc. Since supervisors cannot afford to train every agent to handle any kind of calls [7], the mix of agent skills tend to mirror call profiles [4], i.e. the same group of agents serve the same customer. So, economy of scope can be achieved. The issue of skills and staffing becomes tougher in smaller call centers, which have to deal with small volumes and a rather high variety of customers. So, a same operator serves multiple customers (instead of one). In general, there is many to many relation between customers, skills and agent, as we show in Fig. 1. In smaller call centers, even Quality of Service (QoS) requirements tend to be more strict. For, SLAs (Service Level Agreements) are tighter, since customers have a stronger negotiating power; SLM (Service Level Management) becomes critical for survival [6]. This paper intends to propose a solution for this specific class of multi-agent multi-service (MAMS) call centers. Fig. 1. Relationships between agent, customer, skill and group In short, MAMS operations are characterized by (a) multiservice allocation of agents, and (b) complex seasonality. In the following section we discuss the main approaches to call centers forecasting. After that discussion, we illustrate the use of ARIMA (Auto-Regressive Integrated Moving Average) to forecast call traffic and the use of confidence range on multiple services to avoid situations where call volume is not enough to forecast

2 workload for a single service. The choice of ARIMA specifically reflects the complex, iterative seasonality of the incoming traffic of call centers. FORECASTING IN CALL CENTERS Literature provides several researches on call center operations. Many queuing models [8][9][10][11][12] and optimization models [13][14][15] have been discussed for call centers. Some researches include also multi-skill call centers [3][7][15][16][17][18][19]. However, only few discuss forecasting models in a multi-skill context. Open issues for operations management in multi-skill call centers include (a) long-term workload and workforce forecasting, (b) mid-term scheduling of agents, (c) short-time allocation of agents to pool of services, and (d) real-time routing of calls to agents according to their skillset. Staffing has to be decided in advance because of the administrative and training time needed before operations. This characteristic is critical in MAMS call centers because agents shall handle calls of various classes of services. So, forecasting becomes a central challenge for call center managers. Increased availability of historical databases and similar forecasting problems in other application fields have driven research in the call forecasting area [20]. Weinberg et al. [21] propose a multiplicative model for modeling and forecasting within-day arrival rates to a U.S. commercial bank's call center. They use Markov chain Monte Carlo sampling methods to estimate latent states and model parameters. Their model forecast one-day-ahead call rates and counts for a given time interval and for a given day of the week. They also provide a comparison with classical statistical models. Their approach is computationally intensive, but, because of the intra-day interval basis, the results can be easily integrated with agent scheduling and allocation algorithms. Also Soyer and Tarimcilar [22] use a modulated Poisson process model to describe and analyze arrival data to a call center. They take into account covariate and time effects on the call volume intensity by relating the arrival pattern with advertising strategies. The method is very market-oriented and doesn t consider integration with more specific call center issues. Shen and Huang [23] develop methods for inter-day and dynamic intra-day forecasting of incoming call volumes. Interday forecasting consider day-to-day patterns, intra-day forecasting consider within-day patterns. Their method is computationally faster than Weinberg, Brown, and Stroud method because it is based on the use of singular value decomposition to achieve a substantial dimensionality reduction. Also, it can be easily integrated with real-time call routing. However, it doesn t address multi-skill call centers, as well as Weinberg, Brown, and Stroud approach. So, let us compare a selection of new methods proposed in literature for traffic forecasting in call centres, as shown in Table I. TABLE I. COMPARISON OF FORECASTING METHODS Method Advantages Disadvantages Bayesian analysis Singular value decomposition Unobserved component model 1. It outperforms existing models when used to predict 1-day-ahead arrival rates 2. Feasible to use for realtime dynamic forecasting 1. Computationally fast 2. Feasible to use for realtime dynamic forecasting 1. Forecasting horizon of up to 8 weeks in advance. 1. Unable to accurately predict call volumes at horizons beyond 1 week 2. Computation algorithm is sophisticated to implement and can take a long time to converge 1. Existing case studies run on small historical data (less than 200 days) 2. It focuses on one-dayahead forecasting 1. Good performance only at peak hours. 2. Model estimation is overparameterized Further innovation on statistical process control methods have come from the combination of quality procedures and other areas of statistics like time series. Recent approaches [24][25][26] use control charts on the residuals of an Auto-Regressive Integrated Moving Average (ARIMA) models of Box and Jenkins [27]. Alwan and Roberts [24] suggest the time series modeling because several applications of control charts are misleading because control limits are computed for processes that are not in the state of statistical control. In statistics, signal processing, econometrics and mathematical finance, a time series is a sequence of data points, measured typically at successive time instants spaced at uniform time intervals. Time series analysis includes methods for analyzing time series data in order to extract meaningful statistics and other characteristics of the data [28]. Time series forecasting is the use of a model to predict future values based on previously observed values. They are very frequently plotted via line charts and have a natural temporal ordering [29]. This makes time series analysis distinct from other common data analysis problems, in which there is no natural ordering of the observations [30]. A time series model will generally reflect the fact that observations close together in time will be more closely related than observations further apart. Since 1995, time series forecasting has been discussed for call volume prediction by applying seasonal ARMA modeling [31]. Afterwards, Tych et al. [32] used dynamic harmonic regression and evaluated the forecasting performance in comparison with the seasonal ARIMA approach. In 2007, Taylor [33] indicate strong potential for the use of seasonal ARIMA modeling by comparing the performance of a wide range of methods in forecasting call volumes for several call centers, including also an exponential smoothing model for double seasonality. ARIMA is a generalization of an ARMA model. In call centers, it fits time series data either to interpret the existing data

3 either to forecast future points in the series [34][35]. These models are applied where data show non-stationary evidence and the practice of differencing (corresponding to the "integrated" part of the model, it transforms a time series by subtracting past values of itself) can be applied to remove non-stationary interference [31]. Thus, we have adopted ARIMA to forecast the incoming number of calls in MAMS call centers. MAMS FORECASTING We here present our approach to address monthly planning by traffic and workforce where time series, exponential smoothing and regression analysis can be used. ARIMA procedure analyzes and forecasts equally spaced univariate time series data [36]. An ARIMA model predicts a value in a response time series as a linear combination of its own past values, past errors, and current and past values of other time series [37]. As El Hag [38] states, The ARIMA procedure provides a comprehensive set of tools for univariate time series model identification, parameter estimation, and forecasting, and it offers great flexibility in the kinds of ARIMA or ARIMAX models that can be analyzed. The ARIMA procedure supports (a) seasonal, subset, and factored models, (b) intervention or interrupted time series models, (c) multiple regression analysis with ARMA errors, and (d) rational transfer function models of any complexity [39]. The ARIMA approach deals with data sequences formed over time as a random sequence, then this sequence can be approximated by a mathematical model. Once the objective has been identified, this model can forecast future values according to past and present values of the sequence [40]. For more than half a century, the Box-Jenkins ARIMA linear models have dominated many areas of time series forecasting. One of the attractive features of the Box-Jenkins approach for forecasting is that ARIMA processes are a very rich class of possible models and it is possible to find a process which provides an adequate data description [41]. In time series analysis, the Box Jenkins methodology applies autoregressive moving average ARMA or ARIMA models to find the best fit of a time series to past values of this time series, by using an iterative three-stage modeling approach [42]: 1) Model identification and model selection: assessing that the variables are stationary, identifying seasonality in the dependent series (seasonally differencing it, if necessary), and using plots of the autocorrelation and partial autocorrelation functions of the dependent time series to decide which (if any) autoregressive or moving average component should be used in the model. 2) Parameter estimation using computation algorithms to define coefficients which best fit the selected ARIMA model. The most common methods use maximum likelihood estimation or non-linear least-squares estimation. 3) Model validation by testing whether the estimated model conforms to the specifications of a stationary univariate process. In particular, the residuals should be independent of each other and constant in mean and variance over time. (Plotting the mean and variance of residuals over time and performing a Ljung-Box test or plotting autocorrelation and partial autocorrelation of the residuals are helpful to identify misspecification.) If the estimation is inadequate, a new attempt to define a better model must be performed by returning to step 1. Thus, ARIMA is one of the most suitable models for stochastic phenomena that have a double seasonal distribution. We consider two parameters: (a) day (e.g. on Monday has a characteristic behavior that differs from Thursday), (b) hour (e.g. the arrival rate at 8 AM is different from the arrival rate at 1 PM). The usual formula of ARIMA [43] is shown in (1). Where: t = time index W t = value of the variable (number of calls) at time t, or a transformation of the variable Y t (number of calls) = average B = back-shift operator, i.e. φ(b) = auto regression operator, represented as a polynomial back-shift operator B: θ(b) = moving average operator, represented as a polynomial back-shift operator B: a t = random error So, for the double seasonal ARIMA the formula is shown in (2). Where: x = hours a day (the hours the service runs in one day, e.g. the service starts from 8 AM and ends at 8 PM, so x = 12) y = hours a week (the hours the service runs in one week, e.g. it works 10 hours per day and 5 days per week, so y = 50). The process is designed as follows: Fig. 2. Call forecasting process Outlier detection aims to identify data values that do not respect the characteristic curve. In case of detection, these values (2)

4 should not be used to identify the pattern. We can detect the spike, understand the data records which cause the spike, and remove them in order to normalize the data and make the forecast results more accurate. Identify aims to identify the best model to represent the phenomenon, that is to identify the best number of coefficients (i.e. the order of the polynomial, not the value of the coefficients). The identification cannot be performed online, so we must perform a priori analysis for each service. Schwartz Bayesian Information Criterion (SBC) is used to select the best model which shows lowest prediction error [44]. Estimate aims to quantify the factors identified in the previous step. The coefficients are estimated using a maximum likelihood for each service model, by computing the total number of calls for every hour every day. Then according to the working time of each service, incoming calls data is filtered and the parameters are estimated. Forecast step generates the prediction values and the confidence interval associated. Furthermore, it also checks whether the historical data is enough or not. Confidence interval is used to indicate the reliability of our estimate for each service. It is an observed interval, in principle different from sample to sample, that includes the parameter of interest by repeating the experiment. The frequency of the parameter in the observed interval is determined by the confidence level or confidence coefficient [45]. More specifically, the meaning of the term "confidence level" is that, if confidence intervals are constructed across many separate data analyses of repeated experiments, the proportion of such intervals that contain the true value of the parameter will approximately match the confidence level; this is guaranteed by the reasoning underlying the construction of confidence intervals [46]. Confidence intervals consist of a range of values that act as good estimates of the unknown population parameter. However, in rare cases, none of these values may cover the value of the parameter. The level of confidence of the confidence interval would indicate the probability that the confidence range captures this true population parameter given a distribution of samples. It does not describe any single sample [47]. As Strelen [48] states: if a corresponding hypothesis test is performed, the confidence level corresponds with the level of significance, i.e. a 95% confidence interval reflects a significance level of 0.05, and the confidence interval contains the parameter values that, when tested, should not be rejected with the same sample. Greater levels of confidence give larger confidence intervals, and hence less precise estimates of the parameter. Confidence intervals of difference parameters not containing 0 imply that there is a statistically significant difference between the populations. A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained. We take it into account because certain factors may affect the confidence interval, including size of sample, level of confidence, and population variability [49]. Our discussion points out the use of shared agents on multiple services because it may affect the workforce forecasting. The larger the services sample size is, the better the population parameter is estimated. CASE STUDY Our model has been used on Phonetica ( a MAMS call center located in Italy, with a time-varying traffic and workforce. The project aims to reduce salary costs and improve service level to the customer. Because of their economy of scale, call volumes have been increasing very fast. That growth implies some issues in sizing workforce for each service (different services need different agents who have the related skill to handle the call), how to route the calls and continuous training of agents. Given the availability of their large historical database (roughly 3 years of traffic data), we decided to validate our MAMS approach on this real case study. We have selected the top 9 services provided by the call center; Fig. 3 shows the call volume of these services (dark gray). Services have been selected according to the skills developed in the call center. Agents who run these services share same skill set and training experience. Fig. 3. Call volume generated by selected services (dark gray) against all services (light grey) In order to define the correct forecasting model, we have analyzed the incoming calls of each service per weekday and hour. Fig. 4 shows the amount of calls per day and hour. Different lines indicate different days (1 for Monday, 2 for Tuesday, and so on). On this chart, you can see a peak on Thursday. This outlier has been afterwards identified as a service disruption that generated a peak of calls from disappointed users. This is obviously an event that cannot be forecasted and exception handling is not matter of this paper. Outliers must be removed to have a clearer analysis. Fig. 5 shows the new result.

5 (5) Finally, the confidence interval is used to assess forecasted data. A particular confidence level is intended to give the assurance that, if the statistical model is correct, then the true value of the parameter is in the confidence interval. The result is shown in Fig. 6. Fig. 4. Count of calls per weekday and per hour of day After removing the outliers, we identify the model by selecting the one with the lowest SBC (Schwartz Bayesian Information Criterion) value. The best model is used to estimate the factors identified in the previous section by using a maximum likelihood. The formula is shown in (3). After executing the factorization of (3), we can get the new formula which is shown in (4). Fig. 6. Forecasted data and confidence interval Fig. 6 shows 4 days historical data and 3 days forecasted data. The chart in the middle shows the forecasted number of calls (the dotted line) compared with the real number of call (the solid line). Top chart and bottom chart represent the confidence interval range and the forecasted number of calls have 95% is in this range. Top chart shows the highest value of the confidence interval range (i.e. Average+2STD), while bottom chart shows the lowest value of the confidence interval range (i.e. Average- 2STD). We have implemented a VBA module that supports planners to size workload for each class of services, and to assess whether the result is accurate or not. So, through this module, a planner can forecast the calls (first chart in Fig. 7) and compare them against past trends (Fig. 7 shows last 5 days trend in the second chart and same day a week before trend in the third chart). Fig. 5. Count of calls without outliers (4) Where: According to the algorithm of the double seasonal ARIMA, we can compute the forecasting number of calls for each hour and each service, and the forecasting result is (5). Fig. 7. Forecasted data and past trends The confidence interval function computes the last 9 days error and the total number of calls which have the same weekday of the forecasting date and the last 30 days data. The result is accurate (i.e. green in Fig. 8) if the error on the last 9 weekdays

6 and the last 30 days is lower than 33% or its absolute value is lower than 4 calls (this latter condition has been introduced to fit low volume services). Fig. 8. The confidence interval function in our VBA module CONCLUSIONS We have designed and implemented a forecasting support system that allocates low volume inbound calls to shared agents, thus satisfying the requirements peculiar to multi-agent multiservice (MAMS) call centers. The forecasting model has been implemented on a parametric tool, that enables sensitivity analysis, with clearly stated accuracy levels. In particular, it allows to assess different configurations of services in order to optimize workload forecasting and confidence interval. The field validation is still in progress and first results show that out method generates very competitive two-week-ahead forecasting. Furthermore, first outcomes in the call center are (a) a long-term optimized staffing, that easies human resources management, (b) lower operating costs, and (c) higher service quality. Finally, our forecasting support system enabled supervisors to work on exceptions, mainly to address real-time management. Future works are short-time scheduling and real-time allocation (1) temporary agents, which can be allocated to deal with workload peaks, (2) regular agents, who are daily workers. Our forecasting model will be integrated with our previous research [50] on queue theory, in order to right size workforce in mid-term. Mid-term planning is based on Erlang model and it is performed weekly by using traffic forecasting results as an input to the model. Further research will be carried on about real-time agent allocation, that can solve issues generated during unpredictable peaks (e.g. Fig. 4 in previous section). REFERENCES [1] CFA business work, Contact Centre Operations: Labour Market Report, 2012 [2] Gans, N., Koole, G., and Mandelbaum, A. (2003), Telephone Call Centers: Tutorial, Review and Research Prospects, Manufacturing and Service Operations Management, 5, [3] A. Pot, S. Bhulai, G. Koole, A Simple Staffing Method for Multiskill Call Centers, Manufacturing a Service Operations Management, vol. 10, no. 3, pp , 2008 [4] G. Koole, Call Center Mathematics: A scientific method for understanding and improving contact centers, Vrije Universiteit Amsterdam, 2007 [5] Gurvich, I., & Perry, O. (2012). Overflow Networks: Approximations and Implications to Call Center Outsourcing. Operations Research, 60(4), [6] Motta, G., Barroero, T., Galvani, F., & Longo, A. (2011). IT Service Level Management: Practices in Large Organizations. Communications, [7] A. N. Avramidis, W. Chan, P. L'Ecuyer, Staffing multi-skill call centers via search methods and a performance approximation, IIE Transactions, vol. 41, no. 6, pp , 2009 [8] Feldman, Z., A. Mandelbaum, W. A. Massey, W. Whitt Staffing of time-varying queues to achieve time-stable performance. Management Science. [9] Koole, G., A. Mandelbaum Queueing models of call centers: An introduction. Annals of Operations Research 113(1 4) [10] Mandelbaum, A., S. Zeltyn The impact of customers patience on delay and abandonment: Some empirically-driven experiments with the M/M/n G queue. OR Spectrum 26(3) [11] Mandelbaum, A., S. Zeltyn. 2007a. Service engineering in action the palm/erlang A queue, with application to call centers in Advances in services innovations, D. Spath, K.-P. Fahnrich (eds.), Springer, Berlin. [12] Massey, W. A., R. B. Wallace An optimal design of the M/M/C/K queue for call centers. Queueing Systems, forthcoming. [13] L Ecuyer, P Modeling and optimization problems in contact centers in Proceedings of the Third International Conference on the Quantitative Evaluation of Systems (QEST 2006), University of California, Riversdale, IEEE Computing Society, [14] Atlason, J., M. A. Epelman, S. G. Henderson Call center staffing with simulation and cutting plane methods. Annals of Operations Research 127(1 4) [15] Cezik, M. T. and L'Ecuyer, P Staffing multiskill call centers via linear programming and simulation. Management Science, 54(2), [16] Stolletz, R., S. Helber Performance analysis of an inbound call center with skill-based routing: A priority queueing system with two classes of impatient customer and heterogeneous agents. OR Spectrum 26(3) [17] Jouini, O., Y. Dallery, O. Z. Aksin. 2007a. Queueing models for multiclass call centers with real-time anticipated delays. Working Paper, Koc University [18] Bhulai, S., G. Koole A queueing model for call blending in call centers. IEEE Transactions on Automatic Control 48(8) [19] Koole, G., A. Pot An overview of routing and staffing algorithms in multi-skill customer contact centers. Technical Report, Department of Mathematics, Vrije Universiteit Amsterdam, The Netherlands. [20] Z. Aksin, M. Armony, V. Mehrotra, The Modern Call Center: A Multi- Disciplinary Perspective on Operations Management Research, Production and Operations Management, Vol. 16, No. 6, pp , November-December 2007 [21] Weinberg. J., L. D. Brown, J. R. Stroud Bayesian forecasting of an inhomogeneous poisson process with applications to call center data. Journal of the American Statistical Association, vol. 102, pp [22] R. Soyer, M. Tarimcilar, Modeling and analysis of call center arrival data: A bayesian approach Management Science, vol. 54, no. 2, pp , [23] H. Shen and J.Z. Huang, Interday forecasting and intraday updating of call center arrivals, Manufacturing & Service Operations Management, vol. 10, no. 3, pp , 2008 [24] Alwan, L C, Roberts, H V. (1988). Time-series modeling for statistical process-control. Journal of business & economic statistics, 6(1) [25] Alwan, L C, Bissell, M G. (1988). Time-series modeling for qualitycontrol in clinical-chemistry. Clinical Chemistry, 34(7) [26] Alwan, LC, Roberts, HV The problem of misplaced control limits. Journal of the Royal Statistical Society. Series C, Applied statistics, 44(3), [27] Box, G. E. P., and Jenkins, G. M. (1976), Time Series Analysis, Forecasting and Control( 2nd ed.), San Francisco: Holden-Day [28] Galit Shmueli. Practical Time Series Forecasting: A Hands-On Guide. CreateSpace Independent Publishing Platform.2 edition.2011:12-43

7 [29] Hur, D. A comparative evaluation of forecast monitoring systems in service organizations, 33rd Annual Meeting of the Decision Sciences Institute, Decision Sciences Institute, San Diego, CA,USA, 2002, 5 pp. [30] James D. Hamilton. Time Series Analysis. Princeton University Press. 1994:39~145 [31] Andrews, B. H., S. M. Cunningham L.L. Bean improves call-centre forecasting. Interfaces 25(6) [32] Tych, W., D. J. Pedregal, P. C. Young, J. Davies An unobserved component model for multi-rate forecasting of telephone call demand: The design of a forecasting support system. International Journal of Forecasting 18(4) [33] Taylor, J. W A comparison of univariate time series methods for forecasting intraday arrivals at a call center. Management Science, forthcoming. [34] Bianchi, Lisa, Jeffrey Jarrett and R. Choudary Hanumara. Improving forecasting for telemarketing centers by ARIMA modeling with intervention, International Journal of Forecasting, 14 (4), 1998, [35] Bianchi, Lisa, Jeffrey Jarrett and R. Choudary Hanumara. Forecasting incoming calls to telemarketing centers, The Journal of Business Forecasting Methods & Systems, 12 (2), 1993, [36] Kalpakis, K.; Gada, D.; Puttagunta, V. Distance measures for effective clustering of ARIMA time-series, Data Mining, ICDM 2001, Proceedings IEEE International Conference, 2001: [37] Antipov, A. and N. Meade. Forecasting call frequency at a financial services call centre, The Journal of the Operational Research Society, 53 (9), 2002, [38] El Hag, Sharif. An adjusted ARIMA model for internet traffic, 2007 IEEE International Conference, 2007:1-6. [39] Weiqiang Wang, Ying Guo. Air Pollution PM2.5 Data Analysis in Los Angeles Long Beach with Seasonal ARIMA Model, Energy and Environment Technology, ICEET '09. International Conference, 2009:7-10. [40] Areekul, P.; Senjyu, T.; Toyama, H.; Yona, A. Transmission & Distribution Conference & Exposition: Asia and Pacific, 2009:1-4 [41] Yantai Shu; Minfang Yu; Jiakun Liu; Yang, O.W.W. Wireless traffic modeling and prediction using seasonal ARIMA models, Communications, ICC '03. IEEE International Conference, 2003: [42] Lagarto, Joao; de Sousa, Jorge; Martins, Alvaro; Ferrao, Paulo. Price forecasting in the day-ahead Iberian electricity market using a conjectural variations ARIMA model, European Energy Market (EEM), th International Conference. 2012:1-7. [43] J. Taylor W. et al. A comparison of univariate methods for forecasting electricity demand up to a day ahead. International Journal of Forecasting. Vol 22. pg (2006) [44] Oh, C.O. and Morzuch, B.J., Evaluating Time-Series Models to Forecast the Demand for Tourism in Singapore Comparing Within-Sample And Postsample Results, Journal of Travel Research, Vol 43, No. 4, pp , 2005 [45] Douglas Altman, David Machin, Trevor Bryant, Stephen Gardner. Statistics with Confidence: Confidence Intervals and Statistical Guidelines. BMJ [46] Aksin, O.Z. and P.T. Harker. Computing performance measures in a multiclass multi-resource processor-shared loss system, European Journal of Operational Research, 123 (1), 2000, [47] Yili Hong; Meeker, W.Q.; Escobar, L.A. The Relationship Between Confidence Intervals for Failure Probabilities and Life Time Quantiles, Reliability, IEEE Transactions,2008: [48] Strelen, J.C. The accuracy of a new confidence interval method, Simulation Conference, Proceedings of the 2004 Winter [49] Kasemset, C.; Kachitvichyanukul, V. Effect of confidence interval on bottleneck identification via simulation, Industrial Engineering and Engineering Management (IEEM), 2010 IEEE International Conference,2010: [50] Barroero, T. and Motta, G. and Della Vedova, M., Right Sizing Customer Care: An Approach for Sustainable Service Level Agreements, 2011 International Joint Conference on Service Sciences, May 2011, Taipei, Taiwan