FORECASTING NETWORK TRAFFIC: A COMPARISON OF NEURAL NETWORKS AND LINEAR MODELS

Transcription

1 Session 2. Saisical Mehods and Their Applicaions Proceedings of he 9h Inernaional Conference Reliabiliy and Saisics in Transporaion and Communicaion (RelSa 09), Ocober 2009, Riga, Lavia, p ISBN Transpor and Telecommunicaion Insiue, Lomonosova 1, LV-1019, Riga, Lavia FORECASTING NETWORK TRAFFIC: A COMPARISON OF NEURAL NETWORKS AND LINEAR MODELS Irina Klevecka Insiue of Telecommunicaions / Riga Technical Universiy Azenes Sr. 12, Riga, LV-1048, Lavia Ph.: klevecka@inbox.lv The main aim of he research was o produce he shor-erm forecass of raffic loads by means of neural neworks (a mulilayer percepron) and radiional linear models such as auoregressive-inegraed moving average models (ARIMA) and exponenial smoohing. The raffic of a convenional elephone nework as well as a packe-swiched IP-nework has been analysed. The experimenal resuls prove ha in mos cases he differences in he qualiy of shor-erm forecass produced by neural neworks and linear models are no saisically significan. Therefore, under cerain circumsances, he applicaion of such complicaed and ime-consuming mehods as neural neworks o forecasing real raffic loads can be unreasonable. Keywords: elecommunicaions, packe-swiched neworks, raffic forecasing, neural neworks, ARIMA, exponenial smoohing 1. Inroducion The objec of he research is he ime series characerizing he real raffic of boh radiional elephone neworks (POTS) and packe-swiched IP-neworks. The reliable forecass of raffic generaed by users (subscribers) allow planning he capaciy of ransmission channels, avoiding he overload and susaining he opimal level of qualiy of service. A rapid developmen of packe-swiched neworks and he ransformaion of radiional elephone neworks ino muli-service sysems offer new opporuniies o a user (subscriber) and expand his / her scope of aciviies. Though, no only he archiecure of elecommunicaions neworks bu also he saisical naure of raffic has been changed wha implies a srong influence of such self-similariy. In he analysis of dynamic behaviour of IP-neworks such a complicaed and ime-consuming nonlinear mehod as neural neworks is gaining more and more accepance. However, he soluion of he ask of raffic forecasing is no rivial. As our research proves, under he cerain circumsances he reliable forecass of he raffic of packe-swiched neworks can be produced by applying radiional linear mehods as well. 2. Peculiariies of Forecasing Nework Traffic In order o model and forecas he dynamics of nework raffic, we usually assume ha is values are expressed by discree ime series. A discree ime series is defined as a vecor {x } of observaions made a regularly spaced ime poins = 1, 2,, N. Unlike he observaions of a random variable, he observaions of a ime series are no saisically independen. This relaion ses up he specific base for forecasing an analysed variable (i.e. for producing he esimae xˆ (N+L) of an unknown value x(n+l) aking ino accoun he hisorical values x( 1 ), x( 2 ),, x( N )). The mehods of raffic forecasing are defined by he ITU-T recommendaions E.506 and E.507 [1; 2]. Even he recommendaions are parly obsolee and are supposed o be used for forecasing he raffic of ISDN-neworks, some of he mehods sill can be applied o modern elecommunicaions neworks. In paricular, hese mehods are auoregressive-inegraed moving average (ARIMA) models and exponenial smoohing. As i has been already menioned, he empirically observed raffic of packe-swiched neworks is self-similar in a saisical sense, over a wide range of ime scales. Self-similariy in wide sense means ha a covariance srucure is preserved when he ime series is aggregaed. Objecs wih his self-similar qualiy are called fracals. Two imporan feaures of self-similar raffic are long-range dependence and a slowly decaying variance [3]. These saisical effecs of self-similar raffic make is forecasing more complicaed han he predicion of he raffic of radiional elephone raffic which is characerized by shor-range dependence. 170

2 The 9 h Inernaional Conference RELIABILITY and STATISTICS in TRANSPORTATION and COMMUNICATION Non-linear neural neworks have won populariy in ime series forecasing, among he asks of which is also he predicion of packe-swiched raffic. They provide addiional opporuniies in modelling non-linear phenomena and recognizing chaoic behaviour of processes. However, as he curren research shows, in some cases radiional linear models succeed in forecasing he raffic of packe-swiched neworks no less han such resources- and ime-consuming mehods as neural neworks. I is imporan o keep in mind ha self-similariy of packe-swiched raffic, which comes along wih a slowly decaying variance and long-range dependence, has a prominen influence only in he case of measuremens in a very small scale over he aggregaion period varying from milliseconds o approximaely 15 minues (see Figure 2.1). From he poin of view of ime series forecasing such a fine scale is ofen unreasonable. In his case he selecion of he adequae saisical model can be complicaed due o a srong influence of auocorrelaion beween disan observaions of a imes series as well as due o he influence of exraneous noises and anomalous ouliers which unavoidably enail he measuremens in such a fine scale. Therefore, according o he ITU Recommendaion E.492 [4] he measuremens of nework raffic should be averaged over 15- minues and / or one-hour inervals. In his case we can ofen speak abou he possibiliy of applying radiional mehods of ime series forecasing. Figure 2.1. Saisical effecs of packe-swiched raffic depending on a ime scale [5, wih auhor s changes] A reference period also deermines a forecasing horizon for which reliable forecass can be produced. Condiionally speaking, he possible forecasing horizons for ime series aggregaed over he period of one second or 24 hours are differen. The main accen of his research was pu on he applicaion of neural neworks (i.e. a muli-layer percepron) for forecasing he changes of he raffic of boh radiional elephone neworks and packeswiched IP-neworks. The forecass produced by non-linear models were compared o hose which were produced by radiional linear models. For he purpose of his comparison he models of ARIMA and exponenial smoohing were chosen (as he mehods recommended by he ITU-T). If he comparaive analysis of forecass produced by neural neworks and linear models do no reveal any saisically significan differences, hen he applicaion of such a complicaed and ime-consuming mehod as neural neworks does no make sense. 3. The Mehods of Traffic Forecasing 3.1. Neural Neworks Neural neworks are massively parallel, disribued processing sysems represening a new compuaional echnology buil on he analogy o he human informaion processing sysem. A neural nework consiss of a large number of simple processing elemens called neurons or nodes. Each neuron is conneced o oher neurons by means of direced communicaion links, each wih an associaed weigh. The weighs represen informaion being used by he nework o solve a problem. Neural neworks are suiable for solving various asks including ime series forecasing. The emporal srucure of an analysed sample can be buil ino he operaion of a neural nework in implici or explici way [6]. In he firs case he emporal srucure of he inpu signal is embedded in he spaial srucure of he nework. The inpu signal is usually uniformly sampled, and he sequence of synapic weighs of each neuron conneced o he inpu layer of he nework is convolved wih a differen sequence of inpu samples. Explici represenaion of emporal srucure, when ime is given is own paricular represenaion, is used rarely. Therefore, only he implici represenaion of ime, whereby a saic neural nework is provided wih dynamic properies, has been implemened in his research. For a neural nework o be dynamic, i mus be given memory, which may be divided ino shorerm and long-erm memory. Long-erm memory is buil ino a neural nework hrough supervised 171

3 Session 2. Saisical Mehods and Their Applicaions learning, whereby he informaion conen of he raining daa se is sored in he synapic weighs of he nework. However, if he ask has a emporal dimension, we need some form of shor-erm memory o make he nework dynamic. One way of building shor-erm memory ino he srucure of a neural nework is hrough he use of ime delays which can be implemened a he synapic level inside he nework or a he inpu layer of he nework. a) b) Figure 3.1 Temporal processing using neural neworks: a) nonlinear filer buil on a saic neural nework [6]; b) ime lagged feed-forward nework (TLFN) [6] [7] Temporal paern recogniion requires processing of paerns ha evolve over ime, wih he response a a paricular insan of ime depending no only on he presen value of he inpu bu also on is pas values. Figure 3.1(a) shows he block diagram of a nonlinear filer buil on a saic neural nework. Given a specific inpu signal consising of he curren value x(n) and he p pas values x(n 1),, x(n p) sored in a delay line memory of order p, he free parameers of he nework are adjused o minimize he raining error (he mean square error) beween he oupu of he nework, y(n), and he desired response d(n) [6]. The srucure shown in Figure 3.1(a) can be implemened a he level of a single neuron or a nework of neurons. A ime lagged feed-forward nework is shown in Figure3.1(b). I consiss of a apped delay memory of order p and a mulilayer percepron (MLP). A sandard back-propagaion algorihm can be used o rain his ype of neural neworks ARIMA Models The processes of auo-regression, moving average and heir combinaions refer o he class of linear models, as all he relaions beween he observaions and random errors of a ime series are expressed by means of linear mahemaical operaions. In conras o simulaed raffic, he real raffic usually incorporaes seasonal and / or cyclic componens. In his case one should pay his / her aenion o he seasonal modificaions of ARIMA. The ARIMA is he Box-Jenkins varian of convenional ARMA models, which is predesinaed for applicaions o non-saionary ime series ha become saionary afer heir differencing. In he case of seasonal ARIMA models, seasonal differencing is also applied in order o eliminae a seasonal componen of period s. If d and D are non-negaive inegers, hen {x } is a seasonal ARIMA(p,d,q)(P,D,Q) process given by [8]: p P s d D s q Q s φ ( B) Φ ( B ) x = θ ( B) Θ ( B ) ε, (3.1) where s period of a cyclic componen; B delay operaor; φ(b) auoregressive operaor of order p; θ(b) moving-average operaor of order q; Φ(B s ) seasonal auoregressive operaor of order P; Θ(B s ) seasonal moving-average operaor of order Q; differencing operaor given by = 1 =1 B; s seasonal differencing operaor given by s =1 B s ε whie noise. 172

4 The 9 h Inernaional Conference RELIABILITY and STATISTICS in TRANSPORTATION and COMMUNICATION The operaors φ(b), θ(b), Φ(B s ) and Θ(B s ) have o saisfy he condiions of saionariy and reversibiliy. The indexes p, P, q and Q are inroduced here in order o remind abou differen orders of he operaors. The descripion of he ARIMA process incorporaing wo and more periodic componens is analogous o his Exponenial Smoohing The mehod of exponenial smoohing is he generalizaion of moving average echnique. I allows building he descripion of a process whereby he laes observaions are given larges weighs in comparison wih earlier observaions, and he weighs are exponenially decreasing. There exis differen modificaions of exponenial smoohing which are suiable for modelling and forecasing he ime series incorporaing linear / non-linear rends and / or seasonal flucuaions. Such models are based on he decomposiion of ime series. Jus as in he case of ARIMA models, he ask of forecasing real nework raffic requires applying he seasonal modificaions of exponenial smoohing. In his research he model of exponenial smoohing wih addiive seasonaliy was implemened o consan-level processes. Is mahemaical expression is given by [9]: S I = α ( x I p ) + (1 α) S 1 = δ ( x S ) + (1 δ ) I p where α smoohing parameer for he level of he series; S smoohed level of he series, compued afer x is observed; δ smoohing parameer for seasonal facors; I smoohed seasonal index a he end of he period ; p number of periods in he seasonal cycle. In his case he forecas is calculaed as follows [9]: x ( l) = S + I s+ l, (3.2) ˆ, (3.3) where xˆ ( l) forecas for l periods ahead from origin. Nework raffic measured over long ime periods (several years) usually incorporaes no only seasonal flucuaions bu also a linear rend. Then i is necessary o use seasonal rend modificaions of exponenial smoohing, he descripion of which can be found in [9]. 4. Pracical Research Sixeen ime series of differen lengh and differen aggregaion periods have been analysed in he process of he research. The measuremens were aken on he ransporaion level and characerize hree variables he ransmission rae of ougoing inernaional raffic of he IP-nework, he ransmission rae of oal ougoing raffic of he IP-nework and he inensiy of he oal serviced load of he convenional elephone nework. The main aim of he pracical research was o analyse he saisical properies of he ime series and o develop such a neural nework which is suiable for modelling he underlying process and producing a reliable forecas for a pre-defined forecasing horizon. The selecion of he relevan neural nework closely followed an advanced algorihm inroduced in [10]. A secondary ask of he research was evaluaing he influence of he size of he basic sample on he effeciveness and reliabiliy of forecasing. The size of he basic sample was equal o 9, 12, and, in some cases, 18 weeks. The forecasing period (i.e. he size of a esing sample) varied from one o 14 days. The period of averaging he errors of one-sep-ahead forecass was se up o one day. Following he ITU-T Recommendaion E.492 [4] he iniial raffic measuremens were averaged over 15-minues and one-hour periods. The main characerisics of he ime series are shown in Table 4.1. All he analysed ime series are characerized by he presence of seasonal componens wih periods of 24 hours and one week. Tha was revealed by applying a Fourier analysis. The esimaes of he 173

5 Session 2. Saisical Mehods and Their Applicaions Hurs exponen vary from 0.65 for elephone raffic o 0.85 for packe-swiched raffic. Such values indicae he persisence of analysed ime series and exploi he poenialiies of heir furher forecasing. Table 4.1. The main characerisics of analysed ime series Nework Type IPnework Traffic Type Ougoing inernaional raffic Aggregaion Period 15 min. 1 hour Labelling Basic Sample Size of Basic Sample A May 12 Jul. 13, weeks (6048 obs.) B May 12 Aug. 3, weeks (8064 obs.) С May 12 Jul. 13, weeks (1512 obs.) D May 12 Aug. 3, weeks (2016 obs.) Size of Tesing Sample 1-14 days E Feb. 4 Apr. 6, weeks (6048 obs.) IPnework Toal ougoing raffic 15 min. 1 hour F Feb. 4 Apr. 27, weeks (8064 obs.) G Feb. 4 June 8, weeks (12096 obs.) H Feb. 4 Apr. 6, weeks (1512 obs.) I Feb. 4 Apr. 27, weeks (2016 obs.) 1-14 days J Feb. 4 June 8, weeks (3024 obs.) K Jan. 8 Mar. 11, weeks (6048 obs.) Telephone nework (POTS) Toal serviced raffic 15 min. 1 hour L Jan. 8 Apr. 1, weeks (8064 obs.) M Jan. 8 May 13, weeks (12096 obs.) N Jan. 8 Mar. 11, weeks (1512 obs.) O Jan. 8 Apr. 1, weeks (2016 obs.) 1-14 days P Jan. 8 May 13, weeks (3024 obs.) The specificaion of he developed neural nework is displayed in Table 4.2. Neural neworks belong o he so-called heurisic mehods. I means ha he relevan values of mos parameers of neural neworks have o be evaluaed in experimenal way. Table 4.2. The main parameers of he developed neural nework 1 Sage Parameer / Procedure Parameer Value / Procedure Descripion Selecion of nework opology Training Type of opology Time-lagged feed-forward nework (mulilayer percepron) Number of hidden layers 1 Number of hidden neurons Varying from 1 o 10 Number of oupu neurons 1 Acivaion funcion Hidden layer hyperbolic angen; oupu layer linear funcion Number of raining epochs 600 Training algorihm Back propagaion (100 epochs) & conjugae gradien descen (1000 epochs) Error funcion Mean square error Learning rae 0.1 Momenum erm 0.3 Mehod of iniialisaion of weighs and biases Randomised values from a uniform disribuion wih a range of [-0.5;0.5] Number of imes o randomise weighs and biases 100 Mehods o preven over-learning Cross-validaion, weigh regularizaion [13] The size of raining, validaion and es subses In 3:1:1 proporion Sopping crierion Training error is invariable during 50 epochs In-sample The parameers of in-sample evaluaion R, MAE, RMSE, MAPE, AIC, BIC and ou-ofsample Diagnosic esing of residuals Lagrange muliplier ype es [14], χ 2 - es evaluaion The parameers of ou-of-sample evaluaion RMSE, MAE, MAPE, he Diebold-Mariano crierion 1 Noaions: R correlaion coefficien, MAE mean square error, RMSE roo mean square error, MAPE mean absolue percenage error, AIC Akaike s informaion crierion, BIC Bayesian informaion crierion. 174

6 The 9 h Inernaional Conference RELIABILITY and STATISTICS in TRANSPORTATION and COMMUNICATION The appropriae archiecure of a neural nework was defined as follows. According o he universal approximaion heorem [11] he number of hidden layers was equal o one. The size of he inpu window was equal o he larges period of he cyclic componen idenified by means of a Fourier analysis. The number of oupu neurons was equal o one and implied a one-sep ahead forecasing. In order o idenify he relevan number of hidden neurons all he archiecures wih he number of hidden neurons varying form one o en have been esed and verified. A wo-sage raining process was implemened. During he firs sage a mulilayer percepron was rained by applying he back propagaion during one hundred epochs, wih learning rae 0.1 and momenum 0.3. I usually gives he opporuniy o locae he approximae posiion of a reasonable minimum. During he second sage, a long period of conjugae gradien descen (1000 epochs) is used, wih a sopping window of 50, o erminae raining once convergence sops or over-learning occurs. Once he algorihm sops, he bes nework from he raining run is resored. The final neural nework was chosen in compliance wih he mehod suggesed in [10]. According o ha, among compeing neural neworks he model wih uncorrelaed residuals and he smalles value of he informaion crierion (IC) has o be chosen for furher forecasing. The final forecass produced by neural neworks were compared o hose which are produced by he seasonal ARIMA and seasonal exponenial smoohing. The Diebold-Mariano es [12] has been applied in order o evaluae saisically significan differences beween hese forecass. The es is non-parameric and can be used even if forecasing errors do no comply wih he classic requiremens, i.e. hey are nonnormally disribued, auo-correlaed or serially correlaed. For he sake of space saving, only one empirical example illusraing he producion of he forecass for he ime series (B) is shown here. However, he main conclusions have been drawn aking ino accoun he whole se of produced forecass and he complee resuls of verificaion. As a resul of verificaion procedures, hree models have been chosen for furher forecasing of he ime series (B) a ime lagged mulilayer percepron wih one hidden neuron MLP , a seasonal model (1,0,6)(0,1,1) 672, and he model of exponenial smoohing wih addiive seasonaliy and parameers α = 0.19 и γ = The final forecass produced by hese models over a forecasing horizon up o wo weeks are shown in Figure 4.1. The sandard esimaes of he qualiy of he produced pseudo-forecass are shown in Figure 4.2. The values of hese parameers do no differ significanly and vary depending on a forecasing period. Therefore, i is hard o say which model performs beer han ohers. The Diebold-Mariano es [12] has been implemened in order o idenify saisically significan differences beween produced forecass for hree forecasing horizons such as 24 hours, a week and wo weeks. As we can see in Table 5.1, here are Transmission Rae, Mbps Hisorical Daa Forecas (MLP ) Observaion Number Transmission Rae, Mbps Hisorical Daa Forecas [(1,0,6)(0,1,1)672] Observaion Number Transmission Rae, Mbps Hisorical Daa Forecas ( alfa = 0.19, gamma = 0) Observaion Number Figure 4.1. Final forecass of he ime series (B) produced by differen models 175

7 Session 2. Saisical Mehods and Their Applicaions no saisically significan differences beween forecass produced by he neural nework and over he forecasing horizons of 24 hours and one week. However, as a forecasing horizon increases, he qualiy of forecass produced by he neural nework deerioraes. Thus, he model ouperforms he neural nework over a forecasing horizon of wo weeks. On he oher hand, forecass produced by he neural nework perform beer han hose produced by exponenial smoohing independenly of a forecasing horizon. 2,8 2,4 MAE 2,8 MAE MAE MAE 2 2 1,2 1, Forecasing Horizon Forecasing Period 3,0 RMSE 5,2 3,6 RMSE RMSE 2,0 RMSE 2,0 0,4 1, Forecasing Horizon Forecasing Horizon MAPE, % MAPE,% MAPE, % MAPE,% Forecasing Horizon Forecasing Period a) b) Figure 4.2. Ou-of-sample evaluaion of he qualiy of forecass a) depending on he forecasing horizon; b) for successive forecasing periods of 96 observaions (24 hours) Table 4.3. The evaluaion of saisically significan differences beween final forecass of he ime series (B) by means of he Diebold -Mariano es 2 Models Forecasing Horizon (L) & Sep (h) L=96, h=1 L=672, h=1 L=1344, h=1 Neural Nework vs (0.67) 0.80 (0.42) Neural Nework vs. Seasonal Exponenial Smoohing 2.40 (0.02) vs. Seasonal Exponenial Smoohing 2.59 (0.01) The significance level of he Diebold-Mariano es is shown in brackes and highlighed in bold for he values less han

8 The 9 h Inernaional Conference RELIABILITY and STATISTICS in TRANSPORTATION and COMMUNICATION Despie of he resul of his paricular comparaive sudy he comparison of forecass produced for oher ime series by applying neural neworks, models and exponenial smoohing, in mos cases did no reveal any saisically significan differences. 5. Conclusions The resuls of he research show ha in mos cases he differences in qualiy beween shor-erm forecass of nework raffic produced by neural neworks and linear models are no saisically significan. Therefore, conrary o popular belief, he use of such complicaed and ime-consuming mehods as neural neworks is no always appropriae. This issue requires furher research wih he aim of specifying he condiions under which he mechanism of neural neworks has o be applied o forecasing he raffic of elecommunicaions neworks. Some oher imporan conclusions have been made as well: A long-range dependence and a slowly decaying variance of he packe-swiched raffic are apparen only for he measuremens aken over a very fine scale, usually over he periods up o minues. If according o he ITU-T recommendaions he measuremens of raffic are averaged over larges periods han he radiional linear mehods can be implemened for forecasing purposes as well. Unlike simulaed ime series, real ime series usually incorporae prominen seasonal flucuaions which are he resul of human behaviour. A neural nework can model and forecas seasonal ime series wihou prior de-seasonalisaion. In his case he mos imporan parameer o define is he size of he inpu window which has o be equal o he larges period of a seasonal componen. The ask of forecasing nework raffic incorporaing periodic flucuaions requires focusing on he seasonal modificaions of linear models. Jus as in he case of neural neworks, he correc idenificaion of he periods of seasonal componens is imporan for successful modelling and forecasing. References 1. ITU-T (CCITT) Recommendaion E.506 (rev.1). Forecasing Inernaional Traffic. Geneva, ITU-T Recommendaion E.507. Models for Forecasing Inernaional Traffic. ITU, 1993 (1988). 3. Sheluhin, O.I., Smolskiy, S.M. and Osin, A.V. Self-Similar Processes in Telecommunicaions. John Willey & Sons, ITU-T Recommendaion E.492. Traffic Reference Period (Telephone Nework and ISDN. Qualiy of Service, Nework Managemen and Traffic Engineering). ITU, 02/ Fowler, T.B. A Shor Tuorial on Fracals and Inerne Traffic, The Telecommunicaions Review, No 10, 1999, pp Haykin, S. Neural Neworks: A Comprehensive Foundaion (2 nd ed.). Prenice Hall, Mozer, M. C. Neural Nework Archiecures for Temporal Paern Processing. In: Time Series Predicion: Forecasing he Fuure and Undersanding he Pas / Eds. A.S. Weigend and N. A. Gershenfeld. Perseus Book Publishing, 1993, pp Box, G. and Jenkins, G. Time Series Analysis: Forecasing and Conrol. San Francisco: Holden-Day, Gardner, E.S.Jr. Exponenial Smoohing: The Sae of Ar, Journal of Forecasing, No 4, 1985, pp Klevecka, I. An Advanced Algorihm for Forecasing Traffic Loads by Neural Neworks. In: Scienific Proceedings of Riga Technical Universiy (Series "Telecommunicaions and Elecronics"), No 9, (Submied for publicaion) 11. Cybenko, G. Approximaion by Super posiions of Sigmoidal Funcion, Mahemaics of Conrol, Signals, and Sysems, No 2, 1989, pp Diebold, F.C. and Mariano, R.S. Comparing Predicive Accuracy, Journal of Business & Economic Saisics, Vol. 13, No 3, July 1995, pp Weigend, A.S., Rumelhar, D.E. and Huberman, B.A. Generalizaion by Weigh-Eliminaion wih Applicaion o Forecasing. In: Proceedings of he 1990 Conference on Advances in Neural Informaion Processing Sysems, No 3, 1990, pp Medeiros, M.C., Teräsvira, T. and Rech, G. Building Neural Nework Models for Time Series: A Saisical Approach, Journal of Forecasing, No 25, 2006, pp