TELKOMNIKA, Vol.11, No.11, November 2013, pp. 6547~6555 e-issn: 2087-278X 6547 Software Agg Predcto based o Extreme Learg Mache Xaozh Du 1, Hum Lu* 2, Gag Lu 2 1 School of Software Egeerg, X a Jaotog Uversty, X a 710049, Shaax, Cha 2 School of Software Egeerg, Chagchu Uversty of Techology, Chagchu, 130012, Jl, Cha *Correspodg author, e-mal: luhm.cc@gmal.com Abstract I the research o software agg ad rejuveato, oe of the most mportat questos s whe to trgger the rejuveato acto. Ad t s useful to predct the system resource utlzato state effcetly for determg the rejuveato tme. I ths paper, we propose software agg predcto model based o extreme learg mache (ELM) for a real VOD system. Frst, the data o the parameters of system resources ad applcato server are collected. The, the data s preprocessed by ormalzato ad prcpal compoet aalyss (PCA). The, ELMs are costructed to model the extracted data seres of systematc parameters. Fally, we get the predcted data of system resource by computg the sum of the outputs of these ELMs. Expermets show that the proposed software agg predcto method based o wavelet trasform ad ELM s superor to the artfcal eural etwork (ANN) ad support vector mache (SVM) the aspects of predcto precso ad effcecy. Based o the models employed here, software rejuveato polces ca be trggered by actual measuremets. Keywords: software agg, extreme learg mache, predcto Copyrght 2013 Uverstas Ahmad Dahla. All rghts reserved. 1. Itroducto The software relablty ad avalablty are creasgly beg demaded preset software systems [1]. Whle recet studes show that whe software applcato s executed cotuously for log tervals of tme, some error codtos them are accumulated to result performace degradato or eve a crash falure, whch s called software agg [2]. The pheomeo has bee observed may software systems, such as operatg system [3], web server [4], SOA server [5], ad so o. Because of the effect of software agg, the system relablty decreases. To couteract software agg ad ts related traset software falures, a prevetve ad proactve techque, called software rejuveato, has bee proposed ad s becomg popular [2]. It volves stoppg the rug software occasoally, cleag ts teral state ad or ts evromet ad restartg t. A extreme but well-kow example of software rejuveato s the hardware reboot [6]. I geeral, the cost of software rejuveato s substatally lower tha the cost of a system falure followed by a reactve recovery. Over the recet years, quattatve studes of software agg ad rejuveato have bee take, ad may dfferet approaches have bee developed ad the effects of software rejuveato have bee studed. These studes ca be categorzed to two kds, tme-based rejuveato polcy ad measure-based rejuveato polcy. The tme-based rejuveato polcy s characterzed by the fact that the software s perodcally rejuveated every tme a predefed tme costat has elapsed. I these approaches, a certa falure tme dstrbuto s assumed ad cotuous tme Markov cha (CTMC) [2], sem-markov process [7], Markov regeeratve process (MRGP) [8], Markov decso process (MDP) [9] or stochastc Petr et (SPN) model [10] etc s developed to compute ad optmze system avalablty or related measures. The measure-based rejuveato polcy apples statstcal aalyss to the measured data o resource avalablty to predct the expected tme to resource exhausto [11], ad t provdes a wdow of tme durg whch a rejuveato acto s advsed. The basc dea of the measure-based rejuveato polcy s to motor ad collect data o the attrbutes ad parameters, whch are resposble for determg the health of the rug software system. Garg et al. [3] proposed a methodology to detect ad estmate the agg the UNIX system, they mplemeted a SNMP based tool to collect data, ad adopted o-parametrc statstc method to detect ad estmate Receved Aprl 15, 2013; Revsed Jue 19, 2013; Accepted July 20, 2013
6548 e-issn: 2087-278X agg. Adrzejak et al. [6] used a sple-based descrpto of the agg profles ad adopted a statstcal test to verfy ts correctess of the SOAP server. Adrzejak et al. [12] also used mache learg methods to model ad predct the software agg of a web applcato. Grottke et al. [4] used o-parametrc statstcal methods to detect ad estmate treds of agg, ad adopted AR model to predct the agg of a Web server. Hoffma et al. [13] gave a practce gude to resource forecastg, they adopted several methods to model ad predct the software agg of a Web server, they foud that probablstc wrapper (PWA) was a better method for varable select, ad support vector mache (SVM) was a better approach for resource forecastg. For the predcto of tme seres, artfcal eural etwork (ANN) [14] ad support vector mache (SVM) [15] are wdely adopted. Artfcal eural etworks, especally BP etworks, are powerful tools for fttg olear tme seres. However, there are some dsadvatages mplemetg of artfcal eural etworks. Frstly t s hard to determe the parameters of euros ad the etwork structure. Furthermore, the trag process of eural etworks s tme-cosumg ad the covergece rate s slow, because the etworks ofte settle udesrable local mma of the error surface. Whe support vector mache s used for predcto, t also faces some dsadvatages ad challeges, such as slow learg rate, multparameters to be determed, ad so o. I order to overcome some challeges of ANN ad SVM, extreme learg mache (ELM) proposed by Huage et al.[16] has attracted the more ad more atteto recetly. ELM s adopted for geeralze sgle-hdde layer feedforward etworks (SLFNs), ad the hdde layer eed ot be tued, whch results better geeralzato performace, faster learg speed, ad least huma teracto. I ths paper, we aalyze the software agg pheomeo of a real VOD system, ad propose a software agg predcto model based o extreme learg mache. The ma cotrbutos of ths paper are 1) proposg a software agg predcto model based o ELM, 2) applyg PCA to reduce the dmeso of put varables of ELMs, 3) usg the data collected from a real VOD system to evaluate the software agg predcto performace. 2. Software Agg Predcto Model based o ELM I order to provde support for trggerg software rejuveato actos, we eed to predct the system resource utlzato state precsely to reflect the software system state the future. Extreme learg mache, whch s a learg algorthm, proposed by Huag et al. [16] was developed for sgle-hdde layer feedforward eural etworks (SLFNs). ELM provdes good geeralzato performace at extremely fast learg speed by choosg hdde odes radomly ad determg the output weghts of SFLNs aalytcally. Therefore, we preset a software agg predcto model based o extreme learg mache, show Fgure 1, whch llustrates the k -step predcto procedure. Fgure 1. Software Agg Predcto Model From Fgure 1, we see that the data are frstly preprocessed after they are collected. The the tme seres of target parameter yt () s putted to wavelet trasform module, ad the detal compoets D (), t 1,..., pad the approxmato compoet A() p t of yt () are gotte, where p s the decomposto level assged by user, t s the sample dex. The other TELKOMNIKA Vol. 11, No. 11, November 2013: 6547 6555
TELKOMNIKA e-issn: 2087-278X 6549 parameters x ( t), 1, 2,..., are putted to PCA module, ad the frst m prcpals z ( t), 1, 2,..., m are selected, where m. The we costruct p 1 ELMs to forecast the decomposed compoets of target parameter separately. Each compoet ( A() t or D (), t 1,..., p) ad all the frst m prcpals z ( t), 1, 2,..., m are the puts of the respectve ELM, ad the output of the ELM s the k -step predcto value of the compoet ( A* ( t k) or D* ( tk), 1,..., p). Fally, all the outputs of these p 1 ELMs are summed to obta the k - step predcto value of the object parameter y*( t k). The detal steps of our approach are as follows: 1) Data Preprocess Data preprocess cludes two phases: (1) Parameter reducto ad selecto. I ths phase, the parameters whch are costat values durg the motorg perod ad the parameters that have the same meags are excluded. (2) Normalzato. I ths phase, all the selected parameters are ormalzed to elmate dmeso fluece. After ormalzato, the rage of all these parameters are lmted to [-1, 1]. 2) Prcpal Compoet Aalyss I the expermets, we collect 30 parameters of memory, 15 parameters of CPU, 17 parameters of dsk ad 27 parameters of applcato server. Some parameters are costat durg the observato terval, such as Commt Lmt of memory, C2 Trastos/sec of CPU ad so o. Some parameters have the same meag, such as Avalable Kbytes ad Avalable Mbytes of memory etc. After all these parameters are excluded, there are stll 32 parameters left. The relatoshp betwee software agg ad these parameters ca be expressed as the Equato (1): y f x1 x2 x (,,, ) (1) p p Where y deotes the avalable bytes of memory ad x 1, x 2,, x are the mpact factors of software agg the VOD system, ad here equals 32. If we take all these 32 parameters as the puts of the ELMs drectly, the etwork scale s very large, ad ts effcecy s very low. PCA [17] s a essetal method of multvarate statstcal aalyss, whch selects several represetatve prcple compoets to expla most of the data chages. Therefore, order to mprove the effcecy ad to keep the accuracy of our predcto model, PCA s adopted to reduce the put parameters here. Let the samples of these factors be X ( X1, X2, X ) T, the the procedure of PCA s as follows: (1) Normalze the samples X to remove dmeso fluece. (2) Calculate the relatve matrx P ad covarace matrx C of the sample data, ad egevalues 1, 2, ad egevectors are obtaed. (3) Calculate the cotrbuto rate of each compoet respectvely. The choce of prcpal compoet s determed based o varace cotrbuto rate ad cumulatve cotrbuto rate. The varace cotrbuto rates are calculated by the Equato (2): /( ) k (1,2,, ) (2) k k 1 Ad the cumulatve cotrbuto rate for each prcple compoet s gotte by the Equato (3): m /( ) m(1,2,, ) m j j1 1 (3) The hgher of the meas the stroger of the ablty for the frst prcpal compoet to 1 abstract the formato of x 1, x 2,, x. If the accumulato cotrbuto rate of the frst m Software Agg Predcto based o Extreme Learg Mache (Xaozh Du)
6550 e-issn: 2087-278X compoets s more tha a predetermed threshold (such as 85 percet), the frst m compoets are selected as the puts of the ELM. After PCA s fshed, formula (1) ca be reduced to the Equato (4): y f( z, z, z ) (4) 2 m where y deotes the avalable bytes of memory ad z 1, z 2,, zm s the prcpal compoets of agg mpact factors of the VOD system, ad here m. 3) Extreme Learg Maches Extreme learg mache, whch s a learg algorthm, proposed by Huag et al. [16] was developed for sgle-hdde layer feed forward eural etworks (SLFNs). ELM provdes good geeralzato performace at extremely fast learg speed by choosg hdde odes radomly ad determg the output weghts of SFLNs aalytcally. Here we cosder a sglehdde layer feed forward etwork wth L hdde euros. The put X ( z1, z2,..., zm, y) s a vector wth m+1 elemets, the output of the th hdde euro s Ga (, b, X ), where b s the bas, ad a ( a 1, a2,..., am, ay) s the weght vector, as ( s 1, 2,..., m, y) s the coecto weght betwee the th hdde euro ad the s th put euro. The the output of the SLFN s gve by the Equato (5): L yt ( k) f( X) Ga (, b, X) (5) 1 Where, ( 1,...,, y)' s the weght vector coectg hdde layer wth output layer, s the coecto weght betwee the th hdde euro ad the k th output euro. For the k case of addtve hdde euros, Ga (, b, X ) takes the followg form show by the Equato (6): Ga (, b, X) ga ( ' X b) (6) Where g: R R s the actvato fucto. m Assume that N arbtrary samples ( X, Y) R R are gve, the weght vectors a ad bas b are radomly assged. The the SLFN wth L hdde euros ca approxmate the N samples wth zero error f ad oly f there exsts, so that we get Y j by Equato (7): L Y G( a, b, X ), j 1,2,..., N (7) j j 1 The above N equatos ca be rewrtte the followg compact form show by Equato (8): H Y (8) T T Ga (, b, X1) Ga ( L, bl, X1) 1 Y 1 where H,,. Y Ga (, b, XN) Ga ( L, bl, XN) T NL T L Y L m N NM Oce the hdde ode parameters ( a, b ) are geerated radomly, they rema fxed. The trag the SLFN s equvalet to fdg the mmum orm least-squares soluto *, whch s gve by the Equato (9): * H Y (9) TELKOMNIKA Vol. 11, No. 11, November 2013: 6547 6555
TELKOMNIKA e-issn: 2087-278X 6551 where H s the Moore Perose geeralzed verse of matrx H. For the trag data set {( X, Y)}, the ELM algorthm [16] s descrbed as follows: 1,2,..., N Step 1: Assg the hdde ode umber L, ad the actvato fucto g(.) ; Step 2: For 1, 2,..., L, radomly geerate the put weght vector a ad the bas b Step 3: Calculate the hdde layer output matrx H ad H ; * Step 4: Accordg Equato (9), calculate the output weght vector. For ay put sample x R, the output value y * s calculated by usg the Equato (10): L * * y g( axb) (10) 1 The, we get the k -step predcto value of the object parameter, ad the procedure s fshed. 3. Results ad Aalyss 3.1. Expermetal Setup The expermetal evromet s a real VOD system, ad ts structure s show Fgure 2. The VOD system cossts of a web server, a applcato server (or vdeo server) ad a dsk array. The web server acts as the presetato layer, whch provdes meda data to clets. Whe t receves the order request from a clet, the web server redrects ths request to the applcato server. The the clet makes coecto wth the applcato server ad receves the meda data from the applcato server drectly f the clet gets the permsso. The dsk array s used to store the meda data. I our expermets, we adopt Apache as the web server, ad Helx server as the applcato server. Fgure 2. Structure of the VOD System 3.2. Data Collectos Durg the expermets, we collect 30 parameters of memory, 15 parameters of CPU, 17 parameters of dsk ad 27 parameters of applcato server, the samplg terval s 3 mutes, 7500 samples of the system parameters are collected for 375 hours. The umber of clet access of the VOD system s llustrated Fgure 3, ad the tme seres of system avalable memory s show Fgure 4. From Fgure 3, t s foud that the umber of clet access shows a certa perodcty, however t has a large radomess ad t fluctuates frequetly. From Fgure 4, we see that the avalable memory shows a frequet ad large fluctuato, whch s caused by the stochastc arrval of clet access ad the software agg. Accordg to the tme seres of the collected system avalable memory ad the umber of clet access, we use Ma-Kedall method [4] to test whether there s software agg pheomeo the VOD system. Table 1 shows the results of tred test for the avalable memory ad the umber of clet access. From Table 1, we fd that there s a dowward tred the tme seres of the avalable memory, ad the tme seres of the umber of clet access also has a dowward tred. That s, the dowward tred of the Software Agg Predcto based o Extreme Learg Mache (Xaozh Du)
6552 e-issn: 2087-278X system avalable memory s ot caused by the clets. Therefore, t ca be cocluded that there exsts software agg pheomeo the VOD system. Fgure 3. Number of Clet Access Fgure 4. System Avalable Memory Table 1. Tred Test data Zstatstc commet avalable memory -41.1374 Dowward tred detected umber of clet access -30.7877 Dowward tred detected 3.3. Software Agg Predcto I order to evaluate the performace of software agg predcto, root mea square error (RMSE) s adopted as dcator. RMSE s the square root of the varace of the resduals, ad t ca be terpreted as the stadard devato of the uexplaed varace. The lower the values of RMSE, the better the predcto result. RMSE s defed by the Equato (11):\ RMSE N 1 ( y ( ) y ˆ( )) N 2 (11) Where, y () deotes the actual value of the tme seres of the avalable memory, y ˆ( ) s the respectve predcto value, ad N s the pots of data set. I the expermets, the umber of prcpal compoets m s set to 2. For the ELM, we use a sgmod actvato fucto ad the umber of hdde euros s 47. The frst 4500 samples are adopted to tra these ELM, ad the resdual 3000 samples are used to test whether our method s effectve. Fgure 5 shows the oe-step forward predcto value of the avalable memory. (a) Predcto data (b) Predcto error Fgure 5. Predcto Results of System Avalable Memory TELKOMNIKA Vol. 11, No. 11, November 2013: 6547 6555
TELKOMNIKA e-issn: 2087-278X 6553 From Fgure 5, we see that the error betwee the predcto data ad the actual data of the avalable memory resource s very low. The we coclude that the predcto model proposed by us s sutable for software agg forecastg. We also fd that the predcto error teds to be larger at the valley of the resource cosumpto. The reaso s that there are more clets the system at the valley of the resource cosumpto, whch results more memory resource cosumpto ad more fluctuato, so the predcto precse decreases. Table 2 shows the approxmato performace of our software agg predcto model compared wth support vector mache (SVM), artfcal eural etwork model (ANN) wth BP algorthm. The smulatos for our model, ANN are carred out MATLAB R2007a evromet rug a Core2 Duo CPU, 3GHz. The smulato for SVM s carred out by usg the LIBSVM [18] mplemeted C code rug the same PC. The umber of hdde euros of ANN s set 42, ad the kerel fucto used SVM s radal bass fucto. I our expermets, all the puts ad the outputs have bee ormalzed to [-1, 1]. 20 trals have bee coducted for all the methods ad the average results are adopted. Table 2. Predcto Results of Varous Models Model Trag data Testg data RMSE Tme (s) RMSE Tme(s) Our method 0.0033 0.084 0.0406 0.021 SVM 0.0186 3.797 0.0445 3.734 ANN 0.0214 139.629 0.0698 0.0391 From Table 2, t ca be see that the predcto precso of our method s superor to that of ANN ad SVM. The trag tme ad the testg tme of our method are far lower tha that of SVM, ad the trag tme of our model s far lower tha that of ANN. For the trag data, the RMSE of our method s 0.0033, ad for the testg data, the RMSE s 0.0406. the trag tme ad the testg tme of our method are 0.084 secods ad 0.021 secods, whch show that the effcecy of our method s a very hgh. Therefore, the method we preseted s effectve to forecast the software agg process. 3.4. Sestvty Aalyss We perform sestvty aalyss for our preseted model that predcts the avalable memory of the VOD system by addg a prcpal compoet at a tme, ad calculate the model s chage RMSE. The result s show Fgure 6. Fgure 6. RMSE versus the Number of Prcpal Compoets From Fgure 6, t s show that wth the crease of the umber of prcpal compoets, the RMSE decreases, whe the umber of prcpal compoets arrves to a certa value, the mmum RMSE s gotte. The wth the cotuous crease of the umber of prcpal compoets, the RMSE creases. The reaso s that wth the crease of the umber of prcpal compoet, the more formato of put varables s cluded, but except the frst several prcpal compoets, the resdual compoets have trval formato, ad wth more Software Agg Predcto based o Extreme Learg Mache (Xaozh Du)
6554 e-issn: 2087-278X prcpal compoets, the ELMs are becomg more complex, whch results the crease of RMSE. Therefore, we should select the approprate umber of prcpal compoets based o requremet. 4. Cocluso I ths paper, we have vestgated the software agg pheomeo of a real VOD system, ad have proposed a software agg predcto model base o extreme learg mache. The expermetal results have showe that the proposed software agg predcto model s effectve to forecast the agg progress, ad the PCA s a mportat ad useful method to reduce the redudacy of data. Compared wth SVM ad ANN predcto model, our model s more effectve o precso tha ANN ad SVM. Ad the tme for trag ad testg of our model s far lower tha that of ANN ad SVM. ELM s a quck ad effcet method for resolvg the predcto problem, but how to determe the umber of hdde euros s a ope ssue, though several methods have bee preseted, t s stll a challege for the ELM. Therefore, the future we wll study ths ssue. Though ths paper we have study the software agg predcto problem, the cause resultg software agg s stll pedg, so ext we wll explore ths problem. Ackowledgemets Ths work was supported by the Natoal Natural Scece Foudato of Cha uder Grat No. 60933003, the Natoal Natural Scece Foudato of Cha uder Grat No. 61240029, the Natoal Postdoctoral Scece Foudato of Cha uder Grat No. 2011M500611, the Idustral Techology Research ad Developmet Specal Project of Jl Provce uder Grat No. 2011006-9; the Fudametal Research Fuds for the Cetral Uverstes; the Natoal College Studets' Iovatve Trag Program of Cha uder Grat No. 201210190037. Refereces [1] Paulso LD. Computer System, Heal Thyself. IEEE Computer. 2002; 35(8): 20-22. [2] Huag Y, Ktala C, Koletts N, Fulto D. Software Rejuveato: Aalyss, Module ad Applcatos. Proceedgs of the 25th Symp. O Fault Tolerat Computg. Pasadea, USA. 1995; 381-390. [3] Garg S, Pulafto A, Telek M, Trved KS. A Methodology for Detecto ad Estmato of Software Agg. Proceedgs of the Itl. Symp. O Software Relablty Egeerg. NJ, USA. 1998; 283-292. [4] Grottke M, L L, Vadyaatha K, Trved KS. Aalyss of Software Agg a Web Server. IEEE Trasactos o Relablty. 2006; 55(3): 411-420. [5] Slva L, Madera H, Slva JG. Software Agg ad Rejuveato a Soap-based Server. Proceedgs of the Ffth IEEE Iteratoal Symposum o Network Computg ad Applcatos. Massachusetts, USA. 2006; 56-65. [6] Aloso J, Matas R, Vcete E, et al. A Comparatve Expermetal Study of Software Rejuveato Overhead. Performace Evaluato. 2013; 70: 231-250. [7] Bao Y, Su X, Trved KS. A Workload-based Aalyss of Software Agg ad Rejuveato. IEEE Trasactos o Relablty. 2005; 54(3): 541-548. [8] Garg S, Pulafto A, Telek M, et al. Aalyss of Prevetve Mateace Trasactos Based Software Systems. IEEE Tras. o Computers. 1998; 47(1): 96-107. [9] Okamura H, Doh T. Dyamc Software Rejuveato Polces a Trasacto-based System uder Markova Arrval Processes. Performace Evaluato. 2013; 70: 197-211. [10] Wag D, Xe W, Trved KS. Performablty Aalyss of Clustered Systems wth Rejuveato uder Varyg Workload. Performace Evaluato. 2007; 247-265. [11] Vadyaatha K, Trved KS. A Comprehesve Model for Software Rejuveato. IEEE Trasactos o Depedable ad Secure Computg. 2005; 124-137. [12] Adrzejak A, Slva L. Usg Mache Learg for No-trusve Modelg ad Predcto of Software Agg. Proceedgs of the IEEE Network Operatos ad Maagemet Symposum. Salvador, Brazl. 2008; 25-32. [13] Hoffma GA, Trved KS, Malek M. A Best Practce Gude to Resource Forecastg for Computg Systems. IEEE Trasactos o Relablty. 2007; 56: 615-628. [14] Goh CK, Teoh EJ, Ta KC. Hybrd Multobjectve Evolutoary Desg for Artfcal Neural Networks. IEEE Trasactos o Neural Networks. 2008; 19: 1531-1548. TELKOMNIKA Vol. 11, No. 11, November 2013: 6547 6555
TELKOMNIKA e-issn: 2087-278X 6555 [15] Cao LJ, Tay FEH. Support Vector Mache wth Adaptve Parameters Facal Tme Seres Forecastg. IEEE Trasactos o Neural Networks. 2003; 14(6): 1506-1518. [16] Huag GB, Zhu QY, Sew CK. Extreme Learg Mache: Theory ad Applcatos. Neurocomputg. 2006; 70(1-3): 489-501. [17] Wag X, Kruger U, Irw GW, McCullough G, McDowel N. Nolear PCA wth the Local Approach for Desel Ege Fault Detecto ad Dagoss. IEEE Trasactos o Cotrol System Techology. 2008; 16: 122-129. [18] Ferrar S, Stegel RF. Smooth Fucto Approxmato Usg Neural Networks. IEEE Trasactos o Neural Networks. 2005; 16(1): 24-38. Software Agg Predcto based o Extreme Learg Mache (Xaozh Du)