Research on Cloud Computing and Its Application in Big Data Processing of Railway Passenger Flow

Transcription

1 325 A publcato of CHEMICAL ENGINEERING TRANSACTIONS VOL. 46, 2015 Guest Edtors: Peyu Re, Yacag L, Hupg Sog Copyrght 2015, AIDIC Servz S.r.l., ISBN ; ISSN The Itala Assocato of Chemcal Egeerg Ole at DOI: /CET Research o Cloud Computg ad Its Applcato Bg Data Processg of Ralway Passeger Flow Xao Yog a, Cheg Yg b, Fag Yaju a a Departmet of Automato, Wuha Uversty, Wuha , Cha, b Iformato Cetre, Wuha Uversty, Wuha , Cha. xao14@sa.com Moder ralway has a hgh speed, heavy load ad tesve developmet tred. It ot oly brgs the opportutes of ralway trasport capacty ad volume, but also makes the data of varous types of large-scale cotuous growth. I the stage of the developmet of the ralway by the producto eterprse to the servce eterprse, t s ecessary to vgorously promote the applcato of cloud computg, ad to pla the cloud computg framework. We study a parallel support vector mache model based o mult-level SVM, ad realze the parallel algorthm cloud computg evromet. The algorthm dvdes the large trag data set to a umber of small trag sets by Map, ad the a ew SVM s combed wth these small trag sets. Fally, the data s traed to be a ew SVM by Reduce. At the ed of the paper, we use the SVM parallel forecastg method to predct the passeger flow of Cha Ralway, ad compare the performace of the dstrbuted wth that of o-dstrbuted algorthms. Expermetal results show that the proposed algorthm has better effect tha sgle mache algorthm terms of tme cosumpto ad classfcato accuracy. Wth the crease of odes, the tme cosumpto s sgfcatly shorteed. 1. Itroducto Moder ralway has a hgh speed, heavy load ad tesve developmet tred. It ot oly brgs the opportutes of ralway trasport capacty ad volume, but also makes the data of varous types of large-scale cotuous growth. So t has brought a huge challege to the formato process of the ralway. I the stage of the developmet of the ralway by the producto eterprse to the servce eterprse, t s ecessary to vgorously promote the applcato of cloud computg, ad to pla the cloud computg framework (Le Yu Wa (2011), Sh Xalag (2011), ad Zhou Xag (2009)). Ralway formato ca make tra schedule wth the chage of passeger flow more targeted. The cloud platform ca be used to buld the bookg system cotas tcketg, refud, bookg appotmets ad may other subsystems cludg techcal support, mateace ad motorg of massve tcket formato (Wag Yl (2010) ad Wag Jua (2009)). Therefore, the processg capacty of the ralway passeger flow data the cloud platform s partcularly mportat. Sce MapReduce s a mportat method of bg data mg algorthm, the domestc ad foreg scholars have bee abudat the research of ths aspect. MapReduce programmg model based o cloud computg s a good soluto to the problem of massve formato processg parallel, ad the MapReduce model has the fucto of load balacg ad automatc processg the ode falure (Dea J C (2004) ad Doulkerds (2014)). Cheg Tao et al. (2006) ad M.Y. Eltabakh (2011) developed the MapReduce parallel programmg model to mache learg, ad realzed may kds of mache learg algorthms. Ku Deg (2011) ad C. Egle et al.'s (2012) report gave a algorthm of parallel SMO support vector m ache, whch s based o the sample set uform ad radomly dvded to a umber of data sets that are ot too small. It s more effectve for lear vector mache. Nasullah et al. (2011) ad F. Färber, et al. (2012) used ths dea to lear support vector mache. The expermet proved the effectveess of the proposed dea for lear support vector maches. MapReduce parallel computg framework has bee wdely cocered by researchers because of ts excellet performace ad the smplcty of programmg model dealg wth Please cte ths artcle as: Xao Y., Cheg Y., Fag Y.J., 2015, Research o cloud computg ad ts applcato bg data processg of ralway passeger flow, Chemcal Egeerg Trasactos, 46, DOI: /CET

2 326 large-scale data. So, a lot of data processg algorthms based o MapReduce parallel framework have bee mplemeted ad acheved good performace(y Chuaye (2014) ad Tag Duoyu (2013)). I ths paper, the applcato of cloud computg techology ralway large-scale data processg cludes two aspects: o the oe had, based o the characterstcs of the ralway resource dstrbuto, the paper proposes the method of usg Hadoop cloud computg techology to aalyze large data of ralway passeger flow. Ths method ca effectvely mprove the speed ad effcecy of data aalyss by costructg a cloud computg platform based o Hadoop. O the other had, accordg to the data characterstcs of the ralway, we study a parallel support vector mache model based o mult level SVM, ad realze the parallel algorthm cloud computg evromet. The algorthm dvdes the large trag data set to a umber of small trag sets by Map, ad the a ew SVM s combed wth these small trag sets. Fally, the data s traed to be a ew SVM by Reduce. At the ed of the paper, we use the SVM parallel forecastg method to predct the passeger flow of Cha Ralway, ad compare the performace of the dstrbuted wth that of o dstrbuted algorthms. 2. Cloud computg platform of ralway passeger flow data based o hadoop Wth the characterstcs of ralway passeger trasport, the bg data of ralway passeger flow the desg of the cloud platform usg dstrbuted ad herarchcal structure. As show Fgure 1, t ca be dvded to 3 layers: data layer, model layer, ad applcato layer. Data Layer Algort hm Aalyss The Chage of passeger flow The Chage of destato of passeger flow Applcato layer Tcketg system Tra umber Data warehouse(characterstcs of data) MAP HIVE Calculato Storage REDUCE HBASE Data layer( HDFS) Fgure 1: The archtecture dagram of bg data of ralway passeger flow cloud platform

3 327 (1) Data Layer Ralway passeger flow data cludg etwork data ad busess data. The data layer stores these data by HDFS Hadoop. Ad the, we ca use Hbase, Hve ad other data processg ad maagemet tools to dyamcally geerate Map-Reduce tasks. After that, through the hgh effcecy of the ralway passeger flow data, processg results are stll stored fle format HDFS. At the same tme, accordg to the specfc busess eeds, we ca export the requred format. (2) Model Layer. The model layer uses the processg results of the bg data of ralway passeger flow whch s processed the data layer, ad establshes the aalyss model, such as the statstcal model of the passeger's destato, the passeger's etwork bookg model, the herarchcal demad model, the passeger flow forecast model, etc.. It ca form a tegrated aalyss platform for the passeger's whereabouts ad the etwork bookg, so as to optmze the ralway trasportato system. (3) Applcato Layer The applcato layer aalyss the results of model layer, such as the characterstcs of the passeger's trajectory, ad the cocetrato of the pot of bookg, so as to desg a targeted sales program. It ca effectvely mprove the effcecy of passeger trasport of Cha's ralway. 3. Parallel support vector forecastg model based o map-reduce Support vector mache regresso has two types cludg lear regresso ad o-lear regresso. For lear regresso, lear regresso fucto s developed as follows: f () x x b. (1) Assume trag sample set D cosst of samples (x,y)(=1, 2,...,), where x X, y R are the best assessmets. I order to guaratee flat of Eq. (1), we eed to fd a smallest w. Therefore, uversal umber of mmzed Eucldea space s troduced ad some techologes such as dualty prcple ad lagraga multpler method are appled to fd smallest w. Here, a regresso fucto s obtaed as follows: 1 f ( x) a a x x b (2) The basc deal of o-lear support vector mache regresso s demostrated the followg. Frstly, data ca be mapped to hgh dmesoal feature space by a o-lear mappg. Secodly, lear regresso ca be mplemeted ths space. Therefore, the lear regresso the hgh dmesoal feature space ca correspod to o-lear regresso the low dmesoal put space, whch ca be acheved through core fucto, j j k x x x x. Thus, the followg equato ca be acqured: 1 max, 2 1 j1 1 1 s. t. a a, 1 1 a a a j a j k x x j a a y a a a, a [0,C], 1,2,..., (3) Where costat C >0 s amed pealty factor. If C s bg, ftess bas has bgger pealty. ε s bggest bas that regresso ca be allowed. The, regresso fucto ca be demostrated as follows: f ( x) a a k x, x b = k x, x b 1 1 (4) Parameter b ca be obtaed from the followg equato. b y a a k x, x, fa [0,C] l l l 1 yl a a k x xl fa 1,, [0,C] (5)

4 K( x, y) exp[ ( x y ) ] (6) Dfferet from the tradtoal sgle model method, the parallel mult model method ca be amed at dfferet clusterg to select dfferet kerel parameters for each sub model. Because kerel of Gauss s a typcal local fucto. So, the dstace betwee the data ad the test pot s gettg closer, the fluece o fucto value s hgher. We ca determe the kerel wdth parameter accordg to the average dstace betwee the data ad the clusterg ceter. It s used to esure that the data the cluster has a greater mpact o the value of the kerel fucto, whle the data the o cluster has lttle effect o the value of the kerel fucto. The calculato formula s as follows: l x c l, j 2 j1 1,, k; j 1,, l (7) Amog them: c s the ceter of th cluster, x, j s the of j th sample data of th cluster, l s the sample umber the th cluster. I ths paper, the large scale trag data set of SVM algorthm s dvded to several small trag sets by Map, ad the a ew SVM model s combed wth these small trag sets. That s to say, the ( 0 C ) correspods to the sample ( x, y ) as the put of Reduce. I the cloud computg evromet, the trag data set s automatcally completed by the Map-Reduce system. It s based o the user's Map put block sze, the bg data set s dvded to a umber of small trag sets that are ot greater tha the block sze of the Map put. The, small trag sets are the put for Map task, ad the Map tasks are depedet ad parallel executo. After each Map task completed the trag of ther small trag set, the sample data of the support vector mache s used as the output of Map. 4. Smulato expermet ad result aalyss Due to the legth of the artcle ad the author's lmted eergy, ths paper oly carres o the deploymet of the Hadoop cluster of the ralway passeger flow data, ad expermets are appled to demostrate the effectveess ad feasblty of the proposed method ad the desg of the platform. I order to verfy the performace of the hgh performace computg system uder the backgroud of bg data, ths paper uses the Hadoop cluster evromet s made up of 8 SYSTEM X3850 IBM server, each server s a quad core PC, each core as a Hadoop ode. Oe of them s NameNode, the other s the DataNode. I addto, order to compare the performace of the bechmark, we are rug a cotrast expermet o dstrbuted mode. After that, we use the fast Fourer-Trasform algorthm to geerate the test data sets of 3 dfferet szes: small data sets (S, 100MB), medum data sets (M, 300 MB) ad large data sets (L, 500MB). We take dfferet data sets as the sample data set of the parallel SVM algorthm, ad compare wth the o dstrbuted SVM algorthm. Expermetal results are show fgure 2 ad fgure 3.

5 329 Fgure 2: Data dstrbuto wth dfferet base colums. As ca be see from Fgure 2, the amout of data processed by the two SVM algorthms a same ode dmeso s dfferet. The amout of data processed by parallel (dstrbuted) SVM algorthm s less tha tradtoal (o dstrbuted) SVM algorthm. However, wth the gradual crease of odes, the amout of data processed by the two algorthms have gradually produced a dfferece. Although wth the crease of odes, the two algorthms ca deal wth more data. But the parallel (dstrbuted) SVM algorthm s more obvous ad more outstadg. Fgure 3: Performace evaluato of Map-Reduce executo wth dfferet base colums Ca be see from fgure 3, the tme cosumg of the computato of the two kds of SVM algorthms the same data dmeso s also dfferet. Whe dealg wth small sample data sets, the dfferece betwee the two algorthms s ot sgfcat. Ad wth the crease of the umber of sample data, the tme cosumg of operato has gradually produced the dfferece. Obvously the computg speed of the parallel (dstrbuted) SVM algorthm s faster.

6 Coclusos I ths paper, a dstrbuted cloud computg method s proposed to mprove the speed ad effcecy of bg data aalyss of ralway passeger flow data. We aalyze the problem of low effcecy of the tradtoal data aalyss method the bg data of ralway passeger flow. The, we propose a dstrbuted cloud computg method based o Hadoop, desg a large data cloud computg platform based o Hadoop, ad realze the bg data processg fucto of the parallel SVM passeger flow the cloud platform. Expermets show that the method proposed ths paper s effectve ad feasble. I order to further study the applcato of Hadoop ralway passeger flow data aalyss, ths paper gves the drecto of applcato. Refereces Alham N.K., L M.Z., Hammoud S., Lu Y., Poraj M., 2010, A Dstrbuted SVM for Image Aotato [C] Seveth Iteratoal Coferece o Fuzzy Systems ad Kowledge Dscovery, 2010: Chu C.T., Km S.K., L Y.A., 2006, Map-Reduce for Mache Learg o Multcore [C]. NIPS'06, Dea J., Ghemmawat S., 2004, MapReduce: Smpled data processg o large clusters [C]. OSDI 04: Proceedgs of the 6th Sympesum o Operatg System Desg ad Implemetato. New York: ACM Press, Doulkerds C., Nørvåg K., 2014, A survey of large-scale aalytcal query processg map-reduce [J]. VLDB J. 23 (3) Eltabakh M.Y., Ta Y., Özca F., Gemulla R., Krettek A., McPherso J., 2011, Cohadoop: flexble data placemet ad ts explotato hadoop [J], Proc. VLDB Edow, 4 (9): Egle C., Lupher A., X R., Zahara M., Frakl M.J., Sheker S., Stoca I., 2012, Shark: fast data aalyss usg coarse-graed dstrbuted memory [J]. : Proc. of the ACM SIGMOD It. Cof. o Maagemet of Data, Färber F., Cha S.K., Prmsch J., Borhövd C., Sgg S., Leher W., 2012, Sap haa database: data maagemet for moder busess applcatos [J]. SIGMOD Rec. 40 (4): Ku D., Yh L., Perera A., 2011, Parallel SMO for Trag Support Vector Maches [EB/OL] [ ]. Http: / / / Cotets / Educato /APC-SVM.pdf. Le Y.W., 2011, Cloud computg techology, platform ad applcato [M]. Bejg: Tsghua Uversty press. Sh X.L., 2011, Supply cha maagemet [M]. Bejg: Machery Idustry Press. Tag D.Y., Cao X.H., 2013, Parallel algorthm desgg ad applcato of weghted Voroo dagram usg MapReduce programmg mode [J]. Applcato Research of computers, 30 (5): Wag J., Hog C.Y., She Z., 2009, Desg of EC Webste Search Ege Based o Compass [J]. Moder computer based o, (2): Wag Y.L., Yag X.J., 2010, Full-text Retreval System Research ad Desg Base o Compass [J]. Coal techology, (6): Y C.Y., Zhag Y., Wu C.Z., 2014, Research o Page-Rak algorthm optmzato based o MapReduce [J]. Computer applcato research, 31 (2): Zhou X., Wag L.F., Jag Z.J., Zhag Y., 2009, Desg of eterprse formato portal search ege based o Lucee [J]. Mcroprocessor, (8):