Efficient big data processing strategy based on Hadoop for electronic commerce logistics

Transcription

1 Absrac Efficien big daa processing sraegy based on Hadoop for elecronic commerce logisics Jiaojin Ci Insiue of Economy and Managemen, Nanyang Normal Universiy,Nanyang, , China Corresponding auhor s cijiaojin@163.com Received 01 Ocober 2013, Wih he rapid developmen of cloud compuing, more and more elecronic commerce applicaions are confroned wih he problems of processing big daa, such as big daa from he social media posed by he cusomers of elecronic commerce logisics. In order o improve he big daa processing efficiency in elecronic commerce logisics, an efficien big daa processing sraegy based on Hadoop is designed, which is named ECLHadoop. In ECLHadoop, hose closely relaed daa blocks are placed a he same nodes, which can help o reduce he MapReduce I / O cos, especially he I / O cos a he shuffling sage. The simulaion experimen resuls show ha, based on Hadoop, he ECLHadoop can improve he big daa compuing efficiency for daa-inensive analysis in he elecronic commerce logisics service. Keywords: big daa, daa placemen, big daa analysis, big daa compuing sraegy, elecronic commerce logisics 1 Inroducion Wih he developmen of Inerne echnology and e- commerce, especially in he developmen of cloud compuing and physical neworking echnology, humans, sensors, C, servers, mobile phones and oher equipmen o produce more and more daa in daily producion and life, how o deal wih hese large daa is facing a huge challenge. Currenly, here have been numerous sudies abroad designed some large daa processing framework, mainly large daa processing framework including Google's Google MapReduce [1] and he Apache Hadoop Hadoop MapReduce [2], hese wo frameworks have heir own disribued File Sysem, Respecively GFS [3] and HDFS [4]. Hadoop daa placemen major consideraion in order o balance he disribuion of daa, bu i does no ake ino accoun daa placemen policy relaionship beween he various daa ses. All daa sored in HDFS according o workload requiremens Hadoop cluser o place, so when execuing MapReduce compuing, large amouns of daa will be migraed, and hus increase he number of I / O cos, especially in Shuffling phase I / O cos. Since hen, he Hadoop big daa placemen sraegies on he basis of many academics, research insiuions and universiies o design a number of large daa rocessing frameworks, such as Hadoop ++ [5], CoHadoop [6] and HadoopDB [7]. However, hese large daa processing frameworks need large daa iself larger changes. Hadoop ++ needs Saic daa indexing and join Trojan Trojan connecion, when he new need for hese large daa again by some saic daa odificaions. CoHadoop is based on IBM's proposed Hadoop new opimizaion mehod, which divided according o he applicaion requiremens wih daa blocks. Large daa before being submied o he HDFS, Co-Hadoop applicaion requiremens need o aribue hem ino big daa dividing line; due o he large daa before being sored in HDFS mus ener Line changes, hus increasing he number of reamen coss. Hadoop ++ and CoHadoop carried ou o some exen on he basis of Hadoop The degree of modificaion, and HadoopDB conduced on he basis of Hadoop Greaer changes. While he exising framework of he large daa processing o some exen, raise High efficiency of large daa processing, bu hese processing framework more or Less need for big daa can be modified. This changes some special Useful for a given applicaion, bu will resul in a variey of applicaions of he process frame loss Versailiy. Therefore, in order o improve he processing efficiency of large daa, RESEARCH A new sudy, efficien large daa processing sraegy has imporan meaning Jusice, his sraegy should be based on Hadoop for daa-inensive Applicaions, and does no require any changes Hadoop iself. Given hese requiremens, we design a kind of Hadoop-based Foundaion for e-commerce logisics services such daa-inensive applicaions Effecive large daa processing sraegy ECLHadoop (Efficien Elecronic Commerce Logisics Big Daa rocessing Sraegy in Hadoop for Daa-inensive Analysis). 2. Research saus overview The large daa is divided ino n daa blocks, each corresponding o he Map-asking daa block. This phase belongs Shuffle reamen, will increase he number of I / O cos, especially large number of relaed daa blocks are placed in he daa node. Big Daa in Hadoop ++ processing sraegy as shown in Figure 1 In large daa sources from a daa block in he submission o he Map Reduce Before calculaing, mus be pre-processed. Hadoop ++ will Adding block indexing and Trojan Trojan connecion informaion. When Large daa is divided ino daa blocks, each daa block is added o he previous A Trojan index. Wih he addiion of an index and Trojan Trojan Connecion, so Hadoop ++ sage or in he Map Reduce Order Segmen, all calculaions especially Join 140 Operaion Research and Decision Making

2 query evaluaion will be well Improvemen. However, adding he index informaion and Trojan Trojan Union Access he informaion you need o spend some ime, and when he daa se changes, Need o re-add he index informaion and Trojan Trojan joins leer Ineres, while he associaed daa se ha will no be placed ogeher, So Hadoop ++ in Shuffle is sill a big sage Boleneck. Sorage node Daa block[1] Map[1] Big daa se [1] Map[2] Daa block[k] Map[k] Big daa Spli big daa Daa reques Big daa se [2] Daa block[] Daa block[n] Map[] Daa block[n] Map[x] FIGURE 1 CoHadoop big daa processing sraegy Firsly, according o Co-Hadoop Big Daa applicaion requess will be divided, hen he associaed daa block is assigned o he same daa sorage node. Thus, he query is calculaed especially relaed queries will be addressed in he same compue node. Map Reduce compuaion only needs Map Reduce compuing wihou he need o calculae, so you can cancel Shuffle sage, coss are grealy reduced. In order o improve processing efficiency, and many oher opimizaion mehods or sraegies have also emerged. For example, he Twiser and HaLoop [10] have proposed an ieraive calculaion for large daa problem, UC Berkeley also made Spark [11] o handle large daa. In addiion, here are some oher big daa compuing sraegies such as Hadoop-ML [12] and so on. The sandard GA generaes iniial populaion randomly. This mehod ends o produce a narrow search space, which is disadvanageous o he acquisiion of he global opimal soluion. To improve he convergence characerisics of GA, he iniial populaion is generaed using uniform pariion based generaion mehod. This mehod evenly divides he range of opimized parameers ino regions wih heir number equal o populaion scale Gs. In every small region, an individual will be generaed randomly. This mehod can quicken he convergence speed and increase he possibiliy of converging o global opimal soluion. Based on he above analysis, he crossover probabiliy generaion will be defined as: c where: ( )( f f ) f f emp b avg b avg f favg f f b avg c a -h (1) emp e e e, (2) where and are he imum crossover probabiliy and he imum one, be he imum ieraion imes, be he bigger finess of wo individuals chosen for crossover operaion a -h generaion, and denoe he imum finess and average finess of he populaion a -h generaion. Similarly, he muaion probabiliy a -h generaion will be defined as: f b m where: T l f ( )( f f ) f f emp k avg k avg f favg f f k avg e emp e e l (1 T ) (1 ) (1 ) f avg m where and are he imum muaion probabiliy and he imum one, be finess of he individual chosen for muaion operaion a -h generaion. f k 3 E-commerce logisics of large daa processing sraegy ECLHadoop Through he above analysis we can see ha he exising large daa processing sraegies are some problems, especially for specific applicaions, are presen boleneck resricing is compuer efficiency. Based on his, we propose a large e-commerce logisics daa processing sraegy-ecl Hadoop, which is shown in Figure 2. (3) (4) 141 Operaion Research and Decision Making

3 SNS BIG DATA IN MODULE ARRANGE ADD TIME SIGNAL ADD A SIGNAL ANALYSIS ON ROUTE FOR BIG DATA IN STORAGE OF BIG DATA IN FIGURE 2 Elecronic commerce logisics big daa processing sraegy- ECLHadoop Daa from a large e-commerce logisics commerce logisics service Cusomer Service released on various social neworking sies relevan informaion. These will be based on Hadoop big daa mechanism is divided ino 64MB Daa blocks will add a ime samp o all daa blocks. In addiion, we will also add applicaions associaed ag for each block of daa; hese should be Markers associaed wih he demand from he early, hey and e-commerce associaed logisics services. Tha is, if he analysis needs o ell me hey need daa ses A and B o link daa ses, we can recognize some of he daa block wih he daa ses A and B for he daa se of some number. According o he associaion of he block, hus, adding a clearance for hese daa blocks Union flags. Afer compleion of he calculaion for all he semanic daa blocks, can be Ge big e-commerce logisics daa placemen rouing able. Through which complee e-commerce logisics of large daa placemen, hose closely relaed Daa blocks will be sored in he same node. Because in Map Reduce Shuffle avoid processing sage, i can improve he coun of efficiency, especially relaed inquiries calculaions. To beer illusrae he e-commerce logisics of large daa processing sraegy, e-commerce logisics and big daa correlaion definiion and e-commerce logisics of large daa correlaion calculaion is paricularly imporan o do a brief analysis of he following: (1) E-commerce logisics big daa correlaion. In he calculaion of e-commerce logisics in big daa, Those collecions wih compuing dependen daa ses is called he relaed daa ses. As in E-commerce logisics applicaions, assume here is a very frequen meer Operaors relaionship - calculaed by "Air China" in logisics and ranspor of fas Delivery raffic. As can be seen from he demand, "Order Form" and "Logisics Business Transporaion Company able "is he relaionship beween wo compuing Join Table. Therefore, i can be seen ha he "Order Form" and "public ransporaion logisics business Division able "wo ables wih large daa dependencies are wo ables, which E-commerce logisics is big daa correlaion. (2) E-commerce logisics big daa correlaion calculaion Big daa correlaion is calculaed primarily hrough e- commerce logisics for business needs analysis and calculaion o achieve. Assug ecommerce maer The Company s big daa flow analysis needs, and hey need he disribuion able as follows: 1. Business needs 1: {Table (A), Table (C)} 2. Business needs 2: {Table (A), Table (C)} 3. Business needs 3: {Table (A), Table (C)} Which can be seen, he daa se A daa se C and daa Se D-relaed, and he daa ses and daa ses E B relaed. Map Reduce mainly includes hree aspecs of compuing, namely, Map phase calculaion, Reduce phase calculaion and Shuffle sage Calculaions. If he daa ses associaed wih A and C in accordance wih Hadoop Their random disribuion of he disribuion mechanism, he daa se A And C will be dispersed in he daa cener a number of daa nodes. Whole A Map Reduce compuaion process includes he following imporan aspecs: Firs, Map sages needed by hundreds of housands of daa node coun Calculaion o obain inermediae resuls, and hese resuls need o be migraed o he middle Reduce daa nodes specified final calculaions. Daa (inermediae sages housands meer Map daa nodes in he process calculaion resuls) require daa migraion (Shuffle process), so big Large increases he I / O cos of Map Reduce Shuffle sage Makes he I / O cos of he enire phase of exremely large Map Reduce, Overall compuaional efficiency is very low. Differen from he above process, if he daa in an associaed relaionship A se of daa collecion and placed in he same node C, he daa ses A and Daa se C may be calculaed in he same daa node complees, wihou To Shuffle sage of Map Reduce, hus avoiding he Map Reduce The Shuffle boleneck. So i will grealy enhance he Map Re duce Compuaional efficiency. A he same node, here is no relaionship beween he numbers of calculaions associaed daa sorage Daa block as Hadoop as compleely dispersed ino he daa cener of many housands of compuers. However, we have adoped and Hadoop A copy of he same sraegy, he same se of hree copies, so by hree copies can effecively proec daa reliabiliy. In proecion can on reliabiliy, ECLHadoop sraegy and Hadoop daa sorage Sraegy is he same, here is no conflic. In he previous analysis, we mainly consider he relevan daa ses Can be sored in he case of he same node, bu wih he amoun of daa To he larger, here may be some associaed daa ses is difficul o sore In he case of he same daa node. To solve he above problem, ECLHadoop reamen sraegy is as follows: The daa se is sored in he same A nework bandwidh performance of he bes chassis, he neares wo compuing On he machine, if he wo compuers is no enough, i is sored in he neares hree On he compuer, and so on. In his secion, he proposed e-commerce logisics of large daa processing sraegy ECLHadoop on he basis for such a reamen sraegy Deailed case sudies. Firs, assume ha e-commerce logisics service Service-relaed social neworking sie has four daa ses, respecively: Daase A, daase B, C daa ses and daa ses D. 4 Simulaion In order o prove he correcness and raionaliy of he above 142 Operaion Research and Decision Making

4 mehods were wo relaed simulaion. Experimen 1 compared wih he ime samp added Join ECLHadoop d and no compuaional experimens join query Hadoop compuaional efficiency under he circumsances; and Experimen 2 will compare Tim lus he associaed applicaion and have labeled ECLHadoop Join joins Charles Inquiry case compuaional experimens compuaional efficiency of Hadoop. Simulaion experimens using five compuers, one of which is he meadaa node, and he oher A node ouside of he shadow, and he remaining hree for e-logisics daa Sorage nodes. Each compuer will be insalled he Linux operaing sysem and he Hadoop Disribued File Sysem HDFS. Simulaion resuls from a use of he major social neworking sies he e-commerce logisics service relaed daa. Time simulaion daa is ranging from January 1, 2013 o May 8, In Simulaion Experimen 1, simulaion experimens were carried ou in five of he added Time samp and non-join query ECLHadoop calculaed Hadoop comparing he calculaion ime of boh comparaive resuls are shown in Figure 3. Time flag(s) ECLHadoo Haddop monh FIGURE 3 Compuing ime comparison beween he ECLHadoop wih ime flag and Hadoop wihou Join query compuing From he experimenal resuls, in Hadoop, he pair Any requess, all he daa blocks have o be calculaed; however, in ECLHadoop, since he effec of he ime samp calculaion range,, grealy reduced. For example, when only need o ge he ime range from 2013 For some years beween April 1 o April 30, 2013 daa The resuls of is inquiry, you can discard his ime ouside he scope of I is he daa o reduce he amoun of compuing ime (i.e., coun MapReduce Coun he ime). However, all daa in Hadoop because even Form a block, mus have been calculaed, and herefore he compuaion ime spen significanly More. Simulaion 2 is sill used from he social neworking websie E-commerce logisics services relaed simulaion daa. This se of simulaion daa here are wo daa ses, ie, daa ses and daa ses A B. And Simulaion A similar experimen, his experimen sill conains five sub-group simulaion. The simulaion has five groups, including: (1) daases A connecion Daase B. A size of he daa se for he 1TB, he daase B, Size is 1TB. (2) A daa se joins daase B. Number According o he size of he se A is 1TB, while he size of he daa se B is 2TB, (3) A daa se joins daase B. A large daa ses in Small as 2TB, while he size of he daa se B is 1TB. (4) Daa ses A coupling daase B. A size of he daa se for 2TB, daa he size of he se B is 2TB. (5) A daa se joins daase B. A size of he daa se for 2TB, while he size of he daase B Is 3TB. 5 Conclusions This paper is designed based on Hadoop for E-commerce logisics daa-inensive analysis of large daa processing sraegy-eclhadoop. In order o improve e-commerce logisics for large daa Compuaional efficiency, ECLHadoop sraegy designed e-commerce Daa Service Logisics large daa placemen sraegy will have o calculae he relevance The daa is placed ogeher in he same daa node. Accordingly, he use of ECLHadoop policy can significanly reduce I / O cos. Wihou Sudies, we will be in big daa processing based on open source Implemen and improve he basic sraegy ECLHadoop on Hadoop Sraegy. Moreover, in a furher sudy, some more analysis based on he large daa processing sraegy ECLHadoop e maer Sreag service analysis applicaions, he counbased ECLHadoop Coun he oucome desired by he user, hereby improving cusomer e-commerce he new logisics service saisfacion. References [1] al S K, King R A 1983 On edge deecion of X-ray images using fuzzy ses IEEE Trans aern Analysis and Machine Inelligence 5(1) [2] Nakagawa Y, Rosenfeld Y 1978 A noe on he use of local and operaions in digial picure processing IEEE Trans Sysems, Man & Cyberneics 8(8) [3] Dean J, Ghemawa S 2004 MapReduce: Simplified daa processing on large clusers roc of he he 6h Symposium on Operaing Sysem Design and Implemenaion 1-13 [4] Dirich J, Quian'e-Ruiz J A, Jindal A, e al 2010 Hadoop ++:Making ayellow elephan run like a cheeah (wihou i even noicing) roceeding of he VLDB Endowmen 3(1-2) [5] Ekanayake J, Li Hui, Zhang Bing-jing, e al 2010 Twiser: A runime for ieraive MapReduce roc of he 1s Inernaional Workshop on MapReduce and Is Applicaions [6] Bu Y Y, Howe B, Balazinska M, e al 2010 HaLoop: Efficien ieraive daa processing on large clusers roceeding of VLDB Endowmen 3(1-2) [7] Ghoing A, ednaul E Hadoop-ML: An infrasrucure for he rapid implemenaion of parallel reusable analyics roc of he Large- Scale Machine Learning: arallelism and Massive Daases Workshop [8] Brown J 1994 Some moivaional issues in compuer-based insrucion Educaional Technology 26(4) 27-9 [9] Kapur J N, Sahoo K, Wong A K C 1985 A new mehod for gray level picure hresholding using he enropy of he hisogram Compuer Vision, Graphics and Image rocessing [10] Fu X N, Yin S M, Liu S Q 2003 A improved adapive fuzzy enropy enropy hresholding mehod on image segmenaion Aca hoonica Sinica [11] Gaoufi S, Oral M, Reisman A 2004 Daa envelopmen analysis lieraure: a bibliography updae ( ) Socio-Economic lanning Sciences [12] Ridler T W, Calvard S 1978 icure hresholding using an ieraive selecion mehod IEEE Trans Sysems, Man and Cyberneics Operaion Research and Decision Making

5 Auhor Jiaojin Ci, , Anqing, Anhui, China. Curren posiion, grades: lecurera Insiue of Economy and Managemen in Nanyang Normal Universiy. Universiy sudies: economics. Scienific ineres: logisic managemen and Elecronic commerce. 144 Operaion Research and Decision Making