A Novel Scalable Architecture of Cloud Storage System for Small Files based on P2P

Transcription

1 2012 IEEE Internatonal Conference on Cluster Computng Workshops A Novel Scalable Archtecture of Cloud Storage System for Small Fles based on P2P ZHANG Q-fe, PAN Xue-zeng, SHEN Yan College of Computer Scence and Technology, Zhejang Unversty, Hangzhou , Chna; zhangfeezju@gmal.com LI Wen-juan, College of Qanjang, Hangzhou Normal Unversty Hangzhou, , Chna lelle@163.com Abstract Scalablty and Latency are the two mportant performance ndcators for the dstrbuted fle system, and Google and Apache have acheved a great success wth GFS and HDFS when operatng bg fles, but the latency s too long when readng and wrtng small-sze fles, because the concurrent I/O can t work for small fles, besdes the master node s dffcult to extend n the cloud storage system wth Master/Slave structure. In ths paper, we propose a dstrbuted cloud storage system based on P2P, where a central route node s ntroduced to mprove the resource query effcency, so clents can fnd data usng only one message compared wth Chord s log(n). The central routng node only stores the status and routng nformaton of all data nodes, whch are ndexed by the Tre Tree structure, so query tme meets the requrement of onlne query. The data nodes store fle s content and fle s metadata thus the system s easy to extend because the master node no longer needs to store the metadata. Clents can also cache the routng nformaton, so the read and wrte tme s reduced accordng to the Localty Prncple. Experments show that the readng and wrtng tme s sgnfcantly reduced compared wth Hadoop HDFS. Keywords-P2P; the Small Fle; DHT; Chord Routng Algorthm; Cloud Storage System; Tre Tree I INTRODUCTION Wth the development of computatonal scence, the large-scale parallel computers and large-scale parallel storage system are also renewed to meet the computng and storage requrements of varous busness systems. The development of parallel storage system s ncreasngly specalzed to take as much utlzaton of underlyng hardware as possble. Currently, the parallel storage system wth a Master/Slaver structure has great advantage n readng and wrtng large-sze fles and the performance advantage s partcularly evdent when dealng wth G-level even T-level fles, such as Google s GFS(Google Fle System) [11] and Apache s Hadoop HDFS (Hadoop Dstrbuted Fle System) [13]. But the concurrent I/O can t work for small-sze fles. Accordng to a report from the US Natonal Energy Research Scentfc Computng Center n 2007, 99 percent fles are smaller than 64MB and 43 percent fles are smaller than 64 KB [1] n a fle system wth fles. Researchers of US Pacfc Northwest Natonal Laboratory have also come to the smlar concluson that 94 percent fles are smaller than 64MB and 58 percent fles are smaller than 64 KB [2] n a fle system wth fles. The concluson s also confrmed by the data from clmate, astronomy, bology and other felds, and the average sze s 61 MB [3] of the fles from global atmospherc crculaton patterns and the average sze s 1 MB [4] of the fles from astronomcal mage and the average sze s 190 KB [5] of the fles from human genome sequence map. So researchers begn to focus on the performance mprovement for the small-sze fle n the dstrbute envronment, and Shakh [6] et al. presented an mplementaton of fle stuffng on metadata server. The problem s that stuffng ncreases space pressure of the metadata server when more and more small fles are stuffed. Carns [7] et al. mplemented the smlar stuffng method n PVFS2, and compared wth the method n [6], the method n [7] only worked when the metadata server acted as data server at the same tme, besdes, only fles that smaller than strpe sze can beneft from ther optmzaton. Hendrcks [8] et al. studed the overhead due to the decouplng of clents and servers n /12 $ IEEE DOI /ClusterW

2 dstrbuted fle system. They presented metadata pre-fetchng method to mask the network roundtrp. Snce the metadata s very small, the optmzaton can brng sgnfcant performance gans. Kuhn [9] et al. observed that not all applcatons requred complete metadata for small fle workloads. They mplemented a specal drectory hnt, no metadata to label the drectory whch contans small fles. Ths method shows sgnfcant metadata performance mprovements for small fles but the drectory restrcton lmts ts applyng scope to drectory level. Mackey [10] et al. proposed the metadata compress method, whch can mprove the space utlzaton for metadata, but the method doesn t fundamentally solve the problem and ntroduces addtonal overhead smultaneously. The pre-fetchng method can reduce the tme overhead of a round-trp, but the method wll ntroduce lots of extra tme wth the number of the small fles ncreasng. Frstly, we present the current manstream dstrbuted fle system wth Master/Slaver structure, and then we analyze ts performance bottleneck when dealng wth small fles. After that, we propose a dstrbute fle system for small fle based on P2P n ths paper, and we ntroduce the Tre data structure to store the routng nformaton, whch meets the requrements of onlne query. Clents can also cache the central routng nformaton when the number of small fles s very large. Experments show that the performance of readng and wrtng small fles s sgnfcantly mproved compared wth Hadoop HDFS. and when desgnng a fle system for small fle wth same capacty the master wll become the bottleneck. Fgure 2.1 Google s GFS fle system archtecture B. Dstrbute Storage System Based on P2P In ths paper we proposed a new dstrbute storage system archtecture for small fles wth a central routng node, whch s shown as Fg II DISTRIBUTE STORAGE SYSTEM Fgure 2.2 Dstrbuted fle system based P2P A. Dstrbute Storage System wth Master/Slaver structure Currently, the cloud storage system manly used the Master/Slaver structure, such as Google s GFS, Yahoo s HDFS and Amazon s S3 (Smple Storage Servce) [12]. The typcal structure s shown as Fg.2.1. The master server mantans drectory structure, metadata and the mappng between the fle name and block ID, and data blocks are stored on slaver servers. Clents can read and wrte the data block n parallel for large-sze fles after obtanng Slavers IP and blocks ID n step 3 and step 4. But the concurrent I/O could not work [15] for small fles, The center node mantans the routng and status nformaton of all data node, and the tem s added or updated when the data node sends a heartbeat packet to the center node. By ths way, the center node knows that the data node s onlne, the other hand, the center node wll mark one node offlne when the center node don t receve a heartbeat packet wthn a threshold tme, and then the center node wll ntate backup process (There are 3 copes for each fle). All the data node form a rng, and the system uses current manstream dstrbute hash table DHT [16] to dstrbute the fles, whch s also adopted by Amazon s Dynamo [14]. DHT enables that each node only needs to process the data falls between the current node and ts 42

3 precursor node, and the consstence of hash functon unformly spreads data along the rng. Currently, there are many routng algorthms based on DHT, such as Chord[17], CAN, Kademla[18] et al., but Chord s complexty s O(log(N)), Kademla s complexty s O(k), where N s the number of all data nodes and k s the number of the data set that s nearest to value key. Chord s completely decentralzed and symmetrc, whch s dfferent from our system. In our system we ntroduced a central routng node. The center node stores all nodes routng and status nformaton, whch s showed as Tab.1 that clents can pre-fetch the routng nformaton when the number of the routng table tem s not huge; and clent can also cache the routng nformaton when the routng data s very large. It s shown as Fg.2.2 that clents can drectly read or wrte the data on the specfc data node. But, the Center Node wll become the bottleneck of the system wth the number of small fles ncreasng. There wll be a few problems as follow: (1) The query tme complexty s O(log( n )) when the data set s orderly and the query tme complexty s O(n) when the data set s unordered, so t s mpossble to handle the hgh concurrent request. (2) Too much mantan cost. The tme complexty s up to On ( ) for nsertng and deletng nodes. The number of faled nodes s proportonal to the system scale, and the mantenance cost greatly ncreases the system burden when the system extends. Table 1 Items of central routng node Hash Value Status IP Address 12ABC23ADFKBCJKDFJAF On ABC23ADFKBCJKDFJAF Off EFAER223ADFK12JKD78A On III THE INDEX STRUCTURE OF CENTRAL ROUTING NODE BASED ON TRIE TREE In ths paper, a new ndex structure based on Tre tree s proposed to solve the problem that the query tme s too long and the system s dffcult to extend. The Tre tree that s known as word search tree or key tree s a tree structure, whch s a varant of hash tree, and t s usually used as statstcs n search engne system or sortng a large number of strngs. Tre s more effcent than Hash Table for Tre tree has less character comparson. Fgure 3.1 The shape of Tre tree Frstly, several symbols are defned n order to better descrbe the performance of our system. Defnton 1: the number of all dfferent characters n hash value s m and the character set n the hash value s denoted by { N1,... N m }, where N m and symbol denotes the length of the strng. The symbol n denotes the number of tems n hash table. Defnton 2: the length of the hash value s l and the strng of the hash value s denoted by T T[1... l], where T l. Defnton 3: the length of the search strng s l and the search strng s denoted by P P[1... l], where P l. A. TmeComplexty The Tre tree s bult wth the hash value shown n Tab.1, and the hash value has a length of 20 Bytes and status flag s one Byte, and IP address s 4 Bytes. 1) Tme complexty of constructon Tre tree There s only one node called root node at ntal n Tre tree, and t begns to traverse from root node when structurng a Tre tree n the memory, and a new node wll be added when the strng s not n Tre tree. Average chldren number of each node s m /2, and average heght 43

4 of Tre tree s l /2, so the average number to compare for a new strng s less than ( l /2)*( m /2), so we can get results as follow: (1) The query tme complexty s On ( *(log m)* l/2) when the all nodes chldren are ordered. (2) Tme complexty s On ( *( m/2)*( l/2)) when the all nodes chldren are unordered. 2) Tme complexty of queryng The substance s that usng the Tre tree can reduce the useless comparson. The search begns from root node and only m-tmes comparsons s needed at most at each layer because that there are m chldren at most to each node, so the average comparson tme s ( m/2)*( l /2) for a Tre tree wth heght of l. We can get results as follow: (1)Tme complexty s O((log m)* l/ 2) when the all nodes chldren are ordered (2)Tme complexty s O(( m/ 2)*( l / 2)) when the all nodes chldren are unordered. 3) Tme complexty of addng It s smlar to construct a Tre tree for addng a node, and t begns from the root node and searches the node havng same prefx wth pattern P. The IP address stored n the matchng node wll be renewed f the pattern P matches all the strng along the nodes of the Tre tree, otherwse we add a new node. We can get results as follow: (1)Tme complexty s Om ( * l/2) when the all nodes chldren are ordered. (2)Tme complexty s O(log( m)* l / 2) when the all nodes chldren are unordered. 4) Tme complexty of dealng faled node It s smlar to construct a Tre tree for deletng a faled node, and t begns from root node and searches the node havng same prefx wth pattern P. The node wll be deleted f we fnd a leaf node, otherwse the pattern do not exst n the Tre tree. More, the father node may merge the remanng chldren after deletng the faled node. It s shown as Fg.3.2 that s the shape after deletng node 12. The node n cloud computng envronment could not leave ntatvely unless the node faled, and the node fal s small probablty event. Comparng wth other P2P scenaros, the probablty for data nodes to qut or jon the system s very low, so t s not often to delete or add nodes. Fgure 3.2 Tre tree's shape after deletng node 9 B. Space complexty for pre-fetchng Let l represents the heght of the Tre tree, and let L represents the leaf number of layer- and let M represents the non-leaf number of layer-. Due to T m, so we can get formula (1). L L M 1 m 2 Let M represents the total number of all non-leaf nodes n the Tre tree, so we can get formula (4) accordng to formula (2) and formula (3), at last we can easly calculate that M [ n/ ml, n/2 l], so the space complexty s On ( ). l1 0 (1) M M, (2) l L n, (3) 1 L L l1 l1 M 0 m 0 2 C. Scalablty of Central Server, (4) The query tme complexty based on Tre tree s Oml ( ), whch meets the requrements of onlne query. But the memory consumpton of Tre tree exceeds the 44

5 physcal lmt of the central node when the system extends, for nstance, the number of small fles s and each leaf takes up 30 Bytes, thus the total memory taken by Tre tree s up to 3GB. So the center node wll become the bottleneck when system extends. In ths paper we propose a new scalable dstrbuted archtecture of the center server, whch s shown as Fg.3.3. The server on the top n the Fg. 3.3 s portal to clents, and the server A, B and C respectvely construct the Tre tree usng the substrngs except the characters A, B and C. By ths way, the space of Tre tree s dstrbuted to each server, and just one hop routng tme s added. language, and the software of central node sde that runs on Ubuntu wth kernel s wrtten n C wth compler gcc Fgure 4.1 Network System topology dagram A. Comparson of query tme Fgure 3.3 Topology after the system extended D. Fault tolerance of central node The system uses the double hot backup method, whch could ensure system sustanable work when the center node faled. The backup machne wll take over the faled one mmedately. E. Cache mechansm of clent s routng nformaton Clents often read or wrte the same data many tmes n a perod of tme accordng to the Localty Prncple, so clents can cache the routng nformaton after obtanng the routng nformaton from the central node. IV EXPERIMENTS In experments, there are 10 data nodes and a center node, whch s shown as Fg The Swtch s D-Lnk s 10/100/1000 adaptve swtch. The data node runs on Wndows XP wth Intel s dual-core CPU 2.93 and 2GB memory, and the IDE s.net Framework 3.5 and Vsual Studo The software of node sde s wrtten n C# It s shown as Tab. 4.1 that the query tme of array structure proportonally ncreases wth the number of small fles, but the query tme of Tre structure ncreases lttle when the number of small fles ncrease from one hundred thousand to one hundred mllon, the lttle ncreased tme s manly because more ntermedate nodes are needed to compare. But the worst case s m-tmes comparsons to each layer, and the tme complexty s Oml ( ), whch s ndependent on n and s a constant. The query tme above doesn t nclude round-trp tme n the network, but t s only the tme of searchng Tre tree. Table.4.1 Query Tme of both algorthms The number of small fles Query tme of Tre Ms B. Comparson of memory consumpton Query tme of array Ms It s shown as Tab.4.2 that the memory consumpton of two data structures ncreases wth the number of small fles, and the memory consumpton of Tre structure s 1.6 tmes more than Array structures. The man reason s that the Tre tree needs addtonal ntermedate nodes and addtonal ponters to mantan the tree. 45

6 Table.4.2 Memory Consumpton of both algorthms The number of small fles Memory consumpton of Tre Bytes Memory consumpton of Array Bytes C. Comparson wth Hadoop HDFS Hadoop s the most wdely used open source dstrbuted fle system, whch s smlar wth Google s GFS. We compared the performance between ours system and Hadoop HDFS. The relatonshp chart between the fle sze and tme consumpton s shown as Fg.4.2 and Fg.4.3, and the tme overhead of readng small fles wth Hadoop s far more than our system. It s because that Hadoop s desgned for large-sze fle n parallel, so Hadoop doesn t take ts advantage when readng and wrtng small-sze fle, nstead, more system overhead s added for parallel desgn. Fgure. 4.2 Tme cost for readng fles of 1K to 100 K comparng wth Hadoop Fguer 4.3 Tme cost for readng fles of 10 K to 1000 K comparng wth Hadoop V. CONCLUSION Wth the rapd development of computatonal scence, all knds of data ncludng growng proporton small fles are generated, and t s too costly for storng the small fles on the GFS or HDFS. In ths paper we proposes a new dstrbuted storage system for small fles based on P2P after analyzng the dstrbuted storage system wth Master/Slaver structure. We ntroduce a central routng node to mprove the effcency of resource dscovery, so clents can fnd data usng only one message compared wth Chord s log(n). The central routng node stored the status and routng nformaton of all the nodes, whch s ndexed by the Tre, and query tme meets the requrements of onlne query. The clents can pre-fetch the routng nformaton when the data s not very large, and the clents can cache the nformaton when the number of small fles s very large. The experments show that the performance of our system s sgnfcantly mproved compared wth Hadoop HDFS when readng and wrtng small fles. The system mplemented the basc functon of storage system for small fles based on DHT, and the system reaches the requrement of onlne query. But the central node s the bottleneck when the system extends, so our next-stage works nclude the followng contents: (a)the node n cloud computng envronment could not leave untl the node faled or the node was damaged by natural dsasters, whch s small probablty event. Our next-stage work s to study the relatonshp between the system performance and the probablty the node faled. (b)implement the scalablty based on the Tre structure. 46

7 REFERENCE [1] Patascale Data Storage Insttute, NERSC fle system statstcs. [2007]. [2] Felx. E., Envronmental Molecular Scences Laboratory: Statc Survey of Fle System Statstcs. [2007]. [3] Chervenak A., Schopf J. M., Pearlman L., et al. Montorng the Earth System Grd wth MDS4. e-scence and Grd Computng New York: IEEE, 2006: [4] Nelsen E. H., The Sloan Dgtal Sky Survey Data Archve Server. Computng n Scence Engneerng, 2008, 10(1): [5] Bonfeld J. K., Staden R., ZTR: a new format for DNA sequence trace data. Bonformatcs, 2002, 18(1):3-10. [6] Faraz Shakh, Mkhl Chanan, A case for small fle packng n parallel vrtual fle system (pvfs2), In Advanced and Dstrbuted Operatng Systems Fall 07, [7] Carns P., Lang S., Ross R., Vlayannur M., et al. Small-fle access n parallel fle systems Internatonal Parallel and Dstrbuted Processng Symposum, New York: IEEE Computer Socety, 2009:1-11. [14] Decanda G., Hastorun D., Jampan M., et al. Dynamo: amazon's hghly avalable key-value store. Specal Interest Group on Operatng System New York: ACM : [15] L xu-qao, Dong bn, Xao L-mn, et al. Small Fles Problem n Parallel Fle System. Network Computng and Informaton Securty. New York: IEEE Computer Socety. 2011: [16] Karger D., Lehman E., Leghton T., et al. Consstent hashng and random trees: dstrbuted cachng protocols for relevng hot spots on the World Wde Web. Symposum on Theory of Computng 1997, New York: ACM, 1997: [17] Stoca I., et al. Chord: A scalable peer-to-peer lookup servce for nternet applcatons. n SIGCOMM ' New York, NY, USA: ACM. [18] Maymounkov P., D. Mazères, Kademla: A Peer-to-Peer Informaton System Based on the XOR Metrc, n Lecture Notes n Computer Scence, P. Druschel, F. Kaashoek and A. Rowstron, P. Druschel, F. Kaashoek and A. Rowstron^Edtors. 2002, Sprnger Berln/Hedelberg. p [19] Shvachko, K., et al. The Hadoop Dstrbuted Fle System. n Mass Storage Systems and Technologes (MSST), 2010 IEEE 26th Symposum on [8] Hendrcks J., Sambasvan R. R., Snnamohdeen S.,et al. Improvng small fle performance n object-based storage. [2006]. [9] Kuhn M., Kunkel J. M., Ludwg T., Dynamc fle system semantcs to enable metadata optmzatons n PVF. Concurrency and Computaton: Practce and Experence, 2009, 21(14): [10] Mackey G., Sehrsh S., Jun W., Improvng metadata management for small fles n HDFS. Cluster Computng and Workshops New York: IEEE Computer Socety, 2009:1-4. [11] Ghemawat S., Goboff H., Leung S., The Google fle system. Symposum on Operatng Systems Prncples New York: ACM.2003: [12] Amazon. Amazon Smple Storage Servce [2011]. [13] Shvachko K., Harong Kuang, Rada, S., Chansler R., "The Hadoop Dstrbuted Fle System," Mass Storage Systems and Technologes (MSST), 2010 IEEE 26th Symposum on, vol., no., pp.1-10, 3-7 May