A Distributed Storage Architecture based on a Hybrid Cloud Deployment Model

A Distributed Storage Architecture based on a Hybrid Cloud Deployment Model Emigdio M. Hernandez-Ramirez, Victor J. Sosa-Sosa, Ivan Lopez-Arevalo Information Technology Laboratory Center of Research and Advanced Studies of the National Polytechnic Institute (CINVESTAV) Ciudad Victoria, Mexico (52) 834 107 0220 + 1114, 834 107 0241, 834 107 0220 ehernandez,vjsosa,ilopez{@tamps.cinvestav.mx} Abstract This paper presents a practical case of study that analyzes the behavior of a distributed storage architecture, which was developed on a hybrid cloud computing environment. Open source software was used for implementing this architecture. An elastic service can be supported by virtualization technologies that allow the architecture to increment and decrement resources on demand. Performance and resource consumption were evaluated applying different replication techniques, which offer several levels of data availability and fault tolerance. The obtained results help to identify the benefits and limitations that arise when a system based on this architecture is implemented. The prototype allows us to visualize some trade-offs when applying different replication techniques depending on the availability, efficiency, fault tolerance and privacy required by users. Keywords cloud computing, scalability, virtualization, high availability. I. INTRODUCTION To define the storage requirements for institutions or companies of any size has become a problem with no trivial solutions. This is mainly due to the very fast generation of digital information which behavior is very dynamic[1]. In this context, it is common that managers of storage resources, with the responsibility to make predictions about the resources that will be needed in the medium term, often face the following scenarios: a) predictions are below of real needs, in this case, there will be a problem of resources deficit; b) generation of an excessive expenditure on the purchase of storage resources, producing a complex administration and probably with resources that will not ever be used in the medium term. This situation makes it attractive the acquisition of storage services that implement an elastic concept, i.e., those having the ability to grow and being reduced on demand, wherein the costs of acquisition and management are relatively low. Nowadays, this service model is called cloud computing. In this model, storage resources are provisioned on demand and are paid according to consumption. Services deployment in a cloud computing environment can be implemented in basically three ways: private, public or hybrid. In the private option, resources belong to a company; this implies an initial strong investment for the company, because it is necessary to purchase a big amount of storage resources and assume the administration costs. In the public version, resources belonging to a third party, where costs are a function of the resources used. These costs include administration. Finally, hybrid version contains a mixture of both. The cloud computing model is mainly supported by the development of technologies such as virtualization and service-oriented architecture. Distributed storage services over a cloud environment provide omnipresence and make it easier their deployment. This means that users can access their files from anywhere, while there exists an Internet connection and without requiring the installation of a special application (only it is needed a web browser). Data availability, scalability, elastic service and pay only for consumption are very attractive characteristics found in the cloud service model. Virtualization is playing a very important role in the cloud computing. With this technology, it is possible to have facilities such as multiple execution environments, sandboxing, server consolidation, use of multiple operating systems, ease of software migration, among others. Besides virtualization technologies, emergent tools for creating cloud computing environments that provide dynamic instantiation and release of virtual machines and software migration are also supporting the elastic service offered in this kind of computing model. Although there are currently several proposals for cloud storage, such as Amazon S3 [10], RackSpace[3] or Google Storage[4], which provide high availability, fault tolerance and services and administration at low cost, there still are companies that do not feel confident to store their information in a third-party-owned environment. In these cases, such companies that would like to take advantages of cloud computing, would require to implement a private cloud solution. Unfortunately, this option often is beyond the scope of their budgets. This dilemma makes it attractive to think about a hybrid solution, in which companies or users in general can store some information using a private infrastructure, e.g. sensitive or most frequently used data, and store the rest of the information in a public cloud. To evaluate different alternatives of implementation that might have this kind of companies was the main motivation of

the architecture presented in this paper. With the development and evaluation of a prototype of a storage service implemented on a hybrid cloud environment based on free software, we wanted to analyze the behavior of a service like this, taking mainly into account the low cost of the system implementation, the system efficiency, resource consumption and several levels of data privacy and availability by using different techniques of data replication. The following list summarizes the contributions of this paper: 1) Proposal of a distributed storage architecture implemented on a hybrid cloud computing environment based on free distribution software. 2) Results of evaluations made at different settings applied to the infrastructure, offering several levels of data availability, fault tolerance and privacy, by means of implementing different replication mechanisms. 3) Proposal for an innovative replication mechanisms based on the Information Dispersion Algorithm [2], which was adapted to a hybrid cloud computing model. 4) A Prototype of a web distributed storage system that is supported by the architecture presented in this paper. This system was called DISOC (Distributed Storage on the Cloud) and represents our proof of concept. The rest of the paper is organized as follows; section II includes related work. Section III describes the components that form the distributed storage architecture, which is supported by virtualization technologies. Section IV presents an evaluation of the tests and results obtained from the DISOC prototype and section V offers some final comments and conclusions. II. RELATED WORK Nowadays Amazon S3 is considered a pioneer of cloud storage solutions.it offers to its users different rates for storage, according to the amount of the stored data. These rates vary depending on the data availability required by users. Data availability is related to the replication technique that will be used in the Amazon infrastructure[10]. There exist also solutions that take advantages of public cloud storage using replication techniques that were originated in RAID, for example RACS[8], which is a proxy that is located between multiple cloud storage providers and customers. It is responsible for distributing data in a way that it provides an opportunity for clients to tolerate interruptions in a public cloud storage service or when the price for using the services is getting high. It uses replication in order to support those possible situations. RACS offers to its users an interface similar to Amazon S3, allowing operations such as PUT, GET, DELETE and LIST. Another proposal is HAIL[9], a cryptographic distributed system that allows file servers to provide a secure storage environment. HAIL supports the failure of any of the servers that make up the system, adding a degree of security to stored data using an approach based on the Reed Soloomon error correction codes. Currently there are public cloud storage infrastructures such as Amazon S3[10], Rackspace[3], Google Storage[4] that are being used by distributed file systems such as Dropbox Dropbox[13], Wala[12], and ADrive[11], that allow users to store and share file. A common point in these infrastructures and applications is the use of public clouds. These services are being very useful for users wanting to have an unlimited storage space or to backup their data. However, the use of these type of solutions can be a challenging decision for a business environment. This is because some organizations have fear of storing sensitive data in a third party infrastructure or that the data could not be available at the time they were required. Our approach suggests creating a hybrid cloud storage environment (private + public), with low cost infrastructure, in which only part of the stored data are in the public environment, minimizing the likelihood of unauthorized access. III. INFRASTRUCTURE DESCRIPTION In the previous section, it was mentioned that a small and medium businesses (SMB) could face some economical and technical challenges when trying to obtain the benefits of having their own cloud computing environment (private). Our proposal is trying to help with those challenges by designing and implementing an scalable and elastic distributed storage architecture based on a free and well known open source tools. This architecture is thought for combining private and public clouds by creating a hybrid cloud environment. For this purpose, tools such as KVM[5] and Xen[6], which are useful for creating virtual machines (VM), were evaluated. For managing a cloud environment is possible to use tools such as Open Nebula[15] and Eucalyptus[16]. The hard disks (HDs) integrated into the storage infrastructure are found in commercial computers (commodities). The use of this type of HDs makes this architecrure failure-prone. This was the reason why, we evaluate different replication mechanisms, which provide several levels of data availability and fault tolerance. Figure 1 shows the core components(a) included in the storage architecture (private cloud) and the distributed storage web system (DISOC) that is used as a proof of concept(b). It can be seen that the private cloud has an interface to a public cloud allowing a hybrid environment. The core components of the architecture are the following: Virtual Machine (VM): In our current private cloud implementation, every core in a physical machine is ideally thought for running only one virtual machine. This situation can be changed depending on the level of workload. The open source tools KVM[5] and Xen[6] were evaluated to decide which one could offer a better performance in terms of virtual machine instantiation. Results of those tests are not included in this paper due to space limitation. KVM showed a slightly higher performance than XEN, reason why we chose KVM, similar results can be found at [17]. Each VM has a Linux operating system optimized to work in virtual environments, which requires a minimum consumption of disk space. The VM also includes an Apache web server, PHP and some basic tools that were used to build

Fig. 1. Core components in a private/public cloud storage infrastructure. the DISOC prototype. Every VM is able to access to a pool of disks through the DAM module. Virtual Machine Manager Module (VMMM). It has the function of dynamic instantiation and de-instantiation of virtual machine depending on the current load on the infrastructure. We evaluated two open source tools for managing virtual machines, Open Nebula(ONE)[15] and Eucalyptus[16]. We chose ONE, because it offers more simplicity in the installation/configuration and has more support and documentation available online. Data Access Module (DAM). In order to improve the speed of deployment of VMs and the storage service scalability, it was allocated a minimal physical disk space in every virtual machine (VM). The real disk space used by every VM was given by a Data Access Module Interface (DAM-I), which allows VMs to get access to disk space by means of a Data Access Module (DAM). The main function of DAM is to provide transparent access to the different disks that are part of the storage infrastructure. It allocates and retrieves individual files stored on different file servers. In this context, each VM has the notion of being interacting with a single disk. DAM is implemented over NFS and includes a file allocation algorithm that locates the whole file, or part of it, using a Round Robin policy that follows a sequential identification mechanism. This allocation scheme allows DAM to find the location of a file using a minimum of additional information (metadata). Since DAM is configurable, it is possible to evaluate the performance of the storage service according to several levels of availability, applying different replication techniques. Load Balancer Module (LBM ). It is designed to distribute the load among different VMs instantiated on the physical servers that make up the private cloud. The LBM is configurable, so it is possible to define different balancing policies. The results presented in this paper consider a Round-Robin policy. LBM is the main gateway for the storage service. The NGinx web server[14] was adapted to become LBM, because this server can work as load balancer and has a low consumption of resources, essential point in a virtual environment with limited resources. Load Manager (LM). Basically, this module is responsible for monitoring the load that can occur in the private cloud. In general, it keeps track of the average response time per request in each VM. Exceeding a threshold (configurable), the manager informs the VMMM in order for it to deploy a new VM into the private cloud. LBM is also informed of a new VM that has to be considered in the load distribution. Likewise, when a low load threshold is reached, the VMMM will shut down a VM and the LBM will not consider it in the load balancing process in future requests. Distributed Storage On the Cloud (DISOC). It is a webbased file storage system that is used as a proof of concept of our architecture. A. Replication mechanisms High availability is one of the important features offered in a storage service deployed in the cloud. To accomplish this, the use of replication techniques is very common. DAM is the component that is configured to provide different levels of data availability. It currently includes the following replication policies: no replication, full replication, mirroring and IDAbased replication. No Replication. This replication policy represents the lowest level of data availability. With this scheme, only the original version of a file is stored in our disk pool, following a Round Robin allocation policy, depending on disk availability. This allocation method prevents files from being restricted to a single server, providing a minimal fault tolerance. Figure 2 (a) illustrates this allocation scheme using a pool of disks (D 0...D n ). Mirroring. This replication technique is a simple way to ensure higher availability, without high resource consumption. In this replication, every time a file is stored in a disk, DAM creates a copy and places it on a different disk. As shown in Figure 2(b), the distribution of files follows also a Round Robin policy, adding the copy of

Physical machines Cores Memory Hard disk Network 1 pc 4 4 Gb 640 Gb Ethernet 10/100 4 pc 2 2 Gb 250 Gb Ethernet 10/100 Virtual machines 5 1 1 Gb 1 Gb Virtual 1 1 128 Mb 1 Gb Virtual TABLE I CHARACTERISTICS OF THE PHYSICAL PCS AND VMS USED IN THE PRIVATE CLOUD Fig. 2. Replication mechanisms the file in the next available disk. The total number of bytes stored is F 2, where F is the size of the original file. Total replication. Represents the highest data availability approach. In this technique, a copy of the file is stored in the total file servers available. It is also the strategy that requires the highest consumption of resources. The total sum of bytes stored is F n, where n is the total number of file servers. As it is shown in Figure 2(c). IDA-based replication. In order to provide better data availability, with less impact on the consumption of resources, an alternative approach based on information dispersal techniques can be used. The Information Dispersal Algorithm (IDA)[2] is an example of this strategy. When it is required to store a file using IDA, a file of size F is partitioned into n fragments of size F /m, where m < n. These fragments are distributed in n different disks. IDA only needs to obtain m fragments to reconstruct the original file. Under this scheme, even if n m disks failed, the file would still be recovered, that is why it is desirable that no more than n m file servers fail. IDA provides better fault tolerance than mirroring without needing to totally replicate the original file. In this prototype was evaluated IDA with n = 5 and m = 3 (this means a 60% of the original file is replicated). IDA seems attractive for being used in a hybrid cloud environment, since it is not necessary to save the entire file on a single file server (disk), so it could be possible to send k fragments of the file (where k < m) to a public cloud storage without revealing the content of the original file. As shown in Figure 2(d,e). IV. RESULTS The evaluation scenario used to test our prototype of a infrastructure was built basically using 5 commercial PCs (commodities), which characteristics are shown in first section of table 1. This private cloud is able to be connected to a public cloud, allowing a hybrid cloud environment. The features of the VMs (for this test, there were only 5 VMs, each using one core) that were instantiated on the mentioned PCs are shown in section 2 of table I. In this evaluation, the access to a public storage cloud was emulated by connecting our private storage cloud with an external disk, located at a different network through a public internet connection. For the sake of simplicity (and keeping full control of the test) in this evaluation was not used a connection to the Amazon S3 public storage cloud. Results obtained from this prototype are intended for evaluating: a) the impact of having an elastic service and, b) the behavior of the system when requiring several levels of data availability, applying different replication techniques. A. The impact of having an elastic service As a first step, it was evaluated the impact of having elasticity in the storage service versus a static service (without elasticity). In the elastic service, a new virtual machine is instantiated when a workload exceeds a defined threshold. The evaluation uses different workloads generated by Autobench[7]. The evaluation of the static service was useful for defining a benchmark that allows us to recognize the benefits obtained by an elastic service. In this context, it was compared the behavior of a single physical machine with a hard disk receiving an increasing workload versus applying the same workload on a set of virtual machines that were incrementally instantiated. For this test, the workload basically consists of a set of requests of a dynamically generated PHP web page. This web page emulates a processing time on the server by running a sorting algorithm (bubble type). Trying to emulate different levels of load on the server, it was defined a list containing different quantities of elements that had to be sorted. The results shown in Figure 3 represent the average response time a customer received when the load balancer only accessed to one physical machine (red line), and when the balancer accessed the same physical machine with 1 to 3 (blue line). It can be seen, at the beginning of the test, when the workload is low, how the response time offered by the static service (running only on one physical machine) is better, in some cases up to 4 or 5 orders of magnitude, compared to that obtained in the execution of the service

accessing to one virtual machine. In this test, a maximun response time of 30s was defined as the upper treshold for a new VM instantiation. It means that when the global system response time reaches 30 seconds, a new virtual machine will be instantiated and integrated into the storage service. It can be seen that the response time in the elastic service has some considerable falls during the test. This behavior is not occurring at the time of a new VM instantiation, but at time when the VM is included in the service by the load balancer. The instantiation and activation time of the new VM is between 60 and 90 seconds. At the time the workload increases, it will be necessary to instantiate another VM. The elastic service was always able to finish the workload offering an acceptable response time, while the static service collapsed and could not meet the total requests. Likewise, when this descending activity will be monitored until get running only one VM on all the infrastructure. Fig. 3. Performance comparison between a fixed and elastic storage service B. Data availability, evaluation of different replication policies DAM component allows us to define the level of data availability required in the storage service. This can be done by applying different replication techniques. In this test, it was defined a benchmark that shows the benefits obtained of using a distributed storage system versus a centralized version. For this evaluation, DAM was configurated for having access to a single disk. This test ran the storage system into one VM (emulating a centralized processing) with a single storage server (emulating centralized storage). The rest of the tests were always considered using a distributed processing (5 VMs) and distributed storage (5 disks that were distributed on different storage servers encapsulated by DAM). Since the replication with IDA policy is attractive to a hybrid cloud service, we compared its behavior in cases when it is only used on a private cloud and when it is also considering the use of a public cloud (hybrid model). Two main metrics were taken for these experiments: 1) response time: it considers the time from when the user clicks on the button to upload or download a file, until the point when the file loading or downloading has finished, in this test, until the TCP connection is closed down. 2) service time: the time needed by DAM for locating a file (or part of it) and that the file is ready to be read by the system component that is requesting it. The response time obtained for users during the uploading process is very similar independently of the replication technique that was used, except for the hybrid version of IDA. It can be seen that IDA was very affected when it involved the access to the external infrastructure (public cloud). The impact on hybrid IDA is given because some file fragments have to to be sent to/retrieve from the external infrastructure through a public internet connection. The main benefit of storing some file fragments in the external infrastructure is the fact of having more storage space available in the private cloud. It is important to remember that the number of fragments that are sent to the public infrastructure will never be greater than or equal to m, where m is the number of pieces required to build the original file. As it is shown in Figure 4, the response time for downloading file using the hybrid version of IDA is also the most affected. Response times in the downloading process have similar behavior. For testing the behavior of this version of IDA, DAM was configurated to always obtain a fragment of a file from the the public cloud. It should be noted that this is not the typical case, because in a real scenario, the hybrid version of IDA only would obtain a fragment of a file from the public cloud in the cases when it was not able for DAM to obtain the m needed fragments from the private cloud, which means that more than n-m disks had failed (worst case). Service times observed in Figure 4 suggest that again the higher consumption of time is due to the use of a public provider. It can be seen that the service time generated mainly by DAM is minimal compared to the total response time, independently of the replication technique that is used. The only exception of this is the hybrid version of IDA, which is being forced to get access to the public cloud. V. CONCLUSION This paper described the design, implementation and validation of a distributed storage architecture that takes into account a hybrid cloud model. It was introduced DAM, a simple mechanism for storage consolidation on a hybrid cloud environment, which is able to offer different levels of data availability based on users requirements. DAM uses a lightweight algorithm for file allocation, reducing the amount of metadata needed with a low resources consumption. Another point discussed was the real performance improvement obtained when using an elastic (virtualized) environment, instead of a physical environment. This will be true especially when the system is prone to receive big workloads. Finally, it is shown how the hybrid version of the IDA algorithm can be a viable solution for those SMB that want to obtain the benefits of cloud storage without exposing the content of all of their files in a third-party infrastructure. Acknowledgments. This research was partially funded by project number 173455 from Fondo Mixto Conacyt-Gobierno del Estado de Tamaulipas.

Fig. 4. Average response time and service time for file uploading(fu) and downloading(fd) using different replication techniques REFERENCES [1] John F. Gantz et al, The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010, An IDC White Paper - sponsored by EMC [2] Michael O. Rabin, Efficient dispersal of information for security, load balancing, and fault tolerance, J. ACM 36, 2 (April 1989), 335-348. [3] Rackspace Cloud Files, http://www.rackspace.com/cloud/cloudhostingproducts/files, August 2011 [4] Google Storage for Developers http://code.google.com/apis/storage, August 2011 [5] Kernel Based Virtual Machine, http://www.linux-kvm.org, August 2011 [6] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles (SOSP 03) [7] Autobench, http://www.xenoclast.org/autobench, August 2011 [8] Abu-Libdeh, H et al, RACS: a case for cloud storage diversity, Proceedings of the 1st ACM Symposium on Cloud Computing, June 2010 [9] Bowers K. D. et al, HAIL: a high-availability and integrity layer for cloud storage, Proceedings of the 16th ACM Conference on Computer and Communications Security, November 2009. [10] Amazon Simple Storage Service(S3), http://aws.amazon.com/s3, August 2011 [11] ADrive, Web storage, http://www.adrive.com,august 2011 [12] Wala, Secure online storage http://www.wuala.com, August 2011 [13] Dropbox http://www.dropbox.com/features, August 2011 [14] NGinx web server, http://wiki.nginx.org, August 2011 [15] OpenNebula home page, http://opennebula.org, August 2011 [16] Eucalyptus home page, http://www.eucalyptus.com, August 2011 [17] Comparative of xen and kvm available in: http://virt.kernelnewbies.org/xenvskvm, August 2011