}w!"#$%&'()+,-./012345<ya

Transcription

1 }w!"#$%&'()+,-./012345<ya MASARYKOVA UNIVERZITA FAKULTA INFORMATIKY Cloud backend deployment at Faculty of informatics MU MASTER THESIS Ondrej Faměra Brno, Spring 2013

2 Declaration I hereby declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Advisor: RNDr. Jan Kasprzak ii

3 Acknowledgement I would like to thank Yenya, for technical support and feedback, Hanna Kim for support and language corrections and my colleagues for feedback. iii

4 Abstract This work presents approaches in building cloud storage, network and management subsystems, compares performance of individual implementations and discusses best choices in this field. It also discusses implementation of cloud backend in infrastructure of Faculty of Informatics at Masaryk s university. iv

5 Keywords cloud, CEPH, GlusterFS, OpenStack, OpenNebula, fadmin, layer-bloat v

6 Contents 1 Introduction Cloud Cloud storage Distributed storage Storage layer-bloat Distributed object storage as VM storage Metadata GlusterFS Distributed, replicated and striped volumes CEPH RADOS Block Device (RBD) DRBD Other distributed file systems Moose FS SheepDog Cloud networking VM traffic Management networking DFS traffic Cloud management traffic Inter VM network separation techniques Linux bridge VLANs Tunneled protocols and tunnels Open vswitch Cloud management Layer-bloat problem OpenStack OpenNebula Performance testing Physical testing environment Virtual testing environment Storage performance testing

7 6.1 DFS capabilities requirements DRBD GlusterFS CEPH MooseFS and SheepDog CEPH and GlusterFS performance comparison Cloud management testing OpenStack OpenNebula Deployment at FI MU Existing infrastructure and front-end Current deployment state Conclusion

8 Chapter 1 Introduction The boom of virtualization in IT has never seen such a rapid growth as when it was hidden after the word cloud. This concept made a revolution in this area as stuff around virtualization get highly propagated to non-it people claiming and promising consolidation, money saving, ease of use and many other over-hyped conclusions. This work does not aim at introducing concepts of cloud from user perspective but rather shows the other side of whole cloud ecosystem the machinery behind it. Despite our determination to bring most recent information, this area is silently and rapidly evolving so we ll write about most recent state with consideration of possible future development. First, we ll provide insight into cloud storage concepts. Beginning with definition of most suitable kind of storage for cloud. Also, we ll introduce Storage layer-bloat problem which impacts performance expectations planning. Further, we ll describe distributed file systems and their suitability for storing virtual machines data. Along with these concepts, we ll also debate about redundancy, reliability and robustness of distributed storage systems. As example for our requirements on cloud storage we more deeply introduce two projects: GlusterFS and CEPH. In the end we shorty describe other possible file systems with their main shortcomings that made them out of our list of options for effective deployment. In third chapter we ll take a closer look at anatomy of network traffic in cloud environment. We will not only focus on high-level architecture of how virtual machines in cloud communicate, but also on management traffic around cloud. Recommendations for traffic separation, security and performance are also described here with the aim at distributed file systems presented in previous chapter. 3

9 1. INTRODUCTION Last theoretical chapter will introduce cloud management systems with their main problem similar to one from storage layer-bloat. After general discussion about cloud management, we ll introduce two candidates for implementation at Faculty of Informatics of Masaryk s university (FI MU). First of them is a large project called OpenStack, which is aiming at becoming cloud operating system. The second one is OpenNebula project, which is less ambitious than OpenStack but more focuses on the particular area of cloud management. Part of this work is also performance testing of selected subsystems that are crucial for cloud deployment. First, we ll present here testing equipment and testing methods description. We will take look at storage subsystem performance and compare distributed file systems that had interesting features for deployment at FI MU. Comparisons that we made here are supported by graphs with most important results to make right overview. Later, we ll discuss about suitability of tested distributed file systems for our deployment. One whole chapter we ll dedicate to discussion about cloud management systems. We ll show differences between two selected in earlier chapters. Then we decide on suitability for our deployment and describe most important advantages of them. The last chapter deals with deployment of systems discussed in our work to improve virtualization environment at FI MU. Here we gather our knowledge from beginning and transform it into practical deployment and implementation suggestions. 4

10 1. INTRODUCTION 1.1 Cloud Before we dive deeper into cloud technologies, we need to introduce what we are actually dealing with. Cloud technology provides services at all levels of IT infrastructure, from basic virtualization services to operating system services to application level services.[1] The goal of Cloud Computing is to allow users to take benefit from all of these technologies, without the need for deep knowledge about or expertise with each one of them.[17] Figure 1.1: Cloud computing logical diagram[19] 5

11 1. INTRODUCTION Most recent accepted standardized definition of Cloud Computing is one by the National Institute of Standards and Technology (NIST).[21] Cloud can be classified in many different ways, depending on which aspect we are looking at. In figure 1.1 we show for example division by cloud provided services. This view is most important for end users. It consists of: Software as a Service (SaaS) Platform as a Service (Paas) Infrastructure as a Service (Iaas) From deployments point of view, we can divide cloud into following categories: Public Cloud run and provisioned by 3rd party cloud providers, enabling organizations to outsource computer infrastructure outside to decrease total running cost of IT Private Cloud contained inside control domain of cloud owner providing better control on whats happening inside along with ability to completely customize it to specific requirements Hybrid Cloud mix of public and private cloud bringing the best of both worlds: low cost services and highly customizable own environment This work is aiming at giving more insights to cloud owners into building up private cloud providing IaaS and PaaS services. Our main goal is to find optimal solution for FI MU as a cloud provider. To build up cloud from the bottom, we should take a deeper look at: Cloud storage providing storage for whole cloud infrastructure Cloud networking enabling communication management not only inside the cloud but also managing connection to rest of the world Cloud management giving administrators and users ability to keep everything in cloud under control and provide them with high level interface to work with this environment 6

12 Chapter 2 Cloud storage Cloud storage is made up of many distributed resources, but still acts as one. It s highly fault tolerant through redundancy and distribution of data and highly durable through the creation of versioned copies.[22] Important aspects of Cloud storage are scalability it should be easy to scale-out without loosing the data availability flexibility it should be possible to use commodity hardware as well as highperformance one redundancy data should be protected against errors recovery data should be easy to recover or storage should deploy self-healing mechanism performance we should take into consideration planned usage of storage system ( distributed computations versus service hosting ) As for Cloud storage solutions, they follow two implementation paths: hardware path Big powerful storage systems providing all-in-one solution for storage, however they are expensive and they outdates quickly with new trends. software path Approach that builds upon deployment of distributed storage created by software to lower cost of specialized hardware. Usually, the commodity hardware which has lower price is used than specialized all-in-one solutions. Our work aim at providing better insight into distributed storage implemented in software as we believe it provides better performance/price ratio. 7

13 2. CLOUD STORAGE 2.1 Distributed storage Distributed storage (DS) is closely analogous to a distributed object storage. Distributed object storage (DOS) stores objects (unified data structures) among data nodes and provides access to them. These data nodes forms groups of nodes called clusters. Objects on data nodes are building blocks for more complicated higher-level structures such as distributed file systems or virtual block devices. Before we continue, we ll provide you some basic terminology from DS cluster environment: data nodes nodes that stores the data (objects) monitor nodes nodes that monitors operation of other nodes in cluster distributed file system (DFS) file system running on the top of DOS metadata extra data stored by DFS metadata nodes nodes that takes care about metadata operations in DFS object replication way of storing the same object on multiple data nodes providing redundancy of object in case of data node failure self-healing ability to react on event of data node failure, for example by allocating more objects to keep number of replicas on desired level replica redundant data, exact copy of another object used to increase redundancy and so provide fault tolerance In this work we ll focus on DS suitable to provide storage for virtual machines (VMs) in virtualization environment. As one of typical use cases of storage for VMs is a non-volatile storage that will hold users data from VM. This can be a disk, disk partition, file, loop device etc. The goal is to provide performance close to physical hardware (bare metal) Storage layer-bloat In order to provide high performance storage to VM we face problem that we cannot optimize the storage performance according to underlying hardware. Reason is that there are too many levels (layers) with different characteristics (disk block size, file system block size, DFS block size, etc. ). It s very hard to determine how we should work with such storage to get the optimal performance. 8

14 2. CLOUD STORAGE Lets take for demonstration a virtual disk (1) stored in files (2) on distributed file system (3) which is created of objects (4) lying on other conventional file system (5) placed on RAID 1 (6) on different machines (7). Previous sentence with numbers in brackets indicate layer number in which preceding component is. We need to remember that this kind of abstraction makes it not feasible to easily figure out optimal values for things like disk block size to use Distributed object storage as VM storage There are two possible approaches how to allow DOS to become storage for VMs. One of them is file system approach where we don t consider VM storage to be something special. We treat it as regular file stored on DFS. Problem of this approach is unnecessary overhead of DFS when working with the file such as metadata look-up and update. Second approach is to provide block device created on top of DOS. Here we avoid using DFS and we access DOS directly. Virtual block device maps directly to objects resulting in higher performance as we avoid use of DFS metadata as there is no DFS Metadata Similar to regular local file systems such as ext4, DFS also needs to have metadata. This is mainly due to the fact that DFS are actually file systems and they generally require metadata to properly function. We know about three general approaches to metadata realization in DFS: centralized metadata systems Metadata of whole DFS are stored on one centralized system. This system soon becomes performance bottleneck and single point of failure (SPOF). distributed metadata systems Metadata are distributed among several systems and can be access from any of them. This requires that all systems maintain consistency in distributed metadata in real time which poorly scales with increasing number of systems. algorithmically located metadata Similar to distributed metadata systems, metadata are on different systems. However metadata now share common function according to which it is possible to compute where to look for them without querying metadata systems. So clients can access metadata directly as if they were regular data stored in DFS about which they know their location. 1. Redundant array of inexpensive disks 9

15 2. CLOUD STORAGE 2.2 GlusterFS GlusterFS is an open source DFS capable of scaling to several petabytes and handling thousands of clients. It is based on a stackable purely user space design.[2] Some of interesting advantages includes: Scalability and Performance algorithmically located data and metadata Elastic Hash Algorithm function that allows client to compute where (meta)data are without need to query GlusterFS for this information High Availability internal replication and self-healing mechanisms As basic building block GlusterFS uses a brick. Brick is storage file system that is part of Gluster storage volume (Gluster volume types are discussed later in section 2.2.1). Brick is a regular directory on data node, which holds the DOS objects and is build on top of existing local file system. It s advised for brick to be on separated partition to prevent interference with other systems or user data. Essential requirement for local file system is support for user extended attributes. For locating data and metadata, GlusterFS uses only Elastic Hash Algorithm. This allows the clients to calculate themselves where data resides based just on knowledge of structure of GlusterFS volume. In case of replicated volume or undergoing re-balancing of data it might happen that data are not in calculated place. To cover situations when data are not in position yet, GlusterFS uses references which temporary refers to current location of data. Configuration of GlusterFS is created by interacting with command line interface (CLI) utilities. This is due to complexity of configuration files which gets generated through this CLI in background. From user s point of view, this is somehow better approach as user can interactively configure GlusterFS, which is simpler than manually editing configuration files Distributed, replicated and striped volumes As mentioned earlier, GlusterFS organizes storage networks into bricks which then forms storage volumes. These volumes can have some special properties which improves their parameter in way of performance, fault-tolerance and distribution of data among nodes. Basic volume types are: 10

16 2. CLOUD STORAGE distributed volume files are spread across cluster and are not accumulating on single node replicated volume files get replicated to another bricks, number of replicas can be adjusted. By default number of replicas is one which means no replication. striped volume split files to stripes and store them on different bricks, this improves performance in high concurrency environment These basic types can be further combined together to create more specific attributes. Such an example can be the distributed, striped and replicated volume. This combination brings high performance of stripped volume distributed evenly in cluster to spread the workload, and replicated volume that provides fault tolerance in case of failure of some nodes. All basic types and this example are showed in figure 2.1. Figure 2.1: Basic GlusterFS volume types and example of their combination[15] 11

17 2. CLOUD STORAGE 2.3 CEPH CEPH is a unified, distributed storage system designed for excellent performance, reliability and scalability.[3] It s built on top of Reliable Autonomic Distributed Object Store (RADOS) which provides foundation for all other services provided by CEPH. Essential building blocks of CEPH cluster are[4]: Object Storage Daemons (OSDs) store data as objects on storage nodes Monitors maintain copies of cluster structure, monitors data nodes and provide information for clients about cluster OSDs are closely analogous to bricks in GlusterFS. Apparent difference is only in their internal structure. They also recommend separated file system and requires user extended attributes. For locating data in cluster CEPH uses Controlled, Scalable, Decentralized Placement of Replicated Data (CRUSH) algorithm. Figure 2.2: CEPH cluster services built on top of RADOS[11] 12

18 2. CLOUD STORAGE CEPH cluster builds it s services using librados library which is also available as general purpose library for accessing RADOS. RADOS organizes internally it s structure into storage pools (similar to volumes in GlusterFS) which can have different attributes such as number of replicas or different CRUSH map for data placement. Services that build on RADOS are (also shown in figure 2.2): RBD RADOS Block Device (RBD) (discussed later in section 2.3.1) RADOSGW/RGW RADOS REST 2 full Gateway CEPH FS POSIX-compliant 3 DFS similar to GlusterFS Configuration of CEPH cluster is done through configuration files, which are well documented and provide wide variety of configuration options. All configuration options are stored in single file. Along with configuration, CEPH have it s own authentication protocol similar to Kerberos in order to maintain security in communication between nodes in cluster RADOS Block Device (RBD) RBD service provides access to virtual block devices built on top of RADOS for use mainly in VMs. These devices can be access using Linux kernel module utilizing kernel objects (KO) or by QEMU hypervisor accessing them directly via librbd library. By default RBD stripes a block device image over multiple objects to increase performance.[12] 2. REST - Representational State Transfer 3. POSIX - Portable Operating System Interface standard 13

19 2.4 DRBD 2. CLOUD STORAGE For performance comparison use, we also mention Distributed Replicated Block Device (DRBD) as distributed file system with very limited distribution capabilities. It s suitable for 2-node clusters where it works best. Despite it is able to run in 3 node cluster, its configuration is very complicated and brings very limited use in scenario where we want to use tenths of nodes. As the name suggests its main use is to provide network replicated block device. In figure 2.3 is shown how DRBD works in 2-node cluster scenario. Both nodes can see DRBD block device but only one can actively use it. In case of failure of one node, DRBD relies on external application or manual intervention to change access point for DRBD block device. Figure 2.3: DRBD operation example[16] 2.5 Other distributed file systems Following DFSs have some potential in cloud storage environment. But we consider them not to be ready for production environment because of many reasons, which we ll discuss later. However their concepts are interesting for being included into this work. 14

20 2. CLOUD STORAGE Moose FS This DFS is similar in architecture to GlusterFS, but it implements some interesting unique features. Most interesting one of them is per-file replication, which allows to define number of replicas in granularity of one file (GlusterFS replication level granularity is only on volume level).[5] Another feature such as file system wide trash bin which allows to delete files not instantly but after selected period of time are rather eye-candy. List of disadvantages for this DFS starts with existence of metadata server which creates place for performance bottleneck and also it introduces SPOF.[14]. As workaround to problem of SPOF it is possible to use external high-availability solutions such as pacemaker 4, but it s not as stable and reliable as mechanisms that are parts of DFSs themselves SheepDog Another promising project that addresses exactly the problem of distributed storage for the purpose of virtualization. It supports block devices, replication, snapshots, thin provisioning and other useful features. Current it s limited for user with QEMU/KVM hypervisor and high-availability features are supported by underlying cluster management software (corosync, zookeeper, accord). [6]. The biggest problem of this DFS is immaturity of whole project and lack of activity in it that would make it widely available in distributions in recent versions. Road-map of project promises a lot but reality is somewhere else

21 Chapter 3 Cloud networking Network traffic in cloud can be categorized into two main types: VM traffic generated by running VMs in cloud management/overhead traffic needed by various components of cloud to work properly together VM traffic can be considered as pure as the traffic in the environment where we use bare-metal servers. This traffic is what we can see arriving/leaving the cloud or traveling between VMs in cloud. This represents real user data in network. On other side there is so called management (or more precisely overhead) traffic. In one single machine environment this would be equivalent to communication between components of that machine. In cluster of computers this includes DFS communication between data nodes, monitoring communication, cloud management communication, network overhead because of encapsulations and other communication not related to VM traffic. 3.1 VM traffic On this level, we may consider VM traffic to be regular traffic, which can be seen real world networks. But the difference is that it would be most probably running over virtualized environment that is hidden from our viewing perspective. Virtualization enables us to virtualize whole networks in ways that most of devices in network topology can be virtual. For end users this concept should be not visible or disruptive. The way how this is achieved is discussed in section about management traffic. Away from how the data are transferred over network, we deal with numerous ways how to interconnect with other networks. These concepts are taken from virtualization scenarios and they provide following abilities for communication: 16

22 3. CLOUD NETWORKING Isolated networking VMs, which are connected to isolated network, can only communicate between each other. NAT-ed or routed networking Same as isolated networking but there is a point in network (computer, router,... ) which has connectivity to outer network. This single point which defines how to reach outer network can be limited (Network address translation (NAT)) or restricting (static routing). Direct networking VMs are connected to some existing network without major limitations and acts as part of that network. Typical example of this is network bridge which interconnects real and virtual network on virtualization server. 3.2 Management networking Non-negligible amount of traffic we observer in physical network of cloud is used for management, monitoring and subsystem communication. This category covers many different types of traffic which have different requirements on network parameters such as throughput, latency or jitter DFS traffic The largest amount of this traffic comes from DFS subsystem as it s responsible for providing DS over network. This kind of traffic is very sensitive to network latency and jitter which greatly influences its performance. According to speeds of today s hard drives, DFS have no problem to utilize 1 Gbit network interface. Because 10 Gbit network cards are still not widely used nor cheap, we should try to optimize performance on 1 Gbit network. First possible optimization is dedicating separated network exclusively to DFS communication. This improves processing at switch level where most of traffic are large packets. Second recommended optimization is increase of Message transmission unit (MTU) to reduce overhead of packet header. For example, increasing MTU size from 1500 bytes (standard Ethernet) to 9000 bytes (jumbo frame) requires 6-times less headers and CPU interrupts to send and receive such a packet. This can be very beneficial for DFS traffic as most of it is expected to be large in size. 17

23 3. CLOUD NETWORKING Cloud management traffic Depending on selected cloud management solution, it might be needed to protect communication between cloud management and rest of cloud infrastructure by separating this traffic. Despite this applies mainly to applications unable to secured communication themselves, it s also a good practice to do it for hardening management environment. Another reason can be the fact that some of traffic might be time critical and in case it is disrupted, it can paralyze management of cloud or even cause Denial of Service. 3.3 Inter VM network separation techniques In inter VM communication we face the problem of making communication link between two or more processes (VMs) that can be on one or more virtualization servers. Problem is how to separate this traffic easily and dynamically. We are focusing on situation when VMs are on different virtualization servers. Both of VMs that wants to communicate have Virtual NICs (VNICs) that are visible as VNIC on virtualization hosts. One of possible solutions is to setup static routing based on knowledge of network between VM and route it between servers. Such an approach leads to no separation at all, requires manual setup and is not flexible. Because of that, we describe here better solutions Linux bridge In case we have more of VNICs that we need to put into same network, we can use Linux bridge which groups them together under one bridge. This allows us to easily create distinguishable groups on single virtualization host. But very soon we ll discover that this approach is also not flexible as it limits us to only interfaces that we have available on virtualization host. It doesn t provide any separation on network out of virtualization host (except the case we have the server that has as many physical NICs as VMs). To overcome this limitation we need something that would help us extend the influence of separation we have on local machine to the rest of the network VLANs Answer to problem of network separation can be Virtual Local Area Networks (VLANs) defined in RFC 802.1Q[18]. Thanks to VLANs, we can have more than just one layer 2 network on one single physical NIC. VLANs are amend to standard that allows to put small tag on network packet with 18

24 3. CLOUD NETWORKING identification number with value of By using this tag, the network devices can identify the network to which the packet belongs and treat it accordingly. The catch here is that network devices such as switches and routers must understand this tag and route it to right destination where it belongs to. Setting up VLANs can be done either manually or automatically using GARP/GVRP or MRVP protocols. However this can be problematic for security reasons as administrators might not wish to allow network devices to be reconfigured dynamically as this can cause harm to the rest of network in case this goes out of control. Another problem is that number of VLANs is limited to 4094 which in large deployments can be easily depleted Tunneled protocols and tunnels Another way to create separated layer 2 networks on single physical NIC is the use of tunneling protocols creating tunnels working on top of existing layer 3 network. This can be very useful as it allows not only to solve problem with VLAN limits but also allows the networks to be interconnected beyond the borders of local layer 2 network. This feature comes with price of overhead as these protocols needs to package data inside the regular packet which means they transfer less data compared to ones that are just tagged as VLANs. Despite there are quite many encapsulation protocols we picked following that can server purpose of interconnecting VMs: NVGRE Network Virtualization using Generic Routing Encapsulation STT Stateless Transport Tunneling Protocol for Network Virtualization VXLAN A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks Open vswitch This very interesting open source project tries to take best from Linux bridge and combines it with ability to dynamically create VLANs and various encapsulation tunnels. It s designed to be configured and monitored using standardized protocols such as OpenFlow, sflow or NetFlow. [7]. We may imagine open vswitch as very intelligent Linux bridge capable of collaboration with Software defined network (SDN). 19

25 Chapter 4 Cloud management Cloud is very abstract definition and so it could be a problem to define what the Cloud management should be responsible for. We focus on narrowing our definition to IaaS and PaaS in private cloud but we ll still refer to it as to Cloud management. As essential it should provide unified interface to administration of all cloud subsystems such as storage and network management discussed in earlier chapters. We also believe that this unification should bring easier maintenance of these systems for administrators. Sadly this is not always true and sometimes it can make things more complicated. 4.1 Layer-bloat problem In Cloud storage chapter we mentioned problem with layer-bloat in storage management. Here we are facing similar problem, which affects the way how the management system is deployed and used. Many cloud management systems were created on top of other existing utilities and software which provided parts of its functionality. This is not bad as it uses already existing utilities to do the job. Problem arises when these utilities starts to be dependent on each other as all of them provides its own scope of configuration options and its own layer of functionality that builds on top of the other. The more layers of these utilities are bonded together, the more probable is that failure of any of them would bring breakage of whole system. As an example serves existence of cloud management for other cloud managements which tries to make abstraction over all other abstractions over management of cloud. Existence of such software indicates disastrous state of these kinds of software. 20

26 4. CLOUD MANAGEMENT 4.2 OpenStack OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface[8] Figure 4.1: OpenStack architecture[13] As description from OpenStack homepage says, OpenStack is aiming at providing solution for complete cloud management. Whole system is logically divide into three main categories shown in the figure 4.1. These categories include projects which deals with tasks in different areas: OpenStack Compute (Nova) OpenStack Image service (Glance) OpenStack Networking (Quantum) OpenStack Object Storage (Swift) OpenStack Block Storage (Cinder) 21

27 4. CLOUD MANAGEMENT Furthermore OpenStack have projects, which overcome categorization and work in general scope of whole OpenStack environment: OpenStack Identity (Keystone) Metering (Ceilometer) Orchestration (Heat) OpenStack Dashboard (Horizon) OpenStack Common Library (Oslo) Figure 4.2: OpenStack subsystem logical architecture (version Grizzly) [13] Each of this projects takes care for area of its influence in OpenStack environment and can be possibly stand-alone. This way every project can be developed on its own and it needs just to provide some interface to rest of OpenStack ecosystem. However, some projects might be dependent on each other and need to communicate. To imagine how this communication is architected, we provide look at newest version of OpenStack named Grizzly in figure 4.2. To ease configuration of whole system, developers provide not only manuals, but also definition files for automated deployment used by various tools such as Puppet or Chef. The biggest problem could be the number of nearly independent projects that needs to be configured separately to form OpenStack. 22

28 4. CLOUD MANAGEMENT Distributions powered by OpenStack as of beginning of 2013 include Cloudscaling, Debian, Fedora, Piston Cloud Computing, Red Hat Enterprise Linux, SwiftStack, opensuse, Ubuntu and Stackops. OpenStack is also being integrated into software from HP, which wants to create software that would manage whole datacenter. [20] 4.3 OpenNebula OpenNebula.org is an open-source project developing the industry standard solution for building and managing virtualized enterprise data centers and enterprise private clouds. [9] Architecture of OpenNebula can be simply divided into three main categories by type: Tool, Component, Interface/API. These types then form the whole environment around it as shown in figure 4.3. Figure 4.3: OpenNebula architecture[10] 23

29 4. CLOUD MANAGEMENT The main functionality is located in one central component OpenNebula. Communication with other components is done through XML-RPC API or specialized internal APIs. Specialized APIs are designated to give control to scripts and programs that communicate directly with physical infrastructure of Open- Nebula cloud environment. Objectives that these programs fulfills are: Transfer Manager (TM) transfers configuration files and disk images between nodes in infrastructure Information Manager (IM) gathers information about node operation and monitors state of VMs running on it. Virtual machine Manager (VM) interacts with local hypervisors and manages VMs life cycle on nodes. Authentication API authenticates users. These APIs are on side of physical infrastructure provisioned by scripts mostly written in bash or ruby. This allows great flexibility for future customizations as these scripts can be easily edited. Another special thing is that these scripts are main part of software needed for node to be part of cluster. Except of them only few system utilities are needed depending on used features. Distributions powered by OpenNebula as of beginning of 2013 include Debian, Red Hat Enterprise Linux, CentOS, opensuse and Ubuntu. and Stackops. 24

30 Chapter 5 Performance testing Important part before deployment of any part of new cloud system is performance testing of individual components to maximize performance of future system. We conducted the following tests to determine possible physical limits of available hardware of our deployment and to find best possible combination of components that will bring maximum possible performance for our scenario of deployment. 5.1 Physical testing environment The testing was conducted on 5 nodes running latest 64bit version of Fedora 18 OS. From hardware point of view, there were 4 identical nodes ( later referred to as data nodes) and one more powerful node ( later referred to as master node ). Master node was used as client of DFSs and as master node in cloud management testing. Configuration of master node: Processor : 8-core AMD Opteron 6134 Memory : 16 GB ECC DDR2 RAM Disk : 2x 1 TB Seagate Barracuda Network : 2x Gigabit Intel PRO 1000 Configuration of data node: Processor : 2-core Intel Core 2 Duo E8500 Memory : 8 GB ECC DDR2 RAM Disk : 2x 1 TB Seagate Barracuda Network : 2x Gigabit Intel PRO

31 5. PERFORMANCE TESTING Nodes were interconnected using HP 2848 ProCurve switch. Each node was connected to network with both NICs using 1 Gigabit Full-duplex link: NIC1 - MTU 1500, connected to Internet NIC2 - MTU 9000, connected to local network on switch Disks were directly attached to motherboard without using hardware RAID controller. Each node used one disk for OS and one disk for our tests. Before testing, disks were checked for errors and completely rewritten using random pattern. 5.2 Virtual testing environment To simulate performance in real deployment, some tests were run in VM with following configuration: CPU: 1x kvm64 CPU on KVM enabled host Memory: 2048 MB System: Fedora 18 x86 64 freshly booted from LiveCD Disk: attached using virtio driver with disabled caching of data The tests were conducted with SELinux in enforcing mode. Tests in VMs were run from non-graphical terminal to avoid performance drop caused by graphics emulation. 26

32 Chapter 6 Storage performance testing For comparable performance we have chosen hard drives with similar physical characteristics such as size, linear read and write speed. Comparison of these values was done by a simple test of 2000 reading and writing of 1 MB blocks to disk. The test was conducted using dd program with direct flag set for ensuring that page cache and disk cache are avoided for both reading and writing. The difference between disks was less than 3% during whole testing. Because most of DFSs use some existing underlying file system on disk we have also tested performance of two file systems (FSs). One was ext4, which is FS widely used as default FS on many Linux distributions and represents conservative FS approach. Second is btrfs, which is FS that gathers lots of popularity by re-making way how modern FSs works. Both of FSs were tested using IOzone File system Benchmark 1. All raw results of following tests can be found as attachment of this work. Test were ran in a way that page and disk cache is avoided. For our work we consider following results from IOzone relevant as they represent extremes which are crucial for wide range of deployments of DFS: Linear read performance Random read performance Linear write performance Random write performance On both file systems, ext4 and btrfs, we ran the test two times, once with 200 MB and once with 400 MB testing file. As a record size we have tested power of 2 from 128 bytes to 8096 bytes (8kB). As relevant record sizes we consider of 512 bytes, 1kB, 4kB and 8kB. While 512 bytes and 4kB are common disk block sizes, 1kB and 8kB are their multiplies. Results of IOzone benchmark on these two FSs show the figure 6.1 and figure 6.2. Please notice that these graphs are scaled to show small differences between FSs

33 6. STORAGE PERFORMANCE TESTING Figure 6.1: Linear read/write performance of ext4 and btrfs on physical disks Figure 6.2: Random read/write performance of ext4 and btrfs on physical disks 6.1 DFS capabilities requirements For our deployment we decided on following requirements for DFS: hardware independent It s essential that DFS would be independent of hardware on which it s run as most of it would be commodity HW with possibly different configurations distribution independent There must be at least 2 officially supported Linux distributions with this file system support redundancy of data each piece of user data must be stored at least at two physical nodes to provide redundancy and fault tolerance random read/write performance DFS should be able to provide high performance in random reads and writes of small blocks (512 bytes, 1 kb, 4kB, 8kB) on-line configuration DFS should provide ability to change number of data nodes without need to shut down whole DFS self-healing DFS should be able to heal itself after node failure. Data with reduced redundancy must be always available. It s not feasible to deny access to data because of undergoing replication. maintenance mode DFS should provide reasonable function for exercising 28

34 6. STORAGE PERFORMANCE TESTING maintenance on multiple data nodes at once without great impact on performance of DFS = DFS should not re-balance abruptly 6.2 DRBD Before moving to truly distributed DFS, we have tested performance of DRBD. This was done to compare it to performance of other DFS. DRBD is architected to be most efficient in scenario of disk mirroring between two nodes. We expected write and read performance to be very close to native performance of physical disks as if they were stand alone. In test results we were interested in amount of overhead which brings this networked file system as there is need for synchronization between nodes over network. We also wanted to compare this performance on physical (bare metal) and virtual machine (VM). Results of these tests can be seen in figure 6.3 and figure 6.4. As we can see the overhead of virtualization layer and its impact on I/O (input/output) performance of disks is marginal. Figure 6.3: Linear read/write performance of ext4 and btrfs on DRBD device in virtualized (virt) and physical (bare) environment Figure 6.4: Random read/write performance of ext4 and btrfs on DRBD device in virtualized (virt) and physical (bare) environment 29

35 6. STORAGE PERFORMANCE TESTING 6.3 GlusterFS First of tested DFSs, which meets most of our requirements, was GlusterFS. Performance tests were conducted using version 3.4-alpha which was stable enough for the testing. In general, it was very easy to start using GlusterFS as it has rapid deployment ability, which only requires few commands to have DFS up and running. There was no need to mess around configuration files, everything was configured via command line interface (CLI). Since the version 3.2 the CLI is only supported way of configuring GlusterFS which makes some manuals around Internet useless. We tested GlusterFS in configuration with distributed, replicated and striped volume to meet requirements of high performance and redundancy. Exact commands used for creating volume can be found in attachment of this work. Because GlusterFS doesn t provide block device support yet, we had to use files to simulate block device instead. So the virtual disk for VM was in fact RAW nonpreallocated image file. Graphs bellow shows the performance of GlusterFS attached to VM as storage depending on FS used. In figure 6.5 are tests of linear read and write operations. In figure 6.6 are results for random read and write. Figure 6.5: Linear read/write performance of ext4 and btrfs on GlusterFS in virtualized environment From the graphs can be seen that, there is a measurable difference in read and write performance of GlusterFS depending on underlying FS and FS used in VM. But in most scenarios this difference is too small to bring some benefits. 30

36 6. STORAGE PERFORMANCE TESTING Figure 6.6: Random read/write performance of ext4 and btrfs on GlusterFS in virtualized environment Due to the lack of block device support the previous tests might be discriminating compared to other DFSs that have this functionality. Work on implementing this functionality is planned for the final version 3.4 of GlusterFS. However it would be restricted to cluster with one node only which is still not acceptable for deployment in distributed environment. We believe that even despite of that, the performance of block device over DFS would be in that case better than compared to image file over DFS as we tested now. Another downside we encountered was limited ability to control fail domain. This is very essential thing in planning the outage of several data nodes for maintenance. This is somehow possible in case of replicated volumes where exists concept named Replica-Set. Replica-Set holds list of nodes which can be safely taken down without disruption of whole DFS. However this list of nodes is defined statically when volume is created or extended. GlusterFS doesn t allow dynamic change, which might be needed in some cases. In case we want to turn off machines in one rack that are not in same Replica-Set, there is no possibility to do that or to change it without taking down the DFS. 6.4 CEPH Second tested DFS that we have chosen was CEPH in version Deployment was not so straightforward as in previous DFS and required several stages of file copying over network to establish working cluster. Learning curve to figure out the right configuration was a bit steep. The tests here used RBD block device provided by CEPH, which was stripped and replicated over the cluster to meet requirements. Block device was attached to system via Linux kernel module, which should bring better performance in 31

37 6. STORAGE PERFORMANCE TESTING some cases but doesn t support some features such as copy-on-write clones. Scripts used to configure testing environment can be founded in attachment of this work. Graphs in figure 6.7 and figure 6.8 shows performance of CEPH RBD device used by VM as storage device. Interesting is good read performance of ext4 used in VM. Figure 6.7: Linear read/write performance of ext4 and btrfs of CEPH RBD in virtualized environment Figure 6.8: Linear read/write performance of ext4 and btrfs of CEPH RBD in virtualized environment Despite quite complex configuration CEPH gives us more options that we can use. For example one configuration, which is separated from main configuration file, is CRUSH map. This map describes how the cluster looks and allows us to create logical view over it. So we can organize OSDs into racks, storage rooms, datacenters or anything that we come to into hierarchical structure. But CRUSH map is not about only organizing things but it s about the weight of each part of tree hierarchy in determining where to place data. This can be especially useful when we need to change physical topology of cluster and then we want to rebalance the data distribution in the cluster. Also it can be used for maintenance when we can tag some parts of tree to be offline for a while but we 32

38 6. STORAGE PERFORMANCE TESTING don t want to rebalance data that are stored there. This is case when we want to just reboot one rack of computers for maintenance. 6.5 MooseFS and SheepDog Through initial testing we have also tried other DFSs which got major drawbacks described in earlier chapters. MooseFS was hot candidate as competitor of GlusterFS however it s stability was a concern to us because it was susceptible to crashes which lead to DFS unavailability during our testing. Lack of block device support without any plans for near future made this DFS not interesting for further testing. The main problem of SheepDog project is its immaturity. It still contain some serious flaws and it s not supported by many distributions in recent versions. It s high-availability stands on top of other software such as corosync, which we don t think it s the good solution for DFS used for virtualization. Still concepts presented by this project are really interesting as they describe the need of specialized DFS for virtualization. Despite all tries we were not able to make it run on recent version Fedora. 33

39 6. STORAGE PERFORMANCE TESTING 6.6 CEPH and GlusterFS performance comparison We consider the comparison of CEPH and GlusterFS as very interesting. Comparison of linear read/write speed can be seen in figure 6.9. Despite CEPH clearly wins in speed of linear write, GlusterFS is comparable and sometimes better in linear read. Figure 6.9: Linear read/write performance between CEPH and GlusterFS The situation in figure 6.10 however shows mostly better or comparable speed of random read/write operations for CEPH. The reason of dramatically higher read performance in case of ext4 as VM file system is still a mystery for us. Figure 6.10: Random read/write performance between CEPH and GlusterFS 34

40 Chapter 7 Cloud management testing Our knowledge of cloud management starts from assumption that it could be very similar to libvirt library, which is widely used at Linux platform for unified access to virtualization. As we discovered most of cloud management projects build on top of it. However this new layer of abstraction brings new set of problems in way how the whole system works. Some of them that we encountered are described later in next chapter. Our vision about the cloud management was that it would be able to work with at least one of tested DFSs. Also it would support the use of some more advanced networking setup either using Open vswitch or similar. For consistent testing environment, we first used VMs to test cloud managements as they are easily disposable. This approach saved us hours of sitting in server room testing this on real hardware as VMs were breaking down pretty regularly. The goal of this phase was to test feasibility of deployment and create customized manuals for future deployment of selected projects in our environment. Following Linux distributions were used to host cloud management software: Fedora 18 x86 64 Fedora 19 x86 64 Debian 6 x86 64 CentOS 6 x86 64 Second phase was bare metal deployment in our 4-node data cluster with one master node. We have tested customization of tested systems to our environment. Compared to isolated virtual environment in first test phase, the biggest problems here lied in network customization. As real network was behaving differently than simulated environment and we interacted with living network. 35