PARALLELS CLOUD STORAGE Performance Benchmark Results 1
Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements... 5 High Reliability... 5 Extensive Scalability and Flexibility... 6 Top Performance... 7 Replication In Parallels Cloud Storage... 7 Overview... 7 Results of Replication Performance Tests... 10 I/O Performance Tests... 12 Overview... 12 Test Results for Horizontal Scalability... 13 Test Results for Vertical Scalability... 14 Conclusion... 16 Appendix: Test Lab Configurations and Methodologies... 17 Replication Testing... 17 Performance Testing... 19 2
Executive Summary To support growing demands for both high performance and high data availability, modern data centers need a fast, flexible storage solution. But this requirement presents problems: besides the difficulties involved in managing and maintaining such storage, there are flexibility and cost issues. Redundant storage is typically either not flexible (as is the case with local RAID arrays) or too expensive for many organizations (as is the case with SAN storage). Parallels Cloud Storage is designed to solve both problems. It can run on commodity hardware, thus not requiring any significant infrastructure investment; and it is a very easy solution to set up and grow on demand. This white paper provides a description of the architecture, major features, and performance of Parallels Cloud Storage. Architecture Overview Parallels Cloud Storage is a highly available distributed storage system with built-in replication and disaster recovery. It runs on top of commodity hardware, using locally attached hard drives to create storage clusters that span multiple machines. This approach allows rapid live migration of containers (CTs) and virtual machines (VMs) across hardware nodes, without having to copy CT or VM data. It also ensures high availability, as it makes storage across the cluster available remotely. Figure 1 illustrates the logical structure and basic components of Parallels Cloud Storage. The key components are: Chunk Servers (CSs). All data in a Parallels Cloud Storage cluster, including disk images of both CTs and VMs, is stored in the form of fixed-size chunks on chunk servers, which provide access to the data as needed. The cluster automatically replicates the data chunks and distributes them across the available chunk servers to provide high availability of data. Therefore, to ensure this high availability, a Parallels Cloud Storage cluster must have multiple chunk servers 3
Metadata servers (MDSs). To keep track of data chunks and their replicas, the cluster stores metadata about them (such as their file names) on metadata servers. In addition to managing metadata, the MDSs control how files are split into chunks and where the chunks are stored. They also track versions of chunks, ensure that the cluster has enough replicas, and keep a global log of important events that happen in the cluster. As is the case with CSs, multiple MDSs are needed to provide high availability. Clients. Clients manipulate data stored in the cluster by sending different types of file requests for example, to modify an existing file or create a new one. Clients access a storage cluster by communicating with the MDSs and CSs. Parallels CT and VM clients can be run natively i.e., directly from the storage cluster. You can also mount storage as a conventional file system (although Parallels Cloud Storage is not POSIX-compliant). In addition, you can create and mount image files residing on storage as loop devices by using the ploop feature. You can format these loop devices in any file system you want (ext4, for instance) and can export them using iscsi. Figure 1. Key components of Parallels Cloud Storage 4
A recommended cluster setup would consist of three to five MDS instances (allowing you to survive the loss of one or two of the MDSs, respectively) and three or more CSs to provide storage capacity. (For more information, see the Recommended Configuration chapter in the Parallels Cloud Storage Administrator s Guide.) You can set up any number of computers in the cluster to perform the role of an MDS, CS, or client. You can also assign two or all three roles to the same computer. For example, you can configure a computer to act as a client, and if you want it to allocate its local disk space to the cluster, you can set it up to be a chunk server, as well. In this paper, unless otherwise noted, the term server refers to a physical server. In each of our tests, three servers in each cluster are set up to host all three roles (MDS, CS, and client), and the remaining servers in the cluster have two roles (CS and client). Key Features No Special Hardware Requirements Parallels Cloud Storage has no special hardware requirements: it can be built using commodity hardware (such as SATA or SAS drives and 1Gbit+ Ethernet) and will run on your current hardware stack. Although additional hardware resources, such as SSD drives or 10Gbit Ethernet, can significantly boost performance, they are not required to get the major benefits of Parallels Cloud Storage: a cost-effective distributed storage solution with built-in replication and high availability High Reliability Parallels Cloud Storage is designed to ensure that data availability will not be affected by the unavailability of up to a predefined number of logical or physical components. The following features ensure high reliability: Replicated metadata. Several MDSs typically three or five run simultaneously, enabling them to survive the unavailability of one or two MDSs, respectively. As long as more than half of the MDSs are responsive, clients are guaranteed the ability to access metadata. 5
Replicated chunks. Every piece of data is saved in multiple copies across different CSs to ensure that the data is available even if some CSs are in a failed state (e.g., disconnected from the network or being rebooted). Parallels Cloud Storage also automatically maintains the required number of replicas of each chunk. If some of the replicas become unavailable, Parallels Cloud Storage will start making more replicas and copying them to new locations. High availability and consistency. The existence of multiple CSs and MDSs and the replication of data chunks across the cluster mean that even if some CSs or MDSs are unavailable, clients can still access and modify data. Parallels Cloud Storage also tracks any chunk modifications to guarantee that all pieces of data are in the same state so if a CS is down for a period of time and some of its chunks end up out of date as a result, Parallels Cloud Storage will recognize the problem and update the chunks to latest version once the CS comes back online. Data checksumming and scrubbing (works with SSD cache for a chunk server journal feature only). Although Parallels Cloud Storage itself is highly available and consistent, there is always a chance that local storage could become corrupted. To guard against such silent data corruption which could occur, for example, due to hardware faults Parallels Cloud Storage periodically performs data checks. When data is first written or modified, the storage solution calculates and remembers checksums for each piece of data. Then, in background mode, Parallels Cloud Storage periodically inspects data from local storage and compares saved checksums with newly calculated ones to ensure that there is no corruption. If it detects any corrupted data chunks, it replaces them with replicas of known-good copies. Extensive Scalability and Flexibility Another advantage of Parallels Cloud Storage is its extensive scalability and flexibility. Key capabilities in this regard include: Growth on demand. You can easily extend Parallels Cloud Storage by simply adding new CSs up to a total of 1PB of storage space. Load balancing. Parallels Cloud Storage automatically balances workloads among all available CSs. In storing new chunks, Parallels Cloud Storage takes into account both the amount of free disk space and the distribution of the I/O load, in order to ensure 6
high performance. Additionally, when you add new CSs to a cluster, Parallels Cloud Storage automatically balances existing data across all CSs. Storage tiers. Since fast storage costs more than slower storage, Parallels Cloud Storage lets you define allocation policies for different types of local drives so as to deliver the best price/ performance ratio. For instance, you may choose to store frequently accessed data on fast, but relatively expensive, SSD drives, while placing rarely needed backup data on inexpensive SATA drives. Top Performance It s hard to overestimate the importance of storage performance, which is usually the bottleneck for the whole system s performance. Parallels Cloud Storage offers several options for enhancing performance, including: Thin replication. Parallels Cloud Storage supports thin replication, in which only the changed parts of chunks are updated on recovery, rather than the whole chunks. This approach significantly boosts overall performance because it involves fewer reads, writes, and data transfers. SSD cache for a chunk server journal. You can attach an SSD drive to a CS in the cluster and configure the drive to store a write CS journal a step that can boost the performance of random write operations in the cluster by a factor of two or more. The journal can also maintain chunk checksums to improve storage reliability. SSD read cache on the client. You can attach an SSD drive to a client and configure the drive to store a local cache of frequently accessed data a step that can increase overall cluster read operations performance by a factor of 10 or more. Replication in Parallels Cloud Storage Overview To ensure high availability, Parallels Cloud Storage stores a number of replicas of each chunk across multiple physical servers. If it detects the unavailability of one or more chunk servers for a predefined amount of time, it will automatically start the replication process to ensure the availability of a sufficient number of copies of the data. Figures 2-4 illustrate the replication process in a cluster that consists of three CSs. With the replication level set 7
to 2, each chunk should have two replicas. If CS #1 becomes unavailable for some reason, chunks 1 and 2 end up with only one replica. In such a case, Parallels Cloud Storage immediately detects that these chunks have fewer replicas than they should and creates however many additional replicas are needed to bring the number up to the specified replication level. Figure 2. This healthy cluster has three CSs and two replicas of each of three data chunks. Figure 3. This cluster, with one unavailable CS, has become degraded: Chunks 1 and 2 have only one replica. 8
Figure 4. The cluster is healthy again after replication, with two replicas of every chunk Parallels Cloud Storage replication is similar to the RAID rebuild process, but it has two key differences: Replication is much faster with Parallels Cloud Storage than with a typical online RAID 1/5/10 rebuild, because Parallels Cloud Storage is able to replicate in parallel, across all servers in the cluster. In contrast, RAID is only able to copy data from a single hard drive to another single hard drive. The more CSs you have in a cluster, the less time it takes to replicate each chunk. Replication performance is important because it minimizes the time when the system operates with a lower than normal redundancy level. To ensure the overall reliability of the storage solution, replication time must be much shorter than the expected time between CS faults because the briefer the period of reduced redundancy, the less likely you are to lose any data. In theory, the mean time to data loss (MTTDL) is in reverse proportion to T2, where T is the time to recover. Factors affecting replication performance include: 9
Number of available CSs, because replication is a process that runs in parallel so the more available replication sources and destinations you have, the faster it goes. Performance of the local disk system used for storing chunks. Network performance, because as each chunk is read, it s transferred by the network to write an additional copy. Chunk distribution, because some CSs may have much more data to be replicated than others and therefore may become overloaded during replication. I/O activity in the cluster under replication. Results of Replication Performance Tests Figure 5 shows the results of the replication performance measurements of Parallels Cloud Storage, running on commodity hardware. Each server has software RAID 0 arrays, using two 7200-RPM SATA drives, with servers interconnected in a 1Gbit network. For a more complete description of the testing methodology, please refer to the appendix. Figure 5.The more servers a cluster has, the less time it takes to replicate data. 10
These performance measurements show that with Parallels Cloud Storage, replication performance scales almost linearly as the number of servers in the cluster grows. For instance, with a cluster consisting of 21 servers, if one server is dead (due to kernel panic, network disconnection, or any other factor), it will take only about 20 minutes to replicate 1TB of data. Such linear scalability is not possible with RAID. In short, these tests demonstrate that even with Parallels Cloud Storage running on commodity hardware, its replication time is a fraction of what RAID requires, resulting in much longer MTTDL, and therefore much higher reliability. Note also that, although Parallels Cloud Storage was designed to operate efficiently when running even on modest hardware, it s able to make full use of all the resources of more powerful systems. Figure 6 compares the performance of clusters with 7 and 14 servers running on software RAID 0 (with two 7200-RPM SATA drives) with that of servers running on hardware RAID 0 arrays (with four 7200-RPM SATA drives). In case of the hardwarebased system, servers were interconnected via a 10Gbit network because the 1Gbit network used in the previous tests is limited to 110MB/sec of data transfer a limitation that can become a bottleneck when used with fast disk storage. Figure 6. Using a more powerful hardware configuration enables a terabyte of data to be replicated much more rapidly 11
I/O Performance Tests Overview This section presents the results of tests comparing the I/O performance of software RAID 1 (mirroring) with that of Parallels Cloud Storage, deployed on servers configured with RAID 0 and two replicas of each data chunk. (We chose software RAID 1 for the RAID comparison because it has the same level of redundancy as a Parallels Cloud Storage configuration with two replicas.) According to our statistics, the vast majority of I/O operations on virtualization platforms are random. Based on these statistics, we used mostly random I/O for our tests, although we did include one sequential read, because it s important for some scenarios (creating backups, for instance). Our tests used the following I/O load patterns: 1. A sequential read of a 16M block (measured in MB/sec). This I/O workload is equivalent to a sequential read of a large file. 2. A random read of a 4K block (measured in IOPS). This pattern represents any workload with a random read, such as one involving small files, script codes, file system journals, or file searches. 3. A random write of a 4K block (measured in IOPS). This pattern represents any workload with a random write, such one involving file system journals or small files. 4. A random write of a 4K block + fdatasync (measured in fdatasync operations per second). This workload is the most important, because it simulates work that a relational database must do to save simple database changes during a transaction. We performed two series of tests: one series measuring horizontal scalability (that is, the effects of adding more servers up to a total of 21) and one measuring vertical scalability (that is, the effects of adding more workload threads to a single server up to a total of 16 threads). For a more complete description of the testing methodology, please refer to the appendix. 12
Test Results for Horizontal Scalability Horizontal scalability is very important for distributed systems, as it allows you to add new nodes to a cluster to increase the system s overall performance. For these tests, we compared RAID 1 performance both with that of Parallels Cloud Storage running on commodity hardware and Parallels Cloud Storage with SSD journaling added. In each case, we ran 16 threads of I/O workload on each server, starting with a cluster consisting of a single server, then moving up to 10 servers, and then to 21. Figures 7 and 8 show the performance results. In all cases, Parallels Cloud Storage demonstrates I/O performance that is at least comparable with that of local RAID storage and when SSD caching is added, Parallels Cloud Storage dramatically exceeds RAID s I/O performance. Figure 7. As the number of servers increases, the read performance of Parallels Cloud Storage increases accordingly. 13
Figure 8. As the number of servers increases, the write performance of Parallels Cloud Storage increases accordingly. Test Results for Vertical Scalability Vertical scalability is the ability to maintain the desired I/O performance as the number of workload threads on a server increases. For these tests, we ran 1, 8, and 16 threads of I/O workload on a single server in the cluster. Figures 9 and 10 show the performance results as the number of workload threads increased. As the figures show, because Parallels Cloud Storage is able to use idle HDDs on cluster servers, it thereby exceeds performance of local RAID storage which limited by local HDD speed. However, the bandwidth of 1 Gbit Ethernet may limit Parallels Cloud Storage performance on sequential reads. When SSD caching is added, the performance of Parallels Cloud Storage is several times faster. 14
Figure 9. Read performance of Parallels Cloud Storage scales well as the number of workload threads increases. Figure 10. Write performance of Parallels Cloud Storage scales well as the number of workload threads increases. 15
Conclusion Parallels Cloud Storage demonstrates little to no overhead compared to local RAID storage, while at the same time providing flexibility and scalability far beyond what s available from a local RAID storage system. Its ability to replicate data chunks in parallel, across all servers in the cluster, makes it much faster than RAID and because the faster replication results in a much longer MTTDL, its reliability is also greater. In addition, when running on commodity hardware, Parallels Cloud Storage offers horizontal and vertical scalability comparable to that of RAID and when you add SSD for caching, its performance is dramatically better than what RAID can deliver. In short, Parallels Cloud Storage gives you a storage solution that s as fast or faster than RAID, while also offering greater reliability, flexibility, and scalability. 16
Appendices Replication Testing Two hardware sets were used in replication performance testing: a modest hardware set (software RAID 0, 1Gbit network) and a more powerful hardware set (hardware RAID 0, 10Gbit network). Table 1 shows the test lab configuration for replication testing. 17
The methodology we used for replication testing was as follows: 1. Set up Parallels Cloud Storage cluster with a client mount on each server. 2. Created 100GB of data on each client: dd if=/dev/urandom of=/pstorage/client_$i_test_file bs=1m count=102400, where /pstorage is the pstorage client mount point. 3. Chose a random server and turned off its network connection (ifdown ethx, where ethx external network connection used by pstorage). 4. Checked pstorage event log for messages about replication start ( Replication started. N chunks queued to replication ) and finish ( Replication finished ). 5. Replication speed is an amount of replicated data divided by time it took to recover according to event log. 6. Replication progress can be monitored via pstorage top tool (press v to get access to replication performance stat and ETA). (For more information, see the Monitoring General Cluster Parameters chapter in the Parallels Cloud Storage Administrator s Guide.) 18
Performance Testing Table 2 shows the test lab configuration for performance testing. Table 2. Performance Testing Configuration The methodology used for performance testing was as follows: 1. We used a custom-developed I/O load tool to simulate various workloads (random and sequential reads and writes in multiple threads). The volume of test data was 16GB for each server. 2. The set up for Parallels Cloud Storage clusters consisted of 21 servers, with a client mount on each server. 3. Tested performance in several configurations, with 1, 4, and 16 workload threads running on 1, 10, and 21 servers. 19
4. For a fair comparison between local RAID and Parallels Cloud Storage, giving each the same level of redundancy and a similar logical structure, we used servers with RAID 1 (mirroring) for the local RAID tests, vs. Parallels Cloud Storage configured to store 2 data replicas. 5. For the PCS+SSD cases, SSD drives were added for both CS journaling and client read caching. 20