A Performance Study of Veritas Storage Foundation Cluster File System in a NFS file serving environment. March 2008

Transcription

1 A Performance Study of Veritas Storage Foundation Cluster File System in a NFS file serving environment 1

2 TABLE OF CONTENTS EXECUTIVE SUMMARY... 3 SCOPE & AUDIENCE... 4 INTRODUCTION... 5 About Veritas Storage Foundation Cluster File System (CFS)... 5 About Network File System (NFS)... 5 BENCHMARK CONFIGURATION... 6 Hardware Configuration... 6 Software Configuration... 6 BENCHMARK DETAILS... 8 Tested configurations... 8 BENCHMARK RESULTS /14

3 Executive Summary This paper describes the scalability of the Veritas Storage Foundation Cluster File System 5.0 (CFS) on Linux for a high volume transactional workload with a mix of file operations. Files were served using the Network File System (NFS) protocol for workloads based on the widely used and well understood SPECsfs benchmark (SPEC.org). In these tests, the benchmark suite from SPEC was used as a load generator to allow for measuring the scalability factors of CFS. Scalability measurements were taken for configurations ranging from 1-node up to 16-nodes. To ensure accurate results measurements were taken at different load points throughout the test. This led to two distinct problems. The first problem raised is a direct result of bottlenecking resulting from the configurations for the 1 and 2 node tests. With these smaller configurations, the arrays themselves became a bottleneck and maximum throughput to the arrays was reached long before the CPUs became saturated. With larger configurations, more arrays and disks were brought into play, which improved the storage configurations ability to handle higher throughput levels. The reason for this was that throughout the test a 1 to 1 relationship between nodes and arrays was used. This balanced configuration was used to ensure that the tests were performed on a realistic environment and to simulate a growing environment. The second problem with our tests is that measuring throughput at a sustained 100% CPU utilization is unrealistic in real life environments. As a result, we decided to adjust I/O throughput for each test until a more realistic 80% CPU utilization and base our tests on the amount of throughput required to reach an 80% utilization as the target for each configuration we tested. At this level of CPU utilization, the storage configurations used in these tests could be compared more easily. The scalability factors that we obtained by measuring the level of throughput required to cause 80% CPU utilization was almost linear across the range of configurations. The 16-node configuration scaled to 91% of its theoretical maximum based on 16 times the throughput of the 1-node configuration. With a scalability factor of 91% at 16-nodes, the Veritas Storage Foundation Cluster File System 5.0 provides the most scalable infrastructure for complex workloads of any clustered file system for the Linux operating system. 3/14

4 Scope & Audience This paper explains the value that Veritas Storage Foundation software can bring to customers currently using, or planning to use, Network File System (NFS) for data sharing in a performance critical environment. The paper discusses the results of a performance benchmark for a file serving workload. It does not explain in detail how to set up the configurations studied. Rather, it only describes the outcome and implications of the performance benchmark. The paper is intended for engineering/architecture and operations groups for storage and for UNIX servers in large enterprise data centers who are deploying or evaluating Veritas Storage Foundation Cluster File System. This paper assumes basic knowledge of NFS and Veritas Storage Foundation Cluster File System as well an understanding of data center storage management tools, such as SANs (Storage Area Networks), logical volumes, and file systems. 4/14

5 Introduction This paper gives insight into how the Veritas Storage Foundation Cluster File System (CFS) solution can provide scalability in high-performance file serving environments. In some environments Veritas Storage Foundation Cluster File System has become a natural replacement for NFS (Network File System), but as this paper shows, using NFS together with CFS can provide high availability and high performance for your file serving needs. The Veritas Storage Foundation Cluster File System solution guarantees availability by leveraging the leading cross-platform clustering solution, Veritas Cluster Server (VCS). Close integration with VCS enables the Veritas Storage Foundation Cluster File System solution to provide a highly available cluster file system infrastructure across multiple servers even when individual components or servers fail. As an NFS replacement, CFS is able to increase availability and performance by leveraging the SAN (Storage Area Network) for data transfers instead of slower TCP/IP based networks. When used in conjunction with NFS, CFS provides a highly available, scalable platform for environments with high transaction throughput. This document illustrates how well CFS scales for transactional workloads like file serving. While the benchmark configuration in this study focuses on CFS scalability for NFS file sharing between servers and clients, the Veritas Cluster File System scales equally well for parallel applications running locally. About Veritas Storage Foundation Cluster File System (CFS) Veritas Storage Foundation Cluster File System provides an integrated solution for shared file environments. The solution includes Veritas Cluster File System, Cluster Volume Manager and Cluster Server to help implement robust, manageable, and scalable shared file solutions. With VERITAS Storage Foundation Cluster File System, cluster-wide volume and file system configuration allows for simplified management; and extending clusters is simplified as new servers adopt cluster-wide configurations. Supports the POSIX standard and maintains cache-coherency across all cluster nodes. About Network File System (NFS) NFS is the most popular network protocol on the market to share data between UNIX and Linux servers; it was first introduced in 1984 by Sun Microsystems and later was made an official standard for transferring files between computers over the network. It utilizes TCP/IP communicate between clients and servers to concurrently share files over the network. It does not fully support the POSIX standard and is therefore not suitable for environments where data integrity across nodes is demanded. 5/14

6 Benchmark configuration Hardware Configuration Servers: 16x Sun v20z each with two 2.2 GHz AMD Opteron CPUs and 8 GB RAM Clients: 32x SuperMicro each with 3.0GHz Intel Pentium 4 CPUs and 1GB of RAM Network: Each server and client was equipped with a dedicated Gigabit network adapter for the private network used to benchmark data traffic. Storage: Each server was connected to 16 Sun StorEdge 3510 arrays through a dual ported 2 Gigabit Fibre Channel. Each array was equipped with a dual ported 2 Gigabit controller, 36x 15K RPM 36GB disks and 1 GB of cache. Software Configuration Operating System and Software stack This benchmark was performed on RedHat Enterprise Linux Advanced Server 4 with update 4. The Linux kernel used during the tests was version ELsmp. Veritas Storage Foundation Cluster File System version 5.0 with maintenance pack 1 was used during this benchmark. Storage Layout Each Sun 3510 array was configured to export 6 LUNs. Each LUN was created from 6 disks in a RAID-0 (stripe) to maximize performance. The stripe size was kept at the default 32k. Veritas Volume Manager The Veritas Volume Manager layout was scaled based on the number of nodes and arrays in each test run. Following best practices for deployment in environments with high transaction rates, separate metadata and data volumes were used to minimize the risk of creating a bottleneck at the disk level for journal updates. One disk group consisting of 6 LUNs (1-node test) to 96 LUNs (16-node test) was used 8 RAID-0 volumes were striped across the disk group for file system metadata; Their size calculated as 3GB*number of arrays 8 RAID-0 volumes were striped across the disk group for data storage. Their size calculated as 100GB*number of arrays 8 Volume sets were created. Each consisting of one metadata and one data volume 6/14

7 Veritas File System All tests used 8 file systems The file systems were created with a block size of 1KB and an intent log size of 256Mb The data portion of the file systems ranged from 100 GB to 1600 GB as the environment scaled from 1-node to 16-nodes. The size was calculated as 100 GB*Number of participating nodes All file systems were mounted as cluster file systems ( mount option -o cluster ) Primary nodes for the 8 cluster file systems were rotated among the server nodes in the configuration (for example, in the 4-node test, each server node was the primary mount point for two file systems). In the 12-node and 16-node tests, only eight nodes served as primary nodes (one node for each file system); the remaining nodes were secondary on all eight file systems. NFS tuning parameters To ensure the best possible NFS performance throughout the tests the number of NFS daemons was increased to 96. Network tuning parameters The Linux networking stack is conservative when it assigns memory to resources to network buffers. For workloads generating a lot of network traffic, increasing the defaults can yield significant performance improvements. The table below lists the modified network parameters. Table 1: Network tuning parameters Tunable Value Description /proc/sys/net/core/rmem_default /proc/sys/net/core/rmem_max /proc/sys/net/core/wmem_default /proc/sys/net/core/wmem_max Default and Maximum Socket Receive Buffer size Default and Maximum Socket Send Buffer size /proc/sys/net/ipv4/tcp_rmem /proc/sys/net/ipv4/tcp_wmem Minimum, Initial and Maximum size of TCP socket recieve buffer Minimum, Initial and Maximum size of TCP socket send buffer /proc/sys/net/ipv4/tcp_sack 0 Disables selective TCP Acknowledgement /proc/sys/net/ipv4/tcp_timestamps 0 Disable TCP timestamps 7/14

8 Benchmark details The benchmark suite used to perform these tests was SPECsfs (SFS97_R1 V3.0, formerly known as LADDIS) from (Standard Performance Evaluation Company). This is a benchmark suite specifically designed to measure performance of file serving over the NFS protocol and it is the industry standard used to measure performance among NAS (Network Attached Storage) vendors. As a benchmark tool SPECsfs is widely accepted as a good load generator for a mix of small, transactional read and writes. For file systems, this presents a difficult workload to achieve good performance results, as it requires lots of metadata and data updates across a large data set. Note that while the SPECsfs benchmark was run according to the rules and regulations from SPEC this benchmark has not been submitted as an official result. There was no intention to publish official results of this benchmark as performance of this hardware configuration is not particularly interesting. The SPECsfs benchmark suite was used only as a load generator to ensure that that the file system was put under stress from a known workload while the scalability of the file system was measured. Tested configurations To demonstrate the scalability of the Veritas Storage Foundation Cluster File System product, tests were conducted in multiple configurations. The test matrix below lists all permutations of Servers, Arrays and clients used during the tests: Table 2 Test matrix Server(s) Array(s) Clients This setup allowed for linear scaling of the workload and resources available to CFS In each case, 8 file systems were created and allocated across all of the arrays and accessed by all of the clients and server nodes. To be able to compare the runs on different numbers of nodes at comparable load points, the load generation on the clients was incremented by approximately 1000 operations per second per node. Each run consisted of 15 load points. 8/14

9 Benchmark results As with all benchmarks, one should not look just at the maximum throughput, but also consider other aspects of the configuration. For example CPU and memory utilization at a given load point. The benefit of taking system resource utilization into account is that at similar loads it becomes possible to compare test results from multiple configurations. Figure 1 shows the performance of the six tested configurations, plotting Response Time on the Y-axis and Operations per Second on the X-axis. To illustrate that the performance was about the same for each configuration, the X-axis is the average of Operations per Second per server node (total operations per second divided by the number of server nodes in the test) so that all six configurations can be shown in a single graph. Figure 1: Throughput per Node vs Response Time 9/14

10 Figures Figure 2 through Figure 4 illustrate different ways of calculating CFS scalability as the configuration grew from a 1-node to 16-nodes. In Figure 2, the maximum throughput for each configuration (bar graph) and the average CPU utilization (over all server nodes) at the maximum point (line graph) is plotted. The CPU utilization line graph show that the average CPU utilization at the maximum throughput point increased with more nodes, and that the configuration with the lowest average CPU utilization was the 1-node case. Figure 2: Maximum Throughput and CPU Utilization (at Maximum Throughput) for all Configurations The main reason the cases with fewer server nodes had the lowest CPU utilization was an I/O bottleneck that was primarily a problem with one to four server nodes tests. As previously mentioned a balanced system (The number of arrays was kept equal to the number of server nodes) was maintained throughout the tests. This means that as we grew the configuration more spindles were added to the array and as a result the throughput bottleneck shifted from the array to the CPUs. 10/14

11 Tables Table 3 and Table 4 show values from a subset of the data collected where the same load level was applied for each of the configurations (about OPS/second per node). The number of average I/Os per operation and KBs moved per operation is almost identical for 1- node to 16-nodes; thus, the load on each array should be about the same. However, the iostat value for average wait (time to do an I/O including wait time) is more than four times larger in the 1-node configuration compared to the 16-node configuration. A number of tests were performed to investigate the cause of this I/O bottleneck, but no firm conclusions were reached. Table 3: Comparison of I/O performance at approximately the same throughput (about Ops/s) per Node Number Of Nodes Measured Ops per Second Measured Ops per Second per Node Response Time (ms) (R+W) I/Os per OP (R+W) KBs per OP iostat average wait time iostat average wait time Table 4: Comparison of CPU load at approximately the same throughput (about Ops/s) per Node Number Of Nodes Measured Ops per Second Measured Ops per Second per Node Response Time (ms) sar average user CPU busy sar average sys CPU busy sar average CPU busy Approx. CPU usec busy per OP /14

12 The low CPU utilization caused by I/O throughput bottlenecks in the 1 to 4 node configurations distorts the apparent scalability of the other configurations. Using maximum throughput from each configuration the scalability from 1-node to 8-nodes would be 8.12 (118030/14539). This value (8.12) is unreasonable unless it is noted that the CPU in the 1-node case was only 80.5% busy, but in the 8-node was 96.8% busy. Because the 1-node configuration could only achieve about 80% CPU utilization and also most customers would not be running systems at an average close to 100% utilization, a better assessment of scalability is to compare the different configurations at about the same (high) CPU utilization. Figure 3 is like Figure 2 except the throughput and CPU utilization are from the measured load that came closest to 80% busy for each configuration. Figure 3: Throughput and CPU Utilization at 80% CPU Busy 12/14

13 Figure 4 presents the same data as Figure 3, but instead of reporting throughput, the graph reports the scaled throughput multiple for each configuration relative to the 1-node configuration (Throughput with N-nodes divided by throughput with 1-node at about 80% busy). As can be seen in these two figures the collected data points are in all cases very close to an average CPU utilization of 80% (The line graph is almost parallel to the X-axis). In Figure 3 the measured scalability is more reasonable for comparison and shows a very high level of scalability. For example, the 1-node to 8-nodes throughput ratio is 7.37 and the 1-node to 16- node ratio is One of the reasons the values are not 8.0 and 16.0 is that the CPU busy time per operation is higher when there are more server nodes. In Table 4 above, the second to last column shows the average CPU utilization. The rows all show the performance when doing about the same load per node (12000 operations per second). Note that the average CPU busy time per operation increases as the number of nodes increases. To account for some of the load variability an estimation of the CPU time per operation can be made by dividing the total busy microseconds (calculated from sar cpu busy) per second by the operations per second. This is what is reported in the last column of Table 4 as Approx CPU microsecond busy per operation. For example, at the operations per second per node load point, an operation took about 15% more time on the 16-node configuration as compared to the 1-node configuration. The cause of this increase was not investigated as a part of this project, but this type of increase is commonly seen in multi-processors servers as the number of CPUs is increased. One likely cause for the increase is an increase in the number of instructions with more nodes as the potential for contention and the need for coordination increase. Figure 4: Scalability factor at 80% CPU busy 13/14

14 The tables below show the throughput ratios for the various configurations using the two methods illustrated in Figures Figure 2 through Figure 4. Table 5 and Table 6 show the ratio of the throughput for the N-node configuration (far left column) divided by the throughput for the 1- node, 2-node,, 12-node configurations. The far right hand column shows the scalability as a percentage (scalability vs. 1-node divided by the number of nodes). In Table 5, the values shown are from the maximum throughputs for each test (values shown in Figure 2); in Table 6, the values are from the measured throughput tests when each configuration was about 80% CPU busy (values used in Figures Figure 3 and Figure 4). The values in Table 5 are all very high; again, these values do not factor in the fact that the throughput in the smaller configurations is limited by the array. The values in Table 6 (that factor in the CPU utilization) are still quite high and show that the 16 nodes scalability (as a percentage) is 91% for this particular workload. Table 5: Throughput Ratios of N-nodes vs. M-nodes at Maximum Throughput Number Of Nodes Vs. 1 Node vs. 2 Node vs. 4 Node vs. 8 Node vs. 12 Node Percentage of Perfect Scalability % % % % % % Table 6: Throughput Ratios of N-nodes vs. M-nodes using Throughput at 80% CPU busy Number Of Nodes Vs. 1 Node vs. 2 Node vs. 4 Node vs. 8 Node vs. 12 Node Percentage of Perfect Scalability % % % % % % 14/14