Zimbra Collaboration Server Performance on VMware vsphere 5.0. Performance Study

Transcription

1 Zimbra Collaboration Server Performance on VMware vsphere 5.0 Performance Study Technical WHITE PAPER

2 Table of Contents Introduction Architecture and Description Performance Test Environment Workload Description... 5 Hardware Configuration... 5 Server Hardware Description Storage Hardware and Configuration Software Configuration Software Versions Methodology... 7 Single Virtual Machine Scale-Up Tests Memory Sizing Storage Sizing Multiple Virtual Machine Scale-Out Tests... 7 Performance Results... 8 Virtual Machine Scale-Up Results Virtual Machine Comparison with Native Environment Performance Optimizations Virtual Machine Scale-Out Results Comparison with Microsoft Exchange Server Conclusion References TECHNICAL WHITE PAPER / 2

3 Introduction Zimbra Collaboration Server provides a full-featured and robust suite that includes , calendaring, and collaboration software. Zimbra customers include enterprises such as H&R Block and Bechtel, small-to-medium businesses (SMBs) such as Red Hat and Digg, and cloud and hosting providers, such as Comcast and NTT Communications. Zimbra Collaboration Server has become an important piece of software that VMware is leveraging and extending into its VMware vcloud platform as a software-as-a-service (SaaS) offering. 1 Versions of Zimbra Collaboration Server include a network edition, an open-source edition, and a prepackaged virtual appliance. This paper describes a detailed performance analysis of the Zimbra Collaboration Server on VMware vsphere 5.0. The goals of the performance study are the following: To measure and optimize the scale-up performance of Zimbra Collaboration Server in a four and eight virtual CPU (vcpu) virtual machine and compare the results with those of a native Zimbra environment To compare the performance of Zimbra Collaboration Server with Microsoft Exchange Server 2010 in a similar configuration To measure and optimize the scale-out performance of Zimbra Collaboration Server from one to eight 4-vCPU virtual machines Test results demonstrate the following: Due to optimizations made within Zimbra Collaboration Server and the tickless timer within Red Hat Enterprise Linux (RHEL) 6, Zimbra Collaboration Server in a virtual machine can scale up effectively, and performs within 95% of a physical host. Zimbra Collaboration Server scales out effortlessly, with only a 10% drop in Sendmail latency as up to eight virtual machines are added. Zimbra Collaboration Server consumes less than half of the CPU of Microsoft Exchange Server 2010 for the same number of users. The user provisioning time for the tests is also orders of magnitude better. TECHNICAL WHITE PAPER / 3

4 Architecture and Description Although the mail server is its primary component, Zimbra Collaboration Server includes many other software pieces such as desktop and Web mail clients, and connectors for Microsoft Outlook and mobile clients that form a feature-rich collaboration suite. Zimbra Collaboration Server supports multiple client connectivity protocols, such as MAPI, SOAP, IMAP, POP, CalDAV, and XHTML. This study focuses only on the performance of Zimbra mail server using the most popular Zimbra Web client protocol, which is SOAP. The mail server itself includes many open-source integrations that use industry-standard protocols. The following software is bundled with the mail server and installed as part of the installation process: Jetty: a Java application servlets container that runs the Zimbra software Zimbra Collaboration Server stores all the messages and file attachments (immutable data) in its message store in MIME format. Postfix: a mail transfer agent (MTA) that routes mail messages to the appropriate Zimbra server OpenLDAP: an open-source implementation of the Lightweight Directory Access Protocol (LDAP) that provides user authentication MySQL: a database engine used to store and access the metadata (mutable data) of mailboxes The MySQL data store is the database that contains the metadata contents of each mailbox user s tags, folders, and contacts, as well as the status of messages such as read and unread. Lucene: a full-featured text and search engine The Lucene index store maintains the search index files for each mailbox. Verity: a third-party source that converts certain attachment file types to HTML ClamAV, SpamAssassin, Amavisd: anti-virus and anti-spam components jsieve: a mail-filtering language; jsieve scripts create filters for This study focuses on testing the Zimbra mail server with Jetty, Postfix, LDAP, MySQL, and Lucene. Figure 1 shows all of the previously described components, as well the system under test (SUT). Zimbra Collaboration Server (Jetty+JVM) Message and File Store Search Metadata Directory Database Reliability Attachment Index & View Free/Busy Providers JDBC Third-Party Storage System Standard Zimbra Storage System Lucene Index MySQL Open LDAP External LDAP Active Directory Zimbra Journaling Atonomy Keyview IBM Domino Microsoft Exchange Tested Components Figure 1. Mail Server Architecture TECHNICAL WHITE PAPER / 4

5 The Zimbra user mailboxes are stored as flat files on the local file system. Tunables for optimizing the performance of Zimbra Collaboration Server on ext3 for Linux is available in Performance Tuning Guidelines for Large Deployments in the Zimbra wiki. 2 Performance Test Environment This section describes the workload that was used and its characteristics, the hardware and software configurations, and the testing methodology. The same components were used for both the physical machine and the virtual machine experiments. Workload Description The workload used in the tests was Zimbra Loadgen, which is developed and maintained in the Zimbra performance lab. Zimbra Loadgen is a comprehensive and flexible mail server load generator designed to provide functionality similar to Exchange LoadGen. 3 Zimbra Loadgen can support AJAX, JSP, IMAP, and POP, as well as mailbox profiles. For the tests, a profile that emulates a typical mail user in an SMB was designed with the following properties: Simulates the SOAP Web client 250MB mailboxes per user 30KB message body 100KB attachment: 5% of the s have the attachment during provisioning, 10% during run time Five fixed recipients per Mixture of send mail, read mail, and process mail tasks derived from the very heavy user profile of Microsoft Exchange Load Generator 2010 A Zimbra document 4 describes more details about the predecessor of Zimbra Loadgen, called Soapgen, which was improved to remove various limitations of Soapgen. Hardware Configuration Figure 2 shows the hardware configuration for the Zimbra Collaboration Server experiments. Client Driver Dell PowerEdge R710 Zimbra Mail Server/vSphere 5.0 (SUT) Dell PowerEdge R910 Gigabit Ethernet Switch Storage EMC CLARiiON CX3-40 Figure 2. Hardware Schematic TECHNICAL WHITE PAPER / 5

6 Server Hardware Description Dell PowerEdge R910 Quad-socket eight-core server 2.27GHz Intel Xeon X7350 (Nehalem-EX) processors with hyperthreading disabled 256GB DDR3 DIMM memory Table 1. Mail Server Hardware (SUT) Dell PowerEdge R710 Dual-socket quad-core server 2.93GHz Intel Xeon X5570 (Nehalem) processors with hyperthreading disabled 96GB DDR3 DIMM memory Table 2. Client Hardware Storage Hardware and Configuration EMC CLARiiON CX3-40 array 180 x 15,000-RPM disk drives for 8-vCPU configuration Sample storage configurations: Single 8-vCPU VM Multi-VM 4-vCPU configuration Zimbra message store 27 RAID 0 LUNs (LUNs were larger than single VM): 7 RAID 0 LUNs Zimbra index store 8 RAID 0 LUNs 4 RAID 0 LUNs Zimbra database store 1 RAID 0 LUN 1 RAID 0 LUN LUN striping 5 disk stripe 5 disk stripe Table 3. Storage Hardware Software Configuration Software Versions The operating system (OS) software and benchmark driver programs were identical in both native and virtual environments. VMware vsphere 5.0 ESXi 5.0 build Operating system (guest and native) Zimbra Collaboration Server version Microsoft Exchange Server version RHEL6 GA + kernel update: el6.x86_ _GA_3058.RHEL6_64_ RHEL6_64 NETWORK edition Microsoft Exchange Server 2010 version Table 4. Software Versions TECHNICAL WHITE PAPER / 6

7 Methodology Single Virtual Machine Scale-Up Tests The tests include native and virtual experiments with four and eight vcpus. For the native experiments, the maxcpus and mem Linux boot time options were used to specify the number of processors and memory at boot time. Zimbra Loadgen 4 was used to provision and test 1,000 users per vcpu. The CPU utilization and Sendmail latencies were then used as a metric to measure performance. A key differentiator in the benchmarking of Zimbra Collaboration Server and Exchange Server 2010 is enabling search indexing for messages, which is known to be a resource-intensive operation and especially expensive for Exchange Server Therefore, almost all of the Exchange benchmarking efforts have this feature disabled. In Zimbra Collaboration Server, however, indexing by Lucene is very efficient, and enabled by default. Hence, in the experiments, indexing is enabled for Zimbra Collaboration Server. This contrasts the cost of indexing in Zimbra when compared to that of Exchange without indexing. Memory Sizing The following are some of the memory tuning options used in the tests: 8GB per CPU for virtual and native experiments 30% memory for the JVM heap; 20% for the InnoDB (MySQL) buffer pool This leaves the rest of the memory for the file system page cache Enable large pages for JVM and MySQL InnoDB Kernel settings: huge pages in /etc/sysctl.conf and /etc/security/limits.conf Zimbra configuration files for setting large pages in JVM and mysql.cnf file for MySQL Use CMS garbage collector and use 25% of the heap for young generation (which stores transient objects for immediate garbage collection) Storage Sizing Because the experiments used 250MB mailboxes per user and users were scaled to 8,000 on an 8-CPU configuration, the storage requirements were quite large. As an example, here are the capacities of the various mail stores in the 8-CPU experiments. All stores are maintained on Linux ext3 file systems. Message store: 2TB Database data store: 61GB Lucene index store: 567GB Zimbra Collaboration Server is a mature, robust product and its documentation provides guidelines for tuning the memory, file system, and other options 2. Experiments used appropriate tunables from these guidelines. Multiple Virtual Machine Scale-Out Tests Experiments scaled out Zimbra Collaboration Server from one to eight 4-vCPU virtual machines. Each 4-vCPU virtual machine was configured using properties similar to those listed for single virtual machine tests. One notable change was that the memory size of each virtual machine was changed to 26GB in order to accommodate eight virtual machines on the physical host without memory overcommittment. Table 5 summarizes some of the properties when running eight Zimbra Collaboration Server virtual machines. Number of VMs Number of users Total number of vcpus Total memory Total u s e d s pac e Total number of disk spindles 8 32, GB 10TB 165 Table 5. Properties of Eight Zimbra Virtual Machines TECHNICAL WHITE PAPER / 7

8 As Table 5 shows, the configuration to set up and run 32,000 users on eight virtual machines was complex. The storage was configured to isolate indices and message stores on separate spindles within a virtual machine, but due to a limitation on the number of spindles, these were shared across virtual machines. Performance Results Virtual Machine Scale-Up Results Table 6 summarizes the results of the single virtual machine scale-up tests. Number of vcpus Memory size (GB) Number o f Users AVERAG E S e n d mai l l atency (ms) C PU utilization (usr/sys/ wait) Index write IOPs , /3.4/ , /3.2/ , /6.6/ Table 6. Single Virtual Machine Scale-Up Results Zimbra demonstrates excellent scaling going from four to eight vcpus, with mail latencies not changing much and CPU utilization also remaining steady. The 4-vCPU experiment with 8,000 users demonstrates that Zimbra Collaboration Server can easily support twice the number of users with 50% CPU to spare and no large increase in mail latencies. The Lucene index store generates the largest number of write IOPs while computing the index for new messages, so it is a measured metric. Virtual Machine Comparison with Native Environment Table 7 summarizes the performance of Zimbra Collaboration Server in a virtual machine and in a native environment. CPUs Users Virtual/ Native AVERAGE S e n d mai l l atency (ms) C PU utilization (usr/sys/ wait) 4 4,000 Virtual /3.4/ ,000 Native /3.8/11 8 8,000 Virtual /3.2/ ,000 Native /3.4/9.7 Table 7. Single Virtual Machine Comparison with Native Environment Zimbra Collaboration Server performs with results very close to those in a native environment in the 4- and 8-vCPU configurations. Furthermore, Zimbra Collaboration Server runs in a JVM. For the 8-vCPU configuration, the Java heap was 19.6GB. Even with this large heap, no performance issues were observed within Zimbra Collaboration Server due to garbage collection, and it performed as well in a virtual machine as on a native machine. TECHNICAL WHITE PAPER / 8

9 Performance Optimizations Some performance optimizations are introduced in Zimbra Collaboration Server 7.x: In Zimbra Collaboration Server 6.0.2, CPU fluctuations affected performance at lower vcpu counts. In 7.x, the average CPU utilization during the Zimbra Loadgen test dropped from 28% to 18% in the 4-vCPU virtual machine. Sendmail latency showed a 4x improvement over Zimbra Collaboration Server when using a 2-vCPU, RHEL 5 virtual machine. This change brought Sendmail latency to a little over 80% of a native environment, as shown in Table 8 (0.82 entry). Moving from RHEL 5 to RHEL 6 with the tickless timer kernel implementation 5 gives a 16% improvement in the performance of 8-vCPU Zimbra Collaboration Server virtual machines, bringing virtual performance very close to a native environment, up to 97%, as shown in Table 8 (0.97 entry). vcpus users R H E L version S e n d mai l l atency (derived from ratio o f vi rtual to native) AVERAGE S e n d mai l l atency (ms) 8 8,000 RHEL ,000 RHEL Table 8. RHEL 6 Improvements Virtual Machine Scale-Out Results Figure 3 details the scalability of Zimbra Collaboration Server as up to eight virtual machines are added to the host Zimbra Users Average Sendmail Latencies Zimbra Mailbox Users Average Sendmail Latencies in ms vCPU Virtual Machines 0.0 Figure 3. Zimbra Collaboration Server Scale-Out Results As Figure 3 shows, Zimbra mail server demonstrates excellent scalability, with up to 32,000 users in eight virtual machines. The Sendmail latency, which is the primary metric used in the tests, increases by only 10% when scaling from a single 4-vCPU virtual machine to eight 4-vCPU virtual machines. The total host CPU utilization was 21%. The number of users per virtual machine can be doubled, with over 50% of the CPU capacity still free. TECHNICAL WHITE PAPER / 9

10 Comparison with Microsoft Exchange Server 2010 Microsoft Exchange Server is an industry leader in mail-server deployments, and is also a very popular application for vsphere customers. Comparison testing of virtualized instances of Zimbra Collaboration Server and Exchange Server 2010 provides insight for competitive product development. Two different load generators are used to simulate users sending one from VMware/Zimbra for Zimbra Collaboration Server (Loadgen) and one from Microsoft for Exchange Server 2010 (LoadGen). In order to fairly compare Zimbra and Exchange using these distinct workloads, the test designer matched the mail and run-time profiles so that each product was stressed the same way. See Workload Description in this paper for details of the profile. Figure 4 and Table 9 demonstrate the CPU utilization as well as latencies during 4-vCPU experiments with Exchange Server 2010 and Zimbra Collaboration Server. 60 % CPU Utilization Zimbra MTA+LDAP/Exchange 2010 CAS+HUB Zimbra Database/Exchange 2010 Indexing Mailbox 10 0 Zimbra Mailbox Server Exchange 2010 Figure 4. Comparison of Zimbra and Exchange CPU Utilization vcpus Users Exchange/ Zimbra mai l servers AVERAGE S e n d mai l l atency (ms) Total C PU utilization (mai lbox, D B, MTA, LDAP) Total write IOPs 4 4,000 Zimbra % 1, ,000 Exchange 69 53% 500 Table 9. Comparison of Zimbra and Exchange As Figure 4 demonstrates, Exchange CPU utilization is significantly greater than that of Zimbra Collaboration Server. The mail server alone is 27%, more than the combined utilization of mailbox, database, MTA, and LDAP. Indexing adds an additional 9%, as well as a 17% utilization in CAS/HUB. The results demonstrate that Zimbra Collaboration Server is much more efficient than Exchange Server 2010, even though Zimbra demonstrates higher latencies. The Zimbra latency of 112.5, however, is well below the guideline of 500 milliseconds for an acceptable user experience. During testing, it took five days to provision mailboxes (with indexing enabled) in Exchange LoadGen, while it took only 12 hours for Zimbra Loadgen. TECHNICAL WHITE PAPER / 10

11 Conclusion Zimbra Collaboration Server is a robust, well-architected, and scalable addition to VMware s suite of products. The tests in this performance study show that Zimbra Collaboration Server demonstrates the following: Near-native performance and scalability on vsphere 5.0 Excellent scale-out performance going from one to eight virtual machines on a single host Much better CPU efficiency and ease of provisioning than Microsoft Exchange Server 2010 References 1. Herrod, Steve. VMware to Acquire Zimbra, The Console Blog. VMware, Performance Tuning Guidelines for Large Deployments, The Zimbra Wiki. VMware, Exchange Load Generator 2010, Microsoft Download Center. Microsoft, ccfc915b29ef 4. Zimbra Collaboration Suite Scalability & Performance Benchmarks for Service Providers. Zimbra, Tickless Kernel, Red Hat Enterprise Linux 6 Power Management Guide, Edition 1.0. Red Hat, Tickless-kernel.html VMware, Inc Hillview Avenue Palo Alto CA USA Tel Fax Copyright 2011 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Item No: VMW-WP-ZIMBRA-USLET-101