BlackBerry Enterprise Server for Microsoft Exchange

Transcription

1 BlackBerry Enterprise Server for Microsoft Exchange Version 4.1 Performance Benchmarking

2 BlackBerry Enterprise Server Version 4.1 for Microsoft Exchange Performance Benchmarking Last modified: 14 August 2006 Part number: Version 7 At the time of publication, this documentation is based on BlackBerry Enterprise Server Version 4.1 for Microsoft Exchange Research In Motion Limited. All Rights Reserved. The BlackBerry and RIM families of related marks, images, and symbols are the exclusive properties of Research In Motion Limited. RIM, Research In Motion, BlackBerry, Always On, Always Connected and the envelope in motion symbol are registered with the U.S. Patent and Trademark Office and may be pending or registered in other countries. Compaq is a registered trademark of Hewlett-Packard Development Company, L.P. IBM, Lotus, and Sametime are registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Intel and Pentium are registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Excel, Outlook, PowerPoint, and Windows are registered trademarks of Microsoft Corporation in the United States and/or other countries. Novell and GroupWise are registered trademarks of Novell, Inc. in the United States and other countries. Java, J2RE, and J2SE are trademarks of Sun Microsystems, Inc. in the United States or other countries. All other brands, product names, company names, trademarks and service marks are the properties of their respective owners. The BlackBerry device and/or associated software are protected by copyright, international treaties, and various patents, including one or more of the following U.S. patents: 6,278,442; 6,271,605; 6,219,694; 6,075,470; 6,073,318; D445,428; D433,460; D416,256. Other patents are registered or pending in various countries around the world. Visit for a list of RIM [as hereinafter defined] patents. This document is provided as is and Research In Motion Limited and its affiliated companies ( RIM ) assume no responsibility for any typographical, technical, or other inaccuracies in this document. In order to protect RIM proprietary and confidential information and/or trade secrets, this document may describe some aspects of RIM technology in generalized terms. RIM reserves the right to periodically change information that is contained in this document; however, RIM makes no commitment to provide any such changes, updates, enhancements, or other additions to this document to you in a timely manner or at all. RIM MAKES NO REPRESENTATIONS, WARRANTIES, CONDITIONS, OR COVENANTS, EITHER EXPRESS OR IMPLIED (INCLUDING WITHOUT LIMITATION, ANY EXPRESS OR IMPLIED WARRANTIES OR CONDITIONS OF FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, MERCHANTABILITY, DURABILITY, TITLE, OR RELATED TO THE PERFORMANCE OR NON-PERFORMANCE OF ANY SOFTWARE REFERENCED HEREIN OR PERFORMANCE OF ANY SERVICES REFERENCED HEREIN). IN CONNECTION WITH YOUR USE OF THIS DOCUMENTATION, NEITHER RIM NOR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, OR CONSULTANTS SHALL BE LIABLE TO YOU FOR ANY DAMAGES WHATSOEVER BE THEY DIRECT, ECONOMIC, COMMERCIAL, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, OR INDIRECT DAMAGES, EVEN IF RIM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, INCLUDING WITHOUT LIMITATION, LOSS OF BUSINESS REVENUE OR EARNINGS, LOST DATA, DAMAGES CAUSED BY DELAYS, LOST PROFITS, OR A FAILURE TO REALIZE EXPECTED SAVINGS. This document might contain references to third-party sources of information, hardware or software, products or services and/or third-party web sites (collectively the Third-Party Information ). RIM does not control, and is not responsible for, any Third-Party Information, including, without limitation the content, accuracy, copyright compliance, compatibility, performance, trustworthiness, legality, decency, links, or any other aspect of Third-Party Information. The inclusion of Third-Party Information in this document does not imply endorsement by RIM of the Third-Party Information or the third-party in any way. Installation and use of Third-Party Information with RIM's products and services may require one or more patent, trademark, or copyright licenses in order to avoid infringement of the intellectual property rights of others. Any dealings with Third-Party Information, including, without limitation, compliance with applicable licenses and terms and conditions, are solely between you and the third-party. You are solely responsible for determining whether such third-party licenses are required and are responsible for acquiring any such licenses relating to Third-Party Information. To the extent that such intellectual property licenses may be required, RIM expressly recommends that you do not install or use Third-Party Information until all such applicable licenses have been acquired by you or on your behalf. Your use of Third-Party Information shall be governed by and subject to you agreeing to the terms of the Third-Party Information licenses. Any Third-Party Information that is provided with RIM's products and services is provided as is. RIM makes no representation, warranty or guarantee whatsoever in relation to the Third-Party Information and RIM assumes no liability whatsoever in relation to the Third- Party Information even if RIM has been advised of the possibility of such damages or can anticipate such damages.

3 Research In Motion Limited 295 Phillip Street Waterloo, ON N2L 3W8 Canada Research In Motion UK Limited Centrum House, 36 Station Road Egham, Surrey TW20 9LF United Kingdom Published in Canada

4

5 Contents 1 Overview...9 About this document... 9 Audience BlackBerry Enterprise Server components BlackBerry Dispatcher...11 BlackBerry Messaging Agent...11 MAPI...12 CDO...12 BlackBerry Router...12 BlackBerry Controller...12 BlackBerry MDS Services...12 BlackBerry MDS Connection Service...13 BlackBerry Attachment Service...13 BlackBerry Synchronization Service...14 BlackBerry Policy Service...14 BlackBerry Collaboration Service BlackBerry Enterprise Server implementation considerations...15 Proximity to messaging and collaboration servers...15 BlackBerry Desktop Software connection...15 Internet connection...15 BlackBerry MDS Connection Service traffic...16 BlackBerry Attachment Service traffic...16 Wireless message reconciliation...16 BlackBerry Configuration Database...16 Wireless enterprise activation Test environment Components...19 Microsoft Exchange Server equipment...19 BlackBerry Enterprise Server equipment...20 Microsoft SQL Server 2000 equipment...20

6 Domain controller equipment...20 Load generator computer equipment...21 Performance testing tools...21 Microsoft Load Simulator BlackBerry MDS Connection Service Push Load Testing tool BlackBerry Synchronization Service Load Testing tool Measurement counters Lab initialization Performance tests...23 Overview of the performance tests Microsoft Exchange 2003 performance test results Overview of the test results Processor use Memory use...29 Disk use...29 Network use...31 Performance impact considerations Microsoft SQL Server 2000 performance test results...33 Overview of the test results Processor use Disk use Transactions/sec...36 User connection count Summary Performance impact considerations BlackBerry Enterprise Server performance test results...41 Overview of the performance test results...41 Processor use Memory use Messages queued for delivery...44 Messages received per minute...44 Network use...45 Performance impact considerations...46

7 9 Conclusions Conclusions and recommendations...49 A Appendix: MMB and MMB2 comparison...51 Overview...51 BlackBerry Attachment Service traffic for MMB B Appendix: BlackBerry MDS Connection Service traffic rate derivation...57 Message test load traffic BlackBerry MDS Connection Service load traffic C Appendix: Calculations and data Sample calculations...59

8

9 1 Overview About this document About this document This document describes the performance characteristics of the BlackBerry Enterprise Server Version 4.1 for Microsoft Exchange. It presents the results of performance testing on the BlackBerry Enterprise Server and describes the impact of the BlackBerry Enterprise Server on a Microsoft Exchange 2003 Server and a Microsoft SQL 2000 Server. Finally, this document includes important information for system administrators who are responsible for implementing the BlackBerry Enterprise Solution in a Microsoft Exchange environment. The performance tests that this document describes are based on the ability of the BlackBerry Enterprise Server to support 2000 user accounts. Each test was run for four hours. Research In Motion (RIM) used the data collected in the last two hours of the test, when the load driver reached a steady state load, for analysis. The BlackBerry Enterprise Server performance was measured using Messaging Application Programming Interface (MAPI) Messaging Benchmark 2 (MMB2), a standard developed by Compaq Computer Corporation (now Hewlett-Packard Development Company) and Microsoft to compare Microsoft Exchange Server benchmarks based on the MAPI protocol with those of other servers. See Appendix: MMB and MMB2 comparison on page 51 for more information about the MMB2 workload. As part of the performance testing, the impact on the BlackBerry Enterprise Server performance was measured when the following features and services were enabled: , calendar, and sent folder redirection wireless message reconciliation BlackBerry MDS Connection Service BlackBerry Attachment Service BlackBerry Synchronization Service The performance testing included both incoming and outgoing traffic to the BlackBerry Enterprise Server. The incoming traffic included pull traffic for the BlackBerry MDS Connection Service and synchronization commands sent from BlackBerry devices to the BlackBerry Synchronization Service.

10 BlackBerry Enterprise Server for Microsoft ExchangePerformance Benchmarking Audience This document assumes that you have a working knowledge of the following items: BlackBerry Enterprise Server for Microsoft Exchange installation, configuration, administration, and architecture Microsoft Exchange Server 2000 and, or 2003 installation and administration Microsoft SQL Server 2000 installation and administration System Monitor counters and performance metrics 10

11 2 BlackBerry Enterprise Server components BlackBerry Dispatcher BlackBerry Messaging Agent BlackBerry Router BlackBerry Controller BlackBerry MDS Services BlackBerry MDS Connection Service BlackBerry Attachment Service BlackBerry Synchronization Service BlackBerry Policy Service BlackBerry Collaboration Service BlackBerry Dispatcher The BlackBerry Dispatcher handles traffic to the BlackBerry Infrastructure. It compresses/decompresses and encrypts/decrypts wireless data. The BlackBerry Dispatcher handles all internal Service Routing Protocol (SRP) connections from the BlackBerry Messaging Agent and also manages communication from the BlackBerry MDS Connection Service, the BlackBerry Synchronization Service, the BlackBerry Policy Service, and the BlackBerry Collaboration Service to the BlackBerry Infrastructure. These components connect to the BlackBerry Dispatcher through specific ports and communicate with the BlackBerry Infrastructure through the BlackBerry Router using a unique SRP identifier that the BlackBerry Dispatcher establishes. BlackBerry Messaging Agent The BlackBerry Messaging Agent handles BlackBerry Synchronization Service traffic, including contacts, tasks, memos, message settings, and message filters. The BlackBerry Messaging Agent also has a mechanism for wireless calendar initialization before wireless calendar synchronization is enabled. This mechanism is used to initially synchronize the BlackBerry device and the Microsoft Exchange message store, enabling wireless synchronization to occur. Each BlackBerry Messaging Agent owns a unique internal SRP identifier that connects to the BlackBerry Dispatcher. The BlackBerry Messaging Agent also handles all Microsoft Exchange-related traffic using Messaging Application Programming Interface (MAPI) and Collaboration Data Objects (CDO).

12 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking MAPI The BlackBerry Messaging Agent uses thread pools that have a MAPI session to the Microsoft Exchange Server. New thread pools are created when an existing thread pool exceeds the 50-mailbox maximum or the BlackBerry Enterprise Server connects to another Microsoft Exchange Server. This thread pool system minimizes the number of MAPI sessions to the Microsoft Exchange Server, which provides added product stability. CDO The BlackBerry Messaging Agent uses CDO sessions for wireless calendar activity. A CDO session is established for a user when a calendar-related request is made. If no activity occurs within five minutes, the session ends. BlackBerry Router The BlackBerry Router acts as a gateway to the BlackBerry Infrastructure and user computers. It determines whether data should be transmitted wirelessly or routed through a network connection. When a BlackBerry device is connected to a computer, the BlackBerry Router routes data to the BlackBerry device and bypasses the wireless network. BlackBerry Controller The BlackBerry Controller starts the BlackBerry Messaging Agents and monitors their health. If the BlackBerry Controller detects nonresponsive threads or if a BlackBerry Messaging Agent stops responding, the BlackBerry Controller restarts the BlackBerry Messaging Agent. The BlackBerry Controller assigns BlackBerry user accounts to BlackBerry Messaging Agents based on the following criteria: Microsoft Exchange Server on which the user accounts reside (groups user accounts together or, if most user accounts reside on the same Microsoft Exchange Server, distributes the user accounts evenly) amount of user accounts that currently reside on each BlackBerry Messaging Agent (up to 500 users) The BlackBerry Controller also monitors all BlackBerry processes and restarts them if they are not in a running state. BlackBerry MDS Services In BlackBerry Enterprise Server Version 4.1, RIM introduced a new component called the BlackBerry MDS Services. The BlackBerry MDS Services also require the BlackBerry MDS Connection Service to send data to BlackBerry devices. The BlackBerry MDS Services are responsible for managing interactions and requests between BlackBerry devices and enterprise applications. The BlackBerry MDS Services include the following services: BlackBerry MDS Management Service: deals with policies, such as those that specify which applications users can download, the services available to applications, and so on BlackBerry MDS Provisioning Service: controls and manages which applications users can download to BlackBerry devices 12

13 2: BlackBerry Enterprise Server components BlackBerry MDS Data Optimization Service: transforms data for efficient wireless transmission and use on BlackBerry devices BlackBerry MDS Connection Service: provides TCP/IP and HTTP-based connectivity between BlackBerry MDS Studio Java Applications and enterprise applications and between BlackBerry MDS Browser Applications and enterprise applications BlackBerry MDS Application Integration Service: supports the integration and transmission of data between BlackBerry MDS Studio on the BlackBerry device and web services applications BlackBerry MDS Studio Application Repository: manages published BlackBerry MDS Studio Applications centrally The current version of this document includes benchmarking information for the BlackBerry MDS Connection Service. The document does not include benchmarking information for the BlackBerry MDS Services. BlackBerry MDS Connection Service The BlackBerry MDS Connection Service enables push-based access to enterprise data and applications. Capitalizing on the BlackBerry Enterprise Server architecture, the BlackBerry MDS Connection Service provides a safe connection between the BlackBerry device and corporate application servers. Through this connection, users can access corporate data from enterprise applications. The data travels between the BlackBerry device and the BlackBerry Enterprise Server using the same path as the BlackBerry Collaboration Service, so no extra connections or firewall openings are required. The BlackBerry MDS Connection Service also communicates with the BlackBerry Dispatcher. It permits persistent socket connections from the BlackBerry device to the corporate application server. You can configure the BlackBerry MDS Connection Service to run on a computer that is remote from the BlackBerry Enterprise Server computer. In BlackBerry Enterprise Server Version 4.0.x, the BlackBerry MDS Connection Service was called the BlackBerry Mobile Data Service. RIM benchmarked the BlackBerry MDS Connection Service using the same load characteristics used to benchmark the BlackBerry Mobile Data Service in Version 4.0.x, but with different content. BlackBerry Attachment Service The BlackBerry Attachment Service enables users to open and view message attachments on their BlackBerry devices. Attachment content is formatted and delivered to the BlackBerry device using the Universal Content Stream (UCS) format. When a user requests to view an attachment on the BlackBerry device, the BlackBerry Messaging Agent sends the request information to the BlackBerry Attachment Service, and the BlackBerry Attachment Service performs the conversion of the attachment content to UCS format. The BlackBerry Enterprise Server compresses and encrypts the attachment data and then sends the formatted attachment to the BlackBerry device. The BlackBerry Attachment Service also enables users to view Microsoft PowerPoint presentations in a slide format. Visit to read the BlackBerry Attachment Service guide for more information. 13

14 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking BlackBerry Synchronization Service The BlackBerry Synchronization Service synchronizes contacts, tasks, and memos between the application on a user s computer and the user s BlackBerry device using Microsoft Exchange through the BlackBerry Messaging Agent. The BlackBerry Synchronization Service also synchronizes message settings and message filters. The wireless protocol also synchronizes a variety of database configurations, backs up databases on the BlackBerry device, and synchronizes items when necessary. Initialization of the BlackBerry Synchronization Service is triggered when items on the BlackBerry device and the BlackBerry Configuration Database are not synchronized. This initialization can be triggered by activating a BlackBerry device wirelessly or moving user accounts. The initialization mechanism is also used to initially synchronize the BlackBerry device and message store, enabling wireless synchronization to occur. Because the initialization feature is not considered to maintain a steady state load on the BlackBerry Enterprise Server, its impact on the BlackBerry Enterprise Server performance is not included in this document. There are load conditions that the initialization places on the system that you should consider when initialization occurs. BlackBerry Policy Service The BlackBerry Policy Service pushes IT policies and IT administrative commands to BlackBerry devices wirelessly. It is also responsible for creating and pushing service books to BlackBerry devices that are being activated wirelessly and for sending third-party applications to BlackBerry devices wirelessly. Because the BlackBerry Policy Service is not considered to maintain a steady state load on the BlackBerry Enterprise Server, its impact on the BlackBerry Enterprise Server performance is not included in this document. BlackBerry Collaboration Service The BlackBerry Collaboration Service is designed to keep users in touch with their instant messaging community when they are connected to the wireless network. The BlackBerry Collaboration Service includes contact list management, presence awareness, notifications, and emoticons. It also supports the IBM Lotus Sametime, Microsoft Live Communications Server 2005, and Novell GroupWise platforms. This document does not include benchmarking information for the BlackBerry Collaboration Service. 14

15 BlackBerry Enterprise Server implementation considerations Proximity to messaging and collaboration servers BlackBerry Desktop Software connection Internet connection BlackBerry MDS Connection Service traffic BlackBerry Attachment Service traffic Wireless message reconciliation BlackBerry Configuration Database Wireless enterprise activation Proximity to messaging and collaboration servers The BlackBerry Enterprise Server should have access to messaging and collaboration servers through a local, high-speed, switched network connection. For a decentralized server topology, consider the physical link speeds between the sites and the geographical placement of BlackBerry users. RIM recommends that you place the BlackBerry Enterprise Server as close to the messaging servers as possible and target the messaging servers that host the majority of BlackBerry users. Depending on the WAN infrastructure, you might want to implement a BlackBerry Enterprise Server in more than one site for optimal performance. BlackBerry Desktop Software connection If you use the BlackBerry Desktop Software, a network connection to the BlackBerry Enterprise Server and the messaging server must be available. Internet connection The BlackBerry Enterprise Server requires an outbound-initiated Internet connection on port Any firewall that is placed between the BlackBerry Enterprise Server and the Internet must allow this connection. You must also verify that you have enough Internet bandwidth to manage the additional traffic that the BlackBerry Enterprise Server generates. Messages are sent to BlackBerry devices in 2-KB sections; however, because of compression, only 1 KB goes through the Internet connection. 3

16 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking BlackBerry MDS Connection Service traffic When implementing the BlackBerry MDS Connection Service, consider both the rate at which data is sent to BlackBerry devices and the amount of transcoding that the data requires. Do not push too much data that requires transcoding to BlackBerry devices when messaging traffic is heavy; doing so might cause message delivery to slow. If you often push data that requires transcoding, the system might experience processor bottlenecks. There is less control over the rate at which pages are pulled, so you should control the rate at which content is pushed. If you plan to push a large amount of data that requires transcoding, consider installing the BlackBerry MDS Connection Service on a computer that is remote from the BlackBerry Enterprise Server computer. BlackBerry Attachment Service traffic Potential issues processing attachments might occur in the following situations: when processing concurrent requests when processing large attachments when processing complex attachments (for example,.pdf files or ASCII text) Visit to read article TAE for more information. Wireless message reconciliation When there are pending changes on a BlackBerry device that require immediate reconciliation, users can reconcile messages from the BlackBerry device manually using the Reconcile Now option in the BlackBerry Desktop Software. If users use the option excessively, a significant increase in network traffic might occur. BlackBerry Configuration Database The BlackBerry Enterprise Server Version 4.1 requires either Microsoft Desktop Engine (MSDE) 2000, Microsoft SQL Server 2005 Express Edition, or Microsoft SQL Server There are two limitations that prevent MSDE 2000 from scaling: The workload governor is activated if more than eight concurrent requests are made. If you use MSDE 2000 with a BlackBerry Enterprise Server that is configured for 2000 user accounts, the BlackBerry Enterprise Server limits the number of BlackBerry Messaging Agents to two and minimizes the number of connections to the BlackBerry Configuration Database. These limitations do not guarantee that the workload governor will not be impacted. The risk in running a large number of user accounts on two BlackBerry Messaging Agents is that there are more user accounts for each BlackBerry Messaging Agent to handle, which might result in longer start-up times and more non-responsive threads caused by MAPI or Microsoft Exchange issues. The maximum size of a database cannot exceed 2 GB. If users have 16-MB BlackBerry devices, database memory should not exceed 1 MB for each user account. However, if users have 32-MB BlackBerry devices, the database memory use is greater. 16

17 3: BlackBerry Enterprise Server implementation considerations If you are enabling more than 500 BlackBerry user accounts, you should use Microsoft SQL Server If you enable a large number of user accounts with MSDE 2000, monitor the Event Viewer for warnings that indicate that the workload governor has been activated. There might also be an event from the BlackBerry Enterprise Server warning that MSDE is approaching its 2-GB database size limit. These warnings indicate that you should upgrade from MSDE 2000 to Microsoft SQL Server RIM conducted the benchmarking of the BlackBerry Configuration Database using Microsoft SQL Server RIM has not evaluated the limitations of Microsoft SQL Server 2005 Express Edition at this time. Microsoft SQL Server 2000 is the updated equivalent to MSDE Wireless enterprise activation The wireless enterprise activation feature eliminates the need for users to synchronize their BlackBerry devices with their computers using the BlackBerry Desktop Software. Although RIM did not load test this feature, you should consider how many BlackBerry devices (running BlackBerry Device Software Version 4.0 and later) you want to activate wirelessly at one time. Activating BlackBerry devices wirelessly involves pushing content from a user's mailbox to a BlackBerry device that is not connected to a computer. A large amount of data is sent through the BlackBerry Enterprise Server to the BlackBerry device, which results in increased wireless traffic, increased activity by the BlackBerry Enterprise Server, and increased activity by the messaging and collaboration servers. Stagger the wireless activation of BlackBerry devices so that the load on the BlackBerry Enterprise Server does not impact message delivery for existing users. 17

18 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking 18

19 4 Test environment Components Performance testing tools Measurement counters Lab initialization Components The following tables provide information about the various components that RIM used in the performance test environment. In some cases, these components are greater than the minimum requirements established for the BlackBerry Enterprise Server Version 4.1 for Microsoft Exchange. Microsoft Exchange Server equipment Type hardware Components 2 Compaq DL380 - G3 servers with the following features: Dual PIV GHz processor (Hyper Threading turned off) 2560 MB memory HP NC7781 Gigabit Server Adapter 6 36-GB [15,000 rpm] drives in a RAID 0 array with Smart Array 5312 Controller software Microsoft Windows 2003 Service Pack 1 Microsoft Exchange 2003 Server SP 1 configuration Microsoft Exchange 2003 SP 1 (default options with circular logging enabled) installed on the Microsoft Exchange Servers; each Microsoft Exchange Server had one information store, which was mounted on the 6-drive RAID 0 set

20 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking BlackBerry Enterprise Server equipment Type hardware Components 1 Compaq DL360 G3 server with the following features: Dual PIV 2.8 GHz processors (Hyper Threading disabled) 3920 MB memory HP NC7781 Gigabit Server Adapter 2 36-GB RAID 0 drives with Smart Array 5i Controller software Windows Server 2003 SP 1 Microsoft Exchange 2003 System Manager SP 1 BlackBerry Enterprise Server Version 4.1 for Microsoft Exchange Microsoft XML Version 4.0 SP 2 (MSXML) Java 2 Runtime Environment, SE 1.4.2_03 configuration The BlackBerry Enterprise Server was configured with one administrative service account. The service account was given the following permissions for each Microsoft Exchange Server information store: Send As Receive As Administer Information Store Microsoft SQL Server 2000 equipment Type hardware Components 1 Compaq DL360 G4 server with the following features: PIV 3.0 GHz processor 1024 MB memory NC3163 Fast Ethernet NIC 10/100 LAN 2 72-GB RAID 1+0 drives with Integrated Smart Array Controller software Windows Server 2003 SP 1 Microsoft SQL Server 2000 (Enterprise Edition) Domain controller equipment Type hardware Components 1 Compaq DL360-G3 server with the following features: PIV 2.8 GHz processor 1024 MB memory HP NC7782 Gigabit Server Adapter 2 36-GB RAID 1+0 drives with Integrated Smart Array Controller software Windows Server 2003 SP 1 20

21 4: Test environment Load generator computer equipment Type hardware Components 4 Compaq DL360-G3 servers with the following features: 1024 MB memory PIV 2.8 GHz processors HP NC7782 Gigabit Server Adapter 2 36-GB RAID 1+0 drives with Integrated Smart Array Controller software Windows Server 2003 SP 1 Microsoft Outlook 2003 Testing tools Microsoft Loadsim 2003 BlackBerry MDS Connection Service Push Load Testing Tool BlackBerry Synchronization Service Load Testing Tool Performance testing tools Performance testing was performed in a single, defined environment using the following testing tools: Microsoft Exchange Server 2003 Load Simulator (LoadSim) BlackBerry MDS Connection Service Push Load Testing tool BlackBerry Synchronization Service Load Testing tool Microsoft Load Simulator 2003 In their performance tests, RIM used the MAPI Messaging Benchmark 2 (MMB2), which is a workload defined in LoadSim 2000 and has been used in previous benchmarking for the BlackBerry Enterprise Server Version 4.0. RIM used LoadSim 2003 because they selected Microsoft Exchange 2003 for the benchmarking of BlackBerry Enterprise Server 4.1. LoadSim 2003 has a new workload defined that is derived from the MMB2 workload. It contains additional features that exercise new features in Microsoft Exchange To permit customers to compare previous benchmarking data to the data in this document, and because RIM does not use the new features in LoadSim 2003, RIM decided to configure Loadsim 2003 with a MMB2 workload. LoadSim 2003 created 2000 Microsoft Exchange user accounts (1000 user accounts for each Microsoft Exchange Server) because this type of testing required a fixed baseline load of 2000 MMB2 user accounts. LoadSim 2003 generated the MMB2 load and created 10 folders as messages were filed for each user account. The 10 folders were used during the MMB2 workload to generate the load for wireless message reconciliation. The base MMB2 workload also included attachments. Visit for more information about LoadSim

22 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking BlackBerry MDS Connection Service Push Load Testing tool RIM developed the BlackBerry MDS Connection Service Push Load Testing tool to benchmark the BlackBerry MDS Connection Service. In past benchmarking, RIM used a tool from Microsoft called Web Application Stress Tool. RIM determined that the tool did not have the capabilities to request for notification of a push or to enable transcoding of the content. Using the BlackBerry MDS Connection Service Push Load Testing tool, RIM can specify any URL to push, wait for notification, and enable transcoding. The new tool enables more features of the BlackBerry MDS Connections Service to be exercised. Requests by the BlackBerry MDS Connection Service Push Load Testing tool were sent to unique SMTP addresses. Each request was configured to request transcoding and to wait for notifications. The performance test was configured to send a push request packet every 4.2 minutes for each user account. BlackBerry Synchronization Service Load Testing tool RIM developed the BlackBerry Synchronization Service Load Testing tool to generate Microsoft Exchange side traffic for contacts, tasks, and memos. The tool generates a specified amount of load for each item. It adds, modifies, and deletes a specified number of contacts, tasks, and memos over a specified period of time for a specified number of users. The tool populates all relevant fields with data that the BlackBerry Synchronization Service synchronizes, including recurring tasks. The purpose of the tool is to generate a controlled amount of load from Microsoft Exchange for these types of items. Measurement counters RIM measured performance using the System Monitor on a separate computer. The System Monitor monitored the following items: processor time (%processor time) disk use (disk transfer/sec, disk reads/sec, disk writes/sec, %disk time) network use (total bytes/sec) memory use (committed bytes, pages/sec) BlackBerry counters (messages queued for delivery, messages received/min) Lab initialization For the performance tests, user accounts were added to the BlackBerry Enterprise Server in 500-user increments up to the 2000 user account limit. Two Microsoft Exchange Servers were used for the performance testing, and user accounts were evenly distributed across those servers. For example, a test with 500 BlackBerry user accounts had 250 BlackBerry user accounts enabled on each Microsoft Exchange Server. Throughout the performance testing, there were 1000 user accounts configured on each Microsoft Exchange Server on which MMB2 load was applied. To reset the tests to the correct number of BlackBerry user accounts with no message items quickly, RIM restored the information store and the Microsoft SQL database from backup files. 22

23 5 Performance tests Overview of the performance tests Overview of the performance tests RIM obtained the performance data through a series of tests that were conducted in a controlled environment. Twenty-five tests were run to determine the impact of the BlackBerry Enterprise Server on the Microsoft Exchange Server The performance tests also displayed the performance characteristics of the BlackBerry Enterprise Server. The duration of each test was four hours. The analysis is based on the data from the last two hours of the test, when the load reached a steady state. Test # Environment Description user accounts on 2 Microsoft Exchange Servers (1000 user accounts on each server) user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) established a baseline for the Microsoft Exchange Server so that the impact on the BlackBerry Enterprise Server could be measured in subsequent tests no BlackBerry user accounts enabled MMB2 and synchronization load measured the impact of BlackBerry user accounts (core default features) on the Microsoft Exchange Server and on the Microsoft SQL Server established a baseline for the BlackBerry Enterprise Server so that the impact on the BlackBerry Enterprise Server could be measured in subsequent tests BlackBerry users enabled core features enabled (messaging, calendar, sent item redirection) MMB2 load applied and synchronization load

24 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking Test # Environment Description user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) measured the impact of attachment viewing on the BlackBerry Enterprise Server and on the Microsoft Exchange Server all attachments in messages that were generated by the MMB2 workload were viewed instantly on the BlackBerry device 50% of all attachments were requested from BlackBerry devices % of messages viewed by Loadsim was adjusted accordingly so that messages are either only viewed on the BlackBerry device or in the Microsoft Outlook client MMB2 and synchronization load applied core features enabled (messaging, calendar, sent item redirection) and the BlackBerry Attachment Server BlackBerry user accounts enabled measured the impact of wireless message reconciliation on the BlackBerry Enterprise Server, Microsoft Exchange Server, and the Microsoft SQL Server MMB2 and synchronization load applied core features enabled (messaging, calendar, sent item redirection) and wireless message reconciliation BlackBerry user accounts enabled 24

25 5: Performance tests Test # Environment Description user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) measured the impact of the BlackBerry MDS Collaboration Service load on the BlackBerry Enterprise Server and on the Microsoft SQL Server BlackBerry MDS Connection Service pulled pages equivalent to and pushed a 31-Kilobyte WML page for each request; both types of requests required some image transcoding Pull rates: 500 user accounts: one request every 28.8 seconds 1000 user accounts: one request every 14.4 seconds 1500 user accounts: one request every 9.6 seconds 2000 user accounts: one request every 7.2 seconds Push rates: 500 user accounts: two requests every second 1000 user accounts: four requests every second 1500 user accounts: six requests every second 2000 user accounts: eight requests every second MMB2, synchronization load applied, and BlackBerry MDS load core features enabled (messaging, calendar, sent item redirection) and external services enabled BlackBerry user accounts enabled measured the impact of the BlackBerry Synchronization Service on the BlackBerry Enterprise Server, the Microsoft Exchange Servers, and on the Microsoft SQL Server. BlackBerry Synchronization Service load traffic: contacts (2 adds, 1 modify, 1 delete); tasks (4 adds, 1 modify, 1 delete); memos (2 adds, 1 modify, 1 delete) for each user each 8-hr day; 10 synchronization commands each 8-hr day for each user from the BlackBerry device MMB2 and synchronization load applied core features enabled (messaging, calendar, sent item redirection) and BlackBerry Synchronization Service BlackBerry user accounts enabled 25

26 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking Test # Environment Description user accounts on 2 Microsoft Exchange Servers 500 BlackBerry user accounts 2 BlackBerry Messaging Agents (250 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1000 BlackBerry user accounts 2 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 1500 BlackBerry user accounts 4 BlackBerry Messaging Agents (375 BlackBerry user accounts on each one) user accounts on 2 Microsoft Exchange Servers 2000 BlackBerry user accounts 4 BlackBerry Messaging Agents (500 BlackBerry user accounts on each one measured the impact of all benchmarked BlackBerry features enabled at the same time on the BlackBerry Enterprise Server, Microsoft Exchange Server, and on the Microsoft SQL Server MMB2 and synchronization load applied (as defined above), and BlackBerry MDS load (as defined above) core features enabled (messaging, calendar, sent item redirection), wireless message reconciliation, BlackBerry Attachment Server, and BlackBerry MDS Connection Service BlackBerry user accounts enabled 26

27 Microsoft Exchange 2003 performance test results 6 Overview of the test results Processor use Memory use Disk use Network use Performance impact considerations Overview of the test results The following table shows the average processor time, average network bytes/sec, disk transfers/sec (reads/sec and writes/sec) and pages/sec recorded during performance tests 1 through 25, excluding tests 14 through 17, which had no impact on the Microsoft Exchange Servers. The data was averaged across two Microsoft Exchange Servers, each with exactly the same number of Microsoft Exchange and BlackBerry user accounts enabled for all benchmarking scenarios. See Performance tests on page 23 for more information about each performance test. The data in the following table shows the average processor time, disk transfers/sec, network bytes/sec, and pages/sec for one Microsoft Exchange Server. The counters were collected using the System Monitor on a Windows Server 2003 computer. Test # Average % processor time (Total) Average % processor time (store.exe) Average disk transfers/sec Average network bytes/ sec (Mbps) Average memory pages/sec

28 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking Test # Average % processor time (Total) Average % processor time (store.exe) Average disk transfers/sec Average network bytes/ sec (Mbps) Average memory pages/sec Processor use In the performance testing, the Microsoft Exchange Server processor time remained below 30%, which is well below the upper threshold of 75% that should not be exceeded on a Microsoft Exchange Server. The largest impact on the processor occurred when all features were enabled. The processor time increased by 19.81%, which was a 3.65-time increase above the baseline value with no BlackBerry user accounts enabled. Looking at the individual features, enabling wireless reconciliation along with the core , calendar, and sent item redirection had the largest impact to Microsoft Exchange with respect to processor time. When wireless message reconciliation was enabled along with the core BlackBerry features for 2000 BlackBerry user accounts, the processor time increased by 17.89%, which is a 3.29-time increase above the baseline value. Enabling any other feature resulted in minimal increases to processor time. Even with this increase in processor time, the value remained well below the maximum 75% threshold. See Appendix: Calculations and data on page 59 for more information about the calculations. 28

29 6: Microsoft Exchange 2003 performance test results The impact observed by RIM is specific to the workload applied and might vary site-to-site. The relative impact observed occurs only on messaging traffic specific to what the BlackBerry Enterprise Server redirects. The impact is not a multiplier that can be applied to existing processor use because the impact might also include load from other applications, backups occurring on the Microsoft Exchange Server, or other Microsoft Exchange traffic not specific to BlackBerry redirection. Microsoft Exchange Server average processor time (total) Memory use Because the Microsoft Exchange Server handles its own memory management, memory use cannot be monitored easily on the Microsoft Exchange Server. To determine if enough memory is available for the system, monitor pages /sec. Microsoft recommends that pages /sec should always remain below A memory bottleneck causes the number of pages /sec to exceed this threshold, which results in disk thrashing. Disk thrashing causes more stress on the disk subsystem and can result in poor performance. In the performance test environment, each Microsoft Exchange Server had 2560 MB of memory. The maximum pages /sec that RIM observed was , which is well below any dangerous threshold. During the load testing, the pages /sec counters dropped to zero periodically and remained well below the acceptable thresholds. Microsoft also recommends that available bytes always remain above 50 MB. RIM followed this recommendation when benchmarking memory use. Disk use The number of disk transfers /sec reports the rate of disk reads and writes that are occurring in the disk subsystem when each feature is enabled. It is a good measure of the impact on the disk subsystem used by the messaging server as BlackBerry user accounts are enabled with specific features. The configuration for these tests had 6 disks configured as RAID 0 with 15K rpm drives that, on average, can handle over 120 IOPS for each disk. 29

30 BlackBerry Enterprise Server for Microsoft Exchange Performance Benchmarking From the baseline, the largest increase occurred with all BlackBerry features (that were benchmarked) enabled. Disk transfers /sec grew from to Even though there was a 3.76-time increase in disk transfers /sec, this number is well below the maximum capacity that the disk subsystem can handle. Out of the individual features that were benchmarked, RIM observed that wireless reconciliation, along with the core BlackBerry features enabled ( , calendar, sent item redirection) caused the largest increase in disk use. Disk transfers /sec grew from to , which was a 3.33-time increase in disk transfers /sec. See Appendix: Calculations and data on page 59 for more information about the calculations. In the past, RIM tracked the disk queue length, which tends to be more useful as an indicator of potential bottlenecks for specific direct-attached disk subsystems and not as a general disk use measurement. Also, disk queue length is related to how long the disk system takes to complete a transfer (disk sec/transfer). As a verification in the test environment, RIM observed that the average disk queue length remained well below the maximum threshold of one for each disk spindle in the array. For SAN environments, the disk seconds per transfer is also a good indicator of disk bottlenecks. The value should not exceed 10 ms on average and should not spike to more than 50 ms. As a verification in the test environment, RIM observed that the average disk transfers /sec counter was well below the critical threshold. If you have a high disk queue length, a high disk seconds per transfer measure, or a high number of disk transfers /sec, the observed increase in disk traffic might impact your environment. Visit prodtechnol/exchange/2003/library/e2k3perf.mspx for more information about the accepted maximum levels. The impact observed by RIM is specific to the workload applied and might vary site-to-site. The relative impact observed occurs only on messaging traffic specific to what the BlackBerry Enterprise Server redirects. The impact is not a multiplier that can be applied to existing disk traffic because the impact might also include disk activity from other applications, backups occurring on the disk subsystem, or other Microsoft Exchange traffic not specific to BlackBerry redirection. Microsoft Exchange Server average disk transfers/sec 30