EMC Archiving for Microsoft Exchange 2007

Transcription

1 EMC Archiving for Microsoft Exchange 2007 Enabled by EMC CLARiiON CX4, EMC Centera, EMC SourceOne, and VMware vsphere 4 Proven Solution Guide

2 Copyright 2009 EMC Corporation. All rights reserved. Published October 2009 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part number: H

3 Table of Contents Table of Contents Chapter 1: About this Document...4 Overview...4 Audience and purpose...5 Scope...6 Business challenge...7 Technology solution...8 Objectives...9 Reference Architecture...10 Validated environment profile...11 Hardware and software resources...12 Prerequisites and supporting documentation...13 Chapter 2: Storage Design...14 Overview...14 Exchange storage design layout...15 Spindle calculations...15 Database LUN size...16 Log LUN size...16 Mailbox content...17 EMC SourceOne storage design layout...19 DiskXtender...21 CLARiiON spindle layout...22 Chapter 3: EMC SourceOne Architecture Design...23 Overview...23 Architecture design layout...24 Chapter 4: Testing and Validation...26 Overview...26 Section A: Client experience...27 Section B: Historical archiving and shortcutting...28 Section C: Journaling...48 Section D: CLARiiON performance data...55 Chapter 5: Conclusion...57 Overview

4 Chapter 1: About this Document Chapter 1: About this Document Overview Introduction to about this document This Proven Solution Guide summarizes a series of data points that were discovered, validated, or otherwise encountered during the validation of a solution using EMC archiving for Exchange 2007 enabled by EMC CLARiiON CX4, EMC Centera, EMC SourceOne, and VMware vsphere. EMC's commitment to consistently maintain and improve quality is led by the Total Customer Experience (TCE) program, which is driven by Six Sigma methodologies. As a result, EMC has built Customer Integration Labs in its Global Solutions Centers to reflect real-world deployments in which TCE use cases are developed and executed. These use cases provide EMC with an insight into the challenges currently facing its customers. Use Case definition A use case reflects a defined set of tests that validates the reference architecture for a customer environment. This validated architecture can then be used as a reference point for a Proven Solution. Contents The content of this chapter includes the following topics. Topic See Page Audience and purpose 5 Scope 6 Business challenge 7 Technology solution 8 Objectives 9 Reference Architecture 10 Validated environment profile 11 Hardware and software resources 12 Prerequisites and supporting documentation 13 4

5 Chapter 1: About this Document Audience and purpose Audience The intended audience for the Proven Solution Guide is: Internal EMC personnel EMC partners EMC customers Purpose The purpose of this solution is to: Develop a suggested EMC SourceOne architecture for 10,000 users in the context of a fully virtualized Exchange and management solution Test and document the performance of historical archiving and shortcutting of from Microsoft Exchange 2007 into the EMC SourceOne native archive (extended via DiskXtender to the EMC Centera array) Test and document the performance of realtime journaling activities from the Microsoft Exchange 2007 journaling server to the EMC SourceOne native archive (extended via DiskXtender to the EMC Centera array) 5

6 Chapter 1: About this Document Scope Scope of this document This document describes the architecture of an EMC solution built at EMC s Global Solutions labs. The solution is engineered to allow customers to: Introduce archiving into their environment with consideration for storage configuration, design, sizing, and software. Reduce operational costs while enforcing record-keeping policies in compliance with internal governance as well as industry and government regulations. Deliver the highest levels of application service agreements with the lowest total cost per application workload. Achieve the flexibility of a solution that scales to meet large enterprise needs, yet offers a simple footprint for midsize organizations. The solution provides information on: Creating a well-performing storage design for a virtualized Microsoft Office Exchange 2007 environment on a CLARiiON CX4-480 with a large and very active database (0.48 IOPS, 11.2 TB of Exchange data). Designing and deploying EMC SourceOne. Documenting the observed environment performance and the suggested methods of growing performance as needs dictate. Not in scope Testing EMC SourceOne search and retrieval functions and performance were outside the scope of this testing, as were the functional aspects of the compliance and discovery elements of EMC SourceOne. 6

7 Chapter 1: About this Document Business challenge Overview Protecting a company from data loss is a strategic imperative. Customers are facing the following business challenges on a daily basis, and many are looking for the easiest way to achieve: Optimized IT infrastructure through server and connectivity consolidation Decreased downtime and improved reliability through high availability and automated disaster recovery Increased energy efficiency through fewer servers and the ability to dynamically power down unused servers with green IT solutions Proactive messaging management Increased user productivity through seamless access to archived content Optimized archiving, litigation readiness, and compliance to information governance standards The challenges in meeting these criteria are: Regulatory requirements that mandate secure data Corporate data security policies Compliance reporting Unknown resource requirements Undetermined infrastructure design to provide high availability 7

8 Chapter 1: About this Document Technology solution Overview The following table describes the virtual machines in this environment, their configuration, and how the roles were configured. Quantity Purpose Configuration Software 2 Active Directory (AD) servers 2 vcpus Windows 2008 (64-bit) SP1 4 GB RAM 1 vnic 2 Hub/CAS servers 4 vcpus Windows 2008 (64-bit) SP1 16 GB RAM Exchange 2007 SP1 1 vnic 2 Mailbox servers (users) 8 vcpus Windows 2008 (64-bit) SP1 32 GB RAM Exchange 2007 SP1 1 vnic 2 Mailbox servers (Journal) 4 vcpus Windows 2008 (64-bit) SP1 16 GB RAM Exchange 2007 SP1 1 vnic 3 EMC SourceOne native archive (archive and index roles) 2 EMC SourceOne native archive (search and retrieve roles) 4 vcpus 4 GB RAM 1 vnic 4 vcpus 4 GB RAM 1 vnic 3 EMC SourceOne worker servers 4 vcpus 4 GB RAM 1 vnic 2 Microsoft Failover Clustering nodes for SQL Server 2 Microsoft Failover Clustering nodes for EMC SourceOne master role (also hosts Index and Message Center shares) 8 vcpus 32 GB RAM 1 vnic 4 vcpus 8 GB RAM 1 vnic Windows 2003 (32-bit) SP2 EMC SourceOne native archive Windows 2003 (32-bit) SP2 EMC SourceOne native archive Windows 2003 (32-bit) SP2 EMC SourceOne worker EMC SourceOne Web services Windows 2003 (64-bit) SP2 SQL Server 2005 SP2 Windows 2003 (32-bit) SP2 EMC SourceOne master 8

9 Chapter 1: About this Document Objectives Overview The following table details the objectives of this use case. Number Objective 1 Develop a suggested EMC SourceOne architecture for 10,000 users in the context of a fully virtualized Exchange and management solution. 2 Test and document the performance of historical archiving and shortcutting of from Microsoft Exchange 2007 into the EMC SourceOne native archive (extended via DiskXtender to the EMC Centera array). 3 Test and document the performance of realtime journaling activities from the Microsoft Exchange 2007 journaling server to the EMC SourceOne native archive (extended via DiskXtender to the EMC Centera array). 9

10 Chapter 1: About this Document Reference Architecture Corresponding Reference Architecture This use case has a corresponding Reference Architecture document that is available on Powerlink and EMC.com. Refer to EMC Archiving for Microsoft Exchange 2007 Enabled by EMC CLARiiON CX4, EMC Centera, EMC SourceOne, and VMware vsphere 4 Reference Architecture for details. If you do not have access to the following content, contact your EMC representative. Reference Architecture diagram The following diagram depicts the overall physical architecture of the use case. 10

11 Chapter 1: About this Document Validated environment profile Profile characteristics The use case was validated with the following environment profile. Profile characteristic Exchange 2007 users (0.48 IOPS very heavy user) Value 10,000 divided between: 3,000 users x 2 GB mailboxes 3,000 users x 1 GB mailboxes 4,000 users x 500 MB mailboxes Read / Write ratio 1:1 Number of Exchange 2007 users per server 5,000 1 Number of Exchange 2007 storage groups per server Number of Exchange 2007 mail databases per storage group Total size of all Exchange databases in the environment Number of Exchange 2007 users per mail database Size of Exchange 2007 user mailbox 28 divided between: 15 storage groups for 2 GB users 8 storage groups for 1 GB users 5 storage groups for 500 MB users 1 (maximum size 200 GB) 56 x 200 GB (11.2 TB) Varies based on mailbox size: 100 users per database where the mailbox size is 2 GB 188 users per database where the mailbox size is 1 GB 400 users per database where the mailbox size is 500 MB Varies based on user: 2 GB 1 GB 500 MB 1 The use of 8 vcpus to support 5,000 users does not constitute a new building block for Microsoft Exchange 2007 using VMware vsphere. The goal of this change was to ensure that CPU at the Exchange Server was not a bottleneck, allowing any effects of EMC SourceOne activities to be observed during testing. Baseline performance showed CPU utilization remained under 30% on the Exchange mailbox servers. Therefore, it is entirely possible that an additional 8 vcpu mailbox server would support more than 5,000 users, or users with increased IOPS. 11

12 Chapter 1: About this Document Profile characteristic Size of storage group mailbox database LUN (to accommodate the specified number of databases and the deleted item retention on the defrag space) Storage group log LUN size 256 GB 40 GB Value Exchange 2007 production data RAID type, physical drive size, and speed RAID 1/0, 450 GB, 15k Number of storage groups per journaling server 2 Journaling servers 2 Number of journaling mailboxes per production Exchange storage groups Number of journaling mailboxes per journaling server storage group 1 (= >4 in total) 1 Hardware and software resources Hardware The hardware used to validate the use case is listed below. Equipment Quantity Configuration Storage array 1 CLARiiON CX GB memory 8 Fibre Channel, 4 iscsi ports per storage processor (SP) 16 DAEs 180 Fibre Channel 450 GB 15k 2 FLARE 28 CAS 1 EMC Centera (8 node) Generation 4 EMC Centera 500 GB drives 16 TB raw capacity SAN 2 1 x Brocade 5100 (32-port fabric switch) 1 x Brocade 4100 (32-port fabric switch) Network 2 2 x Cisco Catalyst 3750 (32-port Ethernet switch) Production servers 3 2 x Dell R900 servers (16-core, 2.9 GHz CPU, 128 GB RAM) 1 x Fujitsu Siemens RX600S4 server (16-core, 2.9 GHz CPU, 128 GB RAM) 2 This use case tested multiple different mailbox sizes on the same Exchange server. As a result, the storage layout for this user count differs from other published material. A standard database configuration was required. Therefore, each database was designed to support the maximum number of users per database (400), rather than the actual user counts that were present on each individual database (for instance 100 users on the databases that contained 2 GB mailboxes). 12

13 Chapter 1: About this Document Software The software used to validate the use case is listed below. Microsoft Windows Microsoft Windows Microsoft Exchange Software Version 2003 (SP2) Enterprise Edition 2008 (SP1) 2007 (SP1) EMC Navisphere Agent and CLI VMware vsphere 6.28 (Agent and CLI) 4.0 RTM EMC SourceOne (Build 1035) EMC DiskXtender 6.3 Microsoft Exchange Load Generator Prerequisites and supporting documentation Technology It is assumed the reader has a general knowledge of the following products: EMC CLARiiON CX4 EMC Centera EMC SourceOne EMC DiskXtender VMware vsphere 4 Microsoft Exchange 2007 Supporting documents The following documents, located on Powerlink, provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access to the following content, contact your EMC representative. Enabled by EMC CLARiiON CX4, EMC Centera, EMC SourceOne, and VMware vsphere 4 - Reference Architecture EMC SourceOne Management Installation Guide EMC SourceOne Management Administration Guide EMC DiskXtender 6.3 Microsoft Windows Version - Installation Guide EMC DiskXtender 6.3 Microsoft Windows Version - Administration Guide 13

14 Chapter 2: Storage Design Chapter 2: Storage Design Overview Introduction to storage design Storage design is an important element to ensure the successful development of the EMC Archiving for Microsoft Exchange 2007 enabled by EMC CLARiiON CX4, EMC Centera, EMC SourceOne, and VMware vsphere 4 solution. Contents This chapter contains the following topics: Topic See Page Exchange storage design layout 15 Spindle calculations 15 Database LUN size 16 Log LUN size 16 Mailbox content 17 EMC SourceOne storage design layout 19 DiskXtender 21 CLARiiON spindle layout 22 14

15 Chapter 2: Storage Design Exchange storage design layout Description of the Exchange storage design Each Exchange Server hosting user mailboxes had 28 Exchange Storage Groups (ESGs), each with a single database. Each Exchange Server hosting journal mailboxes had two ESGs, each with a single database. This brought the total number of ESGs to 60. This use case involved a mixed configuration of mailbox sizes, which directly affected the number of users each database could host, while staying within 200 GB for the mailbox database file. However, all users were considered the same in terms of user activity (very heavy) so the database IOPS calculations were based on the databases that contained the maximum number of users. In this case, the maximum number was 400 users. Because Exchange is a bursty application by nature, it is good practice to combine several databases on the same spindles. This way they can share the combined IOPS capability of the drives that they are stored on, since it is unlikely that all the databases will require bursts of IOPS at precisely the same moment. This helps to avoid over-engineering the solution and to keep the overall cost at a minimum. In this case, a decision was taken to combine five databases onto the same spindles. Spindle calculations Exchange spindle calculation For any single database, the IOPS requirement was as follows: IOPS = 400 users x 0.48 IOPS (Very Heavy Profile) = 192 IOPS The total back-end IOPS for RAID and drive types is calculated as follows: (192 *.5) + 2(192 *.5) = = 288 /180 (capacity of 15k drive) = 1.6 drives per database Combining five databases together: 1.6 drives per database * 5 = 8 drives 3 Therefore, for every five ESGs a total of 12 spindles was used and allocated as follows: Four spindles were assigned in a 2+2 RAID 1/0 configuration for logs Eight spindles were assigned in a 4+4 RAID 1/0 configuration for databases In total, 144 disks were used for these 60 ESGs. This allowed for 56 ESGs for the user mailboxes and four ESGs for the journal mailboxes. 3 This use case tested multiple different mailbox sizes on the same Exchange server. As a result, the storage layout for this user count differs from other published material. A standard database configuration was required. Therefore, each database was designed to support the maximum number of users per database (400), rather than the actual user counts that were present on each individual database (for instance 100 users on the databases that contained 2 GB mailboxes). 15

16 Chapter 2: Storage Design Database LUN size Database LUN size calculation Depending on the size of the user mailboxes, there was a different number of users for each database: 2 GB users = 100 users per database 1 GB users = 188 users per database MB users = 400 users per database A standard database LUN size was selected for simplicity in the design, therefore the database LUN size was based on the largest required, which in this case is either the 2 GB user database or the 500 MB user database. Working out the required LUN size for 2 GB users was done as follows: 100 x 2 GB =200 GB + 50 GB* = 250 GB. *Twenty five percent extra space was allocated for deleted items retention, content indexing, and growth. This was rounded up to 256 GB (being the maximum VMFS volume size on VMware vsphere 4 with the default block size). Log LUN size Log LUN size calculation Normally, an allocation of five logs per day per user would be sufficient but in previous testing, depending on the actual user IOPS, as many as 20 logs per day have been observed. In this use case, the maximum number of users per storage group was 400 (in the case of users with 500 MB mailboxes) so the calculation is as follows: 400 x 20 MB (20 logs x 1 MB) = 8,000 MB or 8 GB Five days of log file storage is calculated as follows: 8 GB x 5 = 40 GB Allowing for five days of log file storage allows the system to continue operating, even in the event of backup failure (and the associated absence of log file truncation). In this case, it also provided ample room to accommodate the logs files generated by EMC SourceOne activities (such as shortcutting). 4 Because there were a total of 3,000 users with 1 GB mailboxes in this environment, and a desire to spread these users evenly across the two mailbox servers, it was decided to create 16 storage groups to house these mailboxes (8 per server), and have 188 users per database, rather than 15 storage groups of 200 users each. 16

17 Chapter 2: Storage Design Mailbox content Generating the Exchange mailboxes The content of the Exchange data was generated using the latest beta version of Microsoft Exchange Load Generator (LoadGen) available at the time of testing. This beta version was used as it provided a new feature called Dynamic Mail Generation. Previous versions of LoadGen created content using a static collection of messages. While it was possible to increase the number of static messages it used, it was not feasible to create enough individual messages to populate a 10,000-user environment without significant re-use of the same messages. For normal performance or functionality testing this is acceptable but, in an archiving test, it means that a significant number of messages are dismissed as duplicates, giving a misleading indication of performance. Dynamic Mail Generation uses a randomization engine to create unique messages as it populates the mailboxes. It is also possible to create random messages during performance test runs. Both of these factors were important in ensuring that this use case produced realistic performance numbers. Exchange mailbox structure Three different mailbox sizes were tested in this use case. The following table outlines how each mailbox size was configured. Each of these settings is adjusted through the XML file used to control LoadGen. LoadGen setting 2 GB mailbox 1 GB mailbox 500 MB mailbox InboxMessageCount 5,000 2,500 1,250 DeletedMessageCount NewFolderCount DeepFolderCount DeepFolderDepth NewFolderMessageCount AttachmentFrequency Attachment Pathname Custom Custom Custom The customized Attachment Pathname setting refers to the folder of 960 unique attachments of varying types and sizes that was provided to LoadGen to use as a library for creating the mailboxes. This combined with the 30 percent attachment ratio (increased from LoadGen default of 0.5 percent) ensured that the profile of the Exchange databases was closer to a real life environment than a database generated with default LoadGen settings. 17

18 Chapter 2: Storage Design Attachment types and sizes The attachment library was designed to contain attachments ranging in size from 4 KB to 10 MB, including the following types: PDF DLL BMP HTML JPEG XLS XLSX PPT PPTM DOC DOCX MSG GIF TXT ZIP 18

19 Chapter 2: Storage Design EMC SourceOne storage design layout Drives for installation There are a number of drives associated with the EMC SourceOne installation, including the drives required for: Message Center Volumes folder Index Native archive temporary space SQL Server databases SQL Server logs Message Center The Message Center is a temporary location used while container files are being built. The structure and contents of a container file are determined in the Message Center as mail items come into the system. Mail items are assigned to a container as they are ingested. When a container is considered full, or the time limit expires (24 hours), the container file is then closed and written out to the permanent archive location (volumes folder). Volumes folder The volumes folder is the location on a shared drive where container files are permanently stored. This drive can also be extended using DiskXtender. In this use case, it was extended to the EMC Centera array. Note: The volumes folder and the message center were colocated on the same LUN, which was 500 GB in size on a 2+2 RAID 1/0 configuration. Colocating the two roles does increase overall I/O to the drive, so adequate IOPS capability should be assigned. Index The Index drive is used to store the indexes that are built during EMC SourceOne activities. In this use case, the Index LUN was created on a 2+2 RAID 1/0 configuration, which proved more than adequate. Note, however, that the search function was not part of the scope of this testing and therefore the performance of this LUN may need to be increased, depending on the intensity of the searching performed in the environment in question. 19

20 Chapter 2: Storage Design Native archive temporary space When installing the native archive component, there is a requirement to specify a temporary storage location for use when generating container files and indexes. When complete, these files are then transferred to the Message Center and Index shares respectively, to be accessed by all components of the EMC SourceOne architecture. The EMC SourceOne Management Installation Guide suggests specifying a drive with at least 1 GB of free space for this temporary space. Initially, in testing, a dedicated 2 GB LUN was created and assigned to each server designated to run the archive and index roles. During testing, it was observed that this drive ran out of space and resulted in incomplete or corrupt indexes due to a full disk condition. To overcome this condition a larger volume was assigned to each of the servers with the archive and index role to ensure that indexes could always successfully write to disk. The size of the disk should be based on the predicted or observed level of activity. SQL Server databases A 200 GB LUN was created on a 6+6 RAID 1/0 configuration to host the SQL Server database files for the following databases: EMC SourceOneArchive EMC SourceOneActivity EMC SourceOneSearch SQL Server logs A 100 GB LUN was created on a 3+3 RAID1/0 configuration to host the SQL Server log files for the following databases: EMC SourceOneArchive EMC SourceOneActivity EMC SourceOneSearch 20

21 Chapter 2: Storage Design DiskXtender DiskXtender As shown in the following image, DiskXtender 6.3 was used to extend the Message Center drive to the EMC Centera array. DiskXtender was installed using the default settings and the EMC Centera IP addresses and security file were specified according to the guidance provided in the EMC DiskXtender 6.3 Microsoft Windows Version Installation Guide. EMC SourceOne automatically configures DiskXtender when the properties of the Message Center drive are configured to extend to the EMC Centera array. 21

22 Chapter 2: Storage Design CLARiiON spindle layout CLARiiON spindle layout The following diagram illustrates the CLARiiON spindle layout. 22

23 Chapter 3: EMC SourceOne Architecture Design Chapter 3: EMC SourceOne Architecture Design Overview Introduction to EMC SourceOne architecture design This section describes the EMC SourceOne architecture tested in this solution and indicates the specific tuning parameters that were selected and tested. Contents This section contains the following topic: Topic See Page Architecture design layout 24 23

24 Chapter 3: EMC SourceOne Architecture Design Architecture design layout EMC SourceOne final setup The final EMC SourceOne setup for this environment consisted of the following: Quantity Purpose Configuration Software 3 EMC SourceOne native archive (archive and index roles) 4 vcpus 4 GB RAM 1 vnic 2 EMC SourceOne native archive (search and retrieve roles) 3 EMC SourceOne worker servers 2 Microsoft Failover Clustering nodes for SQL Server 2 Microsoft Failover Clustering nodes for SourceOne master role (also hosts Index and Message Center shares) 4 vcpus 4 GB RAM 1 vnic 4 vcpus 4 GB RAM 1 vnic 8 vcpus 32 GB RAM 1 vnic 4 vcpus 8 GB RAM 1 vnic Windows 2003 (32-bit) SP2 EMC SourceOne native archive Windows 2003 (32-bit) SP2 EMC SourceOne native archive Windows 2003 (32-bit) SP2 EMC SourceOne worker EMC SourceOne Web services Windows 2003 (64-bit) SP2 SQL Server 2005 SP2 Windows 2003 (32-bit) SP2 EMC SourceOne master EMC SourceOne initial setup In the original setup of EMC SourceOne, only two virtual machines were deployed with the archive and index server roles, and both of these virtual machines were only provided with two vcpus. This setup proved sufficient to effectively archive in realtime all of the journal activity created by the 10,000 users during the LoadGen test runs. However, when testing moved to historical archiving and shortcutting, it was observed that the archive and index servers were CPU-bound with just two vcpus, as they were operating at close to 100 percent at all times. Increasing the vcpu count in these virtual machines to four vcpus significantly eased the pressure on these virtual machines and they subsequently operated at an average of approximately 55 percent CPU, but still with significant spikes of up to 100 percent. Similarly, the original setup consisted of only two worker servers with two vcpus each. As the archive server CPU bottleneck was removed, the bottleneck then 24

25 Chapter 3: EMC SourceOne Architecture Design became the CPU on the worker servers, so these virtual machines were also increased to four vcpus. As testing progressed, an additional archive and index server, and an additional worker server were deployed to demonstrate the effect of the scalable architecture of EMC SourceOne. As expected, adding additional components significantly increased the throughput capability of the system. In this case, it had the effect of increasing the number of messages processed from 80,000 messages per hour with two worker and two archive servers to 120,000 messages per hour with three worker and three archive servers. EMC SourceOne server tuning EMC SourceOne limits the number of jobs that can run concurrently against any discovered target mail server. This is to ensure that the mail server does not become overwhelmed by requests to execute EMC SourceOne activities at the expense of the production traffic generated by users. The default setting is six jobs. After initial testing, it was observed that six jobs was not placing significant extra load on the Exchange Server, and therefore the value was increased to nine to increase the processing throughput. It is important to correlate this number with the number of jobs that worker servers are permitted to run, as there is no value in increasing one without a corresponding increase in the other (as shown in the following image). 25

26 Chapter 4: Testing and Validation Overview Introduction to testing and validation This chapter provides details on Microsoft Exchange 2007 performance, the EMC SourceOne components tested in this solution, and a summary of the test results. Contents This section contains the following topics: Topic See Page Section A: Client experience 27 Section B: Historical archiving and shortcutting 28 Section C: Journaling 48 Section D: CLARiiON performance data 55 26

27 Section A: Client experience Overview of client experience Most customer environments that are considering implementing an archiving solution into an existing environment do not have the luxury of ingesting the content of their databases during off peak hours, as it would significantly lengthen the implementation time. There are also many situations where a system is busy 24 hours a day due to the geographical spread of the user base. The following tests were carried out to establish the effect on the user experience of carrying out realtime journaling, archiving, and shortcutting tasks during the working day. The I/O impact of journaling, archiving and shortcutting to production, journal and archiving servers is discussed in the subsequent sections of this chapter. In all tests where Exchange user activity was simulated, the results of the LoadGen test were deemed to be successful by the tool, meaning that the performance of every metric was within the recommended thresholds. To get a better insight to the end-user experience during the various tests, the RPC Average Latency was compared to see if journaling or any of the EMC SourceOne activities affected this value. RPC Average Latency The following chart shows that performing historical archiving or shortcutting tasks has a marginal impact on RPC Average Latency. However, when compared with the recommended threshold of 50 ms, it is apparent that the client experience would have been excellent both before and after the introduction of journaling and EMC SourceOne activities. 27

28 Section B: Historical archiving and shortcutting EMC SourceOne historical archiving Historical archiving is the operation of copying s that meet the desired criteria into an archive. EMC recommended practice is to create multiple groups of Exchange items (either by server hierarchy, address book, or LDAP query) to stagger historical archiving and shortcutting tasks. For instance, the archiving task for Exchange Group 1 might run on Monday night, with its associated shortcutting task configured to run on Tuesday night, giving an opportunity to back up the EMC SourceOne environment and any archived messages before they are actually shortcut. The archiving task for Exchange Group 2 could be run on Tuesday in tandem with the shortcutting task for Exchange Group 1, and so on. EMC SourceOne shortcutting Shortcutting tasks remove the original item and replace it with a shortcut stub to an archived item. With EMC SourceOne, a shortcutting activity will archive and shortcut qualifying items. If a historical archiving activity has been run prior to the shortcutting activity EMC SourceOne will simply report the mail item itself as being a duplicate, that is, present in the archive already, and just shortcut the item. Configuration for this use case Normally, a historical archiving task is designed to run against a subset of the mail items within the databases: for instance, items older than 90 days. This task is normally run on a daily or weekly basis, depending on customer requirements. Because it is run on such a frequent basis. Normal practice would be to set the duration of the task to Run to Completion, meaning it will continue to process mail until it has identified and archived all the mail items meeting the chosen criteria. However, the objective of this testing was to determine the rate at which this EMC SourceOne configuration could ingest mail items. Because this is an artificial environment, and the entire mail item content was generated in a very short period of time, setting the duration of the tasks to Run to Completion would not be feasible since all the dates are in such a short range. A single test would probably ingest the majority of the mail items. Therefore, for tests designed to see the maximum throughput of historical archiving and shortcutting for the configuration, all activities were configured with a task duration of three hours, with the statistics used to calculate the average messages processed per hour by each activity. Also, only a single activity was configured to process mail items from both of the production mailbox servers as separating into two individual activities, and addressing different Exchange objects, offered no additional value. On average, these test runs lasted 10 hours, with the timeline shown in the following diagrams, where Load was the user load, online maintenance, or idle condition. 28

29 Note: In both of the following timelines the term Load is intended to depict a constant that is running without interruption for the duration of the 10-hour test. The following diagrams outline the periods of the test where Load was running in isolation, and the periods where SourceOne was also active. For tests designed to observe the throughput of the system while running historical archiving and shortcutting tasks in tandem against different Exchange objects, then two separate activities were configured with the following timeline for the test. In this instance, Group 1 consisted of Exchange Server 1, and Group 2 consisted of Exchange Server 2. During the historical archiving of Group 1, nine worker jobs were allowed for this process, due to the default limit set on the server s tuning tab. Later in the same test, the same nine jobs slots were used by the shortcutting activity for Group 1, while Group 2 was using nine worker jobs to perform the historical archiving task. If increased throughput is needed above the values provided by this configuration, the procedure is to increase the jobs allowed per mail server, along with either (a) an increase in jobs allowed per worker or (b) the creation of additional worker servers. 29

30 Historical archiving performance The following chart illustrates historical archiving performance during the test run. The final column of the chart, where only nine worker threads were operational, corresponds to the test where EMC SourceOne was archiving from one set of Exchange objects while using the remaining allowed jobs to shortcut mail items from another set of Exchange objects. The reasons for this are explained in Chapter 4: Testing and Validation > Section B: Historical archiving and shortcutting >Configuration for this use case. 30

31 Archive server CPU Utilization The following tests all represent scenarios where LoadGen test runs were running in parallel with the historical archiving tasks. The same tests were run during idle conditions and during online maintenance, with similar results. In the first set of tests, with two worker and two archive servers, each with two vcpus, it was noticed that the CPU utilization of the archive servers was regularly between 90 and 100 percent, as shown in the following graph. Based on this, both archive servers were increased from two vcpus to four vcpus, and the test was re-run. While the number of messages processed over the threehour test did not increase substantially, it is apparent that the process of building a container file is an aggressive user of CPU resources, as shown in the following graph. Based on these observations, careful attention should be paid to the number of CPUs allocated to the native archive role. The number of CPUs per native archive server may vary based on the number of machines deployed in this role, but it is advisable to monitor to ensure CPU is not a bottleneck. 31

32 Later in the testing, an additional archive and worker server were added in an attempt to measure the effectiveness of scaling out the environment and the following graph shows the effect of this on an individual archive server. As shown in the graph above, this substantially increased the message-processedper-hour throughput of the environment, while reducing the intensity of the CPU utilization on the individual archive server. It also highlighted that CPU utilization on the archive server is bursty in nature. 32

33 Effect of historical archiving on the mailbox server CPU During a baseline LoadGen test run, the average CPU utilization on the Exchange Server was approximately 25 percent. As shown in the following graph, irrespective of whether two or three worker servers were active in the environment, the effect of historical archiving activities on the user mailbox server was very similar. At the start of the historical archiving activity, CPU utilization jumps to between 50 and 60 percent and then slowly declines over the course of the 3-hour test. Just before completion, utilization is between 30 and 40 percent. To compare with a baseline for a mailbox server, all of the data in the previous graph was taken while LoadGen tests were running. To verify this, the same historical archiving task was run against an idle mailbox server with the results as shown in the following graph. 33

34 This graph verifies that the three-worker server configuration was placing an average 10 to 15 percent additional CPU load on the mailbox server during the archiving activity. 34

35 Worker CPU utilization during historical archiving activities As shown in the following graph, worker CPU utilization was very similar during all tests irrespective of the number of archive or worker servers configured. This was due to the fact that even when additional workers were added, the number of connections permitted to each Exchange Server (via the server tuning tab in EMC SourceOne) was increased, such that there were always six historical archiving worker jobs permitted per worker operating against the Exchange Servers. Based on these figures, it appears that each four-vcpu worker is capable of running more concurrent historical archiving jobs, assuming that the archive servers and the Exchange Servers have the bandwidth to cope with the extra work generated. 35

36 SQL Server CPU utilization during historical archiving The following graph shows SQL Server CPU utilization during historical archiving. SQL Server CPU utilization was very low, though it is important to note that this testing did not incorporate a search or retrieve load, which would be one of the major consumers of CPU resources on the SQL Server machine. 36

37 Comparing archiving performance under varying conditions Tests were run to determine how archiving performance would be affected, depending on the time of day and any other workloads that might be running in tandem. As can be seen the following graph, all three tests produced similar results, with messages processed during online maintenance being slightly lower due to the extra load online maintenance places on the Exchange Server CPU. 37

38 Mailbox server CPU utilization during varying load As can be seen from the following graph, the mailbox server CPU is under extreme pressure during online maintenance. While historical archiving tasks during online maintenance still achieved 90 percent of the number of items processed per hour in the other two tests, scheduling any tasks that are CPU intensive during online maintenance may not be the best policy. Shortcutting performance Shortcutting operations can be measured in two different ways, depending on whether or not an historical archiving job has been run against the target mail server objects in advance. The first method is where shortcutting tasks are set to Run until completion, at some point after an historical archiving job has been run. This activity just shortcuts mail and does not archive any additional mail (unless new items are present that meet the criteria). The second method is to let shortcutting run without a previous archiving job. This results in all matching mail items needing to be both archived and then shortcut as part of the process. These tests attempt to determine the duration and effects of the first method, as well as the effects and throughput of the second method. The shortcutting process is more intensive with respect to the effect it has on an Exchange Server as it involves changes in the content of the Exchange database (rather than just reading it). This results in the creation of Exchange logs. 38

39 Duration to complete shortcutting after prior archiving Where historical archiving jobs were run prior to shortcutting, they were configured to run for three hours. The subsequent shortcutting tasks were also configured to run for three hours but in all cases, it was observed that it took between 40 and 45 minutes to shortcut all of the mail that had been archived in the preceding archiving task. Effect of shortcutting on the mailbox server CPU As can be seen in the following graph, the start of shortcutting corresponded with an increase of CPU utilization from 25 percent to approximately 60 percent. This stayed relatively constant until all the mail items were processed, at which time it returned to the previous steady state of 25 percent. 39

40 Effect of shortcutting on the SQL Server CPU As shown in the following graph, SQL Server CPU utilization exhibited a similar load as was observed during historical archiving. Once again, however, search and retrieve were not being utilized at this point, so significant extra CPU utilization would be expected if they were also present. Shortcutting performance with no previous archiving Two tests were carried out to gather this data, to see if there was a difference in performance when the mailbox server was under user load and when it was idle. Both tests were run over three hours for fair comparison against the historical archiving tests. As can been seen in the following chart, the results were almost identical. In fact, slightly more mail items were processed under load in this instance. This does not indicate that performance is slightly superior during the user load. The variance can be explained by the fact that new mail was being generated during this user load, and some of this new may have been smaller and easier to process, or may simply have benefited from being in the Exchange database cache on the Exchange Server. In reality, the numbers are so similar that they can be classed as statistically the same. 40

41 Disk I/O observations While online maintenance tests showed the highest utilization figures for the mailbox servers (especially CPU and disk), it was evident that scheduling any other tasks during online maintenance would not be good practice. Therefore, the following disk I/O observations are all taken from tests during which the user load was also running, as they were seen to be the most resource-intensive tests, while still being appropriate times for EMC SourceOne activities to be scheduled. 41

42 Disk I/O observations for the user mailbox server during historical archiving Historical archiving is primarily a read operation from the perspective of the mailbox server being targeted. As can be seen in the following graph, database reads increased substantially during the time that historical archiving was running. As shown in the following graph, database response time was unaffected by the increase, since the spindles assigned for Exchange databases were able to handle the extra load. 42

43 Disk I/O observations for the user mailbox server during shortcutting In contrast to an archiving task, shortcutting caused a lot of write activity on the targeted mailbox server. The following graph looks at the 45-minute window where shortcutting was run (after a preceding three-hour historical archiving task). Database write I/O increased substantially, peaking at over 300 IOPS for this single database. Log writes exhibit a similar dramatic increase, caused as the insertion of stubs and the removal of the complete mail items created a surge in log writes. As shown in the following graph, average log write activity was approximately six times higher than during baseline load, with peaks of up to 100 IOPS per log device in this instance. 43

44 However, as the following chart illustrates, log and database response times were not dramatically affected. 44

45 Disk I/O observations for the SQL Server during historical archiving As expected, regardless of user load being present or not, the amount of disk activity created on the SQL Server was very similar (as shown in the following graph). In this case, all of the EMC SourceOne SQL databases shared the same database and log LUN pair. No meaningful database read activity was observed here, but it should be expected in an environment where search and retrieve functions are being extensively used. In this configuration, there were three worker servers with six threads each. Typically, a thread would connect to approximately 10 mailboxes, meaning a total of 180 mailboxes being archived at any given moment. The activity seen in the following graph (approximately 600 database IOPS and 125 log IOPS) would be expected to increase if additional workers or additional threads were working simultaneously. 45

46 Disk I/O observations for the SQL Server during shortcutting SQL Server database and log activity was seen to be much lower when shortcutting previously archived items, as was the case for the first 45 minutes of this test run. Because this test was running for three hours and the match criteria found additional to be processed, the shortcutting task continued to archive and shortcut the remaining found. This is why a sharp increase in database and log write activity is observed 45 minutes after the test started (shown in the following two graphs). 46

47 Disk I/O observations for the archive server during historical archiving The archive server writes to the workspace were not seen to differ during historical archiving, regardless of the presence of user load (which is expected behavior). The IOPS shown in the following graph is for one of three archive servers in operation during the test. Disk I/O observations for the archive server during shortcutting The archive server writes to the workspace were not seen to differ during shortcutting, regardless of the presence of user load (which is expected behavior). It is noticeable in the following graph that there is very low activity for the first 45 minutes of the test, which represents the shortcutting of previously archived mail items. Once these are complete, and the shortcutting task moves on to finding other items that meet the match criteria, an increase in activity is noticeable. The IOPS shown in the following graph is for one of three archive servers in operation during the test. 47

48 Section C: Journaling Journaling configuration Journaling tests were conducted over the course of 10-hour LoadGen test runs, with the databases configured to journal to one of four journal mailboxes. The journal infrastructure was as follows: Each journal mailbox was placed on its own Exchange storage group. Each of these storage groups had dedicated database and log LUNs. Two journal servers were created, each with two storage groups. Each storage group had one database, and each database had one journal mailbox. The user mailbox database to journal the mailbox relationships was configured to balance, as closely as possible, the number of users per journal mailbox, without becoming more granular than the database-level configuration. This configuration was chosen in order to create an environment where the resource requirements for journaling could be observed in isolation. There is no technical or practical reason why the mailbox databases/mailboxes could not be consolidated onto the user mailbox servers in actual deployment. EMC SourceOne journaling EMC SourceOne carries out journaling by creating a master journaling job per journal activity created in the EMC SourceOne console. This master job in turn spawns child jobs, which are dispatched to the various worker servers that are enabled to execute journal activities. A single journal activity was created to address all four journal mailboxes. Because of the configuration chosen (that is, four journal mailboxes), a maximum of four child jobs can be spawned by the master job. Each child job can connect to a journal mailbox and archive the items it sees at the time of connection. Once complete, this job stops, and a new job is spawned approximately five minutes after the completion of the previous child job. This pattern can be seen in the following image. 48

49 In all the tests carried out, the EMC SourceOne architecture was seen to successfully keep pace with the rate at which the 10,000 very heavy users were generating new . Over the course of 10 hours, over 300,000 messages were archived as a result of journaling alone. Journal server CPU utilization As shown in the following graph, over the course of a 10-hour LoadGen test, CPU usage averaged approximately 7 percent. However, given that there are five-minute intervals between child jobs being spawned, the average is somewhat skewed by this idle time. Zooming in to look at a 30-minute segment of the same test, the following pattern can be seen (shown in the following graph). 49

50 So, while child jobs are actively removing from journal mailboxes, it appears that approximately 15 percent load is placed on the CPU of the dedicated journal mailbox server. This figure returns to nearly zero percent as the child jobs finish and enter a waiting state for the next polling interval. If journal mailboxes are to be homed on user mailbox servers, then this CPU utilization increase should be factored into the specification of that user mailbox server. SQL Server CPU usage during journaling While the SQL Server utilization was low during journaling, it showed the same spiky behavior in relation to the five-minute polling interval of the EMC SourceOne journaling master job, as shown in the following graph. 50

51 Disk I/O observations While online maintenance tests showed the highest utilization figures for the mailbox servers (especially the CPU and disk), it was evident that scheduling any other tasks during online maintenance would not be good practice. Therefore, the following disk I/O observations are all taken from tests during which the user load was also running, as they were seen to be most resource-intensive tests, while still being appropriate times for EMC SourceOne activities to be scheduled. Disk I/O observations for the journal mailbox server Looking at a similar 30-minute window to see what the disk requirements are, the spiky nature continues to follow the polling interval pattern, with just short of 70 IOPS being the requirement for one of the log devices (approximately 2,500 users). As s are bifurcated and sent to the journal mailboxes, a steady stream of about 20 IOPS can be seen. When EMC SourceOne journal child jobs run, they copy the items to the archive and then delete them from the journal mailbox. As shown in the following graph, the act of deletion caused addition log write I/O, resulting in the spike in activity up to the 70 IOPS. Database writes per second during the same period were not seen to follow quite the same pattern as shown in the following graph. This can be expected given the coalescence that should be occurring at an Exchange level. 51

52 Journal server database reads per second were low, but retained the five-minute pattern. Steady-state-ready activity is effectively zero since only the EMC SourceOne child jobs read the mailbox content. This is shown in the following graph. 52

53 Disk I/O observations for the SQL Server during journaling As shown in the following two graphs, both the SQL Server log and database writes show similar characteristics, in line with the polling interval - with database IOPS peaking at between 400 and 500 IOPS, and logs peaking between 150 and 200 IOPS during the interval where EMC SourceOne is archiving from the journal mailboxes. 53

54 Disk I/O observations for the archive server during journaling As noted in Chapter 2: Storage Design > EMC SourceOne storage design layout > Native archive temporary space, the working space assigned to an archive server can consume a lot of space, and, as a result, perhaps warrants more space than the 1 GB recommended in the installation documentation. In the following graph, it can also be seen that the I/O write traffic generated during a journaling test is not always trivial, with peaks of over 350 IOPS. No reads were observed on the same LUN. 54

55 Section D: CLARiiON performance data CLARiiON CX4 storage processor utilization The following graph shows the SP utilization on the CLARiiON CX4-480 array in the heaviest of the tests carried out. The two windows where historical archiving and shortcutting are running in addition to user load are easily identifiable. Shortcutting, in addition to user load, had the greatest impact on SP utilization, bringing it to a total of just over 50 percent. SP I/O throughput also peaks when shortcutting is running in combination with user load, with the array delivering approximately 9,000 IOPS per second at that point. The following graph shows the effect of placing both the SQL Database LUN and the EMC SourceOne Message Center LUN on the same SP. 55

56 The following graph shows the throughput for these two LUNs in the same picture, to illustrate the amount of I/O being generated by these two LUNs alone. While this did not affect performance, it suggests a best practice of keeping the SQL database LUN and the EMC SourceOne Message Center on separate SPs wherever possible. 56