Hitachi Virtual Storage Platform Dynamically Provisioned 160,000 Mailbox Exchange 2010 Mailbox Resiliency Storage Solution

Transcription

1 1 Hitachi Virtual Storage Platform Dynamically Provisioned 160,000 Mailbox Exchange 2010 Mailbox Resiliency Storage Solution Tested with: ESRP Storage Version 3.0 Test Date: February April 2013 Month Year

2 Notices and Disclaimer Copyright 2013 Hitachi Data Systems Corporation. All rights reserved. The performance data contained herein was obtained in a controlled isolated environment. Actual results that may be obtained in other operating environments may vary significantly. While Hitachi Data Systems Corporation has reviewed each item for accuracy in a specific situation, there is no guarantee that the same results can be obtained elsewhere. All designs, specifications, statements, information and recommendations (collectively, "designs") in this manual are presented "AS IS," with all faults. Hitachi Data Systems Corporation and its suppliers disclaim all warranties, including without limitation, the warranty of merchantability, fitness for a particular purpose and non-infringement or arising from a course of dealing, usage or trade practice. In no event shall Hitachi Data Systems Corporation or its suppliers be liable for any indirect, special, consequential or incidental damages, including without limitation, lost profit or loss or damage to data arising out of the use or inability to use the designs, even if Hitachi Data Systems Corporation or its suppliers have been advised of the possibility of such damages. This document has been reviewed for accuracy as of the date of initial publication. Hitachi Data Systems Corporation may make improvements and/or changes in product and/or programs at any time without notice. 1

3 Table of Contents Overview... 3 Disclaimer... 3 Features... 4 Solution Description... 5 Targeted Customer Profile Test Deployment Replication Configuration Best Practices Core Storage Storage-based Replication Backup Strategy Test Results Summary Reliability Storage Performance Results Backup and Recovery Performance Conclusion Appendix A RAID 6 Drive Failure and Rebuild Appendix B Test Reports Performance Test Result: CB Performance Test Checksums Result: CB Stress Test Result: CB Stress Test Checksums Result: CB Backup Test Result: CB Soft Recovery Test Result: CB Soft Recovery Test Performance Result: CB

4 Hitachi Virtual Storage Platform Dynamically Provisioned 160,000 Mailbox Exchange 2010 Mailbox Resiliency Storage Solution Tested with: ESRP Storage Version 3.0 Test Date: February April 2013 Overview This document provides information on a Microsoft Exchange Server 2010 mailbox resiliency storage solution that uses Hitachi Virtual Storage Platform storage systems with Hitachi Dynamic Provisioning. This solution is based on the Microsoft Exchange Solution Reviewed Program (ESRP) Storage program. For more information about the contents of this document or Hitachi Data Systems best practice recommendations for Microsoft Exchange Server 2010 storage design, see Hitachi Data Systems Microsoft Exchange Solutions Web page. The ESRP Storage program was developed by Microsoft Corporation to provide a common storage testing framework for vendors to provide information on its storage solutions for Microsoft Exchange Server software. For more information about the Microsoft ESRP Storage program, see TechNet s overview of the program. Disclaimer This document has been produced independently of Microsoft Corporation. Microsoft Corporation expressly disclaims responsibility for, and makes no warranty, express or implied, with respect to the accuracy of the contents of this document. The information contained in this document represents the current view of Hitachi Data Systems on the issues discussed as of the date of publication. Due to changing market conditions, it should not be interpreted to be a commitment on the part of Hitachi Data Systems, and Hitachi Data Systems cannot guarantee the accuracy of any information presented after the date of publication. 3

5 Features The purpose of this testing was to measure the ESRP 3.0 results on a Microsoft Exchange 2010 environment with 160,000 users and sixteen servers. This testing used Hitachi Virtual Storage Platform with Hitachi Dynamic Provisioning in a two-pool RAID-6 (6D+2P) - one for databases and one for logs - resiliency configuration. These results help answer questions about the kind of performance capabilities to expect with a large-scale Exchange deployment on Hitachi Virtual Storage Platform. Testing used sixteen Hitachi Compute Blade 2000 server blades in two chassis, each with the following: 64 GB of RAM Two quad-core Intel Xeon X GHz CPUs Two dual-port 8 Gb/sec Fibre Channel PCIe HBA (Emulex LPe1205-HI, using two port per HBA) located in the chassis expansion tray Microsoft Windows Server 2008 R2 Enterprise This solution includes Exchange 2010 Mailbox Resiliency by using the database availability group (DAG) feature. This tested configuration uses sixteen DAGs, each containing twenty-four database copies and two servers (one simulated). The test configuration was capable of supporting 160,000 users with a 0.12 IOPS per user profile and a user mailbox size of 1 GB. Hitachi Virtual Storage Platform with the following was used for these tests: GB 10K RPM SAS disks 480 GB of cache 32 8 Gb/sec paths used Hitachi Virtual Storage Platform delivers proven enterprise class functionality advanced virtualization of externally attached storage, logical partitioning, thin provisioning and universal replication with the industry s most reliable, scalable and highest performing storage services platform. Virtual Storage Platform is a large-sized, high-performance, highly reliable enterprise-class storage system that can scale to 2,048 disks and over 65,000 logical units (LUs) while capable of maintaining 100 percent data availability. Virtual Storage Platform is highly suitable for a variety of applications and host platforms that support the most demanding workloads. With internal and external storage virtualization capabilities, advanced replication technologies, tiered storage features and a tightly integrated management suite, Virtual Storage Platform is fully capable of serving as the core underlying storage platform of high performance Exchange Server 2010 architectures, while maintaining the ability to support additional workloads of an organization such as SQL Server and SharePoint Server. 4

6 Solution Description Deploying Microsoft Exchange Server 2010 requires careful consideration of all aspects of the solution architecture. Host servers need to be configured so that they are robust enough to handle the required Exchange load. The storage solution must be designed to provide the necessary performance while also being reliable and easy to administer. Of course, an effective backup and recovery plan should be incorporated into the solution as well. The aim of this solution report is to provide a tested configuration that uses Hitachi Virtual Storage Platform to meet the needs of a large Exchange Server deployment. This solution uses Hitachi Dynamic Provisioning, which is enabled on Hitachi Virtual Storage Platform via a license key. In the most basic sense, Hitachi Dynamic Provisioning is similar to the use of a host-based logical volume manager (LVM), but with several additional features available within Hitachi Virtual Storage Platform and without the need to install software on the host or incur host processing overhead. Hitachi Dynamic Provisioning is a superior solution by providing for one or more pools of wide striping across many RAID groups within Hitachi Virtual Storage Platform. One or more Hitachi Dynamic Provisioning virtual volumes (DPVols) of a user-specified logical size (with no initial physical space allocated) are created and associated with a single pool. Primarily, Hitachi Dynamic Provisioning is deployed to avoid the routine issue of hot spots that occur on logical units (LUs) from individual RAID groups when the host workload exceeds the IOPS or throughput capacity of that RAID group. By using many RAID groups as members of a striped Hitachi Dynamic Provisioning pool underneath the virtual or logical volumes seen by the hosts, a host workload is distributed across many RAID groups, which provides a smoothing effect that dramatically reduces hot spots and results in fewer mailbox moves for the Exchange administrator. Hitachi Dynamic Provisioning also carries the side benefit of thin provisioning, where physical space is only assigned from the pool to the DPVol as needed. Space is allocated as needed as 42 MB pool pages to that DPVol s logical block address range. A pool can also be dynamically expanded by adding more RAID groups without disruption or requiring downtime. Upon expansion, a pool can be rebalanced easily so that the data and workload are wide striped evenly across the current and newly added RAID groups that make up the pool. High availability is also a part of this solution with the use of database availability groups (DAG), which is the base component of the high availability and site resilience framework built into Microsoft Exchange Server A DAG is a group of up to 16 mailbox servers that host a set of databases and logs and use continuous replication to provide automatic database-level recovery from failures that affect individual servers or databases. Any server in a DAG can host a copy of a mailbox database from any other server in the DAG. When a server is added to a DAG, it monitors and works with the other servers in the DAG to provide automatic recovery delivering a robust, highly available Exchange solution without the administrative complexities of traditional failover clustering. For more information about the DAG feature in Exchange Server 2010, see This solution includes two copies of each Exchange database using sixteen DAGs, with each DAG configured with two server blades (one simulated) that host active mailboxes in twenty-four databases. To target the 160,000 user resiliency solution, a Hitachi Virtual Storage Platform storage system was configured with 2048 disks (maximum 2048). Sixteen servers (one per DAG) were used, with each server configured with 10,000 mailboxes. There were 24 active databases and the simulated database copies for the tests. 5

7 Each DAG contained two copies of the databases hosted by that DAG: A local, active copy on a server connected to the primary Hitachi Virtual Storage Platform A passive copy (simulated) on another server connected to a second Hitachi Virtual Storage Platform (simulated). This recommended configuration can support both high-availability and disaster-recovery scenarios when the active and passive database copies are allocated among both DAG members and dispersed across both storage systems. Each simulated DAG server node in this solution maintains a mirrored configuration and possesses adequate capacity and performance capabilities to support the second set of replicated databases. Figure 1 illustrates the two systems that make up the simulated DAG configuration. For more information, see the Hitachi Data Systems Storage Systems web page. Figure 1 6

8 This solution enables organizations to consolidate Exchange Server 2010 DAG deployments on two Hitachi Virtual Storage Platform storage systems. Using identical hardware and software configurations guarantees that an active database and its replicated copy do not share storage paths, disk spindles or storage controllers, making it a very reliable, high-performing, highly available Exchange Server 2010 solution that is cost effective and easy to manage. This helps ensure that performance and service levels related to storage are maintained regardless of which server is hosting the active database. If further protection is needed in a production environment, additional Exchange Server 2010 mailbox servers can be easily added to support these failover scenarios. The disks in Hitachi Virtual Storage Platform were organized into parity groups for use by databases or logs. There were GB 10K RPM SAS disks used in these tests configured as 256 RAID-6 (6D+2P) parity groups for the Exchange databases and logs. Each parity group had two LDEVs of 1.57 TB configured. There were 256 RAID-6 (6D+2P) SAS parity groups. Each parity group contained two 1.57 TB LDEVs. The 460 LDEVs from parity groups 1-1 to 29-6 were added to Hitachi Dynamic Provisioning Pool-0 ( Pool) and the 52 LDEVs from Parity Group 29-7 to were added to Hitachi Dynamic Provisioning Pool-1 (Log Pool). There were 384 V-Vol Groups (DPVol containers) created from Hitachi Dynamic Provisioning Pool-0 ( Pool) and each group had one 1910 GB DPVol created in it. Similarly, from Hitachi Dynamic Provisioning Pool-1 (Log Pool) 384 V-Vol Groups were created and each group had one 191 GB DPVol in it. The DPVols and Log DPVols were then assigned to the hosts as LUNs. 7

9 Table 1 outlines the port layout for the primary storage and servers. An identical configuration would be deployed on the replicated storage and servers for this solution. Table 1. Hitachi Virtual Storage Platform Ports to Server Mapping Configuration Server Primary path Secondary path CB10 1G 1Q 1H 1R CB11 1Q 1G 1R 1H CB12 1A 1J 1B 1K CB13 1J 1A 1K 1B CB22 2C 2L 2D 2M CB23 2L 2C 2M 2D CB24 1C 1L 1D 1M CB37 1L 1C 1M 1D CB14 2E 2N 2F 2P CB15 2N 2E 2P 2F CB16 1E 1N 1F 1P CB17 1N 1E 1P 1F CB18 2G 2Q 2H 2R CB19 2Q 2G 2R 2H CB20 2A 2J 2B 2K CB21 2J 2A 2K 2B 8

10 Table 2 outlines the port layout with the database DPVol assignments for the primary storage and servers. An identical configuration would be deployed on the replicated storage and servers for this solution. Table 2. Hitachi Virtual Storage Platform Ports to DPVol Layout Port DB DPVols 1G s :50-11:5B 1H s :5C-11:67 1Q s :68-11:73 1R s :74-11:7F 1A s :30-10:3B 1B s :3C-10:47 1J s :48-10:53 1K s :54-10:5F 2C s :60-10:6B 2D s :6C-10:77 2L s :78-10:83 2M s :84-10:8F 1C s :90-10:9B 1D s :9C-10:A7 1L s :A8-10:B3 1M s :B4-10:BF 2E s :C0-10:CB 2F s :CC-10:D7 2N s :D8-10:E3 2P s :E4-10:EF 1E s :F0-10:FB 1F s :FC-10:07 1N s :08-11:13 1P s :14-11:1F 2G s :20-11:2B 2H s :2C-11:37 2Q s :38-11:43 2R s :44-11:4F 2A s :00-10:0B 2B s :0C-10:17 2J s :18-10:23 2K s :24-10:2F 9

11 Table 3 outlines the port layout with the log DPVol assignments for the primary storage and servers. An identical configuration would be deployed on the replicated storage and servers for this solution. Table 3. Hitachi Virtual StoragePlatform Ports to Log DPVol Layout Port Log Log DPVols 1G Logs :C0-10:CB 1H Logs :CC-10:D7 1Q Logs :D8-10:E3 1R Logs :E4-10:EF 1A Logs :F0-10:FB 1B Logs :FC-10:07 1J Logs :08-11:13 1K Logs :14-11:1F 2C Logs :20-11:2B 2D Logs :2C-11:37 2L Logs :38-11:43 2M Logs :44-11:4F 1C Logs :00-10:0B 1D Logs :0C-10:17 1L Logs :18-10:23 1M Logs :24-10:2F 2E Logs :40-12:4B 2F Logs :4C-12:57 2N Logs :58-12:63 2P Logs :64-12:6F 1E Logs :70-12:7B 1F Logs :7C-12:87 1N Logs :88-12:93 1P Logs :94-12:9F 2G Logs :A0-12:AB 2H Logs :AC-12:B7 2Q Logs :B8-12:C3 2R Logs :C4-12:CF 2A Logs :80-11:8B 2B Logs :8C-11:97 2J Logs :98-11:A3 2K Logs :A4-11:AF 10

12 Table 4 provides the detailed specifications for the storage configuration which uses RAID-6 (6D+2P) groups and 600GB 10K disks. Dynamic Provisioning Pool 0 is dedicated for the databases and Dynamic Provisioning Pool 1 is dedicated for the logs. Table 4. Hitachi Virtual Storage Platform Configuration Details Host Pool Port DPVol Size (GB) RAID Level Description CB10 CB11 CB12 CB13 CB22 CB23 CB24 CB37 CB14 CB15 CB16 CB17 CB18 CB19 CB20 CB21 CB10 0 1G/1Q 11:50-11:5B 1910 RAID-6 s H/1R 11:5C-11: RAID-6 s Q/1G 11:68-11: RAID-6 s R/1H 11:74-11:7F 1910 RAID-6 s A/1J 10:30-10:3B 1910 RAID-6 s B/1K 10:3C-10: RAID-6 s J/1A 10:48-10: RAID-6 s K/1B 10:54-10:5F 1910 RAID-6 s C/2L 10:60-10:6B 1910 RAID-6 s D/2M 10:6C-10: RAID-6 s L/2C 10:78-10: RAID-6 s M/2D 10:84-10:8F 1910 RAID-6 s C/1L 10:90-10:9B 1910 RAID-6 s D/1M 10:9C-10:A RAID-6 s L/1C 10:A8-10:B RAID-6 s M/1D 10:B4-10:BF 1910 RAID-6 s E/2N 10:C0-10:CB 1910 RAID-6 s F/2P 10:CC-10:D RAID-6 s N/2E 10:D8-10:E RAID-6 s P/2F 10:E4-10:EF 1910 RAID-6 s E/1N 10:F0-10:FB 1910 RAID-6 s F/1P 10:FC-10: RAID-6 s N/1E 11:08-11: RAID-6 s P/1F 11:14-11:1F 1910 RAID-6 s G/2Q 11:20-11:2B 1910 RAID-6 s H/2R 11:2C-11: RAID-6 s Q/2G 11:38-11: RAID-6 s R/2H 11:44-11:4F 1910 RAID-6 s A/2J 10:00-10:0B 1910 RAID-6 s B/2K 10:0C-10: RAID-6 s J/2A 10:18-10: RAID-6 s K/2B 10:24-10:2F 1910 RAID-6 s G/1Q 12:DO-12:DB 191 RAID-6 Logs H/1R 12:DC-12:E7 191 RAID-6 Logs

13 CB11 CB12 CB13 CB22 CB23 CB24 CB37 CB14 CB15 CB16 CB17 CB18 CB19 CB20 CB21 1 1Q/1G 12:E8-12:F3 191 RAID-6 Logs R/1H 12:F4-12:FF 191 RAID-6 Logs A/1J 11:B0-11:BB 191 RAID-6 Logs B/1K 11:BC-11:C7 191 RAID-6 Logs J/1A 11:C8-11:D3 191 RAID-6 Logs K/1B 11:D4-11:DF 191 RAID-6 Logs C/2L 11:E0-11:EB 191 RAID-6 Logs D/2M 11:EC-11:F7 191 RAID-6 Logs L/2C 11:F8-12: RAID-6 Logs M/2D 12:04-12:0F 191 RAID-6 Logs C/1L 12:10-12: RAID-6 Logs D/1M 12:1C-12: RAID-6 Logs L/1C 12:28-12: RAID-6 Logs M/1D 12:34-12:3F 191 RAID-6 Logs E/2N 12:40-12:4B 191 RAID-6 Logs F/2P 12:4C-12: RAID-6 Logs N/2E 12:58-12: RAID-6 Logs P/2F 12:64-12:6F 191 RAID-6 Logs E/1N 12:70-12:7B 191 RAID-6 Logs F/1P 12:7C-12: RAID-6 Logs N/1E 12:88-12: RAID-6 Logs P/1F 12:94-12:9F 191 RAID-6 Logs G/2Q 12:A0-12:AB 191 RAID-6 Logs H/2R 12:AC-12:B7 191 RAID-6 Logs Q/2G 12:B8-12:C3 191 RAID-6 Logs R/2H 12:C4-12:CF 191 RAID-6 Logs A/2J 11:80-11:8B 191 RAID-6 Logs B/2K 11:8C-11: RAID-6 Logs J/2A 11:98-11:A3 191 RAID-6 Logs K/2B 11:A4-11:AF 191 RAID-6 Logs

14 The ESRP Storage program focuses on storage solution testing to address performance and reliability issues with storage design. However, storage is not the only factor to take into consideration when designing a scale-up Exchange solution. These factors also affect server scalability: Server processor utilization Server physical and virtual memory limitations Resource requirements for other applications Directory and network service latencies Network infrastructure limitations Replication and recovery requirements Client usage profiles These factors are all beyond the scope of the ESRP Storage program. Therefore, the number of mailboxes hosted per server as part of the tested configuration might not necessarily be viable for some customer deployments. For more information about identifying and addressing performance bottlenecks in an Exchange system, see Microsoft's Troubleshooting Microsoft Exchange Server Performance. Targeted Customer Profile This solution is designed for medium to large organizations that plan to consolidate their Exchange Server 2010 storage on high-performance, high-reliability storage systems. This configuration is designed to support 160,000 Exchange users with the following specifications: Thirty-two Exchange servers (sixteen tested, sixteen simulated for the database copies) Sixteen database availability groups (DAG) each with two servers (one simulated) and two copies per database Two Hitachi Virtual Storage Platform (one tested) 0.1 IOPS per user (0.12 tested for 20 percent growth) 1 GB mailbox size Mailbox resiliency provides high-availability and used as primary data protection mechanism. Hitachi Virtual Storage Platform RAID protection against physical failure or loss. 24x7 background database maintenance enabled. 13

15 Test Deployment The following tables summarize the testing environment. Table 5. Simulated Exchange Configuration Number of Exchange mailboxes simulated 160,000 Number of database availability groups (DAGs) 16 Number of servers per DAG 2 (1 simulated) Number of active mailboxes per server 10,000 Number of databases per host 24 Number of copies per database 2 Number of mailboxes per database Simulated profile: s per second per mailbox (IOPS, include 20% headroom) 0.12 LU size Log LU siz Total database size for performance testing 1910 GB 191 GB 160,000 GB % storage capacity used by Exchange database** 21.7% **Storage performance characteristics change based on the percentage utilization of the individual disks. Tests that use a small percentage of the storage (~25%) might exhibit reduced throughput if the storage capacity utilization is significantly increased beyond what was tested for this paper. 14

16 Table 6. Storage Hardware Storage connectivity (Fibre Channel, SAS, SATA, iscsi) Storage model and OS/firmware revision Storage cache Fibre Channel 1 Hitachi Virtual Storage Platform Firmware: /00 WHQL listing: Hitachi Virtual Storage Platform 480 GB Number of storage controllers 1 Number of storage ports 32 Maximum bandwidth of storage connectivity to host 256 Gb/sec (32 8Gb/sec ports) Switch type/model/firmware revision HBA model and firmware Number of HBAs per host Host server type Brocade 5300, Fabric OS v7.0.1b Emulex LPe1205-HI FW : 1.11X14 2 dual-ported HBA per host, 2 8 Gb/sec port used per HBA Hitachi Compute Blade E55A GHz Intel Xeon Processors, 64 GB memory Total number of disks tested in solution 2048 Maximum number of spindles that can be hosted in the storage 2048 Table 7. Storage Software HBA driver Storport Miniport HBA QueueTarget setting 0 HBA QueueDepth setting 32 Multipathing Host OS Hitachi Dynamic Link Manager v Microsoft Windows Server 2008 R2 Enterprise ESE.dll file version Replication solution name/version N/A 15

17 Table 8. Storage Disk Configuration (Mailbox Store Disks) Disk type, speed and firmware revision Raw capacity per disk (GB) Number of physical disks in test SAS Disk 600 GB 10KAJ-AJ 600 GB 1840 (dynamic provisioning pool) Total raw storage capacity (GB) 1,104,000 Disk slice size (GB) Number of slices per LU or number of disks per LU RAID level Total formatted capacity N/A N/A RAID-6 (6D+2P) at storage level 738,300 GB Storage capacity utilization 66.9% capacity utilization 66.4% Table 9. Storage Disk Configuration (Transaction Log Disks) Disk type, speed and firmware revision Raw capacity per disk (GB) Number of spindles in test SAS Disk 600 GB 10K AJ-AJ 600 GB 208(dynamic provisioning pool) Total raw storage capacity (GB) 124,800 Disk slice size (GB) Number of slices per LU or number of disks per LU RAID level Total formatted capacity N/A N/A RAID-6 (6D+2P) at storage level 83,460 GB 16

18 Replication Configuration Table 10. Replication Configuration Replication mechanism Exchange Server 2010 Availability Group (DAG) Number of links 2 Simulated link distance Link type Link bandwidth N/A IP GigE (1Gb/sec) Table 11. Replicated Storage Hardware Storage connectivity (Fibre Channel, SAS, SATA, iscsi) Storage model and OS/firmware revision Storage cache Fibre Channel 1 Hitachi Virtual Storage Platform Firmware: /00 WHQL listing: Hitachi Virtual Storage Platform 480 GB Number of storage controllers 1 Number of storage ports 32 Maximum bandwidth of storage connectivity to host 256 Gb/sec (32 8 Gb/sec ports) Switch type/model/firmware revision HBA model and firmware Number of HBAs per host Host server type Brocade 5300, Fabric OS v7.0.1b Emulex LPe1205-HI FW : 1.11X14 2 dual-ported HBA per host, 2 8 Gb/sec port used per HBA Hitachi Compute Blade E55A GHz Intel Xeon Processors, 64 GB memory Total number of disks tested in solution 2048 Maximum number of spindles that can be hosted in the storage

19 Table 12. Replicated Storage Software HBA driver Storport Miniport HBA QueueTarget setting 0 HBA QueueDepth setting 32 Multipathing Host OS Hitachi Dynamic Link Manager v Microsoft Windows Server 2008 R2 Enterprise ESE.dll file version Replication solution name/version N/A Table 13. Replicated Storage Disk Configuration (MailboxStore Disks) Disk type, speed and firmware revision SAS Disk 2TB 7.2K AJ-AJ Raw capacity per disk (GB) Number of physical disks in test 600 GB 1840 (dynamic provisioning pool) total raw storage capacity (GB) 1,104,000 Disk slice size (GB) N /A Number of slices per LU or number of disks per LU Raid level Total formatted capacity N/A RAID-6 (6D+2P) at storage level 738,300 GB Storage capacity utilization 66.9% capacity utilization 66.4% Table 14. Replicated Storage Disk Configuration (Transactional Log Disks) Disk type, speed and firmware revision Raw capacity per disk (GB) Number of spindles in test SAS Disk 2TB 7.2KAJ- AJ 600 GB 208 (dynamic provisioning pool) Total raw storage capacity (GB) 124,800 Disk slice size (GB) Number of slices per LU or number of disks per LU Raid level Total formatted capacity N/A N/A RAID-6 (6D+2P) at storage level 83,460 GB 18

20 Best Practices Microsoft Exchange Server 2010 is a disk-intensive application. It presents two distinct workload patterns to the storage, with 32KB random read/write operations to the databases, and sequential write operations of varying size (between 512 bytes up to the log buffer size) to the transaction logs. For this reason, designing an optimal storage configuration can prove challenging in practice. Based on the testing run using the ESRP framework, Hitachi Data Systems recommends these best practices to improve the performance of Hitachi Virtual Storage Platform running Exchange For more information about Exchange 2010 best practices for storage design, see the Microsoft TechNet article Mailbox Server Storage Design. Core Storage 1. When formatting a newly partitioned LU, Hitachi Data Systems recommends setting the ALU to 64K for the database files and 4K for the log files. 2. Disk alignment is no longer required when using Microsoft Windows Server Keep the Exchange workload isolated from other applications. Mixing another intensive application whose workload differs from Exchange can cause the performance for both applications to degrade. 4. Use Hitachi Dynamic Link Manager multipathing software to provide fault tolerance and high availability for host connectivity. 5. Use Hitachi Dynamic Provisioning to simplify storage management of the Exchange database and log volumes. 6. Due to the difference in patterns, isolate the Exchange database from the log groups. Create a dedicated Hitachi Dynamic Provisioning pool for the databases and a separate pool for the logs. 7. The log LUs should be at least 10 percent of the size of the database LUs. 8. Hitachi Data Systems does not recommend using LU concatenation. 9. Hitachi Data Systems recommends implementing Mailbox Resiliency using the Exchange Server 2010 Availability Group feature. 10. Ensure that each DAG maintains at least two database copies to provide high availability. 11. Isolate active databases and their replicated copies in separate dynamic provisioning pools or ensure that they are located on a separate Hitachi Virtual Storage Platforms. 12. Use fewer, larger LUs for Exchange 2010 databases (up to 2TB) with Background Maintenance (24x7) enabled. 13. Size storage solutions for Exchange based primarily on performance criteria. The number of disks, RAID level and percent utilization of each disk directly affect the level of achievable performance. Factor in capacity requirements only after performance is addressed. 14. Disk size is unrelated to performance with regards to IOPS or throughput rates. Disk size is related to the usable capacity of all of the LUs from a RAID group, which is a choice users make. 19

21 15. The number of spindles, coupled with the RAID level, determines the physical IOPS capacity of the RAID group and all of its LUs. If the disk has too few spindles, the response times grow to large values very quickly. Storage-based Replication N/A Backup Strategy N/A 20

22 Test Results Summary This section provides a high-level summary of the test data from ESRP and the link to the detailed HTML reports that are generated by ESRP testing framework. Reliability A number of tests in the framework check reliability spanning a 24-hour window. The goal is to verify the storage can handle high load for a long period of time. Following these stress tests, both log and database files are analyzed for integrity to ensure that no database or log corruption occurs. No errors were reported in the event log file for the storage reliability testing. No errors were reported for the database and log checksum process. If done, no errors were reported during the backup to disk test process. No errors were reported for the database checksum on the remote storage database. Storage Performance Results Primary storage performance testing exercises the storage with maximum sustainable Exchange type of for two hours. The test shows how long it takes for the storage to respond to an under load. The following data is the sum of all of the logical disk s and average of all the logical disks latency in the two-hour test duration. Individual Server Metrics These individual server metrics show the sum of the across the storage groups and the average latency across all storage groups on a per-server basis. Table 15. Individual Server Metrics for Exchange Server (CB10) Disk Transfers Per Second 2772 Disk Reads Per Second 1571 Disk Writes Per Second 1201 Disk Read Latency (ms) 7.1 Disk Write Latency (ms) 3.5 Transaction Log Log Disk Writes Per Second 1099 Log Disk Write Latency (ms)

23 Table 16. Individual Server Metrics for Exchange Server (CB11) Disk Transfers Per Second 2253 Disk Reads Per Second 1273 Disk Writes Per Second 980 Disk Read Latency (ms) 5.8 Disk Write Latency (ms) 2.6 Transaction Log Log Disk Writes Per Second 915 Log Disk Write Latency (ms) 0.8 Table 17. Individual Server Metrics for Exchange Server (CB12) Disk Transfers Per Second 2306 Disk Reads Per Second 1303 Disk Writes Per Second 1004 Disk Read Latency (ms) 5.6 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 941 Log Disk Write Latency (ms) 0.7 Table 18. Individual Server Metrics for Exchange Server (CB13) Disk Transfers Per Second 2304 Disk Reads Per Second 1302 Disk Writes Per Second 1002 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 937 Log Disk Write Latency (ms)

24 Table 19. Individual Server Metrics for Exchange Server (CB22) Disk Transfers Per Second 2752 Disk Reads Per Second 1561 Disk Writes Per Second 1192 Disk Read Latency (ms) 7.0 Disk Write Latency (ms) 3.5 Transaction Log Log Disk Writes Per Second 1087 Log Disk Write Latency (ms) 0.8 Table 20. Individual Server Metrics for Exchange Server (CB23) Disk Transfers Per Second 2768 Disk Reads Per Second 1570 Disk Writes Per Second 1198 Disk Read Latency (ms) 7.1 Disk Write Latency (ms) 3.5 Transaction Log Log Disk Writes Per Second 1093 Log Disk Write Latency (ms) 0.8 Table 21. Individual Server Metrics for Exchange Server (CB24) Disk Transfers Per Second 2735 Disk Reads Per Second 1551 Disk Writes Per Second 1184 Disk Read Latency (ms) 7.1 Disk Write Latency (ms) 3.5 Transaction Log Log Disk Writes Per Second 1082 Log Disk Write Latency (ms)

25 Table 22. Individual Server Metrics for Exchange Server (CB37) Disk Transfers Per Second 2813 Disk Reads Per Second 1595 Disk Writes Per Second 1218 Disk Read Latency (ms) 7.1 Disk Write Latency (ms) 3.5 Transaction Log Log Disk Writes Per Second 1110 Log Disk Write Latency (ms) 0.8 Table 23. Individual Server Metrics for Exchange Server (CB14) Disk Transfers Per Second 2299 Disk Reads Per Second 1298 Disk Writes Per Second 1001 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 937 Log Disk Write Latency (ms) 0.7 Table 24. Individual Server Metrics for Exchange Server (CB15) Disk Transfers Per Second 2283 Disk Reads Per Second 1290 Disk Writes Per Second 993 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 927 Log Disk Write Latency (ms)

26 Table 25. Individual Server Metrics for Exchange Server (CB16) Disk Transfers Per Second 2301 Disk Reads Per Second 1299 Disk Writes Per Second 1001 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 936 Log Disk Write Latency (ms) 0.7 Table 26. Individual Server Metrics for Exchange Server (CB17) Disk Transfers Per Second 2285 Disk Reads Per Second 1291 Disk Writes Per Second 994 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 930 Log Disk Write Latency (ms) 0.7 Table 27. Individual Server Metrics for Exchange Server (CB18) Disk Transfers Per Second 2295 Disk Reads Per Second 1296 Disk Writes Per Second 998 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 934 Log Disk Write Latency (ms)

27 Table 28. Individual Server Metrics for Exchange Server (CB19) Disk Transfers Per Second 2278 Disk Reads Per Second 1287 Disk Writes Per Second 992 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 928 Log Disk Write Latency (ms) 0.7 Table 29. Individual Server Metrics for Exchange Server (CB20) Disk Transfers Per Second 2295 Disk Reads Per Second 1297 Disk Writes Per Second 999 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 933 Log Disk Write Latency (ms) 0.7 Table 30. Individual Server Metrics for Exchange Server (CB21) Disk Transfers Per Second 2275 Disk Reads Per Second 1285 Disk Writes Per Second 990 Disk Read Latency (ms) 5.7 Disk Write Latency (ms) 2.5 Transaction Log Log Disk Writes Per Second 925 Log Disk Write Latency (ms)

28 Aggregate Performance Across All Servers Metric The aggregate performance across all server metrics shows the sum of across all servers in the solution and the average latency across all servers in the solution. Table 31. Aggregate Performance for Exchange Server 2010 Disk Transfers Per Second Disk Reads Per Second Disk Writes Per Second Disk Read Latency (ms) 6.12 Disk Write Latency (ms) 2.83 Transaction Log Log Disk Writes Per Second Log Disk Write Latency (ms) 0.78 Backup and Recovery Performance This section has two tests: The first measures the sequential read rate of the database files and the second measures recovery/replay performance (playing transaction logs in to the database). Read-only Performance This test measures the maximum rate at which databases can be backed up via VSS. The following tables show the average rate for a single database file. Table 32. Read-only Performance MB Read Per Second Per MB Read Per Second Total Per Server Transaction Log Recovery/Replay Performance This test measures the maximum rate at which the log files can be played against the databases. The following table shows the average rate for 500 log files played in a single storage group. Each log file is 1MB in size. Table 33. Transaction Log Recovery/Replay Performance Time to Play One Log File (sec)

29 Conclusion This document details a tested and robust Exchange Server 2010 Resiliency solution capable of supporting 160,000 users with a 0.12 IOPS per user profile and user mailbox size of 1 GB using sixteen DAG s, each configured with 2 server nodes (one simulated). A Hitachi Virtual Storage Platform storage system, with 480 GB of cache and sixteen 8 Gb/sec Fibre Channel host paths, using Hitachi Dynamic Provisioning (with two pools) and GB 10K RPM SAS disks in a RAID-6 (6D+2P) configuration was used for these tests. Testing confirmed that Hitachi Virtual Storage Platform is more than capable of delivering the IOPS and capacity requirements needed to support the active and replicated databases for 160,000 Exchange mailboxes configured with the specified user profile, while maintaining additional headroom to support peak throughput. The solution outlined in this document does not include data protection components, such as VSS snapshot or clone backups, and relies on the built-in Mailbox Resiliency features of Exchange Server 2010 coupled with Hitachi Virtual Storage Platform RAID technology to provide high-availability and protection from logical and physical failures. Adding additional protection requirements may affect performance and capacity requirements of the underlying storage configuration, and as such need to be factored into the storage design accordingly. For more information to about planning Exchange Server 2010 storage architectures for Hitachi Virtual storage Platform, see This document is developed by Hitachi Data Systems and reviewed by the Microsoft Exchange product team. The test results and data presented in this document are based on the tests introduced in the ESRP test framework. Do not quote the data directly for pre-deployment verification. It is still necessary to validate the storage design for a specific customer environment. The ESRP program is not designed to be a benchmarking program; tests do not generate the maximum throughput for a given solution. Rather, it is focused on producing recommendations from vendors for Exchange application. Thus, do not use the data presented in this document for direct comparisons among the solutions 28

30 Appendix A RAID 6 Drive Failure and Rebuild These ESRP tests used RAID-6 (6D+2P) rather than RAID-5 (7D+1P) or RAID-10 (2D+2D or 4D+4D). RAID-6 (6D+2P) on Hitachi Virtual Storage Platform is more capacity efficient than RAID-10 (4D+4D). In the case of 8 disks, RAID-6 has a capacity efficiency of 75%. RAID-10 has a capacity efficiency of 50%. One downside with the use of parity RAID instead of mirrored and striped RAID-10 is that the internal write penalty for host random writes is higher. RAID-6 volumes require six physical s (3 reads, 3 writes) on the backend for every host write, while RAID-5 volumes require four physical disk s (2 reads, 2 writes). In comparison, RAID-10 volumes require two physical s (2 writes) on the backend for every host write. The other downside is the rebuild time for the RAID group after a sudden disk failure. Hitachi Virtual Storage Platform always scans the storage system looking for soft fails, because an excessive soft fail count is a predictor of a hard failure in the future. If the number of soft fails exceeds the user-set failure threshold in a 24-hour period, Hitachi Virtual Storage Platform does the following, in order: 1. Executes a disk-to-disk copy to a global hot spare to avoid having to perform a rebuild. 2. Switches to using the spare disk and marks the source disk as failed. 3. Alerts maintenance people to replace the disk. If a hard fail of a disk in a RAID volume does occur, the following happens: If using RAID-10, the contents of the good disk are mirrored onto a spare disk. These hot spares are userdefined in several disk enclosures on a storage system. If using RAID-5 or RAID-6, all disks in the RAID group must be read to recreate the missing data or parity that was located on the failed disk onto the spare disk. This rebuild mode is called corrective copy. An associated array setting called [Drive] Restore Options determines how aggressive the rebuild operation is while there are still ongoing host s. This setting has three levels: aggressive, moderate, and background. 29

31 Appendix B Test Reports This appendix contains Jetstress test results for one of the servers used in testing this storage solution. These test results are representative of the results obtained for all of the servers tested. Performance Test Result: CB10 Test Summary Overall Test Result Machine Name Pass CB10 Test Description Test Start Time Test End Time Collection Start Time Collection End Time 4/5/2013 1:09:47 PM 4/5/2013 4:46:41 PM 4/5/2013 1:13:38 PM 4/5/2013 3:13:35 PM Jetstress Version ESE Version Operating System Windows Server 2008 R2 Enterprise Service Pack 1 ( ) Performance Log C:\VSP_C1B10_SAS10K_ESRP_R6_ 1 GB _mbox_10000 Users\Performance Test\Performance_2013_4_5_13_10_38.blg Sizing and Throughput Achieved Transactional per Second Target Transactional per Second 1200 Initial Size (bytes) Final Size (bytes) Files (Count) 24 30

32 Jetstress System Parameters Thread Count Minimum Cache Maximum Cache 3 (per database) MB MB Insert Operations 40% Delete Operations 20% Replace Operations 5% Read Operations 35% Lazy Commits 70% Run Background Maintenance True Number of Copies per 2 31

33 Configuration Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Log path: C:\logluns\log1 : C:\dbluns\db1\Jetstress edb Log path: C:\logluns\log2 : C:\dbluns\db2\Jetstress edb Log path: C:\logluns\log3 : C:\dbluns\db3\Jetstress edb Log path: C:\logluns\log4 : C:\dbluns\db4\Jetstress edb Log path: C:\logluns\log5 : C:\dbluns\db5\Jetstress edb Log path: C:\logluns\log6 : C:\dbluns\db6\Jetstress edb Log path: C:\logluns\log7 : C:\dbluns\db7\Jetstress edb Log path: C:\logluns\log8 : C:\dbluns\db8\Jetstress edb Log path: C:\logluns\log9 : C:\dbluns\db9\Jetstress edb Log path: C:\logluns\log10 : C:\dbluns\db10\Jetstress edb Log path: C:\logluns\log11 : C:\dbluns\db11\Jetstress edb Log path: C:\logluns\log12 : C:\dbluns\db12\Jetstress edb Log path: C:\logluns\log13 : C:\dbluns\db13\Jetstress edb Log path: C:\logluns\log14 : C:\dbluns\db14\Jetstress edb Log path: C:\logluns\log15 : C:\dbluns\db15\Jetstress edb Log path: C:\logluns\log16 : C:\dbluns\db16\Jetstress edb Log path: C:\logluns\log17 : C:\dbluns\db17\Jetstress edb Log path: C:\logluns\log18 32

34 : C:\dbluns\db18\Jetstress edb Instance Instance Instance Instance Instance Instance Log path: C:\logluns\log19 : C:\dbluns\db19\Jetstress edb Log path: C:\logluns\log20 : C:\dbluns\db20\Jetstress edb Log path: C:\logluns\log21 : C:\dbluns\db21\Jetstress edb Log path: C:\logluns\log22 : C:\dbluns\db22\Jetstress edb Log path: C:\logluns\log23 : C:\dbluns\db23\Jetstress edb Log path: C:\logluns\log24 : C:\dbluns\db24\Jetstress edb 33

35 Transactional Performance MSExchange ==> Instances Reads Latency (msec) Writes Latency (msec) Reads/sec Writes/sec Reads Bytes Writes Bytes Reads Latency (msec) Writes Latency (msec) Reads/sec Writes/sec Reads Bytes Writes Bytes Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance

36 Background Maintenance Performance MSExchange ==> Instances Maintenance IO Reads/sec Maintenance IO Reads Bytes Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance

37 Log Replication Performance MSExchange ==> Instances Reads/sec Reads Bytes Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance

38 Total Performance MSExchange ==> Instances Reads Latency (msec) Writes Latency (msec) Reads/sec Writes/sec Reads Bytes Writes Bytes Reads Latency (msec) Writes Latency (msec) Reads/sec Writes/sec Reads Bytes Writes Bytes Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance Instance