Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

Transcription

1 Roland Mueller Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks Solution Reference Number: BAPEXCX1344 Contents Introduction Business problem and business value Architectural overview Software and hardware overview Functional requirements Best practice and implementation guidelines Solution validation methodology Summary References and helpful links About the Author Appendix A: Bill of materials Appendix B: Configuring the RAID1 boot array on the ServeRAID M Appendix C: Configuring the RAID0 single-disk arrays on the ServeRAID M Appendix D: Creating volumes Appendix E: Microsoft Exchange Jetstress 2013 Stress Test Result Report Copyright Lenovo All rights reserved. ibm.com/redbooks 1

2 Introduction This document describes a scalable, reference architecture for a 4-node Microsoft Exchange Server 2013 mail environment that uses System x3650 M5 servers and internal storage. The solution as described supports a user population of 5,000 users with 3,500 MB mailboxes; however, when it is deployed in aggregate, it can support any number of mailboxes in multiples of 5,000. By using the native high availability features of Exchange Server 2013, the environment that is described in this publication allows administrators to eliminate traditional backup methods, which frees critical enterprise resources. In addition, the use of internal disks in a cache-enabled JBOD configuration drastically reduces the overall cost of the solution by eliminating SAN or DAS-based storage and reduces the need for storage administrators. This paper is intended to provide the planning, design considerations, and best practices for implementation. This document is for clients who are implementing Exchange Server 2013 and for IT engineers familiar with the hardware and software that make up this reference architecture. Also, the System x sales teams and their customers who are evaluating or pursuing Exchange Server 2013 solutions can benefit from this validated configuration. Comprehensive experience with the various reference configuration technologies is recommended before implementing this solution. Business problem and business value Today s IT managers are looking for efficient ways to grow their IT infrastructure, while simultaneously reducing administration and power costs. Add the requirement for high availability, and organizations must maintain a delicate balance between availability of the mail environment and the cost of purchasing and maintaining the solution. Rapidly responding to changing business needs with simple, fast deployment and configuration while maintaining systems and services directly corresponds to the vitality of your business. Natural disasters, malicious attacks, and even simple software upgrade patches can cripple services and applications until administrators resolve the problems and restore any backed up data. Business value The last three generations of Microsoft Exchange Server have seen dramatic improvements in native high availability and site resiliency features. Starting with the Cluster Continuous Replication foundational technology (which first appeared in Exchange Server 2007 and later evolved into the Database Availability Group in Exchange Server 2010), Exchange Server 2013 further improves availability and site resiliency by de-coupling the mailbox and Client Access Server (CAS) roles and simplifying the namespace requirements. In addition to the improvements in high availability and site resiliency, optimizations at the database level reduced IOPS requirements by 50% over Exchange Server This reduction in IOPS allows Exchange Server 2013 to use less expensive, larger capacity hard disk drives (HDDs). 2 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

3 Lenovo made a significant investment in the System x platform to use the improvements in Exchange Server The large memory capacity of the server improves search query performance and reduces IOPS, while the impressive number of internal drive slots provides organizations with the option of using internal disks for their mail environment rather than purchasing expensive DAS or SAN-based storage subsystems. The improved architecture of the x3650 M5 makes it a perfect fit. This System x3650 M5 solution for Exchange Server 2013 provides businesses with an affordable, interoperable, and reliable industry-leading solution. This reference architecture combines Microsoft software, System x servers, consolidated guidance, and validated configurations to provide a high level of redundancy and fault tolerance to ensure high availability of resources. Architectural overview The design consists of four x3650 M5 servers that are part of a database availability group (DAG), which spans two data centers. Two servers are in each of the data centers. Four copies of each mailbox database (two in each datacenter) are used to provide fault tolerance, site resiliency, and to eliminate the need for traditional backups. Figure 1 shows the overall architecture of the solution. Management Servers Witness Server Domain Controllers Primary Datacenter (Active) Management Servers Corporate Network (MAPI and Replication traffic on separate VLANs) Domain Controllers Secondary Datacenter (Passive) Layer 4 Load Balancer Network Switches Layer 4 Load Balancer Network Switches Database Availability Group Mailbox/CAS Server-1 Copy-1 Copy-2 Copy-3 Copy-4 Mailbox/CAS Server-2 Mailbox/CAS Server-3 Mailbox/CAS Server-4 Figure 1 Architectural overview For the storage configuration, the Exchange databases and logs are hosted on the servers internal disks rather than an external disk system. Each disk that is hosting Exchange mailbox databases and logs is defined as a discreet, single-disk RAID0 array, which creates a JBOD-like design while it uses the RAID controller s 2 GB cache for maximum performance. Each server has the mailbox and the CAS role installed (multi-role). System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 3

4 The native high availability features in Exchange Server 2013 help eliminate single points of failure without having to rely on hardware high availability features as heavily as in previous releases of Microsoft Exchange Server. Therefore, users have near-continuous access to business-critical resources. Multiple paths connect the servers to the networking infrastructure to maintain access to critical resources if there is a planned or unplanned outage. A hardware network load balancer is also used to balance network connections between the CAS servers. A witness server is at the primary site. The witness server gives the primary site three votes as opposed to two votes (the mailbox servers). If there is a WAN outage, the mailbox servers at the primary site remain active. Because all mail users are at the same site as the primary datacenter, no users lose access to . When the WAN is restored, replication resumes. Software and hardware overview A short summary of the software and hardware components that are used in this reference architecture is listed in this section. The reference configuration is constructed of the following enterprise-class components: Four System x3650 M servers in DAG cluster and installed with the Exchange Server Mailbox and CAS roles Microsoft Exchange Server 2013 SP1 Microsoft Windows Server 2012 R2 Together, these software and hardware components form a cost-effective solution that supports an Exchange Server 2013 mail environment that is flexible and scalable enough to support any number of user populations (in multiples of 5,000 users) when it is deployed in aggregate (four servers support 5,000 users, eight servers support 10,000 users, and so on). This design consists of four System x3650 M5 servers that are running the Microsoft Windows 2012 R2 operating system and installed with Microsoft Exchange Server 2013 SP1. System x3650 M5 At the core of this solution, the System x3650 M5 server delivers the performance and reliability that is required for business-critical applications, such as Exchange Server System x3650 M5 servers can be equipped with up to two 18-core Intel E v3 processors, and up to 3 TB of TruDDR4 of memory. Up to seven PCIe 3.0 expansion slots, four 1-Gb network ports, and an optional embedded dual-port 10 GbE network adapter provide ports for both your data and storage connections. The x3650 M5 includes an on-board RAID controller and the choice of spinning hot swap SAS or SATA disks and SFF hot swap solid-state drives (SSDs). The large number of HDD slots in the x3650 M5 makes it the perfect platform for running Microsoft Exchange Server mail environments from local disk. The x3650 M5 supports the following components: Up to inch Gen3 Simple Swap HDDs Up to inch Simple Swap HDDs Up to SAS/SATA 2.5-inch Gen3 Hot Swap HDDs Up to 12+2 SAS/SATA 3.5-inch Hot Swap HDDs 4 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

5 The x3650 M5 also supports remote management via the Integrated Management Module (IMM), which enables continuous management capabilities. All of these key features, including many that are not listed here, help solidify the dependability that customers are accustomed to with System x servers. Figure 2 shows the System x3650 M5. Figure 2 System x3650 M5 with inch drive bays at the front ServeRAID M5210 RAID controller The ServeRAID M5210 SAS/SATA controller for System x is part of the ServeRAID M Series family that offers a complete server storage solution consisting of RAID controllers, cache and flash modules, energy packs, and software feature upgrades in an ultra-flexible offerings structure. M5210 comes as a small form factor PCIe adapter. Two internal x4 HD Mini-SAS connectors provide connections to up to 32 internal drives (depending on the server model). To use cache, the optional 2 GB onboard data cache (DDR3 that is running at 1866 MHz) with battery backup upgrade is used. Figure 3 shows the ServeRAID M5210 Controller with an optional cache installed. Figure 3 ServeRAID M5210 SAS/SATA Controller with optional cache installed Note: When the ServeRAID M5210 RAID controller is used in a pure JBOD (just a bunch of disks) configuration, the controller cannot be installed with the optional onboard data cache (DDR3 that is running at 1866 MHz) with battery backup; the disk drives are passed directly through to the operating system. Pure JBOD deployments are significantly affected by the lack of battery-backed cache; therefore, to allow the use of cache, RAID0 is used to create discreet, single-disk arrays. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 5

6 Microsoft Windows Server 2012 R2 Windows Server 2012 R2 provides the enterprise with a scalable and highly elastic platform for mission-critical applications. The operating system supports up to 4 TB of RAM, 320 logical processors, and 64 nodes per cluster. It also includes updates to Active Directory to help the performance of applications such as Exchange. Deployment Figure 4 shows the hardware as it is deployed in the data centers. As shown in Figure 4, each rack includes two System x3650 M5 servers, two top-of-rack network switches, and a network load balancer. Primary Datacenter (Active) Secondary Datacenter (Passive) Top-of-Rack Networking Switches Top-of-Rack Networking Switches Network Load Balancer Network Load Balancer x3650 M5 servers x3650 M5 servers Figure 4 Deployed hardware Note: The x3650 M5 servers are the only hardware components that are covered in-depth in this reference architecture. Functional requirements This section describes the designed function of the reference architecture and the customer profile and design requirements. To demonstrate functionality and design decision points, this paper presents a case study of a fictitious organization with 5,000 employees. The employee population is at the same site as the primary data center. A single namespace spans both sites to use new site resiliency features that are included in Exchange Server System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

7 User profile The company determined the average number of s that are sent and received per day for each user is approximately 100, with an average size of 75 KB. Each user is assigned a 3500 MB mailbox. The organization requires a deleted item retention window of 14 days to give users ample time to recover unintentionally deleted s. High availability and site resiliency If an organization has multiple data centers, the Exchange Server infrastructure can be deployed in one site or distributed across two or more sites. Typically, the service level agreement that is in place determines the degree of high availability and the placement of the servers. In this solution, the organization has two data centers with a user population that is co-located with the primary datacenter. The organization determined that site resiliency is required; therefore, the Exchange Server 2013 design is based on a multiple site deployment with site resiliency. The secondary data center is passive and used for disaster recovery, if needed. The organization also decided a 24-hour recovery point objective is sufficient. Backup and recovery Exchange Server 2013 includes several native features that provide data protection that, when implemented correctly, can eliminate the need for traditional backups. Such backups often are used for disaster recovery, recovery of accidentally deleted items, long-term data storage, and point-in-time database recovery. Each of these scenarios is addressed with features in Exchange Server 2013, such as high availability database copies in a DAG, the Recoverable Items folder with the optional Hold Policy, archiving, multiple mailbox search, message retention features, and lagged database copies. If there is a server failure and recovery (or reseeding) is necessary, rebuilding a failed database might take hours or even days when Exchange Server 2013 native data protection features are used. Having a traditional backup can greatly reduce the time that is required to bring a failed database back online. However, the downsides to the use of backups are not insignificant; the administrative overhead, licensing costs, and the extra storage capacity that is required for the backup files can greatly increase the total cost of ownership of the solution. In this solution, the organization decided to forgo traditional backups in favor of the use of an Exchange Server 2013 native data protection strategy. Database copies Before you determine the number of database copies that are needed, it is important to understand the following types of database copies: High availability database copy This type of database copy has a log replay time of zero seconds. When a change is made in the active database copy, changes are immediately replicated to passive database copies. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 7

8 Lagged database copy This type of database copy has a pre-configured delay that is built into the log replay time. When a change is implemented in the active database copy, the logs are copied over to the server hosting the lagged database copy, but are not immediately implemented. This configuration provides point-in-time protection, which can be used to recover from logical corruption of a database (logical corruption occurs when data is added, deleted, or manipulated in a way the user or administrator did not expect). Lagged database copies allow up to 14 days of lagged data. Log replay time for lagged database copies: Lenovo recommends the use of a replay lag time of 72 hours. This lag time gives administrators time to detect logical corruption that occurred at the start of a weekend. Another factor to consider when you are choosing the number of database copies is serviceability of the hardware. If only one high availability database copy is present at each site, the administrator is required to switch over to the database copy that is hosted at a secondary datacenter when a server must be powered off for servicing. To prevent this issue, maintaining a second database copy at the same geographic location as the active database copy is a valid option to maintain hardware serviceability and reduce administrative overhead. Improvements to the Exchange Server 2013 namespace design reduced the complexity of this action, which makes datacenter switchover more transparent than in Exchange Microsoft recommends having a minimum of three high availability database copies before removing traditional forms of backup. Microsoft also recommends having at least two database copies at each site when a JBOD storage design is used. Our example organization chose to forgo traditional forms of backups and is using cache-enabled JBOD rather than traditional RAID arrays; therefore, the organization requires at least two copies of each database at each site. The organization determined a 14-day deleted item retention window is sufficient protection and decided not to deploy lagged database copies. Database Availability Groups The database availability group (DAG) is the building block for highly available or disaster recoverable solutions. It evolved from continuous cluster replication (CCR) in Exchange DAGs were first introduced in Exchange A DAG is composed of up to 16 mailbox servers that host a set of replicated databases and provide automatic database-level recovery from failures that affect individual servers or databases. Microsoft recommends minimizing the number of DAGs that are deployed for administrative simplicity. However, multiple DAGs are required in the following circumstances: More than 16 mailbox servers are deployed. Active mailbox users are in multiple sites (active/active site configuration). Separate DAG-level administrative boundaries are required. Mailbox servers are in separate domains (DAG is domain bound). In this reference architecture, the organization is deploying an active/passive site configuration (active users at the primary site only); therefore, the organization uses one DAG, which spans both sites. A file share witness is in the primary datacenter, which gives it three votes as opposed to two votes at the secondary datacenter (the two mailbox servers). 8 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

9 If there is a WAN outage, the database copies in the primary datacenter remain active. No users lose access to during the WAN outage. After the WAN is restored, replication resumes to the secondary datacenter. Best practice and implementation guidelines A successful Microsoft Exchange Server deployment and operation can be significantly attributed to a set of test-proven planning and deployment techniques. Proper planning includes sizing required server resources (CPU and memory), storage (capacity and IOPS), and networking (bandwidth and VLAN assignment) are needed to support the infrastructure. This information can then be implemented by using industry-standard best practices to achieve optimal performance and growth headroom that is necessary for the life of the solution. Configuration best practices and implementation guidelines, which aid in planning and configuration of the solution, are described in the following sections: Racking and power distribution System x3650 M5 Setup Network configuration on page 10 Storage design and configuration on page 12 Exchange Server 2013 database and log placement on page 13 Racking and power distribution Power distribution units (PDUs) and their cabling should be installed before any system is racked. When you are cabling the PDUs, consider the following points: System x3650 M5 Setup Ensure sufficient, separate electrical circuits and receptacles to support the required PDUs. To minimize the chance of a single electrical circuit failure taking down a device, ensure that there are sufficient PDUs to feed redundant power supplies by using separate electrical circuits. For devices that have redundant power supplies, plan for individual electrical cords from separate PDUs. Maintain appropriate shielding and surge suppression practices and use appropriate battery back-up techniques. The Exchange Server 2013 architecture consists of four dual-socket System x3650 M5 servers with the following configuration: 2x Intel Xeon E v3 8-core 2.6 GHz processors 12x 16 GB TruDDR4 Memory (2Rx4, 1.2 V) PC CL MHz LP RDIMM 10x 4 TB NL SAS 3.5-inch Hot Swap HDDs 2x 2 TB NL SAS 3.5-inch Hot Swap HDDs ServeRAID M5210 SAS/SATA Controller for System x ServeRAID M5200 Series 2GB Flash/RAID 5 Upgrade On-board quad-port Broadcom BCM5719 1Gb network adapter Setup involves the installation and configuration of Windows Server 2012 R2, storage, and networking configuration on each server. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 9

10 The following pre-operating system installation steps are used: 1. Validate that firmware levels are consistent across all servers. 2. Verify that the two 2TB NL SAS local disks are configured as an RAID1 array (for the operating system). (For more information, see Appendix B: Configuring the RAID1 boot array on the ServeRAID M5210 on page 20.) The following Windows installation and configuration steps are used: 1. Install Windows Server 2012 R2. 2. Set your server name and join the domain. 3. Install the Exchange Server 2013 prerequisite features. For more information about prerequisites, see this website: 4. Run Windows Update to ensure that any new patches are installed. Note: All the servers must have the same software updates (patches) and service packs. Network configuration This section describes the networking topology and best practices to correctly configure the network environment. In our solution, we rely heavily on the use of virtual LANs (VLANs), which are a way to logically segment networks to increase network flexibility without changing the physical network topology. With network segmentation, each switch port connects to a segment that is a single broadcast domain. When a switch port is configured to be a member of a VLAN, it is added to a group of ports that belong to one broadcast domain. Each VLAN is identified by a VLAN identifier (VID). A VID is a 12-bit portion of the VLAN tag in the frame header that identifies an explicit VLAN. Network topology design Two isolated networks are recommended to support this reference architecture: a MAPI network and a Replication network. The use of two network ports in each DAG member provides you with one MAPI network and one Replication network, with redundancy for the Replication network and the following recovery behaviors: If there is a failure that affects the MAPI network, a server failover occurs (assuming there are healthy mailbox database copies that can be activated). If there is a failure that affects the Replication network, log shipping and seeding operations revert to using the MAPI network if the MAPI network is unaffected by the failure, even if the MAPI network has its ReplicationEnabled property set to False. When the failed Replication network is restored and ready to resume log shipping and seeding operations, you must manually switch over to the Replication network. To change replication from the MAPI network to a restored Replication network, you can suspend and resume continuous replication by using the Suspend-MailboxDatabaseCopy and Resum boxDatabaseCopy cmdlets, or restart the Microsoft Exchange Replication service. 10 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

11 Microsoft recommends the use of suspend and resume operations to avoid the brief outage that is caused by restarting the Microsoft Exchange Replication service. A combination of physical and virtual isolated networks is configured at the server and the switch layers to satisfy isolation best practices. Each x3650 M5 server is equipped with an on-board quad-port Broadcom BCM Gb network adapter that is used for all data traffic. Each server maintains two 1 Gb connections to each of the network switches that are used in this reference architecture. The connections between the servers and the switches are shown in Figure 5. Note: If more network ports are needed for management or monitoring networks, extra network adapters can be purchased and installed. Corporate Network Network Switch Network Switch DB Mailbox/CAS Server DB Mailbox/CAS Server Figure 5 Network connections between host servers and network switches at one of the data centers Windows Server 2012 R2 NIC teaming can be used to provide fault tolerance and load balancing to all of the networks (MAPI network and Replication network). This setup allows the most efficient use of network resources with a highly optimized configuration for network connectivity. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 11

12 At the switch layer, VLANs should be used to provide logical isolation between the various networks. A key element is properly configuring the switches to maximize available bandwidth and reduce congestion. However, based on individual environment preferences, there is flexibility regarding how many VLANs are created and what type of role-based traffic they handle. After a final selection is made, ensure that the switch configurations are saved and backed up. Before continuing, test the network implementation thoroughly to ensure that communication is not lost despite the loss of a network switch or connection. Storage design and configuration This section describes the storage topology and includes instructions to correctly configure the internal storage on the x3650 M5. Key storage concepts and terminology This section describes the following basic concepts and terminology that are used throughout the next sections: JBOD An architecture that includes multiple drives, while making them accessible as independent drives or a combined (spanned) single logical volume with no actual RAID functionality. RAID0 RAID0 comprises striping (but not parity or mirroring). This level provides no data redundancy or fault tolerance. RAID0 has no error detection mechanism; therefore, the failure of one disk causes the loss of all data on the array. RAID1 RAID1 creates an exact copy (or mirror) of a set of data on two disks. This configuration is useful when read performance or reliability is more important than data storage capacity. A classic RAID 1 mirrored pair contains two disks. Cache-Enabled JBOD This storage design uses discreet, single-disk RAID0 arrays to maintain the independent disk design of a traditional JBOD configuration. It also uses the cache of the storage controller. AutoReseed AutoReseed is a new high availability feature for Database Availability Groups in Exchange Server This feature automatically reseeds a database if there is a disk failure to a pool of volumes that were pre-configured for this purpose. If the failed drive included an active database, Exchange fails over to one of the other passive copies and reseeds the failed database to the AutoReseed volume. If the failed drive contained one of the passive copies, Exchange reseeds the database on the same drive. This feature allows administrators to replace failed drives at their leisure, rather than being on-site to repair the failed hardware when the failure occurs. Storage Partitioning This section describes the storage partitioning that is required to support the configuration as described in this document. Figure 6 on page 13 shows the volumes and the underlying RAID architecture. 12 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

13 An RAID1 array that uses the 2 TB disks in slots 0 and 1 is used for the Windows operating system and the Exchange Transport Database. In our Exchange testing, we found the cache to have a significant performance effect. With cache enabled, the performance bottleneck shifts from the disk subsystem to the server s processors. Therefore, to use the cache, 10 drives are configured as discreet and individual RAID0 arrays. These arrays host Exchange database files, log files, and a restore logical unit number (LUN) and an AutoReseed LUN. RAID1 Array OS/Transport RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array RAID0 Array Figure 6 RAID1 array for the OS and RAID0 arrays formed from individual drives Storage Configuration Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server. For more information about creating a RAID1 boot array for the Windows operating system, see Appendix B: Configuring the RAID1 boot array on the ServeRAID M5210 on page 20. For more information about creating the RAID0 sing-disk arrays for the Exchange databases and the Restore and AutoReseed volumes, see Appendix C: Configuring the RAID0 single-disk arrays on the ServeRAID M5210 on page 24. After the RAID0 arrays are created, create the volumes from within the Windows operating system. For more information about creating volumes, see Appendix D: Creating volumes on page 27. Exchange Server 2013 database and log placement This section describes the Exchange Server 2013 mailbox database and log placement. Because our example organization chose to forgo traditional forms of backups and is using cache-enabled JBOD rather than a standard RAID configuration, the organization requires at least two copies of each database at each site for a total of four copies of each database. A total of 16 active databases are required to support the user population of 5,000 users. Each of the 16 active databases has three more passive copies to provide high availability. Therefore, 64 database copies are needed to support the entire user population. The databases are divided between four servers; therefore, each server hosts 16 databases. Each volume (except for the operating system, Restore, and AutoReseed volumes) hosts two database copies. The log files for each of the databases are placed on the same volume in separate folders. Figure 7 on page 14 shows the database distribution between the servers. The logs follow the same pattern and are on the same drives as their respective databases. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 13

14 Primary Datacenter (Active) RAID1 Array OS/Transport DB1 DB2 RAID1 Array OS/Transport DB3 DB4 DB9 DB10 DB15 DB16 DB5 DB6 DB11 DB12 DB7 DB8 DB13 DB14 Restore Volume AutoReseed DB3 DB4 DB9 DB10 DB15 DB16 DB5 DB6 DB11 DB12 Restore Volume DB1 DB2 DB7 DB8 DB13 DB14 AutoReseed DAG-1 Secondary Datacenter (Passive) RAID1 Array OS/Transport DB1 DB2 RAID1 Array OS/Transport DB3 DB4 DB9 DB10 DB15 DB16 DB5 DB6 DB11 DB12 DB7 DB8 DB13 DB14 Restore Volume AutoReseed DB3 DB4 DB9 DB10 DB15 DB16 DB5 DB6 DB11 DB12 Restore Volume DB1 DB2 DB7 DB8 DB13 DB14 AutoReseed Active Database Passive Database Figure 7 Database copy distribution between the servers Solution validation methodology Correctly designing an Exchange Server 2013 infrastructure requires accurate sizing of the servers and the storage and stress testing of the storage to ensure it can handle peak loads. Microsoft provides two tools to evaluate both aspects of an Exchange environment: Exchange Server 2013 Server Role Requirements Calculator for sizing the servers and storage and the Microsoft Exchange Server Jetstress 2013 Tool for testing the performance of the storage. Storage validation Storage performance is critical in any type of Exchange deployment. A poorly performing storage subsystem results in high transaction latency, which affects the user experience. It is important to correctly validate storage sizing and configuration when Exchange is deployed in any real-world scenario. 14 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

15 To facilitate the validation of Exchange storage sizing and configuration, Microsoft provides the Microsoft Exchange Server Jetstress 2013 Tool. Jetstress simulates an Exchange I/O workload at the database level by interacting directly with the Extensible Storage Engine (ESE). The ESE is the database technology that Exchange uses to store messaging data on the Mailbox server role. The Jetstress utility can simulate a target profile of user count and per-user IOPS and validate that the storage subsystem can maintain an acceptable level of performance with the target profile. Test duration is adjustable and can be set to an extended period to validate storage subsystem reliability. Testing storage systems by using Jetstress focuses primarily on database read latency, log write latency, processor time, and the number of transition pages that are repurposed per second (an indicator of ESE cache pressure). Jetstress returns a Pass or Fail report, depending on how well the storage is performing. To ensure performance results, the storage passed rigorous testing to establish a baseline that conclusively isolates and identifies any potential bottleneck as a valid server performance-related issue. Note: In normal operations, two servers host the 16 active databases (8 active databases per server in the active datacenter). However, to ensure that a single server can handle the workload if the second server was down for maintenance, a single server was tested with 16 active databases. Performance Testing Results Table 1 lists the parameters that were used to evaluate the storage performance. Table 1 Testing parameters Parameter Value Database Sizing Database Files (Count) 16 Number of Users 5000 IOPs Per User.10 Mailbox Size (MB) 3500 Jetstress System Parameters Thread Count 10 Minimum Database Cache Maximum Database Cache 512 MB MB Insert Operations 40% Delete Operations 20% Replace Operations 5% Read Operations 35% Lazy Commits 70% Run Background Database Maintenance True Number of Copies per Database 4 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 15

16 Table 2 presents the target IOPs and the IOPs that are achieved during the 24-hour testing cycle. The achieved IOPs exceeded the target by IOPs. Table 2 Database Sizing and Throughput Parameter Value Achieved Transactional I/O per Second Target Transactional I/O per Second 500 Jetstress evaluates latency values for Database Reads and Log writes because these affect the user experience. Table 3 displays the test results of the load on the database files. The second column (Database Reads Avg Latency) should not exceed 20 msec. A value over 20 indicates that the storage cannot handle sustained peak workload. In our testing, the maximum value is 9.79, which is well below the 20 msec limit. Table 3 Testing results for database files Database Instances Database Reads Avg Latency (msec) Database Writes Avg Latency (msec) Database Reads/sec Database Writes/sec Database Reads Avg Bytes Database Writes Avg Bytes Database Database Database Database Database Database Database Database Database Database Database Database Database Database Database Database System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

17 Table 4 Testing results for log files Table 4 displays the test results of the load on the log files. The third column (Log Writes Avg Latency (msec)) should not exceed 10 msec. A value over 10 indicates that the storage cannot handle the workload. In our testing, the maximum value is 0.066, which is well below the 10 msec limit. Database Instances Log Reads Avg Latency (msec) Log Writes Avg Latency (msec) Log Reads/sec Log Writes/sec Log Reads Avg Bytes Log Writes Avg Bytes Database Database Database Database Database Database Database Database Database Database Database Database Database Database Database Database For more information about the complete results from the 24-hour JetStress test, see Appendix E: Microsoft Exchange Jetstress 2013 Stress Test Result Report on page 28. Summary The System x3650 M5 with cache-enabled JBOD that uses internal disks performed admirably throughout the test duration. This test demonstrates the capability of the x3650 M5 in supporting 5,000 mail users with 3,500 MB mailboxes. The System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 provides a highly available and site-resilient answer to organizations that are ready to upgrade their existing mail environment or implement a new low-cost infrastructure. The environment that is described in this document allows administrators to eliminate traditional backup method, which frees critical enterprise resources. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 17

18 The use of internal disks in a cache-enabled JBOD configuration drastically reduces the overall cost and complexity of the solution by eliminating SAN or DAS-based storage and reducing the need for storage administrators. This solution also provides a framework for scaling up to meet the needs of any sized user population. References and helpful links For more information about the topics in this paper, see the following resources: High availability and site resilience: The New Exchange: Exchange Server 2013 storage configuration options: Namespace Planning in Exchange Server 2013: change-2013.aspx Non-RAID drive architectures: ServeRAID M5210 and M5210e SAS/SATA Controllers Product Guide IBM Support: IBM Firmware update and best practices guide: About the Author Roland G. Mueller works for Lenovo in Kirkland, Washington. He has a second office at the Microsoft main campus in Redmond, Washington to facilitate close collaboration with Microsoft. He specializes in Exchange Server infrastructure sizing, design, and performance testing. Before Lenovo, Roland worked for IBM from , specializing in various technologies, including virtualization, bare-metal server deployment, and Microsoft Exchange Server. 18 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

19 Appendix A: Bill of materials Each of the four servers is configured as listed in Table 5. Table 5 Bill of materials Feature code Description Quantity 5462AC1 Server1: System x3650 M5 1 A5FP System x3650 M5 PCIe Riser 1 (2 x8 FH/FL + 1 x8 FH/HL Slots) Select Storage devices: No IBM configured RAID required 1 A5R5 System x3650 M5 PCIe Riser 2 (1 x16 FH/FL + 1 x8 FH/HL Slots) 1 A5FF System x3650 M5 12x 3.5-inch Base without Power Supply 1 A5EW System x 900W High Efficiency Platinum AC Power Supply 2 A5EL Extra Intel Xeon Processor E v3 8C 2.6 GHz 20 MB 1866 MHz 90 W No Preload Specify 1 A5B7 16 GB TruDDR4 Memory (2Rx4, 1.2V) PC CL MHz LP RDIMM 12 A5EY System Documentation and Software-US English 1 A5GT Intel Xeon Processor E v3 8C 2.6 GHz 20 MB Cache 1866 MHz 90 W 1 A3YZ ServeRAID M5210 SAS/SATA Controller 1 A3Z2 ServeRAID M5200 Series 2 GB Flash/RAID 5 Upgrade 1 A5GE x3650 M5 12x 3.5-inch HS HDD Assembly Kit 1 A5VQ IBM 4 TB 7.2K 12 Gbps NL SAS 3.5-inch G2HS 512e HDD 10 A5VP IBM 2 TB 7.2K 12 Gbps NL SAS 3.5-inch G2HS 512e HDD m, 13 A/ A/250 V, C13 to IEC 320-C14 Rack Power Cable 2 A5FV System x Enterprise Slides Kit 1 A5G1 System x3650 M5 IBM EIA Plate 1 A2HP Configuration ID 01 1 A5V5 System x3650 M5 Right EIA for Storage Dense Model 1 A5FM System x3650 M5 System Level Code 1 A5FH System x3650 M5 Agency Label GBM 1 A5EA System x3650 M5 Planar 1 A5G5 System x3650 M5 Riser Bracket 2 A5FT System x3650 M5 Power Paddle Card 1 A47G Super Cap Cable 425mm for ServRAID M5200 Series Flash CM1 HIPO: Configuration Instruction 1 A2HP Configuration ID 01 1 A46P ServeRAID M5210 SAS/SATA Controller Placement 1 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 19

20 Feature code Description Quantity A46S ServeRAID M5200 Series 2GB Flash/RAID 5 Upgrade Placement 1 A2JX Controller ND6 Service pack1: Lenovo RTS for System x - Base - 3yr HG Service pack2: 3 Year Onsite Repair 24x7 4 Hour Response 1 Appendix B: Configuring the RAID1 boot array on the ServeRAID M5210 Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server. Complete the following steps to configure the internal storage: 1. Power on the server you want to configure and press F1 to enter UEFI Setup when the UEFI splash window opens. 2. From the UEFI Setup menu, select System Settings and press Enter, as shown in Figure 8. Figure 8 UEFI Setup main menu 3. From the System Settings menu, scroll down and select Storage and press Enter. 20 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

21 4. From the Storage menu, select the storage adapter and press Enter, as shown in Figure 9. Figure 9 Available RAID adapters 5. From the Main menu, select Configuration Management and press Enter, as shown in Figure 10. Figure 10 ServeRAID M5210 Main Menu 6. From the Configuration Management menu, select Create Virtual Drive - Advanced and press Enter. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 21

22 7. From the Create Virtual Drive - Advanced menu, ensure that RAID1 is selected as the RAID level and 256 KB is selected as the Stripe size (be aware that it is spelled Strip size ). Select Select Drives and press Enter, as shown in Figure 11. Figure 11 Create Virtual Drive menu that shows correct RAID level and Stripe size 8. From the Select Drives menu, highlight the HDDs in slots 0 and 1 and press the Enter to select them. Then, select Apply Changes and press Enter, as shown in Figure 12. Figure 12 Select Drives menu 9. From the Success page, select OK and press Enter. You are returned to the Create Virtual Drive Advanced menu. 10.From the Create Virtual Drive - Advanced menu, select Save Configuration and press Enter. 22 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

23 11.From the Warning page (as shown in Figure 13), confirm your choice and press Enter to select it. Next, select Yes and press Enter. Figure 13 Warning page 12.From the Success page, select OK and press Enter. 13.Press Esc twice to return to the Main Menu. Select Controller Management and press Enter, as shown in Figure 14. Figure 14 Selecting Controller Management System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 23

24 14.From the Controller Management page (as shown in Figure 15), scroll down and select Boot Device and press Enter. Figure 15 Selecting a boot device 15.Select the RAID1 array and press Enter. 16.Press Esc to exit to the Main Menu. 17.After you configure the RAID1 boot array, exit UEFI and boot to your installation media to install the Windows operating system. Appendix C: Configuring the RAID0 single-disk arrays on the ServeRAID M5210 Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server. Complete the following steps to configure the internal storage: 1. Power on the server that you want to configure and press F1 to enter UEFI Setup when the UEFI splash window opens. 2. From the UEFI Setup menu, select System Settings (as shown in Figure 16 on page 25) and press Enter. 24 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

25 Figure 16 UEFI Setup main menu 3. From the System Settings menu, scroll down and select Storage and press Enter. 4. From the Storage menu, select the storage adapter (see Figure 17) and press Enter. Figure 17 Available RAID adapters 5. From the Main menu, select Configuration Management (as shown in Figure 18) and press Enter. Figure 18 ServeRAID M5210 Main Menu System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 25

26 6. From the Configuration Management menu, select Create Virtual Drive - Advanced and press Enter. 7. From the Create Virtual Drive - Advanced menu, ensure that RAID0 is selected as the RAID level and 256 KB is selected as the Stripe size and then select Select Drives and press Enter (see Figure 19). Figure 19 Create Virtual Drive menu that shows correct RAID level and Stripe size 8. From the Select Drives menu, highlight the first HDD and press the Enter to select it as shown in Figure 20. Then, select Apply Changes and press Enter. Figure 20 Select Drives menu 26 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks

27 9. From the Success page, select OK and press Enter. You are returned to the Create Virtual Drive Advanced menu. 10.From the Create Virtual Drive - Advanced menu, select Save Configuration and press Enter. 11.From the Warning page (see Figure 21), highlight Confirm and press Enter to select it. Next, select Yes and press Enter. Figure 21 Warning page 12.From the Success page, select OK and press Enter. 13.Repeat steps 7-12 to build RAID0 arrays on each individual remaining drive. 14.After you configure the drives, exit UEFI and boot to Windows normally. When you are finished, each of the remaining 10 HDDs in the front of the server should be a discreet RAID0 array. Appendix D: Creating volumes After you create the RAID0 single-disk arrays, volumes must be defined and assigned mount points or drive letters from within the Windows operating system. Complete the following steps: 1. Open Server Manager and click File and Storage Services from the navigation tree. 2. Click Disks, and then bring each of the disks online and initialize them as GPT by right-clicking each disk and clicking Bring Online or Initialize Disk from the menu. 3. Right-click each disk and click New Volume. 4. Click Next to bypass the Before You Begin page. 5. On the Server and Disk page, verify that the correct disk is selected and click Next. 6. Click Next to accept the default size, which uses the entire disk. 7. On the Drive Letter or Folder page, select a drive letter or choose a folder as a mount point for the volume and click Next to continue. System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks 27

28 8. On the File System Settings page, verify that NTFS is selected as the File System and use the drop-down menu to select 64K as the Allocation unit size, as shown in Figure 22. Click Next to continue. Figure 22 File System Settings window 9. In the Confirmation window, click Create to create the volume. Click Close after the process completes. 10.Repeat these steps for the remaining 11 disks. Appendix E: Microsoft Exchange Jetstress 2013 Stress Test Result Report Test Summary Overall Test Result Machine Name Test Description Pass WIN-6RCTC0BTSAJ 5,000 Users 4 database copies 3500 mb mailbox.10 IOPS 2 databases per volume with logs 8 disks used + 1 restore +1 autoseed 10 threads Test Start Time 11/11/2014 9:08:30 AM Test End Time 11/12/2014 9:17:42 AM Collection Start Time 11/11/2014 9:17:18 AM Collection End Time 11/12/2014 9:17:19 AM 28 System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks