Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

Transcription

1 Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces Last update: 02 July 2015 Microsoft Software Defined Storage using Lenovo rack-based servers Designed for Enterprises, MSPs/CSPs, and HPC Affordable, high performance, scalable, and highly available Low footprint and high density to achieve cost savings on space, power and cooling David Ye

2 Table of Contents 1 Introduction Business problem and business value Business problem Business value Requirements Functional requirements Non-functional requirements Architectural overview Component model Operational model Hardware components Lenovo System x3550 M Lenovo System x3650 M GbE RDMA network adapters Lenovo RackSwitch G8124E Lenovo RackSwitch G NetApp DE Storage nodes and disk enclosures Networking Systems management Windows Server 2012 R Deployment example Deployment considerations Systems Management Storage sizing and tuning Storage sizing Storage resiliency ii Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

3 7.2.3 Storage write-back cache Storage column and interleave sizing MPIO load balancing policy Allocation unit size Network adapters and switches RDMA configurations Lenovo G8264 OS application guide Hardware certifications Deployment best practices Appendix: Lenovo Bill of materials Recommended configurations Storage Node BOMs Networking BOMs Resources iii Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

4 1 Introduction This document describes the Lenovo Reference Architecture for Microsoft Scale-Out File Server (SOFS) that uses Storage Spaces. Lenovo reference architecture offerings create virtually turnkey solutions that are built around the latest Lenovo servers, networking, and storage, which takes the complexity out of the solution. This Lenovo reference architecture combines Microsoft software, consolidated guidance, and validated configurations for compute, network, and storage. Software-defined storage (SDS) is an evolving concept for computer data storage that manages policy-based provisioning and data storage independent of hardware. SDS definitions typically include a form of storage virtualization to separate the storage hardware from the software that manages the storage infrastructure. The software that enables an SDS environment might also provide policy management for feature options, such as deduplication, replication, thin provisioning, snapshots, and backup. The key benefits of SDS over traditional storage are increased flexibility, automated management, and cost efficiency. One approach to SDS that is gaining traction is Microsoft SOFS. Microsoft SOFS is a high-performance, scalable, and continuously available storage solution for Microsoft Hyper-V and SQL servers that offer an alternative to traditional Storage Area Network (SAN) solutions with the same or better performance. This Lenovo solution for Microsoft SOFS combines the Storage Spaces and Failover Cluster features of Windows 2012 R2 with Lenovo industry standard x86 servers, Lenovo RackSwitch network switches, and NetApp high density disk enclosures to provide turnkey solutions for enterprises. The architecture that is described here was validated by Lenovo and certified for Microsoft Storage Spaces. The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and best practices for implementing the Lenovo Reference Architecture for Microsoft SOFS that uses Storage Spaces. 1 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

5 2 Business problem and business value This section describes the challenges organizations face and how this Reference Architecture for Microsoft Scale-Out File Server (SOFS) with Storage Spaces can help meet those challenges. 2.1 Business problem The cloud and mobile innovations in the last few years present tremendous amount of growth opportunity for those enterprises that are equipped with proper IT infrastructure. However, companies are discovering that their IT infrastructure is not always up to the task at hand; finding that budgetary constraints or outdated architectures are hindering their ability to compete. Enterprises that use proprietary systems are finding that they are locked into expensive maintenance contracts and obligations that force them to continue buying expensive proprietary technologies. With digital data growing rapidly annually in the enterprises, companies who deployed traditional proprietary SAN storage are seeing a significant amount of their budget being allocated for storage purchases. This factor is one of the inhibiting factors that limit companies growth and competitiveness because of a lack of investments in other key areas, such as new applications. 2.2 Business value The Lenovo solution for Microsoft SOFS combines the skills and technologies of the leading enterprise software and hardware vendors to create a non-proprietary storage solution that lowers overall storage costs, increases storage reliability, and frees you from expensive maintenance and service contracts. Gaining access to near-zero downtime with exceptional fault tolerance, dynamic pooling, enhanced virtualization resources, end-to-end architectural and deployment guidance, predefined, out-of-box solutions and much more gives you the tools to compete today and into the future. Table 1 lists a high-level comparison of SAN shared storage and Microsoft SOFS capabilities. 2 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

6 Table 1: Comparison of SAN and Microsoft SOFS capabilities FC/iSCSI SAN RAID resiliency Disk pooling Data deduplication High availability (controller failover) Storage tiering Snapshots Persistent write back cache Copy offload FC, FCoE, iscsi Microsoft SOFS with Storage Spaces Two/three way mirroring, single/dual parity Storage Spaces disk pooling Data deduplication Continuous availability (transparent failover) Storage tiering Snapshots Persistent write back cache SMB copy offload 10 GbE, 40 GbE, 56 Gb IB SMB Multichannel, SMB Direct (RDMA) The basic building block of this Lenovo solution can scale 2 4 storage nodes and up to 240 disks. In some cases, it can scale up to 360 disks. This 2 4 storage nodes building block model can scale horizontally and linearly as much as your compute infrastructure requires. It also provides high performance and continuous availability. All of these features can be achieved by using standard Lenovo 2-socket x86 server hardware and ultimately lower total cost of ownership. 3 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

7 3 Requirements The functional and non-functional requirements for this reference architecture are described in this section. 3.1 Functional requirements The following functional requirements are featured: Integrate with Windows 2012 R2 Storage features: o o Create storage pools with just a bunch of disks (JBOD) Create RAID equivalent virtual disks or logical unit numbers (LUNs): Enable virtual disk for storage tiering Allocate Write-back cache for virtual disks from SSD tier o o o Create Cluster Shared Volumes Support Multipathing Support SMB 3.0 and 3.02 for storage access protocol: Create Continuous Availability (CA) Files Shares Create multiple SMB connections for each network adapter on-demand Detect and utilize RDMA-capable network adapters Encrypt storage traffic between hosts and storage nodes Provide transparent failover capability o Support File Management: Enable/Disable deduplication on per volume basis Configure as DFS Namespace folder target server Enable/Disable Folder Redirection Enable/Disable Offline Files (client side caching) Support Roaming User Profiles Support Home Directories o Support IO intensive application workloads: Microsoft Hyper-V Servers Microsoft SQL Servers Integrate with Windows Failover Cluster for high availability (HA): o Create Windows Failover Cluster: Create Cluster Management Network 4 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

8 Create Cluster Communication Network Create Client/Server Network for Storage Create Client Access Point for CA Shares o Single Management User Interface: Manage all Microsoft SOFS storage functionalities Provide wizard driven tools for storage related tasks o Enterprise Management: Support integration with Microsoft System Center 3.2 Non-functional requirements Table 2 lists the non-functional requirements for a server-based storage solution. Table 2: Non-functional requirements Requirement Scalability Load balancing Fault tolerance Physical footprint Ease of installation Ease of management/operations Flexibility Security High performance Certification Description Scale out linearly with building block approach SMB storage traffic is distributed across storage nodes Storage nodes and JBOD enclosures are redundant Low profile servers and high density JBOD enclosures reduce rack space Standard server hardware and software with cluster setup wizard Managed through Windows Failover Cluster Manager Different choices for various hardware components and SSD and HDD drives mixed Encryption of storage traffic High speed low latency Remote Direct Memory Access over 10GbE Hardware should be certified for Microsoft Windows 2012 R2 and Storage Spaces 5 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

9 4 Architectural overview Microsoft SOFS is considered an SDS solution. It is not just using standard x86 servers and disk enclosures to build a storage solution to lower cost; it also provides continuous availability storage for applications. Two primary use cases for this solution are Microsoft SQL Server and Hyper-V Server. Figure 1 shows the Microsoft SQL Server and Hyper-V clusters that are connected to Microsoft SOFS as storage backend through Server Message Blocks (SMB 3.02) protocol over high speed Ethernet networks. Figure 1: Microsoft Hyper-V and SQL Servers with Microsoft SOFS 6 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

10 Figure 2 shows more details of the software and hardware layers. Figure 2: Microsoft Hyper-V and SQL Servers with Microsoft SOFS Storage Logical View The following software and hardware layers are featured: Server Message Block (SMB) protocol SOFS Cluster Shared Volumes (CSV) Storage Space Server Message Block protocol is the Windows file sharing protocol in which Microsoft Hyper-V or SQL Servers use to communicate with the back-end Microsoft SOFS storage. Microsoft SOFS is a Windows operating system feature that provides scale-out file shares that are continuously available for file-based server application storage. Microsoft SOFS also provides the ability to share the same folder from multiple nodes of the same cluster. A CSV is a shared disk that contains a volume that is made accessible for read and write operations by all nodes within a Microsoft Windows SOFS cluster. Microsoft SQL and Hyper-V data are stored in CSV. Storage Space is a Windows feature with which you can create a virtual disk from a storage pool and present it a volume or LUN to Microsoft SOFS cluster nodes. Virtual disk can be created with resiliency options, such as 2-way mirror, 3-way mirror, single parity, dual parity, and no parity. 7 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

11 Storage Pool A storage pool is a grouping of industry-standard disks, such as solid-state disks (SSDs) or hard disk drives (HDDs) to provide the aggregate performance of all disks in the pool. Storage Node A storage node is an industry-standard x86 server that is running Windows 2012 R2 to provide storage controller functionalities. Shared JBOD A shared JBOD is a physical disk enclosure that houses the physical disks, such as SSDs and HDDs. Microsoft SOFS hosts SQL Server s databases files and Hyper-V s VHDX files. When one of the Microsoft SOFS storage nodes fails, the SMB client that is running on the SQL or Hyper-V server automatically selects the next best available SOFS storage node to connect to and resumes IO operations on behalf of the SQL or Hyper-V. From SQL and Hyper-V servers perspective, the IO appears stalled for a short amount of time. 8 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

12 5 Component model Windows Server 2012 R2 provides a rich set of storage features with which you can use lower-cost, industry-standard hardware without compromising performance or availability. Figure 3 shows an overview of the Microsoft software storage and virtualized components in the Microsoft SOFS storage solution. Figure 3: Windows Server 2012 R2 Storage Stack Diagram The Microsoft SOFS storage solution consists of the following components: Microsoft Directory Service Microsoft Active Directory (AD), Domain Name Servers (DNS), and Dynamic Host Configuration Protocol (DHCP) servers provide user authentications, domain name service, and dynamic IP service infrastructure for Hyper-V and SOFS storage nodes. 9 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

13 Microsoft Virtualization Stack Hyper-V Storage NUMA I/O Windows Server 2012 R2 supports large virtual machines, up to 64 virtual processors. Any large virtual machine configuration typically also needs scalability in terms of I/O throughput. Windows Server 2012 R2 provides a Hyper-V storage NUMA I/O capability to create several communication channels between the guest devices and host storage stack with a specified dedicated set of VPs for the storage IO processing. Hyper-V storage NUMA I/O offers a more efficient I/O completion mechanism that interrupts distribution amongst the virtual processors to avoid expensive inter-processor interruptions. With those improvements, the Hyper-V storage stack can provide scalability improvements in terms of I/O throughput to support the needs of large virtual machine configuration with data intensive workloads like Exchange, SQL, and SharePoint. Storage Quality of Service (QoS) and Hyper-V IO Balancer Storage QoS is a new feature that is provided in Windows Server 2012 R2. It offers the capability to set certain QoS parameters for storage on the virtual machines. Storage QoS and Hyper-V IO balancer provide the ability to specify maximum I/O operations per second (IOPS) values for an individual virtual hard disk. Windows Server 2012 R2 Hyper-V with Storage QoS can throttle storage that is assigned to VHD/VHDX in the same volume to prevent a single virtual machine from using all I/O bandwidth and help to control the balance of VM storage demand and storage performance capacity. VHDX VHDX is a new virtual hard disk format with which you can create resilient high-performance virtual disks up to 64 TBs. Microsoft recommends the use of VHDX as the default virtual hard disk format for VMs. VHDX provides more protection against data corruption during power failures by logging updates to the VHDX metadata structures and the ability to store custom metadata. The VHDX format also provides support for the TRIM command, which results in smaller file size and allows the underlying physical storage device to reclaim unused space. The support for 4KB logical sector virtual disk and the larger block sizes for dynamic and differential disks allows for increased performance. Microsoft SMB 3.0 SMB Multichannel SMB multichannel provides the capability to automatically detect multiple networks for SMB connections. It offers resilience against path failures and transparent failover with recovery without application service disruption with much improved throughput by aggregating network bandwidth from multiple network interfaces. Server applications then can use all available network bandwidth, which makes them more resistant to network failure. SMB Direct SMB Direct (SMB over RDMA) makes available Remote Direct Memory Access (RDMA) hardware support for SMB to provide high-performance storage capabilities. SMB Direct is intended to lower CPU usage and latency on the client and server while delivering high IOPS and bandwidth utilization. It can deliver enterprise-class performance without relying on expensive Fibre Channel SAN. With the 10 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

14 CPU offloading and the ability to read and write directly against the memory of the remote storage node, RDMA network adapters can achieve extremely high performance with low latency. SMB Direct is also compatible with SMB Multichannel to achieve load balancing and automatic failover. SOFS Cluster Failover Clustering A failover cluster is a group of independent computers that work together to increase the availability and scalability of clustered roles. If one or more of the cluster nodes fail, the service automatically fails over to other node without disruption of service. SOFS role The SOFS role provides a continuously available SMB service and a mechanism for clustered file servers in an active-active configuration to aggregate bandwidth across the cluster nodes. Cluster Shared Volumes (CSV) CSVs in a Windows Server 2012 R2 failover cluster allow multiple nodes in the cluster to simultaneously access shared storage with a consistent and distributed namespace. To each of the cluster nodes in the cluster, the CSV provides a consistent single file namespace. Therefore, CSVs simplify the management of many LUNs in a failover cluster. Continuous Availability (CA) File Share A feature in Windows Failover Clustering, CA tracks file operations on highly available file shares so that storage clients (such as Hyper-V or SQL servers) can fail over to another node of the cluster without interruption. Failover Cluster Manager Failover Cluster Manager is a user interface with which you can manage all aspects of Failover Cluster management. By using this feature, you can run tasks, such as create Failover Cluster, Storage Pools, CSVs, and CA File Share server. Microsoft Storage Stack Storage Spaces Storage Spaces is a storage virtualization feature in Windows Server 2012 R2 provides many features including write-back cache, storage tiering, thin provisioning, mirroring, and parity with just a bunch of disks (JBOD). It gives you the ability to consolidate all of your Serial Attached SCSI (SAS) and Serial ATA (SATA) connected disks into Storage Pools. Storage Spaces are compatible with other Windows Server 2012 R2 storage features, including SMB Direct and Failover Clustering. Microsoft Multipath I/O (MPIO) MPIO is a Microsoft-provided framework that allows storage providers to develop multipath solutions to optimize connectivity and create redundant hardware paths to storage systems. In this solution, we use Microsoft Device Specific Module (DSM) working with Microsoft Storage Spaces to provide fault tolerant connectivity to the disk enclosures or JBOD. 11 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

15 6 Operational model It is useful to view this solution in the following layers: Physical Layer: This layer includes the physical hardware, such as rack, servers, switches, and disk enclosures units (JBODs). Software Layer: This layer includes the Windows operating system and its built-in software features that enable the physical hardware to provide storage functionality. 6.1 Hardware components The following section describes the components in a Microsoft SOFS deployment Lenovo System x3550 M5 At the core of this reference architecture, the Lenovo System x3550 M5 server delivers the performance and reliability that is required for mission-critical storage solutions, such as SOFS. Lenovo System x3550 M5 servers can be equipped with up to two 18-core E v3 series processors and up to 1.5 TB of TruDDR4 memory. Up to three PCIe 3.0 expansion slots, four integrated 1 GbE network ports, and an optional embedded dual-port 10/40 GbE network adapter provides ports for your data and storage connections. The Lenovo System x3550 M5 includes an on-board RAID controller and the choice of spinning hot swap SAS or SATA disks and small form factor (SFF) hot swap SSDs. The x3550 M5 supports a maximum of 24 TB of internal storage. The x3550 M5 supports the following components: Up to 10 front and two rear SFF HDDs or SSDs Up to four 3.5-inch HDDs The x3550 M5 also supports remote management via the Lenovo Integrated Management Module (IMM), which enables continuous management capabilities. All of these key features (including many that are not listed here) help solidify the dependability Lenovo customers are accustomed to with System x servers. Figure 4 shows the Lenovo x3550 M5 server. Figure 4: Lenovo System x3550 M5 For more information, see this website: lenovopress.com/tips1194.html Lenovo System x3650 M5 The Lenovo System x3650 M5 server (as shown in Figure 5) is an enterprise class 2U two-socket versatile server that incorporates outstanding reliability, availability, and serviceability (RAS), security, and high efficiency for business-critical applications and cloud deployments. It offers a flexible, scalable design and simple upgrade path to inch HDDs, with doubled data transfer rate via 12 Gbps SAS internal storage connectivity and up to 1.5TB of TruDDR4 Memory. Its onboard Ethernet solution provides four standard 12 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

16 embedded Gigabit Ethernet ports and two optional embedded 10 Gigabit Ethernet ports without occupying PCIe slots. Figure 5: Lenovo System x3650 M5 Combined with the Intel Xeon processor E v3 product family, the Lenovo x3650 M5 server offers a high density of workloads and performance that is targeted to lower the total cost of ownership (TCO) per virtual machine. Its flexible, pay-as-you-grow design and great expansion capabilities solidify dependability for any kind of virtualized workload with minimal downtime. The Lenovo x3650 M5 server provides internal storage density of up to 84 TB (with up to 14 x 3.5-inch drives) in a 2U form factor with its impressive array of workload-optimized storage configurations. The x3650 M5 offers easy management and saves floor space and power consumption for the most demanding storage virtualization use cases by consolidating the storage and server into one system. For more information, see the following websites: System x3650 M5 Overview System x3650 M5 Product Guide GbE RDMA network adapters This Lenovo solution for Microsoft SOFS offers the following choices of 10 GbE RDMA network adapters: Emulex VFA5 Melllanox CX3 These adapters provide high throughput and low latency capabilities for a SOFS solution. For more information, see the following websites: lenovopress.com/tips1142.html lenovopress.com/tips0897.html Lenovo RackSwitch G8124E The Lenovo RackSwitch G8124E is a 10 Gb Ethernet switch that is specifically designed for the data center and provides a virtualized, cooler, and easier network solution. The G8124E offers GbE ports in a high-density, 1U footprint. Designed with ultra-low latency and top performance in mind, the RackSwitch G8124E provides line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. The G8124E also supports Converged Enhanced Ethernet (CEE) and Data Center Bridging for support of FCoE and can be used for NAS or iscsi. 13 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

17 The G8124E is virtualized and supports VMready technology, which is an innovative, standards-based solution to manage virtual machines in small to large-scale data center and cloud environments. VMready works with all leading VM providers. The G8124E also supports Virtual Fabric, which allows for the carving up of a physical NIC into 2-8 vnics for improved performance, availability, and security while reducing cost and complexity. The G8124E is easier to use and manage, with server-oriented provisioning via point-and-click management interfaces. Its industry-standard command-line interface (CLI) and easy interoperability simplifies configuration for those users who are familiar with Cisco environments. Figure 6 shows the Lenovo RackSwitch G8124E. Figure 6: Lenovo RackSwitch G8124E The RackSwitch G8124E includes the following benefits: A total of 24 SFP+ ports that operate at 10 Gb or 1 Gb Ethernet speeds Optimal for high-performance computing and applications that require high bandwidth and low latency All ports are nonblocking, 10 Gb Ethernet with deterministic latency of 570 nanoseconds VMready helps reduce configuration complexity and improves security levels in virtualized environments Variable-speed fans automatically adjust as needed, which helps to reduce energy consumption Easy, standards-based integration into Cisco and other networks helps reduce downtime and learning curve For more information, see this website: lenovopress.com/tips0787.html Lenovo RackSwitch G8264 Designed with top performance in mind, Lenovo RackSwitch G8264 (as shown in Figure 7) is ideal for today s big data, cloud, and optimized workloads. Figure 7: Lenovo RackSwitch G8264 The G8264 switch offers up to Gb SFP+ ports in a 1U form factor and can be used in the future with four 40 Gb QSFP+ ports. It is an enterprise-class and full-featured data center switch that delivers line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. Redundant power and fans and numerous HA features equip the switches for business-sensitive traffic. 14 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

18 The G8264 switch is ideal for latency-sensitive applications, such as client virtualization. It supports Virtual Fabric to help clients reduce the number of I/O adapters to a single dual-port 10 Gb adapter, which helps reduce cost and complexity. The G8264 switch supports the newest protocols, including Data Center Bridging/Converged Enhanced Ethernet (DCB/CEE) for support of FCoE, iscsi, and NAS. For more information, see the Lenovo RackSwitch G8264ttttttttttttt Product Guide NetApp DE6600 NetApp DE6600 JBOD is a low-profile, high-density external disk enclosure, as shown in Figure 8. Figure 8: NetApp DE6600 Disk Enclosure This enclosure can provide cost savings for your data center with space saving, power and cooling, and its low footprint can house up to 60 HDDs or a mix of SDDs and HDDs, in a 4U form factor footprint. It features dual SAS modules that can provide redundancy for fault tolerant and dual paths to support storage multipathing capability. It can also scale vertically by cascading multiple enclosures together to provide higher storage performance and with capacity over 1 PB. NetApp DE6600 is certified for Microsoft Windows 2012 R2 Storage Spaces. It is easy to deploy and manage with Windows 2012 R2 and is an ideal platform for the following applications: Managed Service Providers (MSP) and Cloud Service Providers (CSP) with low cost and high density requirements High Performance Computing (HPC) deployments that require high density storage to lower cost on space, power and cooling High throughput applications, such as oil and gas, manufacturing, video surveillance, and call center Enterprises with low-cost, high-density, and performance storage requirements For more information, see this website: netapp.com/us/products/storage-systems/e5400/e5400-tech-specs.aspx 15 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

19 6.2 Storage nodes and disk enclosures This solution can use the Lenovo System x3550 M5 or Lenovo System x3650 M5 server as storage nodes. Each server features the following components: Two E v3 Intel Xeon processors 128 GB of memory Two 2.5-inch HDDs in a RAID1 pair Four on-board 1 Gbps Ethernet ports One dual-port 10 Gbps Ethernet adapter with RDMA capability Two dual-port SAS HBAs For more information, see Storage Node BOMs on page 27 In addition to the storage nodes, physical disk enclosures and disks are based on JBOD and SSD plus 7.2k rpm Near-Line SAS (NL-SAS) disks, respectively. Each storage node has dual connections or paths to the JBOD enclosures for fault tolerant and performance. SSD provides high performance storage tier for daily work set for various application workloads. NL-SAS disks provide high storage capacity for data that are not used daily. The basic building block for the Microsoft SOFS storage solution is four storage nodes and four JBODs (4-Node/4-JBOD). However, business customers can start with a minimum of 2-Node/1-JBOD and add storage nodes or JBODs as needed. Customers can scale the Microsoft SOFS storage solution linearly with 4-Node/4-JBOD increment. Depending on the capacity and performance requirements, the storage nodes are connected to up to 4 NetApp DE6600 JBOD disk enclosures and can have up to 240 SSDs or HDDs in any combination. All disk enclosures are directly connected with SAS cables and shared among all storage nodes. All connections from each SOFS storage node to the shared JBODs are redundant for HA. There are no SAS switches involved in this solution to reduce complexity and costs. The storage nodes are clustered to provide continuous availability back-end storage for Microsoft Hyper-V or SQL servers. Microsoft Storage Spaces is used to pool all the disk resources in the enclosures. Cluster virtual disks are created from the pool to present them to SOFS storage nodes as logical unit numbers (LUNs). Storage Spaces provides RAID resiliency options, such as 2-way mirror or 3-way mirror during the creation of virtual disks. Single or dual parity is also supported and is suitable for backup, archiving, or sequential workloads. SSDs can be used for storage tiering or as a write-back cache for virtual disks. Virtual disks are formatted by using the Microsoft Clustered Shared Volume File System (CSVFS). CSV volumes are accessed by all storage nodes in the SOFS cluster. Server Message Block (SMB) 3.02 file shares are created by using CSV volumes on the SOFS cluster. Each Hyper-V host and storage node is equipped with a dual-port 10 Gbps Ethernet adapter. The Ethernet adapter is RDMA-capable, which is used by the SMB-Direct feature of the SMB 3.02 protocol to provide high speed and low latency network access with low CPU utilization. These RDMA adapters are dedicated for SMB file share storage traffic. 16 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

20 With the file shares in place, Hyper-V or SQL server hosts can use the files shares as back-end storage to store VHDX VM or SQL server databases files. If there is a storage node failure while a Hyper-V or SQL host is accessing the storage, the host s SMB and witness client is notified immediately by the SOFS cluster witness service and then Hyper-V or SQL server automatically switches to the next best available storage node, re-establishes the session, and continues IO operations from the point of failure. This process is transparent to Hyper-V or SQL server and it appears that the IO operation was stalled for a short time only. This transparent failover capability or continuous availability is what makes SOFS a highly available storage solution. Table 3 lists the three most common deployment configurations. Table 3: Microsoft SOFS Sizing Configurations Number of storage nodes Number of JBODs Maximum disk counts 2-Node/1-JBOD Node/2-JBOD Node/4-JBOD The SAS cabling diagrams for the three configurations are shown in Figure 9, Figure 10, and Figure 11. Figure 9: 2-Node/1-JBOD SAS Cabling Diagram (up to 60 disks) 17 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

21 Figure 10: 2-Node/2-JBOD SAS Cabling Diagram (up to 120 disks) 18 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

22 Figure 11: 4-Node/4-JBOD SAS Cabling Diagram (up to 240 disks) 19 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

23 6.3 Networking Depending on the network scalability requirements, this solution uses two Lenovo RackSwitch G8124E or G8264 rack switches for HA and performance capabilities and is ideal for I/O-intensive storage software applications, such as Microsoft SOFS. These Lenovo top-of-the-rack switches support the latest CEE standard, which is required for network adapters that have RDMA capability. Network adapters that have RDMA can function at full speed with low latency while using little CPU. For workloads (such as Hyper-V or Microsoft SQL Server), this feature enables SOFS storage to resemble local direct-attached block storage. As shown in Figure 12 and Figure 13, each storage node has one independent 10 Gbps Ethernet connection to each Lenovo rack switches. Figure 12: Logical SOFS Networking view There are two 10 Gbps connections total for each storage node. There is no network adapter teaming requirement for the storage node. Microsoft SOFS load balances I/O traffic across both connections, which means there is no requirement for inter-switch links (ISL) between two switches in this solution. This solution focused on the hardware components for Microsoft SOFS. It is assumed that there is a network infrastructure that supports client connections, Active Directory, DNS, and DHCP servers that is in place at the customer premise. For best practices, the switches that are described in this solution can be dedicated for storage traffic between Hyper-V or SQL servers and storage nodes. This configuration ensures the best throughput and low I/O latency for storage. For more information, see Networking BOMs on page Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

24 Figure 13: Physical Layer Rack and Networking view 6.4 Systems management Lenovo XClarity Administrator is a centralized resource management solution that reduces complexity, speeds up response, and enhances the availability of Lenovo server systems and solutions. The Lenovo XClarity Administrator provides agent-free hardware management for Lenovo s System x rack servers and Flex System compute nodes and components, including the Chassis Management Module (CMM) and Flex System I/O modules. Figure 14 shows the Lenovo XClarity administrator interface, where Flex System components and rack servers are managed and are seen on the dashboard. Lenovo XClarity Administrator is a virtual appliance that is quickly imported into a virtualized environment server configuration. 21 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

25 Figure 14: XClarity Administrator interface 6.5 Windows Server 2012 R2 The Microsoft SOFS solution requires the Windows Failover Cluster feature from Windows Server 2012 R2 Standard or DataCenter edition. 22 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

26 6.6 Deployment example Although the scope of this paper focuses on SOFS storage, it is beneficial to see how it is deployed in the enterprise production environment. Figure 15 shows an example of the complete solution in a production environment. The customer has an initial capacity requirement of 120 hard drives with mixed of SSDs and HDDs. The configuration for this deployment is two storage nodes and two JBODs. Figure 15: End-to-end SOFS solution In this example, Microsoft SOFS storage is used in a Hyper-V private cloud or Microsoft SQL server environment. As shown in Figure 15, it is assumed that the enterprise customer has an infrastructure that includes Windows Active Directory domain, client computers, network switches and router, Hyper-V, and SQL server hosts. When SOFS storage is deployed, it is providing storage resources for Hyper-V and SQL server hosts that are connected via the dedicated dual 10 GbE RDMA network interfaces (R-NIC) and 10 GbE switches (switch 5 and 6). The two other network adapter ports in each SOFS storage node are connected to the switch 3 and 4 for Active Directory authentication, cluster management, and cluster private network communication. 23 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

27 7 Deployment considerations This section describes noteworthy deployment considerations. 7.1 Systems Management Systems management for this solution uses the built-in Windows Failover Cluster Manager Tool or Windows Powershell cmdlets. Optionally, this solution can be managed by Microsoft System Center if it is deployed in the environment. Lenovo System x servers provide System Center Management Packs so that you can manage the entire solution from a single console. 7.2 Storage sizing and tuning SOFS storage sizing and tuning is an important element of providing optimal storage performance for Microsoft Hyper-V and SQL servers. The following outlines some of the key aspects of storage sizing and tuning for SOFS that uses Storage Spaces Storage sizing There are two considerations when it comes to storage sizing: IO requirements and usable capacity. The ideal use case for Microsoft SOFS storage solution is that the daily application working set is stored in the SSD storage tier for performance and the infrequently used data is stored in the slower NL-SAS storage tier for capacity. The best approach is to monitor the production environment for some time and analyze how much data is used daily. Each environment is unique; however, as general best practice and recommendation, the SSD-to-NL-SAS disk capacity ratio is usually 5-10%. For example, if the NL-SAS storage tier is 100TB, the SSD tier should be 5-10 TB Storage resiliency Similar to RAID capabilities in RAID adapters or SAN storage, Storage Spaces in SOFS provides storage resiliency. Table 4 lists the supported resiliency types and general deployment recommendations or use cases. Table 4: SOFS Resiliency Type and Target Workloads Resiliency Type Number of Data Copies Maintained Deployment Recommendations Mirror Parity 2 (two-way mirror) 3 (three-way mirror) 2 (single parity) 3 (dual parity) All Workloads Sequential workloads such as archival or backup (not recommended for random workloads) Simple 1 Workloads that do not need resiliency or provide alternative resiliency mechanism, such as data replication that is built into the application. 24 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

28 7.2.3 Storage write-back cache Storage Spaces can use SSDs for storage tiering and a write-back cache (WBC). WBC can improve performance by handling bursty small random write I/O operations to virtual disks. By default, Storage Spaces allocates 1 GB from SSD tier and you can increase this size to improve write performance. For more information, see the following websites: technet.microsoft.com/en-us/library/dn aspx#bkmk_wbc technet.microsoft.com/en-us/library/hh848643(v=wps.630).aspx Storage column and interleave sizing Similar to the RAID Group concept, the number of columns in Storage Spaces specifies how many physical disks data is striped across at once. More columns provide better performance because of parallelism. Interleave is stripe size or the amount of data written at once to a column. You can adjust the interleave parameter to fit the average I/O size of the application to avoid split I/O and improve performance. For more information, see the following websites: social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-quest ions-faq.aspx?pageindex=3#what_are_columns_and_how_does_storage_spaces_decide_how _many_to_use social.technet.microsoft.com/wiki/contents/articles/15200.storage-spaces-designing-for-performa nce.aspx MPIO load balancing policy Adjusting the MPIO load balancing policy other than the default can improve performance. For more information, see this website: technet.microsoft.com/en-us/library/jj649803(v=wps.630).aspx Allocation unit size As a best practice, format the CSV volumes with 64 K allocation unit size. This size provides better performance for Hyper-V, SQL, and Exchange Servers Network adapters and switches RDMA configurations RDMA tuning and configurations are essential to the performance of the SOFS solution. For more information about RDMA setup, see these websites: implementerslab.com/artifacts/2c2c78a8-24d b2f3-970eeb1d73e9/elx_sis_all_smbdir ect_oce14000_windows2012_appnote_microsoftnew.pdf blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direc t-smb-over-rdma-and-the-mellanox-connectx-3-using-10gbe-40gbe-roce-step-by-step.aspx mellanox.com/related-docs/whitepapers/wp_deploying_windows_server_eth.pdf 25 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

29 7.2.8 Lenovo G8264 OS application guide For configurations of G8264 RackSwitch to support RDMA capability, read Part 4 (section on CEE) of the G8264 Application Guide. The following most common features are used for supporting network adapters that have RDMA capability: Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic that is based on the 802.1p priority value in each packet s VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth that is based on the 802.1p priority value in each packet s VLAN tag. Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. For more information, see this website: ibm.com/support/docview.wss?uid=isg3t Hardware certifications This entire solution is certified for Windows and Storage Spaces and the Bill of Materials in section 8 reflects the Lenovo certified hardware. When newer hardware is released from Lenovo, you can replace the hardware in this solution with the newer hardware if the hardware is Windows and Storage Spaces certified. For more information about Microsoft certification, see this website: windowsservercatalog.com/ 7.4 Deployment best practices To be successful in deploying Microsoft SOFS, adhere to the following general deployment best practices: Verify purchased hardware Verify the disks and enclosures to make sure that the disk performance of the same disk model is consistent. This verification can be useful in isolating issues, such as hardware components that might have different firmware levels because of different manufacturing and purchase dates. For more information, see the following websites: o o gallery.technet.microsoft.com/scriptcenter/storage-spaces-physical-7ca9f304 technet.microsoft.com/en-us/library/jj aspx Deploy and configure Microsoft SOFS. o How-To Deploy Clustered Storage Spaces: o Step-By-Step for Tiered Storage Spaces: -ti ring-in-windows-server-2012-r2.aspx Validate Microsoft SOFS performance: Use an I/O performance tool to verify the SOFS performance. Test workloads against Microsoft SOFS: Run your application workloads against SOFS to ensure SOFS meets your I/O requirements. 26 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

30 8 Appendix: Lenovo Bill of materials This appendix features the Bill of Materials (BOMs) for different configurations of hardware for Microsoft SOFS deployments. There are sections for storage servers and networking. The BOM lists in this appendix are not meant to be exhaustive and must always be confirmed with the configuration tools. Any description of pricing, support, and maintenance options is outside the scope of this document. 8.1 Recommended configurations Table 5 lists the three recommended configurations with the number and type of SAS cables that are needed for each configuration. Table 5: Recommended configurations of servers and JBOD enclosures Configurations x3550 M5 or x3650 M5 JBOD 6 Gb SAS 12 Gb SAS cables Server Nodes Enclosures cables (msas-hd to msas) P/N 39r6529 P/N 00MJ166 2-Node/1-JBOD Node/2-JBOD Node/4-JBOD Storage Node BOMs The Bill of Materials for the storage nodes are listed in Table 6, Table 7, and Table 8. Table 6: Netapp JBOD Code Description Quantity 00D1653 NetApp DE6600 Disk Enclosure 1 46W " drive carrier for DE Y6210 4TB 7.2K 6Gbps NL SAS 3.5" G2HS HDD 48 00D " driver carrier for DE6600 (SSD) 12 00W GB SAS 3.5" MLC HS Enterprise SSD 12 46W0474 DE6600 Rear Rack Mount Bracket (L&R) 1 27 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

31 Table 7: System x3550 M5 Code Description Quantity 5463AC1 Lenovo System x3550 M5 1 A59F System Documentation and Software-US English 1 A5AY System x 750W High Efficiency Platinum AC Power Supply 1 A5B7 16GB TruDDR4 Memory (2Rx4, 1.2V) PC CL MHz LP RDIMM 8 A5AY System x 750W High Efficiency Platinum AC Power Supply 1 A59V System x3550 M5 Planar Select Storage devices - configured RAID 1 A2K7 Primary Array - RAID 1 (2 drives required) 1 ASBF 600GB 10K 12Gbps SAS 2.5" G3HS 512e HDD 2 A45W ServeRAID M1215 SAS/SATA Controller 1 A5BK Intel Xeon Processor E v3 12C 2.3GHz 30MB Cache 2133MHz 120W 1 A5BZ Addl Intel Xeon Processor E v3 12C 2.3GHz 30MB 2133MHz 120W 1 A1ML Integrated Management Module Advanced Upgrade 1 A5AL System x Enterprise 1U Cable Management Arm (CMA) 1 A59W System x3550 M5 4x 2.5" HS HDD Kit 1 A5AK System x3550 M5 Slide Kit G4 1 A5AB System x Advanced LCD Light Path Kit m, 10A/ V, C13 to IEC 320-C14 Rack Power Cable 2 A5AF System x3550 M5 PCIe Riser 2, 1-2 CPU (LP x16 CPU1 + LP x16 CPU0) 1 A5M0 N2225 SAS/SATA HBA 2 A5AH System x3550 M5 PCIe Riser 1 (1x ML2 x16 CPU0) 1 A40Q Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter * 1 A413 2U Bracket for Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter 1 A5Q6 2U bracket for N2225 SAS/SATA HBA 1 A1PJ 3m Passive DAC SFP+ Cable 2 28 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

32 Table 8: System x3650 M5 Code Description Quantity 5462AC1 Lenovo System x3650 M5 1 A5FR System x3650 M5 PCIe Riser 1 (1 x16 FH/FL + 1 x8 ML2 Slots) 1 A5FY System x3650 M5 2.5" ODD/LCD Light Path Bay 1 A5FX System x Enterprise 2U Cable Management Arm (CMA) 1 A5R5 System x3650 M5 PCIe Riser 2 (1 x16 FH/FL + 1 x8 FH/HL Slots) 1 A5FD System x3650 M5 2.5" Base without Power Supply 1 A5EU System x 750W High Efficiency Platinum AC Power Supply 2 A5EN Addl Intel Xeon Processor E v3 12C 2.3GHz 30MB 2133MHz 120W 1 A5B7 16GB TruDDR4 Memory (2Rx4, 1.2V) PC CL MHz LP RDIMM 8 A5EY System Documentation and Software-US English 1 A1ML Integrated Management Module Advanced Upgrade 1 A5GV Intel Xeon Processor E v3 12C 2.3GHz 30MB Cache 2133MHz 120W 1 A40Q Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter 1 A5G3 System x3650 M5 2.5" ODD Bezel with LCD Light Path 1 A5M0 N2225 SAS/SATA HBA 2 A45W ServeRAID M1215 SAS/SATA Controller 1 A5G6 x3650 M5 8x 2.5" HS HDD Assembly Kit (Single RAID) 1 ASBF 600GB 10K 12Gbps SAS 2.5" G3HS 512e HDD m, 13A/125-10A/250V, C13 to IEC 320-C14 Rack Power Cable 2 A5FV System x Enterprise Slides Kit 1 ASFE Notice for Advanced Format 512e Hard Disk Drives 1 A4EL HDD Filler ASM GEN 3 Quad Filler 2 A5G5 System x3650 M5 Riser Bracket 2 A5FT System x3650 M5 Power Paddle Card 1 A5G4 System x3650 M5 ODD Filler 1 A4C2 HDD Filler ASM GEN 3 6 A5G1 System x3650 M5 EIA Plate 1 A5V4 System x3650 M5 Right EIA 1 A4VH Lightpath LCD Op Panel 1 A5EA System x3650 M5 Planar 1 29 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

33 8.3 Networking BOMs The Bill of Materials for the networking switches are listed in Table 9 and Table 10. Table 9: RackSwitch G8124E Code Description Quantity 7159BR6 Lenovo System Networking RackSwitch G8124E (Rear to Front) m, 10A/ V, C13 to IEC 320-C14 Rack Power Cable m Blue Cat5e Cable 1 A1DK Lenovo 19" Flexible 4 Post Rail Kit 1 Table 10: RackSwitch G8264 Code Description Quantity 7159G64 Lenovo System Networking RackSwitch G8264 (Rear to Front) m, 10A/ V, C13 to IEC 320-C14 Rack Power Cable 2 A3KP Lenovo System Networking Adjustable 19" 4 Post Rail Kit SFP+ SR Transceiver 2 A1DP 1m QSFP+-to-QSFP+ cable 1 A1DM 3m QSFP+ DAC Break Out Cable 0 30 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

34 Resources For more information about the topics that are described in this document, see the following resources: Storage Spaces Frequently Asked Questions (FAQ): social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-question s-faq.aspx Control Scale-Out File Server Cluster Access Point Networks: jeffgraves.me/2014/10/27/control-scale-out-file-server-cluster-access-point-networks/ The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0: blogs.technet.com/b/josebda/archive/2012/05/13/the-basics-of-smb-multichannel-a-feature-of-windo ws-server-2012-and-smb-3-0.aspx SMB Transparent Failover making file shares continuously available: blogs.technet.com/b/clausjor/archive/2012/06/07/smb-transparent-failover-making-file-shares-contin uously-available.aspx Storage Spaces - Designing for Performance: social.technet.microsoft.com/wiki/contents/articles/15200.storage-spaces-designing-for-performance.aspx Cluster Shared Volume (CSV) Inside Out: blogs.msdn.com/b/clustering/archive/2013/12/02/ aspx 31 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

35 Trademarks and special notices Copyright Lenovo References in this document to Lenovo products or services do not imply that Lenovo intends to make them available in every country. Lenovo, the Lenovo logo, ThinkCentre, ThinkVision, ThinkVantage, ThinkPlus and Rescue and Recovery are trademarks of Lenovo. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used Lenovo products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-lenovo products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by Lenovo. Sources for non-lenovo list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. Lenovo has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-lenovo products. Questions on the capability of non-lenovo products should be addressed to the supplier of those products. All statements regarding Lenovo future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local Lenovo office or Lenovo authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in Lenovo product announcements. The information is presented here to communicate Lenovo s current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard Lenovo benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-lenovo websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this Lenovo product and use of those websites is at your own risk 32 Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces