A Balanced Approach to Optimizing Storage Performance in the Data Center

Transcription

1 A Balanced Approach to Optimizing Storage Performance in the Data How a mix of HDDs and SSDs can maximize the performance of your workloads October 19, 2015 Mark T. Chapman Lenovo.com/systems

2 Executive Overview Not long ago, data centers had but one option for real-time storage of data: hard disk drives. The only real questions were about which capacities, speeds, form-factors, and interfaces to use. HDDs were, and continue to be, the workhorses of most data centers. There are many workloads that perform extremely well with all-hdd arrays; however, the all-hdd data center is quickly becoming a thing of the past. Solid-state drives (SSDs) emerged a few years ago and have been accepted in many data centers. This is because SSDs offer benefits for specific workloads that HDDs cannot match. On the other hand, SSDs have limitations of their own and are not a panacea for all data center storage woes. There is a vast array of storage technologies available, offering a wide range of performance and pricing across all of the various products, and many ways in which to implement a storage solution. So how can IT personnel decide what is the best approach to use in their data center: all-hdd, all-ssd, or a hybrid tiered combination of the two? And if the latter, how should the combination be configured for a data center s workloads? This paper highlights the advantages and disadvantages of each type of storage, in terms of cost, performance, capacity, and endurance. It also provides an overview of how storage can be tiered for best performance and the workloads that make the best use of SSDs and a hybrid storage infrastructure. Note: Except where specified otherwise, the information in this paper refers to technologies in general and not to particular products or vendors. 2

3 An overview of the advantages / limitations of different storage media Each type of storage media has its strengths and weaknesses 1. Much or all of this section may already be familiar to many readers, but it is presented for those who may not be. HDDs are a well-known factor. They have been used effectively with virtually every workload for decades. Therefore, this paper does not describe all the ways in which HDDs can be used. We will primarily mention the pros and cons of HDDs. However, because SSDs are a relatively new technology to many people we will describe the various types of SSDs and their respective strengths and weaknesses. In the following sections, we will present ways in which HDDs and SSDs can be used together in a data center to maximize the benefits of each type of storage and offset their respective limitations. Hard disk drives HDDs are available in many combinations of speeds, interfaces, form-factors, and capacities. As a class (meaning taking all types of HDDs into account), HDDs offer a number of advantages that meet nearly all requirements for storage: High-density storage capacity (8TB or more) using 3.5-inch nearline storage Relatively low cost per-drive and per-gigabyte of storage (especially nearline drives) A variety of interface types, including 2Gb to 16Gb Fibre Channel (FC), 6Gb and 12Gb SAS, and 6Gb SATA Fast, balanced read/write performance, using 15k and 10k drives (especially 12Gb SAS and 16Gb FC drives) Redundancy and high availability, using RAID and hot-swap drives Ease of installation and replacement, using hot-swap drives Self-encrypting drives (SEDs) are available to provide strong data security There also are disadvantages to using HDDs: Large numbers of HDDs in a data center create a lot of heat, require a large amount of energy (for power and cooling), and create quite a bit of noise. 1 The advantages and disadvantages presented here may be drive-specific. Not all advantages and disadvantages apply to all drives. 3

4 A performance bottleneck can occur for certain applications that require high read throughput, low latency, or high IOPS (I/O operations per second). The drives may not be able to keep up with the demand. Uneven failure rate. Although a category of drives for example 3.5-inch 10k has a predictable failure rate, individual drives can fail unexpectedly much sooner than average. Long rebuild times for RAID-5 arrays after a drive failure. HDD throughput is not increasing as quickly as HDD capacities, creating the potential for further bottlenecks in the future. And others. Many data centers use a two- or three-tier design to work around many of these limitations. This structured approach uses a number of 15k SAS HDDs to store performance-critical ( hot ) data. The majority of the data is stored on 10k SAS or SATA HDDs in a second ( warm ) tier where performance is not as critical, and to reduce cost. An optional third tier holds the infrequently accessed (archival) or noncritical cold data on high-capacity nearline 7200 rpm SATA drives, to further reduce cost. Tier-3 data is typically accessed sequentially, in large blocks, where high random read/write performance is unnecessary. This approach is effective for many data centers. However, for some workloads even this is not fast enough. Other strategies incorporate SSDs into the design to create a hybrid storage infrastructure. (See Creating a hybrid storage infrastructure using HDDs and SSDs, below.) Solid-state drives Typically, SSDs look to an OS exactly like an HDD, requiring no learning curve or software changes 2 to implement SSDs in a data center. In general, an all-ssd array can replace an all- HDD array directly, especially for read-intense workloads. However, for a number of reasons it may be more advantageous to implement a hybrid tiered infrastructure for the majority of workloads, to make the most effective use of both SSDs and HDDs. SSDs are available in different technologies, each with its own strengths and weaknesses. By far, the most common type of flash memory used in enterprise SSD products is NAND, which is available in two classes: SLC (single-level cell) and MLC (multi-level cell). SLC-based SSDs 2 There are exceptions, where applications may need to be changed to point to different SAN addresses. 4

5 store one bit per memory cell, as opposed to MLC-based drives, which store 2, 3, or more bits per cell. These extra bits multiply the capacity of the drives compared to SLC drives. SLC drives are faster and have longer write lifecycles than MLC drives. (Every NAND cell has a known finite number of times the cell can be written to before it wears out.) On the other hand, MLC drives have greater storage capacity and lower cost per gigabyte, but are somewhat slower and with shorter lifecycles than SLC. In fact, as MLC capacity grows the lifecycle shrinks. Ironically, in some cases SSDs are too fast, in the sense that they can overload a 6Mbps SAS interface. This can mean a server is unable to take advantage of all the throughput available from the SSDs attached to it. Fortunately, the faster 12Gb SAS and 10Gb or 16Gb FC interfaces can overcome this limitation in most configurations. Yet, some SSD environments can produce enough throughput to overload even these pipelines. Relief in that regard is on the way. The first NVMe (Non Volatile Memory express) controllers should be rolling out in late These connect a drive directly to a server s PCIe bus, rather than though a SAS or FC interface. 3 This promises significantly faster bandwidth for storage transfer. How much faster? A typical SATA connection tops out at about 500 MBps throughput, and a SAS connection at 1.5 GBps. An NVMe connection on a PCIe bus can theoretically support almost 4 GBps, using a x4 PCIe 3.0 connection (or 2 GBps for PCIe 2.0), and 2x or 4x that much for a x8 or x16 PCIe channel, respectively. (Naturally, implementations may vary by server and by vendor.) Some workloads require the fastest possible read performance. Many data centers use SSDs to cache certain types of files, including logs, journals, temporary tables and hot tables, instead of using server memory. (This frees up that memory for application use). Other data centers cache all of a workload s read-intensive files, but keep the write-intensive files on HDDs in a lower tier. In addition, fast, low-capacity SSDs are often used as OS boot drives, due to the high read performance and minimal writes required. 3 NVMe-connected SSDs should not be confused with PCIe SSDs, which are mounted on adapters in PCIe slots. NVMe SSDs are installed in drive bays, like HDDs and most SSDs. 5

6 In addition, SSDs can be used as the first tier of a two-, three-, or even four-tier hybrid strategy. (See Creating a hybrid storage infrastructure using HDDs and SSDs, below.) As a class, SSDs offer a number of advantages that can best satisfy the storage requirements for read-intensive applications that stress read performance, low latency, and high IOPS. SSDs can also be used in conjunction with server DRAM as a means of supporting larger inmemory databases with extremely high performance. SSD advantages include: Extremely high read performance. Extremely high IOPS performance. Extremely low latency. Extremely low cost-per-iops. Low energy consumption and low heat output for active drives; idle SSDs consume almost no energy and produce almost no heat. Silent operation; there are no moving parts to make noise Some SSDs offer high capacity (up to 3.84TB) combined with much higher IOPS than HDDs. Redundancy and high availability, using RAID and hot-swap drives Extremely fast rebuild times for RAID-5 arrays Ease of installation and replacement, using hot-swap drives Self-encrypting drives (SEDs) are available to provide strong data security Predictable failure rate. Using an SSD s wear-tracking software, a user can see approximately when a drive is likely to fail (essentially, wearing out more than an acceptable number of memory cells) in time to take preventive action. A new generation (2015) of enterprise-class SSDs provide higher capacities than 10k and 15k enterprise HDDs, with better cost-per-gigabyte than 15k SAS HDDs. (For example, the recently announced 2.5-inch hot-swap Lenovo 3.84TB 6Gb SAS Enterprise-Capacity SSD has a list price that is approximately 10.5x the cost of Lenovo s 300GB 15k 6Gb 2.5-inch Hot- Swap SAS HDD, but provides almost 13x the capacity. 4 4 Prices current as of October 19, Source: 6

7 There also are disadvantages to using SSDs: On a cost-per-drive or cost-per-gigabyte basis, most SSDs are still significantly more expensive than HDDs. Due to the high throughput of SSDs, 1Gb Ethernet may not be fast enough to prevent bottlenecks. Upgrading to 10GbE increases cost. 2Gbps and 4Gbps Fibre Channel likewise may require replacement with 8GB, 10Gb, or 16Gb FC. Most SSDs provide relatively low capacity (<2TB). The highest capacity SSDs provide lower write performance than many enterprise HDDs and fewer IOPS than the fast, lower-capacity SSDs No FC implementations Not all servers (especially at the low end) support SSDs. Newer SSDs incorporate technologies to offset some of their limitations. For example, to minimize flash memory wear-out and extend drive endurance, SSDs use wear-leveling firmware to spread data writes evenly across all memory cells. This prevents some cells from being written to significantly more than others and wearing out prematurely. Also, the SSDs implement overprovisioning. Overprovisioning is the difference between the stated and actual capacities. Depending on the type (SLC or MLC) and purpose (entry, mainstream, performance, or enterprise capacity) of the SSD, drives may contain anywhere from 7-50% more flash memory than the drive specs indicate. This spare memory is used to substitute for cells as they wear out, thus extending the useful life of the drive. Creating a hybrid storage infrastructure using HDDs and SSDs Different workloads stress different aspects of a storage infrastructure. Some are more readintensive and some are more write intensive. Many require balanced read/write performance. Others require high IOPS/low latency. However, even within a workload, there may be some files that benefit from high read performance, latency, and IOPS, while other files require high write performance and maximum capacity. Most of these workloads can make effective use of SSD storage in a hybrid HDD/SSD infrastructure. Only the ratio of SSDs to HDDs changes by workload. Many data centers still run an all-hdd infrastructure, however their numbers are dwindling. Today, an all-ssd tiered infrastructure is also feasible for some data centers, using small SSDs with the fastest read/iops performance (typically SLC flash) as a front-end for slower high- 7

8 capacity (MLC) SSDs that hold the bulk of the data. Yet this kind of tiering is not yet appropriate for a large majority of data centers. For most data centers, the advantages of a hybrid HDD/SSD tiered configuration make it the ideal solution. It offers the potential for a large variety of configurations that allow a data center to customize the design for their specific needs. This type of tiered configuration can provide higher read performance and IOPs than an all-hdd configuration, along with higher write performance and potentially larger storage capacity and lower acquisition cost than an all- SSD configuration. And tiering can even be combined with SSD caching, if caching works better for some workloads and tiering for others within the same data center. Note: The main difference between using SSDs for data caching versus tiering is that with caching files are kept permanently on HDDs and then copied temporarily to the SSDs as needed. With tiering, read-intensive files are stored on the SSDs. To reiterate, in an all-hdd tiered infrastructure, the three tiers are segmented by HDD speed. 15k SAS drives hold the hot data, with warm data stored on 10k drives and cold data on 7200 rpm SATA drives. A hybrid tiered infrastructure, in contrast, stores the most performance-critical read-intensive or IOPS-intensive (hot) data on a small number of SSDs (in place of HDDs). The majority of the active (warm) data is stored on 15k SAS and 10k SAS or SATA HDDs in the second tier, where write performance (or a balance of reads and writes) is emphasized. (A variation uses 15k drives for the second tier and moves the 10k drives into a third tier.) An optional bottom tier holds the cold data on high-capacity low-cost SATA drives, as before. Which approach works best in a given data center (all-hdd arrays, hybrid SSD/HDD arrays, or some of each) will depend on the unique combination of workloads used within that data center. Many if not most workloads can make effective use of SSDs to some extent in the storage infrastructure, either to cache hot data, or to store it there permanently. Examples of the types of data that can benefit from the speed of SSDs include: Applications that require lightning-quick read times, high IOPS, and extremely low latency Database files/records that need to be read quickly and concurrently 8

9 Files that are kept in DRAM for the fastest possible computational speed, but where there is insufficient memory to hold everything; also in-memory databases with the same limitation Files used for indexing, cataloguing, and searching Online sales catalogs Client desktop images subject to delays from VDI boot storms The following are just some of the types of workloads that can take advantage of SSDs: Big Data/Analytics Collaboration Data warehousing ecommerce High-Frequency Trading High-Performance Computing (HPC) / Technical Computing Media streaming Medical Imaging Online Analytical Processing (OLAP) Online Transaction Processing (OLTP) Video-on-demand (VOD) Virtualization / Virtual Desktop Infrastructure (VDI) Web serving Web-based applications Storage tiering offers many advantages over using a single tier for all data. But not if an administrator has to spend a large amount of time monitoring data to determine which files are hot, warm, or cold at any given time. Or having to manage the movement of all that data, which can heat up and cool off rapidly. Fortunately, that does not have to be the case. Automated tiering software monitors the I/O patterns of workload data usage and then determines which files would benefit from being stored on SSDs and which on HDDs. Usage patterns can vary according to the time of day (for example, a morning VDI boot storm vs. overnight batch processing), the day of the week (payroll processing on Tuesdays, or weekdays vs. weekends), and even seasonally (due to holiday purchases or tax season). Therefore, the tiering software also determines when to move data between SSDs and HDDs. (Depending on the vendor, the software may run either on the server, or on the storage devices themselves.) 9

10 Data movement generally is done via policies, which are set by the administrator. Default policies supplied with the software typically can be extensively customized as needed. For example, with one workload, 90% of the data access may occur on 5% of the data. Another workload may see 70% of the access occurring on 40% of the data during the day; at night, 95% of the I/O accesses occur on 12% of the data. And so on. Each workload can have multiple unique policies. As an example of an automated tiering solution, the Lenovo Storage S2200 and S3200 SAN storage units include Lenovo Intelligent Real Time Tiering. The software continuously monitors data on SSDs and HDDs for changes in usage. Every 5 seconds the software automatically moves newly hot data to the Tier 1 SSD layer and warm and cold data to the Tier 2 and Tier 3 HDD layers. The result of doing real-time tiering (as opposed to periodically or daily, as competitive offerings do) in a hybrid tier with 5-10% flash capacity is overall performance that is approximately 80% of an all-ssd environment, for about 75% lower acquisition cost than for all-ssd. 5 What is the future of data center storage? A number of evolving industry and societal trends, as well as governmental regulations, will force data centers to adapt or fall behind their competition. One such trend is toward greater data security, which in part can be addressed by using self-encrypting HDDs and SEDs (among many other security solutions). Another is a gradual shift from an all-hdd data center to one that incorporates SSDs into a hybrid solution. Will we ever see an all SSD data center? SSD technology is advancing quickly, as evidenced by the recent announcements of 3.84TB enterprise-class SSDs, and projections that 30TB SSDs will arrive by Per-drive and pergigabyte costs are dropping rapidly. (HDD per-drive and per-gigabyte costs also continue to drop, although presently at a slower rate.) Industry experts differ in their views of where the future of data center storage is headed: 5 Using list pricing from shop.lenovo.com, for a 24-drive array: hybrid array (4 x 2.5-inch 1.6TB SAS SSDs, 12 x 2.5-inch 900GB SAS HDDs, 8 x 3.5-inch 6TB HDDs, 1 x S SAN storage unit, and 1 x SSD Data Tiering Upgrade license) vs. an all-ssd array (24 x 2.5-inch 1.6TB SAS SSDs, and 1 x S SAN storage unit). 10

11 Some predict that enterprise SSD prices will drop to below that of HDDs in the next year or two, while SSD capacities will increase to beyond that of HDDs potentially with even greater endurance than HDDs. 6 This belief is aided by new 3D technologies that allow SSD memory cells to be stacked in layers for greater density/capacity, and thereby lower the cost-pergigabyte significantly. Write endurance is expected to increase dramatically as well, because the greater storage density will allow for more spare capacity that can be activated automatically as memory cells wear out. In this view, just as punch cards gave way to reel-to-reel tapes, and HDDs eventually replaced tape drives as a primary storage medium, HDDs will ultimately cede dominance of the data center to SSDs. On the other hand, other pundits believe it may be many years before dominance shifts from SSDs to HDDs, due in part to upcoming advances in HDD technology and their belief that while SSDs may close the price gap, HDDs will continue to be less expensive (due in part to these technology advances). 7 Upcoming advances include perpendicular recording which creates denser, higher-capacity drives and helium-filled drives that enable more platters in the same space, also increasing density and capacity. Technologies such as these will allow vendors to continue driving down the cost of HDD storage, and HDDs may continue to stay ahead of the cost curve, compared to SSDs. In addition, currently installed drives will continue to run for years, perhaps being replaced then by SSDs; or perhaps not. And even then, HDDs will probably still be used for inexpensive, high-capacity archival storage for many years to come. Summary SSDs and HDDs each offer benefits the other lacks, and each can offset the limitations of the other. Although there are certainly workloads that dictate an all-hdd or all-ssd infrastructure, the vast majority of workloads and data centers will benefit from a hybrid approach that incorporates both types of medium and applies each where it is best suited. 6 For example,

12 It is probable that SSDs will eventually replace HDDs. However, it is impossible at this point to predict when the shift to SSDs will reach a tipping point. It may take two years, or a decade. Until then, SSDs will accelerate the performance of some workloads, and HDDs will remain the bedrock of many, if not most, workloads. Important note: The views of the two sources referenced in this section largely apply to consumer HDDs and SSDs for use in PCs and laptops. The economics and requirements of consumer versus enterprise storage are quite different. Thus the predictions, may differ considerably for enterprise drives as used in servers. The predictions should be used in the context of general trends of enterprise storage, rather than as specific timeframes for change. 12

13 LENOVO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. This information could include technical inaccuracies or typographical errors. Changes may be made to the information herein; these changes will be incorporated in new editions of the publication. Lenovo may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any performance data contained herein was determined in a controlled environment; therefore, the results obtained in other operating environments may vary significantly. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Any references in this publication to non-lenovo Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of those Web sites is at your own risk. Lenovo, the Lenovo logo, System x, and ThinkServer, are trademarks of Lenovo in the United States, other countries, or both. Intel and Xeon are trademarks of Intel Corporation in the United States, other countries, or both. Microsoft, Windows Storage Server 2012, Windows Server 2012, and the Windows Logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Lenovo.com/systems 13