Flash In The Enterprise Technology and Market Overview Chris M Evans, Langton Blue Ltd Architecting IT January 2014 Doc ID: AI1401-01S
Table of Contents The Need for Flash Storage... 3 IOPS Density... 3 The I/O Blender... 3 Flash to The Rescue... 3 What is Flash?... 4 Flash Memory Technology... 4 Managing NAND... 4 Flash Form Factors and Packages... 4 Deployment Scenarios... 6 Server Deployment... 6 Caching Options... 6 Shared Storage... 7 Flash Optimised... 7 Flash Enhanced... 7 All Flash... 7 Requirements... 7 Justifying The Use of Flash Technology... 8 More Information... 9 The Author... 9 Copyright 2014 Langton Blue Ltd Page 2 of 9
The Need for Flash Storage Flash storage has become one of the most talked about technologies of the last 12 months and has moved from being a niche product to a mainstream consideration for storage in the Enterprise. This white paper discusses flash technology, what it is, why it should be used and how to use it. Since the mid 1950 s, storage deployments have always used spinning disks - hard disk drives (HDDs) as the main permanent storage medium. HDDs consist of one or more magnetic platters (or disks) accessed by a mechanical read/write head that moves across the surface of the platter, accessing data on concentric tracks. Although hard drive manufacturers have made great gains in increasing the areal or bit density of hard drives, managing to store ever increasing amounts of data within the same physical form factor, gains in performance have not kept pace with capacity. IOPS Density The relationship between performance and capacity can be quantified by a measurement called IOPS density. This is the ratio of input/output operations (I/Os) per second per unit of capacity, typically expressed as gigabytes or terabytes. Application IOPS requirements have increased over time as the performance of processors, memory and I/O backplanes has improved. The I/O Blender As server and desktop virtualisation becomes more prevalent, the workload profile of I/O on shared storage becomes much more unpredictable and as a result today s workload is highly randomised. This presents a particular problem for hard drives that have to mechanically move the write head into position and wait for the drive to rotate to the right position before data can be read. At worst case, each I/O could be random enough to include full rotational delay (waiting for the drive to turn) and seek time (moving the head to the right position), in total a value known as latency. Flash to The Rescue Flash storage offers a solution to both the I/O Density and I/O blender problems, delivering a high number of IOPS without the latency impact of a mechanical device. This makes it much more suited to today s demanding data centre environments where storage latency has become the main scalability bottleneck. Copyright 2014 Langton Blue Ltd Page 3 of 9
What is Flash? Flash storage is solid-state non-volatile memory, similar to that used in consumer devices, such as ipods and USB memory sticks. The contents of flash storage are not lost when power is removed from the device, making it a permanent storage medium. Today s flash devices are based around NAND logic technology and packaged with controller logic circuitry that manages the process of reading and writing from NAND enabling them to be suitable for enterprise-class environments. Flash has a number of benefits over traditional spinning media. Performance flash devices deliver high performance, both in terms of throughput and latency, even for random I/O. Environmental flash devices are lighter and consume much less power than their hard drive counterparts. They create no vibration and require less cooling. Although solid state drives are packaged to look like hard drives, in reality they can be made much smaller than the typical hard drive form factor and so have a high space density (capacity to space occupied) characteristic. Flash Memory Technology NAND Flash memory was originally implemented using a technology known as SLC or Single Level Cell. This stores one bit of data in each memory cell. SLC is expensive but more resilient than other forms. MLC or Multi-level Cell records multiple (typically 2) bits of information per cell. Both MLC and SLC technology have finite lifetimes, being capable of only a fixed number of write operations. SLC lasts longer than MLC, as it only has to store two possible values or states. Some vendors have introduced MLC technology using three or four bits per cell, by storing up to eight or sixteen states respectively and this is being used to create high capacity devices. Managing NAND Flash memory has a finite lifespan, limited by the number of times data can be written to the device. Data is written to a device in blocks (typically 4KB) and once the contents of a block are set it must be erased before it can be written again. Flash devices have a fixed number of erase cycles, usually around 100,000 for SLC and 10,000 for MLC devices. As a result, without some management, flash devices would have a short lifetime and be very unreliable. The flash storage vendors (those selling SSDs and arrays) use a range of management techniques in order to prolong the life of flash, including wear levelling, overprovisioning, write amplification and garbage collection. Flash Form Factors and Packages Flash can be packaged in a number of different ways. Today we see the following implementations of flash storage. Solid State Disk (SSD) these devices package flash in the form factor of a traditional hard disk and use existing disk-based protocols for connectivity, including SATA and SAS. SSDs have on-board controllers that both manage the interface with the host and the flash itself and implement management algorithms. Performance of SSDs is usually measured in submillisecond figures. PCIe SSD this is a host bus adaptor that uses the PCI Express bus to connect flash modules to the host computer. These devices deliver higher performance and lower latency than a traditional solid-state disk by bypassing the storage I/O bus and allowing direct access across the PCIe backplane. Some vendors implementations include built-in RAID and offloading of management tasks to dedicated on-board processors and memory. As there is no standard Copyright 2014 Langton Blue Ltd Page 4 of 9
access protocol in place, PCIe SSD devices usually require some host driver software and this can consume processor cycles and memory. PCIe performance is typically measured in tens of microseconds. NVDRAM this is a server memory slot or DIMM form factor device that uses the existing onboard memory slots normally reserved for DRAM on the server motherboard. By plugging directly onto the memory bus, NVDRAM delivers exceptionally high performance, typically measured around five to ten microseconds. The technology looks like DRAM and so BIOS changes and O/S drivers are required to differentiate the media as solid state and not volatile memory. Flash storage arrays take flash storage in one or more of the above forms and implement it in a number of ways, including all-flash and hybrid models. Copyright 2014 Langton Blue Ltd Page 5 of 9
Deployment Scenarios Flash technology can be deployed in a number of places within the data centre, including in the server itself, or in a centralised storage array or appliance. Server Deployment Flash technology can be deployed directly in the server, with three form-factor choices currently available - SSDs, PCIe SSD adaptors and NVDIMM (non-volatile DIMM) modules. Solid-state disks can be used simply as replacement for traditional hard drives and will boost performance by improving throughput and lowering latency. However when only used as a tier of storage, SSDs may be under utilised and of course not managed effectively to ensure optimum lifetime. SSDs can also be used as cache devices, speeding up I/O to and from traditional hard drives. These solutions need additional caching software, either as an add-on driver or software product or built into the operating system. PCIe SSD devices can be used either as cache or primary storage devices. Some PCIe SSD drives are automatically recognised as disk drives but most require additional software in form of drivers or software that integrate with the operating system and cache active data to improve performance. NVDIMM devices are now available from a number of vendors, most notably IBM in the form of their exflash offering. NVDIMMs have similar characteristics to PCIe SSD in that they bypass the storage I/O bus but are even closer to the processor backplane and have very low latency. Deployment of flash directly into the server removes the I/O overhead of the storage area network (SAN), lowering latency and improving throughput and performance. However directly attached devices are not shared and isolated to access only by that server itself. This causes problems if the server crashes or fails due to a hardware error. It also requires planning to cater for redundancy of the data if the server copy is the only copy available. Consequently many vendors offer solutions that act only as a read cache. Caching Options A cache stores copies of the most frequently accessed data in a server in order to improve I/O performance, however the cache copy is not the primary copy of data. Cache solutions can be used to improve either read or write performance or both. Read performance is improved by retaining frequently accessed blocks of data in cache and servicing I/O requests directly from the cache copy. If a block of data is updated, the cache is written with the new block and also stored on external disk, a process known as write-through caching. Cache devices may also be used for caching write I/O, a process known as write-back caching, where the I/O is acknowledged to the host as soon as it is stored in the flash cache and before de-staging to external disk. Although write caching improves write I/O performance, there is a data integrity risk if a hardware or software failure occurs before the cache copy has been flushed to disk. In addition, if a write cache is incorrectly sized and becomes full, then performance will revert back to disk I/O performance as every write operation will need to be flushed to disk before another one can be accepted into the cache. This means that correctly sizing and monitoring cache devices is extremely important. Copyright 2014 Langton Blue Ltd Page 6 of 9
Shared Storage Storage arrays, deployed as part of a storage area network or directly attached to a server can be ameliorated with flash devices. The initial use of flash in recent years has been in adding SSDs to traditional disk arrays. This provided some performance improvement but typically didn t make best use of the flash resources as the underlying architecture of these arrays was designed to cope with slow permanent storage media. As a result the implementations were only a stepping-stone to better devices. As the use of flash in arrays evolved, we have seen a split into three distinct categories of device, the flash-optimised traditional array, the flash-enhanced array and the all-flash array. Flash Optimised These arrays are based on traditional storage array devices but have been amended to cope with flash as the main storage media. Amendments include redesigning the array software to cope with flash characteristics and adding extra processing and memory capacity. Regardless of the amendments, the hardware itself still accesses the devices using the same interface as used with hard disk drives. Flash Enhanced These devices use flash to improve the performance of the array, through a variety of software-based techniques and new architectures. For example, the ZFS file system is implemented as the underlying storage file-system layer and this caches all writes to flash before committing asynchronously to disk. These devices are also known as hybrid arrays as they combine flash and hard drives together for a better price/performance ratio. All Flash These devices have been designed, architected and built to specifically work with only flash storage media. Their design intrinsically caters for the characteristics of flash devices and includes flash media delivered as solid-state disks, PCIe SSD or custom flash modules. The all-flash array category represents the fastest of devices, includes a greater set of space management features such as compression and de-duplication and are typically architected for long service life. Requirements Whatever category of product is chosen, the underlying requirements should be considered. following table highlights some of the considerations when choosing a specific platform. The Characteristic Flash Optimised Flash Enhanced All-Flash Performance Good, but can be hindered by Good, but can suffer Excellent, architecture designed the underlying HDD-based performance issues on read for high performance and architecture and needs I/O if data is not available scalability. software features to ensure through the cache. flash is used effectively. Cost Features Scalability Good, flash can be added as a complement to HDDs to target performance but needs Good, typically all the mature features of traditional arrays are available for flash disks. Good, these arrays are based on traditional technology, however they may not be capable of scaling fully in allflash designs. Acceptable; costs are contained by using flash in a limited fashion. Good, hybrid solutions usually support advanced features. Good, scaling is achieved through capacity with hard drives and through performance with flash. High. These devices are targeted at high performance workloads and flash is still relatively expensive. Some vendors are using de-duplication to reduce per GB costs. Acceptable. Most manufacturers have not supported advanced features like replication and snapshots. Acceptable. Most manufacturers have designed their solutions with limited scalability at this stage, although some vendors have built scalability into their design. Copyright 2014 Langton Blue Ltd Page 7 of 9
Justifying The Use of Flash Technology It is clearly obvious that flash storage can add significant performance improvements over traditional storage media. However, flash technology is more expensive than hard drive-based storage when compared purely on a $/GB basis and this can make justification for using the technology more difficult. Here are some thoughts on developing a flash deployment strategy. Identify requirements. Without a clear idea of either the business benefit delivered or problem solved, justification is almost impossible. This means a clear set of requirements need to be established with details on exactly where within the infrastructure flash would be used. Determine how much flash is needed. It s easy and tempting to think all-flash arrays are the solution to all requirements, however these devices are expensive and may provide way more throughput than can be utilised. By evaluating the performance improvement needed as part of requirements, the specific need for flash can be pinpointed, making it easier to determine whether server or array-based solutions are best. Consider $/IOPS. The traditional way of measuring storage cost is to look at the per TB or per GB price. This method was useful when storage was delivered as a single tier and became more complex with multi-tier solutions. The benefits from flash are not delivered in capacity terms but by throughput and latency, therefore it makes sense to use this metric to calculate the cost/benefit of implementing flash compared to traditional hard drives. Calculate business benefit or loss. Implementing flash may provide a number of quantifiable benefits, including higher transaction/order throughput, offsetting additional server or array purchases or significant savings in environmental charges. These can all be built into a business case. Evaluate. Flash is a new technology and the market is full of mature solutions in each category. Evaluating specific vendor offerings can provide more detail and knowledge on how flash will best work in a particular environment and validate whether vendor claims stand up to scrutiny. Use your vendor. There are a large number of vendors, both incumbent and start-up, looking for business. They have plenty of collateral in the form of white papers and guides that help justify the use of their products. Use Langton Blue. Vendors have great material and place their spin on the industry. Langton Blue provides independent advice without vendor hype. Copyright 2014 Langton Blue Ltd Page 8 of 9
More Information The full version of this report is available on request, containing additional technical background and vendor reviews. Contact enquiries@langtonblue.com for more information and pricing details. Langton Blue Ltd is hardware and software independent, working for the business value to the end customer. Contact us to discuss how we can help you bank the benefits in your technology investments. Website: www.langtonblue.com Email: enquiries@langtonblue.com Twitter: @langtonblue Phone: 0845 275 7085 Post: Langton Blue Ltd 78 York Street London W1H 1DP The Author Chris M Evans has worked in the technology industry since 1987, starting as a systems programmer on the IBM mainframe platform, while retaining an interest in storage. After working abroad, he cofounded an Internet-based music distribution company during the.com era, returning to consultancy in the new millennium. In 2009 he co-founded Langton Blue Ltd (www.langtonblue.com), a boutique consultancy firm focused on delivering business benefit through efficient technology deployments. Chris writes a popular blog at http://blog.architecting.it, attends many conferences and invitation-only events and can be found providing regular industry contributions through Twitter (@chrismevans) and other social media outlets No guarantees or warranties are provided regarding the accuracy, reliability or usability of any information contained within this document and readers are recommended to validate any statements or other representations made for validity. Copyright 2014 Langton Blue Ltd. All rights reserved. No portions of this document may be reproduced without the prior written consent of Langton Blue Ltd. Details are subject to change without notice. All brands and trademarks of the respective owners are recognised as such. Copyright 2014 Langton Blue Ltd Page 9 of 9