Performance Beyond PCI Express: Moving Storage to The Memory Bus A Technical Whitepaper

Transcription

1 : Moving Storage to The Memory Bus A Technical Whitepaper By Stephen Foskett

2 April

3 Introduction In the quest to eliminate bottlenecks and improve system performance, the state of the art has continually moved storage ever closer to the server CPU. External storage arrays, once the champions of performance, are now decidedly second tier compared to solid state storage located inside the server. Until now, the fastest storage attachment was PCI Express, with such flash storage devices commanding attention. But what if there was something better? Moving storage to the memory bus promises to make I/O faster, more scalable and more consistent. 3

4 The Limits of PCI Express PCI Express (PCIe) would seem to be the ultimate location for storage. After all, previous storage attachments (Ethernet, Fibre Channel, and InfiniBand) relied on PCIe cards for server connectivity. This is why it was so impressive when companies placed flash memory directly on PCIe cards, with no additional intermediate bus. But PCIe is a limited resource: Servers have just a few dozen PCI lanes, so issues of contention arise. These are the fundamental limits of PCIe-based storage: 1. Because PCIe is not a native storage interface, PCIe storage solutions need an onboard controller to allocate resources and mediate between the flash chips on the card and the demands of server I/O coming from the bus. 2. Because PCIe lanes are limited, systems can only hold a few PCIe storage devices, or they must make do with fewer lanes and less available bandwidth. 3. Because PCIe slots are limited, it can be difficult to scale a PCIe storage solution without a major investment, and slots are in demand for other peripheral cards. 4. Because PCIe is bus mastering, it can interrupt system memory access when a DMA transfer is invoked. While this task is in process, no other movements of data to or from the flash storage can take place. These limits do not always make themselves felt, but performance-sensitive applications are already reaching them. NAND flash chips are incredibly fast, but placing them behind a PCIe controller keeps applications from exploiting their performance potential. Another concern is the amount of time an application must wait for a read or write operation to be completed, a concept known as I/O latency. Layers of virtualization and translation, as well as many outstanding I/O operations, can make this aspect of performance unpredictable. Finally, many applications simply require more flexible capacity than a single PCIe card can deliver: Some will use less than the capacity of the smallest card, while others will 4

5 require slightly more than the capacity of the cards available. Flooding the PCIe SSD Controller PCI Express is a scalable systems bus using a number of serial interconnects known as lanes to transfer data between expansion cards and a computer. These PCIe lanes typically terminate at the system chipset, with most modern systems having 16 or 32 lanes. Originally supporting 250 MB/s for each lane, PCIe Gen 2 can transfer 500 MB/s over each lane. This means that a 32-lane Gen 2 system has a maximum aggregate throughput of 16 GB/s over PCIe using all of the available lanes. Each PCIe card is limited to a fraction of the available lanes in the system. Lowend solid-state drives (SSDs) and host bus adapters (HBAs) might use 4 lanes, while extreme-performance options could use as many as 16. This gives a maximum aggregate bandwidth of between 2 GB/s and 8 GB/s for this type of solution. And of course other peripheral cards must also use some of the PCIe lanes. Most high-performance applications are limited more by the total number of I/O operations per second (IOPS) they require than by the maximum amount of bytes they can send. These applications can queue hundreds of thousands of read and write operations to the I/O subsystem in seconds, overwhelming conventional storage solutions and forcing further processing to wait until the I/O operations complete. Consider a typical storage device: For each read or write request, it must receive and validate the operation, look up the proper location for the data, perform the read or write, and return the result to the requesting system. This process is the same whether the device is a hard disk drive, an enterprise storage array, or a PCIe card. The only difference is that the PCIe card is located on a fast, low-latency bus and can see many more operations than a remote device. 5

6 This relationship between latency, IOPS, and throughput has long been a focus of storage system optimization. No solution can maintain maximum performance when crushing levels of I/O requests are outstanding. The challenge in system design is to provide sufficient performance at an acceptable level of latency. 1 This flood of data can cause puzzling performance issues for PCIe storage devices. As shown below in a graph of real-world benchmark data produced by Diablo Technologies, a typical PCIe storage device can easily handle tens of thousands of IOPS without any trouble. But once enough operations are pending, the wait time (latency) for each operation increases. In this example, 72,000 IOPS sent the latency shooting up past 1 millisecond 2. The PCIe SSD controller was simply overwhelmed with I/O requests. Figure 1: IOPS vs. Latency This 100% writes example is extreme, to be sure, since writes are slower to 1 For example, see the article, The Fundamental Characteristics of Storage at FlashDBA, 2 For another example, see The SSD Review s Micron P320h HHHL 700GB PCIe Enterprise SSD Review, 6

7 perform with NAND flash than reads. But it shows how difficult it is for even a high-performance PCIe storage controller to keep up with tens of thousands of writes every second. In this situation, the application would see each I/O operation becoming slower, even as the storage system was able to keep accepting more work. Indeterminate Latency This brings up the issue of predictable latency. If an application can have I/O operations completed in well under 100 microseconds most of the time, yet latency shoots up 10x or more once the workload gets heavy, it can interfere with application performance. Even worse, this happens just when the application is the busiest. Imagine a system designed to handle high-volume transactions. A developer might profile a new PCIe storage device by generating what he considers to be a reasonable volume of requests and determine that it can process tens of thousands of I/O operations per second with under 100 microseconds of latency. Happy with this level of performance, he architects his new trading system to assume that all reads and writes will be processed in less than this time. But once he puts it into production, a surge of data could easily cause performance to drop by a factor of 10 or more. This unpredictable latency is illustrated in the graph below 3. Here, a PCIe subsystem is able to perform just as described above. Yet the controller periodically delays I/O operations, causing latency to spike. In this case, the performance degradation happened even under moderate load, as the controller performed tasks like wear leveling and garbage collection. If an application was relying on predictable performance, it would be unsuccessful. 3 This chart shows performance measurements using the storage subsystem from AMPS, a highperformance messaging database platform from 60East Technologies. 7

8 Figure 2: AMPS 15% Read Mix Contention for PCIe or flash controller resources can make performance unpredictable in similar ways. If a system has multiple high-performance PCIe cards, a workload spike on one can interfere with the others. This is true of I/O cards like Ethernet and InfiniBand adapters as well as PCIe storage devices. The sort of applications that demand extreme levels of performance also require consistency. They must be written to tolerate worst-case latency because these I/O subsystems simply cannot guarantee performance at all times. So storage must be over-specified to ensure that worst-case latency does not exceed tolerable thresholds. 8

9 Introducing Memory Channel Storage (MCS ) When examining the graph above, the orange line immediately draws attention. What is this MCS with lower latency than PCIe and, more importantly, predictability of performance under load? Although most server I/O passes over the PCIe bus, systems have another, separate interface for memory. This memory bus can push tens of gigabytes per second at latencies measured in nanoseconds. This interface has only been used for dynamic RAM in the past, rather than any type of non-volatile storage. As the industry moved storage to the PCIe bus, technologists considered what might come next. Since CPUs already have prodigious memory capacity, and solid-state storage already relies on memory chips (albeit NAND flash rather than dynamic RAM), attention turned to adding persistent storage on these memory channels. Dynamic RAM is very expensive, and many systems have empty slots left unused. Memory Channel Storage is the creation of Diablo Technologies, and this is the MCS shown on the graph above. It places NAND flash on cards which reside alongside DRAM in system memory slots. Of course, flash is not the same as DRAM, so systems need special drivers to make use of this type of storage. But Diablo has developed the supporting technologies needed to make MCS feasible. The benefits of moving storage to the memory bus are many: 1. Applications can access persistent storage more quickly. Moving flash to the memory channel places storage even closer to the CPU than PCIe, reducing latency and improving performance. 2. Applications can have both high IOPS and low latency simultaneously. Memory subsystems are designed to be highly parallel, and systems can spread the I/O load across multiple MCS cards more easily than trying to use multiple PCIe cards. 9

10 3. Applications can create, read, update, and delete data with predictable response times. Memory channels must be highly deterministic to support system RAM, so there is a much lower chance of contention hampering performance. 4. Memory access is integrated with the CPU. Moving data from MCS-based storage devices to system memory is no more than a copy from one address to another within the memory controller, keeping the processor or system bus access from being interrupted each time data is transferred. Thus, many of the limitations that can become apparent when performancesensitive applications are moved to PCIe flash storage are reduced or eliminated through the use of Memory Channel Storage. Predictable performance can be critical for proper system sizing, and the storage architecture can match application needs more exactly, both in terms of capacity and performance. 10

11 Moving Storage to Memory Channels Storage architectures are being optimized per application, with capacity and performance diverging dramatically. Data-heavy applications are moving to scaleout storage while high-performance applications are using server-side flash. The sort of applications that demand extreme performance are the same that will be affected by unpredictable latency and limited scalability. Memory Channel Storage promises an alternative architecture that delivers more of both. This is a clever approach to utilize resources already present in today s servers, namely empty memory slots. In the short term, applications gain scalable persistent storage with predictable performance. But the future could see servers shrink as fewer PCIe slots and drive bays are needed. One can also envision a future where applications are written to take advantage of non-volatile memory just as they currently use DRAM. 11

12 About the Author Stephen Foskett is an active participant in the world of enterprise information technology, focused on enterprise storage, server virtualization, networking, and cloud computing. A long-time voice in the storage industry, Stephen has authored numerous articles for industry publications, and is a popular presenter at industry events. His contributions to the enterprise IT community have earned him recognition as both a Microsoft MVP and VMware vexpert. Foskett organizes the popular Tech Field Day event series for Gestalt IT. His company, Foskett Services, is focused on content and community, creating technical events, writing, and video productions and connecting the IT world through social media. He can be found online at FoskettServices.com, TechFieldDay.com, blog.fosketts.net, and on Twitter 12