Moneta: A High-Performance Storage Architecture for Next-generation, Non-volatile Memories

Transcription

1 Moneta: A High-Performance Storage Architecture for Next-generation, Non-volatile Memories Adrian M. Caulfield Arup De, Joel Coburn, Todor I. Mollov, Rajesh K. Gupta, Steven Swanson Non-Volatile Systems Laboratory, Department of Computer Science and Engineering University of California, San Diego

2 The Future of Storage Hard Drives PCIe-Flash 2007 PCIe-NVM 2013? NVM Lat.: 7.1ms BW: 2.6MB/s 1x 1x 68us 250MB/s 104x 96x 12us 1.7GB/s 591x 669x = 2.89x/yr = 2.95x/yr *Random 4KB Reads from user space 2

3 Log Request Latency (us) Software Overheads File System OS Hardware Disk Flash PCM

4 Architecting a High Performance SSD Hardware architecture and software layers limit performance HW/SW interface critical to good performance Careful co-design provides significant benefits Increased bandwidth Decreased latencies Increased concurrency 4

5 Overview Motivation Moneta Architecture Optimizing Moneta Performance Analysis Conclusion 5

6 Ring (4 GB/s) Moneta Architecture Host via PIO Request Queue Scoreboard Ring Control Transfer Buffers DMA Control 16GB PCM 16GB PCM 16GB PCM Tag Status Registers 16GB PCM Host via DMA 6

7 Moneta Architecture Overview Application File System OS IO Stack Moneta Driver PCIe 1.1 x8 (2GB/s Full Duplex) 7

8 Advanced Memory Technology Characteristics Work with any fast NVM: DRAM-like speed DRAM-like interface Phase Change Memory Coming soon Simple wear leveling: Start Gap [Micro 2009] 8

9 Moneta: Modeling Advanced NVMs Built on RAMP s BEE3 board PCIe 1.1 x8 host connection 250MHz design DDR2 DRAM emulates NVMs Adjust timings to match PCM RAS-CAS Delay for reads Precharge latency for writes Read Write Projected PCM Latency 48 ns 150 ns Latency projections from [B.C. Lee, ISCA 09] 9

10 Overview Motivation Moneta Architecture Optimizing Moneta Interface & Software Microarchitecture Performance Analysis Conclusion 10

11 Latency (us) Software Optimization is Critical Baseline Latency (4KB) Hardware: 8.2 us Software: 13.4 us Optimizations Remove IO Scheduler Atomic Operations Lock Free Data Structures Atomic HW/SW Interface Spin-waits vs. sleeping Final 5us SW latency 62% reduction vs. Base Hardware costs PCM Ring DMA Wait Interrupt Issue Copy Schedule OS/User 11

12 Bandwidth (GB/s) Moneta Bandwidth Random Write Accesses 1M IOPS Access Size (KB) Base Base NoSched Base Base NoSched Atomic NoSched Ideal Atomic SpinWait Ideal Ideal Ideal 12

13 CPUs / GB/s CPU Utilization Random 4KB Read/Write Accesses Baseline NoSched Atomic Spin 13

14 Overview Motivation Moneta Architecture Optimizing Moneta Interface & Software Microarchitecture Performance Analysis Conclusion 14

15 Bandwidth (GB/s) Balancing Bandwidth Usage Full duplex PCIe should see better R/W BW Smarter HW request scheduling = more BW Two request queues: one for reads, one for writes Alternate between Qs on 0.5 each buffer allocation 0.0 Random R/W Accesses Access Size (KB) Single Queue RW Queues Ideal 15

16 Round-Robin Scheduling Prevent requests from starving other requests Allocate a buffer and then put request at back of queue Attains much better small request throughput in the presence of large requests 12x improvement in small request BW 16

17 NVMs for Storage vs DRAM Replacement Write coalescing Storage must guarantee durability by closing row DRAM leaves row open to enable coalescing Row buffer size should match access size Large accesses for storage Cache-line sized for memory Peak memory activity limited by PCIe BW Storage and DRAM replacement are different and should be optimized differently 17

18 Overview Motivation Moneta Architecture Optimizing Moneta Performance Analysis Conclusion 18

19 System Overview Moneta Memory and Device Interconnect Capacity Fusion-IO IODrive PCIe 4x 80GB SLC NAND Flash SW RAID-0 PCIe 4x SATA 2 Controller 128GB Disk HW RAID-0 PCIe 4x RAID Controller 4TB Moneta PCIe 8x 64GB Moneta-4x PCIe 4x 64GB 19

20 Bandwidth (GB/s) XDD Bandwidth Comparison 50/50 Read/Write Random Accesses Disk SSD FusionIO Moneta Moneta-4x Access Size (KB) 20

21 Bandwidth (GB/s) Bandwidth w/o File System Read Write Read Write Large Accesses (4MB) Small Accesses (4KB) Moneta FusionIO SSD-RAID DISK-RAID 21

22 Bandwidth (GB/s) Bandwidth with XFS Read Write Read Write Raw IO Large 4MB Accesses (4MB) Small 4KB Accesses (4KB) Moneta FusionIO SSD-RAID DISK-RAID 22

23 Log Speedup vs DISK-RAID Database & App Performance An Opportunity for Leveraging Hardware Optimizations XDD 4KB RW HashTable PTF Btree Bio XDD 4KB RW DGEMM lu bt sp is 23

24 Conclusion Fast advanced NVMs are coming soon We built Moneta to understand the impact of NVMs Found that the interface and microarchitecture are both critical to getting excellent performance Many opportunities to move software optimizations into hardware Many open questions exist about the architecture of fast SSDs and the systems they interface with 24

25 Thank You! Any Questions? 25