itransformer: Using SSD to Improve Disk

Transcription

1 itransformer: Using SSD to Improve Disk Scheduling for High performance I/O Xuechen Zhang Song Jiang Wayne State t University it Kei Davis Los Alamos National Laboratory

2 Challenges of data management using parallel file systems Processing huge amount of data LANL combustion application generates 5 TB data per run. Checkpointing p gdatasets are 22 TB per timestep. Particle accelerator Large Hadron Collider at CERN generates over 15 PB of data per year. Handling very high I/O concurrency Intrepid p at Argonne has 164k cores. I/O intensive applications are concurrently executed. Increased I/O concurrency can compromise the efficiency of hard-disk-based parallel file systems, which are designed to handle huge amount of data. 2

3 Handling concurrency without disk scheduling Track number of incoming requests Disk service order w/o scheduler 1, 5, 7, 21, 3, 4, 2, 10 Seek distance 1, 5, 7, 21, 3, 4, 2,

4 Handling concurrency with I/O scheduler Track number of incoming requests 1, 5, 7, 21, 3, 4, 2, Scheduling of the first four requests Scheduling of the second four requests. Request service order: 1, 5, 7, 21, 2, 3, 4, 10 Using I/O scheduler (assuming queue size is 4): disk seek distance is 42. I/O scheduler helps reduce disk seek distance. 4

5 Handling concurrency with I/O scheduler of extended dispatchqueue Track number of incoming requests , 5, 7, 21, 3, 4, 2, Request service order: 1, 2, 3, 4, 5, 7, 10, 21 Using I/O scheduler (assuming queue size is 8): disk seek distance is 20. Increasing the queue length can further reduce the seek distance and help improve I/O efficiency.

6 Effect of queue size on I/O performance 1) Iozone a file system benchmark; 2) Various data a access patterns; 3) CFQ I/O scheduler; 4) 8GB data file; 5) 256 threads; 6 A large queue can effectively recover spatial locality if available in requests. When individual threads issue fully random requests, increasing queue size generates limited improvements.

7 Limitations of increasing queue size in disk scheduling Having a large queue would allow many write requests to be outstanding in volatile DRAM, running the risk of losing data. Keeping requests in the queue without being completed may increase requestresponse response times and violate QoS requirements. Increasing thequeuesize may not besufficient to address the issue of concurrency among streams of random requests. 7

8 Our solution: usingsolid state solid state disksto improve diskschedulingscheduling In-memory Queue This queue is only responsible for dispatching requests if relatively strong locality can be identified in the queue. itransformer In-SSD Queue Extension 1) Requests are sent to the queue extension on the SSD for further scheduling. 2) SSD is non-volatile, keeping dirty data in the queue extension safe. 3) A write request is considered complete once it arrives at the queue. 8

9 Talk outline Introduction Design and Implementation Performance evaluation Related work Conclusions 9

10 10 itransformer: a new disk schedulingframework usingsolid state solid state disks Design objectives Recover and exploit spatial locality lit in the I/O requests; Hide random access latency on the hard disk. The challenges How to effectively handle scientific workloads which have large working data sets and relatively weak locality? When scientific program processes a large data set, data accesses exhibit weak temporal locality, which is hard to exploit by a relatively small SSD for effective caching. When I/O concurrency is high, handling a large amount of random access can overwhelm SSD as a small storage device for random data. How to reduce overhead for multi level scheduling using both inmemory queue and in SSD queue extension? Serving sequential requests using SSD as buffer may increase I/O lt latency.

11 Scheduling modes Determining the scheduling modes MEM mode (in memory queue) When spatial locality is strong, only in memory queue is used for dispatching. Requests are directly sent from in memory queue to disk. SSD mode (in memory queue and in SSD queue extension) When spatial locality is weak, requests are sent from in memory queue to in SSD queue extension and then dispatched from SSD to disk. Weak Locality MEM mode SSD mode 11 Strong Locality

12 Locality measurement of dispatch queues 12 Locality measurement of requests from queues The average distance of disk head movement for serving requests To statistically quantify the spatial locality, we use the method which is similar to the one developed in Linux. Locality ofrequests fromin memoryqueue (Lin_mem) Locality of requests from in SSD queue extension (Lin_SSD) Locality estimation when SSD is not used When SSD is not in use, we still need an intelligent method to estimate the best locality which can be achieved if SSD had been used. itransformer maintains i a separate ghost queue to hold the metadata of any reqeusts dispatched out of the in memory queue. We run the standard disks scheduler over the requests in the ghost queue. Locality of requests from the ghost queue (Lghost) Comparing the localities li i to determine the transition ii of modes Lin_mem VS. Lin_SSD or Lin_mem VS. Lghost

13 Out of band writeback for random writes itransformer handles random write via out of band writeback to opportunistically reduce I/O latencies of random writes. When random writes are detected, dt tditransformer uses SSD to cache the data and delays the service of write requests as long as the SSD queue is not full. Write requests are later dispatched from SSDswhen hard disks are not busy. 13

14 Data prefeching for random reads Random reads Random reads are difficult to handle even when SSD is used. itransformer tracks history of random reads to predict and prefetch data to be requested into SSDs. Tracking I/O read activities Disk address space is partitioned into slots of equal size. A slot is the minimal unit for prefetching. We designed an algorithm to identify slots that random read requests have moved into and out of multiple l times. The slots which are randomly accessed in high frequency becomesthe candidates for prefethcing. 14

15 Illustration of the LRU stack S: slot number; C: access frequency; Slot Sequence: 3, 1, 1, 2 1, 1, 2 1, 2 2 Null Top S:1 C:1 S:3 C:2 S:1 C:2 S:1 C:2 S:2 C:1 S:9 C:1 S:1 C:1 S:3 C:2 S:3 C:2 S:1 C:2 S:5 C:1 S:9 C:1 S:9 C:1 S:9 C:1 S:3 C:2 S:3 C:1 S:5 C:1 S:5 C:1 S:5 C:1 S:9 C:1 When a slot s counter value is greater than a threshold, the slot becomes a prefetchable slot. 15

16 itransformer implementation itransformer is prototyped with instrumentation of the Linux device mapper as a stand alone kernel module in the generic block layer; To activate itransformer in a cluster system administrators only need load the module into the kernel. To maintain i the data consistency itransformer writes dirty data on the SSD back to disk on unloading of the module. During initializationitransformer itransformer checks if there are anydirty data left in the SSD because of system failure and rebuilds the mapping table. 16

17 Talk outline Introduction Design and Implementation Performance evaluation Related work Conclusions 17

18 Experimental setting Darwin cluster at Los Alamos National Laboratory 120 nodes and 8 data servers 48 core 2GHz AMD Opteron 6168, 64GB memory RAID0 consisting of two 500GB 7200rpm disks 120GB SSD Software configuration Kernel CFQ for RAID, NOOP for SSD MPICH compiled with ROMIO PVFS2 parallel file system 18

19 Comparison of storage devices SSD Hard disk RAID0 Capacity 120GB 1TB Interface SATA SAS Seq. Read 160MB/s 170MB/s Seq. Write 140MB/s 160MB/s Ran. Read 60MB/s 15MB/s Ran. Write 30MB/s 5MB/s 19

20 The ior mpi io benchmark Random access pattern Performance improvement up to 2.4X 20

21 On disk data access pattern Disk Disk SSD 21

22 The Hpio benchmark 22 Read Write 4.5X improvement on average 1) large number of random small writes 2) sync flush dirty data and meta-data t

23 The BTIO benchmark Performance increasingly suffers since request size is reduced to 200Bytes at 1024 processes. 23

24 The S3asim benchmark itransformer reduces I/O times by up to 66%. 24

25 Related work Hardware Solutions Use storage devices which are not sensitive to random access, e.g. solid state disk. It is not cost effective for scientific applications Large working set; Weak Spatial locality; Software strategies Adopt techniques inparallel I/Omiddleware, e.g. collective IO. Requires specific interfaces; It is not designed for multiple applications; Disk schedulers in Linux kernel, e.g. CFQ, NOOP, 25

26 Conclusions Existing schedulingmechanism makes disks vulnerable to high I/O concurrency. We propose itransformer a two level scheduling framework usingin SSD queueextension extension to handlei/o concurrency and randomness of workloads. Evaluation with representative parallel I/O benchmarks show itransformer increases I/O throughput by upto 3X and 35% on average. 26

27 Q & A Contacts: Xuechen Zhang, xczhang@wayne.edu Kei Davis, kei.davis@lanl.gov Song Jiang, sjiang@eng.wayne.edu 27

28 28 Performance impactof prefetch area size

29 Related work Middleware approaches to handlingi/o concurrency Collective I/O [Thakur et al. FRONTIERS 99] SSD based cache in thememoryhierarchy hierarchy Flashcache [Srinivasan et al. at Facebook] Readyboost [Microsoft] SSD based hybrid storage system Combo drive [Payer et al. 2009] I CASH [Ren et al. HPCA 11] Hystor [Chen et al. ICS 2011] 29