Early Experience and Evaluation of File Systems on SSD with Database Applications

Transcription

1 Early Experience and Evaluation of File Systems on SSD with Database Applications Yongkun WANG, Kazuo GODA, Miyuki NAKANO, Masaru KITSUREGAWA The University of Tokyo 1

2 Outline Motivation Flash SSD Basic Performance Study Performance Evaluation by TPC C Benchmark Conclusion and Future Work 2

3 Motivation Flash SSDs are likely to be used in enterprise storage platforms for achieving high performance in data intensive applications IO path management techniques should be evaluated carefully Existing systems are designed for traditional hard disks IO performance features of flash SSD are different from that of hard disk For better utilization of SSDs in DBMS Evaluate basic performance of SSDs Evaluate performance of IO path in conventional DBMS With different file systems and IO schedulers 3

4 Flash SSD Flash SSD (Solid State Drive) A package of multiple flash memory chips FTL (Flash Translation Layer) provides block device emulation Performance properties of flash Memory (Samsung K9XXG8UXM) READ (4KB) takes 25us PROGRAM (4KB) takes 2us ERASE (256KB) takes 15us Erase before program design can lead to poor performance in a normal in place write system Flash SSD SATA Port Bus FTL Controller Chip SDRAM Buffer Flash Memory Bus NAND NAND Flash Flash Flash Memory Memory Memory Chip NAND NAND Flash Flash Flash Memory Memory Memory Chip NAND NAND Flash Flash Flash Memory Memory Memory Chip 4

5 Outline Motivation Flash SSD Basic Performance Study Performance Evaluation by TPC C Benchmark Conclusion and Future Work 5

6 Purpose of Basic Performance Study Clarify the performance between SSD and HDD Clarify the performance difference among SSDs Clarify the erase problem on SSDs 6

7 Experimental System Dell Precision 39 Workstation Dual core Intel Core 2 Duo 1.86GHz 2GB Memory SATA 3.Gbps Controller CentOS bit Kernel Hard Disk (HDD) Hitachi HDS7217, 3.5, 72RPM, 32M Cache, 75GB Flash SSD Mtron PRO 75 SLC, GB Flash SSD Intel X25 E SLC, GB Flash SSD OCZ VERTEX EX SLC, GB Inside each device, read ahead pre fetching and write back caching are enabled

8 Micro Benchmark One million requests for each case Request Size: 512B to 256KB Access patterns Sequential Read/Write Random Read/Write Mixed Random (5% Read plus 5% write) Number of outstanding IOs One outstanding IO: submit one IO request at a time 3 outstanding IOs: submit 3 IO requests at a time 8

9 Basic Performance of Flash SSDs ~ Sequential Access ~ IO Throughput [MB/s] K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K64K128K256K IO Size [bytes] IO Size [bytes] IO Size [bytes] IO Size [bytes] HDD Mtron Intel OCZ 1% Read 1% Write The read throughput of Intel s SSD and OCZ s SSD is much higher The write throughput of Intel s SSD is higher Write throughput of Intel s SSD drops quickly after the request size is larger than 32KB The performance gap between read and write throughput of OCZ s SSD is large 9

10 Basic Performance of Flash SSDs ~ Random Access (Single outstanding IO) ~ IO Throughput [K IOPS] K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K 128K 256K 512 1K 2K 4K 8K 16K 32K 64K 128K 256K512 1K 2K 4K 8K 16K 32K 64K128K256K IO Size [bytes] IO Size [bytes] IO Size [bytes] IO Size [bytes] HDD Mtron The read IOPS of SSD is much higher than that of HDD. Intel OCZ The performance of random write drops drastically on Mtron s SSD and OCZ s SSD. The performance of mixed access also drops drastically on Mtron s SSD and OCZ s SSD. [Bathtub effect, by Freitas on FAST21 tutorial] % Read Read 1% Write Write 5% MixRead 5% Write 1

11 Basic Performance of Flash SSDs ~ Random Access (3 outstanding IOs) ~ IO Throughput [K IOPS] K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K128K256K 512 1K 2K 4K 8K 16K 32K 64K128K256K IO Size [bytes] IO Size [bytes] IO Size [bytes] IO Size [bytes] HDD Mtron Intel OCZ 1% Read Read (3 outstanding IOs) 1% Read Read (one outstanding IOs) The read throughput is improved Intel s SSD and OCZ SSD. 11

12 Basic Performance of Flash SSDs ~ Response Time Distribution of 4KB Random Access ~ Cumulative Frequency [%] Response Time [us] 1 HDD Mtron Random Read (blue line) Most of random reads could complete in a very small range of response times on SSDs Random Write (red line) Response Time [us] The random write behavior is different among three SSDs Response Time [us] Response Time [us] Intel OCZ 1% Read Read 1% Write Write 5% MixRead 5% Write 12

13 Outline Motivation Flash SSD Basic Performance Study Performance Evaluation by TPC C Benchmark Conclusion and Future Work 13

14 Purpose of Evaluation By TPC C Provide evaluation on the IO behaviors of SSDs running an actual database application Two file systems, two DBMSs and four IO schedulers Investigate the detailed behavior of IO path 14

15 Experimental System Dell Precision 39 Workstation Dual core Intel Core 2 Duo 1.86GHz 2GB Memory SATA 3.Gbps Controller CentOS bit Kernel Hard Disk (HDD) Hitachi HDS7217, 3.5, 72RPM, 32M Cache, 75GB Flash SSD Mtron PRO 75 SLC, GB Flash SSD Intel X25 E SLC, GB Flash SSD OCZ VERTEX EX SLC, GB Inside each device, read ahead pre fetching and write back caching are enabled

16 System Configuration TPC C benchmark 5.1 Database settings MySQL: InnoDB Commercial DBMS File system options Ext2fs (ext2) Nilfs2 IO scheduler No operation (Noop) Anticipatory Deadline Completely Fair Queuing (CFQ) Kernel Tracer Database Application (TPC-C Benchmark) DBMS (MySQL, Commercial DBMS) OS kernel File System (, ) IO Schedulers Device Driver (SATA) Disk for OS HDD for Database Flash SSDs for Database 16

17 Configuration of TPC C Benchmark 3 warehouses, with 3 virtual users Key and Think time was DBMS configuration for TPC C benchmark Commercial DBMS MySQL(InnoDB) Data buffer size 8MB 4MB Log buffer size 5MB 2MB Data block size 4KB 16KB Data file fixed, 5.5GB, database size is 2.7GB Synchronous IO Yes Yes Log flushing method flushing log at transaction commit 17

18 File Systems Ext2fs (ext2) In place update Seek then read Seek then update a b c d read write data page Buffer c b d a Disk Nilfs2 An example of log structured file system Seek then read Random writes => sequential writes a b c d obsolete data page Buffer c b d a a b c d Disk 18

19 Experimental Study Transaction Throughput IO Throughput Buffer Size Workload Property IO Scheduler 19

20 Transaction Throughput Intel s SSD is better than HDD. Mtron s SSD is better than HDD with LFS. OCZ s SSD is better than HDD with. The performance difference is caused by the combination of SSDs and file system Transaction Throughput [tpm] 14, 12, 1, 8, 6, 4, 2, HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL 2

21 IO Path Investigation Logical IO is captured at the system call level, where DBMS call the service routine of OS kernel. Physical IO is captured at the device driver level, where the IO requests are sorted and merged, ready to be served by the device. Logical IO Kernel Tracer Physical IO Database Application (TPC-C Benchmark) DBMS (MySQL, Commercial DBMS) OS kernel File System (, ) IO Schedulers Device Driver (SATA) Disk for OS HDD for Database Flash SSDs for Database 21

22 Logical IO Throughput The transaction throughput follows the results of the logical IO throughput. Transaction Throughput Logical IO Throughput Write Read Transaction Throughput [tpm] 14, 12, 1, 8, 6, 4, 2, Read/Write Rate by DBMS [MB/s] HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS 22 MySQL

23 Physical IO Throughput Logical IO Throughput Physical IO Throughput Write Read Write Read HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL HDD Mtron Intel OCZ HDD Mtron Intel OCZ 23 Commercial DBMS MySQL Read/Write Rate by DBMS [MB/s] Read/Write Rate to device [MB/s]

24 Physical IO Throughput (Read) Large amount of reads are absorbed by the file system buffer cache Logical IO Throughput Physical IO Throughput Write Read Write Read HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL HDD Mtron Intel OCZ HDD Mtron Intel OCZ 24 Commercial DBMS MySQL Read/Write Rate by DBMS [MB/s] Read/Write Rate to device [MB/s]

25 Physical IO Throughput (Write,) Large amount of reads are absorbed by the file system buffer cache. For, write throughput are almost the same between logical throughput and physical throughput. ( Synchronous IO) Logical IO Throughput Physical IO Throughput Write Read Write Read HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL HDD Mtron Intel OCZ HDD Mtron Intel OCZ 25 Commercial DBMS MySQL Read/Write Rate by DBMS [MB/s] Read/Write Rate to device [MB/s]

26 Physical IO Throughput (Write, ) Large amount of reads are absorbed by the file system buffer cache. For, write throughput are almost the same between logical throughput and physical throughput. ( Synchronous IO) LFS() produces additional writes at the physical IO layer, which has a serious impact on the overall transaction throughput. Logical IO Throughput Physical IO Throughput Write Read Write Read Read/Write Rate by DBMS [MB/s] Read/Write Rate to device [MB/s] HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL HDD Mtron Intel OCZ HDD Mtron Intel OCZ 26 Commercial DBMS MySQL

27 Physical IO Size The average request size of Physical IO Read Size () Write Size () Read Size () Write Size () Average Read/Write Size [bytes] 6, 5, 4, 3, 2, 1, 18,392 17, , ,352 HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL 27

28 Physical IO Size (HDD, Mtron) The average request size of Physical IO The avg. write size of LFS is much larger than that of, which is beneficial for hard disk and some SSD such as Mtron s SSD HDD Read Size () Write Size () IO Throughput [MB/s] Average Read/Write Size [bytes] 6, 5, 4, 3, 2, 1, Read Size () Write Size () 18,392 17, , ,352 HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS MySQL IO Throughput [MB/s] ,24 2,48 4,96 8,192 16,384 32,768 65, ,72 262,144 IO Size [bytes] Mtron 512 1,24 2,48 4,96 8,192 16,384 32,768 65, ,72 262,144 IO Size [bytes] 28

29 Physical IO Size (Intel, OCZ) The average request size of Physical IO The avg. write size of LFS is much larger than that of, which is beneficial for hard disk and some SSD such as Mtron s SSD. Large write size is not beneficial on Intel s and OCZ s SSD, as shown in the basic performance study. This helps to explain the inferior transaction throughput on. Average Read/Write Size [bytes] 6, 5, 4, 3, 2, 1, Read Size () Read Size () 18,392 17, ,115 HDD Mtron Intel OCZ HDD Mtron Intel OCZ Commercial DBMS Write Size () Write Size () MySQL 186,352 IO Throughput [MB/s] IO Throughput [MB/s] Intel 512 1,24 2,48 4,96 8,192 16,384 32,768 65, ,72 262,144 IO Size [bytes] OCZ ,24 2,48 4,96 8,192 16,384 32,768 65, ,72 262,144 IO Size [bytes]

30 Database Buffer Size (Mtron) The throughput is improved when increasing the buffer size Commercial DBMS MySQL 18, 4, 16, 3,5 Transaction Throughput [tpm] 14, 12, 1, 8, 6, 4, 2, Transaction Throughput [tpm] 3, 2,5 2, 1,5 1, 5 8M 16M 32M 64M 128M 256M 512M 1G Buffer Size [bytes] 4M 8M 16M 32M 64M 128M 256M 512M 1G3 Buffer Size [bytes]

31 Workload Property Measure with three types of workloads Speedup of over is increasing when the percentage of readwrite transactions is increased 25, speedup 12 Transaction Type IO Property read intensive % of mix normal write intensive New Order Read Write Payment Read Write Delivery Read Write Stock Level Read Only Order Status Read Only Transaction Throughput [tpm] 2, 15, 1, 5, read normal write intensive intensive read normal write intensive intensive speedup Commercial DBMS Buffer Size [bytes] MySQL 31

32 IO Schedulers Noop No operation Anticipatory Merge the IO requests, and re order in an elevation manner Deadline Impose the deadline for each request Completely Fair Queuing (CFQ) Balance the service time of IOs among processes 32

33 Transaction Throughput with IO Schedulers IO scheduling does not affect the transaction throughput largely. Transaction Throughput [tpm] Noop Anticipatory Deadline CFQ Mtron Intel OCZ Mtron Intel OCZ Commercial DBMS MySQL 33

34 Conclusion and Future Work We study the basic performance characteristics of flash SSDs We measure and analyze the application performance and the IO behavior on three flash SSDs and two file systems with TPC C benchmark. Transaction Throughput Logical IO Throughput Physical IO Throughput We plan to study IO path management techniques for database applications running on flash SSDs. 34

35 Q&A Thank you very much! 35