Measuring Interface Latencies for SAS, Fibre Channel and iscsi Dennis Martin Demartek President Santa Clara, CA 1
Demartek Company Overview Industry analysis with on-site test lab Lab includes servers, networking and storage infrastructure Fibre Channel: 4 & 8 Gbps (16Gb soon) Ethernet: 1 & 10 Gbps (including NFS, CIFS, iscsi & FCoE) Servers: 8+ cores, very large RAM Virtualization: ESX, Hyper-V, Xen We prefer to run real-world applications to test servers and storage solutions Currently testing various SSD and FCoE implementations Web: www.demartek.com Santa Clara, CA 2
Demartek News Demartek Deployment Guides Completed: iscsi May 31, 2011 In-progress: SAS, SSD & 16Gb Fibre Channel Free Monthly Newsletter http://www.demartek.com/newsletter/newsletter_main.html Text DemartekLabNotes to 22828 Storage Interface Comparison http://www.demartek.com/demartek_interface_comparison.html Internet search for Storage Interface Comparison Santa Clara, CA 3
Latency is Important OLTP applications require short response times (small latencies) Some transactions require several successive queries in order to provide complete response HDDs have same maximum RPM (15000) as a decade ago, limiting latency improvements RAID5 striping and latency Latencies can actually increase with large disk groups! http://en.wikipedia.org/wiki/standard_raid_levels#raid_5_latency Santa Clara, CA 4
Host Storage Interfaces The host storage system interface matters We compared the performance and latency of the same storage system with multiple host interfaces What kinds of performance and latency would you expect from these four host interfaces? 1Gb iscsi, 10Gb iscsi, 6Gb SAS, 8Gb FC Santa Clara, CA 5
Evaluation Environment Santa Clara, CA 6
Performance Random IOPS 35,000 SQLIO - IOPS (Random) 30,000 25,000 20,000 15,000 10,000 5,000 0 4KB rnd 8KB rnd 16KB rnd 32KB rnd 64KB rnd 4KB rnd 8KB rnd 16KB rnd 32KB rnd 64KB rnd Direct 6G SAS Switched 6G SAS Direct 8G FC Switched 8G FC Direct 10G iscsi Switched 10G iscsi Direct 1G iscsi Switched 1G iscsi Santa Clara, CA 7
Performance Random MBPS 1200 SQLIO - MBPS (Random) 1000 800 600 400 200 0 4KB rnd 8KB rnd 16KB rnd 32KB rnd 64KB rnd 4KB rnd 8KB rnd 16KB rnd 32KB rnd 64KB rnd Direct 6G SAS Switched 6G SAS Direct 8G FC Switched 8G FC Direct 10Gb iscsi Switched 10Gb iscsi Direct 1Gb iscsi Switched 1Gb iscsi Santa Clara, CA 8
Performance Sequential IOPS 35,000 SQLIO - IOPS (Sequential) 30,000 25,000 20,000 15,000 10,000 5,000 0 4KB seq 8KB seq 16KB seq 32KB seq 64KB seq 128KB seq 256KB seq 4KB seq 8KB seq 16KB seq 32KB seq 64KB seq 128KB seq 256KB seq Direct 6G SAS Switched 6G SAS Direct 8G FC Switched 8G FC Direct 10G iscsi Switched 10G iscsi Direct 1G iscsi Switched 1G iscsi Santa Clara, CA 9
Performance Sequential MBPS 1400 SQLIO - MBPS (Sequential) 1200 1000 800 600 400 200 0 4KB seq 8KB seq 16KB seq 32KB seq 64KB seq 128KB seq 256KB seq 4KB seq 8KB seq 16KB seq 32KB seq 64KB seq 128KB seq 256KB seq Direct 6G SAS Switched 6G SAS Direct 8G FC Switched 8G FC Direct 10Gb iscsi Switched 10Gb iscsi Direct 1Gb iscsi Switched 1Gb iscsi Santa Clara, CA 10
Performance Latencies up to 5ms 100 Latencies - Up to 5 ms 90 80 Percentage of Results 70 60 50 40 30 20 10 0 0 1 2 3 4 5 Latencies (milliseconds) Direct 6G SAS Switched 6G SAS Direct 8G FC Switched 8G FC Direct 10G iscsi Switched 10G iscsi Direct 1G iscsi Switched 1G iscsi Santa Clara, CA 11
Performance Summary Performance rankings for this test: 1. 6Gb SAS 2. 8Gb FC (close second) 3. 10Gb iscsi 4. 1Gb iscsi iscsi imposes additional overhead, and even at 10Gb, has higher latency than 6Gb SAS and 8Gb FC Santa Clara, CA 12
Commentary Some real-world applications are moving towards larger I/O block sizes and more importance on bandwidth Virtualized servers result in higher percentage of random I/O for storage systems Deployment of SSDs can increase CPU utilization and network bandwidth needs I believe that at the current rate of price decreases and capacity increases, SSDs (probably NAND flash) will become the new standard for tier-1 storage by 2012 Santa Clara, CA 13
Demartek SSD Resources Demartek SSD Zone www.demartek.com/ssd.html Demartek involved in ongoing real-world testing of SSDs Contact me regarding our upcoming Deployment Guides (SAS, SSD, 16Gb FC) Santa Clara, CA 14
Contact Information (303) 940-7575 www.demartek.com http://twitter.com/demartek YouTube: www.youtube.com/demartek Skype: Demartek Dennis Martin, President dennis@demartek.com www.linkedin.com/in/dennismartin Santa Clara, CA 15