Accelerate MySQL Open Source Databases with SanDisk Non-Volatile Memory File System (NVMFS) and Fusion iomemory SX300 PCIe Application Accelerators

WHITE PAPER Accelerate MySQL Open Source Databases with SanDisk Non-Volatile Memory File System (NVMFS) and Fusion iomemory SX300 PCIe Application Accelerators March 2015 951 SanDisk Drive, Milpitas, CA 95035 2013 SanDIsk Corporation. All rights reserved www.sandisk.com

Table of Contents 1. Executive Summary... 3 1.1 Audience...4 2. The Open Source Database Market Summary and Key Trends... 4 3. NVMFS Architectural Description and Technical Overview... 4 3.1 NVMFS and the MySQL Open Source Database...5 3.2 TRIM and Compression...6 4. Key Customer Use Cases and Major Challenges... 7 4.1 Improvements to Flash Endurance...7 4.3 Improvements to MySQL Transactional Latency...7 5. Test Bed and Workloads... 8 5.1 Linkbench...8 5.2 TPC-C...8 6. Test Results... 8 6.1 Atomic Writes...9 6.2 NVM Compression...10 6.3 Impact on Endurance (Increase in PCIe Life Expectancy)...11 7. Conclusion... 11 7.1 Special Corner Cases and Key Limitations...12 Appendix A Benchmark parameters... 12 Linkbench...12 Sysbench...12 TPC-C...13 2

1. Executive Summary The adoption of flash-based storage in the data center has provided customers with revolutionary application performance improvements, based on the ability of flash devices to process I/O functions much faster than traditional hard disk drives (HDDs). However, deploying flash devices as simply a faster HDD solution and using existing software infrastructure, misses a tremendous opportunity to improve application performance with new software interfaces. These interfaces take advantage of the byte-addressable, memory-like features of a flash storage device like SanDisk s Fusion iomemory PCIe Application Accelerators. Working with the Oracle MySQL, Percona and MariaDB communities, SanDisk has created an optimized solution that focuses on solving specific performance problems in the database. This solution uses a new flash-aware file system called NVMFS (Non-Volatile Memory File System), which is uniquely capable of translating standard Portable Operating System Interface (POSIX)-compliant file interfaces into flash-aware calls to the underlying device. With NVMFS, open source databases like Oracle MySQL, Percona Server and MariaDB can become flash-aware and solve many daunting performance problems. While the performance of open source databases has improved in many areas over many releases, there are two problem areas in MySQL that can significantly benefit from the use of these flashaware interfaces: Oracle MySQL, Percona Server and MariaDB write all table space data twice to ensure data integrity during system failure, which leads to twice the amount of actual data per write. This double-write problem exists for both spinning media and flash. These databases have supported compression for a long time, but the implementations have been shown to perform poorly on flash. Generally, most customers do not run compression with production workloads on flash, due to severe latency and throughput penalties. This white paper shows how SanDisk NVMFS significantly improves I/O performance, provides consistently low latency, and reduces latency variation: The SanDisk NVMFS Atomic Writes feature solves the double-write problem. That results in a significantly increased life expectancy of devices at similar throughputs, while providing consistently lower I/O latency. The Atomic Writes feature uses flash-aware interfaces to write an entire database commit in one operation. Enhancements to compression enable customers to get the benefit of compression with little performance impact. This NVMFS compression provides up to a 50% increase in capacity 1 for applications, while driving a transaction-per-second rate that is within 10% of the uncompressed rate. Combining Atomic Writes with NVMFS compression reduces write operations by close to 75% 1, compared to running uncompressed without these features. This significantly enhances the life expectancy of flash devices and reduces latency. This white paper outlines customer use cases, test criteria, and benchmark results for running these open source databases on SanDisk NVMFS. The Conclusion section describes key customer benefits and special cases. 1 For workloads that compress well. Improvements will vary depending on the load. 3

1.1 Audience This white paper is intended for database administrators, storage administrators, enterprise database application architects, and decision makers tasked with evaluating, acquiring, managing, operating, or deploying Oracle MySQL, Percona Server or MariaDB in the data center. 2. The Open Source Database Market Summary and Key Trends MySQL is a development community that develops a popular open source version of a relational database management system (RDBMS). Today, many large enterprises are using this RDBMS for e-commerce, online transaction processing (OLTP), and embedded databases across vertical industries such as education, financial services, healthcare, media technology, telecom, retail, e-commerce, and social networking. These corporations include Google, Facebook, Amazon, Paypal, Linkedin, Verizon, Rackspace, and the Chicago Mercantile Exchange. Some of the key highlights for the MySQL community include: Leading open source database for web applications Leading open source database as a service (DBaaS) in the cloud DBaaS market is gaining momentum. Amazon RDS offers an Oracle MySQL RDBMS engine. Rackspace Cloud Databases offer fully managed instances of MariaDB, Oracle MySQL and Percona Server, with container-based virtualization. Integrated with Hadoop in Big Data platforms. 3. NVMFS Architectural Description and Technical Overview NVMFS is the first POSIX-compliant file system implemented natively on an enterprise flash memory system, providing file system convenience with near raw-block-device performance. Using native access APIs provided by the VSLTM software, NVMFS provides a general-purpose, POSIX-compliant file system name space on top of Fusion iomemory PCIe Application Accelerators, within the Linux operating system environment. NVMFS leverages the virtual storage layer (VSL) software, which is a fully-associative, log-structured, non-volatile flash management layer running on the host. Inherently, as a side-effect of its native flash management, the VSL software provides much of the functionality that traditional file systems require, including: Block allocation and mapping Transactional consistency (such as logging or journaling) Physical data layout strategies 4

Additionally, the lower random-access penalty of flash means there is less need for optimal data structure and access patterns. For example, it is less important to implement complex data structures, reservation schemes, read-ahead, and other such heuristics to reduce the occurrence of random accesses. By leveraging the performance characteristics of the Fusion iomemory PCIe Appliation Accelerators, and using the capabilities native to the VSL software, NVMFS can avoid implementing complex block allocation or mapping layers, and it does not require a journaling layer. This leads to dramatically reduced code complexity, the potential for reduced CPU and memory utilization via simpler data structures, and improved performance via reduced I/O. Additionally, NVMFS communicates data usage information back to the VSL software, which can be used to optimize flash management performance, such as improving garbage collection. This communication can additionally improve performance via reduced CPU and memory usage, as well as lower I/O consumption. The following block diagram describes how NVMFS leverages functionalities provided by the underlying flash translation layer (VSL software). Fig. 1. NVMFS interaction with VSL software 3.1 NVMFS and the MySQL Open Source Databases Oracle MySQL, Percona Server and MariaDB are the first major applications targeted for NVMFS acceleration. NVMFS exports several primitives from the underlying flash translation layer (VSL software), and these are accessed directly by MySQL to accelerate workloads and extend the endurance of the flash device for real customer workloads. 5

Atomic Writes are transactional I/O primitives, allowing NVMFS and the VSL software to guarantee complete ACID (Atomicity, Consistency, Isolation, Durability) writes to the storage device. Oracle MySQL, Percona Server and MariaDB have updated their database designs to take advantage of the NVMFS atomics primitives. For customers deploying these databases, this means no doublebuffering is required during database writes. Traditionally, these databases write all table space data twice to ensure data integrity during system failure, which leads to twice the amount of actual data per write. Legacy storage systems such as spinning media do not suffer from limited write cycles, but for flash this can have a negative impact on both endurance and performance. The more writes the flash translation layer is handling, the more garbage collection will take place subsequently. The difference between Atomic Writes and traditional MySQL writes is shown below. Fig. 2. Contrasting traditional writes and Atomic Writes 3.2 TRIM and Compression Native persistent TRIM is another primitive used by NVMFS to accelerate certain workloads. Persistent TRIM, developed by SanDisk, allows NVMFS to delete multiple logical block ranges atomically and deterministically, in a single I/O operation. Certain MySQL distributions use this feature to implement NVM compression. Compared to legacy MySQL compression, known in the MySQL community as ROW compression, NVM compression not only compresses data much more efficiently using pluggable compression algorithms, but also does so with minimal performance loss. As with Atomic Writes, compression reduces the number of writes to the underlying flash translation layer, and thus can meet or even surpass the throughput achieved with uncompressed workloads over time. 6

4. Key Customer Use Cases and Major Challenges NVMFS provides multiple benefits to real customer production workloads. SanDisk has worked with the top vendors to implement native support for NVMFS running on Fusion iomemory devices. By doing this, SanDisk enables customers to gain the direct benefits of native flash characteristics exported by NVMFS. 4.1 Improvements to Flash Endurance Many customers are concerned about device endurance, especially as newer NAND technologies are sustaining fewer writes over their lifetime, because of higher density. Oracle MySQL, Percona Server and MariaDB with Atomic Writes enabled, running on Fusion iomemory PCIe Application Accelerators, can double the expected endurance of the flash device at similar throughput. This is because the total number of table space writes is cut in half. As with many applications, the write path is usually the most complex in any application. By writing only half the amount of data compared to other file systems on flash, these databases achieve higher scalability and performance on modern hardware. Atomic Writes show the biggest performance increase for workloads that are larger than the available amount of DRAM. Because Atomic Writes reduce the cost of writes, environments with high amounts of memory pressure can have their database tier more quickly between in-dram and on-flash storage. Transactional latency is also one of the major areas where Atomic Writes add a significant amount of value. OLTP workloads must persist small amounts of data as quickly as possible. By using Atomic Writes, the database can write data more quickly and thus lower latency, even for fixed-throughput workloads. This also means customers can run more transactions per system within the same latency envelope 4.2 Improvements to Device Cost per Gigabyte Customers have long hoped to use compression with flash-based storage devices as a way to reduce cost by storing more data. Oracle MySQL, Percona Server, and MariaDB have supported compression for a long time, but the implementations have been shown to perform very poorly on flash. Generally, most customers do not run compression with production workloads on flash due to severe latency and throughput penalties. Working with the MySQL community, SanDisk has helped to implement a new version of compression called NVM compression. This technology allows the database to use the persistent TRIM interface of NVMFS to mark logical block addresses as being unallocated after compression. Compared to legacy compression, NVM compression is much simpler, faster, and more versatile for real production workloads. Using pluggable compression algorithms, NVM compression can also achieve higher compression ratios at much better performance. Compared to not using compression, NVM compression can cut the actual cost per gigabyte in half for workloads that compress well. 4.3 Improvements to MySQL Transactional Latency Combining Atomic Writes with NVM compression allows Oracle MySQL, Percona Server, and MariaDB to write four times less data, compared to running uncompressed without Atomic Writes. This has a major impact on how much garbage collection takes place during high-write workloads. 7

Less garbage collection results in more consistent transactional latency, as the Fusion iomemory PCIe Application Accelerator has less work to perform in the background. 5. Test Bed and Workloads Three applications were used to show the benefits of the NVMFS file system in combination with MySQL. All tests were performed using the MariaDB 10.0.15 database, a drop-in replacement for Oracle MySQL that is considered fully compatible. The following benchmarks were used. 5.1 Linkbench The Linkbench benchmark was developed by Facebook to simulate the workload characteristics of their social graph. All Linkbench workloads used a 10x configuration, resulting in approximately 110GB of data. 5.2 TPC-C The TPC-C benchmark is a reimplementation of the classic TPC-C workload and is made available from Percona. It is not the official implementation by TPC, but instead an implementation done for MySQL using the same set of transactions. TPC-C is a write-heavy workload, often used to show transactional performance of MySQL in a high-write environment. All TPC-C workloads were configured for 1,000 warehouses, resulting in approximately 100GB of data. 5.3 Sysbench Sysbench is a modular benchmark used to test all areas of the system. For MariaDB, specific OLTP workloads and custom transaction rates can be used to measure both maximum throughput and sustained latency during fixed workloads. Two servers were used for all testing: the HP DL380 Gen8 and Supermicro X10DRU-i+. Sysbench TPC-C Linkbench Server HP DL380 Gen8 SuperMicro X10DRU-i+ CPU 16 x E5-2690 2.90 GHz with HT 28 x E5-2690 v3 with HT (56 cores total) DRAM 128GB 256GB Storage Fusion iomemory SX300-1600 Fusion iomemory SX300-1600 VSL software 4.1.1 4.1.1 Software Centos 6.5 MariaDB 10.0.15 Centos 6.5 MariaDB 10.0.15 6. Test Results To obtain the most comprehensive data set, multiple tests were run using multiple benchmarks and with different data-to-dram ratios. The I/O characteristics within the MariaDB storage engine will change significantly, not only based on the workload but also on how much of the main data fits into the MariaDB buffer pool. This was simulated by using 10GB and 50GB of DRAM for various MariaDB runs. Customers often see their workloads increase past the amount of available DRAM, and thus 8

testing workloads with a large dataset is important. 6.1 Atomic Writes The following chart shows the effects of using Atomic Writes with the NVMFS file system when running Linkbench. For this test, the size of the active data set is twice as big as the amount of DRAM available to MariaDB. As this causes MariaDB to tier aggressively between on-disk and in-dram storage, Atomic Writes significantly improves maximum throughput. Fig. 3. NVMFS atomics vs. XFS and EXT4 (Linkbench : 110GB data 50GB MySQL Buffer pool, MariaDB 10.0.15) The benefits of Atomic Writes can also be observed for fixed-throughput workloads. This is often very representative of real customer workloads, as maximum throughput is rarely needed or seen. The following chart shows Sysbench injecting a constant amount of OLTP transactions every second and calculating the 99% latency. 9

Fig. 4. XFS vs. NVMFS atomics (Sysbench - MariaDB 10.0.15, 4000 OLTP TXN injection/second, 99% latency, 220GB data 10GB buffer pool) As NVMFS with Atomic Writes is writing less data than XFS, the 99% latency is significantly lower and more consistent during this one-hour run. 6.2 NVM Compression As TPC-C is very write-heavy, the efficiency of the compression path within MariaDB is important for high-throughput workloads. The following chart shows how NVMFS compression achieves performance very close to the speed of running uncompressed. Fig. 5. NVMFS compression on MariaDB workload (TPC-C-like benchmark: 1000 warehouses 75GB MySQL Buffer pool, MariaDB 10.0.15) For workloads where the amount of reads vs. writes is more balanced, NVM compression also adds value by compressing better and faster. The chart below shows Linkbench using 110GB of data with a 50GB MariaDB buffer pool. 10

Fig. 6 NVMFS compression vs. EXT4 cases (Linkbench, 110GB data 50GB MySQL Buffer pool, MariaDB 10.0.15) 6.3 Impact on Endurance (Increase in PCIe Life Expectancy) An interesting characteristic of compressing data stored on flash is the reduced amount of writes submitted to the storage device. Together with atomics, compression can reduce the writes by 4x, resulting in less garbage collection and more consistent latencies. The chart below shows the physical bytes written during a Linkbench workload, with uncompressed, atomics-only, and atomics combined with compression. Fig. 7. NVMFS atomics/compression comparisons (Linkbench: 110GB data 50GB MySQL Buffer Pool, MariaDB 10.0.15, Physical bytes written) 7. Conclusion To obtain the most comprehensive data set, multiple tests were run using multiple benchmarks and The NVMFS file system was designed to allow applications to take advantage of the full potential of flash. Common Linux file systems such as XFS and EXT4 were mainly developed for HDDs and thus deploy complexity that is not needed on flash. NVMFS allows applications to take advantage of atomic I/O and persistent trim commands exposed by the underlying flash translation layer of Fusion iomemory PCIe Application Accelerators. As these technologies are implemented in the database stack, higher throughput, lower latency, increased flash endurance, and more efficient compression are observed. 11

With NVMFS, customers can store more data on their flash storage and serve workloads faster, with more consistent response times. 7.1 Special Corner Cases and Key Limitations When NVMFS is used to accelerate workloads, almost every production workload can improve in latency, throughput, data size, or device endurance. However, some workloads will be accelerated more than others. NVM compression relies on multiple cores to efficiently compress data in parallel. Thus, workloads running on smaller (less than 8-core) systems may see reduced performance. Larger systems (16+ cores) may see improved performance, especially for workloads where most of the data fits into DRAM. Atomic Writes shows the biggest benefit to performance (latency and throughput) when the data set is smaller than available DRAM. Combining Atomic Writes with NVM compression will ensure good performance for most workloads, even if the data set fits entirely in DRAM. While NVM compression can be used without Atomic Writes, it is not advised, as standard writes will serialize the data access to the storage device. Appendix A Benchmark parameters Below are the command lines used for all three benchmarks. Note that two systems were used for testing, but all comparisons on a single chart were done on the same system, with the exact same specifications. Linkbench To load the data, the following command was used:./bin/linkbench -D dbid=linkdb -D host=localhost -D sock=/tmp/mysql.sock -D user=root -D port=3306 -D password= -D maxid1=100000001 -c config/myconfig.properties l To run the benchmark itself, the following command was used:./bin/linkbench -D requesters=64 -D dbid=linkdb -D host=127.0.0.1 -D user=root -D port=3306 -D password= -D maxid1=100000001 -c config/myconfig.properties -csvstats final-stats.csv -csvstream streaming-stats.csv -D requests=500000000 -D maxtime=28800 Sysbench Sysbench v0.5 was used for this test. Loading data was done using the following command:./sysbench --test=tests/db/parallel_prepare.lua --oltp-tables-count=32 --numthreads=16 --oltp-table-size=30000000 run For transaction injection, the following command was used: 12

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=32 --oltp-tablesize=30000000 --report-interval=1 --percentile=99 --tx-rate=4000 --num-threads=32 --max-time=3600 --max-requests=0 run TPC-C As part of the tpcc-mysql benchmark by Percona, the load.sh script was used to populate the tables using the following command:./load.sh tpcc1000 1000 For the benchmark itself, the following command was used:./tpcc_start -h localhost -P 3306 -d tpcc1000 -u root -p -w 1000 -c 32 -r 30 -l 3600 13

Products, samples and prototypes are subject to update and change for technological and manufacturing purposes. SanDisk Corporation general policy does not recommend the use of its products in life support applications wherein a failure or malfunction of the product may directly threaten life or injury. Without limitation to the foregoing, SanDisk shall not be liable for any loss, injury or damage caused by use of its products in any of the following applications: Special applications such as military related equipment, nuclear reactor control, and aerospace Control devices for automotive vehicles, train, ship and traffic equipment Safety system for disaster prevention and crime prevention Medical-related equipment including medical measurement device Accordingly, in any use of SanDisk products in life support systems or other applications where failure could cause damage, injury or loss of life, the products should only be incorporated in systems designed with appropriate redundancy, fault tolerant or back-up features. Per SanDisk Terms and Conditions of Sale, the user of SanDisk products in life support or other such applications assumes all risk of such use and agrees to indemnify, defend and hold harmless SanDisk Corporation and its affiliates against all damages. Security safeguards, by their nature, are capable of circumvention. SanDisk cannot, and does not, guarantee that data will not be accessed by unauthorized persons, and SanDisk disclaims any warranties to that effect to the fullest extent permitted by law. This document and related material is for information use only and is subject to change without prior notice. SanDisk Corporation assumes no responsibility for any errors that may appear in this document or related material, nor for any damages or claims resulting from the furnishing, performance or use of this document or related material. SanDisk Corporation explicitly disclaims any express and implied warranties and indemnities of any kind that may or could be associated with this document and related material, and any user of this document or related material agrees to such disclaimer as a precondition to receipt and usage hereof. EACH USER OF THIS DOCUMENT EXPRESSLY WAIVES ALL GUARANTIES AND WARRANTIES OF ANY KIND ASSOCIATED WITH THIS DOCUMENT AND/OR RELATED MATERIALS, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR INFRINGEMENT, TOGETHER WITH ANY LIABILITY OF SANDISK CORPORATION AND ITS AFFILIATES UNDER ANY CONTRACT, NEGLIGENCE, STRICT LIABILITY OR OTHER LEGAL OR EQUITABLE THEORY FOR LOSS OF USE, REVENUE, OR PROFIT OR OTHER INCIDENTAL, PUNITIVE, INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES, INCLUDING WITHOUT LIMITATION PHYSICAL INJURY OR DEATH, PROPERTY DAMAGE, LOST DATA, OR COSTS OF PROCUREMENT OF SUBSTITUTE GOODS, TECHNOLOGY OR SERVICES. No part of this document may be reproduced, transmitted, transcribed, stored in a retrievable manner or translated into any language or computer language, in any form or by any means, electronic, mechanical, magnetic, optical, chemical, manual or otherwise, without the prior written consent of an officer of SanDisk Corporation. All parts of the SanDisk documentation are protected by copyright law and all rights are reserved. 2015 SanDisk Corporation. All rights reserved. SanDisk is a trademark of SanDisk Corporation, registered in the United States and other Countries. Fusion iomemory is a trademark of SanDisk Enterprise IP LLC. Other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s). NVMFS_WP 03.20.2015. 14