Accomplish Optimal I/O Performance on SAS 9.3 with Intel Cache Acceleration Software and Intel DC S3700 Solid State Drive ABSTRACT Ying-ping (Marie) Zhang, Jeff Curry, Frank Roxas, Benjamin Donie Intel Corporation, Chandler, Arizona Because of the large volumes of data processed, the dramatically increased CPU speed as well as the availability of multiple CPUs on a system, the I/O throughput of the SAS applications could easily become a system bottleneck and a key factor in determining SAS performance. Intel Cache Acceleration Software (CAS) complements the Intel Solid State Disk (SSD) data center family of products for a comprehensive caching solution and prioritizes application performance with unique, policy-based cache control that accelerates applications without backend storage or application changes. It can potentially provide a high performance and low cost solution for SAS like Business Intelligent customers. The goal of this case study is to explore how to achieve optimal performance on SAS 9.3 with Intel CAS and Intel DC S3700 SSDs. In the study, experiments have been designed with different storage configurations, like pure Hard Drive Disks (HDDs), Intel CAS, and pure SSDs, based on the characterization of file systems in SAS Mixed Analytics Workload (MAWD: add D t differentiate from SAS MAW). The results of the study indicate that SAS performance has significantly improved with Intel CAS and Intel DC S3700 SSDs and reduced cost ($) per job. For example, while using the performance of pure HDDs configuration as baseline, creating an Intel CAS drive with a single 100 GB Intel DC S3700 SSD for the input file system, the performance (jobs per hour) of 40 user MAWD has increased 36%. It resulted in the cost ($) per job reduction by 20%. The data collected from each test case has been compared and analyzed in the paper for better understanding: How did Intel CAS significantly improve the performance of SAS applications? What is the Best Known Method (BKM) to accomplish optimal I/O performance with Intel CAS and Intel DC S3700 SSDs on SAS 9.3? INTRODUCTION In general, SAS application performance is being held back by storage I/O speeds which prevent the multi-core, multi-cpu servers from reaching their full performance potential. Based on case studies, this paper is going to provide guidance to SAS customers about how to accomplish optimal I/O performance with Intel CAS and Intel SSD technology, instead of adding more storage, servers, or memory which adds to infrastructure expenses, as well as complexity of management. The case study covers SAS Mixed Analytics Workload (MAWD) characterization, performance measurement with different storage configurations and data analysis. The major components used include: Intel Xeon E5-2600 server, Red Hat* Enterprise Linux 6.3, Intel Cache Acceleration Software (CAS) Linux version and Intel DC S3700 SSD SAS 9.3 applications SAS MAWD.
Intel Cache Acceleration Software Overview Intel CAS combined with Intel SSD data center family is an industry unique solution that delivers application specific performance on physical servers and virtual machines via Selective Optimized Caching [1]. Intel CAS accelerates Linux* applications by caching active (hot) data to a local flash device inside servers, such as SSD and memory. Intel CAS implements caching at the server level; utilizing local high-performance flash media as the cache drive media inside the application server as close as possible to the CPU, thus reducing storage latency as much as possible. The Intel CAS installs into the Linux* operating system itself, as a kernel module. The nature of the integration provides a cache solution that is transparent to users and applications, and your existing storage infrastructure. No storage migration effort or application changes are required. As shown in Figure 1-1, on an initial read, data is retrieved from backend storage and copied to the Intel CAS cache. A second read promotes data to system memory. Subsequent reads are returned at high-performance RAM or flash speed. All data is written synchronously to both the backend storage and the cache. When the cache is full, newly identified active data evicts stale data from the cache, utilizing the Intel CAS proprietary eviction algorithm. Figure1-1: Intel CAS Block Diagram Intel CAS for Linux*, by default, employs a block-based caching architecture that caches all of the activity on the selected device. Starting with release (Ver.2.1+) onwards, it will be possible to list a certain file or a list of multiple files to be cached (delimited by space) exclusively using the include_files command option or the include.conf config file. (See [1] for details) SAS Mixed Analytics Workload Overview Multi-user benchmarking scenarios created in SAS MAWD are used to simulate a typical SAS Foundation environment and utilize real-world data volumes and structures of a typical SAS customer. This benchmark consists of typical analytic jobs designed to replicate a light to heavy workload. These jobs were launched with a certain time delays from each other via a script to simulate scheduled jobs and interactive users launching at different times. Input data types of the benchmark are text, SAS dataset and SAS transport files. The workload utilized SAS procedures including: SAS DATA step, PROC RISK, PROC LOGISTIC, PROC GLM (general linear model), PROC REQ, PROC SQL, PROC MEANS, PROC SUMMARY, PROC FREQ and PROC SORT [2] Inside the benchmark, the tests are divided into two sub-sets [3]: COMP (Computational, CPU-focused): Heavily computational with very light I/O to fully utilize CPU power Row counts up to 187,000, under 10 GB total I/O data
MIXED: A mix of CPU and I/O-intensive jobs with a mix of short and longer running jobs Row counts up to 90 million, variable counts up to 297, and over 800 GB total I/O data In the study, we used the MIXED sub-set of the benchmark to stress I/O on the storage system. The data of the workload are clarified as input, output and saswork that are stored in three separate file systems. Intel Solid-State Drive DC S3700 Series Overview [5] Fast and Consistent Performance: Deliver data at a breakneck pace, with consistently low latencies and tight IOPS distribution 75K Random Read IOPS1 Latencies: Typical 50µs; Max <500µs2 Stress-Free Protection: Protect your data center applications with multiple secure checkpoints that provide protection against data loss and corruption. o Full data path and non-data path protection o Power safe write cache with built in self-test High-Endurance Technology: Meet your most demanding needs with marathon-like write endurance of 10 full drive writes per day over five years REFERENCE CONFIGURATION System Configuration System Configuration Intel SNB-EP White Box Platform OS Red Hat* Enterprise Linux 6.3 Application SAS 9.3 Workload SAS MAWD 4.0 Processor 4 x Intel Xeon E5 2.9 GHz processors Total Memory 256 GB (16 x 16GB DDR3) HBA 2 x LSI SAS MegaRAID SAS 9285-8e Storage 2 x LSI 620J enclosures HD Disk 12 x 480 GB HDDs SD Disk Intel DC s3700 100GB/800GB SSDs Table 2-1: System Configuration OS and Storage Configuration OS Tuning on Red Hat* Enterprise Linux [4] Use the tuned tool in RHEL6.3 with the enterprise-storage profile Use a read ahead size of 16,384 KB Use an I/O request queue size of 1,024
Storage Configuration and Tuning [4] Create three hardware RAID 0 partitions on each enclosure, 6 disks per enclosure Group all of the RAID 0 partitions into one volume group with 256K stripe size Divide the volume group into three ext4-formatted three file systems: input, output, and SAS work FILE SYSTEM CHARACTERIZATION The concept of Intel CAS technology is to cache hot data with high speed SSD or RAM to speed up I/O access. It is important to understand the I/O pattern between the system and storage, such as how big the hot data size is, how many reads/writes (R/W) will operate, the R/W is random or sequential, and how big IOPS or throughput are required to avoid I/O bottleneck. It is important to determine: Will Intel CAS or SSD provide performance gain? What is the right capacity of SSD / RAM to select for best performance per dollar? In this study, we have used Intel RAID SSD Cache Sizing & Performance Predictor Tool [6] and Intel CAS status report to characterize the file systems of SAS MAWD with different number of users. The characterization results of the three file systems of 40 user SAS MAWD: input, output and saswork are reported in Table 3-1. Type of File System Type of the data Type of I/O Operation Ratio of Reads/Writes Size of I/O Operation Size of Data Size of Hot Data input Generated SAS dataset to be analyzed Sequential 100%/0% 94% 256K read ~ 520 GB ~ 120 GB output Analytical results Sequential 0%/100% 99% 4K Writes ~ 480 GB < 1 GB saswork Temporary stage ground for analytical work Sequential / Random Mixed 40%/60% 35% 256K read 62% 4k write 440 GB ~ 200 GB Table 3-1: File system Characterization of 40 User SAS MAWD Furthermore, Figure 3-1 indicates that the Input file system is sequential read intensive, output file system is sequential write intensive and saswork file system is read and write intensive with mixed sequential and random. Figure 3-1: I/O Pattern of 40 user MAWD
Based on the I/O pattern of the file systems, Intel CAS has potential to improve the performance of the Input file system with minimum cost increase, because it has relatively small size of hot data which has been accessed frequently. It is obvious that Intel SSD can bring best performance for saswork file system because of its excellent performance on random read and writes. However, for output file system, if the budget is limited, using HDDs for low cost and high capacity is reasonable. EXPERIMENT DESIGN According to the file system characterization, experiments have been designed to evaluate the performance with different storage configurations for 40 user SAS MAWD (Figure 4-1). 1) Pure HDD: Use 12 JBOD Hard Drive Disks configured as a LVM which is partitioned into three file systems 2) Intel CAS for Input file system: Use one 100 GB S3700 SSD to cache the hot data for input file system, and keep output and saswork file systems as HDD configuration unchanged. 3) Intel CAS for Input file system and Intel 100 GB S3700 SSD for SAS Work: Use one 100 GB S3700 SSD to cache the hot data for input file system, use one 800 GB S3700 SSD for saswork file system and keep output file system HDD configuration unchanged 4) Pure SSD: Use 7 x 800 GB Intel 100 GB S3700 SSDs for the three file systems: 3 for input, 2 for output and 2 for saswork. Figure 4-1: Different Storage Configuration for the Case Study EXPERIMENTAL RESULTS Table 5-1 reports the performance results in which pure HDD configuration is used as the baseline. Type of drives Pure HDD Intel CAS + HDD Total # of HDDs 12 x 480 GB HDDs 12 x 480 GB HDDs Total # of SSDs Configuration 0 12 HDDs, 1 volume group, 3 logical volumes 1 x 100 GB SSDs 12 HDDs for output and saswork, 1x100GB SSD for input 40 User Real Time (Sec) Jobs per Hour Performance Difference 196,520 2.49 1.00 144,957 3.38 1.36
Intel ICAS+HDD+SSD 12 x 480 GB HDDs 1x100 GB SSD 1x800 GB SSD Intel SSD 0 7 x 800GB SSDs 12 HDDs for input and saswork, 1x100GB SSD for output 7 SSD, 1 volume group, 3 logical volumes Table 5-1: Performance of SAS MAWD with Different Storage Configurations 100,706 4.86 1.95 64,390 7.60 3.05 Figure 5-1 shows that by using a single Intel 100 GB DC S3700 SSD to cache hot data in the input file system, Intel CAS has improved the performance by 36% resulting in 20% reduction in cost ($) per job. On other hand, keeping Intel CAS for input file system and using one Intel 800 GB DC S3700 SSD for saswork file system, the performance improvement jumped to 95% with 40% reduction in cost ($) per job. Then, if using Intel 800 GB DC S3700 SSDs for all three file systems, the performance jumps up to 205% and the cost ($) per job reduced by 60%. Figure 5-1: Performance Comparison between difference Storage Configurations PERFORMANCE ANALYSIS The results are very impressive. Now, let us to take a deep dive on the collected data for better understanding: How did Intel CAS significantly improve the performance of SAS applications? What is the Best Known Method (BKM) to accomplish optimal I/O performance with Intel CAS and Intel DC S3700 SSD on SAS 9.3? 1. CPU Utilization First, check the CPU utilization of the baseline experiments shown in Figure 6-1 (40 users with pure HDDs). It demonstrates that most of the time the system is in idle state because the user CPU utilization and system CPU utilization are very low and I/O wait is relatively high. It is obvious that the system is I/O bound and wasted most of time waiting for I/O response. As such, improving I/O performance is critical to reduce the I/O wait time.
1 23 45 67 89 111 133 155 177 199 221 243 265 287 309 331 353 375 CPU Utilization (%) < Accomplish Optimal I/O Performance on SAS 9.3 with Intel CAS and SSD >, Continued CPU Utilization User System Idle IOWait 100 50 0 Time (10 second intervals) Figure 6-1: CPU utilization of 40 user SAS MAWD with pure HDDs 2. Comparison of I/O Wait We will now compare I/O wait times of the tests to see how the different storage configurations changed the I/O performance. Figure 6-2 shows that with Intel CAS for the input file system configuration the I/O wait time has reduced from 26% (for pure HDDs) to 15%. With pure SSD configuration, except about 5% I/O wait time at the beginning, the I/O Wait was down to zero for most of the test. It implies that Intel CAS and Intel SSD technologies can significantly improve the performance of SAS 9.3 by reducing I/O wait by allowing the system to spend more time executing the jobs. Figure 6-2: Comparison of I/O Wait time between Different Storage Configurations 3. Accomplish Optimal I/O Performance on SAS 9.3 with Intel CAS and Intel SSD First, characterize the existing SAS file system or storage systems with block tracing tools, such as Intel RAID SSD Cache Sizing & Performance Predictor Tool to determine: a) How big is the size of hot data? b) What is the I/O pattern of the file systems? Then, collect data on the system to check:
a) What is the expectation of performance improvement on the system? b) What is the budget for the storage upgrade? c) Is it possible to rebuild a new storage system and restore data from the previous one? If the budget allows pure SSD configuration then it is an ideal solution which provides the highest performance because of the high I/O throughput and low I/O Wait. Otherwise, Intel CAS for input like data and Intel SSD for saswork like data are good choices. Current available Intel CAS Linux version is focused on read cache for read only data. Therefore, it is not applicable for write only data or mixed read and write data. However, in the upcoming Intel CAS Linux version, new features such as write back and pre-fetch will be added, In addition, the I/O performance of different type of SSDs can make a difference. For example, the R/W throughput of Intel latest 910 series SSD is up to 3.6 times - 2.8 times over the Intel DC S3700 SSD used in the tests for this paper. It is important to first assess the I/O pattern of the file to assess if there is a problem that can benefit from the use of Intel CAS From the experiments, the results show that once the capacity of the SSD is big enough to hold the entire hot data, increasing the capacity of the SSD has no performance impact unless caching the entire data set. As such, optimum number of SSDs for Intel CAS depends on the size of the hot data. CONCLUSION This paper shows the performance results of SAS MAWD on SAS 9.3 with four different storage configurations. A pure HDD configuration was used as the baseline. As compared to the baseline, use of Intel CAS for read only input file system improved I/O performance by 36%. Using SSD for saswork file system resulted in an improvement of 95% over the baseline. The best performance came from the use of SSD for all the file system to the tune of 205%. The choice that SAS customers make about Intel CAS and SSD is a balance between available budget and expected performance improvement. The data presented in this paper demonstrates the improvement in performance of SAS 9.3 applications by the use of Intel CAS and Intel DC S3700 SSD. It is important for SAS customers to assess the input files to identify the performance problem and to determine if Intel CAS can be a solution. The case study confirmed that Intel CAS and Intel DC S3700 SSD significantly improves performance of SAS9.3 applications by caching hot data or storing entire data into high speed SSD storage system with reduced I/O Wait. ACKNOWLEDGMENTS We would like to recognize Ken Gahagan at SAS for his guidance on SAS performance evaluation. We recognize the team members at Intel, in particular, Arakere Ramesh and Martin Dimitrov for technical review of this paper. We would also like to thank John Skerchock from Intel and Carl Strickland an intern at Intel. REFERENCE [1] Intel Cache Acceleration Software --New and innovative caching solution http://www.intel.com/content/www/us/en/software/intel-cache-acceleration-software.html [2] SAS 9.2 on RHEL Reference Architecture http://people.redhat.com/dshaks/rh-sas-perf-ex-4-8-10.pdf [3] HP reference configuration for entry-level SAS Grid Manager solutions http://h20195.www2.hp.com/v2/getpdf.aspx%2f4aa1-2765enw.pdf [4] SAS 9.3 BI Case Study: Performance Scalability and Tuning on Servers with Intel Xeon Processor E7 Family, http://www.wuss.org/proceedings12/134.pdf [5] Intel Solid-State Drive DC S3700 Series http://www.intel.com/content/www/us/en/solid-state-drives/solid-statedrives-dc-s3700-series.html
[6] RAID Solutions http://intelraid.com/tech.php CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Ying-ping (Marie) Zhang Intel Cooperation MS CH7-401 5000 W. Chandler Blvd. Chandler, AZ 85226 480-554-9145 (O) Email: ying.m.zhang@intel.com Jeff Curry Intel Cooperation MS CH7-201 5000 W. Chandler Blvd. Chandler, AZ 85226 480-554-6366 (O) Email: jeff.curry@intel.com Roxas, Frank MS CH7-201 5000 W. Chandler Blvd. Chandler, AZ 85226 480-554-6763 (O) Email: frank.roxas@intel.com Donie, Benjamin J Intel Cooperation MS CH7-201 5000 W. Chandler Blvd. Chandler, AZ 85226 480-552-1906 (O) Email: benjamin.j.donie@intel.com SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. NOTICES INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations, and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license. Intel, Xeon, and the Intel logo are trademarks of Intel Corporation in the US and/or other countries. Copyright 2012 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others Optimization Notice Intel s compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804