Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers

Transcription

1 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Table of contents Executive summary... 2 Introduction... 2 Test topology... 3 System design test bed... 3 Cluster scale test bed... 4 Test methodology... 5 System design test methodology... 5 Cluster scale test methodology... 5 Test results... 6 System design results... 6 Cluster scale results... 8 Test analysis summary Recommendations For more information... 17

2 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Executive summary This white paper covers two parts of evaluating the impact of system design on Hadoop clusters, specifically in regards to performance for various Hadoop workloads. The first part covers three different worker node designs configured in small clusters and compares the results against performance, cost, capacity and power consumption metrics. The second part covers the evaluation of cluster scaling to multiple racks, as well as the impact of the network fabric and intermediate data storage on the performance results. Comparing the performance results of the three different worker node system designs clearly showed an increase in performance when utilizing higher performance components instead of utilizing the standard Hadoop deployment of midrange CPUs with Large Form Factor (LFF) disk drives. The performance increase was very dependent on the type of workload, and also needed to be evaluated against the increase in cost and changes in storage capacity. The end results indicated that for very CPU intensive workloads, such as machine learning, the additional performance can outweigh the cost increase, but for more generic MapReduce workloads, the cost increase of higher end components may outweigh the performance increase and will need to be considered carefully in the deployment context. Overall the Hadoop cluster scaled very well for most workloads, with some workloads demonstrating better scale behavior than others. In addition the results indicate that for very large clusters the scaling curve should be extrapolated at rack level rather than extrapolating node scaling within the first rack. In addition it was found that midrange CPUs in the worker nodes could not fully capitalize on faster network fabrics and utilizing solid state storage for intermediate data and results, and that higher performance CPUs may be needed to gain the most benefit out of these technologies. Target audience: This document is intended for system and solution architects, as well as technical decision makers. An intermediate knowledge of Apache Hadoop and scale out infrastructure is recommended. Purpose: This white paper presents the results of performance testing on Hadoop clusters and provides comparison metrics for cost, capacity and power consumption. This white paper describes testing performed in March to October 2013 timeframe. Introduction Hadoop is an evolving technology gaining widespread adoption that is becoming a very important element in most data management environments. Due to the large scale and architecture of Hadoop, as well as the typical cost pressures, the design of the worker nodes should be considered very carefully during the decision cycle. There are also large variances in Hadoop workloads with different worker node resource utilization profiles, which can impact the node design. The purpose of this paper is to better understand the impact of worker node design choices on various Hadoop workloads comparing design choices against performance, capacity, cost and power utilization metrics. In addition to the node design, further work was done to better understand how Hadoop clusters scale with various workloads, as well as the impact of changing some elements of the cluster such as network fabric and intermediate data storage. The information in this paper can serve as data points for Hadoop cluster design to gain a better understanding for the tradeoffs involved in design choices of the worker nodes for different workload environments; this paper also provides a better understanding of the scaling profile for cluster growth projections. Important This test can provide a benchmark for comparing hardware and/or software products; it is not intended to be used as a sizing guideline. In the real world, server performance is highly dependent upon the application design and workload profiling. As with any laboratory testing, the performance metrics quoted in this paper are idealized. In a production environment, these metrics may be impacted by a variety of factors. As a matter of best practice for all deployments, HP recommends implementing a proof-of-concept using a test environment that matches as closely as possible the planned production environment. In this way, appropriate performance and scalability characterizations can be obtained. For help with a proof-of-concept, contact an HP Services representative (hp.com/large/contact/enterprise/index.html) or your HP partner. 2

3 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Test topology System design test bed To evaluate the data node design, small Hadoop clusters were built, each with a single head node and three worker nodes. The head node contained the NameNode and JobTracker and was connected directly to the same network fabric as the worker nodes. Figure 1 shows the architectural diagram of the base test bed. Figure 1. Architectural diagram of the base test bed Three different worker node designs were utilized to perform the testing. The detailed hardware description for each cluster design is provided in Table 1. All the clusters were tested with an Apache based Hadoop distribution with MapReduce version 1 processing framework. Table 1. Detailed hardware description of clusters Base System Mid Tier System High End System Worker Nodes HP ProLiant DL380e Gen8 2 CPU Intel Xeon E (12 cores 2.4GHz) 48GB RAM 12 x 2TB SATA LFF HDD(7.2k RPM) 2 x SATA OS Drives 4 x 1GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 4 x 1GbE Bonded Network ports Network HP 5830AF-48G 48 x 1GbE Ports Ultra-deep packet buffers Worker Nodes HP ProLiant DL380p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 14 x 1TB SAS SFF HDD(7.2k RPM) 2 x SAS OS Drives 2 x P420 Storage Controllers 4 x 1GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 4 x 1GbE Bonded Network ports Network HP 5830AF-48G 48 x 1GbE Ports Ultra-deep packet buffers Worker Nodes HP ProLiant DL380p Gen8 2 CPU Intel E (16 cores 2.9GHz) 128GB RAM 14 x 900GB SAS SFF HDD(10k RPM) 2 x SAS OS Drives 2 x P420 Storage Controllers 2 x 10GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 4 x 1GbE Bonded Network ports Network HP 5900AF-48XG-4QSFP+ 48 x 10GbE Ports 4 x 40GbE Ports 3

4 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Cluster scale test bed To evaluate the cluster scaling, a 2-rack cluster was configured with 36 worker nodes in total. Two identical head nodes were configured, the first as the NameNode and the second as the JobTracker. Figure 2 shows the architectural diagram of the cluster test bed. The base cluster configuration was tested first, then the network fabric was upgraded repeating the test suite and lastly the intermediate data storage was also upgraded. The detailed hardware description for each cluster design is provided below in Table 2. All tests were performed with an Apache-based Hadoop distribution with MapReduce version 1 processing framework. Figure 2. Cluster test bed Table 2. Detailed hardware description of clusters Base Cluster Network Upgrade Network and Storage Upgrade Worker Nodes HP ProLiant DL380e Gen8 2 CPU Intel E (12 cores 2.4GHz) 48GB RAM 12 x 2TB SATA LFF HDD(7.2k RPM) 2 x SATA OS Drives 4 x 1GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 4 x 1GbE Bonded Network ports Worker Nodes HP ProLiant DL380e Gen8 2 CPU Intel E (12 cores 2.4GHz) 48GB RAM 12 x 2TB SATA LFF HDD(7.2k RPM) 2 x SATA OS Drives 2 x 10GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 2 x 10GbE Bonded Network ports Worker Nodes HP ProLiant DL380e Gen8 2 CPU Intel E (12 cores 2.4GHz) 48GB RAM 10 x 2TB SATA LFF HDD(7.2k RPM) 2 x 800GB SSD LFF (Intermediate Data) 2 x SATA OS Drives 2 x 10GbE Bonded Network ports Head Node HP ProLiant DL360p Gen8 2 CPU Intel E (12 cores 2.9GHz) 64GB RAM 8 x 900GB SAS SFF HDD 2 x 10GbE Bonded Network ports 4

5 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Table 2. Detailed hardware description of clusters (continued) Base Cluster Network Upgrade Network and Storage Upgrade Network Top of Rack 2 x HP 5830AF-48G HP Intelligent Resilient Framework (IRF) Backbone 2 x HP 5900AF-48XG-4QSFP+ IRF Network Top of Rack 2 x HP 5900AF-48XG-4QSFP+ IRF Backbone 2 x HP 5900AF-48XG-4QSFP+ IRF Network Top of Rack 2 x HP 5900AF-48XG-4QSFP+ IRF Backbone 2 x HP 5900AF-48XG-4QSFP+ IRF Test methodology A number of different Hadoop workloads were used to analyze the cluster performance to get a broader understanding of how different cluster designs will perform. The following Hadoop workloads were used for all phases of the testing: TeraSort Standard Hadoop workload providing good utilization of all system resources including CPU, memory, disk and network I/O Intel HiBench workloads Web-Search Nutch Indexing Machine Learning K-means clustering Data Analytics Hive HiveQL queries on data warehouse type data using complex queries with multiple joins and grouping/sorting operations System design test methodology To evaluate the impact of system design, the three small clusters with three worker nodes each were configured as described in the test topology section of this white paper. The workloads listed above were executed on the clusters in parallel to reduce overall test time. For each workload, a small initial data set size was used to determine execution time on the cluster, then the data set size was increased for each workload until the execution time for the workload was at least 30 minutes, but not exceeding three hours. The system resource utilization was also monitored to ensure that the systems were appropriately configured at the hardware, OS and Hadoop layers; and that the workloads consumed substantial system resources to make the comparisons valid. The same data set size was then used in testing for all the clusters for each workload. Total execution time of the workload was used as the indicator of cluster performance. Since Hadoop is an effective parallel execution framework and all the workloads parallelized very well, it was not deemed necessary to execute multiple workloads in parallel on a single cluster to gain insight when comparing the performance of the system design choices of the worker nodes. In addition to capturing the performance of the clusters, the power consumption of each design was measured and also the pricing and storage capacity was taken into consideration. Cluster scale test methodology To evaluate the scale profile of the cluster design, the base cluster was configured as described in the test topology section with both racks and all 36 worker nodes. The scale testing was performed in three steps, with the first step activating nine of the worker nodes in the cluster, the next step activating 18 worker nodes in the cluster and the final step with all 36 worker nodes active in the cluster. For each scale step all workloads were tested with multiple data set sizes; the larger worker node clusters were tested with the same data set size as the smaller worker node cluster, but additional larger data set sizes were also tested on the larger clusters to evaluate any influence the data set size may have on the workload and cluster size. After completion of scale testing on the base cluster, the network fabric was upgraded with different top of rack network switches, network cards used in the worker and head nodes, as well as adding additional network links from the top of rack network switches to the backbone switches. 5

6 Execution Time (s) Execution Time (s) Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers The same scale steps and workload data set size testing was then performed on the upgraded cluster, by starting with the cluster with nine active worker nodes and increasing the active worker node size to the full two racks with 36 worker nodes. The final phase was to upgrade the storage on the cluster. The upgraded network components were kept in place. To upgrade the disk storage, two of the SATA data drives were removed and replaced with SSD drives. The cluster was then reconfigured to use 10 SATA drives for data only and use the two SSD drives for all the intermediate results/data; typically worker nodes are configured to use all data drives for both data and intermediate results/data. The aim of this test was to understand the impact of separating the more sequential Hadoop data access from the more intermittent intermediate result/data access, as well as to evaluate if there was any impact on the scaling of the cluster. Total execution time of the workload was used as an indicator of cluster performance and only a single workload was executed on the cluster at one time. Test results System design results For the system design tests, three worker node configurations were evaluated with four different workloads; only the results of the most relevant data set sizes for each workload were used for comparison purposes and included in this section. Figures 3, 4, and 5 show the execution time for each workload and comparing each worker node design. As the results are expressed in total workload execution time, a lower number indicates better performance. Figure 3. TeraSort execution time in seconds with 2-TB data set 12,000 10,000 8,000 6,000 4,000 2,000 - Base Mid Tier High End Figure 4. Nutch Indexing and K-means clustering execution time in seconds 4,000 3,500 3,000 2,500 2,000 1,500 Nutch K-means 1, Base Mid Tier High End 6

7 Execution Time (s) Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Figure 5. Data analytics Hive query execution time in seconds 9,000 8,000 7,000 6,000 5,000 4,000 3,000 2,000 Query 1 Query 2 1,000 - Base Mid Tier High End The results were then normalized for all the workloads to be able to better visualize how much the worker node design impacted performance, utilizing the base worker node design as the baseline. As shown in Figure 6, the results were inverted from workload execution time making a higher number better and helping to visualize how much the performance improved over the baseline. Figure 6. Relative performance for all workloads Higher is better Base Mid Tier High End TeraSort K-means Nutch Hive (Q1) Hive (Q2) 7

8 Worker Node Power (W) Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers The power consumed by each worker node design was also measured. The measurements were taken for a single worker node and measured at idle with no workload being executed, and also during workload execution to capture the average and peak power consumption as shown in Figure 7. Figure 7. Worker node power consumption in watts Idle Average under Load Peak Base Mid Tier High End The cost comparison between the systems design were done by utilizing the list pricing of the worker node at the time of testing. The pricing was normalized to the base system design to make visual comparison easier as shown in Figure 8. In addition to the total cost of the worker node, the storage or usable disk capacity cost was also compared. This was based on $ per TB of usable disk space on the worker node, and again normalized to the base system design for easier comparison. This helped to better represent and understand the cost impact of the system design enhancements. Figure 8. Worker node cost comparisons Lower is better Node Cost Disk Capacity Cost Base Mid Tier High End Cluster scale results For the scale testing, the results were calculated on a per worker node throughput number to more effectively visualize the actual performance and scale of the cluster. The throughput number per worker node is either based on data volume or, for some workloads, on data sample size. For all figures shown in this section, a higher per node throughput number demonstrates better performance. As there were a large number of data points with the use of multiple cluster scale points and different workloads with multiple data set sizes per workload, as well as three different cluster configurations, only key data points were used for comparison purposes to evaluate scale and comparing changes to the cluster configuration. The base cluster scale results are represented in Figures 9, 10, 11 and 12. 8

9 MB/s K-Samples/s MB/s Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Figure 9. Base cluster TeraSort scale results Throughput per worker node TeraSort 1TB TeraSort 4TB TeraSort 10TB 2 Figure 10. Base cluster K-means scale results Throughput per worker node K-Means 2B 4 2 Figure 11. Base cluster Nutch Indexing scale results Throughput per worker node Nutch 60GB

10 MB/s MB/s Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Figure 12. Base cluster Hive Query scale results Throughput per worker node Hive Q21-500GB The network fabric and storage was then upgraded on the base cluster. The scale results are shown in Figures 13, 14, 15 and 16. Figure 13. Network and storage upgraded cluster TeraSort scale results Throughput per worker node TeraSort 1TB TeraSort 4TB TeraSort 10TB 2 10

11 MB/s MB/s K-Samples/s Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Figure 14. Network and storage upgraded cluster K-means scale results Throughput per worker node K-Means 2B K-Means 4B 4 2 Figure 15. Network and storage upgraded cluster Nutch Indexing scale results Throughput per worker node Nutch 60GB Nutch 120GB 5.00 Figure 16. Network and storage upgraded cluster Hive Query scale results Throughput per worker node Hive Q21-500GB Hive Q21-1TB 11

12 K-Samples/s MB/s Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers To compare the impact of the upgrades to the base cluster the results were graphed next to each other for select workloads and data points, as shown in Figures 17 and 18. Figure 17. Base and upgraded cluster TeraSort scale results Throughput per worker node Base 10GbE + SSD 2 Figure 18. Base and upgraded clusters K-means scale results Throughput per worker node Base 10GbE 10GbE + SSD 2 12

13 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers In order to interpret the impact of upgrading the cluster components some key system metrics were also graphed for the TeraSort and K-means workloads. Also observed, but not graphed, was that worker node memory was fully consumed for most workloads and tests. Figure 19. Upgraded cluster TeraSort CPU utilization %usr %sys %iowait 0 Figure 20. Upgraded cluster K-means CPU utilization %usr %sys %iowait 0 13

14 TPS TPS Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers The number of disk I/O transfer data was also captured to understand how the SSD drives were utilized for intermediate results/data. In Figures 21 and 22, the SSD drives were identified by OS devices sdl and sdm. Figure 21. Upgraded cluster TeraSort disk transfers sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm Figure 22. Upgraded cluster K-means disk transfers sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm 14

15 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers Test analysis summary As clearly shown in Figure 6 of the test results, the performance of Hadoop clusters can be substantially improved by using upgraded components for the worker nodes from the typical midrange components commonly utilized for Hadoop today. The performance improvement varies by workload and the difference between the base system design and the high end system design ranges from 40% to over 100%. Workloads that are processing and memory-intensive, such as machine learning and search indexing, demonstrated the biggest improvements. The improved performance came with an increase in power consumption and higher system acquisition cost. While the idle power consumption of the all the systems were very similar, the higher end systems consumed more power while under load with up to 50% more power consumed by the high end worker nodes compared to the base worker nodes shown in Figure 7. The worker node cost also increased substantially with an increase of over 2x between the high end worker nodes and the base system nodes. In addition, the disk or storage cost per capacity increased at an even higher rate, with the high end design costing over four times more per TB of usable storage compared to the base design. (Shown in Figure 8.) Note that the disk drives and storage accounts for more than 50% of the cost of all the worker node systems, and that both the mid-tier and high end designs utilized Small Form Factor (SFF) drives, which are significantly more expensive than the LFF drives used on the base system. By utilizing LFF drives on the mid-tier and high end designs the node acquisition cost and cost per storage capacity per node can be substantially reduced. As expected, the Hadoop cluster scaled well for most workloads. TeraSort and K-means scaled well with most data set sizes, while Nutch and Hive required increased data set sizes to demonstrate effective scaling with the large cluster configurations. Also observed was a tendency for almost all workloads to have a slight drop in scaling once a multi-rack cluster was utilized. Upgrades to the base cluster for the networking fabric to 10GbE and also adding SSD drives for intermediate data storage did yield an improvement in performance, but one that was not very significant for most workloads as shown in Figures 17 and 18. Upon further analysis, it was clear that the cluster worker nodes were mostly CPU-bound and the upgrades to the storage and networking could not be capitalized upon as shown in Figures 19 and 20. It is also important to note that the base 1GbE fabric (with multiple bonded links per worker node) utilized ultra-deep packet buffer switches, which minimizes packet drops due to network incast in the Hadoop cluster that commonly affect workload performance. Analyzing the disk transfers on the cluster utilizing SSD drives for intermediate data revealed that a very large number of disk transfers were being handled by the SSD drives, and also that the large slower LFF data drives had far fewer disk transfers than in typical configurations where both the main data and intermediate results are being stored on the same drives. This is a good indication of improving the access pattern on the main data disks, which is ideal for Hadoop workloads where large volumes of data are normally read and written to disk in large block sizes. Recommendations The results of the different system designs for the worker nodes demonstrated that the performance benefit of upgrading the worker node design yields a varying benefit pending the specific workload. For some workloads, the increase in performance meets or can exceed the increase in cost of the system. It is therefore very important to understand the workload profile for which a Hadoop cluster will be deployed to maximize the benefit of upgrading system design components as even a small sample of workload profiles clearly presented the variance it can have on cluster performance. The storage component of the worker node makes up more than half the total cost of the system, and it is an area where the capacity and performance requirement should be carefully assessed for the deployment environment. For the workloads tested, the CPU and memory upgrades yielded substantial performance improvement for a number of the tested workloads. It is still important to make sure the overall system design is balanced for the intended Hadoop workloads. For environments where power and space is a key decision criteria, and the typical workloads are processing-intensive, a higher end system design can provide positive returns. Overall, the Hadoop cluster scaled very well for most workloads, but for very large scale deployments with over 100 worker nodes in multiple racks, it is advised that cluster scale is projected on a per-rack basis with multi-rack testing rather than simply measuring initial scale on a very small cluster in a single rack. As indicated by the test data, not all workloads scale equally well and some require larger data sets to more effectively utilize the resources available in a large cluster. 15

16 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers To gain the full benefit from upgrading the networking and storage components on the base cluster, a corresponding upgrade in the CPU power, and potentially memory, are required. Most of the workloads were CPU bound and did not yield the full benefit from these upgrades. By utilizing solid state storage for intermediate data, the disk I/O pattern on the main data drives was substantially improved for many workloads. While the data in the tests performed did not reflect true impact, it does hold promise as the cost of solid state storage drops and the capacity of main data drives keep increasing with a corresponding increase in processing power. As Hadoop 2.0 and YARN resource management becomes more prevalent, more efficient utilization of system resources are expected. In future updates, processing models will be deployed in the Hadoop framework to complement MapReduce with a corresponding increase in utilization of CPU and memory resources. 16

17 Technical white paper Design and performance report for Hadoop on Intel Xeon-based HP ProLiant DL380 servers For more information Hadoop on HP, hp.com/go/hadoop HP ProLiant servers, hp.com/go/proliant HP Networking, hp.com/go/networking To help us improve our documents, please provide feedback at hp.com/solutions/feedback. Sign up for updates hp.com/go/getupdated Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. 4AA5-1266ENW, February 2014