How To Test The Performance Of An Ass 9.4 And Sas 7.4 On A Test On A Powerpoint Powerpoint 9.2 (Powerpoint) On A Microsoft Powerpoint 8.4 (Powerprobe) (

Transcription

1 White Paper Revolution R Enterprise: Faster Than SAS Benchmarking Results by Thomas W. Dinsmore and Derek McCrae Norton

2 In analytics, speed matters. How much? We asked the director of analytics from a leading U.S. marketing services provider, a Revolution Analytics customer. Her team supports more than 1,000 predictive models currently in production; her clients expect the team to build a predictive model in 30 minutes or less. Previously, we shared results 1 from a performance test comparing Revolution R Enterprise (RRE) to SAS. 2 That test showed how our unique Parallel External Memory Algorithms (PEMA) technology produces vastly better performance for advanced analytics. Some readers noted that SAS and RRE were not tested running on the same hardware, because the test was limited to a single task. It was also pointed out that SAS offers software to enable deployment in clustered computing environments, similar to what was used for RRE. We listened and we set up a new test using the same hardware for both products. To help ensure a fair comparison, we hired an experienced SAS consultant to review the SAS programs, enable them for Grid computing and run the test. We used SAS 9.4 and defined a list of commonly used analytics tasks for the test. The results: g ScaleR ran the analysis tasks 42 times faster than SAS. g ScaleR outperformed SAS on every task. g The ScaleR advantage ranged from 10 to 300 times the performance. g The ScaleR advantage increased when we tested on larger data sets. g The new SAS HP PROCs, where available, only marginally improved SAS performance. In this white paper, we ll report on the approach we used for the test, together with detailed results. Approach For this test, Revolution Analytics engaged a consulting firm experienced with SAS Grid Manager. 3 The consultant set up a clustered computing environment consisting of five four-core machines running CentOS, all networked using Gigabit Ethernet connections and a separate NFS Server. The team deployed SAS Release 9.4, with the following major components: g Base SAS g SAS/STAT g SAS Grid Manager We used a desktop running SAS Management Console and SAS Enterprise Guide as the Grid Client. To test Revolution R Enterprise, we first deployed IBM Platform LSF and Platform MPI Release 9 on the grid, then installed Revolution R Enterprise Release 7 on each node. SAS Grid Manager uses an OEM version of IBM Platform LSF that cannot run concurrently with the standard version from IBM used by Revolution R Enterprise, so we ran the tests sequentially and re-configured the environment for each test SAS is a registered trademark of the SAS Institute, Inc. 3 The consulting firm prefers to remain anonymous. 2

3 To simplify test replication across different environments, we used data manufactured through a random process. Time needed to manufacture the data is not included in the benchmark results. Prior to running the actual tests, we loaded the randomized data into each software product s native file system: for SAS, an SAS data set; for Revolution R Enterprise, an XDF file. Although we have benchmarked Revolution R Enterprise on data sets as large as a billion rows, typical data sets used by even the largest enterprises for the statistical procedures investigated tend to be much smaller. We chose to perform the tests on wide files of 591 columns and row counts ranging from 100,000 to 5 million file sizes that represent what we consider to be typical for many analysts. We also ran scoring tests on narrow files of 21 columns with row counts ranging up to 50 million. For the tests, we defined a sequence of ten frequently used analysis tasks, plus one scoring task. The table below shows the tasks together with the actual SAS 9.4 and Revolution R Enterprise 7 (RRE 7) functions used in the benchmark programs. Table 1: Benchmark Tasks Task RRE 7 Capability SAS 9.4 Capability Descriptive statistics (n, min, max, mean, std) on 1 numeric variable rxsummary PROC SURVEYMEANS Median and deciles for 1 numeric variable rxquantile PROC SURVEYMEANS Frequency distribution for 1 text variable rxcube PROC FREQ Linear regression with 1 numeric response variable and 20 numeric predictors Linear regression with 1 numeric response variable and 20 mixed predictors Stepwise linear regression with 100 numeric predictors Logistic regression with 1 binary response variable and 20 numeric predictors Generalized linear model with numeric response variable, 20 numeric predictors, gamma distribution and link function rxlinmod rxlinmod rxlinmod rxlogit rxglm PROC REG PROC HPREG PROC GENMOD PROC REG PROC LOGISTIC PROC GENMOD k-means clustering with 20 active variables rxkmeans PROC FASTCLUS k-means clustering with 100 active variables rxkmeans PROC FASTCLUS Using the first linear model, score a file with 10x the number of records in the analysis file rxpredict PROC SCORE 3

4 We performed all benchmark tests sequentially, with no other operations running concurrently. The actual SAS 9.4 and RRE 7 programs used for this test are freely available to anyone at GitHub: We invite readers to test these scripts in any environment and compare your results to those we have published below. Results The table below shows results for the larger data set of five million records. Using SAS 9.4, the complete script takes on average 4 5,192 seconds (about 1.5 hours) to complete in the benchmark environment. The same tasks performed in Revolution R Enterprise 7 take 124 seconds (about two minutes) to complete. Table 2 shows the performance of SAS 9.4 and RRE 7 for each of the 10 components of the script. Table 2: Benchmark Results n = 5,000,000 Runtime (Seconds) RRE 7 Speed Task RRE 7 SAS 9.4 Multiple Descriptive statistics X Median and deciles X Frequency distribution X Linear regression with 20 numeric predictors X Linear regression with 20 mixed predictors X Stepwise linear regression, 100 numeric predictors X Logistic regression with 20 numeric predictors X Generalized linear model, 20 numeric predictors X k-means clustering, 20 active variables , X k-means clustering, 100 active variables , X Total, all tasks , X 4 Results averaged over multiple runs; no significant variation across runs. 4

5 RRE s performance advantage increases as the number of records analyzed increases, as shown below: Table 3: Results by Size of Analysis Data Set Total, All Tasks Runtime (Seconds) RRE 7 Speed Analysis File Size RRE 7 SAS 9.4 Multiple n = 1,000, X n = 5,000, , X The scoring test uses the predictive model produced in the first linear regression run and an independent table with 10 times as many rows as the analysis data set. The table below shows the results from this test: Table 4: Results for Scoring Scoring Task Runtime (Seconds) RRE 7 Speed Scoring File Size RRE 7 SAS 9.4 Multiple n = 10,000, X n = 50,000, X With SAS 9.4, SAS bundles HP PROCs designed for use with SAS High Performance Analytics Server. We substituted PROC HPREG for PROC REG in one of the tests. In the benchmark environment, use of this High Performance Analytics PROC does not materially improve SAS performance. 5 Table 5: Results for PROC HPREG Linear Regression Analysis File Size PROC REG Runtime (Seconds) PROC HPREG RRE 7 n = 5,000, According to SAS documentation, HP PROCs run in Single Machine mode unless the customer licenses SAS High Performance Analytics Server. 5

6 Discussion Speed matters. In the time it takes to run our benchmark script in SAS 9.4 under our test conditions, a user can perform the same tasks 42 times in RRE 7. Based on the outcomes of this test, in practice, an analyst using RRE 7 should be able to build more models, build better models by reducing the learning cycle, serve more customers and produce more revenue. Why does RRE 7 run faster than SAS 9.4? Revolution Analytics uses unique technology called Parallel External Memory Algorithms (PEMA) to distribute operations over multiple machines in a clustered architecture. When a data set is larger than memory on a single machine, Revolution R Enterprise 7 streams the data across all of the available computing resources. By contrast, SAS/STAT software swaps data between memory and disk when a data set is larger than memory, a process that is much slower than in-memory operations. When these SAS programs run in a grid configuration even when fully enabled for Grid operations most SAS PROCs do not take advantage of the available computing resources. According to SAS, only four 6 of the PROCs in SAS/STAT are able to take advantage of multiple computing threads. As we demonstrated by testing the HPREG PROC, the SAS HP PROCs do not improve performance unless the customer also licenses SAS High Performance Analytics Server. Revolution Analytics takes performance and efficiency seriously, and we continuously improve the efficiency and speed of our analytics engine. We invite everyone, customers and competitors alike, to run our benchmarks and share your results with us

7 About Revolution Analytics, Inc. As the leading commercial provider of software and services based on the open source R project for statistical computing and headquartered in Mountain View, California Revolution Analytics brings Big Data scalability, performance and cross-platform enterprise readiness to R, the world s most widely used statistics software. The company s flagship Revolution R Enterprise software is designed to meet the production needs of leading organizations in data-driven industries, including finance, retail, manufacturing and digital media. Used by over 2 million data scientists in academia and at cuttingedge organizations such as Google, Lloyd s of London and the U.S. Food and Drug Administration, R is the standard of innovation in statistical analysis. Revolution Analytics is committed to supporting the continued growth of the R community by sponsoring R user groups and conferences worldwide, and the company offers free licenses for Revolution R Enterprise to everyone in academia Revolution Analytics, Inc. All rights reserved. Printed in the United States of America. This white paper is for informational purposes only. REVOLUTION ANALYTICS MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Revolution R Enterprise, RRE, Revolution Analytics, the Revolution Analytics and Revolution R Enterprise logos, and Big Data, Big Analytics Platform are trademarks or registered trademarks of Revolution Analytics, Inc., in the United States and/or other countries. Other product names mentioned herein may be trademarks of their respective companies.