Table of Contents. June 2010

Transcription

1 June 2010 From: StatSoft Analytics White Papers To: Internal release Re: Performance comparison of STATISTICA Version 9 on multi-core 64-bit machines with current 64-bit releases of SAS (Version 9.2) and PASW (formerly SPSS) Statistics Version 18; basic data management, basic statistics, and aggregation operations Note:. Table of Contents Abstract... 2 Overview... 2 Ability to Take Advantage of the Most Recent High-End Hardware... 2 Different Performance Priorities across Platforms... 2 Multithreading in SAS... 3 Multithreading in PASW (SPSS)... 3 Choosing Algorithms and Use Cases to Compare... 3 Data File Sizes, Buffering Mechanisms (Best Achieved Performance), and Repeatability... 4 Methods... 4 Hardware... 4 Software... 4 Data... 5 Data Management Operations, Data Analysis Operations... 5 Results... 6 Conclusion... 7

2 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 2 of 7 Abstract The purpose of this document is to provide a snapshot of key comparisons regarding the speed and efficiency of STATISTICA Version 9 data management and basic statistical computations, with reference to the performance of key competitors in the market place, namely SAS Version 9.2 and PASW (formerly SPSS) Statistics Version 18. This "snapshot" was taken in May 2010, after the most recent releases of 64-bit versions of SAS and PASW became available for Microsoft Windows based 64-bit multi-core computers (and servers). This comparison does not compare the speed of advanced analytic computations or data mining algorithms, because those comparisons are in practice almost impossible to prepare in a methodologically clean and meaningful manner: They inevitably will be confounded by the selection of specific (and incompatible between the three programs) options and implementation details, default statistics and results that are reported, and so on. However, generally speaking, the results of the comparisons regarding advanced analytics and data mining procedures mirror those reported here for basic statistics (descriptives, correlations, and data management operations). In order to measure the effectiveness of the basic implementation of multithreading into the STATISTICA Version 9 architecture, the most common and basic data management and basic statistics were chosen for the comparisons, where the performance capabilities (and limitations) of basic data reading and processing operations can be compared unambiguously. In summary, in the majority of test cases, STATISTICA Version 9 significantly exceeds the performance of the competing computing platforms. Overview The comparison of the performance of statistical analysis platforms is an inherently difficult and somewhat ambiguous task. Specifically, the following issues and challenges have to be considered. Ability to Take Advantage of the Most Recent High-End Hardware As of this writing (June 2010) multi-core 64-bit laptop, desktop, and Windows server computers are commonly available for the fraction of the price of analytics software that can take advantage of this hardware. In fact, on the Windows operating system, STATISTICA, PASW (formerly SPSS), and SAS only recently became available for 64-bit computing platforms, and multithreading capabilities (to take advantage of multiple processor cores, to simultaneously work on the same computational task) have been introduced to different specific operations only in the most current releases of these software packages. Therefore, in order to construct fair comparisons, the most recent versions of the competing packages for 64-bit platforms were installed on identical hardware for this comparison. Different Performance Priorities across Platforms The goal of the next generation of computational optimizations and dynamic multithreading support in STATISTICA Version 9 was to further improve the speed of the specific operations where customers had told us that performance was critical. Therefore, emphasis was placed on: 1. Data management operations (data reading, sub-setting, etc.)

3 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 3 of 7 2. Data aggregation (basic statistical summaries) and other basic statistics 3. Key data mining and predictive modeling algorithms commonly applied to large data sets (Classification and Regression Trees, Stochastic Gradient Boosting (Boosted Trees), Random Forests (Tree Nets), Generalized Linear models, etc.) Multithreading in SAS According to the most recent documentation, multithreading is not implemented in most of the predictive modeling algorithms in SAS (with the exception of linear models). Sorting, basic data reading operations, and basic statistical summaries (e.g., Proc Means), however, will take advantage of multiple processors (multithreading). Multithreading in PASW (SPSS), Availability for 64-Bit Platforms According to the most recent documentation, multithreading is not implemented in most predictive modeling algorithms, except Regression, CSLogistic, and CSCoxReg, and Distriminant. Also, a few other methods including Correlations and Factor have been re-written to take advantage of multiple processor cores. It appears (as of this writing), that the computation of basic aggregate statistics (means) and other data management operations are not multithreaded. PASW Modeler (SPSS Clementine) not available for Windows 64-bit platform. At the time of this review, the data mining solution from IBM (PASW Modeler, formerly known as SPSS Clementine, which is a different application than the SPSS core software) is not available for 64-bit Windows platforms. Choosing Algorithms and Use Cases to Compare Given these differences in priorities (i.e., what specific data management and computational algorithms have been upgraded in the SAS and IBM/SPSS software to take advantage of modern multi-core hardware), as well as the methodological concerns regarding how to achieve the greatest comparability of test cases (given the diverse options offered in the three applications) mentioned above in the Overview, it was decided to focus in this review only on the most common statistics, data management and data aggregation and analysis methods. For example, it is currently not possible to evaluate the effectiveness of STATISTICA Data Miner algorithms (e.g., of Boosted Trees to process multi-gigabyte input files) because these capabilities do not exist in other platforms. Also, specific features and implementation details of, for example, PASW (SPSS) CSLOGISTIC or CSSELECT (for probability sampling) or SAS Proc GenMod are sufficiently different from the respective implementations in STATISTICA Version 9 that it is difficult to compare "speed of computations." For example, to fit a Poisson regression model in SAS with predictor designs to large datasets can (a) be a bit faster, (b) take about the same amount of time, or (c) take substantially longer than in STATISTICA Generalized Linear Models, depending on the parameterization of the model and other options that are chosen; in PASW (SPSS) this analysis must be performed via the GENLIN procedure, which currently (as of this writing) is not multithreaded, so the comparison would be somewhat "unfair." Also, it is currently not possible to evaluate the effectiveness of many of the STATISTICA Data Miner core algorithms (e.g., of Boosted Trees to process multi-gigabyte input files) because these capabilities do not exist on other platforms.

4 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 4 of 7 Data File Sizes, Buffering Mechanisms (Best Achieved Performance), and Repeatability Various comparisons were performed using data sets of various sizes and dimensions (rows, columns). Preliminary tests confirmed that the speed differences between different platforms were mostly a function of the data file size, and not the numbers of rows or columns of the data file. Further, it is common (in STATISTICA and other platforms) that a substantial amount of processing time is spent simply reading the input data into the respective data analysis application (e.g., simply copying a file of the size of several gigabytes will take time that depends mostly on the performance of the hard disk). To minimize the effect of hard disk access times, which could mask real performance differences in the speed with which different computing platforms perform basic data management and analysis operations, multiple runs where performed in each case. The Windows Vista 64 operating system on the testing computer with 8 Gigabytes of RAM that was used in these tests performed buffering of the data, so that consecutive runs of the same analyses became faster as data was already buffered for reading (in memory) for more efficient access. STATISTICA 9 and SAS both use efficient mechanisms to leverage those buffering mechanisms available on Windows platforms to improve performance on consecutive runs with the same data. In practice, this is an important opportunity to improve the speed of basic data management operations for ETL applications with very large datasets, because many such operations do indeed access the same data multiple times in consecutive analysis and/or data transformation passes. Therefore, in all cases, the reported times in this White Paper compare the "best achieved performance" in consecutive runs, to gage the capabilities of the respective platforms for real-life data management and ETL operations. Note that the "worst-achieved performance" differences were sometimes smaller, because the bottleneck of the operations (e.g., to compute means and other moment statistics) was the speed with which the system could access data from the hard disk, while the respective computational algorithms were not used to full capacity, thus blurring the differences between data analysis platforms. Hardware Methods All tests were performed on a 2.2 GHz Quadcore Dell computer, running Windows Vista Professional for 64-bit platforms. The system had 8 GB of RAM. Software The comparisons were performed using the most recent releases of the respective software, at the time of this writing: 1. STATISTICA Version 9.1a, for 64-bit systems 2. SAS Version 9.2 for 64-Bit systems 3. PASW (SPSS) Statistics Version 18 (18.0.0) for 64-bit systems

5 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 5 of 7 Data All comparisons were performed with: 1. a dataset with 30 variables and 9,000,000 cases (approximately 2.2 Gigabytes), and 2. a dataset with 500 variables with 1,000,000 cases (approximately 4 Gigabytes). In STATISTICA, these files have a size of approximately 2.2 Gigabytes and 4 Gigabytes, respectively. These files were exported to text format, and then imported into the respective comparison analysis platforms (SAS 64 bit, PASW [SPSS]). The file sizes in those other platforms were comparable with the size of the STATISTICA data file. Data Management Operations, Data Analysis Operations The following 4 key tests were performed on all three data analysis platforms: 1. Computing basic descriptive statistics (means, standard deviations) for all variables 2. Computing correlation matrix using casewise (listwise) deletion of missing data, for all variables or a subset of variables 3. Sorting the file by two keys into descending order; Key 1 contains 50% 1's and 50% 2's, Key 2 contains uniform random numbers 4. Creating a subset of approximately 25% of the data using logical case selection conditions involving two variables For the reasons described in the Overview section, and also because the multithreading support (or even support for 64-bit platforms) is not uniformly implemented or available across the analysis platforms in this test, these basic tests were chosen to provide as accurate a picture as possible of the "real capabilities" with respect to basic data management and data analysis (aggregation) operations. However, it should be noted that the complete testing process involved many additional tests that yielded generally comparable results (e.g., for logistic regression, general and generalized linear models, etc. when compared to SAS which also offers a multithreaded implementation of this procedure) to those reported here.

6 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 6 of 7 Results In the following table, all results are reported as the number of seconds required to finish the respective task. Test/Task Notes Basic Descriptive Statistics 1 (30 variables, 9,000,000 cases) Basic Descriptive Statistics (500 variables, 1,000,000 cases) Correlation matrix, casewise (listwise) deletion of missing data 2, 30 variables, 9,000,000 cases STATISTICA ver. 9.1a 64 Bit SAS ver Bit PASW/SPSS ver Bit Correlation matrix, casewise (listwise) deletion of missing data, 200 variables, 1,000,000 cases Sorting 9,000,000 Cases, 30 Variables; 2 keys descending Subsetting 2.2 gig file; extracting approx. 25% of the data using a compound logical case selection condition In SAS, Proc Means will take advantage of multiple processors 2 In SPSS, Correlations will take advantage of multiple processors All tests performed after processing data at access to active data All tests performed after processing data at access to active data All tests performed after processing data at access to active data; verified that Threads=4 in PASW (SPSS) All tests performed after processing data at access to active data; verified that Threads=4 in PASW (SPSS) Includes making file ready for analysis, i.e., actually creating it. 3 Speed varied widely depending on the initial sort order of the input file, and whether the file is sorted into ascending or descending order

7 Performance Comparison of STATISTICA 9 (64-bit) vs. SAS, PASW (SPSS); 2010 Page 7 of 7 Conclusion With the exception of the sorting-of-a-large-dataset task (where STATISTICA's score is lower, most likely because its general sorting engine is optimized for more complex data sets), the performance of STATISTICA 9.1a exceeds significantly that of SAS and IBM PASW (SPSS) on all of the key data management and aggregation (summarization) tasks. In summary, STATISTICA Version 9.1a when implemented on a 64-bit multi-core platform offers significant performance gains over previous releases, and over the major competing software platforms SAS 9.2 and IBM PASW (SPSS) 18. To reiterate, this comparison did not include the various algorithms for predictive modeling (data mining) available in STATSITICA Data Miner, most of which are now also implemented via highly optimized, multithreaded algorithms that can take advantage of modern multi-core hardware (e.g., C&RT, Boosted Trees, Random Forests, Generalized Linear Models). Our tests with STATISTICA Data Miner to date have shown significant performance advantages: For example, using a real-world data set from a complex manufacturing application with 320 predictors and 278,000 cases (over 700 MB) can be analyzed with the Classification and Regression Trees procedure in just 83 seconds after an initial pass through the data; using the Stochastic Gradient Boosting procedure (Boosted Trees) with this file (building hundreds of trees), the predictive modeling analysis completes in under 6 minutes. The results reported here also do not reflect the substantial capabilities for parallel processing available on the STATISTICA server platform (WebSTATISTICA), which offers load balancing technology and takes advantage of not only multiple CPU's but also CPU's distributed across servers and clusters of computers. However, internal testing at StatSoft has documented the capabilities of the WebSTATISTICA server platform, for example as the analytics server platform hosting the scoring engine for the STATISTICA Live Score solution for credit and insurance applications. When combined with the performance for data management and data analysis operations documented here, STATISTICA Live Score provides substantial capabilities even when implemented on a single multi-core server and flexibly scales to clusters of servers (WebSTATISTICA manages load balancing across multiple servers). For more information, please contact StatSoft or visit