Tackling BigData: Strategies for Parallelizing and Porting Geoprocessing Algorithms to High-Performance Computational Environments

Transcription

1 Tackling BigData: Strategies for Parallelizing and Porting Geoprocessing Algorithms to High-Performance Computational Environments A. Sorokine 1, A. Myers 1, C. Liu 1, P. Coleman 1, E. Bright 1, A. Rose 1, P. Nugent 1, B. Bhaduri 1 1 OAK RIDGE NATIONAL LABORATORY Oak Ridge, Tennessee {sorokina,myersat,liuc,colemanpr,brightea,rosean,nugentpj,bhaduribl}@ornl.gov 1. Introduction In a world of increasing volume of earth observation and simulation data, the success of scientific advancement and discovery is strongly connected to our capabilities in storing, analyzing, and creating meaningful information from the enormous databases within a potentially useful timeframe. Although the progress of individual processor speed, cache performance, and graphics capability has been impressive, it has not been adequate to match the growth of available spatial data. Consequently, a number of new high-performance computing technologies have been developed and made available to GIScientists in the recent decade. New computational technologies have the potential to improve geoprocessing software performance and reduce computation time by orders of magnitude. Such improvements open new opportunities for GIScientists as the same algorithms and models can be used for larger datasets and applied at a larger scale or for larger number of geographic regions. Most new computational technologies rely on the decomposition of programming tasks into a number of smaller simultaneously executed units. However, most of the existing geoprocessing algorithms have been developed for single-threaded computing environments which in turn imposes rather low limits on the volumes of data that can be effectively processed. Conversion of the existing algorithms to parallel data-intensive architectures presents major challenges in terms of programming effort and testing. This paper describes the preliminary result of an ongoing study aiming to evaluate and analyze strategies for porting legacy geoprocessing algorithms to parallel high-performance and high data volume computational environments. 2. The Challenges of Parallel Geoprocessing and Earlier Work Parallel or distributed computing can be an effective solution to a number of spatial data integration, analysis, and visualization problems (Healy et al., 1998). Creating a geoprocessing application for parallel computational environments is a challenging problem. The complexity of the implementation of a parallel geoprocessing algorithm is significantly higher than development in the traditional serial software paradigm (McKenney, 2011). A number of both technical and non-technical issues specific to parallel processing have to be addressed. Plausible computing solutions include segmenting computational tasks across processors in a multiprocessor machine or across a cluster to alleviate the bottleneck associated with computational processes (Armstrong et al., 1994a, 1994b; Armstrong and Marciano, 1995; Rokos and Armstrong, 1995). This has been well demonstrated in the field of satellite remote sensing. Because of the raster nature of the data and ease of geometric partitioning (single program multiple data sets), several efforts have attempted to facilitate remote sensing algorithms using parallel and distributed computing (Hawick and James, 1997 and 1998; Yang, 2004). Parallel algorithm designers also should account for other issues such as synchronization among the threads accessing the same memory regions, sharing of I/O, and

2 the need to minimize interaction between nodes and threads (Hunt and John, 2011, Quinn, 2003). As a result, implementations of the same algorithm for parallel and serial environments are typically rather different as the peculiarities of parallel platforms have a significant impact upon the overall algorithm design. In addition, in data-intensive applications, input/output operations have to be given a special consideration (Bryant, 2007). In practice, the decision to pursue parallel development has to be based on a large number of factors, with expected improvements in computational efficiency being only one of them. The cost of developing or porting a geoprocessing application to a parallel environment is one of the most important factors that have to be balanced with expected improvement in computational efficiency. Other factors include the availability of the expertise for a specific computational platform, viability of a parallel computational platform for which development is performed in the long term, and the cost of acquiring hardware and maintaining it in the long term. The goal of this study is to demonstrate quantitative comparison of different strategies for porting an existing geoprocessing algorithm to parallel computing environments and to develop a methodology for balancing gains in computational efficiency with development effort. 3. Materials and Methods For the purposes of this study we have examined several re-implementations of a population disaggregation algorithm. This algorithm is used as one of the components of LandScan Global and LandScan USA, two high-resolution population distribution models (Bhaduri et. al., 2002, 2007, Dobson et. al., 2000, Sabesan et. al., 2007). LandScan data are used by tens of thousands of users throughout government, academic, and humanitarian organizations for consequence assessments. Applications using the LandScan data range from plume dispersion models, emergency evacuation planning and support, and natural hazard assessment. Users often require rapid updates of population distributions to account for rapidly changing population mobility such as refugee and other displaced population movements. The need for improving computational performance of the algorithm was initiated by the increased spatial and temporal resolution of the production database for LandScan and LandScan USA. The original version of the algorithm was developed in the Avenue programming language for ArcView 3.2 and has been used for LandScan releases since mid-1990s. The systematic study of the algorithm performance was devised after porting the algorithm to ArcGIS 9.x did not result in the expected performance improvements. Selection of the porting strategy was conditioned on a number of requirements. First, we expected significant improvement of the computational performance that would allow analysts to run the algorithm multiple times to test various combinations of the parameters and data sources. Second, the re-implemented algorithm should fit into the existing geoprocessing infrastructure but at the same time should be usable on highperformance systems without major redesign. Last but not least, the high-performance version of the algorithm should be viable and maintainable in the long term. The goals of this algorithm are to produce the initial LandScan population grid from the provided coefficient grid generated from the LandScan model and the census boundaries in grid format, and normalize the population within each boundary to the census values. First, the total population for each census area is divided by the sum of the coefficients within each of the respective census areas. Next, the coefficient grid is used to generate the initial population grid that will be normalized against the census population. The sum of the resulting grid within each census area is compared with the population values provided; the difference is stored for each zone. The raster cells are converted to points for the normalization process. From here, the population is either added to or subtracted from each point within a census zone until the population difference is zero. Population can be removed

3 from any point within a zone regardless of it coefficient, and the distribution of points are randomized to provide a better population distribution. However, the coefficient does limit the minimum value needed to add one person to a cell. The normalized population points are then converted back into ESRI GRID format and saved for review by an analyst (Bhaduri et. al., 2007). Multiple implementations of the algorithm were created with the purposes of understanding performance improvements that result from various software design decisions. The first reimplementation was done in C# in the ArcGIS 9.x framework. The advantage of using C# in ArcGIS is that it fits well into the ESRI products-based production framework. However, this implementation did not result in any significant performance advantages. Two other implementations were built using open-source tools and frameworks: a serial C++ implementation (compiled with GCC 4.6 and linked to standard library) and a multi-threaded program in Java 6 (IcedTea 1.10). Both implementations use gdal ( for reading and writing GIS formats. C++ is a language of choice for many of high-performance applications and has extensive support in term of libraries and development tools on all high-performance platforms. However, C++ programming generally is regarded as more difficult and requiring more programming skill. On the other hand, Java offers a shorter development cycle and easier availability of the programming expertise in the geoprocessing area. At the same time, some advantages of Java such as late binding and garbage collection come at the cost of performance penalties and lesser adoption and support of Java in high-performance applications. Additional versions of the implementations were created to test the performance of various data structures and memory management approaches. The C++ implementation was tested using either a hash array or indirect index for accessing a table of population counts in memory. For Java, two approaches were tested: storing the intermediate computation result in one array of objects, or storing the results in the arrays of primitive types. All versions of the program were verified to produce identical results with randomization disabled. 4. Preliminary Results and Discussion All implementations were benchmarked using representative sub-datasets of LandScan Global and LandScan USA: C# and Avenue versions were tested on a 39 million pixel subset of LandScan Global (~1 km spatial resolution), and C++ and Java versions were run with 1.3 billion pixel subset of LandScan USA (~90 m spatial resolution). On the Linux systems, execution time was measured using the time system command and rounded to the nearest minute. The Java and C++ versions of the program were executed multiple times to prefill the file cache. The systems were monitored for activity of other programs and services to exclude system-level factors affecting performance. Results of the performance tests are shown in Table 1 and mean execution times of the Java implementation are shown on Fig Time (sec) number of threads Figure 1. Mean execution time of the Java implementation on an 8-core PC.

4 Table 1. Benchmarking Results. Implementation Computing threads Operating System CPU, RAM size, disk Avenue for 1 Windows Intel Core GHz ArcView 3.2 XP 4GB RAM C# for ArcGIS 1 Windows dual 6-core AMD Opteron 9.3 Server GHz 8GB RAM C++ with hash 1 Ubuntu dual 4-core Xeon 2.4GHz, tables C++ with arrays 1 Ubuntu dual 4-core Xeon 2.4GHz, Java with arrays 1 Ubuntu dual 4-core Xeon 2.4GHz, of objects Java with arrays of primitives Java with arrays of primitives 1 Ubuntu Linux 12 Ubuntu Linux dual 4-core Xeon 2.4GHz, 32GB RAM, RAID5 dual 4-core Xeon 2.4GHz, 32GB RAM, RAID5 Dataset size 5,400x 7,200 5,400x 7,200 Execution Time 70 minutes 80 minutes minutes below 2 minutes did not finish because of memory limit 5-6 minutes 2-3 minutes Table 1 shows drastic performance improvements with the transition to implementation in C++ and Java. Performance of the programs written in C++ and Java are rather similar, however, it strongly depends upon the programming techniques used. C++ program execution time was reduced 7-fold by switching from using C++ hash maps to arrays. Two implementations of the algorithm in Java were tested. In the first implementation the data in RAM was organized as a single array of Java objects. However, this implementation had poor performance and excessive memory requirements due to overhead related to creation and destruction of Java objects. In the second Java implementation we relied on the arrays of primitive types and achieved performance that was much closer to that of C++. Further performance improvement was achieved by using multiple threads for computation executed on multiple CPU cores (Fig. 1). The computation was decomposed along the census area boundaries. 5. Conclusions and Current Work In this on-going study we have systematically evaluated strategies for improving performance of a geoprocessing algorithm. We were able to reduce computation time from hours or days to minutes allowing researchers to process much larger datasets and test a larger number of scenarios and model parameters. Such improvements are critically important for bringing GIScience research into the domain of BigData computations. The same algorithm was reimplemented several times using various software platforms and languages. Implementing the algorithm as a standalone program in C++ and Java resulted in the largest performance improvement. However, the performance difference between Java and C++ was not as big as expected. Overall performance of the program depends more on the programming techniques and data structures used; for example, the efficiency of hash table lookup functions in the C++ standard library or speed of allocation of various types of arrays in Java. Improvements from a parallel version of the algorithm were relatively small as the disk system performance was the major bottleneck. Current research is addressing the reimplementation of the same algorithm for high-performance distributed memory system and a version for mixed architecture with GPGPU is under consideration.

5 Acknowledgements This manuscript has been authored by employees of UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the U.S. Department of Energy. Accordingly, the United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The authors would like to acknowledge the financial support for this research from the US Government for the development of LandScan USA model and database. References Armstrong, M. P., Pavlik, C. E., & Marciano, R., 1994a. Parallel processing of spatial statistics. Computers & Geosciences, 20 (2), Armstrong, M. P., Pavlik, C. E., & Marciano, R., 1994b. Experiments in the measurement of spatial association using a parallel supercomputer. Geographical Systems, 1 (4), Armstrong, M. P. & Marciano, R., Massively parallel processing of spatial statistics. International Journal of Geographical Information Systems, 9 (2), Bhaduri, B., E. Bright, P. Coleman, and J. Dobson. LandScan: Locating People Is What Matters. Geoinformatics 5, no. 2 (2002): Bhaduri, B, E. Bright, P. Coleman, and M. Urban. LandScan USA: a High-resolution Geospatial and Temporal Modeling Approach for Population Distribution and Dynamics. GeoJournal 69, no. 1 (2007): Bryant, R. (2007). Data-Intensive Supercomputing: The Case for DISC. Computer Science Department. Retrieved from Dobson, J. E, E. A Bright, P. R Coleman, R. C Durfee, and B. A Worley. LandScan: a Global Population Database for Estimating Populations at Risk. Photogrammetric Engineering and Remote Sensing 66, no. 7 (2000): Hawick, K.A., James, H.A., Distributed high performance computing for remote sensing, Proceedings of Supercomputing 97 Pacific Rim. Hawick, K.A., James, H.A., A Web-based interface for on-demand processing of satellite imagery archives. Proceedings of the Australian Computer Science Conference (ACSC) 98, Perth, Australia. Healy, R., Dowers, S., Gittings, B., & Mineter, M., Parallel processing algorithms for GIS. Taylor & Francis, London. 460 pp. Hunt, C., & John, B. (2011). Java Performance (1st ed.). Prentice Hall. Rokos, D. & Armstrong, M.P., Using Linda to compute spatial autocorrelation in parallel. Computers & Geosciences, 22 (5), McKenney, Paul E. Is Parallel Programming Hard, And, If So, What Can You Do About It? Corvallis, OR, USA: kernel.org, Quinn, M. (2003). Parallel Programming in C with MPI and OpenMP (1st ed.). McGraw-Hill Science/Engineering/Math. Sabesan, A., K. Abercrombie, A. Ganguly, B. Bhaduri, Eddie Bright, and Phillip Coleman. Metrics for the Comparative Analysis of Geospatial Datasets with Applications to High-resolution Grid-based Population Data. GeoJournal 69, no. 1 (2007): Yang, X., Z. Chang, H. Zhou, X. Qu, and C.-J. Li. Services for Parallel Remote-Sensing Image Processing Based on Computational Grid. In Grid and Cooperative Computing - GCC 2004 Workshops, edited by H. Jin, Y. Pan, N. Xiao, and J. Sun, 3252: Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2004.