Geoprocessing on the Amazon cloud computing platform - AWS

Transcription

1 Geoprocessing on the Amazon cloud computing platform - AWS Yuanzheng Shao, Liping Di, Yuqi Bai Center for Spatial Information Science and Systems George Mason University Fairfax, VA USA {yshao3,ldi,ybai1}@gmu.edu Bingxuan Guo, Jianya Gong State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing Wuhan University, Wuhan, China mobilemap@gmail.com Abstract With the continuously increment of the available amount of spatial data sets, science, industry and administration require web-based geo-information concerning storage, availability and processing. The development of spatial data infrastructures (SDIs) bring about the Web-based sharing of large volumes of distributed geospatial data and computational resources. A powerful, dependable and flexible information infrastructure is required to process heterogeneous and distributed data into information and knowledge. The emergence of Cloud Computing technology brings a new computing Information Technology (IT) infrastructure to general users. With cloud computing platform, the users can requisition compute power, storage, and other services gaining access to a suite of elastic IT infrastructure services as demands. This paper presents an implementation of geoprocessing service that integrates Amazon cloud computing and geoprocessing functions to provide geoprocessing competence in a distributed web environment. The integration combines the geospatial processing functions with the flexible cloud computing platform to provide Web-based geoprocessing functionalities. Challenges, approaches and architecture for integrating are discussed. Amazon cloud computing platform is adopted to demonstrate the implementation. Keywords- Cloud Computing, Amazon AWS, Web Processing Service, Geoprocessing I. INTRODUCTION Geoprocessing functions are crucial to discover underlying and useful geospatial information and knowledge, and are widely used in Earth science modeling and applications. The fundamental purpose of geoprocessing is to allow the user to automate Geographical Information Science (GIS) tasks. Almost all uses of GIS involve the repetition of work, and this creates the need for methods to automate, document, and share multiple-step procedures. The geospatial processing functions in GIS have been developed over decades and can be used well in the desktop-based environment. The rapid development of the Web technology makes it possible to share and process large volumes of distributed geospatial data and computational resources through the Web. A powerful, dependable and flexible information infrastructure is required to process heterogeneous and distributed data into information. Cloud computing is rapidly emerging as a technology almost every industry that provides or consumes software, hardware, and infrastructure can leverage. The technology and architecture that cloud service and deployment models offer are a key area of research and development for GIS technology. From a provider perspective, the key aspect of the cloud is the ability to dynamically scale and provide computational resource in a cost efficient way via the internet. From a client perspective, the ability to access the cloud facilities on-demand without managing the underlying infrastructure and dealing with the related investments and maintenance costs is the key. Some Cloud Computing platforms are already available, including Amazon Web Service (AWS), Google App Engine, and Microsoft Azure. Powered by Cloud Computing platform, the users can requisition compute power, storage, database, and other services gaining access to a suite of elastic IT infrastructure services as demands. This paper presents an implementation of geoprocessing service that integrates Amazon cloud computing and geoprocessing functions to provide geoprocessing competence in a distributed web environment. The integration combines the geospatial processing functions with the flexible cloud computing platform to provide Webbased geoprocessing functionalities. Challenges, approaches and architecture for integrating are discussed. Amazon cloud computing platform is adopted to demonstrate the implementation. The reminder of this paper is structured as followed. Section 2 of this paper introduces the related work. Section 3 addresses the challenges when deploying geoprocessing service in the cloud computing platform. The approaches are discussed in Section 4 and the implementation details are described in Section 5. Section 6 presents the conclusions and discusses planned future work. II. BACKGROUND This Chapter gives a brief introduction into the basic concepts about geoprocessing and cloud computing, the related works are also addressed. A. Web Processing Service Geoprocessing services provide building blocks for higher geoprocessing modeling and complex geoprocessing

2 tasks, and are the fundamental utilities for the Geospatial Processing Web (Yue et al., 2009). The fundamental purpose of geoprocessing is to allow the users to automate GIS tasks. Almost all uses of GIS involve the repetition of work, and this creates the need for methods to automate, document, and share multiple-step procedures known as workflows (ESRI Developer Network). In the context of web-based geoprocessing, the Open Geospatial Consortium (OGC) Web Processing Service (WPS) interface specification evolved as the de facto standard. It provides a straight-forward approach to publish and execute geoprocesses over the web (Baranski et al. 2009). According to WPS specification, a geoprocess is defined as any calculation operating on spatially referenced data. The data required by the service can be available locally, or delivered across a network using data exchange standards. The calculation can be as simple as subtracting one set of spatially referenced numbers from another (e.g. determining the difference in influenza cases between two different seasons), or as complicated as a global climate change model. The WPS interface specification includes three operations: GetCapabilities, DescribeProcess and Execute. The GetCapabilities operation is used to determine the capabilities of an implementation of this service, the operations it provides, and the data it serves. The DescribeProcess operation provides a means for a client to determine what the mandatory, optional, and default parameters are for a particular process, as well as the format of the data inputs and outputs. Based on this information, the client could perform the Execute operation upon the designated process. As every OGC Web Service, the WPS communicates through HTTP- GET and HTTP-POST based on an OGC-specific XML-message encoding. B. Cloud computing Cloud Computing is associated with a new paradigm to provide computing infrastructure (Vaquero et al. 2009), which can support geoprocessing over the Web. Some new aspects covered by Cloud Computing paradigm are the efficient use of computational resources, on-demand access, instant scalability, unlimited data storage, unlimited processing power and low start-up barrier (52 North Cloud Computing). A cloud could be public, private or hybrid, depending on the deployment scenarios. The public cloud is available to the public, while the private cloud is used inside an organization. Since this hybrid cloud solution is commonly bound together by proprietary technology, it will only be embraced by enterprise computing in the future as standards are developed. According to the type of provided capabilities, major categories of Cloud Computing include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS) (NIST, 2009). Since Amazon AWS platform is adopted in the implement in this paper, a brief introduction about Amazon AWS will be given in this part. Amazon Web Services is more than a collection of infrastructure services. With pay as you go pricing, the user can save time by incorporating compute, database, storage, messaging, payment, and other services. All AWS services can be used independently or deployed together to create a complete computing platform in the cloud. Amazon Elastic Compute Cloud (EC2) is a web service that provides resizable compute capacity in the cloud. Amazon Simple Storage Service (S3) can be used to store and retrieve large amounts of data, at any time, from anywhere on the web. C. Related work Some GIS enterprises, such as ESRI, have made certain progress in moving their products and services into Cloud Computing. ESRI, collaborated with Amazon, uses the public cloud environment in several different ways, and currently the following options are available: the ability to deploy ArcGIS Server on AWS; ArcLogistics (a cloud application for optimizing routing); Business Analyst Online (a cloud application for geographic analysis of demographic, consumer, business, and other data). There are some progresses about using Cloud Computing platform in geospatial application. Baranski et al. (2009) creates a cloud enabled spatial buffer analysis service in the Google App Engine and conducts a stress test for scalability evaluation. The interface of the service follows the Web Processing Service (WPS) specification and the implementation is based on the Java-implemented 52North open source WPS software. Shao et al. (2011) provide the solution for implementing Web Coverage Service on Amazon Cloud Computing platform. Gong et al. (2010) presents an implementation of geoprocessing service that integrates geoprocessing functions and Microsoft Cloud Computing technologies to provide geoprocessing capabilities in a distributed environment. Hoffa et al. (2009) explores the use of Cloud Computing for scientific workflow, which focus on a widely used astronomy application-montage. Schaeffer et al. (2010) reviews Cloud Computing technology and identifies the paradigm behind it with regard to SDIs. Schaffer et al. (2010) presents an approach for enabling the commercial use of OGC Web Processing Services (WPS) in SDIs. Lu (2010) investigated the service and cloud computing oriented architecture for constructing a distributed and web service enabled geographical information platform. III. CHALLENGES FOR THE INTEGRATION In the professional GIS software, the geospatial processing functions have been developed for over decades. However, the GIS software cannot support an open processing and analysis environment, which means that those functions can only be used under their own proprietary environments and configurations. Taking a slope-aspect computation (r.slope.aspect) in Geographic Resources Analysis Support System (GRASS) GIS as example, which is used to generate raster maps of slope, aspect, curvatures

3 and partial derivatives from a elevation raster map. To execute r.slope.aspect processing function, a proper related environment variables need to be pre-configured. Moreover, both the input and output depends to GRASS internal raster map. The first challenge is that the existing geospatial processing function need to be adapted to the running environment in the Cloud Computing environment. Particularly, we should take the following question into consideration based on Amazon AWS component: How to upload the existing geospatial data into Cloud and discovery data in Cloud? How to migrate the existing geospatial processing functions into Cloud? How to scale dynamically the computation capacity when request sharply changed? The second challenge is with regard to the interoperable service interface that geospatial applications interact with Cloud services. With the development of Earth Observation technologies, geospatial data is gathered by various sources and is highly complex and heterogeneous. Particularly, the temporalspatial coverage, data format, map projection, resolution may be incompatible. Owing to the multidisciplinary nature of Earth science modeling and applications, the involved geoprocessing functions are always diverse and complicated. The support of interoperability, therefore, is crucial to implement geospatial information share and interoperability in a distributed environment. The third challenge is the ability to scale the computing resource up or down automatically according to the defined conditions. The single geoprocessing in cloud could not show the advantage of Cloud Computing, the flexibleness to increase the computing resource during demand spikes, which is a major feature of Cloud Computing, is necessary to maintain the performance. And when the demand decrease, it is also necessary to scale down the computing resource to minimize costs. The geoprocessing functions in Cloud Computing platform should bestow such flexibleness in a transparent way to the end-users. IV. APPROACH Based on the preceding analysis of the services in Amazon AWS platform and geoprocessing functions, we propose the three main points for integrating geoprocessing functions and Amazon AWS platform. Fig. 1 illustrates the proposed architecture of the integrated system. A. Data Storage The first point is to use storage service in Amazon AWS to manage the application data and store the output data. Amazon AWS platform provides two storage services: Amazon S3 and Amazon Elastic Block Store (EBS). Amazon EBS provides block level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are offinstance storage that persists independently from the life of an instance, which allows the user to create volumes that can be mounted as devices by EC2 instances. Compared with Amazon EBS, S3 is subject to eventual consistency which means that there may be a delay in writes appearing in the system whereas EBS has no consistency delays. Also EBS can only be accessed by one machine at a time whereas snapshots on S3 can be shared. Amazon S3 provides a highly durable storage infrastructure designed for mission-critical and primary data storage. Objects are redundantly stored on multiple devices across multiple facilities in an Amazon S3 Region. Moreover, Amazon S3 has the higher latency throughput. In view of those differences, Amazon S3 is adopted to store and manage application data in our implementation. B. Geoprocessing function migration The second point is to move the geoprocessing functions into the virtual machine in Amazon Cloud platform. Those functions in professional software depend on its own standalone environment and libraries. Since the difference environment in Cloud platform, the extra efforts need to be made to move local geoprocessing functions into local environment. As described in Section 3, Amazon AWS provides EC2 to give the resizable compute capacity in the cloud, which presents a true virtual computing environment, allowing the user to use web service interfaces to launch instances with a variety of operating systems, load them with the custom application environment, manage network s access permissions. To build a self-contained environment for geoprocessing functions in Cloud, Amazon Machine Image (AMI) is created to start EC2 instance. An Amazon Machine Image (AMI) is a special type of pre-configured operating system and virtual application software which is used to create a virtual machine within the EC2. It serves as the basic unit of deployment for services delivered using EC2. C. Interoperable interface design Because of the multidisciplinary nature of geospatial data and geoprocessing functions, it s necessary to implement an interoperable service interface that geospatial applications interact with Cloud services. One possible solution is to adopt the existing OGC standards, which provides a method for applications to access data and geoprocessing services seamlessly in a distributed environment, regardless of heterogeneity of involving geospatial data and geoprocessing platforms. Figure 1. High level framework for geoprocessing in Amazon AWS

4 Standardized geoprocessing functionality is available through the OGC WPS interface. All operations are accessible over an internet protocol (HTTP) using structured messages (XML). The WPS is not restricted to any type of data or type of process and allows service providers thereby to host any type of process combined with the appropriate data format via WPS interface. Typical types of data formats supported by WPS are GML, KML, and shape file for vector data or GeoTIFF for raster data. V. IMPLEMENTATION Amazon AWS is adopted to implement WPS. The AWS services could be managed through Amazon Management Console or command line tools. The implementation includes the following steps, using the terrain slope and aspect computation as an illustration. A. Uploading data to Amazon S3 Amazon S3 stores data as objects within buckets. An object is comprised of a file and optionally any metadata that describes that file. In order to store an object in Amazon S3, the user could upload the file to a bucket, and set the permission on the object as well as any metadata. The user could manage the upload process through Amazon Management Console, which provide a point-and-click webbased interface for accessing and managing all of the user s Amazon S3 resources, as Fig.2 illustrated. Figure 2. Amazon S3 upload interface through AWS console To get started with Amazon S3, the following steps should be implemented: (1) Create a Bucket to store your data. The user can choose a Region where the bucket and object(s) reside to optimize latency, minimize costs, or address regulatory requirements. (2) Upload Objects to the Bucket. The data is durably stored and backed by the Amazon S3 Service Level Agreement. (3) Optionally, set access controls. The user can grants others access to the data from anywhere in the world. Once the data are uploaded to Amazon S3, the other Amazon Web Service and applications could access the data through its URL. B. Creating Amazon EC2 for geoprocessing functions To move the geoprocessing functions into Amazon Cloud from standalone environment and offer those functions through OGC WPS interface, the Amazon EC2 instances, which provide both the environment for geoprocessing algorithm and the framework for interoperable web interface, should be created. As to the geoprocessing function, GRASS GIS software is adopted because it provides functions to manipulate raster and vectors, and to process satellite image data. GRASS GIS is proven to be applicable to enable geoprocessing capabilities based on some previous related research. As to the WPS framework, 52North WPS open source software is chosen to enable the deployment of geo-processes on the web in a standardized way. It features a pluggable architecture for processes and data encodings and supports all OGC WPS specification (Version and 1.0.0). To make GRASS GIS work with 52n WPS, some extra efforts and tedious workloads about configuration and programming need to be finished in the previous version. In the latest version (GRASS GIS version 7.0, 52n WPS version 2.0 RC6), however, the integration are getting easy owe to a framework wps-grass-bridge, which is used to make the integration of GRASS GIS 7 in WPS server more easily. Many GRASS GIS 7 modules can be attached out of box. The following steps are used to describe how to set up the 52n North with GRASS GIS backend under Amazon EC2 instance (Suppose the user has created an account in Amazon AWS). (1) Select a basic 32-bit Amazon Linux AMI (AMI Id: ami-76f0061f); and launch the Amazon EC2 instance; Keep the public DNS name for the instance. (2) Access the created instance using any SSH client based on the public DNS name and the private key file. (3) Download and Install Apache Tomcat 6, Python 2.5 or later and Java 1.6 in the created instance. Download and unzip the wps-grass-bridge. (4) Download the latest 52n WPS RC6 release, rename the war file to wps.war and put the war file in the webapps folder of tomcat. (5) Download GRASS GIS 7.0 (Linux version) and its dependent libraries. Change the GISDBASE parameter in the file.grassrc70 to the path of GRASS installation. (6) Start Apache Tomcat 6, and go to Web Admin page through web browser. Activate the table Algorithm Repositories and enable the GRASS Repository. Set the related variables.

5 (7) Save and activate configuration. Request the GetCapabilities operation of WPS to check the availability of GRASS geoprocessing service from the response XML. Once the geoprocessing functions are deployed on Amazon EC2 and WPS framework is built, an Amazon AMI could be created from the EC2 instance. The user could launch the multiple same Amazon EC2 instances from the created AMI. C. Supporting auto scaling Auto scaling enables the user to closely follow the demand curve for the cloud application, and reduce the need to provision Amazon EC2 capacity in advance. Auto Scaling is enabled by Amazon CloudWatch, which allows the user to scale Amazon EC2 capacity dynamically based on the predefined conditions, such as average CPU utilization, network activity and disk utilization. To set the conditions for CloudWatch, the user could access the set up page in Amazon Management Console, or using CloudWatch command line tools to apply certain scaling conditions. Based on the AMI, which is created from Amazon EC2 instance, the user could launch multiple new EC2 instance and manage those instances with Auto Scaling services, and specify the multiple threshold value to determine the number of running instances. D. Validation The following is an implementation of the KVP encoded DescribeProcess request for retrieving the process description of the terrain slope aspect computation: 1.amazonaws.com:8080/wps/WebProcessingService?Reques t=describeprocess&service=wps&identifier=r.slope.aspect <?xml version="1.0" encoding="utf-8"?> <wps:execute service="wps" version="1.0.0" xmlns:wps=" xmlns:ows=" xmlns:xlink=" xmlns:xsi=" xsi:schemalocation=" <ows:identifier>r.slope.aspect</ows:identifier> <wps:datainputs> <wps:input> <ows:identifier>elevation</ows:identifier> <ows:title>srtm DEM data</ows:title> <wps:reference xlink:href=" M_f03_n036e008.tif"/> </wps:input> </wps:datainputs> <wps:responseform> <wps:rawdataoutput> <ows:identifier>aspect</ows:identifier> </wps:rawdataoutput> </wps:responseform> </wps:execute> Table 1 shows the XML encoding of the Execute request for invoking the WPS process r.slope.aspect. Post the XML content to WPS server, and the results, as Fig. 3 illustrated, will be returned. TABLE I. WPS EXECUTE REQUEST IN XML (a)

6 demonstrate the feasibility and flexibility of deploying geoprocessing into Cloud Computing platform for academic purposes. Comparison for economic cost between different Cloud Computing platforms will be made in the future work. (b) Figure 3. Result image of WPS Execute request. (a) Original DEM image, (b) Aspect image returned from WPS server. VI. CONCLUSION AND FUTURE WORK This paper presents an implementation of geoprocessing service that integrates Amazon cloud computing and geoprocessing functions to provide geoprocessing competence in a distributed web environment. Approaches on how various services in Amazon platform can be utilized to meet the storage and computing requirements of geoprocessing services are described, how to scale the computing resources based on predefined conditions are also addressed. Moving existing geoprocessing functions into a Cloud Computing platform require considerable work, since the infrastructure, deployment requirements and APIs of different Cloud Computing platform are also different. The platform-depend APIs complicate the development of geoprocessing functions in different Cloud Computing platform. By adopting the bridge framework of 52n WPS and GRASS GIS 7, making a WPS server is easy in cloud environment. Based on the unlimited computing capacity provided by Cloud Computing platform, the users could concentrate on domain business logic without worrying about the hardware and software limitation. Since the most current commercial Cloud Computing platforms do not have free offers, the user should take the economic cost into account when deploying application in Cloud environment. The implement in this paper is to REFERENCES [1] Xiaoyan L., Liping D., Weiguo H., Peisheng Z., Dadi, U., Sharing and reuse of service-based geospatial processing through a Web Processing Service. In: th International Conference on Geoinformatics, pp.1-5, Aug [2] Yuanzheng S., Liping D., Jianya G., Yuqi B., Peisheng Z., GIS in the Cloud: Implementing a Web Coverage Service on Amazon Cloud Computing Platform. Lecture Notes in Electrical Engineering. 2011, Volume 98, pp [3] Jianya G., Peng Y., Geoprocessing in the Microsoft Cloud Computing platform Azure. A special joint symposium of ISPRS Technical Commission IV & AutoCarto in conjunction with ASPRS/CaGIS 2010 Fall Specialty Conference. 6 pp. [4] J. D. Blower, GIS in the cloud: implementing a web map service on Google App Engine. In: Proceedings of the 1st International Conference and Exhibition on Computing for Geospatial Research and Application. Washington D.C, June 21-23, [5] Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, Matei Zaharia, A view of cloud computing. Communications of the ACM, Vol.53. [6] Brauner, J., Providing GRASS with a Web Processing Service Interface. GI-Days 2008, Münster, Germany. [7] Armbrust, M., Fox, A., Griffith, R., Joseph, Anthony D., Katz, Randy H., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., and Zaharia, M., Above the Clouds: A Berkeley View of Cloud Computing. Technical Report. EECS Department, University of California, Berkeley. 23 pp. [8] Baranski, B., Schaeffer, B., and Redweik, R., Geoprocessing in the Clouds. In: GeoInformatics, van Rees, E. (Ed.), Volume 12, pp [9] Schaeffer, B., Baranski, B., and Foerster, T., Towards Spatial Data Infrastructures in the Clouds. In : Proceedings of the 13th AGILE International Conference on Geographic Information Science, Guimarães, Portugal, Painho, M., Santos, M., and Pundt, H., (Eds.), Geospatial Thinking, Lecture Notes in Geoinformation and Cartography, Springer Verlag, pp [10] Schut, P., OpengGIS Web processing service. Version 1.0.0, OGC r7, Open Geospatial Consortium, Inc.. 87 pp. [11] Wang, Y., Wang, S., and Zhou, D., Retrieving and Indexing Spatial Data in the Cloud Computing Environment, In: Proceedings of the 1st International Conference on Cloud Computing (CloudCom 2009), Jaatun, M.G., Zhao, G., and Rong, C., (Eds.), Lecture Notes in Computer Science (LNCS), vol 5931, pp