Cloud-based Geospatial Data services and analysis

Transcription

1 Cloud-based Geospatial Data services and analysis Xuezhi Wang Scientific Data Center Computer Network Information Center Chinese Academy of Sciences

2 Outlines 1 Introduction of Geospatial Data Cloud 2 Cloud-based Geospatial Data Analysis platform 3 Cloud-based Geospatial Data visualization platform

3 Geospatial Data Cloud I. Platform overviews II. III. IV. Geospatial data services Geospatial model services Services achievements

4 Why do we do? In 2008, USGS opened all geospatial datasets to the public for free. Landsat, MODIS is the most popular remote sensing dataset for scientists and researchers. Especially, the Landsat series. Due to the low international network speed, many Chinese scientists cannot take advantage of these resources easily. Based on the CNIC s mass storage and high speed network( CSTNET: Chinese Science and Technology Network), we mirrored the geospatial datasets covered China and surrounding area, chinese scientists and researchers

5 Platform History USGS opened its Landsat datasets. we contacted the USGS staffs through . We got an oral permit for Landsat redistribution from USGS. we download the Landsat datasets, and provide public services for Chinese scientists Geospatial data cloud opened for public services Host the second user conference with ESRI China Cooperated with National Center for Atmospheric Research (NCAR) Mirrored the NCAR datasets provide public services for Chinese Scientists Dowload MODIS dataset covered CN Host the first user conference Design a new cloud-based data platform Design a new cloud-based geospatial analysis and visualization platform We got a permit for SRTM dataset redistribution from International Centre for Tropical Agriculture(CIAT). We cache SRTM global elevation data, and provide online data services. Accroding to the user requirements, We make some remote sensing products from Landsat dataset and Modis dataset. such as vegetation index, water cover dataset.

6 Platform objectives Based on our cloud infrastructure, we hope to construct an 'one-stop' geospatial data service platform for Chinese scientists and researchers. An open cloud-based platform meeting the requirements of data search, data acquisition, online storage, scientific models, online calculation and map visualization. Search Acquisition Online Calculation Storage Model Visualization

7 General framework Data integration Automatic Metadata crawling User data spaces Web map services Model services Data management Map publish Model package Task management Map visualization Online execution Map management Map style edit Model management Service management Map share Model flow System managem ent Users Services Data Automatic data entities cache Data services Data query Smart data Order Data category Favorite data Integrated search Data application Complex search Batch download One-key indexes Models Indexes Tasks Cloud environments Distributed file system Virtual Machines Task scheduling Resources monitor Mass Storage Database cluster Online storage Near-line storage

9 Online data resources Open geospatial datasets: LANDSAT MODIS EO-1 DEM NCAR NOAA and LUCC More than 350TB data entities, and over 7 million data records. Covered China and surrounding area Online services for Chinese scientists and researchers with free of charge LANDSAT MODIS MODIS_L1B DEM EO-1 Landsat1-3 MSS Landsat 4-5 MSS Landsat 4-5 TM Landsat 7 SLC-on Landsat7 SLC-off Landsat 8 OLI TERRA AQUA TERRA AQUA GDEM 30m SRTM 90m Ocean elevation HYP_L1R HYP_L1G AL1

10 Generated geospatial products Water-cover of China from Landsat TM/ETM in Summer Slope, Aspect, Topographic position indexes et. of China Land Surface Temperature, Vegetation indexes (5 days, 15 days, and 30 days) of China, from 2000 now Water Cover ASPECT LST

11 Geospatial data query Based on geospatial storage cluster and search engine, we provide three ways for users to access their preferred data: 1: by address name 2: by geo-location 3: by administrative unit I need Landsat data in summer, 2000 Input: geo-location or address PostgreSQL+PostGIS Cluster I need DEM data of Beijing Massive geospatial metadata

12 1. search by address name Platform Screenshots

13 2. Search by geographic coordinates Platform Screenshots

14 3. Search by polygon features Platform Screenshots

15 4. Search by administrative unit Platform Screenshots

16 Platform Screenshots The result page that matched the query criteria Extra attribute filters

18 Geospatial model services Why do we need this services? For example: If we need Beijing DEM data, there have two ways. Traditional way: search DEM data, download DEM data, mosaic DEM data with ArcGIS, and then clip DEM data with Beijing shape mask. Convenience way: select Beijing administrative unit in the web, submit a clip task to system, and then download Beijing DEM data. Which is the best way? I prefer the second way. That s why we provide online geospatial model services.

19 Overall frameworks We developed an elastic online task processing system with cloud techniques. Users fill model parameters on the web page and submit tasks to the system; The tasks were run under cloud virtual machines; virtual machines in Cloud The results were stored in cloud storage. Model page Model page VM VM Task VM Task Task scheduling Task Computing resource pool Storage pool

20 DEM tools: Online models and tools DEM clip and mosaic; Reprojection; Aspect, Slope, Topographical index calculation of specified region LANDSAT tools: Online gap-fill model for Landsat ETM7 SLC-off dataset MODIS tools: Online Land Surface Temperature calculation of user specified time range Online vegetation indexes calculation of user specified time range

21 User manage their own data here, including upload, download, rename, delete, create folder and move Private cloud storage We provide user with private cloud storage to store their own model inputs and outputs data.

22 Platform Screenshots Online category page for Geospatial tools and models

23 Platform Screenshots Model parameters page of DEM clip and mosaic tool User only need to fill task name, select a region in the web map, and choose output folder

24 Platform Screenshots Task tracking page list user s submitted tasks They can tracking task states, stop and start task, adjust task priority here.

25 Platform Screenshots Task result page When the task was finished, the results can be download from here, or from user private cloud storage

27 Service target Our platform was wildly used in many research fields in China: earth science, environmental science, ecology, hydrology, meteorology, oceanography, natural disasters, city planning...

28 User characteristics Based on statistics in November 2013 Others Company CAS Government University Doctor Others Currently, over 70,000 users Master Bachelor

29 Service statistics Avg. downloaded over: 200 GB/day; online downloaded over: 250 TB; offline copied over: 137 TB;

31 Cloud-based Geospatial Data Analysis Platform Why do we need this platform? For example: If we want to calculate Chinese monthly temperature variation from 2000 to 2014, how can I achieve this? Traditional way: download all data, and calculate with workstations. It may spent several months to download data. This is a very bad thing. Shortcuts: using cloud computing technology. Apply virtual machines in the cloud; Write code in the virtual machines; read all data from cloud storage; Run program in parallel Which way is the best? I prefer the second way. That s why we want to construct this platform.

32 Overall frameworks VM VM VM VM virtual machine pool Data pool Cloud manager with OpenStack Public geospatial data Distributed file system User private geospatial data Computing resource pool Storage resource pool...

33 Currently progress of the platform We have finished the prototype system. The system is under inner tests for access pressure and stability. In the end of this year, it will be provided for the public access.

35 Cloud-based Geospatial Data Visualization Platform Why do we need this platform? In many cases, users want to publish their geospatial data, model results, vector map to the web site; They also want to customize the map layer styles through the web pages; They also want to customize the components and styles of the webgis pages; To achieve this goal, we construct this platform.

36 Overall frameworks HTML5 map client... Dynamical render map in Cloud Map request VM VM Map styles editor Optimized Web Map Service VM Dynamical cache Tiles Storage Computing resource pool Storage pool

37 Currently progress of the platform Most of system functions were already finished, under inner tests and bug fixed. Published services provided: OGC WMS(web map service), Google TMS, OGC WFS (web feature service) GeoJson, WorldWind...

38 System screenshots I. Render multi-bands raster layer with RGB false-color, custom band composition

39 System screenshots II. Render thematic raster layer with unique color theme

40 System screenshots III. Render raster layer with classified color theme

41 System screenshots IV. Render raster layer with gradient color theme

42 System screenshots I. Render vector layer with Simple symbol

43 System screenshots II. Render vector layer with colorful symbol by unique attributes

44 System screenshots III. Render vector layer with colorful symbol by classified attributes

45 System screenshots IV. Render vector layer with sized symbols by classified attributes

46 System screenshots Vector attributes query and edit

47 Thanks Any suggestions?