GeoSciWave: Grid services for data streaming, mining and visualization

Transcription

1 GeoSciWave: Grid services for data streaming, mining and visualization M. Zhizhin*, A. Andreev*, A. Platonov**, A. Soldatov**, V. Velikhov**, M. Boyarsky*** and R. Nazirov*** *Geophysical Center, ***Space Research Institute, Russian Acad. Sci. **Russian Research Center Kurchatov Institute

2 3-years Grid collaboration Space Research Institute, RAS Geophysical Center, RAS CS Department, Moscow State University Russian Foundation for Basic Research SKIF-Grid, Russia-Belorussia Union DEGREE SSA, EGEE II National Geophysical Data Center, NOAA Microsoft Research, Cambridge Microsoft Research, Redmond

3 Geo e-science Grid Services Parallel data processing pipeline: network + storage + mining + visualization TeraFlow Network with GLORIAD: 1 Gb/s LAN from Chicago to Moscow to stream data and video Data Center with NOAA and MSR: parallel Active Storage storage using MS SQL Server database cluster Data Rendering Farm with MSR and EGEEIII: parallel data mining cluster for climate and space weather modeling Tiled Display Video-Wall with EVL: 12 displays, 30 MPixel

4 Active Storage, Modeling, Data Mining and Visualization Services 7LPH VHULHV * ULGV 7UDMHFWRULHV

5 Global Lambda Integrated Facility Available Advanced Network Resources GLIF is a consortium of institutions, organizations, consortia and country National Research & Education Networks who voluntarily share optical networking resources and expertise to develop the Global LambdaGrid for the advancement of scientific collaboration and discovery. Visualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC. Source: Joe Mambrotti

6 Source: John Silvester

7 GLORIAD: 10Gb Worldwide Ring Source: Natalia Bulashova

8 USA-Russia Lightpath for Fast Data Transfer of Terabyte-sized Scientific Datasets National Center for Data Mining (NCDM) at the University of Illinois at Chicago, Geophysical Center RAS and Space Research Institute RAS have successfully moved 1.4 TB of data in 4.5 hours over a 1 Gbps lightpath between Chicago and Moscow as part of the Teraflow Network initiative Using NCDM s open-source UDP-based Data Transfer protocol (UDT), we were able to transfer the MS SQL database with SDSS astronomy catalog. The 2.5 TB database dump was compressed to 1.4 TB, split into 60 files, transferred over a 1 Gbps lightpath and then decompressed in Moscow and loaded back to MS SQL Server The SkyServer portal and the SDSS database were developed by Jim Gray at MSR and Alex Szalay at JHU. Russian language mirror now resides at in Moscow Direct Lightpath link from IKI in Moscow to NGDC NOAA in Boulder has been successfully tested

9 TeraFlow Test Bed Topology Servers w/ 1 G NICs TFS 1*1GE TFS 10 G =TeraFlow Server Netherlight Amsterdam 3 G WAN OC192/STM64 10GE LAN 10G WAN Gloriad 1 G Moscow TFS L2 TFS TFS WAN Termination L2 TFS TFS Starlight, Chicago SL E1200 Nortel HDXc IRNC GLORIAD OME Nortel HDXc SL TF Server IP addresses Are , 15, and 16 All Paths Are L1/L2 3*1 G Within 10 WAN

10 UDT parallel UDP data UDT protocol is developed by the National Center for Data Mining, UIC UDT is an application level data transport protocol for emerging distributed data intensive applications over wide area highspeed networks UDT uses UDP to transfer bulk data and it has its own reliability control and congestion control mechanisms UDT is not only for private or QoSenabled links, but also for shared networks IKI RAS has developed UDT tunneling application for HTTP/FTP/ over UDT streaming

11 Russian Mirror:

12 GLORIAD Press Release, June 4, 2007

13 National Geophysical Data Center NOAA Space Physics Interactive Data Resource - SPIDR NOAA Satellite Data Products Cloud - CLASS Comprehensive Large Array Stewardship System Microsoft Research Cambridge Environmental Scenario Search Engine: statistics of extreme weather events Microsoft Research Redmond Climate Induced Vegetation Change Analysis Tool Vegetation types map GLC days average temperature of the beginning of vegetation period Time series as a trajectory in the two-dimensional phase space (P-pressure, T-temperature)

14 National Geophysical Data Center NOAA Space Physics Interactive Data Resource - SPIDR =

15 Defense Meteorological Satellite Program, DMSP orbit section Mosaics geolocated Stable night time lights Night time lights difference

16 Gas Flares Monitoring

17 Space Weather Re-Analysis Input: ground and satellite data from SPIDR Space weather numerical models Output: high-resolution representation of the near-earth space

18 Active Storage for Common Data Model Common Data Model is an array of values of a parameter at different times on regular grids, at specified locations (point or station data) or space-time trajectories W Index space explosion Database cluster design, 1999 ODW ORQ MySQL databases, $ % Active Storage for Common Data Model 2 TB of climate history data The data had been stored on a cluster of 58 separate databases, each containing 1 year data, stored in a separate binary field for each grid point. The databases were queried sequentially or in parallel threads at the application level Parallel database cluster Performance results: 4D array lat-lon-height-time air (4D array) 1,8 1,6 1,4 MS SQL2008 single server storage Time, s 1,2 1 0,8 0,6 0,4 0, SINGLE MULTI NCEP_G NCEP_FULL Query < Space Time >

19 Environmental Data Mining and Visualization Fuzzy Logic State Space OGSA-DAI Activities Web Interface for Fuzzy Queries Open Source ESSE team JVM ant, axis, xerces, Linux/Java EsseDataResource Tomcat OGSA-DAI WS-I OGSA -DAI Activities GetXmlData Fuzzy GzipCompression GzipDecompression MySQ Database JDBC MySQL Fuzzy Engine.NET Port Open Source Framework ESSE team (common) 2.0 ESSE team (C#) Microsoft EsseDataResource IIS OGSA-DAI.NET OGSA -DAI Activities GzipDecompression GzipCompression Fuzzy GetXmlData SQL Server Database JDBC ODBC Fuzzy Engine

20 GLORIAD Press Release, August 24, 2007

21 Tiled Display Video Wall: SAGE 3.0 ported to.net Tiled Display UI client UI client SAGE Receiver SAGE Receiver SAGE Receiver SAGE Receiver FreeSpace Manager SAIL SAIL SAIL App1 App2 App3 SAGE Messages Pixel Stream Synchronization Channel

22

23 Examples of Scientific Visualization: subject areas and SAGE applications Astrophysics Images from the Space Hubble Telescope: Magic Carpet Large-scale structure models of our Universe: mediaplayer Environmental studies and remote sensing DMSP night time lights database: Magic Carpet Reanalysis of the climate history: NASA World Wind Art museums Frescos and large icons: Magic Carpet HD and 4K-video streams: mediaplayer

24 Scientific Visualization Applicatons

25 Computational Photography for Digital Art Maps St. Ferapont Monastery in Ferapontovo, XVI century