Data-centric Renovation of Scientific Workflow in the Age of Big Data

Transcription

1 Data-centric Renovation of Scientific Workflow in the Age of Big Data Ryong Lee, Ph. D. Dept. of Scientific Big Data Research Korea Institute of Science and Technology Information Korea

2 Outline Overview Scientific Data Projects in KISTI On-going Work: Data-centric Renovation of Scientific Workflow for Oceanographers Towards Further Scientific Data Customization Services Conclusions and Future Work

3 Scientific Data Projects in KISTI Towards an Advanced Foundation for Data-Intensive Scientific Research National Scientific Data Governance Scientific Data Mgt. and Sharing Scientific Data Analytic Platforms Design and Development of Scientific Data Governance Systems Establishment of International Co-operative Network Development and Distribution of Scientific Data Management and Sharing Platforms R&Ds of Advanced Technologies for Big Data based Scientific Work Supercomputer Storage HPC Network

4 A Novel Data Service specialized in Scientific Big Data Customization Motivated by Oceanographers exploration and analyses with high-resolution and long-term remote sensing data from Satellites Collaborating with KIOST(Korea Institute of Ocean Science & Technology) and KOPRI(Korea Polar Research Institute) in research projects on climate change and red-tide analysis/detection Climate Change Analysis Red-tide Analysis/Detection Supporting Data-intensive Scientific Analysis Tasks Remote Sensing Data taken by Satellites are overwhelming their processing capabilities Customizing the big data for analytics is an growing strong demand

5 (Korea Polar Research Institute) Case Study 1: Exploring the effects of climate changes Non-linear Relationship between Biology and Environmental Changes For better estimation, Multiple remote sensing data need to be compared Unprecedented scale of (Hundreds of TBs) Array-based remote sensing data (via satellites) should be handled efficiently

6 Case Study 2: Red-tide Analysis/Prediction (Korea Institute of Ocean Science & Technology 2002~2013 Red-Tide in South Coast of Korea: A digitized data from hand-made estimation Damage (KRW) Red-tides cause serious economic damages quite unexpectedly Long-term and high-res. remote sensing data should be examined intensively

7 Enhancing Remote Data Handling Capability High Res. & Global Range High Res. A Novel Challenge: Overcoming the practical limits for Handling Global & High-Res. Remote Sensing Big Data Untouchable Domain 250m Scientists Practice: Boundary of Computational Limit in most small-scale science Labs. 500m Low Res. 1000m Local Global 7

8 Realizing Oceanographers Dream to equip with better working env. Renovation of Remote Sensing Big-Data Customization Process As-Is: Require ments MODIS Data Transform (L1Aà..à L3 SMI) File-based Data Analysis (L3 Level) To-Be: Customized Data Transform As-Is: MODIS Data Transform (L1Aà..à L2 ) To-Be: Slow Array-based Data Management (Global Area, 1km) File-based Data Analysis (L2 Level) (Local Area, 1/0.5/0.25km) Customized Data Transform Array-based Data Analysis Slow Array-based Data Management Customized Data Transform Array-based Data Management SciDB based Data Mgt. Base Array Derived Arrays Platform Design (KISTI) Bottleneck A2 A1 L3SMI in SciDB MODIS L2 A3 UDF Extension f(a2, A3) MODIS L3 BIN MODIS L3 SMI Selective Loading Array Manipulation & Computation Complex Analysis in SciDB Goal: Global Climate Change Analysis Goal: Red-tide Analysis/Detection User Environment for Scientific Data Analytics UI for Customizing Data Transform Functions - Transform Customization - Monitoring of Transform - Loading into Array DBMS UI for Array-based A nalysis Functions - SciDB Viewer - Array Data Manipulation - Array Data Provenance UDF f(a1, A2) Parallel Com puting Quick and Smart!! Array-based Data Analysis MODIS L1A MODIS L1B Quick and Smart!! Data Visualization Functions - SciDB Array Visualization - Display of Satellite Image - Big Data Summarization

9 A-DISC (Array-based Data-Intensive Scientific Computing Platform) What We Pursue: To support a variety of Scientists' Discovery with Big Data in their friendly manner What We Enable Array-based Data-Intensive Scientific Computing Platform (A-DISC) Scalable Versatile Scientists-friendly How We Approach Big Data Preprocessing on Parallelism Scientific Data Customization Interactive Scientific Data Workspace What We Conduct HPC-based MODIS Satellite Remote Sensing Data Translation Array DBMS based Scientific Data Analysis Support Scientific Understanding Global Ecology Analysis BigSat-Converter Scientific Data Workspace Red-tide Analysis and Detection

10 Renovate the Workflow: HPC* and Array-based Data Processing User User User Customization Transforming on Parallelism Loading into Array DBMS Customizing Options Big Data Store (Transformed Data Mgt.) Array-based Data Analytics Web-based Interactive Big Data Analytics Customized Data Transform Parallel and Distributed Computing (SGE) Big Data Store User-customized Data, ready for Analysis HDF-to-Array Transform & Load (L1A, L1B, L2, L3) s n a r T Fast form NASA MODIS Aqua * HPC (High-Performance Computing) Array Data Manipulation & Analysis Scientific Data on Array DBMS

11 BigSat-Converter A Server-Client System for Customized Transform of Remote Sensing Big Data Currently, MODIS Aqua/Terrain L1A, L2, RGB, L3 converting on a HPC cluster Planning to support various remote sensing data customizations

12 Enhancing the Power of Data Transformation with HPC User Administrator Customization: Specification of Region/Period/Product/etc. User Customization Administration: Managing the utilization of Computing and Storage Resources Transforming on Parallelism Customizing Options Master Parallel Computing (Sun Grid Engine) 10G Infiniband Big Data Store (Transformed Data Mgt.) Workers HDF-formatted Data Big Data Store (L1A, L1B, L2, L3) NAS Storage 15 TB Format NTFS CPU User-customized Data, ready for Analysis Master & 9 Worker Nodes Memory Storage 2260 MHz, 8 Core 18 GB 250GB 12

13 Significant Benefits of HPC in Remote Sensing Data Transform L1B L2 Total Single 60m 15s 74m 12s 134m 27s BigSat-Con verter 7m 18s 10m 25s 17m 43s Speed-Up 8.25X 7.12X 7.59X Satellite images of Korean peninsula Improvement of Process Improvement of Data model Legacy Code Re-designed Code Performance Improvement: 7.59 times faster 1 month 4 days

14 Array-based Remote Sensing Data Management and Manipulation L1B RGB Array - Dimension: longitude x latitude x time ( x x 3 days ) - Attributes: <Red, Green, Blue> Swath - 1 swath (in RGB), every 5 mins - Image size : 136 x day: 288 swaths L3 SMI in SciDB Jan. 1-31st, 2014 Dimension: (lon. x lat x time) 4320 x 2160 x 31 (days) Attribute: <chlorophyll>

15 Scientific Data Workspace An integrated workspace for scientific data manipulations and analyses Array data on clusters are accessible easily on a graphic user interface R-based analytical programming is supported on a HPC cluster

16 Applied to the field for Red-Tide Analysis A domestic news article regarding to our achievement with KIOST on the red-tide analysis issue A screenshot of our system application

17 Summary: How we enhanced Scientists Capabilities to Big Data Customization of Remote Sensing Big Data Increasing public access to scientific data of huge volume Scientists data requests are various, not being satisfied by the forms given from data publishers Customization service will become important more and more Array-based Data Manipulation and Analysis Scientific data are often stored in a formatted file (HDF, etc.) Adopting Array DBMS Technology can boost the processing Data loading into Array DBMS (for R based analysis) Finally, letting scientists work much easily with Array DBMS

18 Towards Further Scientific Data Customization Services Customization Platforms for Utilizing Scientific Big Data are growing demands from Public Requests for resolving many Natural and Social Problems Enhancing and Boosting Scientific Big Data Processes are an interdisciplinary work to understand domain knowledge for developing unprecedented technology and systems It s not a simple combination of various IT technologies, but rather a well-crafted artwork which should work as part of our society, constructing an elementary foundation for the whole organic integration of various systems

19 Conclusions and Future Work At the moment, we are conducting R&Ds of Scientific Data Customization Service focusing on remote sensing data transform for practical requests We will continue to enhance the developing system for various scientific data as well as remote sensing data towards generic scientific customization services We are always soliciting international cooperation for sharing further R&D issues and valuable experience on common interests Contact Me: Ryong Lee, Ph.D. ( ryonglee@kisti.re.kr )

20 Thank you very much for your kind attention!!