National Institute of Advanced Industrial Science and Technology Yoshio Tanaka

Transcription

1 Migrating from Grid to Cloud: Case Study from GEO Grid National Institute of Advanced Industrial Science and Technology Yoshio Tanaka

2 What is the GEO Grid? The GEO (Global Earth Observation) Grid is aiming at providing a Cyber Infrastructure for worldwide Earth Sciences communities to accelerate GEO sciences based on the concept that relevant data and computation are virtually integrated with a certain access control and ease-of-use interface those are enabled by a set of Grid and Web service technologies. AIST: OGF Gold sponsor (a founding member) AIST: OGC Associate member (since 2007) Satellite Data Map Geo* Contents GIS data Field data Grid Technologies Resources Geology Applications Environment Disaster mitigation

3 Example: Flood simulation GIS Data Rain Sensors Visualize Hi-Resolution Hazard Map GEO MET ASTER Digital Elevation Map Created by ASTER Satellite Images Neural-Map based Data Mining / Landslide Simulation on Large Cluster Computers

4 Why Grid? federation of distributed resources - security SSO Federation of distributed DBs High performance CSW computing

5 GEO Grid Security: GSI + VOMS

6 user login account (GAMA) server Account DB Terra/ASTER credential VO DB TDRS portal server VO (VOMS) server APAN/TransPAC GET GSI + VOMS WFS WCS WMS query CSW GSI + VOMS OGSA DAI exec GSI + VOMS GRAM GridFTP ERSDIS/NASA GIS server Data map server Maps catalogue/ metadata server Meta data gateway server Storage (DEM) GEO Grid Cluster L0 L0 L0 L0 L0 L0 L0 L0 L0 L0 L0 L0

7 Demo Environments in SIMS (ASTER+MODIS+Formsat2) Java Program Integration AIST Framework with OGSA-DAI OGSA-DAI Client SIMS portlet - query data - create web page which shows thumbnail images SQL SQL SQL SQL SQL Application Server Globus OGSA- DAI SQL w/ JDBC Database Server (Sybase) FORMOSAT-2 VOMS NSPO@TW VOMS AIST@JP Globus OGSA- DAI Database Server (PostgreSQL) ASTER SQL w/ JDBC OGSA- DAI MODIS

8 SIMS Search Results FORMOSAT-2 MODIS ASTER

9 GEO Grid Service Examples Satellite data archive and processing ASTER, PALSAR, MODIS, etc. Satellite data application Application of Satellite-Field data Integrator (SFI) for aerosol monitoring Description SDCP (Science Degree Confluence Project) Community validation tool for global land-cover & digital elevation models Hazard information QuiQuake (Quick Estimation System for Earthquake Maps Triggered by Observation Records) Volcanic Gravity Flow Simulations on Volcanic Area Geoscience data Geological maps, Active fault data, etc.

10 Migration from Grid to Cloud

11 Motivation for migrating to Cloud Deployment of applications is not easy Procedures for including new resources (deployments of applications) are troublesome. Need easy-to-use. Write once, run everywhere! Do we need Grid protocols? Do we need Grid Security? Delegation is necessary for third-party file transfer. But key management is burden for end users. Installation/configuration of VOMS is not easy. Do we need Grid protocol (e.g. GRAM)? GEO Grid applications use not Grid middleware/protocol but the other standards (e.g. OGC). Need to adapt the direction for wider use GEO Grid system is stably in operation, but not extendable (elastic). Data server and computing server are tightly coupled. It s hard to use resources outside organization. Is GEO Grid Design appropriate for use by business partners? Japanese government has a plan of promoting use of satellite data for wide use.

12 Goals of and approaches by PRAGMA Enable Specialized Applications to run easily on distributed resources Build once, run everywhere!! Investigate Virtualization as a practical mechanism Supporting Multiple VM Infrastructures (Xen, KVM, OpenNebula, Rocks, WebOS, EC2) Share VM images in PRAGMA VM repository so that we can boot our application VMs at any site by any PRAGMA colleagues. Discussed in PRAGMA 20 HK, March 3 rd and 4 th, 2011, 1 week before the big earthquake in Japan

13 2011 Tohoku Earthquake changed our R&D environments

14 Terra/ASTER Satellite Data Flow and Services Prior to March 11 ALOS/PALSAR TDRS - Data providing - Portal 70 GB/day (ASTER) JAXA 360 GB/day (PALSAR) ERSDAC AIST NASA - Archive (tape, B-ray) - Archive (on-disk) - Processing - WMS ASTER data: NASA ERSDAC AIST PALSAR data: JAXA ERSDAC AIST (processing, WMS, portal site, and data providing by AIST) 14

15 Terra/ASTER Data Flow and Services from March 11 till April 20 ALOS/PALSAR TDRS - Portal JAXA ERSDAC (AIST) Orkney Google NASA - Processing - WMS ASTER data: NASA ERSDAC (AIST) PALSAR data: JAXA ERSDAC (AIST) (processing and WMS by Orkney, portal site by Google) 15

16 ALOS/PALSAR Data Flow and Services from April 21 TDRS Terra/ASTER - Portal - Processing JAXA (AIST) Google OCCI ERSDAC UCSD NASA NCHC - Processing - Processing - WMS ASTER data: NASA ERSDAC (AIST) PALSAR data: JAXA ERSDAC (AIST) (processing by NCHC, SDSC, and OCCI, WMS by NCHC, portal site by Google)

17 Insights Fortunately, we already had VM images for satellite data processing. We have prepared for using cloud. Need to make it routine use! PRAGMA members had disasters/accidents. Japan earthquake Thailand flooding California power outage PRAGMA members has common interests/needs to build a sustainable infrastructure which could be used to support each other in case of emergency. We accelerated the development/deployment of PRAGMA Cloud.

18 Slide by courtesy of PRAGMA PRAGMA Grid/Clouds UZH Switzerland LZU China CNIC China JLU China KISTI KMU Korea AIST OsakaU UTsukuba Japan SDSC USA IndianaU USA UoHyd India NECTEC KU Thailand MIMOS USM Malaysia HKU HongKong ASGC NCHC Taiwan ASTI Philippines HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam CeNAT-ITCR Costa Rica UValle Colombia MU Australia BESTGrid New Zealand UChile Chile 26 institutions in 17 countries/regions, 23 compute sites,

19 Deploy Three Different Software Stacks on the PRAGMA Cloud QuiQuake Simulator of ground motion map when earthquake occurs Invoked when big earthquake occurs HotSpot Find high temperature area from Satellite Run daily basis (when ASTER data arrives from NASA) WMS server Provides satellite images via WMS protocol Run daily basis, but the number of requests is not stable. All these applications run as Condor workers

20 VM Image copied from gfarm slave SDSC (USA) Rocks Xen VM Image copied from gfarm slave S S gfc gfs gfc gfs NCHC (Taiwan) Ezilla/OpenNebula KVM Put all together Store VM images in Gfarm systems Run vm-deploy scripts at PRAGMA Sites Copy VM images on Demand from gfarm Modify/start VM instances at PRAGMA sites Manage jobs with Condor gfs gfs GFARM Grid File System (Japan) AIST QuickQuake + Condor NCHC Fmotif gfs UCSD Autodock + Condor AIST Web Map Service + Condor AIST AIST Geogrid HotSpot + Bloss + Condor gfc Condor Master AIST (Japan) OpenNebula KVM gfc S S VM Image copied from gfarm slave VM Image copied from slave gfs gfarm IU (USA) Rocks Xen VM Image copied from gfarm slave S LZU (China) Rocks KVM gfc gfs S gfc gfs gfs = VM deploy Script = Grid Farm Client = Grid Farm Server gfs gfc gfs S VM Image copied from gfarm slave Osaka (Japan) Rocks Xen Slide by courtesy of PRAGMA

21 Slide by courtesy of P. Papadopoulos, UCSD Essential Steps 1. AIST/GEO Grid creates their VM image 2. Image made available in centralized storage (currently Gfarm is used) 3. PRAGMA sites copy GEO Grid images to local clouds 1. Assign IP addresses 2. What happens if image is in KVM and site is Xen? 4. Modified images are booted 5. GEO Grid infrastructure now ready to use

22 Slide by courtesy of P. Papadopoulos, UCSD Cloud Sites Integrated in GEO Grid Executio PRAGMA Compute Cloud UoHyd India LZU China CNIC China JLU China Pool NCHC Taiwan ASTI Philippines AIST OsakaU Japan SDSC USA IndianaU USA MIMOS Malaysia

23 New Security Model (in progress) browser End users Yahoo Google, etc. OpenID AuthN Biglobe OpenID (accountable) AuthN Web Portal AuthZ Request services Portal Provider Request Services OAuth AuthZ Server AuthZ Server scope Resources Use OpenID/OAuth for AuthN/AuthZ Planning to use OpenID Connect Resource Server Resource Owner (e.g. data owner) Service Provider Resource (e.g. satellite data)

24 Summary We learned a lot through Grid experiments. Migrating from Grid to Cloud Virtualization technologies is useful for making distributed infrastructure easy to use. Better for business use. Still have many research issues. Data Network virtualization Resource managements Security Making it routine-use

25 25 Thank you very much for your attention! Global Earth Observation Grid