Use cases for Cloud Vasileios A. Baousis (Ph.D), Ricardo Correa Network Applications Team

Transcription

1 Use cases for Cloud Vasileios A. Baousis (Ph.D), Ricardo Correa Network Applications Team Slide 1

2 Agenda Definition of Cloud Computing (CC) Requirements and risks of ECMWF CC use case at ECMWF - Public Cloud deployment data flow - Private Cloud deployment data flow CC activities at ECMWF JASMIN project About JASMIN Facility Communication outline Integration with ECMWF software (ECPDS & ECACCESS) FMI CloudSigma case SDS (WebOps Team) activities Questions Slide 2

3 Cloud Computing definition is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models. Slide 3

4 Requirements Real-time access to large data sets Requirement Issue-Approach-Consideration High Disk I/O performance NAS/SAN device? Local disk? Magnetic or SSD? Processes from different users should not impact each other Platform should be able to run multiple Operating Systems Users should be able to instantiate/manage their own virtual machines Shared computing resources Avoid contention User choice? Software requirements and/or conflicts Submit their own VMs? BOYVM? Use VM image from catalogue? Cost to manage on behalf of users is prohibitive Data sets should be accessible at or soon after schedule time 24x7 support and Highly Available Network bandwidth (Internet) to user s host should be enough for purpose Analytical tools needed for capacity planning and optimization Member States should be able to offer the service to third parties. Regardless of size MARS SLAs Without compromising Operations Slide 4

5 Requirements for Legal Issues Requirement Privacy Virtual images Catalogue Issue-Approach-Consideration User data should be kept confidential. Shall contain only open source software? Third party applications on VMs Should be installed by the users. For Public Cloud based solutions Data, images and applications binaries should be physically located in Europe. Users to contract their VMs directly with Public Cloud Provider Slide 5

6 Summary of Requirements ECGATE and/or HPCF can not be enhanced to cover basic requirements. Private Cloud at ECMWF - Pros: Closer to the data - Cons: Investment, staff, Security, legal Public Cloud Provider (Commercial) - Pros: Flexibility, Security (for ECMWF operations), legal - Cons: Farther from the data Recommended Actions - A. Remain calm. - B. Start slow with the Launching of Pilot projects for a number of use cases. Slide 6

7 Cloud computing case (1/2) Reason The global weather model (IFS) and related software produces millions of products on a daily basis. The products generated are highly customized to the individual requirements of our users and are transferred to their home sites either via private networks or via the Internet. The number of products generated is not only high in number but also some of the products have very large volumes in the order of terabytes of data. The transfer of these very large data-sets to our customer s home sites is not always feasible specially when having strict time delivery constraints for 24x7 operations. Slide 7

8 Cloud computing case (2/2) A solution To run the user processes on computing resources close to where the datasets are stored. To implement this solution we have two options, A. Build a Private Cloud in our data centre and offer the resources to external users from other Organizations B. Establish a partnership with a Public Cloud Service Provider (CSP) connected to ECMWF with a high bandwidth network link (GEANT) such that the data is stored directly on the Cloud. Benefit(s) External users from other Organizations could then use Cloud computing resources from the same provider to post-process the data and generate their own output and products. Overcome existing limitations such as software installation and licensing into ECMWF infrastructure. Slide 8

9 Current - JASMIN project - FMI CloudSigma case - SDS (WebOps Team) activities Slide 9

10 About JASMIN Facility A "super-data-cluster" which delivers infrastructure for data analysis. Half super-computer and half data-centre. - Funded by the Natural Environment Research Council (NERC) & the UK Space Agency (UKSA) - Delivered by the Science and Technology Facilities Council (STFC). Provides compute and storage linked together by a high bandwidth network in a unique topology. Provides four basic services to the community: - Storage (including disk and tape) - Batch Computing, - Hosted Computing, & - Cloud Computing. Slide 10

11 Unmanaged Cloud Jasmin Managed Cloud Dedicated VM for data pre post processing dissftp Ecaccess gateway ECPDS destination JAS spacewalk Puppet X ecmwf-app.ceda.ac.uk ecmwf-svc1.ceda.ac.uk Application front-end Vcloud director jasmin-login1.ceda.ac.uk jasmin-xfer1.ceda.ac.uk jasmin-sci1.ceda.ac.uk Access from anywhere app.ecmwf.int ECMWF Slide 11 ECPDS -mspds ECACCESS ectrans

12 Example transferring a file though ecpds/ectrans: ecgb11:--> /usr/local/bin/mspds -destination JAS -source test.tar -target test3.tar 1 DataTransfer(s) will start at: Jun 10 10:01:45 (DatafileId= ) /usr/local/bin/ectrans -gateway ecmwf-svc1.ceda.ac.uk -remote JASMIN1 -source test.tar Slide 12

13 JAS destination Slide 13

14 HPC ECMWF Cloud systems Post processing systems from Member states Data flow Public Cloud deployment Member states &customers ECPDS ECMWF Products to ECPDS Scheduled Time Dissemination Time Slide 14

15 HPC ECMWF ECMWF Post processing systems from Member states Data flow Private Cloud deployment ECPDS Products to ECPDS Scheduled Time Slide 15 Member states &customers Dissemination Time

16 HPC FMI-CloudSigma case CloudSigma CSP/Zurich DC ECMWF Cloud systems FMI VM Member states &customers Products to ECPDS ECPDS ECMWF Scheduled Time Dissemination Time Slide 16

17 SDS (WebOps Team) activities Ultimate goal : Deploy application stacks to several/different cloud platforms. Current activities: Migrating to a Configuration Management System. Automating the processes to create, provision and dispose infrastructure. Working prototype for the corporate web site ( in a development (local) environment. Slide 17

18 Use cases for Cloud Vasileios A. Baousis (Ph.D), Ricardo Correa Network Applications Team Slide 18