The EGI pan-european Federation of Clouds

Transcription

1 The EGI pan-european Federation of Clouds CGW12 Cracow, Oct 2012 Matteo Turilli Senior Research Associate Chair EGI Federated Clouds Task Force Oxford e-research Centre University of Oxford 1

2 Outline EGI, cloud computing and use cases. TF objectives, deliverables, mandate and membership. Federation model and capabilities. Federation test bed and test bed demos. Blueprint document, joining procedure and know how. Conclusions.

3 EGI, New Challenges and Cloud Computing Challenges Need for long running services, not only jobs. Workflows that integrate local and remote systems. Integrating community-specific resources (sensors, antennas, repositories,...). Flexible, efficient, maintainable deployment of computing infrastructures. Users and cloud computing Resource providers: (production) infrastructure managers. Platform/gateways developers: applied research and development groups. End users: researchers in multiple disciplines, user communities, teaching staff.

4 Use Cases Structural biology We-NMR 1. Validating biomolecular NMR structures. VM images pre-packaged with VirtualCing and ToPoS. 2. Training. VM images pre-packaged with Gromacs. Musicology Peachnote 1. Music score processing. VM image pre-packaged with processing software (convert images into music XML) and communicating with Amazon's SQS and HBase. 2. Hadoop and Hbase. Deploy H+HBase clusters on multiple providers.

5 Use Cases Linguistics CLARIN British National Corpus service (BNCWeb). VM image pre-package with the BNC and a analytics software. Distribution. Teaching. Development. Federation. Ecology BioVel remote hosting of OpenModeller service. Space science ASTRA-GAIA data integration with scalable workflows.

6 Use Cases Software Engineering SCI-BUS Gateway services for user communities (guse/ws-pgrade portal). Adapting tools for interoperability among grids environments to multiple/federated clouds. Software Engineering DIRAC framework for building ready to use distributed computing systems. Already ported to EC2 and other proprietary cloud management interfaces, use the federation as yet another back end.

7 IaaS EGI.eu Coordination Core software and support EGI and Cloud Computing Personalised environments for individual research communities in the European Research Area. Community Services Community Services Globus dcache glite ARC Globus UNICORE Community Platform VM Mgmt Data Image Sharing VM Mgmt Data Image Sharing VM Mgmt Data Image Sharing NGI NGI NGI Commercial Monitoring Accounting Notification Monitoring Accounting Notification Monitoring Accounting Notification Monitoring Accounting Notification EGI-wide message bus

8 TF Objectives and Deliverables Engagement: identify and work with resources providers, technology providers, and user communities. Integration: integration of cloud resources within EGI s production infrastructure e.g. cloud computing is not meant to replace grid computing. Recommendations: identify issues that need to be addressed by other areas of EGI e.g. policies, operations, support and dissemination. Blueprint document: advice/full documentation to resource providers/users on how to engage with the federated virtualised environment. A living document on the EGI Wiki. Test bed: implement interfaces and services for a federated cloud on the basis of the Task Force blueprint and the available standards and technologies.

9 Task Force Mandate and Organisation Mandate: 18 months, September 2011 March Activities: 3 blocks of 6 months each. 1. Setup: Sep 2011 Mar Engagement of resource and technology providers. Federation model. Draft of the Blueprint document and demo. 2. Consolidation: Mar 2012 Sep Engagement of user communities. Test bed and first use case. Draft of the Blueprint document and demo. 3. Integration: Sep 2012 Mar Test bed and early adopters. Publication of the Blueprint document and demo.

10 Task Force Members and Technologies Utrecht SARA TUD KTH DANTE EGI.eu GWDG TCD STFC OerC SixSq CNRS FCTSG BSC IFAE Members 66 individuals. 26 institutions. 15 countries. Stakeholders 17 Resource Providers. 7 Technology Providers. 6 User Communities. 4 Liaisons. Technologies 8 OpenNebula. 3 StratusLab. 4 OpenStack. 1 Okeanos. 1 WNoDeS. FZJ CESNET Cyfronet Masaryk LMU SRCE INFN GRNET

11 Federation Model Federation models/implementations ~Sep 2011 No open implementations available. Some on-going research on clouds federation, i.e. StatusLab project. Several published models: IBM Reservoir (Rochwerger et al.); application-centric management (Harmer et al.); Aneka- Federation, InterCloud (Buyya et al.). Extension of management layer. Single type of cloud management system. Brokering. User Communities User Communities User Communities Interfaces Interfaces Interfaces Interfaces Interfaces Cloud Management Cloud Management Cloud Management Cloud Management Hardware Cloud Management Hardware Hardware Hardware Hardware Comprehensive infrastructure solutions.

12 Federation Model Cloud initiatives and landscape ~Sep 2011 Early development stage of open source solutions. Prevalence of test bed and/or pilot projects for cloud evaluation by resource providers. Multiple cloud management platform with proprietary interfaces. At best, early stage of user requirements elicitation. On-going security policy evaluation, early stage of integration with IT departments. User Communities User Communities User Communities Interfaces Interfaces Interfaces Interfaces Interfaces Cloud Management Cloud Management Cloud Management Cloud Management Hardware Cloud Management Hardware Hardware Hardware Hardware

13 Federation Model Resource integration: cloud computing to be integrated into the existing production infrastructure. Heterogeneous implementation: no mandate on the cloud technology. Provider agnosticism: no distinction among type of providers (e.g. academic, commercial). Standard-based interfaces: emerging standards for management interfaces and image container. Core services: core services already deployed within the grid infrastructure. User Communities Federated interfaces User Communities User Communities Core services Cloud Management Cloud Management Cloud Management Cloud Management Hardware Cloud Management Hardware Hardware Hardware Hardware

14 Capabilities Based on the profile documented in EGI Cloud Integration Profile, S. Newhouse, M. Drescher. Minimal set of capabilities to satisfy the given use cases. Evaluation, integration, deployment but no development effort allocated. Infrastructure VM Management Data Management Information Discovery Community Federated AAI VM Image Sharing Clients and Brokering Accounting Monitoring Notification

15 Federation Test bed Sep 2012 Composed of 4 services, 2 management interfaces, 9 cloud infrastructures operated by 7 Resource Providers. 4 more providers are in the process of being federated.

16 Core Services Information system. Each cloud infrastructure exposes a LDAP server publishing information by means of a customised GLUE2 schema. Each LDAP server is polled by a top-bdii server. ldap://test03.egi.cesga.es:2170

17 Core Services Accounting. Each cloud infrastructure generates usage records based on an extended version of the EGI UR format recommendation. Records are uploaded to a central server by means of a client customised for each type of infrastructure.

18 Core Services Monitoring. A standard Nagios installation is used to monitor the availability of the management interfaces exposed by each cloud infrastructure. Probes to test the state of the federated services are under development.

19 Core Services Marketplace. A repository were Resource Providers and EGI can publish metadata about images from which virtual machines can be instantiated. When needed, a single image can be signed and then endorsed by multiple providers.

20 Federation Demo Sep 2012 LDAP MP/UR Clients LDAP MP/UR Clients LDAP MP/UR Clients Resource Provider GWDG (ON/OS) OCCI 1.1 CDMI 1.0 Resource Provider OCCI 1.1 CESNET (ON) CDMI 1.0 Resource Provider CYFRONET (ON) CDMI 1.0 OCCI 1.1 Information GLUE 2.0 BDII Monitoring Nagios LDAP MP/UR Clients LDAP MP/UR Clients LDAP MP/UR Clients LDAP MP/UR Clients Resource Provider OCCI 1.1 KTH (ON) CDMI 1.0 Resource Provider CESGA (ON) Resource Provider Venus-C CDMI 1.0 OCCI 1.1 Venus-C CDMI 1.0 Resource Provider FZJ (OS) Resource Provider IN2P3-CC (OS) OCCI 1.1 OCCI 1.1 ON = OpenNebula. OS = OpenStack. MP = Marketplace. UR = Usage Records. VM metadata Marketplace Accounting OGF UR UR+ & StAR Message Bus

21 Blueprint Document

22 Blueprint Document Documented Knowledge OpenStack, OpenNebula installation and configuration. OCCI and CDMI. Marketplace. Nagios probes for cloud resources. GLUE2 and UR for cloud resources. Latest developments in cloud brokering and clients. User communities leveraging cloud computing. Join the federation test bed Expose an OCCI interface. Install an LDAP server with a GLUE2 schema tailored for cloud resources. Allows the Nagios probes to monitor the interfaces and services. Upload usage records to the EGI centralised repository. Publish the image metadata into the federation Marketplace. Install, if needed, a CDMI server.

23 From Task Force to Task within EGI-InSPIRE May 1 st 2012: the Task Force becomes an official Task within EGI-InSPIRE. Effort: Effort officially contributed by: IIAS, KTH, Jülich, LUH, SARA, CISC, INFN, CNRS, CESNET and OeRC. EGI Technical Outreach team: Collaboration with the EGI Technical Outreach team to coordinate and promote the support of Virtual Research Communities that need or could benefit from Cloud Computing. Community Engagement: Collaboration with the EGI Community Engagement to organise the TF community activities. 1 st EGI Federated Clouds TF PlugFest, July 12 th /13 th, Amsterdam. Set up of a track for requirement gathering dedicated to Cloud Computing.

24 Conclusions Output Adoptions of standards for VM and data management. Federation model compatible and consistent with current EGI infrastructure. Contribution to EGI user communities engagement and support. Documentation made available to the community. Interoperability across multiple cloud management platforms. Cycle #3, Sep 2012 Mar 2013: Integration Focus on dev tools for management interfaces and clients for the test bed. Integration of the test bed services into the EGI infrastructure. Cloud brokering evaluation and deployment. Focus on use cases coordination and implementation. Opening of the test bed to early adopters.

25 Thank you. Matteo Turilli Senior Research Associate Chair EGI Federated Clouds Task Force Oxford e-research Centre University of Oxford Task Force resources Mailing List: Wiki site: GitHub: Indico site: 25