Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing

Transcription

1 Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing Dr. Robert Lovas Deputy head of Laboratory of Parallel and Distributed Systems Coordinator of EU FP7 e-infrastructure project IDGF-SP International Liaison of European Grid Infrastructure/NGI_HU Secretary of Stichting International Desktop Grid Federation

2 Established in 1964 EU Centre of Excellence in IT, Computer Science and Control Basic and applied research Contract-based R&Đ&I activity mainly on complex systems, turnkey realizations Transferring up-to-date results to industry and universities About the institute Basic research Computer science Systems- and control theory Engineering and business intelligence Machine perception and humancomputer interaction Applied research and innovation Vehicles and transportation systems Production informatics and logistics Energy and sustainable development Security and surveillance Networking systems and services, cloud / distributed computing Key figures Budget MEUR/year ~30% basic funding Staff % scientific Fraunhofer Project Center W3C member ERCIM member Contributor to the European Grid Infrastructure

3 Introduction to the research lab Research division of MTA SZTAKI from 1998 Head: Peter Kacsuk, Prof. editor in chief: Journal of Grid Computing (Springer) coordinator of four e-infrastructure projects in the EU 7th Framework Programme (FP7) Deputy head: Robert Lovas, PhD coordinator of two projects in EU FP7 international liaison (European Grid Infrastructure) secretary of International Desktop Grid Federation (Dutch foundation) 25 research fellows (full/part time) Foundation member o Hungarian National Grid Initiative (NGI_HU) o International Desktop Grid Federation (IDGF) o OpenNebula (ONE) User Group, Hungary Participant and/or coordinator in European and national Grid & Cloud research, e-infrastructure, and educational projects (from 2000) By far the most successful research lab in the region in the field ICT / e-infrastructures in terms of the number of coordinated FP7 EU projects 3

4 Our major on-going projects (2014) AgroDAT.hu Grid/Crowd Computing EGI-InSPIRE aginfra ER-FLOW Science Gateways/ Workflows IDGF-SP SCI-BUS Position indicates the covered area. Dimension shows the size of the project. SZTAKI logo, where the institute is the coordinator. EU or Hungarian flags are depending on the targeted regions. SZTAKI Cloud Cloud accreditation CloudSME Cloud computing BIG DATA & AGRICULTURE 4

5 Agrodat.hu project Consortium: Main objective: Establish an agricultural knowledge centre and decision support system based on data gathered by an innovative, complex sensor system and from international open repositories Duration: Budget: 8 MEUR URL: 11/21/2014 5

6 AgroDAT.hu computing and data centre in numbers o o o o o 844 CPU Core Network o o o o o 5274 GB Memory 564 TB solid state and spinning storage CUDA Core 488 Xeon Phi Core 1 Gb copper for management network 10 Gb copper for general usage 8/16 Gb fiber channel for SAN 40 Gb Infiniband for HPC Connected to HBONE/Geant Power consumption o Max. 16 kw

7 How to build e-infrastructure? Cluster computing: in a single location (organisation) operated dedicated computers (typically multicore servers or PCs) Grid computing: separate organizations agree to share their dedicated computing resources (supercomputers, clusters, servers, PCs) Desktop grid computing: desktop PCs within an organization (such as a department or university) are used as non-dedicated computing resource (i.e. spare capacities and CPU cycles) o Volunteer or crowd computing: similar to desktop grid computing but computing resources are volunteered by the public Cloud computing: typically a third-party (company) provides access to computers on a pay-as-you-go basis (or private version)

8 International Desktop Grid Federation - Support Project Fostering interoperability and sustainability of Distributed Computing Infrastructures (HPC part) - Bridged to other DCIs - Marketing: ambassadors and citizen-scientists - Sustainability by selfmaintained resource pool from volunteers (not FP7/H2020 funds) - Roadmap available Hundreds of thousands of volunteers Dozens of applications Bridged more than 1 Million CPU hours/month IDGF-SP 07/03/

9 European Grid Infrastructure Sustainable services from EGI to VREs Slides from Gergely Sipos, Sy Holsinger EGI.eu EGI-InSPIRE RI

10 European Grid Over 35 countries Services for the sharing of IT resources Infrastructure Computers Data Applications. and beyond!! Sustainability NGI membership fees FP7-H2020 projects EDG EGEE EGI European Grid Infrastructure For European researchers, industry and their international collaborators EGI.eu EGI-InSPIRE RI

11 Enabling e-infrastructures Distributed, federated storage and compute facilities Grid and Cloud compute platforms Virtual Research Environments > 200 user research projects EGI-InSPIRE RI resource centres in 40 countries 400,000 logical CPU cores 190 PB disk, 180 PB tape > 99.6% reliability. Evaluatin model 11

12 EGI Solutions Portfolio Individual Researchers Research Communities Resource Centres High- Throughput Data Analysis Community Driven Innovation Federated Operations Federated Cloud /05/14 12 EGI-InSPIRE RI

13 High-Throughput Data Analysis Federated grid resources Batch queues Mid-term storage (file-based) With file transfer, metadata, etc. services Security services Virtual Organisation and user management Software providers glite, ARC, Unicore, dcache, EDGI (desktop grids) High level workflow and workflow tools EGI-InSPIRE RI

14 Federated operations Service Research institutes Service Service Service Domain-specific scientific services Community portals EGI offers: Service catalogue Service monitors (with test probes) Helpdesk system Accounting system Operations portals Registry Accounting Monitoring Heldesk EGI & local NGI Operation portals Infrastructure services Examples: BioVeL: Availability & correctness monitoring of Web Services EUDAT: Service catalogue for custom monitors EGI-InSPIRE RI

15 Federated Cloud Domain specific services in Virtual Machine Images EGI FedCloud interfaces Standards enable federation OCCI: VM Image management OVF: VM Image format BDII: Information system X509: Authentication APEL: Accounting (CDMI: Cloud storage) + VM image Marketplace Cloud hypervisors OS OS OS OS Cloud resources private/public academic/commercial OS Cloud hypervisor is a local choice. Eg. OpenStack OpenNebula Okeanos (OpenStack impl. in GR) EGI-InSPIRE RI

16 Federated Cloud Domain specific services in Virtual Machine Images EGI FedCloud interfaces Standards enable federation OCCI: VM Image management OVF: VM Image format BDII: Information system X509: Authentication APEL: Accounting (CDMI: Cloud storage) + VM image Marketplace Cloud hypervisors OS OS OS OS Cloud resources private/public academic/commercial OS Cloud hypervisor is a local choice. Eg. OpenStack OpenNebula Okeanos (OpenStack impl. in GR) EGI-InSPIRE RI

17 Community driven innovation Based on two pillars: Virtual Team framework for Short (>12 month) Distributed (>2 countries) Focused projects (stated goal, tasks) Contact points in the NGIs Support Operations Examples: Technology study for CTA (gateway + SSO) Support for genome analysis and protein folding VM Image Marketplace for ELIXIR EGI-InSPIRE RI

18 Service Management Use Service Oriented Approach - Create a service portfolio based on high-level strategy discussions as early as possible - Identify the customers of the service Terminology should not imply paying but who negotiations the contract, which applies to even a free at point of use service. - Decide what will/can be managed in-house and what will be outsourced - Understand what agreements need to be put in place and by whom - Link the promises made to technical capabilities Using this mentality and thought process as early as possible will make everything much simpler moving forward - EGI.eu experience in both consultancy and implementation of service management in large-scale e-infrastructure based on FitSM - EGI.eu has one of the most concentrated set of FitSM trainers See extra slides for more information EGI-InSPIRE RI

19 Access policies: Infrastructure and knowledge Free for European academics and their non-european collaborators Pay-for-use pilot for infrastructure access 29 Sites publishing pricing information 20 Grid, 9 Cloud Legal and Policy solutions emerging Business Cases being explored Helix Nebula, ESA, Cloud for Europe, Arctur, Engineering EGI Business Engagement programme Opportunities for industry and SMEs Computing; Software; Knowledge; Expertise; Big-data; Market intelligence Details are emerging Contact support@egi.eu to get involved! EGI-InSPIRE RI

20 IDGF-SP 1/10/

21 IDGF-SP 1/10/

22 IDGF-SP 1/10/

23 IDGF-SP 1/10/

24 Founding members of Actually 50+ organisations IDGF-SP 07/10/

25 Local Desktop Grid: University of Westminster (London, UK) as a best practice New Cavendish Street 576 nodes 2. Marylebone Campus 559 nodes 3. Regent Street 395 nodes 4. Wells Street 31 nodes 5. Little Tichfield Street 66 nodes 6. Harrow Campus 254 nodes Total: 1881 nodes Lifecycle of a node Green mode: 1. PCs basically used by students/staff 2. If unused, switch to Desktop Grid mode 3. No more work from DG server -> shutdown (green solution) IDGF-SP 07/10/

26 Saving GBP over 4 years IDGF-SP 07/10/

27 Russian chapter of IDGF IDGF-SP 07/10/

28 IDGF-SP 07/10/

29 Source: (Nikolay Kutovskiy / LIT JINR) IDGF-SP 07/10/

30 A best practice Kurdyumov Institute IDGF-SP 07/10/

31 Collaboration opportunities (highlights) Join the International Desktop Grid Federation (individual or institute) Sign Memorandum of Understanding between your institute/university and IDGF Learn at and contribute to IDGF events Be adaptor of recommendations, best practices and roadmaps Get support and consultancy: deployment and maintenance of new (volunteer) Desktop Grids worldwide Join the infrastructure with your BOINC project and benefit from the unused capacities of other grids, or with your glite site Application porting support for scientists Support for dissemination and outreach of results and plans Etc. IDGF-SP 07/10/

32 IDGF-SP International Desktop Grid Federation - Support Project Globe adapted from wikipedia/commons/f/fa/ Globe.svg 32

33 Framework suggested by the High-Level Group on Scientific Data 33

34 WS-PGRADE/gUSE overview Based on Liferay General purpose Workflow-oriented gateway framework Supports the development and execution of workflow-based applications Supports the fast development of domainspecific gateways by customization methodologies Most important design aspect is flexibility

35 Statistical Seismology Gateway

36 Supported e-science communities

37 guse based gateways Close to 130 deployments worldwide More than downloads from SourceForge 37

38 Flexibility of using compute infrastructures Key component: DCI Bridge Flexible management of Security: Individual users certificate (X.509, SAML, pubkey, ) Robot certificates Flexible access to various types of DCIs: Clusters (PBS, LSF, MOAB, SGE) Cluster grids (ARC, glite, GT2, GT4, GT5, UNICORE) Supercomputers (e.g. via UNICORE) Desktop grids (BOINC) Clouds (Direct Access, EGI FedCloud, CloudBroker Platform) XSEDE

39 Projects building on WS- PGRADE/gUSE Portal: SCI-BUS CloudSME VIALACTEA aginfra Workflow: SHIWA ER-Flow Desktop Grid: EDGeS EDGI IDGF(-SP) Cloud: CloudSME

40

41

42

43 Azerbaijan Hungary strategic partnership agreement 11/11/2014 Source: AZERBAIJAN STATE TELEGRAPH AGENCY

44 THANK YOU! Questions?