E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS

Size: px

Start display at page:

Download "E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS"

Henry Ferguson
8 years ago
Views:

1 E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS Peter Kacsuk, MTA-SZTAKI, Peter Stefan, NIIFI, Imre Szeberenyi, BME,

Peter Kacsuk, MTA-SZTAKI, kacsuk@sztaki.

2 Overview Background Infrastructure in a nutshell Scientific computing on desktop grids Scientific computing on clouds NIIFI Cloud Summary e-irg Workshop, Budapest, 4-6 April

3 Background Message: how to serve scientific computing with compute and data storage infrastructure in more efficient ways? Scientific computing is the quintessence of IT: it is extensive, ranges from low level hw to higher level applications; it is present in the history of IT since the very beginning (Neumann solving differential equations); its purpose is noble: find answers to scientific questions (cancer research, climate modeling, car industry ). Our purpose: to give suggestions on the answers and to consider about future trends in this field. e-irg Workshop, Budapest, 4-6 April

of IT since the very beginning (Neumann solving differential equations); its purpose is noble: find answers to scientific questions (cancer research,

4 Infrastructure in a nutshell Single computers Parallel computers, vector computers Mainframes, metacomputers, geographically centered clusters, Data/resource centers Supercomputers Grids Clouds Services All are about serving the science! e-irg Workshop, Budapest, 4-6 April

clusters, Data/resource centers Supercomputers Grids Clouds Services

5 Scientific Computing on Desktops e-irg Workshop, Budapest, 4-6 April

6 Why volunteer desktop grids are important? Volunteer desktop grids can collect very large number of resources e-irg Workshop, Budapest, 4-6 April

7 Potential of volunteer computing Number of privately owned PCs: currently 1.5 billion grow to 2 billion by 2015 ~1 million computers are actively participating in volunteer computing supplying ~10PetaFLOPS of computing power Fastest supercomputers supply few PetaFLOPS Largest grids (e.g. EGI) have several hundreds of thousands of hosts. In the area of so-called bag of tasks, parameter sweep applications volunteer computing is competitive. Near-term potential of volunteer computing goes well beyond Exa-scale. e-irg Workshop, Budapest, 4-6 April

computing power Fastest supercomputers supply few PetaFLOPS Largest grids (e.g. EGI) have several hundreds of thousands of hosts.

8 Types of Desktop Grids Global (volunteer) Desktop Grid Aim is to collect resources for grand-challenge scientific problems. Examples: Community World Grid, IberCivis, SZTAKI Desktop Grid. Local (institutional) Desktop Grid Aim is to enable the quick, easy and inexpensive creation of grid for any community (company, university, etc.) to solve their own applications. Example: SZTAKI Desktop Grid (SZDG) local version (used within EDGeS, EDGI, DEGISCO). e-irg Workshop, Budapest, 4-6 April

Local (institutional) Desktop Grid Aim is to enable the quick, easy and inexpensive creation of grid for any community (company,

9 Local DGs in practice University of Westminster DG system based on SZDG 6 1. New Cavendish Street 576 nodes 2. Marylebone Campus 559 nodes 3. Regent Street 395 nodes 4. Wells Street 31 nodes 5. Little Tichfield Street 66 nodes 6. Harrow Campus 254 nodes Total: 1881 nodes 2 e-irg Workshop, Budapest, 4-6 April Lifecycle of a node: 1. PCs basically used by students/staff 2. If unused, switch to Desktop Grid mode 3. No more work from DG server -> shutdown (green solution)

Harrow Campus 254 nodes Total: 1881 nodes 2 e-irg Workshop, Budapest, 4-6 April 2011 9 3 1 5 4 Lifecycle of a node: 1.

10 The EDGI Infrastructure Extends EMI/EGI (glite, ARC, Unicore) grids with DGs Extend Desktop Grids with Clouds for QoS Supports both compute and data intensive applications e-irg Workshop, Budapest, 4-6 April

Clouds for QoS Supports both compute and data

11 Hungarian project to integrate the advantages of Web2 and DGs e-irg Workshop, Budapest, 4-6 April

12 Desktop grids for escience - A Road map Helps organisations in deciding on setting up a Desktop Grid. Linking it into escience. infrastructures: EGI, Clouds, Grids. Advice on all levels from technical to legal and political. e-irg Workshop, Budapest, 4-6 April

13 Join to the volunteer computing community today and offer your notebook for volunteer computing! e-irg Workshop, Budapest, 4-6 April 2011 Source: 13

14 Scientific Computing on Clouds e-irg Workshop, Budapest, 4-6 April

15 Scientific Computing on Clouds Cloud is invented by the business area. Main goal is: selling everything as services Clouds still developing SaaS (Google Docs, app. portals); PaaS (web services, grid, databases, HA clusters); IaaS (virtualized resources). Furthermore the three above are not layers! In SC we tend to take the IaaS definition. European initiative: Stratuslab. e-irg Workshop, Budapest, 4-6 April

portals); PaaS (web services, grid, databases, HA clusters); IaaS (virtualized resources).

16 IaaS Benefits in Scientific Computing IaaS = decoupling from hw + managing OS images. Debate over the benefits here: pure hw services vs. virtualized resources. There are some obvious: Better utilization of hw resources, more even spread of load as well as functionalities; Lower price, lower sustainability costs, lower TCO; Gradual scalability; User ownership and involvement via better sandboxing; Better customization in the sandbox. NIIF Cloud. e-irg Workshop, Budapest, 4-6 April

There are some obvious: Better utilization of hw resources, more even spread of load as well as functionalities; Lower

17 NIIF Cloud Designed to be a private cloud that can be transformed to public. Mostly for NIIFI services. Builds upon OpenNebula + KVM + libvirt. Significant extensions. OS images are stored on storage infrastructure and are served over iscsi protocol. Geographically redundant layout: Servers are located in provincial regional centers; Interconnected by high quality data network. e-irg Workshop, Budapest, 4-6 April

OS images are stored on storage infrastructure and are served over iscsi protocol.

18 NIIF Cloud Architecture e-irg Workshop, Budapest, 4-6 April

19 NIIF Cloud Features Virtual machines and a set of virtual machines interconnected by virtual networks. L2 connectivity and L2 separation of networks. Live migration of virtual machines between sites (machines, networks, images). Guaranteed CPU performance for virtual machines. Web based console. Grid and cluster software stack over the virtual clusters (ARC+SGE). e-irg Workshop, Budapest, 4-6 April

Live migration of virtual machines between sites (machines, networks, images).

20 Future Cloud Trends The purpose of IaaS and scientific computing is different. Yet, they can coexist on the infrastructure. From the IaaS viewpoint: grid is a powerful tool connecting geographically distant sites and allow a specific service on it. From the grid viewpoint: IaaS cloud pushes into the grid software stack below the operating system level, and facilitates easy job migration, better resource utilization and better computing environment customization (support for multiple operating systems). e-irg Workshop, Budapest, 4-6 April

From the grid viewpoint: IaaS cloud pushes into the grid software stack below the operating system level, and facilitates easy job

21 Summary Infrastructure solutions for serving scientific computing has been presented: Desktop PC integration via SZTAKI DG; Virtualization via NIIF Cloud; Future of grids and clouds. e-irg Workshop, Budapest, 4-6 April

22 Thanks & Questions e-irg Workshop, Budapest, 4-6 April

Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing

Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing Dr. Robert Lovas Deputy head of Laboratory of Parallel and Distributed Systems Coordinator of EU FP7 e-infrastructure