HPC and Grid Concepts

Transcription

1 HPC and Grid Concepts Divya MG CDAC Knowledge Park, Bangalore 16 th Feb 2012 Ahmedabad 1

2 Presentation Overview What is HPC Need for HPC HPC Tools Grid Concepts GARUDA Overview Grid Middleware 16 th Feb 2012 Ahmedabad 2

3 What is HPC "High-Performance Computing," or HPC, is the application of "supercomputers" to computational problems that are either too large for standard computers or would take too long. A HPC system, is essentially a network of nodes, each of which contains one or more processing chips, as well as its own memory. These nodes are Symmetric Multi- Processors (SMP) units High-performance computing (HPC) uses parallel processing for running advanced application programs efficiently, reliably and quickly. 16 th Feb 2012 GBC@PRL Ahmedabad 3

4 HPC Architecture Cluster services Portal server Job Scheduler (LRM) SMP box/node having many CPUs 16th Feb 2012 Ahmedabad 4

5 Performance of HPC Clusters Performance of HPC is measured in terms of FLOPS ( Floating-point Operations per Second). 1FLOP = Number of Instructions per Cycle X Clock Speed in Hertz X Number of CPUs HPC term applies especially to systems that function above a teraflop or FLOPS 16 th Feb 2012 GBC@PRL Ahmedabad 5

6 Paradigms of Parallel Programming on HPC Types of Applications PThreads, OpenMP on SMP/Nodes MPI/PVM on Cluster i.e across nodes 16 th Feb 2012 Ahmedabad 6

7 Need for HPC Applications Bio Chemistry: Protein Folding Chemistry: Materials Research Physics: Simulation of Heavenly bodies Environment Modeling: Earthquake, Weather Industry: Product Prototyping, Models to Calculate Risk, Processing seismographic data 16 th Feb 2012 Ahmedabad 7

8 C-DAC s HPCC Software Products C-MPI : Optimized implementation of MPI for Cluster of Multi Processors (CLUMPS). Both point-to-point and collective calls have been optimized. Effectively uses both shared and distributed memory of CLUMPS. C-PFS : Parallel File System Provides MPI-IO file system interface to parallel applications F90IDE : Integrated Development Environment for Fortran 77/90 that includes compiler, debugger, profiler, source code browser and Fortran 77 to F90 convertor. PCF90: An automatic parallelizing compiler for Fortran for SMP based architecture. DIViA: Parallel program correctness and performance debugger. Detects communication bottlenecks and supports message debugging. PARMON: Cluster monitoring tool. Monitors the cluster as a unified resource. Provides Web Interface for monitoring over internet. RMS: Resource Management Software for effective load balancing and Load Scheduling on clusters. 16 th Feb 2012 GBC@PRL Ahmedabad 8

9 Grid Concepts 16 th Feb 2012 Ahmedabad 9

10 What is Grid computing Definition of Grid by Ian Foster and Carl Kesselman A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities. Three-point Check list given by Ian Foster,, Father of Grid Computing coordinates resources that are not subject to centralized control using standard, open, general-purpose protocols and interfaces to deliver nontrivial qualities of service. Reference: Book on The Grid: Blueprint for a New Computing Infrastructure. Grid puts one abstraction layer above High Performance Computing System(s), facilitating co-ordinated and distributed sharing of resources across multiple geographical and administrative domains. 16 th Feb 2012 GBC@PRL Ahmedabad 10

11 Origin of Grid computing Vision: The central idea is that computing should be as reliable, pervasive and transparent as a utility Information or computation power should be delivered on demand. (Apart from type and location) Origin: Conceived by academic and research communities. Internet computing grew from communication needs and GC originated from the needs of the scientific community. Create a dynamic computing environment for sharing resources and results Scale to accommodate petabytes of data, and teraflops of computing power, and keep costs down 16 th Feb 2012 GBC@PRL Ahmedabad 11

12 Popular Grids 16th Feb 2012 Ahmedabad 12

13 What is Grid architecture? Grid Architecture can be described as the layers of building blocks, where each layer has a specific function, to accomplish Grid Computing Infrastructure. 16 th Feb 2012 GBC@PRL Ahmedabad 13

14 Components of Grid Nagios, Monalisa Ganglia,GridICE Bioinformatics,Disaster management Applications Grid Portals, PSE s, API s Access Methods Debuggers, Compiler IDE, Profilers, Workflow tools SRB/SRM, Visualization, softwares Globus/ Glite/ Unicore, Torque/Loadleveler/SGE, Gridway/MOAB/Condor-G, GSI, Grid services, MPICH & MPICH-G2 Program Development Environment Storage & Visualization Scheduler, Middleware & Security Monitoring & Management CPU, Satellite terminals, Telescopes Computational Resources & devices GARUDA, NKN and Internet links Network/Communication Fabric 16 th Feb 2012 GBC@PRL Ahmedabad 14

15 GARUDA Overview 16 th Feb 2012 Ahmedabad 15

16 Objectives Motivation: To Collaborate on Research and Engineering of Technologies, Architectures, Standards and Applications in Grid Computing. Garuda Grid is targeted at providing a facility for the scientific community, which would enable them to seamlessly access the distributed resources. 16 th Feb 2012 GBC@PRL Ahmedabad 16

17 Connectivity Summary 16 th Feb 2012 Ahmedabad 17

18 GARUDA Backbone A ccess Terminal Submit No de/ Grid Head node Generic User Internet Access Access Terminal LAN Port Submit No de/ Grid Head node Access Termi nal Swit ch Local User C-DAC, Bangalore Swit ch Head Node Port Head No de Hea d Node Head No de Database LAN M P L S Access Local User Comp ut e Nodes Head No de Head Node Database Head No de Local User C-DAC, Pune LAN Port A ccess Terminal Head No de Port Swit ch Hea d Node Switch Partner without resources Teles cope Head No de Grid Hea d node Database Storage A ccess Terminals 16th Feb 2012 Partner with resources GBC@PRL Ahmedabad 18

19 National Knowledge Network NKN 16 th Feb 2012 Ahmedabad 19

20 GARUDA Grid: Architecture Grid-Enabled Applications Resource Enabler & Monitoring CLI Federated Information Server Access Portal Workflow tool Grid PSE Visualization Job Scheduler WSRF+GT4 + other Services +Cloud S/W Virtualization support Grid Programming & Development Environment Grid Security and High-Performance Grid Networking Data Grid NKN CDAC Resource centers Non-Research Organizations Research Organizations Computing Resources and Virtual Organizations Educational institutions Computing Centers Resources Security Middleware Resource Management User Environments Programming Environments Data Grid Grid Applications 16 th Feb 2012 GBC@PRL Ahmedabad 20

21 GARUDA S/W Architecture Management, Monitoring & Accounting Paryaveekshanam GARUDA Information Service Ganglia Nagios GARUDA Accounting GARUDA Resources Compute, Data, Storage, Scientific Instruments, Application Specific Software,.. Resource Mgmt & Scheduling GridWay Meta-scheduler Resource Reservation Torque, Load Leveler Globus 4.x (WS Components) Access Methods Access Portal Problem Solving Environments Cmd line interface Visuvalization gateways workflows Security Framework IGCA Certificates MyProxy VOMS 16 th Feb 2012 Ahmedabad 21

22 GARUDA : Garuda Partner Resources Institutions Space Application Centre Indian Institute of Science Raman Research Institute Institute of Mathematical Sciences Madras Institute of Technology Indian Institute of Technology Jawaharlal Nehru University Location Ahmedabad Bangalore Bangalore Chennai Chennai Delhi Delhi Institute of Genomics and Integrative Biology Indian Institute of Technology University of Hyderabad Indian Institute of Technology Physical Research Laboratory Institute of Microbial Technology University of Pune Delhi Guwahati Hyderabad Kharagapur Ahmedabad Chandigarh Pune 16 th Feb 2012 GBC@PRL Ahmedabad 22

23 Grid Middleware 16 th Feb 2012 Ahmedabad 23

24 What is a Grid Middleware Grid Middleware is a layer of S/W to enforce all the properties of: Scalability, Transparency, Heterogeneity, Fault Tolerance and Security of the Grid below the application layers. It provides an uniform interface of the Grid to users and handles all the complexity generated due to heterogeneous systems. Middleware S/W is a layer between grid applications and low level functionality of grid. 16 th Feb 2012 GBC@PRL Ahmedabad 24

25 Popular Grid Middlewares GT - Globus Toolkit (Argonne National Laboratory, Chicago) GRIA Grid Resources for Industrial Application (University of Southampton ) Moab Grid Suite Cluster Resources Inc. Condor Project grid computing engine by Univ. of Wisconsin Gridbus Univ. of Melbourne, for e-business and e-science Unicore Uniform Interface to Computing resources - Univ. of Virginia, for NPACI (National Partnership for Advanced Computational Infrastructure.edu) NorduGrid Middleware ARC (Advanced Resource Connector) Legion Univ. of Virginia Glite - LightWeight Middleware for Grid computing( EGEE) 16 th Feb 2012 GBC@PRL Ahmedabad 25

26 Core functionalities of Grid Middleware 16 th Feb 2012 Ahmedabad 26

27 Middleware functionalities Security Job Management Data Management Information Management 16 th Feb 2012 Ahmedabad 27

28 Grid Middleware: Security User/ Resource Authentication User/ Resource Authorization Access Control Lists User and Resource Policies Virtual Organization 16 th Feb 2012 Ahmedabad 28

29 Grid Middleware: Job Management Submission, Status Query, Cancel & Destroy, Getting Output & Error Support an open Job Description Language RSL, JDL, JSDL Transferring input/output data from/to remote source/destination Support Serial/ Parallel Jobs (Heterogeneous & Homogeneous) Integration with all Local Resource Managers 16 th Feb 2012 Ahmedabad 29

30 Grid Middleware: Data Management Two Basic Categories of Data Management Data Movement Secure Robust Efficient Third party movement Data Replication One or more copies or replicas Survive loss Easy availability Reduce access latency, increase robustness, scalability and performance for distributed applications. 16 th Feb 2012 Ahmedabad 30

31 Grid Middleware: Information Management System information is critical to operation of the grid and construction of applications How does an application determine what resources are available? What is the state of the computational grid? How can we optimize an application based on configuration of the underlying system? We need a general information infrastructure to answer these questions 16 th Feb 2012 GBC@PRL Ahmedabad 31

32 Examples of Useful Information Characteristics of a compute resource IP address, software available, hostname, nodes available, OS version, load, softwares, libraries & their licenses. Characteristics of a network Bandwidth and latency, protocols Characteristics of the Middleware infrastructure Hosts, local resource managers 16 th Feb 2012 GBC@PRL Ahmedabad 32

33 Thank You 16 th Feb 2012 Ahmedabad 33