Grid Computing: A Ten Years Look Back. María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es

Transcription

1 Grid Computing: A Ten Years Look Back María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es

2 Outline Challenges not yet solved in computing The parents of grid Computing What the hell is grid computing? Grid computing nowadays and in the near future References 2

3 Challenges not yet solved in computing 3

4 Grand Challenge Applications Aerospace: Life sciences: E-commerce: Earth sciences: Biology: 4

5 Possible Scenarios A biochemist exploits 10,000 computers to screen 100,000 compounds in an hour 1,000 physicists worldwide pool resources for petaop analyses of petabytes of data Civil engineers collaborate to design, execute, & analyze shake table experiments Climate scientists visualize, annotate, & analyze terabyte simulation datasets An emergency response team couples real time data, weather model, population data Source: Slides The Challenges of Grid Computing Ian Foster 5

6 Means of Solving the Problem Cluster Computing Intranet Computing Grid Computing

7 The Grid Scenario Flexible, secure, coordinated resource sharing among individuals and institutions Enable communities (virtual organizations) to share geographically distributed resources in order to achieve a common goal In applications which cannot be solved by resources of an only institution Or the results can be achieved faster and/or cheaper 7

8 Idiosyncrasy of the scenario Dynamic virtual organizations A set of individual and/or institutions which share rules Large or small Static or dynamic Kind of resources Heterogeneous resources Computers, storage, sensors, networks, etc. Coordinated problem solving Challenges Distribution Collaboration Trust, policies, negotiation, payment Security Authentication Authorization Resource access Resource discovery Scheduling Data management 8

9 The parents of Grid Computing 9

10 Intensive Supercomputing High Performance Computing (HPC) Solving few large problems in a small amount of time Server-based solutions: High performance processors Large amount of memory Fast networks: Low latency time High bandwidth High Throughput Computing (HTC) Solving many small problems in a small amount of time Server-based solutions: Large number of processors Limited size of memory Networks with medium-high latency time

11 Drawbacks of this solution Lack of scalability Expensive equipments Expensive maintenance Waste of resources No reliability

12 Cluster Computing A collection of interconnected computers Used as a single, unified computing resource Tightly coupled computers Centralized job management and scheduling system Parallel programming model MPI, PVM Advanced network based on Fast Ethernet or Myrinet Good balance: costeffectiveness 12

13 Drawbacks of this solution Communications overhead Maintenance Wasted resources 13

14 Intranet Computing Use of resources of a department network Use of a management tool for running sequencial or parallel jobs Advantages: More computational resources More low-cost CPU cycles Higher scalability More reliability Simplify the administration and management Examples: Condor (University of Wisconsin) Sun Grid Engine (Sun Microsystems) LSF (Platform Computing) 14

15 Drawbacks of this solution Not use of resources out of the domain of administration Basic protocols and interfaces not based on open standards Resource to be managed: CPU. What s happen if we need to share data? 15

16 Current problem Moore's law: metaphor for exponential growth in the performance of IT hardware The capacity storage doubles every 12 months The network bandwidth doubles every 9 months The processor efficiency doubles every 18 months Difficult for just one system to analyse the stored data Difficult for a center to analyse the volume of generated information Solution: Share resources with other organizations -> Grid computing 16

17 What the hell is Grid Computing? 17

18 Grid Computing Grid Computing is based on the philosophy of information and electricity sharing, allowing us to access to another kind of heterogeneous and geographically separated resources Grid provides the sharing of: Computational resources Storage elements Specific applications Equipment Other Thus, Grid is based on: Internet protocols Ideas of parallel and distributed computing 18

19 Grid technology Grid technology is an important part of several researching areas because it provides computational and storage support to applications that needs a great computational capacity and analyses a great amount of data 19

20 What is Grid Computing? Sharing characteristics: uniformity transparency reliability security efficiency Different domains of administration, enabling their internal security policies and their intranet resources management software Virtual organizations 20

21 A Three Point Checklist A Grid is a system that... 1)...coordinates resources that are not subject to a centralized control... 2)...using standard, open, general-purpose protocols and interfaces... 3)...to deliver nontrivial qualities of services. Ian Foster What is the Grid? A Three Point Checklist (2002) 21

22 Usual Grids Grid Computing is used in applications with the following characteristics: Have a community of distributed users Need a great computational power Need a great storage capacity Above all, it is used in researching areas which face up to complex problems and store numerous data High Energy Physics Earth Observation Bio-medicine 22

23 Grid Architecture Application High level Middleware EDG, Crossgrid Low level Middleware Globus, Unicore, Legion Operating systems Unix, Linux, Windows Hardware Source: Grid Café: What is the Grid? 23

24 Grid architecture Application Coordination of several resources Resources sharing and access negotiation Communication by means of Internet protocols and security Local resources access and control Collective Resource Connectivity Fabric Application Transport Network Link 24

25 Elements Resource providers Publish the availability of their resources by means of information systems Define their own security policies Broker Register and categorize the published services providing collective search Requesters Use brokering services to find and use resources 25

26 Example Resources providers Broker Requester Communication 26

27 Basic pillars 27

28 Need of security Distributed resources No centralized control Different resource providers Each resource provider uses different security policies 28

29 Security in Grid Generic Security Services (GSS) Authentication, delegation, integrity and confidentiality Public Key Infrastructure (PKI) with X.509 certificates Kerberos Secure Socket Layer (SSL) Grid Security Infrastructure (GSI) Delegation Single Sign-On Proxy certificates 29

30 Certificate request An user ask for a certificate to a Certification Authority (CA) The CA checks the user identity Then, the CA signs the request, creating a certificate, and return it to the user Certificates can be cancelled Certificate Revocation List (CRL) The aim of the certificates is described in the certificate policy (CP) 30

31 Information Systems Provide information on: The Grid itself The user may query about the status and performance of the Grid Grid applications Register and monitor resources Standardization is required to interoperate among different grids projects Globus: MDS (Monitoring and Discovery Service) European Data Grid: R-GMA (Relational Grid Monitoring Architecture) UNICORE: Incarnation Database (IDB) 31

32 Data Grid Set of storage resources and data retrieval components which allows applications to access data by means of special software mechanisms Data grid problems: Data location Replication I/O performance 32

33 Data Transfer GridFTP: Protocol to data transfer in a secure way in a grid environment Extends FTP protocol Use Grid Security Infrastructure (GSI) Several storage systems provide GridFTP interfaces: Castor EDG s SRM Reliable File Transfer (RFT): Grid Service which provides interfaces to manage and monitor file transfers by using GridFTP servers 33

34 Data replication Due to the complexity of a grid environment, the existence of file replicas could be advisable Need of identifying and locating replicas Replica Location Service (RLS): a Grid Service for registering data replicas and later discovering Mappings between logic and fisical identifiers Database for metadata 34

35 Resource management system Resource Management includes the efficient use of computing and storage resources Processor time Memory Storage Network User-transparent Interacts with the rest of Grid components 35

36 Job Execution A job can be any kind of executable that requires CPU or storage Resource manager: Resource Brokering Find suitable resources Matchmaking Assign a job to a resource that satisfies job requirements Job execution Execute the jobs and retrieve output Error management Job execution requires to find the right Computing Element 36

37 Job submission UI Data Localization Replica Catalogue In/Output Job Local storage In/Output Workload Manager Job Computing Element Status CE status Inform. Service Grid enabled data transfers/ accesses SE status Storage Element 37

38 Intensive jobs Used in parallel and distributed environments Parallel machines Clusters A Grid is understood as a set of clusters or parallel machines Possibility to execute MPI jobs MPICH-G2 LAM-MPI Matchmaking Resource broker must select nodes that have MPI installed, and at least n CPUs 38

39 Job queue managers Condor-G: Condor High-throughput computing project Portable Batch System (PBS) Sun Grid Engine (SGE) 39

40 OGSA Defined by The Global Grid Forum (now Open Grid Forum) Open Grid Services Architecture Grid Computing + Web Services Concepts of both technologies 40

41 OGSA Everything is represented as a service (a Grid service) Service: Entity able to communicate through a network and that provides functionalities by means of message interchange Computational resources, data resources, networks, programs, databases,... Why we need Grid services: Web services have several shortcomings 41

42 Web Services vs. Grid Services Web Services The service receives the request, process it and returns the answer Usually stateless Grid Services Stateful services OGSA (Open Grid Service Architecture): standard for all the services of a grid application [Phisiology of the Grid, 42

43 mix-and-match Object-oriented Grid Computing approaches Internet-WWW Problem Solving Approach Web-based technologies Market/Computational Economy 43

44 Grid Projects Globus: EGEE: TeraGrid: CrossGrid: EU-DataGrid: IrisGrid: mygrid: 44

45 Grid Computing nowadays and in the near future 45

46 Cloud Computing Future scenario: No computing on local computers Third-party compute and storage facilities It is not a new idea: John McCarthy, 1961: computation may someday be organized as a public utility Cloud Computing: A large-scale distribured computing paradigm that is driven by economies of scale, in which a pool of abstracted, virtualized, dynamically-scalable, managed computing power, storage, platforms, and services are delivered on demand to external customers over the Internet * Is it just a fashion name for grid computing? 46

47 Grid Computing vs Cloud Computing Business model: Grid computing: project-oriented, in which it is possible to spend an amount of service units, generally CPU hours Example: TeraGrid, proposals for the increasement of computational power Cloud computing: customers pay providers on a consumption basis (such as electricity) Example: EC2 from Amazon (instance-hour consumed), S3 from Amazon (GB-Month of storage) 47

48 Grid Computing vs Cloud Computing Architecture: Grid Computing Application Cloud Computing Application Collective Platform Resource Connectivity Unified Resource Fabric Fabric 48

49 Grid Computing vs Cloud Computing Resource management: Grid computing: Batch-based computing model: Use of LRM (local resource managers), such as Condor, PGS or Sun Grid Engine. Data model: location transparency, use of a distributed metadata catalog. Data storage usually depends on a shared file system (PVFS, Lustre). Virtualization is not so important, although there are some initiatives Widely use of Ganglia as monitoring system Cloud computing: Computing model: Resources in the cloud shared by all users. More number of users. Data model: Google s MapReduce system running on top of the Google File system (Replicated chunks of data) Virtualization is key in Cloud Computing Difficult to obtain a high level of detail in monitoring. In the future, clouds will be self-maintained. 49

50 Grid Computing vs Cloud Computing Programming model: Grid computing: MPICH-G2 GridRPC Workflow systems WSRF Cloud computing: MapReduce model: Map : Applying a specific operation to a set of items, obtaining a new set of items Reduce : Aggregation on a set of items Hadoop: Open source implementation of the MapReduce model Scripting (Java Script, PHP, Python) 50

51 Grid Computing vs Cloud Computing Security model: Grid computing: GSI Cloud computing: Simpler model and less secure Use of SSL and Web forms A challenge not solved in clouds 51

52 Grid Computing vs Cloud Computing They share many goals They are different in many aspects But, they are complementary [Cloud computing] is indeed evolved out of Grid Computing and relies on Grid Computing as its backbone and infrastructure support. * 52

53 References 53

54 References The Anatomy of the Grid: Enabling Scalable Virtual Organizations. I. Foster, C. Kesselman, S. Tuecke, International J. Supercomputer Applications, 15(3), The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. I. Foster, C. Kesselman, J. Nick, S. Tuecke, Open Grid Service Infrastructure WG, Global Grid Forum, June 22, 2002 Performance Tools for the Grid: State of the Art and Future. M. Gerndt, R. Wismueller, Z. Balaton, et al. LRR-TUM Research Report Series. January A Performance Study of Monitoring and Information Services for Distributed Systems. X. Zhang, J. Freschl, and J. Schopf. Proceedings of High Performance Distributed Computing (HPDC 03), August

55 References The Globus Project, Open Grid Forum, Modeling Stateful Resources with Web Services, The WS-Resource Framework, [*] "Cloud Computing and Grid Computing 360- Degree Compared," I. Foster, Y. Zhao, I. Raicu, S. Lu, Grid Computing Environments Workshop, GCE '08, vol., no., pp.1-10, Nov

56 To know more Gridipedia, Grid Café, GridTalk, 56

57 Grid Computing: A Ten Years Look Back María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es