Introduction to Grid Computing

Transcription

1 Front cover Introduction to Grid Computing Learn grid computing basics Understand architectural considerations Create and demonstrate a grid environment Bart Jacob Michael Brown Kentaro Fukui Nihar Trivedi ibm.com/redbooks

2

3 International Technical Support Organization Introduction to Grid Computing December 2005 SG

4 Note: Before using this information, read the information in Notices on page ix. First Edition (December 2005) Copyright International Business Machines Corporation All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Notices ix Trademarks x Preface xi The team that wrote this redbook xi Become a published author xiii Comments welcome xiii Part 1. Grid fundamentals Chapter 1. What grid Computing is Chapter 2. Benefits of grid computing Exploiting under utilized resources Parallel CPU capacity Virtual resources and virtual organizations for collaboration Access to additional resources Resource balancing Reliability Management Summary Chapter 3. Grid terms and concepts Types of resources Computation Storage Communications Software and licenses Special equipment, capacities, architectures, and policies Jobs and applications Scheduling, reservation, and scavenging Grid software components Management components Distributed grid management Donor software Submission software Schedulers Communications Observation and measurement Copyright IBM Corp All rights reserved. iii

6 3.5 Intragrid and intergrid Summary Chapter 4. Grid user roles Using a grid: A user s perspective Enrolling and installing grid software Logging onto the grid Queries and submitting jobs Data configuration Monitoring progress and recovery Reserving resources Using a grid: An administrator s perspective Planning Installation Managing enrollment of donors and users Certificate authority Resource management Data sharing Summary Part 2. Grid architecture considerations Chapter 5. Standards for grid environments Overview OGSA OGSI OGSA-DAI GridFTP WSRF Web services related standards Chapter 6. Application considerations General application considerations CPU-intensive application considerations Data considerations Summary Chapter 7. Security Introduction to grid security Grid security requirements Security fundamentals Important grid security terms Symmetric key encryption Asymmetric key encryption iv Introduction to Grid Computing

7 7.1.6 The Certificate Authority Digital certificates Grid security infrastructure Getting access to the grid Grid secure communication Grid security step-by-step Grid infrastructure security Physical security Operating system security Grid and firewalls Host intrusion detection PKI security policies and procedures Certificate Authority Security controls review Summary Chapter 8. Design Building a grid architecture Solution objectives Grid architecture models Computational grid Data grid Grid topologies Intragrid Extragrid Intergrid e-utilities Phases and activities Basic methodology Recommended steps A conceptual architecture Infrastructure Summary Chapter 9. Web services resource framework Resource state management using Grid services What a Grid service is Grid services vs. Web services OGSA Grid service requirements Open Grid Services Interface (OGSI) Grid services OGSI to WSRF refactoring WSRF fundamentals What a WS-Resource is Contents v

8 9.2.2 Implied resource pattern for stateful resources WS-Resource Framework specifications WS-Resource Framework and Globus Toolkit WSRF references Summary Part 3. Creating a grid environment with the Globus Toolkit Chapter 10. Globus Toolkit 4 components Overview of Globus Toolkit Common runtime components Java WS Core C WS Core Python WS Core Security components WS authentication and authorization Pre-WS authentication and authorization Community Authorization Service (CAS) Delegation service SimpleCA MyProxy GSI-OpenSSH Data management components GridFTP Reliable File Transfer (RFT) Replica Location Service (RLS) OGSA-DAI Data Replication Service (DRS) Monitoring and Discovery Services Index service Trigger service Aggregator Framework WebMDS Execution management WS GRAM Community Scheduler Framework 4 (CSF4) Globus Teleoperations Control Protocol (GTCP) Workspace Management Service (WMS) Summary Chapter 11. Globus Toolkit 4 installation and configuration How to obtain Globus Toolkit Packages of Globus Toolkit Binary packages vi Introduction to Grid Computing

9 Source packages Grid environment Installation Installing required software for Globus Toolkit 4 installation Preparing the OS for Globus Toolkit 4 installation Installing Globus Toolkit Configuration and testing of grid environment Configuring environmental variables Security set up Configuration of Java WS Core Configuration and testing of GridFTP Configuration and testing of RFT Configuration and testing of WS GRAM Testing of MDS Uninstallation Summary Part 4. Grid demonstration application Chapter 12. Demonstration application RenderClient The Graphical User Interface (GUI) RenderClient source code RenderWorker RenderSourceService Alternative architecture DirectoryTree of important files in demo Part 5. Appendixes Appendix A. IBM software portfolio for grid computing IBM Application Workload Modeler IBM Cloudscape/Apache Derby DB2 Connect Family DB2 Everyplace Family DB2 Universal Database Family Mathematical Acceleration Subsystem (MASS) Rational Application Developer for WebSphere Software IBM Tivoli Access Manager Family IBM Tivoli Configuration Manager IBM Tivoli Enterprise Console IBM Tivoli Intelligent Orchestrator IBM Tivoli License Manager The IBM Tivoli Management Framework Contents vii

10 IBM Tivoli Monitoring for Virtual Servers IBM Tivoli OMEGAMON XE Family IBM Tivoli Provisioning Manager IBM Tivoli System Automation for Multiplatforms IBM Tivoli Universal Agent WebSphere Application Server WebSphere Application Server Network Deployment WebSphere Extended Deployment IBM WebSphere MQ WebSphere Studio Application Monitor IBM Director IBM Remote Deployment Manager IBM ServerGuide IBM Virtual Machine Manager Cluster Systems Management Parallel ESSL LoadLeveler General Parallel File System Appendix B. Additional material Locating the Web material Using the Web material System requirements for downloading the Web material How to use the Web material Related publications IBM Redbooks Other publications Online resources How to get IBM Redbooks Help from IBM Index viii Introduction to Grid Computing

11 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. Copyright IBM Corp All rights reserved. ix

12 Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AFS AIX 5L AIX Cloudscape DB2 Connect DB2 Universal Database DB2 developerworks DFS Domino Eserver eserver Everyplace ibm.com IBM iseries LoadLeveler Lotus OMEGAMON Perform pseries Rational Redbooks (logo) Redbooks RS/6000 ServerGuide Summit Tivoli Enterprise Console Tivoli Enterprise Tivoli WebSphere World Community Grid xseries zseries The following terms are trademarks of other companies: Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. x Introduction to Grid Computing

13 Preface In the past several years, grid computing has emerged as a way to harness and take advantage of computing resources across geographies and organizations. In this IBM Redbook, we describe a generalized view of grid computing including concepts, standards, and ways in which grid computing can provide business value to your organization. In a nutshell, grid computing is all about virtualization that enables businesses to take advantage of a variety of IT resources in order to be more responsive to demands of the business and increase availability of applications while reducing both infrastructure and management costs. There are many products and toolkits available from IBM and other companies that enable different aspects of grid computing. One of the most well known toolkits is the Globus Toolkit. Globus Toolkit 4 provides components and services conforming to existing and evolving standards that can be used as the basis for a grid computing solution. In the second half of this book we provide instructions for installing and configuring a simple Globus environment that can be used to demonstrate various aspects of grid computing and to build a proof-of-concept environment. We also describe, and provide as additional material, a sample grid application that can be used to demonstrate, test, and instruct about the grid computing concepts introduced in this book. The team that wrote this redbook This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center. Bart Jacob is a Senior Consulting IT Specialist at IBM Corp - International Technical Support Organization, Austin Center. He has over 25 years of experience providing technical support across a variety of IBM products and technologies, including communications, object-oriented software development, and systems management. He has over 14 years of experience at the ITSO, where he has been writing IBM Redbooks and creating and teaching workshops around the world on a variety of topics. He holds a Masters degree in Numerical Analysis from Syracuse University. Michael Brown is the Technical Project Leader for the Americas and Asia Pacific sites of IBM's Linux Integration Center, headquartered in Austin, Texas. He leads teams that perform technical support for customers who are evaluating IBM software running on Linux platforms. He is a certified Java Programmer, Developer, and Architect and has worked on several previous IBM grid redbooks Copyright IBM Corp All rights reserved. xi

14 and presented on grid computing at the Colorado Software Summit. He holds HBSc and MSc degrees in Computer Science from the University of Western Ontario, Canada. Kentaro Fukui is an IT Specialist for IBM and a Red Hat Certified Engineer working in IBM Global Services, Japan. He has more than two years of experience with grid technologies as well as more than eight years of experience with UNIX -like operating systems, Windows servers, and Lotus Domino servers. He holds a MSc Degree in Information and Computer Science from Keio University, Japan. Currently, he is also working as a PhD candidate student at Keio University. He received the IEEE Computer Society Best Paper Award in Nihar Trivedi is a Consultant and a IBM Grid Technical Sales certified professional working for IBM Business Consulting Services in Australia. Nihar has more than eight years of experience in delivering complex e-business applications in Financial Services, Utility, Government, and Telecommunication industries. Nihar is a PhD student affiliated with the University of Sydney and National ICT Australia. Nihar's main research interests include self-adaptive middleware systems and grid computing. Thanks to the following people for their contributions to this project: Sean Slevin Suguru Hamazaki System Design Center - West, Business Infrastructure Solution, IBM Japan Systems Engineering Julie Czubik International Technical Support Organization, Poughkeepsie Center The team that created a predecessor redbook (Introduction to Grid Computing with Globus, SG ) from which we have reused a wide range of material: Luis Ferreira Viktors Berstis Jonathan Armstrong Mike Kendzierski Andreas Neukoetter Masanobu Takagi Richard Bing-Wo Adeeb Amir Ryo Murakawa Olegario Hernandez James Magowan xii Introduction to Grid Computing

15 Norbert Bieberstein Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html Comments welcome Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an to: redbook@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. JN9B Building Burnet Road Austin, Texas Preface xiii

16 xiv Introduction to Grid Computing

17 Part 1 Part 1 Grid fundamentals Copyright IBM Corp All rights reserved. 1

18 2 Introduction to Grid Computing

19 1 Chapter 1. What grid Computing is Grid computing can mean different things to different individuals. The grand vision is often presented as an analogy to power grids where users (or electrical appliances) get access to electricity through wall sockets with no care or consideration for where or how the electricity is actually generated. In this view of grid computing, computing becomes pervasive and individual users (or client applications) gain access to computing resources (processors, storage, data, applications, and so on) as needed with little or no knowledge of where those resources are located or what the underlying technologies, hardware, operating system, and so on are. Though this vision of grid computing can capture one s imagination and may indeed someday become a reality, there are many technical, business, political, and social issues that need to be addressed. If we consider this vision as an ultimate goal, there are many smaller steps that need to be taken to achieve it. These smaller steps each have benefits of their own. Therefore, grid computing can be seen as a journey along a path of integrating various technologies and solutions that move us closer to the final goal. Its key values are in the underlying distributed computing infrastructure technologies that are evolving in support of cross-organizational application and resource sharing in a word, virtualization virtualization across technologies, platforms, and organizations. This kind of virtualization is only achievable through the use of open standards. Open standards help ensure that applications can transparently take advantage of whatever appropriate resources can be made available to Copyright IBM Corp All rights reserved. 3

20 them. An environment that provides the ability to share and transparently access resources across a distributed and heterogeneous environment not only requires the technology to virtualize certain resources, but also technologies and standards in the areas of scheduling, security, accounting, systems management, and so on. Grid computing could be defined as any of a variety of levels of virtualization along a continuum. Exactly where along that continuum one might say that a particular solution is an implementation of grid computing versus a relatively simple implementation using virtual resources is a matter of opinion. But even at the simplest levels of virtualization, one could say that grid-enabling technologies are being utilized. This continuum is illustrated in Figure 1-1 on page 5. Starting in the lower left you see single system partitioning. Virtualization starts with being able to carve up a machine into virtual machines. As you move up this spectrum you start to be able to virtualize similar or homogeneous resources. Virtualization applies not only to servers and CPUs, but to storage, networks, and even applications. As you move up this spectrum you start to virtualize unlike resources. The next step is virtualizing the enterprise, not just in a data center or within a department but across a distributed organization, and then, finally, virtualizing outside the enterprise, across the Internet, where you might actually access resources from a set of OEMs and their suppliers or you might integrate information across a network of collaborators. 4 Introduction to Grid Computing