Computing. M< MORGAN KAUfMANM. Distributed and Cloud. Processing to the. From Parallel. Internet of Things. Geoffrey C. Fox. Dongarra.

Transcription

1 Distributed and Cloud From Parallel Computing Processing to the Internet of Things Kai Hwang Geoffrey C. Fox Jack J. Dongarra AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO ELSEVIER Morgan Kaufmann is an imprint of Elsevier M< MORGAN KAUfMANM

2 Contents Preface About the Authors Foreword xv xix xxi PART 1 SYSTEMS MODELING, CLUSTERING, AND VISUALIZATION 1 CHAPTER 1 Distributed System Models and Enabling Technologies 3 Summary Scalable Computing over the Internet The Age of Internet Computing Scalable Computing Trends and New Paradigms The Internet of Things and Cyber-Physical Systems Technologies for Network-Based Systems Multicore CPUs and Multithreading Technologies GPU Computing to Exascale and Beyond Memory, Storage, and Wide-Area Networking Virtual Machines and Virtualization Middleware Data Center Virtualization for Cloud Computing System Models for Distributed and Cloud Computing Clusters of Cooperative Computers Grid Computing Infrastructures Peer-to-Peer Network Families Cloud Computing over the Internet Software Environments for Distributed Systems and Clouds Service-Oriented Architecture (SOA) Trends toward Distributed Operating Systems Parallel and Distributed Programming Models Performance, Security, and Energy Efficiency Performance Metrics and Scalability Analysis Fault Tolerance and System Availability Network Threats and Data Integrity Energy Efficiency in Distributed Computing Bibliographic Notes and Homework Problems 55 Acknowledgments 56 References 56 Homework Problems 58 vii

3 viii Contents CHAPTER 2 Computer Clusters for Scalable Parallel Computing 65 Summary Clustering for Massive Parallelism Cluster Development Trends Design Objectives of Computer Clusters Fundamental Cluster Design Issues Analysis of the Top 500 Supercomputers Computer Clusters and MPP Architectures Cluster Organization and Resource Sharing Node Architectures and MPP Packaging Cluster System Interconnects Hardware, Software, and Middleware Support GPU Clusters for Massive Parallelism Design Principles of Computer Clusters Single-System Image Features High Availability through Redundancy Fault-Tolerant Cluster Configurations Checkpointing and Recovery Techniques Cluster Job and Resource Management Cluster Job Scheduling Methods Cluster Job Management Systems Load Sharing Facility (LSF) for Cluster Computing MOSIX: An OS for Linux Clusters and Clouds Case Studies of Top Supercomputer Systems Tianhe-IA: The World Fastest Supercomputer in Cray XT5 Jaguar: The Top Supercomputer in IBM Roadrunner. The Top Supercomputer in Bibliographic Notes and Homework Problems 120 Acknowledgments 121 References 121 Homework Problems 122 CHAPTER 3 Virtual Machines and Visualization of Clusters and Data Centers 129 Summary Implementation Levels of Virtualization Levels of Virtualization Implementation VMM Design Requirements and Providers Virtualization Support at the OS Level Middleware Support for Virtualization Virtualization Structures/Tools and Mechanisms Hypervisor and Xen Architecture Binary Translation with Full Virtualization Para-Virtualization with Compiler Support 143

4 Contents ix 3.3 Virtualization of CPU, Memory, and I/O Devices Hardware Support for Virtualization CPU Virtualization Memory Virtualization I/O Virtualization Virtualization in Multi-Core Processors Virtual Clusters and Resource Management Physical versus Virtual Clusters Live VM Migration Steps and Performance Effects Migration of Memory, Files, and Network Resources Dynamic Deployment of Virtual Clusters Virtualization for Data-Center Automation Server Consolidation in Data Centers Virtual Storage Management Cloud OS for Virtualized Data Centers Trust Management in Virtualized Data Centers Bibliographic Notes and Homework Problems 179 Acknowledgments 179 References 180 Homework Problems 183 PART 2 COMPUTING CLOUDS, SERVICE-ORIENTED ARCHITECTURE, AND PROGRAMMING 189 CHAPTER 4 Cloud Platform Architecture over Virtualized Data Centers 191 Summary Cloud Computing and Service Models Public, Private, and Hybrid Clouds Cloud Ecosystem and Enabling Technologies Infrastructure-as-a-Service (IaaS) Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS) Data-Center Design and Interconnection Networks Warehouse-Scale Data-Center Design Data-Center Interconnection Networks Modular Data Center in Shipping Containers Interconnection of Modular Data Centers Data-Center Management Issues Architectural Design of Compute and Storage Clouds A Generic Cloud Architecture Design Layered Cloud Architectural Development Virtualization Support and Disaster Recovery Architectural Design Challenges 225

5 x Contents 4.4 Public Cloud Platforms: GAE, AWS, and Azure Public Clouds and Service Offerings Google App Engine (GAE) Amazon Web Services (AWS) Microsoft Windows Azure Inter-cloud Resource Management Extended Cloud Computing Services Resource Provisioning and Platform Deployment Virtual Machine Creation and Management Global Exchange of Cloud Resources Cloud Security and Trust Management Cloud Security Defense Strategies Distributed Intrusion/Anomaly Detection Data and Software Protection Techniques Reputation-Guided Protection of Data Centers Bibliographic Notes and Homework Problems 261 Acknowledgements 261 References 261 Homework Problems 265 CHAPTER 5 Service-Oriented Architectures for Distributed Computing 271 Summary Services and Service-Oriented Architecture REST and Systems of Systems Services and Web Services Enterprise Multitier Architecture Grid Services and OGSA Other Service-Oriented Architectures and Systems Message-Oriented Middleware Enterprise Bus Publish-Subscribe Model and Notification Queuing and Messaging Systems Cloud or Grid Middleware Applications Portals and Science Gateways Science Gateway Exemplars HUBzero Platform for Scientific Collaboration Open Gateway Computing Environments (OGCE) Discovery, Registries, Metadata, and Databases UDDI and Service Registries Databases and Publish-Subscribe Metadata Catalogs Semantic Web and Grid Job Execution Environments and Monitoring 312

6 Contents xi 5.5 Workflow in Service-Oriented Architectures Basic Workflow Concepts Workflow Standards Workflow Architecture and Specification Workflow Execution Engine Scripting Workflow System Swift Bibliographic Notes and Homework Problems 322 Acknowledgements 324 References 324 Homework Problems 331 CHAPTER 6 Cloud Programming and Software Environments 335 Summary Features of Cloud and Grid Platforms Cloud Capabilities and Platform Features Traditional Features Common to Grids and Clouds Data Features and Databases Programming and Runtime Support Parallel and Distributed Programming Paradigms Parallel Computing and Programming Paradigms MapReduce, Twister, and Iterative MapReduce Hadoop Library from Apache Dryad and DryadLINQ from Microsoft Sawzall and Pig Latin High-Level Languages Mapping Applications to Parallel and Distributed Systems Programming Support of Google App Engine Programming the Google App Engine Google File System (GFS) BigTable, Google's NOSQL System Chubby, Google's Distributed Lock Service Programming on Amazon AWS and Microsoft Azure Programming on Amazon EC Amazon Simple Storage Service (S3) Amazon Elastic Block Store (EBS) and SimpleDB Microsoft Azure Programming Support Emerging Cloud Software Environments Open Source Eucalyptus and Nimbus ' OpenNebula, Sector/Sphere, and OpenStack Manjrasoft Aneka Cloud and Appliances Bibliographic Notes and Homework Problems 399 Acknowledgement 399 References 399 Homework Problems 405

7 xii Contents PART 3 GRIDS, P2P, AND THE FUTURE INTERNET 413 CHAPTER 7 Grid Computing Systems and Resource Management 415 Summary Grid Architecture and Service Modeling Grid History and Service Families CPU Scavenging and Virtual Supercomputers Open Grid Services Architecture (OGSA) Data-Intensive Grid Service Models Grid Projects and Grid Systems Built National Grids and International Projects NSF TeraGrid in the United States DataGrid in the European Union The ChinaGrid Design Experiences Grid Resource Management and Brokering Resource Management and Job Scheduling Grid Resource Monitoring with CGSP Service Accounting and Economy Model Resource Brokering with Gridbus Software and Middleware for Grid Computing Open Source Grid Middleware Packages The Globus Toolkit Architecture (GT4) Containers and Resources/Data Management The ChinaGrid Support Platform (CGSP) Grid Application Trends and Security Measures Grid Applications and Technology Fusion Grid Workload and Performance Prediction Trust Models for Grid Security Enforcement Authentication and Authorization Methods Grid Security Infrastructure (GSI) Bibliographic Notes and Homework Problems 470 Acknowledgments 471 References < 471 Homework Problems 473 CHAPTER 8 Peer-to-Peer Computing and Overlay Networks 479 Summary Peer-to-Peer Computing Systems Basic Concepts of P2P Computing Systems Fundamental Challenges in P2P Computing Taxonomy of P2P Network Systems P2P Overlay Networks and Properties Unstructured P2P Overlay Networks 492

8 Contents xiii Distributed Hash Tables (DHTs) Structured P2P Overlay Networks Hierarchically Structured Overlay Networks Routing, Proximity, and Fault Tolerance Routing in P2P Overlay Networks Network Proximity in P2P Overlays Fault Tolerance and Failure Recovery Churn Resilience against Failures Trust, Reputation, and Security Management Peer Trust and Reputation Systems Trust Overlay and DHT Implementation PowerTrust: A Scalable Reputation System Securing Overlays to Prevent DDoS Attacks P2P File Sharing and Copyright Protection Fast Search, Replica, and Consistency P2P Content Delivery Networks Copyright Protection Issues and Solutions Collusive Piracy Prevention in P2P Networks Bibliographic Notes and Homework Problems 538 Acknowledgements 538 References 539 Homework Problems 541 CHAPTER 9 Ubiquitous Clouds and the Internet of Things 545 Summary Cloud Trends in Supporting Ubiquitous Computing Use of Clouds for HPC/HTC and Ubiquitous Computing Large-Scale Private Clouds at NASA and CERN Cloud Mashups for Agility and Scalability Cloudlets for Mobile Cloud Computing Performance of Distributed Systems and the Cloud Review of Science and Research Clouds Data-Intensive Scalable Computing (DISC) Performance Metrics for HPC/HTC Systems Quality of Service in Cloud Computing Benchmarking MPI, Azure, EC2, MapReduce, and Hadoop Enabling Technologies for the Internet of Things The Internet of Things for Ubiquitous Computing Radio-Frequency Identification (RFID) Sensor Networks and ZigBee Technology Global Positioning System (GPS) Innovative Applications of the Internet of Things Applications of the Internet of Things 591

9 xiv Contents Retailing and Supply-Chain Management Smart Power Grid and Smart Buildings Cyber-Physical System (CPS) Online Social and Professional Networking Online Social Networking Characteristics Graph-Theoretic Analysis of Social Networks Communities and Applications of Social Networks Facebook: The World's Largest Social Network Twitter for Microblogging, News, and Alert Services Bibliographic Notes and Homework Problems 614 Acknowledgements 614 References 614 Homework Problems 618 Index 623