ADVANCED COMPUTER ARCHITECTURE: Parallelism, Scalability, Programmability

Transcription

1 ADVANCED COMPUTER ARCHITECTURE: Parallelism, Scalability, Programmability * Technische Hochschule Darmstadt FACHBEREiCH INTORMATIK Kai Hwang Professor of Electrical Engineering and Computer Science University of Southern California., J fl *. * Iriventar-Nr.: Sachgebiete:...< Standort: McGraw-Hill, Inc. New York St Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico Milan Montreal New Delhi Paris San Juan Singapore Sydney Tokyo Toronto

2 Contents Foreword Preface xvii xix PART I THEORY OF PARALLELISM 1 Chapter 1 Parallel Computer Models The State of Computing...! Computer Development Milestones Elements of Modern Computers Evolution of Computer Architecture System Attributes to Performance Multiprocessors and Multicomputer^ Shared-Memory Multiprocessors Distributed-Memory Multicomputers A Taxonomy of MIMD Computers Multivector and SIMD Computers Vector Supercomputers SIMD Supercomputers PRAM and VLSI Models Parallel Random-Access Machines VLSI Complexity Model Architectural Development Tracks Multiple-Processor Tracks Multivector and SIMD Tracks Multithreaded and Dataflow Tracks Bibliographic Notes and Exercises 45 IX

3 : Contents Chapter 2 Program and Network Properties Conditions of Parallelism Data and Resource Dependences Hardware and Software Parallelism The Role of Compilers Program Partitioning and Scheduling Grain Sizes and Latency Grain Packing and Scheduling Static Multiprocessor Scheduling Program Flow Mechanisms ' Control Flow Versus Data Flow Demand-Driven Mechanisms Comparison of Flow Mechanisms System Interconnect Architectures Network Properties and Routing Static Connection Networks.' Dynamic Connection Networks Bibliographic Notes and Exercises 96 Chapter 3 Principles of Scalable Performance Performance Metrics and Measures Parallelism Profile in Programs Harmonic Mean Performance 'Efficiency, Utilization, and Quality Standard Performance Measures Parallel Processing Applications Massive Parallelism for Grand Challenges Application Models of Parallel Computers Scalability of Parallel Algorithms Speedup Performance Laws Amdahl's Law for a Fixed Workload Gustafson's Law for Scaled Problems Memory-Bounded Speedup Model Scalability Analysis and. Approaches Scalability Metrics and Goals Evolution of Scalable Computers Research Issues and Solutions Bibliographic Notes and Exercises 149 PART II HARDWARE TECHNOLOGIES 155

4 Contents xi Chapter 4 Processors and Memory Hierarchy Advanced Processor Technology Design Space of Processors Instruction-Set Architectures CISC Scalar Processors RISC Scalar Processors Superscalar and Vector Processors Superscalar Processors The VLIW Architecture Vector and Symbolic Processors Memory Hierarchy Technology Hierarchical Memory Technology Inclusion, Coherence, and Locality Memory Capacity Planning Virtual Memory Technology Virtual Memory Models TLB, Paging, and Segmentation Memory Replacement Policies Bibliographic Notes and Exercises 208 Chapter 5 Bus, Cache, and Shared Memory Backplane Bus Systems Backplane Bus Specification Addressing and Timing Protocols Arbitration, Transaction, and Interrupt The IEEE Futurebus+ Standards Cache Memory Organizations Cache Addressing Models Direct Mapping and Associative Caches Set-Associative and Sector Caches Cache Performance Issues Shared-Memory Organizations Interleaved Memory Organization Bandwidth and Fault Tolerance' Memory Allocation Schemes Sequential and Weak Consistency Models Atomicity and Event Ordering Sequential Consistency Model Weak Consistency Models Bibliographic Notes and Exercises 256 Chapter 6 Pipelining and Superscalar Techniques 265

5 xii Contents 6.1 Linear Pipeline Processors Asynchronous and Synchronous Models Clocking and Timing Control Speedup, Efficiency, and Throughput Nonlinear Pipeline Processors Reservation and Latency Analysis Collision-Free Scheduling Pipeline Schedule Optimization Instruction Pipeline Design Instruction Execution Phases Mechanisms for Instruction Pipelining Dynamic Instruction Scheduling Branch Handling Techniques Arithmetic Pipeline Design Computer Arithmetic Principles Static Arithmetic Pipelines Multifunctional Arithmetic Pipelines Superscalar and Superpipeline Design Superscalar Pipeline Design Superpipelined Design Supersymmetry and Design Tradeoffs Bibliographic Notes and Exercises 322 PART III PARALLEL AND SCALABLE ARCHITECTURES 329 Chapter 7 Multiprocessors and Multicomputers Multiprocessor System Interconnects ; Hierarchical Bus Systems Crossbar Switch and Multiport Memory, Multistage and Combining Networks Cache Coherence and Synchronization Mechanisms The Cache Coherence Problem Snoopy Bus Protocols Directory-Based Protocols Hardware Synchronization Mechanisms Three Generations of Multicomputers Design Choices in the Past...' Present and Future Development The Intel Paragon System Message-Passing Mechanisms Message-Routing Schemes 375

6 Contents xiii Deadlock and Virtual Channels Flow Control Strategies Multicast Routing Algorithms Bibliographic Notes and Exercises 393 Chapter 8 Multivector and SIMD Computers Vector Processing Principles Vector Instruction Types Vector-Access Memory Schemes Past and Present Supercomputers Multivector Multiprocessors. ' Performance-Directed Design Rules Cray Y-MP, C-90, and MPP Fujitsu VP2000 and VPP Mainframes and Minisupercomputers Compound Vector Processing Compound Vector Operations Vector Loops and Chaining...' Multipipeline Networking SIMD Computer Organizations Implementation Models The CM-2 Architecture The MasPar MP-1 Architecture The Connection Machine CM A Synchronized MIMD Machine The CM-5 Network Architecture Control Processors and Processing Nodes Interprocessor Communications Bibliographic Notes and Exercises 468 Chapter 9 Scalable, Multithreaded, and Dataflow Architectures Latency-Hiding Techniques Shared Virtual Memory Prefetching Techniques Distributed Coherent Caches " Scalable Coherence Interface Relaxed Memory Consistency Principles of Multithreading : Multithreading Issues and Solutions Multiple-Context Processors Multidimensional Architectures Fine-Grain Multicomputers 504

7 xiv Contents Fine-Grain Parallelism The MIT J-Machine The Caltech Mosaic C Scalable and Multithreaded Architectures The Stanford Dash Multiprocessor The Kendall Square Research KSR The Tera Multiprocessor System Dataflow and Hybrid Architectures The Evolution of Dataflow Computers The ETL/EM-4 in Japan ' The MIT/Motorola *T Prototype : Bibliographic Notes and Exercises 539 PART IV SOFTWARE FOR PARALLEL PROGRAMMING 545 Chapter 10 Parallel Models, Languages, and Compilers Parallel Programming Models Shared-Variable Model Message-Passing Model Data-Parallel Model Object-Oriented Model Functional and Logic Models 559, 10.2 Parallel Languages and Compilers Language Features for Parallelism Parallel Language Constructs Optimizing Compilers for Parallelism Dependence Analysis of Data Arrays Iteration Space and Dependence Analysis Subscript Separability and Partitioning Categorized Dependence Tests Code Optimization and Scheduling Scalar Optimization with Basic Blocks Local and Global Optimizations Vectorization and Parallelization Methods Code Generation and Scheduling Trace Scheduling Compilation Loop Parallelization and Pipelining Loop Transformation Theory Parallelization and Wavefronting Tiling and Localization Software Pipelining 610

8 Contents xv 10.6 Bibliographic Notes and Exercises 612 Chapter 11 Parallel Program Development and Environments Parallel Programming Environments Software Tools and Environments Y-MP, Paragon, and CM-5 Environments Visualization and Performance Tuning Synchronization and Multiprocessing Modes Principles of Synchronization Multiprocessor Execution Modes Multitasking on Cray Multiprocessors Shared-Variable Program Structures Locks for Protected Access Semaphores and Applications Monitors and Applications Message-Passing Program Development Distributing the Computation Synchronous Message Passing Asynchronous Message Passing Mapping Programs onto Multicomputers Domain Decomposition Techniques ' Control Decomposition Techniques Heterogeneous Processing Bibliographic Notes and Exercises 661 Chapter 12 UNIX, Mach, and OSF/1 for Parallel Computers Multiprocessor UNIX Design Goals Conventional UNIX Limitations Compatibility and Portability Address Space and Load Balancing Parallel I/O and Network Services Master-Slave and Multithreaded UNIX Master-Slave Kernels Floating-Executive Kernels Multithreaded UNIX Kernel Multicomputer UNIX Extensions Message-Passing OS Models Cosmic Environment and Reactive Kernel Intel NX/2 Kernel and Extensions Mach/OS Kernel Architecture Mach/OS Kernel Functions Multithreaded Multitasking 688

9 xvi ' Contents Message-Based Communications, Virtual Memory Management OSF/1 Architecture and Applications The OSF/1 Architecture The OSF/1 Programming Environment Improving Performance with Threads Bibliographic Notes and Exercises 712 Bibliography 717 Index 739 Answers to Selected Problems 765